Create training server AMI

Source: Notion | Last edited: 2024-04-25 | ID: d214ff74-ca4...

---

Use AWS to train

创建 experiment EC2

使用 AMI： Experiment training server 系列的

然后用multiple gpu的EC2 - 比如 g6.12xLarge

SSH

ssh -i "~/.ssh/el-prod-pred.pem" ubuntu@ec2-34-221-35-39.us-west-2.compute.amazonaws.com

本地运行 dev container

export deepTradeDir=DeepTrade
export imageName=eonlabsteam/el-nigma

containerName=training && cd ~/Repositories/${deepTradeDir}/el-nigma && sudo docker container run --net=host --gpus device=0 -it --name ${containerName} -h ${containerName} -v $(pwd):/el-nigma -v ~/.aws:/root/.aws ${imageName}:aws-linux-amd64

# 如果想map所有GPU到 docker container，用 --gpus all

使用script

export imageName=eonlabsteam/ml-tf1

or

export imageName=eonlabsteam/ml-tf2

export deepTradeDir=DeepTrade0

vi run.sh

https://raw.githubusercontent.com/ChenLi0830/DeepTrade-ZL/sonata-tf2-v3/setup_training_server_ubuntu.sh

https://github.com/ChenLi0830/DeepTrade-ZL/blob/master/setup_training_server_ubuntu.sh

chmod +x ./run.sh

./run.sh

输入git credentials

ChenLi0830

多个GPU设置（添加新的Docker container)

export deepTradeDir=DeepTrade1

or

export deepTradeDir=DeepTrade2

or

export deepTradeDir=DeepTrade3

~/run.sh

设置branch

if [ “$imageName” == “eonlabsteam/ml-tf1” ]; then branch=“sonata-v3”; else branch=“sonata-tf2-v3”; fi

cd ~/Repositories/${deepTradeDir}/DeepTrade-ZL/

git checkout $branch && git pull

运行container

containerName=training-{deepTradeDir} && cd ~/Repositories/{deepTradeDir}/DeepTrade-ZL && sudo docker container run —net=host —gpus device=${deepTradeDir: -1} --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -it --name${containerName} -h ${containerName} -v$(pwd):/DeepTrade-ZL -v $(pwd)/../Deeptrade-Dataset:/Deeptrade-Dataset -v ~/.aws/credentials:/root/.aws/credentials${imageName}:aws-linux-amd64

开始训练

sudo docker container attach training-${deepTradeDir}

tmux new -s training

export AWS_PROFILE=el-prod

cd /el-nigma/

python3 continuous_training.py —exchanges binance coinbasepro bitstamp —data_folder_base=../Deeptrade-Dataset  —pred_intervals 2h 15m 20m 25m 30m 35m 40m 45m 50m 1h 70m 80m 90m —symbols BTC ETH

python3 continuous_training.py —exchanges binance coinbasepro bitstamp —data_folder_base=../Deeptrade-Dataset  —pred_intervals 30m —symbols BTC ETH

python3 continuous_training.py —exchanges binance coinbasepro bitstamp —data_folder_base=../Deeptrade-Dataset  —pred_intervals 45m —symbols BTC ETH

python3 continuous_training.py —exchanges binance coinbasepro bitstamp —data_folder_base=../Deeptrade-Dataset  —pred_intervals 1h —symbols BTC ETH

python3 continuous_training.py —exchanges binance coinbasepro bitstamp —data_folder_base=../Deeptrade-Dataset  —pred_intervals 2h 90m —symbols BTC ETH

多个GPU设置（添加新的Docker container)

之后重复设置Branch，运行docker，和开始训练的步骤：

if [ “$imageName” == “eonlabsteam/ml-tf1” ]; then branch=“sonata-v3”; else branch=“sonata-tf2-v3”; fi

cd ~/Repositories/${deepTradeDir}/DeepTrade-ZL/

git checkout $branch && git pull