Awesome

GPT-home-private

Setup:

Use AWS Deep Learning Base AMI

Install PyTorch env (old):

pip3 install torch==1.9.0+cu111 torchtext -f https://download.pytorch.org/whl/torch_stable.html
pip3 install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch==1.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

# Magic, not sure why cupy-cuda111 would not work, it seems that cupy-cuda111 will use different PTX from torch.
pip3 install cupy-cuda111==8.6.0
pip3 install transformers

Install PyTorch env (latest):

pip3 install --pre torch==1.12.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install cupy-cuda11x==11.0.0
python3 -m cupyx.tools.install_library --cuda 11.x --library nccl
pip3 install transformers

Install PyTorch env (CPU-latest):

pip3 install --pre torch==1.12.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Install deepspeed for some micro-benchmark (optional)
```
pip install deepspeed
```

Clone this repo:

git clone https://github.com/BinhangYuan/GPT-home-private.git

set the github cache (Optional):

git config credential.helper 'cache --timeout=30000'

Download a tiny dataset:

wget https://binhang-language-datasets.s3.us-west-2.amazonaws.com/glue_qqp_dataset/data.tar.xz -P ./glue_dataset/

tar -xvf ./glue_dataset/data.tar.xz -C ./glue_dataset/

Setup network configuration:

export GLOO_SOCKET_IFNAME=ens3

export NCCL_SOCKET_IFNAME=ens3

Use TC scripts to control network delay and bandwidth:

Run Distributed Gpipe:

On each node, run:

python dist_pipeline_runner.py --dist-url tcp://XXX.XXX.XXX.XXX:9000 --world-size N --rank i (i=0,...,N-1)

Run deepspeed benchmark:

Update public-ip and hostname in the ./scripts/ip_list.sh file
Update the host name with slots (number of GPUs) in ./scripts/ds_hostnames.sh file
Sync code to all nodes
Setup password free ssh cluster by executing (under the ./script/ directory):
```
bash ssh_pass_free.sh
```
SSH to the rank-0 node, on that node run:
```
source activate pytorch_p38
```

A sample run:

deepspeed --hostfile=./scripts/ds_hostnames dist_deepspeed_zero_s3.py --embedding-dim 2048 --seq-length 2048 --batch-size 1024 --num-layers 40 --micro-batch-size 4

Batch run all settings:

 cd ./scripts
 bash local_run_deepspeed_batch_on_rank0.sh #CASE

Run with Advanced Scripts (under scripts directory):

First update the public IPs and private IP of the rank-0 node in ip_list.sh.
Allow SSH connects:
```
bash accept_ssh_keys.sh
```
Update local repository:
```
bash aws_sync_code.sh #GIT_TOKEN
```
Enable environment: (This is optional but load conda env seems to be slow for the first time)
```
bash aws_foo_load_lib.sh
```
Setup heterogeneous network (update the private IPs in ./scheduler/generate_heterogeneous_tc.py, sync the code to AWS!):
```
bash aws_generate_heter_tc.sh #HETER_CASE (3/4/5)
```
Optional(Play with it by starting TC, this is not needed if you use the next block of cmds to start benchmarks):
```
bash aws_start_heter_tc.sh #HETER_CASE (3/4/5)
```
Run Schedulers (under scheduler/heuristic_evolutionary_solver directory) to get assignments and estimated cost
```
python scheduler.py
```
Update correspond aws_run_gpt3_*_training.sh with the above output:

Run Tasks (e.g.,):

bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh
bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh #HETER_CASE
bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh #DELAY #BANDWIDTH

Clear logs:
```
bash aws_clear_logs.sh
```
Copy training logs from Rank-0 node (For my implementation the benchmark result is on the rank-0 node.)
```
bash copy_rank0_logs.sh
```

Download and generate trace:

bash copy_traces.sh #PREFIX
bash generate_traces.sh #PREFIX

Run the System on Euler with Coordinator

First log in to ETH Zurich Euler cluster (need to use VPN if not on campus), our working directory:
```
/nfs/iiscratch-zhang.inf.ethz.ch/export/zhang/export/fm/
```
Get the Euler ssh client's node IP, and start the coordinator server (under ./coordinator directory), e.g.:
```
python coordinate_server.py --coordinator-server-ip 129.132.93.88 --coordinator-type inference/train
```

Alternatively, we can use the following script to submit multiple jobs:

bash multi_inference_jobs_submit.sh 129.132.93.88 lsf_gptJ_inf_4RTX2080Ti 3 10

On a different terminal, start a job submission client to submit a job:

python job_submit_client.py --coordinator-server-ip 129.132.93.88 --submit-job inference --job-name lsf_gptJ_inf_4RTX2080Ti

So far, we need to manually change the IP in the job-submit templates, e.g. change lsf_gptJ_inf_4gpu.bsub file.
Check submitted job states:
```
bjobs
```