Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabled high-performance Automatic Tensor Parallelism (auto TP) for the MoE models on multiple GPUs/HPUs #6964

Open
wants to merge 64 commits into
base: master
Choose a base branch
from
Open

Enabled high-performance Automatic Tensor Parallelism (auto TP) for the MoE models on multiple GPUs/HPUs #6964

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
c9b12af
Reduced the experts allreduce number per layer to ONCE for the Qwen2-…
gyou2021 Jan 21, 2025
590ea36
Fixed format
gyou2021 Jan 21, 2025
889c275
Removed print
gyou2021 Jan 21, 2025
2ec6c34
Fix a bug about set.
gyou2021 Jan 21, 2025
504d696
Add the missing view operations from sequence parallel(async). (#6750)
inkcherry Jan 21, 2025
c266dc9
Update `torch.norm` to `torch.linalg.norm` and `torch.linalg.vector_n…
loadams Jan 21, 2025
ae12993
Using explicit GPU upcast for ZeRO-Offload (#6962)
xylian86 Jan 21, 2025
deb09a3
Update version.txt after 0.16.3 release (#6965)
loadams Jan 21, 2025
128d436
Precisely track nvme optimizer offload (#6963)
tjruwase Jan 23, 2025
864472b
Update build_win.bat script to exclue GDS op as it lacks Windows supp…
loadams Jan 24, 2025
1ac398c
Add CUDA 12.8 support and comment on CUDA 12.7 (#6975)
loadams Jan 28, 2025
eda53d8
Update torch versions to support 2.6 (#6977)
loadams Jan 29, 2025
112a7c6
generalize deepspeed linear and implement it for non cuda systems (#6…
oelayan7 Jan 29, 2025
7d2c5fe
Update recommended Windows whl building versions (#6983)
loadams Jan 30, 2025
f1d326c
Title: Fix setup_env_ranks to Properly Set Environment Variables Inst…
fabiosanger Jan 30, 2025
46545d7
Specify torchvision in nv-ds-chat workflow (prevents errors with torc…
loadams Jan 30, 2025
af1ba94
Remove assumption that padding only occurs on last rank (#6974)
xylian86 Jan 31, 2025
e235921
Use ds-specific module id to avoid conflicts (#6847)
tjruwase Jan 31, 2025
f5e9796
Update A6000 workflows to use newer docker container - 24.09 vs 24.03…
loadams Jan 31, 2025
07634b9
Allow NVIDIA Blackwell (#6991)
fabiendupont Feb 4, 2025
0e57fa0
Update GH org references (#6998)
tjruwase Feb 5, 2025
e86c0c3
Update CNAME
loadams Feb 5, 2025
0d7f0eb
Update CNAME
loadams Feb 5, 2025
cd8a988
[XPU] max1100 workflow update for docker and softwares (#7003)
Liangliang-Ma Feb 5, 2025
18c712f
autotp training(fix dco) (#7004)
inkcherry Feb 5, 2025
c5bf6f6
import triton files when triton is supported and installed (#6989)
oelayan7 Feb 6, 2025
590de5f
Update A6000 tests transformers version (#7016)
loadams Feb 8, 2025
693c39f
Fix ds-chat CI regression (#7015)
tjruwase Feb 10, 2025
322a05a
[Ulysses tutorial] typos (#7024)
stas00 Feb 11, 2025
8869d78
fix hostname -I for macOS #6497 (#6990)
fitzjalen Feb 12, 2025
e4d03af
Update workflows to cuda 12.4 (#7000)
loadams Feb 12, 2025
8c6251d
[ROCm] Enable fp_quantizer on ROCm (#7027)
rraminen Feb 13, 2025
e3e179c
add gds chinese blog (#7034)
GuanhuaWang Feb 13, 2025
fd2787b
Add chinese blog for deepspeed windows, and fix format (#7035)
hwchen2017 Feb 14, 2025
ba8ef57
AIO on ROCM (#7023)
jomayeri Feb 14, 2025
f4b0f58
Control trace cache warnings (#7039)
tjruwase Feb 18, 2025
3ca3e2f
Update CUDA compute capability to support Blackwell (#7047)
hwchen2017 Feb 18, 2025
5612778
Update setup.py handling of ROCm cupy (#7051)
loadams Feb 19, 2025
af8c190
nv-ds-chat breaks with latest transformers (#7052)
loadams Feb 19, 2025
225471a
Rename aio_thread_count to intra_op_parallelism (#7056)
tjruwase Feb 19, 2025
1df293a
add autoTP training zero2 tests (#7049)
inkcherry Feb 19, 2025
94abf68
Fix, bf16 optimizer remove dup loop (#7054)
wukong1992 Feb 20, 2025
4a4ff9b
Update version.txt after 0.16.4 release (#7063)
loadams Feb 20, 2025
e5eda47
fix an outdated doc wrt CUDA_VISIBLE_DEVICES (#7058)
stas00 Feb 20, 2025
675ec9a
Tecorigin sdaa accelerator (#6903)
siqi654321 Feb 20, 2025
81c1fee
Handle special case of libuv for Windows (#7064)
loadams Feb 20, 2025
17f544c
Update README with info on newest accelerator (#7065)
loadams Feb 21, 2025
20fd872
Bug Fix for offload_states API (#7050)
U-rara Feb 21, 2025
0b289a2
Fix TOCTOU issues, switch to fstat (#7067)
loadams Feb 24, 2025
4a86d02
config torch to avoid graph breaks caused by logger (#6999)
ShellyNR Feb 24, 2025
594b5bb
Fix meta load tensor imcompatible issue (#7073)
Yejing-Lai Feb 24, 2025
a843e39
Replace calls to `python setup.py sdist` with `python -m build --sdis…
loadams Feb 24, 2025
4cbc52c
Revert "Handle special case of libuv for Windows (#7064)" (#7076)
loadams Feb 25, 2025
586e436
Add DeepseekV3 AutoTP. (#7045)
Yejing-Lai Feb 26, 2025
5e379ad
Improve inference tutorial docs (#7083)
loadams Feb 26, 2025
13bf866
Added support for the environment variable DS_MOE_EXPERTS_REDUCE_ONCE…
gyou2021 Feb 27, 2025
d5115be
Changed env variable name to 'DS_MOE_TP_SINGLE_ALLREDUCE'
gyou2021 Feb 28, 2025
f0044cb
Pin transformers version on tests that use latest. (#7085)
loadams Feb 27, 2025
16ad5fd
Update README.md with ICS '23 MoE paper link (#7087)
siddharth9820 Feb 27, 2025
47d4420
Update parallelism for nv-torch-latest/nightly tests due to more GPUs…
loadams Feb 27, 2025
b3c64dd
Remove workflows for very old torch versions (#7090)
loadams Feb 28, 2025
9b1fe98
Fixed conflicts
gyou2021 Feb 28, 2025
6b96dd9
Update auto_tp.py
gyou2021 Mar 5, 2025
e7883e7
Merge branch 'master' into autoTP_Qwen2Moe_DeepSeekv2
hwchen2017 Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update A6000 workflows to use newer docker container - 24.09 vs 24.03 (… 
…#6967)

- Issues with nv-sd updates, will follow up with a subsequent PR

Signed-off-by: gyou2021 <[email protected]>
  • Loading branch information
@loadams @gyou2021
loadams authored and gyou2021 committed Feb 28, 2025
commit f5e97963fd52e717ee7aa3c149c5ac123846c6f3
6 changes: 3 additions & 3 deletions .github/workflows/nv-a6000.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
unit-tests:
runs-on: [self-hosted, nvidia, a6000]
container:
image: nvcr.io/nvidia/pytorch:24.03-py3
image: nvcr.io/nvidia/pytorch:24.09-py3
ports:
- 80
options: --gpus all --shm-size "8G"
Expand Down Expand Up @@ -57,8 +57,8 @@ jobs:
run: |
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
cd tests
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2' unit/ --torch_ver="2.3" --cuda_ver="12"
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2_ops' unit/ --torch_ver="2.3" --cuda_ver="12"
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2' unit/ --torch_ver="2.5" --cuda_ver="12"
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2_ops' unit/ --torch_ver="2.5" --cuda_ver="12"
- name: MII unit tests
run: |
BRANCH="main"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/nv-flash-attn.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
unit-tests:
runs-on: [self-hosted, nvidia, a6000]
container:
image: nvcr.io/nvidia/pytorch:24.03-py3
image: nvcr.io/nvidia/pytorch:24.09-py3
ports:
- 80
options: --gpus all --shm-size "8G"
Expand Down Expand Up @@ -53,7 +53,7 @@ jobs:
run: |
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
cd tests
python -m pytest --color=yes --durations=0 --verbose -rF unit/sequence_parallelism/test_ulysses.py --torch_ver="2.3" --cuda_ver="12"
python -m pytest --color=yes --durations=0 --verbose -rF unit/sequence_parallelism/test_ulysses.py --torch_ver="2.5" --cuda_ver="12"
- name: Open GitHub issue if nightly CI fails
if: ${{ failure() && (github.event_name == 'schedule') }}
uses: JasonEtco/create-an-issue@v2
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/nv-human-eval.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
unit-tests:
runs-on: [self-hosted, nvidia, a6000]
container:
image: nvcr.io/nvidia/pytorch:24.03-py3
image: nvcr.io/nvidia/pytorch:24.09-py3
ports:
- 80
options: --gpus all --shm-size "8G"
Expand Down Expand Up @@ -50,4 +50,4 @@ jobs:
run: |
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
cd tests
python -m pytest --color=yes --durations=0 --verbose -rF -m 'evaluation' -k "test_human_eval" unit/ --torch_ver="2.3" --cuda_ver="12"
python -m pytest --color=yes --durations=0 --verbose -rF -m 'evaluation' -k "test_human_eval" unit/ --torch_ver="2.5" --cuda_ver="12"