Set weights_only=False in torch.load in EMA callback and AdapterMixin #12198
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: chtruong814 <[email protected]>
|
Tests passed for EMA here. Checking others. |
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: chtruong814 <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: chtruong814 <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
|
adapter mixin tests are passing now. Looks like another location we need to pass in weights_only for rest of the expmanager tests. Will hunt for that to just get it all in for this PR. |
…ch.load Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: chtruong814 <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
|
All the tests passed here Will go ahead and merge this now into the other "bump" branch we have. We'll open a PR for the accumulated fixes next week into main branch and cherry-pick to release branch. |
chtruong814
merged commit f28e055
into
ko3n1g/build/bump-pyt-25.01
212 of 213 checks passed
nithinraok
pushed a commit
that referenced
this pull request
Feb 17, 2025
…#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]>
nithinraok
added a commit
that referenced
this pull request
Feb 18, 2025
* fix_5089931 Signed-off-by: Nithin Rao Koluguri <nithinraok> * Set weights_only=False in torch.load in EMA callback and AdapterMixin (#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]> * removed Confidense ensembles notebook Signed-off-by: Nithin Rao Koluguri <nithinraok> --------- Signed-off-by: Nithin Rao Koluguri <nithinraok> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: chtruong814 <[email protected]>
chtruong814
added a commit
that referenced
this pull request
Feb 21, 2025
* f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * try to fix checkpoint loading Signed-off-by: Chen Cui <[email protected]> * update ci checkpoint Signed-off-by: Chen Cui <[email protected]> * fix ckpt loading for nemo1 and t5 tests Signed-off-by: Chen Cui <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * bump Signed-off-by: oliver könig <[email protected]> * Rename neva datamodule (#12121) * Rename dataset Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update Signed-off-by: yaoyu-33 <[email protected]> * pylink Signed-off-by: yaoyu-33 <[email protected]> * fix f string Signed-off-by: yaoyu-33 <[email protected]> * fix intern vit default factory Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> * tests: Run FSDP2 on dual-gpu (#12145) Signed-off-by: oliver könig <[email protected]> * fix cmd Signed-off-by: Chen Cui <[email protected]> * Revert "build: Force re-install VCS dependencies (#12155)" (#12163) This reverts commit 4b19ade. Signed-off-by: Charlie Truong <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove unsed variable Signed-off-by: dimapihtar <[email protected]> * Fix nsys callback tests (#12177) * Fix nsys callback tests Signed-off-by: Hemil Desai <[email protected]> * Simplify tests Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> --------- Signed-off-by: Hemil Desai <[email protected]> * Fixes for bumping pyt to 25.01 (#12165) * fix attention impl config Signed-off-by: Maanu Grover <[email protected]> * safeguard against empty key added by mcore Signed-off-by: Maanu Grover <[email protected]> * docu Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused import Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> * Use pip --no-deps --force-reinstall when building the test container (#12175) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load in EMA callback and AdapterMixin (#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]> * Fix nemo-run stdin exception (#12197) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> * mock invoke context Signed-off-by: Maanu Grover <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * tweak Signed-off-by: Maanu Grover <[email protected]> * more patches needed Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Co-authored-by: Charlie Truong <[email protected]> * Add legacy_ckpt arg to scripts/llm/gpt_distillation.py Signed-off-by: Charlie Truong <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" (#12202) This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * Ckpt fixes pytorch update (#12228) * Fix checkpoint loading for ptq tests Signed-off-by: Charlie Truong <[email protected]> * Pass --ckpt_load_strictness log_all to L2_NeMo_2_SSM_Finetuning test Signed-off-by: Charlie Truong <[email protected]> * Fix tests in L2_NMT_Attention_is_All_You_Need_Finetuning by using default_factory Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * bump modelopt Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 (#12265) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 Signed-off-by: Piotr Żelasko <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Co-authored-by: oliver könig <[email protected]> * remove retro test Signed-off-by: oliver könig <[email protected]> * L2_VLM_HF_Transformer_SFT_FSDP2 Signed-off-by: oliver könig <[email protected]> * Fix distillation state-dict loading bug (#12270) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Add default param dtype in mistral configs (#12186) Signed-off-by: Alexandros Koumparoulis <[email protected]> * ci: Fix pypi link of dry-run (#12267) Signed-off-by: oliver könig <[email protected]> * Fix tiny missing asterisk bug Signed-off-by: Asha Anoosheh <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Add optimizer fix (#12253) * Add optimizer fix Signed-off-by: Boxiang Wang <[email protected]> * Update hf_auto_model_for_causal_lm.py Signed-off-by: BoxiangW <[email protected]> --------- Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> * num gpus Signed-off-by: oliver könig <[email protected]> * Build bitsandbytes (#12279) Co-authored-by: Charlie Truong <[email protected]> * Fixing error when loading T5 checkpoint created with TE<1.13 (#12264) * fix error when loading checkpoint created with TE<1.13 * fix formatting * fix formatting * fix formatting * Apply isort and black reformatting Signed-off-by: huvunvidia <[email protected]> --------- Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> * no cancel for manual Signed-off-by: oliver könig <[email protected]> * Set L2_Speech_Batch_Size_OOMptimizer_Canary to be optional (#12299) Signed-off-by: Charlie Truong <[email protected]> * make expressions r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove parallelize_fn Signed-off-by: Alexandros Koumparoulis <[email protected]> * rstring Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: chtruong814 <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> Signed-off-by: huvunvidia <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Hemil Desai <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: chtruong814 <[email protected]> Co-authored-by: Piotr Żelasko <[email protected]> Co-authored-by: Asha Anoosheh <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: BoxiangW <[email protected]> Co-authored-by: Huy Vu <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]>
ko3n1g
added a commit
that referenced
this pull request
Feb 22, 2025
* f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * try to fix checkpoint loading Signed-off-by: Chen Cui <[email protected]> * update ci checkpoint Signed-off-by: Chen Cui <[email protected]> * fix ckpt loading for nemo1 and t5 tests Signed-off-by: Chen Cui <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * bump Signed-off-by: oliver könig <[email protected]> * Rename neva datamodule (#12121) * Rename dataset Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update Signed-off-by: yaoyu-33 <[email protected]> * pylink Signed-off-by: yaoyu-33 <[email protected]> * fix f string Signed-off-by: yaoyu-33 <[email protected]> * fix intern vit default factory Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> * tests: Run FSDP2 on dual-gpu (#12145) Signed-off-by: oliver könig <[email protected]> * fix cmd Signed-off-by: Chen Cui <[email protected]> * Revert "build: Force re-install VCS dependencies (#12155)" (#12163) This reverts commit 4b19ade. Signed-off-by: Charlie Truong <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove unsed variable Signed-off-by: dimapihtar <[email protected]> * Fix nsys callback tests (#12177) * Fix nsys callback tests Signed-off-by: Hemil Desai <[email protected]> * Simplify tests Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> --------- Signed-off-by: Hemil Desai <[email protected]> * Fixes for bumping pyt to 25.01 (#12165) * fix attention impl config Signed-off-by: Maanu Grover <[email protected]> * safeguard against empty key added by mcore Signed-off-by: Maanu Grover <[email protected]> * docu Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused import Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> * Use pip --no-deps --force-reinstall when building the test container (#12175) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load in EMA callback and AdapterMixin (#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]> * Fix nemo-run stdin exception (#12197) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> * mock invoke context Signed-off-by: Maanu Grover <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * tweak Signed-off-by: Maanu Grover <[email protected]> * more patches needed Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Co-authored-by: Charlie Truong <[email protected]> * Add legacy_ckpt arg to scripts/llm/gpt_distillation.py Signed-off-by: Charlie Truong <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" (#12202) This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * Ckpt fixes pytorch update (#12228) * Fix checkpoint loading for ptq tests Signed-off-by: Charlie Truong <[email protected]> * Pass --ckpt_load_strictness log_all to L2_NeMo_2_SSM_Finetuning test Signed-off-by: Charlie Truong <[email protected]> * Fix tests in L2_NMT_Attention_is_All_You_Need_Finetuning by using default_factory Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * bump modelopt Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 (#12265) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 Signed-off-by: Piotr Żelasko <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Co-authored-by: oliver könig <[email protected]> * remove retro test Signed-off-by: oliver könig <[email protected]> * L2_VLM_HF_Transformer_SFT_FSDP2 Signed-off-by: oliver könig <[email protected]> * Fix distillation state-dict loading bug (#12270) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Add default param dtype in mistral configs (#12186) Signed-off-by: Alexandros Koumparoulis <[email protected]> * ci: Fix pypi link of dry-run (#12267) Signed-off-by: oliver könig <[email protected]> * Fix tiny missing asterisk bug Signed-off-by: Asha Anoosheh <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Add optimizer fix (#12253) * Add optimizer fix Signed-off-by: Boxiang Wang <[email protected]> * Update hf_auto_model_for_causal_lm.py Signed-off-by: BoxiangW <[email protected]> --------- Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> * num gpus Signed-off-by: oliver könig <[email protected]> * Build bitsandbytes (#12279) Co-authored-by: Charlie Truong <[email protected]> * Fixing error when loading T5 checkpoint created with TE<1.13 (#12264) * fix error when loading checkpoint created with TE<1.13 * fix formatting * fix formatting * fix formatting * Apply isort and black reformatting Signed-off-by: huvunvidia <[email protected]> --------- Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> * no cancel for manual Signed-off-by: oliver könig <[email protected]> * Set L2_Speech_Batch_Size_OOMptimizer_Canary to be optional (#12299) Signed-off-by: Charlie Truong <[email protected]> * make expressions r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove parallelize_fn Signed-off-by: Alexandros Koumparoulis <[email protected]> * rstring Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * reduce grad-acc-steps Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * update tests Signed-off-by: Alexandros Koumparoulis <[email protected]> * force-switch from MegatronCheckpointIO to HFCheckpointIO Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix setup_environment call Signed-off-by: Alexandros Koumparoulis <[email protected]> * update test Signed-off-by: Alexandros Koumparoulis <[email protected]> * update verify_sft_checkpoint_structure to handle sharded checkpoints Signed-off-by: Alexandros Koumparoulis <[email protected]> * make io_bytes.pt optional Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * set broken tests to optional Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: chtruong814 <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Hemil Desai <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: chtruong814 <[email protected]> Co-authored-by: Piotr Żelasko <[email protected]> Co-authored-by: Asha Anoosheh <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: BoxiangW <[email protected]> Co-authored-by: Huy Vu <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]>
ko3n1g
added a commit
that referenced
this pull request
Feb 22, 2025
* f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * try to fix checkpoint loading Signed-off-by: Chen Cui <[email protected]> * update ci checkpoint Signed-off-by: Chen Cui <[email protected]> * fix ckpt loading for nemo1 and t5 tests Signed-off-by: Chen Cui <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * bump Signed-off-by: oliver könig <[email protected]> * Rename neva datamodule (#12121) * Rename dataset Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update Signed-off-by: yaoyu-33 <[email protected]> * pylink Signed-off-by: yaoyu-33 <[email protected]> * fix f string Signed-off-by: yaoyu-33 <[email protected]> * fix intern vit default factory Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> * tests: Run FSDP2 on dual-gpu (#12145) Signed-off-by: oliver könig <[email protected]> * fix cmd Signed-off-by: Chen Cui <[email protected]> * Revert "build: Force re-install VCS dependencies (#12155)" (#12163) This reverts commit 4b19ade. Signed-off-by: Charlie Truong <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove unsed variable Signed-off-by: dimapihtar <[email protected]> * Fix nsys callback tests (#12177) * Fix nsys callback tests Signed-off-by: Hemil Desai <[email protected]> * Simplify tests Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> --------- Signed-off-by: Hemil Desai <[email protected]> * Fixes for bumping pyt to 25.01 (#12165) * fix attention impl config Signed-off-by: Maanu Grover <[email protected]> * safeguard against empty key added by mcore Signed-off-by: Maanu Grover <[email protected]> * docu Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused import Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> * Use pip --no-deps --force-reinstall when building the test container (#12175) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load in EMA callback and AdapterMixin (#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]> * Fix nemo-run stdin exception (#12197) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> * mock invoke context Signed-off-by: Maanu Grover <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * tweak Signed-off-by: Maanu Grover <[email protected]> * more patches needed Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Co-authored-by: Charlie Truong <[email protected]> * Add legacy_ckpt arg to scripts/llm/gpt_distillation.py Signed-off-by: Charlie Truong <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" (#12202) This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * Ckpt fixes pytorch update (#12228) * Fix checkpoint loading for ptq tests Signed-off-by: Charlie Truong <[email protected]> * Pass --ckpt_load_strictness log_all to L2_NeMo_2_SSM_Finetuning test Signed-off-by: Charlie Truong <[email protected]> * Fix tests in L2_NMT_Attention_is_All_You_Need_Finetuning by using default_factory Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * bump modelopt Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 (#12265) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 Signed-off-by: Piotr Żelasko <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Co-authored-by: oliver könig <[email protected]> * remove retro test Signed-off-by: oliver könig <[email protected]> * L2_VLM_HF_Transformer_SFT_FSDP2 Signed-off-by: oliver könig <[email protected]> * Fix distillation state-dict loading bug (#12270) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Add default param dtype in mistral configs (#12186) Signed-off-by: Alexandros Koumparoulis <[email protected]> * ci: Fix pypi link of dry-run (#12267) Signed-off-by: oliver könig <[email protected]> * Fix tiny missing asterisk bug Signed-off-by: Asha Anoosheh <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Add optimizer fix (#12253) * Add optimizer fix Signed-off-by: Boxiang Wang <[email protected]> * Update hf_auto_model_for_causal_lm.py Signed-off-by: BoxiangW <[email protected]> --------- Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> * num gpus Signed-off-by: oliver könig <[email protected]> * Build bitsandbytes (#12279) Co-authored-by: Charlie Truong <[email protected]> * Fixing error when loading T5 checkpoint created with TE<1.13 (#12264) * fix error when loading checkpoint created with TE<1.13 * fix formatting * fix formatting * fix formatting * Apply isort and black reformatting Signed-off-by: huvunvidia <[email protected]> --------- Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> * no cancel for manual Signed-off-by: oliver könig <[email protected]> * Set L2_Speech_Batch_Size_OOMptimizer_Canary to be optional (#12299) Signed-off-by: Charlie Truong <[email protected]> * make expressions r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove parallelize_fn Signed-off-by: Alexandros Koumparoulis <[email protected]> * rstring Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * reduce grad-acc-steps Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * update tests Signed-off-by: Alexandros Koumparoulis <[email protected]> * force-switch from MegatronCheckpointIO to HFCheckpointIO Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix setup_environment call Signed-off-by: Alexandros Koumparoulis <[email protected]> * update test Signed-off-by: Alexandros Koumparoulis <[email protected]> * update verify_sft_checkpoint_structure to handle sharded checkpoints Signed-off-by: Alexandros Koumparoulis <[email protected]> * make io_bytes.pt optional Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * set broken tests to optional Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: chtruong814 <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Hemil Desai <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: chtruong814 <[email protected]> Co-authored-by: Piotr Żelasko <[email protected]> Co-authored-by: Asha Anoosheh <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: BoxiangW <[email protected]> Co-authored-by: Huy Vu <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> Signed-off-by: oliver könig <[email protected]>
ko3n1g
added a commit
that referenced
this pull request
Feb 22, 2025
* build: Bump PyT to 25.01 (#11973) * f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * try to fix checkpoint loading Signed-off-by: Chen Cui <[email protected]> * update ci checkpoint Signed-off-by: Chen Cui <[email protected]> * fix ckpt loading for nemo1 and t5 tests Signed-off-by: Chen Cui <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * bump Signed-off-by: oliver könig <[email protected]> * Rename neva datamodule (#12121) * Rename dataset Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update Signed-off-by: yaoyu-33 <[email protected]> * pylink Signed-off-by: yaoyu-33 <[email protected]> * fix f string Signed-off-by: yaoyu-33 <[email protected]> * fix intern vit default factory Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> * tests: Run FSDP2 on dual-gpu (#12145) Signed-off-by: oliver könig <[email protected]> * fix cmd Signed-off-by: Chen Cui <[email protected]> * Revert "build: Force re-install VCS dependencies (#12155)" (#12163) This reverts commit 4b19ade. Signed-off-by: Charlie Truong <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * remove unsed variable Signed-off-by: dimapihtar <[email protected]> * Fix nsys callback tests (#12177) * Fix nsys callback tests Signed-off-by: Hemil Desai <[email protected]> * Simplify tests Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> --------- Signed-off-by: Hemil Desai <[email protected]> * Fixes for bumping pyt to 25.01 (#12165) * fix attention impl config Signed-off-by: Maanu Grover <[email protected]> * safeguard against empty key added by mcore Signed-off-by: Maanu Grover <[email protected]> * docu Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * noqa? Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused import Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> * Use pip --no-deps --force-reinstall when building the test container (#12175) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load in EMA callback and AdapterMixin (#12198) * Set weights_only=False in torch.load within EMA callbck Signed-off-by: Charlie Truong <[email protected]> * Set weights_only=False in torch.load call in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix lint errors in ema and adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Fix undefined name error in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Remove unnecessary import of __futures__ in adapter_mixins Signed-off-by: Charlie Truong <[email protected]> * Fix more locations where weights_only=False needs to be passed to torch.load Signed-off-by: Charlie Truong <[email protected]> * Fix lint issues in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> * Apply isort and black reformatting Signed-off-by: chtruong814 <[email protected]> * Correct noqa F821 line in nemo_model_checkpoint Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: chtruong814 <[email protected]> Co-authored-by: chtruong814 <[email protected]> * Fix nemo-run stdin exception (#12197) * Use pip --no-deps --force-reinstall when building the test container Signed-off-by: Charlie Truong <[email protected]> * Explicitly reinstall nvidia-resiliency Signed-off-by: Charlie Truong <[email protected]> * Do not have pytest capture output in lightning unit tests Signed-off-by: Charlie Truong <[email protected]> * mock invoke context Signed-off-by: Maanu Grover <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * tweak Signed-off-by: Maanu Grover <[email protected]> * more patches needed Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Co-authored-by: Charlie Truong <[email protected]> * Add legacy_ckpt arg to scripts/llm/gpt_distillation.py Signed-off-by: Charlie Truong <[email protected]> * Revert "Do not have pytest capture output in lightning unit tests" (#12202) This reverts commit 162c730. Signed-off-by: Maanu Grover <[email protected]> * Ckpt fixes pytorch update (#12228) * Fix checkpoint loading for ptq tests Signed-off-by: Charlie Truong <[email protected]> * Pass --ckpt_load_strictness log_all to L2_NeMo_2_SSM_Finetuning test Signed-off-by: Charlie Truong <[email protected]> * Fix tests in L2_NMT_Attention_is_All_You_Need_Finetuning by using default_factory Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> * bump modelopt Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 (#12265) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Fix 2D bucketing test on Python 3.12 Signed-off-by: Piotr Żelasko <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Co-authored-by: oliver könig <[email protected]> * remove retro test Signed-off-by: oliver könig <[email protected]> * L2_VLM_HF_Transformer_SFT_FSDP2 Signed-off-by: oliver könig <[email protected]> * Fix distillation state-dict loading bug (#12270) * ci: Bump release workflows (#12259) Signed-off-by: oliver könig <[email protected]> * Add default param dtype in mistral configs (#12186) Signed-off-by: Alexandros Koumparoulis <[email protected]> * ci: Fix pypi link of dry-run (#12267) Signed-off-by: oliver könig <[email protected]> * Fix tiny missing asterisk bug Signed-off-by: Asha Anoosheh <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Add optimizer fix (#12253) * Add optimizer fix Signed-off-by: Boxiang Wang <[email protected]> * Update hf_auto_model_for_causal_lm.py Signed-off-by: BoxiangW <[email protected]> --------- Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> * num gpus Signed-off-by: oliver könig <[email protected]> * Build bitsandbytes (#12279) Co-authored-by: Charlie Truong <[email protected]> * Fixing error when loading T5 checkpoint created with TE<1.13 (#12264) * fix error when loading checkpoint created with TE<1.13 * fix formatting * fix formatting * fix formatting * Apply isort and black reformatting Signed-off-by: huvunvidia <[email protected]> --------- Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> * no cancel for manual Signed-off-by: oliver könig <[email protected]> * Set L2_Speech_Batch_Size_OOMptimizer_Canary to be optional (#12299) Signed-off-by: Charlie Truong <[email protected]> * make expressions r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove parallelize_fn Signed-off-by: Alexandros Koumparoulis <[email protected]> * rstring Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * r-strings Signed-off-by: Alexandros Koumparoulis <[email protected]> * reduce grad-acc-steps Signed-off-by: Alexandros Koumparoulis <[email protected]> * r-string Signed-off-by: Alexandros Koumparoulis <[email protected]> * update tests Signed-off-by: Alexandros Koumparoulis <[email protected]> * force-switch from MegatronCheckpointIO to HFCheckpointIO Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix setup_environment call Signed-off-by: Alexandros Koumparoulis <[email protected]> * update test Signed-off-by: Alexandros Koumparoulis <[email protected]> * update verify_sft_checkpoint_structure to handle sharded checkpoints Signed-off-by: Alexandros Koumparoulis <[email protected]> * make io_bytes.pt optional Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * set broken tests to optional Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: chtruong814 <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Hemil Desai <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: chtruong814 <[email protected]> Co-authored-by: Piotr Żelasko <[email protected]> Co-authored-by: Asha Anoosheh <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: BoxiangW <[email protected]> Co-authored-by: Huy Vu <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]> Signed-off-by: oliver könig <[email protected]> * Update reinstall.sh Signed-off-by: oliver könig <[email protected]> * fix Signed-off-by: oliver könig <[email protected]> * remove retro Signed-off-by: oliver könig <[email protected]> * weights_only=False Signed-off-by: oliver könig <[email protected]> * weights_only=False Signed-off-by: oliver könig <[email protected]> * bump mcore Signed-off-by: oliver könig <[email protected]> * weights_only=False Signed-off-by: oliver könig <[email protected]> * fixes Signed-off-by: oliver könig <[email protected]> * ++model.dist_ckpt_load_strictness=log_all Signed-off-by: oliver könig <[email protected]> * f Signed-off-by: oliver könig <[email protected]> * Distillation NeMo run entrypoint and recipe (#12143) * Remove duplicate test script Signed-off-by: Asha Anoosheh <[email protected]> * Minor renamings Signed-off-by: Asha Anoosheh <[email protected]> * Create entrypoint + recipe Signed-off-by: Asha Anoosheh <[email protected]> * Appease the linter Signed-off-by: Asha Anoosheh <[email protected]> * Fix circular import Signed-off-by: Asha Anoosheh <[email protected]> * Update interface Signed-off-by: Asha Anoosheh <[email protected]> * Appease linter Signed-off-by: Asha Anoosheh <[email protected]> * Appease even more linter Signed-off-by: Asha Anoosheh <[email protected]> * Make TP8 default recipe Signed-off-by: Asha Anoosheh <[email protected]> * Fix tokenizer override Signed-off-by: Asha Anoosheh <[email protected]> --------- Signed-off-by: Asha Anoosheh <[email protected]> * set starcoder to optional Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: chtruong814 <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]> Signed-off-by: Asha Anoosheh <[email protected]> Signed-off-by: Boxiang Wang <[email protected]> Signed-off-by: BoxiangW <[email protected]> Signed-off-by: huvunvidia <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Yu Yao <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Hemil Desai <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: chtruong814 <[email protected]> Co-authored-by: Piotr Żelasko <[email protected]> Co-authored-by: Asha Anoosheh <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: BoxiangW <[email protected]> Co-authored-by: Huy Vu <[email protected]> Co-authored-by: Huy Vu2 <[email protected]> Co-authored-by: huvunvidia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Set weights_only=False in torch.load in EMA callback and AdapterMixin because the default for torch.load changed to
weights_only=True.Changelog
GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information