Merge branch 'slym/pyt21.08_nvfuser' into 'main'

Use nvfuser at pytorch >= 1.10

See merge request ADLR/megatron-lm!314

Merge branch 'sc21' into 'main'

scripts for sc21

See merge request ADLR/megatron-lm!298

Merge branch 'github-main' into 'main'

Bringing github main and internal main back together.

See merge request ADLR/megatron-lm!275

Merge branch 'main' into github-main

Cherry-pick fix from development repo.

Use torch.cuda.synchronize() right after calling batch_isend_irecv() …

…communication API

Merge branch 'TOC-fix' into 'main'

TOC fix

See merge request ADLR/megatron-lm!208

Merge branch 'readme_update' into 'main'

Update scaling numbers in README and other small tweaks.

See merge request ADLR/megatron-lm!130

Updated with minor fix to l2 norm computation.

Merge branch 'torchddp' into 'master'

Use DDP command line argument instead of source flag in pretrain_bert.py.

See merge request ADLR/megatron-lm!2