Best practice for training LLaMA models in Megatron-LM
-
Updated
Jan 2, 2024 - Python
#
megatron-lm
Here are 11 public repositories matching this topic...
Annotations of the interesting ML papers I read
nlp
machine-learning
deep-learning
transformers
gpt
research-paper
bert
gpt-2
xlnet
annotated-paper
megatron-lm
papers-annotations
-
Updated
Nov 10, 2024
Super-Efficient RLHF Training of LLMs with Parameter Reallocation
distributed-systems
reinforcement-learning
distributed-computing
transformers
large-scale-machine-learning
deepspeed
megatron-lm
large-language-models
llm
reinforcement-learning-from-human-feedback
llm-training
llm-framework
-
Updated
Dec 6, 2024 - Python
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
transformers
moe
data-parallelism
distributed-optimizers
model-parallelism
megatron
mixture-of-experts
pipeline-parallelism
huggingface-transformers
megatron-lm
tensor-parallelism
large-scale-language-modeling
3d-parallelism
zero-1
sequence-parallelism
-
Updated
Dec 14, 2023 - Python
Odysseus: Playground of LLM Sequence Parallelism
-
Updated
Jun 17, 2024 - Python
A LLaMA1/LLaMA12 Megatron implement.
-
Updated
Dec 13, 2023 - Python
Training NVIDIA NeMo Megatron Large Language Model (LLM) using NeMo Framework on Google Kubernetes Engine
-
Updated
Nov 19, 2024 - HCL
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
-
Updated
Nov 16, 2023 - Python
Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP
-
Updated
Feb 6, 2024 - Python
Running Large Language Model easily.
-
Updated
Dec 6, 2024 - Jupyter Notebook
Improve this page
Add a description, image, and links to the megatron-lm topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the megatron-lm topic, visit your repo's landing page and select "manage topics."