Friends of OLMo and their links.

Minimalistic large language model 3D-parallelism training

Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

Entropy Based Sampling and Parallel CoT Decoding

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

Tools for merging pretrained large language models.

An Open Source Toolkit For LLM Distillation

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

Train transformer language models with reinforcement learning.

LLM inference in C/C++

A framework for the evaluation of autoregressive code generation language models.

Video+code lecture on building nanoGPT from scratch

LLM training in simple, raw C/CUDA

The simplest, fastest repository for training/finetuning medium-sized GPTs.

GPT-2 (124M) quality in 5B tokens