🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

8-bit CUDA functions for PyTorch

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

QLoRA: Efficient Finetuning of Quantized LLMs

Ongoing research training transformer language models at scale, including: BERT & GPT-2

A tool for extracting plain text from Wikipedia dumps

University Domains and Names Data List & API

Kerberos credential support for batch environments

Open deep learning compiler stack for cpu, gpu and specialized accelerators

Mirror of Apache Hive

Livy is an open source REST interface for interacting with Apache Spark from anywhere

Apache Airflow (Incubating)

Build J2EE Web applications in Scala.

Mirror of Apache Spark

Scalable PCA (sPCA) is a scalable implementation of Principal component analysis algorithm on top of Spark