Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
isaacperez committed Aug 12, 2024
1 parent 8061562 commit 20ba569
Showing 1 changed file with 32 additions and 7 deletions.
39 changes: 32 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# TinyGPT
Like [NanoGPT](https://github.com/karpathy/nanoGPT) but tiny. It is also inspired by [Tinygrad](https://github.com/tinygrad/tinygrad), [Pytorch](https://github.com/pytorch/pytorch) and [MLX](https://github.com/ml-explore/mlx).
TinyGPT is a minimalistic library for implementing, training, and performing inference on GPT models from scratch in Python, with no external dependencies. Inspired by [NanoGPT](https://github.com/karpathy/nanoGPT), [Tinygrad](https://github.com/tinygrad/tinygrad), [Pytorch](https://github.com/pytorch/pytorch), and [MLX](https://github.com/ml-explore/mlx), TinyGPT aims to be as educational as possible, avoiding complex optimizations that might obscure the underlying concepts.

The main objective of this project is to be as didactic as possible, avoiding optimizations that make the code difficult to understand.

I hope we can understand how to train and run a model like GPT-3 using as few libraries as possible and programming everything from scratch, including the library to train and run the model.
## Features
- **Pure Python Implementation**: TinyGPT is written entirely in Python with no external dependencies, making it easy to follow and modify.
- **Didactic Focus**: Prioritizes readability and understanding over optimization, making it an excellent learning tool.
- **Modular Design**: The library is divided into several modules, each focusing on a specific aspect of training and inference.

## Installation
The current recommended way to install TinyGPT is from source.
Expand All @@ -16,12 +17,36 @@ $ python -m pip install -e .
```
Don't forget the `.` at the end!

## Usage
[TO DO]
## Project Structure
- `buffer.py`: Provides a low-level implementation of array operations, similar to NumPy arrays.
- `dataset.py`: Handles data loading and preprocessing.
- `losses.py`: Contains implementations of loss functions used for training.
- `mlops.py`: Provides a low-level implementation of array operations, similar to NumPy arrays.
- `module.py`: Defines the base module from which all model components inherit.
- `nn.py`: Contains the neural network components, including layers and activation functions.
- `optimizers.py`: Implements optimization algorithms used during training.
- `tensor.py`: Provides a minimal tensor implementation to support basic tensor operations.
- `tokenizer.py`: Handles tokenization of text data for model input.
- `utils.py`: Contains miscellaneous utility functions used across the library.

## Examples
The `examples/` directory contains scripts demonstrating how to use TinyGPT for various tasks:

- `gpt.py`: A basic example of training and using a GPT model.
- `mnist.py`: An example of using TinyGPT's neural network components on the MNIST dataset.

To run these examples, navigate to the examples/ directory and execute the Python scripts:

```bash
$ cd examples
$ python gpt.py
$ python mnist.py
```

These examples will guide you through setting up the model, training it, and performing inference, providing a hands-on understanding of how TinyGPT works.

## Documentation
Documentation along with a quick start guide can be found in the docs/ directory.
Documentation along with a quick start guide can be found in the `docs/` directory.

## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Expand Down

0 comments on commit 20ba569

Please sign in to comment.