This course will take you from no knowledge of deep learning to training your own GPT model. As AI moves out of the research lab, the world needs more people who can understand and apply it. If you want to be one of them, this course is for you.
We’ll start with the fundamentals - the basics of neural networks, how they work, and how to tune them. You need some math to understand deep learning, but we won’t get bogged down in it.
This course focuses on understanding concepts over theory. We’ll solve real problems, like predicting the weather and translating languages. Over time, we'll move to more complex topics, like transformers, GPU programming, and distributed training.
To use this course, go through each chapter sequentially. Read the lessons or watch the optional videos - they have the same information. Look through the implementations to solidify your understanding, and try to recreate them on your own.
0. Introduction
An overview of the course and topics we'll cover. Includes some math and NumPy fundamentals you'll need for deep learning.
- Course intro
- Video coming soon
1. Gradient descent
Gradient descent is how neural networks train their parameters to match the data. It's the "learning" part of deep learning.
2. Dense networks
Dense networks are the basic form of a neural network, where every input is connected to an output. These can also be called fully connected networks.
3. Classification with neural networks
Classification is how we get neural networks to categorize data for us. Classification is used by language models like GPT to predict the next word in a sequence.
- Lesson
- Video coming soon
4. Recurrent networks
Recurrent neural networks are optimized to process sequences of data. They're used for tasks like translation and text classification.
- Lesson
- Video (optional)
- Implementation
5. Backpropagation in depth
So far, we've taken a loose look at backpropagation to let us focus on understanding neural network architecture. We'll build a miniature version of PyTorch, and use it to understand backpropagation better.
6. PyTorch
PyTorch is a framework for deep learning that automatically differentiates functions. It's widely used to create cutting-edge models.
- Lesson coming soon
7. Regularization
Regularization prevents overfitting to the training set. This means that the network can generalize well to new data.
- Lesson coming soon
8. Data
If you want to train a deep learning model, you need data. Gigabytes of it. We'll discuss how you can get this data and process it.
- Lesson coming soon
9. Encoders and decoders
Encoder/decoders are used for NLP tasks when the output isn't the same length as the input. For example, if you want to use questions/answers as training data, the answers may be a different length than the question.
- Lesson coming soon
- Implementation
10. Transformers
Transformers fix the problem of vanishing/exploding gradients in RNNs by using attention. Attention allows the network to process the whole sequence at once, instead of iteratively.
- Lesson coming soon
- Implementation
11. GPU programming with Triton
To train a large neural network, we'll need to use GPUs. PyTorch can automatically use GPUs, but not all operators are fused and optimized. For example, flash attention can speed up transformers by 2x or more. We'll use OpenAI Triton to implement GPU kernels.
- Lesson coming soon
- Implementation coming soon
12. Efficient transformers
GPT models take a long time to train. We can reduce that time by using more GPUs, but we don't all have access to GPU clusters. To reduce training time, we'll incorporate some recent advances to make the transformer model more efficient.
- Lesson coming soon
- Implementation
Convolutional networks
Convolutional neural networks are used for working with images and time series.
Gated recurrent networks
Gated recurrent networks help RNNs process long sequences by helping networks forget irrelevant information. LSTM and GRU are two popular types of gated networks.
If you want to run these notebooks locally, you'll need to install some Python packages.
- Make sure you have Python 3.8 or higher installed.
- Clone this repository.
- Run
pip install -r requirements.txt
You can use and adapt this material for your own courses, but not commercially. You must provide attribution to Vik Paruchuri, Dataquest if you use this material.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
