1000+ stars (Sep 2023)

The book received 1000+ stars! Thank everybody!

Major update (Aug 2023)

The second version of the draft of the book is online now!!

Compared to the first version, which was online one year ago, the second version has been improved in various ways. For example, we replotted most of the figures, reorganized some contents to make them clearer, corrected some typos, and added Chapter 10, which was not included in the first version.

I put the first draft of this book online in August 2022. Up to now, I have received valuable feedback from many readers worldwide. I want to express my gratitude to these readers.

Why a new book on reinforcement learning?

This book aims to provide a mathematical but friendly introduction to the fundamental concepts, basic problems, and classic algorithms in reinforcement learning. Some essential features of this book are highlighted as follows.

• The book introduces reinforcement learning from a mathematical point of view. Hopefully, readers will not only know the procedure of an algorithm but also understand why it was designed in the first place and why it works effectively.

• The depth of the mathematics is carefully controlled to an adequate level. The mathematics is also presented in a carefully designed manner to ensure that the book is friendly to read. Readers can selectively read the materials presented in gray boxes according to their interests.

• Many illustrative examples are given to help readers better understand the topics. All the examples in this book are based on a grid world task, which is easy to understand and helpful for illustrating concepts and algorithms.

• When introducing an algorithm, the book aims to separate its core idea from complications that may be distracting. In this way, readers can better grasp the core idea of an algorithm.

• The contents of the book are coherently organized. Each chapter is built based on the preceding chapter and lays a necessary foundation for the subsequent one.

Contents

The topics addressed in the book are shown in the figure below. This book contains ten chapters, which can be classified into two parts: the first part is about basic tools, and the second part is about algorithms. The ten chapters are highly correlated. In general, it is necessary to study the earlier chapters first before the later ones.

Readership

This book is designed for senior undergraduate students, graduate students, researchers, and practitioners interested in reinforcement learning.

It does not require readers to have any background in reinforcement learning because it starts by introducing the most basic concepts. If the reader already has some background in reinforcement learning, I believe the book can help them understand some topics more deeply or provide different perspectives.

This book, however, requires the reader to have some knowledge of probability theory and linear algebra. Some basics of the required mathematics are also included in the appendix of this book.

Lecture videos

The lecture videos (in Chinese) are online. You can check the Bilibili channel https://space.bilibili.com/2044042934 or the Youtube channel https://www.youtube.com/channel/UCztGtS5YYiNv8x3pj9hLVgg/playlists

All the online lecture videos have more than 500,000 views up to now.

About the author

You can find my info on my homepage https://www.shiyuzhao.net/ (GoogleSite) and my research group website https://shiyuzhao.westlake.edu.cn

I have been teaching a graduate-level course on reinforcement learning since 2019. Along with teaching, I have been preparing this book as the lecture notes for my students.

I sincerely hope this book can help readers smoothly enter the exciting field of reinforcement learning.

Citation

@book{zhao2024RLBook,
  title={Mathematical Foundations of Reinforcement Learning},
  author={S. Zhao},
  year={2024},
  publisher={Springer Nature Press and Tsinghua University Press}
}

Third-party code and materials

Some enthusiastic readers sent me the source code or notes that they developed when they studied this book. I would like to share the links here and hope it may be helpful to other readers. I must emphasize that I have not verified the code. If you have any questions, you can directly contact the developers.

Code:

https://github.com/jwk1rose/RL_Learning Language: Python

https://github.com/NewbieToEverything/Code-Mathmatical-Foundation-of-Reinforcement-Learning Language: R

Notes:

https://lyk-love.cn/tags/reinforcement-learning/ by Yukuan Lu, a student from UC Davis

Update history

(Aug 2023)

The second version of the draft of the book is online now!

Compared to the first version, which was online one year ago, the second version has been improved in various ways. For example, we corrected some typos, reorganized some contents to make them clearer, and added Chapter 10, which was not included in the first version.

I put the first draft of this book online in August 2022. Up to now, I have received valuable feedback from many readers worldwide. I want to express my gratitude to these readers.

(Nov 2022)

This book will be published jointly by Springer Nature and Tsinghua University Press. It will probably be printed in the second half of 2023.

I have received some comments and suggestions about this book from some readers. Thanks a lot, and I appreciate it. I am still collecting feedback and will probably revise the draft in several months. Your feedback can make this book more helpful for other readers!

(Oct 2022)

The lecture slides have been uploaded in the folder "Lecture slides."

The lecture videos (in Chinese) are online. Please check our Bilibili channel https://space.bilibili.com/2044042934 or the Youtube channel https://www.youtube.com/channel/UCztGtS5YYiNv8x3pj9hLVgg/playlists

(Aug 2022)

The first draft of the book is online.