# List of Figures

| \# | Caption | Page |
| :--- | :--- | ---: |
| Figure 1.1 | Robot in a maze. | 2 |
| Figure 1.2 | PacMan in Atari 2600. | 4 |
| Figure 1.3 | A record for a game of Go. | 4 |
| Figure 1.4 | Bipedal walker. | 5 |
| Figure 1.5 | Large language models. | 5 |
| Figure 1.6 | Agent–environment interface. | 5 |
| Figure 1.7 | Taxonomy of RL. | 8 |
| Figure 1.8 | Relationship among RL, DL, and DRL. | 11 |
| Figure 2.1 | State transition graph of the example. | 26 |
| Figure 2.2 | Compare trajectories of DTMP, DTMRP, and DTMDP. | 28 |
| Figure 2.3 | State transition graph of the example "Feed and Full". | 29 |
| Figure 2.4 | Backup diagram that state values and action values represent each other. | 40 |
| Figure 2.5 | State values and action values back up themselves. | 42 |
| Figure 2.6 | Backup diagram for optimal state values and optimal action values backing up each other. | 64 |
| Figure 2.7 | Backup diagram for optimal state values and optimal action values backing up themselves. | 65 |
| Figure 2.8 | Grid of the task `CliffWalking-v0`. | 72 |
| Figure 3.1 | Policy improvement. | 91 |
| Figure 3.2 | Illustration of bootstrap. | 95 |
| Figure 4.1 | An example task of Monte Carlo. | 106 |
| Figure 4.2 | An example where the optimal policy may not be found without exploring start. | 113 |
| Figure 4.3 | State value estimates obtained by policy evaluation algorithm. | 128 |
| Figure 4.4 | Optimal policy estimates. | 129 |
| Figure 4.5 | Optimal state value estimates. | 130 |
| Figure 5.1 | Backup diagram of TD return and MC return. | 138 |
| Figure 5.2 | Maximization bias in Q learning. | 153 |
| Figure 5.3 | Backup diagram of $\lambda$ return. | 156 |
| Figure 5.4 | Compare different eligibility traces. | 158 |
| Figure 5.5 | ASCII map of the task `Taxi-v3`. | 160 |
| Figure 6.1 | MDP in Baird's counterexample. | 184 |
| Figure 6.2 | Trend of parameters with iterations. | 185 |
| Figure 6.3 | The task `MountainCar-v0`. | 195 |
| Figure 6.4 | Position and velocity of the car when it is always pushed right. | 196 |
| Figure 6.5 | One-hot coding and tile coding. | 197 |
| Figure 7.1 | The cart-pole problem. | 224 |
| Figure 8.1 | Illustration of MM algorithm. | 244 |
| Figure 8.2 | Relationship among $g_{\pi\left({\mathbf\uptheta}\right)}$, $l\left({\mathbf\uptheta}\middle\vert{\mathbf\uptheta_k}\right)$, and $l_c\left({\mathbf\uptheta}\middle\vert{\mathbf\uptheta_k}\right)$. | 252 |
| Figure 8.3 | The task `Acrobot-v1`. | 259 |
| Figure 9.1 | The task `Pendulum-v1`. | 300 |
| Figure 12.1 | Some Atari games. | 390 |
| Figure 12.2 | Neural network for Categorical DQN. | 398 |
| Figure 12.3 | Neural network for IQN. | 403 |
| Figure 14.1 | Search tree. | 434 |
| Figure 14.2 | Steps of MCTS. | 436 |
| Figure 14.3 | First two steps of the reversi opening "Chimney". | 446 |
| Figure 14.4 | Game tree of Tic-Tac-Toe. | 448 |
| Figure 14.5 | Maximin decision of Tic-Tac-Toe. | 449 |
| Figure 14.6 | MCTS with self-play. | 450 |
| Figure 14.7 | Reverse the color of all pieces on the board. | 452 |
| Figure 14.9 | Residual network. | 452 |
| Figure 14.8 | Example structure prediction network for the game of Go. | 453 |
| Figure 15.1 | MDP of the task "Tiger". | 501 |
| Figure 15.2 | Trajectories maintained by the environment and the agent. | 503 |
| Figure 15.3 | Belief MDP of the task "Tiger". | 507 |
| Figure 16.1 | Learning from feedbacks. | 526 |
| Figure 16.2 | Agent–environment interface of IL. | 531 |
| Figure 16.3 | Compounding error of imitation policy. | 541 |
| Figure 16.4 | Training GPT. | 545 |
| Figure 16.5 | Principal axes and Euler's angles. | 548 |
