# List of Tables

| \# | Caption | Page |
| :--- | :--- | ---: |
| Table 1.1 | Major Python modules that the codes in this book depend on. | 13 |
| Table 2.1 | Example initial state distribution in the task "Feed and Full". | 29 |
| Table 2.2 | Example dynamics in the task "Feed and Full". | 29 |
| Table 2.3 | Transition probability from state–action pair to the next state derived from Table 2.2. | 31 |
| Table 2.4 | Expected state–action reward derived from Table 2.2. | 32 |
| Table 2.5 | Expected reward from a state–action pair to the next state derived from Table 2.2. | 32 |
| Table 2.6 | An example policy in the task "Feed and Full". | 33 |
| Table 2.7 | Another example policy in the task "Feed and Full". | 33 |
| Table 2.8 | Alternative presentation of the deterministic policy in Table 2.7. | 33 |
| Table 2.9 | Initial state–action distribution derived from Tables 2.1 and 2.6. | 34 |
| Table 2.10 | Transition probability from a state to the next state derived from Tables 2.2 and 2.6. | 35 |
| Table 2.11 | Transition probability from a state–action pair to the next state–action pair derived from Tables 2.2 and 2.6. | 35 |
| Table 2.12 | Expected state reward derived from Tables 2.2 and 2.6. | 35 |
| Table 2.13 | State values derived from Tables 2.2 and 2.6. | 46 |
| Table 2.14 | Action values derived from Tables 2.2 and 2.6. | 46 |
| Table 2.15 | State values derived from Tables 2.2 and 2.7. | 47 |
| Table 2.16 | Discounted state visitation frequency derived from Tables 2.2 and 2.6. | 56 |
| Table 2.17 | Discounted state–action visitation frequency derived from Tables 2.2 and 2.6. | 57 |
| Table 2.18 | Optimal state values derived from Table 2.2. | 68 |
| Table 2.19 | Optimal action values derived from Table 2.2. | 68 |
| Table 4.1 | Value of the cards in Blackjack. | 124 |
| Table 5.1 | Taxi stands in the task `Taxi-v3`. | 161 |
| Table 5.2 | Actions in the task `Taxi-v3`. | 161 |
| Table 6.1 | Convergence of policy evaluation algorithms. | 182 |
| Table 6.2 | Convergence of policy optimization algorithms. | 183 |
| Table 6.3 | Tricks used by different algorithms. | 194 |
| Table 7.1 | Observations in the cart-pole problem. | 225 |
| Table 9.1 | Compare the task `Pendulum-v1`. | 301 |
| Table 10.1 | Actions in the task `LunarLander-v2`. | 334 |
| Table 10.2 | Actions in the task `LunarLanderContinuous-v2`. | 334 |
| Table 11.1 | Observations in the task `BipedalWalker-v3`. | 359 |
| Table 11.2 | Actions in the task `BipedalWalker-v3`. | 360 |
| Table 12.1 | Compare Categorical DQN, QR-DQN, and IQN. | 388 |
| Table 12.2 | Different versions of the Pong game. | 391 |
| Table 14.1 | Compare selection policy and decision policy. | 440 |
| Table 14.2 | Neural networks in MCTS. | 441 |
| Table 14.3 | Some two-player zero-sum deterministic sequential board games. | 445 |
| Table 14.4 | 8 equivalent boards. | 451 |
| Table 14.5 | Some DRL algorithms that apply MCTS to board games. | 454 |
| Table 14.6 | APIs for environment dynamics. | 458 |
| Table 15.1 | Initial probability in the task "Tiger". We can delete the reward row since the initial reward is trivial. | 501 |
| Table 15.2 | Initial emission probability in the task "Tiger". We can delete this table since the initial observation is trivial. | 501 |
| Table 15.3 | Dynamics in the task "Tiger". We can delete the reward column if rewards are not considered. | 501 |
| Table 15.4 | Observation probability in the task "Tiger". | 502 |
| Table 15.5 | Conditional probability $\omega {\left ({\mathsfit {s^\prime },\mathsfit {o}}\middle \vert {{b,\mathsfit {a}}}\right )}$ in the task "Tiger". | 504 |
| Table 15.6 | Conditional probability $\omega {\left ({\mathsfit {o}}\middle \vert {{b,\mathsfit {a}}}\right )}$ in the task "Tiger". | 505 |
| Table 15.7 | Belief updates in the task "Tiger". | 505 |
| Table 15.8 | $r{{\left ({{b,\mathsfit {a}}}\right )}}$ in the task "Tiger". | 506 |
| Table 15.9 | Discounted optimal values and optimal policy of the task "Tiger". | 511 |
