# List of Algorithms

| \# | Caption | Page |
| :--- | :--- | ---: |
| Algorithm 2.1 | Check whether the policy is optimal. | 49 |
| Algorithm 2.2 | Policy improvement. | 50 |
| Algorithm 3.1 | Model-based numerical iterative policy evaluation to estimate state values. | 88 |
| Algorithm 3.2 | Model-based numerical iterative policy evaluation to estimate action values. | 89 |
| Algorithm 3.3 | Model-based numerical iterative policy evaluation to estimate action values (space-saving version). | 90 |
| Algorithm 3.4 | Model-based numerical iterative policy evaluation (space-saving version, alternative implementation). | 90 |
| Algorithm 3.5 | Model-based policy iteration. | 92 |
| Algorithm 3.6 | Model-based policy iteration (space-saving version). | 92 |
| Algorithm 3.7 | Model-based VI. | 93 |
| Algorithm 3.8 | Model-based VI (space-saving version). | 93 |
| Algorithm 4.1 | Evaluate action values using every-visit MC policy evaluation. | 110 |
| Algorithm 4.2 | Every-visit MC update to evaluate state values. | 110 |
| Algorithm 4.3 | First-visit MC update to estimate action values. | 111 |
| Algorithm 4.4 | First-visit MC update to estimate state values. | 112 |
| Algorithm 4.5 | MC update with exploring start (maintaining policy explicitly). | 113 |
| Algorithm 4.6 | MC update with exploring start (maintaining policy implicitly). | 114 |
| Algorithm 4.7 | MC update with soft policy (maintaining policy explicitly). | 116 |
| Algorithm 4.8 | MC update with soft policy (maintaining policy explicitly). | 117 |
| Algorithm 4.9 | Evaluate action values using off-policy MC update based on importance sampling. | 121 |
| Algorithm 4.10 | Find an optimal policy using off-policy MC update based on importance sampling. | 123 |
| Algorithm 5.1 | One-step TD policy evaluation to estimate action values. | 139 |
| Algorithm 5.2 | One-step TD policy evaluation to estimate action values with an indicator of episode end. | 140 |
| Algorithm 5.3 | One-step TD policy evaluation to estimate state values. | 141 |
| Algorithm 5.4 | $n$-step TD policy evaluation to estimate action values. | 142 |
| Algorithm 5.5 | $n$-step TD policy evaluation to estimate state values. | 143 |
| Algorithm 5.6 | SARSA (maintaining the policy explicitly). | 144 |
| Algorithm 5.7 | SARSA (maintaining the policy implicitly). | 145 |
| Algorithm 5.8 | $n$-step SARSA. | 146 |
| Algorithm 5.9 | Expected SARSA. | 147 |
| Algorithm 5.10 | $n$-step expected SARSA. | 148 |
| Algorithm 5.11 | $n$-step TD policy evaluation of SARSA with importance sampling. | 150 |
| Algorithm 5.12 | Q learning. | 152 |
| Algorithm 5.13 | Double Q Learning. | 154 |
| Algorithm 5.14 | TD ${\left ({\lambda }\right )}$ policy evaluation or SARSA ${\left ({\lambda }\right )}$. | 158 |
| Algorithm 5.15 | TD ${\left ({\lambda }\right )}$ policy evaluation to estimate state values. | 159 |
| Algorithm 6.1 | Policy evaluation with function approximation and SGD. | 177 |
| Algorithm 6.2 | Policy optimization with function approximation and SGD. | 177 |
| Algorithm 6.3 | Semi-gradient descent policy evaluation to estimate action values or SARSA policy optimization. | 178 |
| Algorithm 6.4 | Semi-gradient descent policy evaluation to estimate state values, or expected SARSA policy optimization, or Q learning. | 179 |
| Algorithm 6.5 | TD ${\left ({\lambda }\right )}$ policy evaluation for action values or SARSA. | 181 |
| Algorithm 6.6 | TD ${\left ({\lambda }\right )}$ policy evaluation for state values, or expected SARSA, or Q learning. | 181 |
| Algorithm 6.7 | DQN policy optimization with experience replay (loop over episodes). | 187 |
| Algorithm 6.8 | DQN policy optimization with experience replay (without looping over episodes explicitly). | 188 |
| Algorithm 6.9 | DQN with experience replay and target network. | 191 |
| Algorithm 7.1 | VPG policy optimization. | 220 |
| Algorithm 7.2 | VPG policy optimization with baseline. | 222 |
| Algorithm 7.3 | Importance sampling PG policy optimization. | 223 |
| Algorithm 8.1 | Action-value on-policy AC. | 239 |
| Algorithm 8.2 | Advantage AC. | 240 |
| Algorithm 8.3 | A3C (one-step TD version, showing the behavior of one worker). | 240 |
| Algorithm 8.4 | Advantage AC with eligibility trace. | 242 |
| Algorithm 8.5 | Clipped PPO (simplified version). | 246 |
| Algorithm 8.6 | Clipped PPO (with on-policy experience replay). | 246 |
| Algorithm 8.7 | Vanilla NPG. | 253 |
| Algorithm 8.8 | CG. | 255 |
| Algorithm 8.9 | NPG with CG. | 255 |
| Algorithm 8.10 | TRPO. | 257 |
| Algorithm 8.11 | OffPAC. | 258 |
| Algorithm 9.1 | Vanilla on-policy deterministic AC. | 292 |
| Algorithm 9.2 | OPDAC. | 294 |
| Algorithm 9.3 | DDPG. | 295 |
| Algorithm 9.4 | TD3. | 297 |
| Algorithm 10.1 | SQL. | 326 |
| Algorithm 10.2 | SAC. | 328 |
| Algorithm 10.3 | SAC with automatic entropy adjustment. | 331 |
| Algorithm 11.1 | ES. | 356 |
| Algorithm 11.2 | ARS. | 358 |
| Algorithm 12.1 | Categorical DQN to find the optimal policy (to maximize expectation). | 377 |
| Algorithm 12.2 | Categorical DQN to find the optimal policy (to maximize VNM utility). | 378 |
| Algorithm 12.3 | QR-DQN to Find the Optimal Policy (To Maximize Expectation). | 382 |
| Algorithm 12.4 | IQN to find the optimal policy (to maximize expectation). | 384 |
| Algorithm 12.5 | Categorical DQN to find the optimal policy (use Yarri distortion function). | 387 |
| Algorithm 13.1 | $\varepsilon $-greedy. | 414 |
| Algorithm 13.2 | UCB (including UCB1). | 415 |
| Algorithm 13.3 | Bayesian UCB. | 421 |
| Algorithm 13.4 | Thompson Sampling. | 422 |
| Algorithm 13.5 | UCBVI. | 423 |
| Algorithm 14.1 | MCTS. | 435 |
| Algorithm 14.2 | AlphaZero. | 441 |
| Algorithm 14.3 | MuZero. | 442 |
| Algorithm 15.1 | Semi-gradient descent policy evaluation to estimate action values or SARSA policy optimization. | 484 |
| Algorithm 15.2 | Semi-gradient descent differential expected SARSA policy optimization, or differential Q learning. | 485 |
| Algorithm 15.3 | Model-based VI for fixed-horizontal episode. | 492 |
| Algorithm 16.1 | GAIL-PPO. | 543 |
