# Table of Codes

| \# | Caption |
| --- | --- |
| [Code 1-1](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_ClosedForm.html) | Check the observation space and action space of the environment |
| [Code 1-2](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_ClosedForm.html) | Closed-form agent for task `MountainCar-v0` |
| [Code 1-3](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_ClosedForm.html) | Play an episode |
| [Code 1-4](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_ClosedForm.html) | Test the performance by playing 100 episodes |
| [Code 1-5](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCarContinuous-v0_ClosedForm.html) | Check the observation space and action space of the task `MountainCarContinuous-v0` |
| [Code 1-6](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCarContinuous-v0_ClosedForm.html) | Closed-form agent for task `MountainCarContinous-v0` |
| [Code 2-1](https://zhiqingxiao.github.io/rl-book/en2024/code/HungryFull_demo.html) | Use the example Bellman expectation equation |
| [Code 2-2](https://zhiqingxiao.github.io/rl-book/en2024/code/HungryFull_demo.html) | Example to solve Bellman optimal equation |
| [Code 2-3](https://zhiqingxiao.github.io/rl-book/en2024/code/CliffWalking-v0_Bellman_demo.html) | Import the environment `CliffWalking-v0` and check its information |
| [Code 2-4](https://zhiqingxiao.github.io/rl-book/en2024/code/CliffWalking-v0_Bellman_demo.html) | Find states values and action values using Bellman expectation equations |
| [Code 2-5](https://zhiqingxiao.github.io/rl-book/en2024/code/CliffWalking-v0_Bellman_demo.html) | Find optimal values using LP method |
| [Code 2-6](https://zhiqingxiao.github.io/rl-book/en2024/code/CliffWalking-v0_Bellman_demo.html) | Find an optimal deterministic policy from optimal action values |
| [Code 3-1](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Check the metadata of `FrozenLake-v1` |
| [Code 3-2](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Play an episode using the policy |
| [Code 3-3](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Calculate the episode rewards of the random policy |
| [Code 3-4](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Implementation of Policy Evaluation |
| [Code 3-5](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Evaluate the random policy |
| [Code 3-6](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Policy improvement |
| [Code 3-7](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Improve the random policy |
| [Code 3-8](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Policy iteration |
| [Code 3-9](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Use policy iteration to find the optimal policy and test it |
| [Code 3-10](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | VI |
| [Code 3-11](https://zhiqingxiao.github.io/rl-book/en2024/code/FrozenLake-v1_DP_demo.html) | Find the optimal policy using the value iteration algorithm |
| [Code 4-1](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | Play an episode |
| [Code 4-2](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | On-Policy MC evaluation |
| [Code 4-3](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | Visualize a 3-dimension np.array, which can be indexed by a state |
| [Code 4-4](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | On-policy MC update with exploring start |
| [Code 4-5](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | MC update with soft policy |
| [Code 4-6](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | Policy evaluation based on importance sampling |
| [Code 4-7](https://zhiqingxiao.github.io/rl-book/en2024/code/Blackjack-v1_MonteCarlo_demo.html) | Importance sampling policy optimization with soft policy |
| [Code 5-1](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_SARSA_demo.html) | Initialize and visualize the task |
| [Code 5-2](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_SARSA_demo.html) | SARSA agent |
| [Code 5-3](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_SARSA_demo.html) | Train the agent |
| [Code 5-4](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_ExpectedSARSA.html) | Expected SARSA agent |
| [Code 5-5](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_QLearning.html) | Q Learning agent |
| [Code 5-6](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_DoubleQLearning.html) | Double Q Learning agent |
| [Code 5-7](https://zhiqingxiao.github.io/rl-book/en2024/code/Taxi-v3_SARSALambda.html) | SARSA $(\lambda)$ agent |
| [Code 6-1](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_SARSA_demo.html) | Import the environment of `MountainCar-v0` |
| [Code 6-2](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_SARSA_demo.html) | The agent that always pushes right |
| [Code 6-3](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_SARSA_demo.html) | Tile coding |
| [Code 6-4](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_SARSA_demo.html) | SARSA agent with function approximation |
| [Code 6-5](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_SARSAlambda.html)| SARSA $(\lambda)$ agent with function approximation |
| [Code 6-6](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DQN_tf.html) | Experience replayer |
| [Code 6-7](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DQN_tf.html) | DQN agent with target network (with TensorFlow) |
| [Code 6-8](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DQN_torch.html) | DQN agent with target network (with PyTorch) |
| [Code 6-9](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DoubleDQN_tf.html) | Double DQN agent (with TensorFlow) |
| [Code 6-10](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DoubleDQN_torch.html) | Double DQN agent (with PyTorch) |
| [Code 6-11](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DuelDQN_tf.html) | Dueling network (with TensorFlow) |
| [Code 6-12](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DuelDQN_torch.html) | Dueling network (with PyTorch) |
| [Code 6-13](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DuelDQN_tf.html) | Dueling DQN agent (with TensorFlow) |
| [Code 6-14](https://zhiqingxiao.github.io/rl-book/en2024/code/MountainCar-v0_DuelDQN_torch.html) | Dueling DQN agent (with PyTorch) |
| [Code 7-1](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_VPG_tf.html) | On-policy VPG agent (with TensorFlow) |
| [Code 7-2](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_VPG_torch.html) | On-policy VPG agent (with PyTorch) |
| [Code 7-3](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_VPGwBaseline_tf.html) | On-policy VPG agent with baseline (with TensorFlow) |
| [Code 7-4](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_VPGwBaseline_torch.html) | On-policy VPG agent with baseline (with PyTorch) |
| [Code 7-5](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_OffPolicyVPG_tf.html) | Off-policy PG agent (with TensorFlow) |
| [Code 7-6](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_OffPolicyVPG_torch.html) | Off-policy PG agent (with PyTorch) |
| [Code 7-7](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_OffPolicyVPGwBaseline_tf.html) | Off-policy PG agent with baseline (with TensorFlow) |
| [Code 7-8](https://zhiqingxiao.github.io/rl-book/en2024/code/CartPole-v0_OffPolicyVPGwBaseline_torch.html) | Off-policy PG agent with baseline (with PyTorch) |
| [Code 8-1](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_QActorCritic_tf.html) | Action-value AC agent (with TensorFlow) |
| [Code 8-2](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_QActorCritic_torch.html) | Action-value AC agent (with PyTorch) |
| [Code 8-3](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_AdvantageActorCritic_tf.html) | Advantage AC agent (with TensorFlow) |
| [Code 8-4](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_AdvantageActorCritic_torch.html) | Advantage AC agent (with PyTorch) |
| [Code 8-5](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_EligibilityTraceAC_tf.html) | Eligibility-trace AC agent (with TensorFlow) |
| [Code 8-6](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_EligibilityTraceAC_torch.html) | Eligibility-trace AC agent (with PyTorch) |
| [Code 8-7](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_PPO_tf.html) | Replayer for PPO |
| [Code 8-8](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_PPO_tf.html) | PPO agent (with TensorFlow) |
| [Code 8-9](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_PPO_torch.html) | PPO agent (with PyTorch) |
| [Code 8-10](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_NPG_tf.html) | Calculate CG (with TensorFlow) |
| [Code 8-11](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_NPG_torch.html) | Calculate CG (with PyTorch) |
| [Code 8-12](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_NPG_tf.html) | NPG agent (with TensorFlow) |
| [Code 8-13](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_NPG_torch.html) | NPG agent (with PyTorch) |
| [Code 8-14](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_TRPO_tf.html) | TRPO agent (with TensorFlow) |
| [Code 8-15](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_TRPO_torch.html) | TRPO agent (with PyTorch) |
| [Code 8-16](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_OffPAC_tf.html) | OffPAC agent (with TensorFlow) |
| [Code 8-17](https://zhiqingxiao.github.io/rl-book/en2024/code/Acrobot-v1_OffPAC_torch.html) | OffPAC agent (with PyTorch) |
| [Code 9-1](https://zhiqingxiao.github.io/rl-book/en2024/code/Pendulum-v1_DDPG_tf.html) | OU process |
| [Code 9-2](https://zhiqingxiao.github.io/rl-book/en2024/code/Pendulum-v1_DDPG_tf.html) | DDPG agent (with TensorFlow) |
| [Code 9-3](https://zhiqingxiao.github.io/rl-book/en2024/code/Pendulum-v1_DDPG_torch.html) | DDPG agent (with PyTorch) |
| [Code 9-4](https://zhiqingxiao.github.io/rl-book/en2024/code/Pendulum-v1_TD3_tf.html) | TD3 agent (with TensorFlow) |
| [Code 9-5](https://zhiqingxiao.github.io/rl-book/en2024/code/Pendulum-v1_TD3_torch.html) | TD3 agent (with PyTorch) |
| [Code 10-1](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_ClosedForm.html) | Closed-form solution of `LunarLander-v2` |
| [Code 10-2](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLanderContinuous-v2_ClosedForm.html) | Closed-form solution of `LunarLanderContinuous-v2` |
| [Code 10-3](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SQL_tf.html) | SQL agent (with TensorFlow) |
| [Code 10-4](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SQL_torch.html) | SQL agent (with PyTorch) |
| [Code 10-5](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SACwoA_tf.html) | SAC agent (with TensorFlow) |
| [Code 10-6](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SACwoA_torch.html) | SAC agent (with PyTorch) |
| [Code 10-7](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SACwA_tf.html) | SAC with automatic entropy adjustment (with TensorFlow) |
| [Code 10-8](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLander-v2_SACwA_torch.html) | SAC with automatic entropy adjustment (with PyTorch) |
| [Code 10-9](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLanderContinuous-v2_SACwA_tf.html) | SAC with automatic entropy adjustment for continuous action space (with TensorFlow) |
| [Code 10-10](https://zhiqingxiao.github.io/rl-book/en2024/code/LunarLanderContinuous-v2_SACwA_torch.html) | SAC with automatic entropy adjustment for continuous action space (with PyTorch) |
| [Code 11-1](https://zhiqingxiao.github.io/rl-book/en2024/code/BipedalWalker-v3_ClosedForm.html) | Closed-form solution of `BipedalWalker-v3` |
| [Code 11-2](https://zhiqingxiao.github.io/rl-book/en2024/code/BipedalWalker-v3_ES.html) | ES agent |
| [Code 11-3](https://zhiqingxiao.github.io/rl-book/en2024/code/BipedalWalker-v3_ES.html) | Train and test ES agent |
| [Code 11-4](https://zhiqingxiao.github.io/rl-book/en2024/code/BipedalWalker-v3_ARS.html) | ARS agent |
| [Code 12-1](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_ClosedForm.html) | Closed-form solution of `PongNoFrameskip-v4` |
| [Code 12-2](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_CategoricalDQN_tf.html) | Wrapped environment class |
| [Code 12-3](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_CategoricalDQN_tf.html) | Categorical DQN agent (with TensorFlow) |
| [Code 12-4](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_CategoricalDQN_torch.html) | Categorical DQN agent (with PyTorch) |
| [Code 12-5](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_QRDQN_tf.html) | QR-DQN agent (with TensorFlow) |
| [Code 12-6](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_QRDQN_torch.html) | QR-DQN agent (with PyTorch) |
| [Code 12-7](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_IQN_tf.html) | Quantile network (with TensorFlow) |
| [Code 12-8](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_IQN_torch.html) | Quantile network (with PyTorch) |
| [Code 12-9](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_IQN_tf.html) | IQN agent (with TensorFlow) |
| [Code 12-10](https://zhiqingxiao.github.io/rl-book/en2024/code/PongNoFrameskip-v4_IQN_torch.html) | IQN agent (with PyTorch) |
| [Code 13-1](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | The environment class `BernoulliMABEnv` |
| [Code 13-2](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | Register the environment class `BernoulliMABEnv` into Gym |
| [Code 13-3](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | $\epsilon$-greedy policy agent |
| [Code 13-4](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | Evaluate average regret |
| [Code 13-5](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | UCB1 agent |
| [Code 13-6](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | Bayesian UCB agent |
| [Code 13-7](https://zhiqingxiao.github.io/rl-book/en2024/code/BernoulliMABEnv-v0_demo.html) | Thompson sampling agent |
| [Code 14-1](https://github.com/ZhiqingXiao/boardgame2/blob/master/boardgame2/env.py) | The constructor of the class `BoardGameEnv` |
| [Code 14-2](https://github.com/ZhiqingXiao/boardgame2/blob/master/boardgame2/env.py) | The member function `is_valid()`, `has_valid()`, and `get_valid()` in the class `BoardGameEnv` |
| [Code 14-3](https://github.com/ZhiqingXiao/boardgame2/blob/master/boardgame2/kinarow.py) | The member function `get_winner()` in the class `KInARowEnv` |
| [Code 14-4](https://github.com/ZhiqingXiao/boardgame2/blob/master/boardgame2/env.py) | The member function `next_step()` and `get_next_state()` in the class `BoardGameEnv` |
| [Code 14-5](https://github.com/ZhiqingXiao/boardgame2/blob/master/boardgame2/env.py) | The member function `reset()`, `step()`, and `render()` in the class `BoardGameEnv` |
| [Code 14-6](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_ExhaustiveSearch.html) | Exhaustive search agent |
| [Code 14-7](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_ExhaustiveSearch.html) | Self-play |
| [Code 14-8](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_AlphaZero_tf.html) | Replay buffer of AlphaZero agent |
| [Code 14-9](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_AlphaZero_tf.html) | Network of AlphaZero agent (with TensorFlow) |
| [Code 14-10](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_AlphaZero_torch.html) | Network of AlphaZero agent (with PyTorch) |
| [Code 14-11](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_AlphaZero_tf.html) | AlphaZero agent (with TensorFlow) |
| [Code 14-12](https://zhiqingxiao.github.io/rl-book/en2024/code/TicTacToe-v0_AlphaZero_torch.html) | AlphaZero agent (with PyTorch) |
| [Code 15-1](https://zhiqingxiao.github.io/rl-book/en2024/code/Tiger-v0_ClosedForm.html) | The environment class `TigerEnv` for the task “Tiger” |
| [Code 15-2](https://zhiqingxiao.github.io/rl-book/en2024/code/Tiger-v0_ClosedForm.html) | Register the environment class `TigerEnv` |
| [Code 15-3](https://zhiqingxiao.github.io/rl-book/en2024/code/Tiger-v0_ClosedForm.html) | Optimal policy when discounted factor $\gamma=1$ |
| [Code 15-4](https://zhiqingxiao.github.io/rl-book/en2024/code/Tiger-v0_Plan_demo.html) | Belief VI |
| [Code 15-5](https://zhiqingxiao.github.io/rl-book/en2024/code/Tiger-v0_Plan_demo.html) | PBVI |
| [Code 16-1](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_ClosedForm_demo.html) | Adjust the camera |
| [Code 16-2](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_ClosedForm_demo.html) | Visualize the interaction with the environment |
| [Code 16-3](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_BC_tf.html) | Experience replayer for state–action pairs |
| [Code 16-4](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_BC_tf.html) | BC agent (with TensorFlow) |
| [Code 16-5](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_BC_torch.html) | BC agent (with PyTorch) |
| [Code 16-6](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_GAILPPO_tf.html) | GAIL-PPO agent (with TensorFlow) |
| [Code 16-7](https://zhiqingxiao.github.io/rl-book/en2024/code/HumanoidBulletEnv-v0_GAILPPO_torch.html) | GAIL-PPO agent (with PyTorch) |
