.. _ppo1:

.. automodule:: stable_baselines.ppo1


PPO1
====

The `Proximal Policy Optimization <https://arxiv.org/abs/1707.06347>`_ algorithm combines ideas from A2C (having multiple workers)
and TRPO (it uses a trust region to improve the actor).

The main idea is that after an update, the new policy should be not too far from the ``old`` policy.
For that, ppo uses clipping to avoid too large update.

.. note::

  PPO1 requires :ref:`OpenMPI <openmpi>`. If OpenMPI isn't enabled, then PPO1 isn't
  imported into the ``stable_baselines`` module.

.. note::

  PPO1 uses MPI for multiprocessing unlike PPO2, which uses vectorized environments.
  PPO2 is the implementation OpenAI made for GPU.

Notes
-----

-  Original paper:  https://arxiv.org/abs/1707.06347
- Clear explanation of PPO on Arxiv Insights channel: https://www.youtube.com/watch?v=5P7I-xPq8u8
-  OpenAI blog post: https://blog.openai.com/openai-baselines-ppo/
- ``mpirun -np 8 python -m stable_baselines.ppo1.run_atari`` runs the algorithm for 40M frames = 10M timesteps on an Atari game. See help (``-h``) for more options.
- ``python -m stable_baselines.ppo1.run_mujoco`` runs the algorithm for 1M frames on a Mujoco environment.
- Train mujoco 3d humanoid (with optimal-ish hyperparameters): ``mpirun -np 16 python -m stable_baselines.ppo1.run_humanoid --model-path=/path/to/model``
- Render the 3d humanoid: ``python -m stable_baselines.ppo1.run_humanoid --play --model-path=/path/to/model``

Can I use?
----------

-  Recurrent policies: ❌
-  Multi processing: ✔️ (using MPI)
-  Gym spaces:


============= ====== ===========
Space         Action Observation
============= ====== ===========
Discrete      ✔️      ✔️
Box           ✔️      ✔️
MultiDiscrete ✔️      ✔️
MultiBinary   ✔️      ✔️
============= ====== ===========


Example
-------

.. code-block:: python

  import gym

  from stable_baselines.common.policies import MlpPolicy
  from stable_baselines import PPO1

  env = gym.make('CartPole-v1')

  model = PPO1(MlpPolicy, env, verbose=1)
  model.learn(total_timesteps=25000)
  model.save("ppo1_cartpole")

  del model # remove to demonstrate saving and loading

  model = PPO1.load("ppo1_cartpole")

  obs = env.reset()
  while True:
      action, _states = model.predict(obs)
      obs, rewards, dones, info = env.step(action)
      env.render()


Parameters
----------

.. autoclass:: PPO1
  :members:
  :inherited-members:


Callbacks - Accessible Variables
--------------------------------

Depending on initialization parameters and timestep, different variables are accessible.
Variables accessible "From timestep X" are variables that can be accessed when
``self.timestep==X`` in the ``on_step`` function.

    +--------------------------------+-----------------------------------------------------+
    |Variable                        |                                         Availability|
    +================================+=====================================================+
    |- self                          |From timestep 0                                      |
    |- total_timesteps               |                                                     |
    |- callback                      |                                                     |
    |- log_interval                  |                                                     |
    |- tb_log_name                   |                                                     |
    |- reset_num_timesteps           |                                                     |
    |- new_tb_log                    |                                                     |
    |- writer                        |                                                     |
    |- policy                        |                                                     |
    |- env                           |                                                     |
    |- horizon                       |                                                     |
    |- reward_giver                  |                                                     |
    |- gail                          |                                                     |
    |- step                          |                                                     |
    |- cur_ep_ret                    |                                                     |
    |- current_it_len                |                                                     |
    |- current_ep_len                |                                                     |
    |- cur_ep_true_ret               |                                                     |
    |- ep_true_rets                  |                                                     |
    |- ep_rets                       |                                                     |
    |- ep_lens                       |                                                     |
    |- observations                  |                                                     |
    |- true_rewards                  |                                                     |
    |- rewards                       |                                                     |
    |- vpreds                        |                                                     |
    |- episode_starts                |                                                     |
    |- dones                         |                                                     |
    |- actions                       |                                                     |
    |- states                        |                                                     |
    |- episode_start                 |                                                     |
    |- done                          |                                                     |
    |- vpred                         |                                                     |
    |- _                             |                                                     |
    |- i                             |                                                     |
    |- clipped_action                |                                                     |
    |- reward                        |                                                     |
    |- true_reward                   |                                                     |
    |- info                          |                                                     |
    |- action                        |                                                     |
    |- observation                   |                                                     |
    +--------------------------------+-----------------------------------------------------+
    |- maybe_ep_info                 |After the first episode termination                  |
    +--------------------------------+-----------------------------------------------------+
