.. _trpo:

.. automodule:: stable_baselines.trpo_mpi


TRPO
====

`Trust Region Policy Optimization (TRPO) <https://arxiv.org/abs/1502.05477>`_
is an iterative approach for optimizing policies with guaranteed monotonic improvement.

.. note::

  TRPO requires :ref:`OpenMPI <openmpi>`. If OpenMPI isn't enabled, then TRPO isn't
  imported into the ``stable_baselines`` module.

Notes
-----

-  Original paper:  https://arxiv.org/abs/1502.05477
-  OpenAI blog post: https://blog.openai.com/openai-baselines-ppo/
- ``mpirun -np 16 python -m stable_baselines.trpo_mpi.run_atari`` runs the algorithm
  for 40M frames = 10M timesteps on an Atari game. See help (``-h``) for more options.
- ``python -m stable_baselines.trpo_mpi.run_mujoco`` runs the algorithm for 1M timesteps on a Mujoco environment.

Can I use?
----------

-  Recurrent policies: ❌
-  Multi processing: ✔️  (using MPI)
-  Gym spaces:


============= ====== ===========
Space         Action Observation
============= ====== ===========
Discrete      ✔️      ✔️
Box           ✔️      ✔️
MultiDiscrete ✔️      ✔️
MultiBinary   ✔️      ✔️
============= ====== ===========


Example
-------

.. code-block:: python

  import gym

  from stable_baselines.common.policies import MlpPolicy
  from stable_baselines import TRPO

  env = gym.make('CartPole-v1')

  model = TRPO(MlpPolicy, env, verbose=1)
  model.learn(total_timesteps=25000)
  model.save("trpo_cartpole")

  del model # remove to demonstrate saving and loading

  model = TRPO.load("trpo_cartpole")

  obs = env.reset()
  while True:
      action, _states = model.predict(obs)
      obs, rewards, dones, info = env.step(action)
      env.render()


Parameters
----------

.. autoclass:: TRPO
  :members:
  :inherited-members:

Callbacks - Accessible Variables
--------------------------------

Depending on initialization parameters and timestep, different variables are accessible.
Variables accessible "From timestep X" are variables that can be accessed when
``self.timestep==X`` in the ``on_step`` function.

+--------------------------------+-----------------------------------------------------+
|Variable                        |                                         Availability|
+================================+=====================================================+
|- total_timesteps               |From timestep 0                                      |
|- callback                      |                                                     |
|- log_interval                  |                                                     |
|- tb_log_name                   |                                                     |
|- reset_num_timesteps           |                                                     |
|- new_tb_log                    |                                                     |
|- writer                        |                                                     |
|- self                          |                                                     |
|- policy                        |                                                     |
|- env                           |                                                     |
|- horizon                       |                                                     |
|- reward_giver                  |                                                     |
|- gail                          |                                                     |
|- step                          |                                                     |
|- cur_ep_ret                    |                                                     |
|- current_it_len                |                                                     |
|- current_ep_len                |                                                     |
|- cur_ep_true_ret               |                                                     |
|- ep_true_rets                  |                                                     |
|- ep_rets                       |                                                     |
|- ep_lens                       |                                                     |
|- observations                  |                                                     |
|- true_rewards                  |                                                     |
|- rewards                       |                                                     |
|- vpreds                        |                                                     |
|- episode_starts                |                                                     |
|- dones                         |                                                     |
|- actions                       |                                                     |
|- states                        |                                                     |
|- episode_start                 |                                                     |
|- done                          |                                                     |
|- vpred                         |                                                     |
|- clipped_action                |                                                     |
|- reward                        |                                                     |
|- true_reward                   |                                                     |
|- info                          |                                                     |
|- action                        |                                                     |
|- observation                   |                                                     |
|- maybe_ep_info                 |                                                     |
+--------------------------------+-----------------------------------------------------+
