# RLPyTorch: A Simple Reinforcement Learning Package in PyTorch

Overview    
==============
Here we provide a simple reinforcement learning package as a backend for ELF. In the root directory there is basic framework for Rlpytorch.

* `args_provider.py`
Provide basic argument management mechanism. In RLPyTorch, each component (methods, model loader, etc) has its own argument set.  
Note that `args_utils.py` is a symlink for backward compatibility.

* `model_base.py`
Wrapper of `nn.Module`, providing save/load/counting steps etc.

* `model_interface.py`
Wrapper of multiple models.

* `model_loader.py`
Utility for loading saved models.

In the subfolders there are multiple components in RLPyTorch:

1. `methods`
Basic RL methods, e.g., PolicyGradient, ActorCritic, etc. Some methods are constructed from other basic ones. For example, `ActorCritic = PolicyGradient + DiscountedReward + ValueMatcher`. This makes the library easier to extend. For example, you might replace `DiscountedReward` with the final reward, which becomes REINFORCE. You might also replace it with Generalized Advantage Estimation (GAE) or $\lambda$-return, etc.  

2. `trainer`
Provider example callback functions for ELF framework. A simple example looks like the following:
```
GC.Start()
GC.reg_callback("train", train_callback)
GC.reg_callback("actor", actor_callback)
while True:
    GC.Run()
GC.Stop()
```
The `GC.Run()` function waits until the next batch with a specific tag arrives, then call registered callback functions.  

3. `runner`
Customize how to run the loop of `GC`. E.g., whether to run it with progress bar, in a single process or multiple processes, etc.

4. `sampler`
Different ways to sample the actor model (to get the next action).   

5. `stats`
Supporting classes to collect various statistics.

6. `utils`
Internal utilities.  


Actor Critic model  
-------------
We implemented advantageous actor-critic models, similar to Vanilla A3C, but with off-policy corrections with importance sampling.

Specifically, we use the trajectory sampled from the previous version of the actor in the actor-critic update.
