# On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting

<img src="https://i.imgur.com/pczVKAq.png">

This directory contains source code accompanying the paper [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://arxiv.org/abs/2206.00761) (NeurIPS 2022).

## Contributors

<img width=35px src="https://tomekkorbak.com/images/fotka.jpg"> [Tomasz Korbak](https://tomekkorbak.com), tomasz.korbak@gmail.com

<img width=35px src="https://i.imgur.com/YLwtSt1.png"> [Hady Elsahar](http://hadyelsahar.io), hady.elsahar@naverlabs.com

<img width=35px src="https://pbs.twimg.com/profile_images/1268967379758915593/q0ofGxv9_400x400.jpg"> [Germán Kruszewski](https://germank.github.io/), german.kruszewski@naverlabs.com

<img width=35px src="https://i.imgur.com/gqHaXeG.png"> [Marc Dymetman](https://scholar.google.com/citations?user=D6J5pooAAAAJ&hl=en), marc.dymetman@naverlabs.com


## Citation

```bibtex
@inproceedings{
korbak2022on,
title={On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting},
author={Tomasz Korbak and Hady Elsahar and Germ{\'a}n Kruszewski and Marc Dymetman},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=XvI6h-s4un}
}
```
