## Language Modeling

#### 2020

| Title | Summary | Paper Source | TL;DR |
| ----- | ------- | ----- | ---- |
| Turing-NLG: A 17-billion-parameter language model by Microsoft | [GitHub](https://github.com/dair-ai/nlp_paper_summaries/blob/master/Language%20Modeling/turing-nlg.md) | [Source](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling)
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) | [Paper](https://arxiv.org/abs/1910.10683) | [GitHub](https://github.com/dair-ai/nlp_paper_summaries/blob/master/Language%20Modeling/t5-text-to-text-transformer.md)
| Reformer: The Efficient Transformer | [Pragmatic ML](https://www.pragmatic.ml/reformer-deep-dive/) | [Paper](https://arxiv.org/abs/2001.04451) | [GitHub](https://github.com/dair-ai/nlp_paper_summaries/blob/master/Language%20Modeling/reformer.md)
| ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators | [Google AI Blog](https://ai.googleblog.com/2020/03/more-efficient-nlp-model-pre-training.html) [Summary](https://medium.com/dair-ai/bert-is-extremely-inefficient-this-is-how-to-solve-it-688b09350f10) | [Paper](https://openreview.net/pdf?id=r1xMH1BtvB) | [GitHub](https://github.com/dair-ai/nlp_paper_summaries/blob/master/Language%20Modeling/electra.md)
| REALM: Retrieval-Augmented Language Model Pre-Training | - | [Paper](https://kentonl.com/pub/gltpc.2020.pdf) | -
| Longformer: The Long-Document Transformer | [Medium](https://medium.com/@viktor2karlsson/longformer-what-bert-should-have-been-78f4cd595be9?source=friends_link&sk=706c3dc289d5a7ee745a54f46225e90d) | [Paper](https://arxiv.org/abs/2004.05150) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling)

#### 2019

| Title | Summary | Paper Source | TL;DR |
| ----- | ------- | ----- | ----- |
| Language Models are Unsupervised Multitask Learners | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) | [Paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) | [Richard Csaky](https://github.com/ricsinaruto/Seq2seqChatbots/wiki/Chatbot-and-Related-Research-Paper-Notes-with-Images#language-models-are-unsupervised-multitask-learners-s2s)
| Plug and Play Language Models: A Simple Approach to Controlled Text Generation | [Uber Engineering](https://eng.uber.com/pplm/) | [Paper](https://arxiv.org/abs/1912.02164)| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| ALBERT: A Lite BERT for Self-Supervised Learning Of Language Representations | [Amit Chaudhary](https://amitness.com/2020/02/albert-visual-summary/) | [Paper](https://arxiv.org/abs/1909.11942)| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| Fine-Tuning GPT-2 from Human Preferences | [Open AI](https://openai.com/blog/fine-tuning-gpt-2/) | [Paper](https://arxiv.org/abs/1909.08593)| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| XLNet: Generalized Autoregressive Pretraining for Language Understanding | [dair.ai](https://medium.com/dair-ai/xlnet-outperforms-bert-on-several-nlp-tasks-9ec867bb563b) | [Paper](https://arxiv.org/abs/1906.08237)| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context | [dair.ai](https://medium.com/dair-ai/a-light-introduction-to-transformer-xl-be5737feb13) | [Paper](https://arxiv.org/abs/1901.02860) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| Parameter-Efficient Transfer Learning for NLP | [dair.ai](https://medium.com/dair-ai/adapters-a-compact-and-extensible-transfer-learning-method-for-nlp-6d18c2399f62) | [Paper](https://arxiv.org/abs/1902.00751) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter | [dair.ai](https://dair.ai/TL;DR_DistillBERT/) | [Paper](https://arxiv.org/abs/1910.01108) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |
| TinyBERT: Distilling BERT for Natural Language Understanding | [Summary](https://medium.com/dair-ai/tinybert-size-does-matter-but-how-you-train-it-can-be-more-important-a5834831fa7d) | [Paper](https://arxiv.org/abs/1909.10351) | -
| Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks | [Summary](https://medium.com/dair-ai/tl-dr-sentencebert-8dec326daf4e) | [Paper](https://arxiv.org/pdf/1908.10084.pdf) | -


#### 2018

| Title | Summary | Paper Source | TL;DR |
| ----- | ------- | ----- | ---- |
| BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding| [Summary](https://medium.com/dair-ai/a-light-introduction-to-bert-2da54f96b68c) | [Paper](https://arxiv.org/pdf/1810.04805.pdf) | [Richard Csaky](https://github.com/ricsinaruto/Seq2seqChatbots/wiki/Chatbot-and-Related-Research-Paper-Notes-with-Images#bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding-s2s)
| Language Models are Unsupervised Multitask Learners| [Jay Alammar](http://jalammar.github.io/illustrated-gpt2/) | [Paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling)

#### 2017

| Title |  Summary | Paper Source | TL;DR |
| ----- | ------- | ----- | ---- |
| Adversarial Generation of Natural Language| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) |[Paper](https://arxiv.org/pdf/1705.10929.pdf) | [Richard Csaky](https://github.com/ricsinaruto/Seq2seqChatbots/wiki/Chatbot-and-Related-Research-Paper-Notes-with-Images#adversarial-generation-of-natural-language-n-c)
| Attention Is All You Need| [Jay Alammar](http://jalammar.github.io/illustrated-transformer/) [Summary](https://medium.com/dair-ai/summary-attention-is-all-you-need-the-transformer-architecture-eb76e5f8e6fb) |[Paper](https://arxiv.org/pdf/1706.03762.pdf) | [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling)
| Training RNNs as Fast as CNNs| [Contribute ✍️](https://github.com/dair-ai/nlp_paper_summaries/new/master/Language%20Modeling) | [Paper](https://arxiv.org/pdf/1709.02755.pdf)  | [Richard Csaky](https://github.com/ricsinaruto/Seq2seqChatbots/wiki/Chatbot-and-Related-Research-Paper-Notes-with-Images#training-rnns-as-fast-as-cnns-n-c)
