---
title: Introduction
number: 1
---

Natural language processing (NLP) is a theory-motivated range of computational techniques for the automatic analysis and representation of human language. NLP research has evolved from the era of punch cards and batch processing, in which the analysis of a sentence could take up to 7 minutes, to the era of Google and the likes of it, in which millions of webpages can be processed in less than a second ([Cambria and White, 2014](http://sentic.net/jumping-nlp-curves.pdf)). NLP enables computers to perform a wide range of natural language related tasks at all levels, ranging from parsing and part-of-speech (POS) tagging, to machine translation and dialogue systems. 

Deep learning architectures and algorithms have already made impressive advances in fields such as computer vision and pattern recognition. Following this trend, recent NLP research is now increasingly focusing on the use of new deep learning methods (see Figure 1). For decades, machine learning approaches targeting NLP problems have been based on shallow models (e.g., SVM and logistic regression) trained on very high dimensional and sparse features. In the last few years, neural networks based on dense vector representations have been producing superior results on various NLP tasks. This trend is sparked by the success of word embeddings (Mikolov et al., [2010](http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf), [2013a](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)) and deep learning methods ([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)). Deep learning enables multi-level automatic feature representation learning. In contrast, traditional machine learning based NLP systems liaise heavily on hand-crafted features. Such hand-crafted features are time-consuming and often incomplete. 

![alt txt](img/trend.png)
*Figure 1: Percentage of deep learning papers in ACL, EMNLP, EACL, NAACL over the last 6 years (long papers).*

[Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) demonstrated that a simple deep learning framework outperforms most state-of-the-art approaches in several NLP tasks such as named-entity recognition (NER), semantic role labeling (SRL), and POS tagging. Since then, numerous complex deep learning based algorithms have been proposed to solve difficult NLP tasks. We review major deep learning related models and methods applied to natural language tasks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and recursive neural networks. We also discuss memory-augmenting strategies, attention mechanisms and how unsupervised models, reinforcement learning methods and recently, deep generative models have been employed for language-related tasks. 

To the best of our knowledge, this work is the first of its type to comprehensively cover the most popular deep learning methods in NLP research today. The work by [Goldberg (2016)](https://arxiv.org/abs/1510.00726) only presented the basic principles for applying neural networks to NLP in a tutorial manner. We believe this paper will give readers a more comprehensive idea of current practices in this domain.

The structure of the paper is as follows: [Section 2](https://nlpoverview.com/#2) introduces the concept of distributed representation, the basis of sophisticated deep learning models; next, Sections [3](https://nlpoverview.com/#3), [4](https://nlpoverview.com/#4), and [5](https://nlpoverview.com/#5) discuss popular models such as convolutional, recurrent, and recursive neural networks, as well as their use in various NLP tasks; following, [Section 6](https://nlpoverview.com/#6) lists recent applications of reinforcement learning in NLP and new developments in unsupervised sentence representation learning; later, [Section 7](https://nlpoverview.com/#7) illustrates the recent trend of coupling deep learning models with memory modules; finally, [Section 8](https://nlpoverview.com/#8) summarizes the performance of a series of deep learning methods on standard datasets about major NLP topics.