---
title: Recurrent Neural Networks
number: 4
---

RNNs ([Elman, 1990](https://crl.ucsd.edu/~elman/Papers/fsit.pdf)) use the idea of processing sequential information. 
The term "recurrent" applies as they perform the same task over each instance of the sequence such that the output is dependent on the previous computations and results. Generally, a fixed-size vector is produced to represent a sequence by feeding tokens one by one to a recurrent unit. In a way, RNNs have "memory" over previous computations and use this information in current processing. This template is naturally suited for many NLP tasks such as language modeling (Mikolov et al., [2010](http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf), [2011](https://ieeexplore.ieee.org/document/5947611); [Sutskever et al., 2011](https://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf)), machine translation ([Liu et al., 2014](http://www.aclweb.org/anthology/P14-1140.pdf); [Auli et al., 2013](http://www.aclweb.org/anthology/D13-1106); [Sutskever et al., 2014](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf)), speech recognition ([Robinson et al., 1996](https://pdfs.semanticscholar.org/03bc/854feaee144b54924b440eff02ed9082cc6b.pdf); [Graves et al., 2013](https://arxiv.org/abs/1303.5778); [Graves and Jaitly, 2014](http://proceedings.mlr.press/v32/graves14.pdf); [Sak et al., 2014](https://arxiv.org/abs/1402.1128)), and image captioning ([Karpathy and Fei-Fei, 2015](https://cs.stanford.edu/people/karpathy/cvpr2015.pdf)). This made RNNs increasingly popular for NLP applications in recent years. 

## A. Need for Recurrent Networks

In this section, we analyze the fundamental properties that favored the popularization of RNNs in a multitude of NLP tasks. Given that an RNN performs sequential processing by modeling units in sequence, it has the ability to capture the inherent sequential nature present in language, where units are characters, words or even sentences. Words in a language develop their semantical meaning based on the previous words in the sentence. A simple example stating this would be the difference in meaning between "dog" and "hot dog". RNNs are tailor-made for modeling such context dependencies in language and similar sequence modeling tasks, which resulted to be a strong motivation for researchers to use RNNs over CNNs in these areas. 

Another factor aiding RNN's suitability for sequence modeling tasks lies in its ability to model variable length of text, including very long sentences, paragraphs and even documents ([Tang et al., 2015](http://aclweb.org/anthology/D15-1167)). Unlike CNNs, RNNs have flexible computational steps that provide better modeling capability and create the possibility to capture unbounded context. This ability to handle input of arbitrary length became one of the selling points of major works using RNNs ([Chung et al., 2014](https://arxiv.org/abs/1412.3555)).

Many NLP tasks require semantic modeling over the whole sentence. This involves creating a gist of the sentence in a fixed dimensional hyperspace. RNN's ability to summarize sentences led to their increased usage for tasks like machine translation ([Cho et al., 2014](https://arxiv.org/abs/1406.1078)) where the whole sentence is summarized to a fixed vector and then mapped back to the variable-length target sequence.

RNN also provides the network support to perform time distributed joint processing. Most of the sequence labeling tasks like POS tagging ([Santos and Zadrozny, 2014](http://proceedings.mlr.press/v32/santos14.pdf)) come under this domain. More specific use cases include applications such as multi-label text categorization ([Chen et al., 2017](https://ieeexplore.ieee.org/document/7966144)), multimodal sentiment analysis ([Poria et al., 2017](http://www.aclweb.org/anthology/P17-1081); [Zadeh et al., 2017](http://aclweb.org/anthology/D17-1115); [Tong et al., 2017](https://aclanthology.coli.uni-saarland.de/papers/P17-1142/p17-1142)), and subjectivity detection ([Chaturvedi et al., 2017](http://sentic.net/bayesian-elm-for-subjectivity-detection.pdf)).

The above points enlist some of the focal reasons that motivated researchers to opt for RNNs. However, it would be gravely wrong to make conclusions on the superiority of RNNs over other deep networks. Recently, several works provided contrasting evidence on the superiority of CNNs over RNNs. Even in RNN-suited tasks like language modeling, CNNs achieved competitive performance over RNNs ([Dauphin et al., 2016](https://arxiv.org/abs/1612.08083)). Both CNNs and RNNs have different objectives when modeling a sentence. While RNNs try to create a composition of an arbitrarily long sentence along with unbounded context, CNNs try to extract the most important n-grams. Although they prove an effective way to capture n-gram features, which is approximately sufficient in certain sentence classification tasks, their sensitivity to word order is restricted locally and long-term dependencies are typically ignored. 

[Yin et al. (2017)](https://arxiv.org/abs/1702.01923) provided interesting insights on the comparative performance between RNNs and CNNs. After testing on multiple NLP tasks that included sentiment classification, QA, and POS tagging, they concluded that there is no clear winner: the performance of each network depends on the global semantics required by the task itself. 

Below, we discuss some of the RNN models extensively used in the literature. 

## B. RNN models

### 1. Simple RNN

In the context of NLP, RNNs are primarily based on Elman network ([Elman, 1990](https://crl.ucsd.edu/~elman/Papers/fsit.pdf)) and they are originally three-layer networks. Figure 9 illustrates a more general RNN which is unfolded across time to accommodate a whole sequence. In the figure, $$x_t$$ is taken as the input to the network at time step $$t$$ and $$s_t$$ represents the hidden state at the same time step. Calculation of $$s_t$$ is based as per the equation:

$$
{\bf s_t} = f( U{\bf x_t} + W{\bf s_{t-1}} )
$$

Thus, $$s_t$$ is calculated based on the current input and the previous time step's hidden state. The function $$f$$ is taken to be a non-linear transformation such as $$tanh, ReLU$$ and $$U, V, W$$ account for weights that are shared across time. In the context of NLP, $$x_t$$ typically comprises of one-hot encodings or embeddings. At times, they can also be abstract representations of textual content. $$o_t$$ illustrates the output of the network which is also often subjected to non-linearity, especially when the network contains further layers downstream. 

![alt txt](img/elmanrnn_Page_1.png)

*Figure 9: Simple RNN network (Figure Source: [Le Cun et al. (2015)](https://www.nature.com/articles/nature14539))*

The hidden state of the RNN is typically considered to be its most crucial element. As stated before, it can be considered as the network's memory element that accumulates information from other time steps. In practice, however, these simple RNN networks suffer from the infamous *vanishing gradient* problem, which makes it really hard to learn and tune the parameters of the earlier layers in the network.

This limitation was overcome by various networks such as long short-term memory (LSTM), gated recurrent units (GRUs), and residual networks (ResNets), where the first two are the most used RNN variants in NLP applications.

### 2. Long Short-Term Memory

LSTM ([Hochreiter and Schmidhuber, 1997](https://www.bioinf.jku.at/publications/older/2604.pdf); [Gers et al., 1999](https://www.semanticscholar.org/paper/Learning-to-Forget%3A-Continual-Prediction-with-LSTM-Gers-Schmidhuber/11540131eae85b2e11d53df7f1360eeb6476e7f4)) (Figure 10) has additional "forget" gates over the simple RNN. Its unique mechanism enables it to overcome both the *vanishing* and *exploding* gradient problem.

![alt txt](img/LSTMGRU.png)

*Figure 10: Illustration of an LSTM and GRU gate (Figure Source: [Chung et al. (2014)](https://arxiv.org/abs/1412.3555))*

Unlike the vanilla RNN, LSTM allows the error to back-propagate through unlimited number of time steps. Consisting of three gates: input, forget and output gates, it calculates the hidden state by taking a combination of these three gates as per the equations below: 

$$
\begin{gathered}
\small
{\bf x} = \left[ {\begin{array}{c}
	h_{t-1}\\
	x_t \\
	\end{array} } \right]\\
f_t = \sigma(W_f . {\bf x} + b_f)\\
i_t = \sigma(W_i . {\bf x} + b_i) \\
o_t = \sigma(W_o . {\bf x} + b_o)\\
c_t = f_t \odot c_{t-1} + i_t \odot \tanh(W_c . {\bf x} + b_c) \\
h_t = o_t \odot \tanh(c_t)
\end{gathered}
$$

### 3. Gated Recurrent Units

Another gated RNN variant called GRU ([Cho et al., 2014](https://arxiv.org/abs/1406.1078)) (Figure 10}) of lesser complexity was invented with empirically similar performances to LSTM in most tasks. GRU comprises of two gates, reset gate and update gate, and handles the flow of information like an LSTM sans a memory unit. Thus, it exposes the whole hidden content without any control. Being less complex, GRU can be a more efficient RNN than LSTM. The working of GRU is as follows:

$$
\begin{gathered}
{\bf z_t} = \sigma(U_z . {\bf x_t} + W_z . {\bf h_{t-1}})\\
{\bf r_t} = \sigma(U_r . {\bf x_t} + W_r . {\bf h_{t-1}})\\
{\bf s_t} = \tanh(U_s . {\bf x_t} + W_s .( \bf h_{t-1} \odot {\bf r_t}) )\\
{\bf h_t} = (1 - {\bf z_t}) \odot {\bf s_t} + {\bf z_t} \odot {\bf h_{t-1}} 
\end{gathered}
$$

Researchers often face the dilemma of choosing the appropriate gated RNN. This also extends to developers working in NLP. Throughout the history, most of the choices over the RNN variant tended to be heuristic. [Chung et al. (2014)](https://arxiv.org/abs/1412.3555) did a critical comparative evaluation of the three RNN variants mentioned above, although not on NLP tasks. They evaluated their work on tasks relating to polyphonic music modeling and speech signal modeling. Their evaluation clearly demonstrated the superiority of the gated units (LSTM and GRU) over the traditional simple RNN (in their case, using $$tanh$$ activation) (Figure 11). However, they could not make any concrete conclusion about which of the two gating units was better. This fact has been noted in other works too and, thus, people often leverage on other factors like computing power while choosing between the two.

![alt txt](img/LSTMvsGRU.jpg)

*Figure 11: Learning curves for training and validation sets of different types of units with respect to (top) the number of iterations and (bottom) the wall clock time. y-axis corresponds to the negative log	likelihood of the model shown in log-scale (Figure source: [Chung et al. (2014)](https://arxiv.org/abs/1412.3555))*

## C. Applications

### 1. RNN for word-level classification

RNNs have had a huge presence in the field of word-level classification. Many of their applications stand as state of the art in their respective tasks. [Lample et al. (2016)](https://arxiv.org/abs/1603.01360) proposed to use bidirectional LSTM for NER. The network captured arbitrarily long context information around the target word (curbing the limitation of a fixed window size) resulting in two fixed-size vector, on top of which another fully-connected layer was built. They used a CRF layer at last for the final entity tagging. 

RNNs have also shown considerable improvement in language modeling over traditional methods based on count statistics. Pioneering work in this field was done by [Graves (2013)](https://arxiv.org/abs/1308.0850), who introduced the effectiveness of RNNs in modeling complex sequences with long range context structures. He also proposed deep RNNs where multiple layers of hidden states were used to enhance the modeling. This work established the usage of RNNs on tasks beyond the context of NLP. Later, [Sundermeyer et al. (2015)](https://ieeexplore.ieee.org/document/7050391) compared the gain obtained by replacing a feed-forward neural network with an RNN when conditioning the prediction of a word on the words ahead. In their work, they proposed a typical hierarchy in neural network architectures where feed-forward neural networks gave considerable improvement over traditional count-based language models, which in turn were superseded by RNNs and later by LSTMs. An important point that they mentioned was the applicability of their conclusions to a variety of other tasks such as statistical machine translation ([Sundermeyer et al., 2014](https://aclanthology.coli.uni-saarland.de/papers/D14-1003/d14-1003)).

### 2. RNN for sentence-level classification

[Wang et al. (2015)](http://www.aclweb.org/anthology/P15-1130) proposed encoding entire tweets with LSTM, whose hidden state is used for predicting sentiment polarity. This simple strategy proved competitive to the more complex DCNN structure by [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062) designed to endow CNN models with ability to capture long-term dependencies. In a special case studying negation phrase, the authors also showed that the dynamics of LSTM gates can capture the reversal effect of the word *not*. 

Similar to CNN, the hidden state of an RNN can also be used for semantic matching between texts. In dialogue systems, [Lowe et al. (2015)](https://arxiv.org/abs/1506.08909) proposed to match a message with candidate responses with Dual-LSTM, which encodes both as fixed-size vectors and then measure their inner product as the basis to rank candidate responses. 

### 3. RNN for generating language

A challenging task in NLP is generating natural language, which is another natural application of RNNs. Conditioned on textual or visual data, deep LSTMs have been shown to generate reasonable task-specific text in tasks such as machine translation, image captioning, etc. In such cases, the RNN is termed a decoder. 

In ([Sutskever et al., 2014](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf)), the authors proposed a general deep LSTM encoder-decoder framework that maps a sequence to another sequence. One LSTM is used to encode the ``source'' sequence as a fixed-size vector, which can be text in the original language (machine translation), the question to be answered (QA) or the message to be replied to (dialogue systems). The vector is used as the initial state of another LSTM, named the decoder. During inference, the decoder generates tokens one by one, while updating its hidden state with the last generated token. Beam search is often used to approximate the optimal sequence. 

[Sutskever et al. (2014)](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf) experimented with 4-layer LSTM on a machine translation task in an end-to-end fashion, showing competitive results. In ([Vinyals and Le, 2015](https://arxiv.org/abs/1506.05869)), the same encoder-decoder framework is employed to model human conversations. When trained on more than 100 million message-response pairs, the LSTM decoder is able to generate very interesting responses in the open domain. It is also common to condition the LSTM decoder on additional signal to achieve certain effects. In ([Li et al., 2016](https://arxiv.org/abs/1603.06155)), the authors proposed to condition the decoder on a constant persona vector that captures the personal information of an individual speaker. In the above cases, language is generated based mainly on the semantic vector representing textual input. Similar frameworks have also been successfully used in image-based language generation, where visual features are used to condition the LSTM decoder (Figure 12). 

Visual QA is another task that requires language generation based on both textual and visual clues. [Malinowski et al. (2015)](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Malinowski_Ask_Your_Neurons_ICCV_2015_paper.pdf) were the first to provide an end-to-end deep learning solution where they predicted the answer as a set of words conditioned on the input image modeled by a CNN and text modeled by an LSTM (Figure 13). 

![alt txt](img/image_caption.png)

*Figure 12: LSTM decoder combined with a CNN image embedder to generate image captioning (Figure source: [Vinyals et al. (2015)](https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Vinyals_Show_and_Tell_2015_CVPR_paper.pdf))*

[Kumar er al. (2015)](https://arxiv.org/abs/1506.07285) tackled this problem by proposing an elaborated network termed dynamic memory network (DMN), which had four sub-modules. The idea was to repeatedly attend to the input text and image to form episodes of information improved at each iteration. Attention networks were used for fine-grained focus on input text phrases.

![alt txt](img/Neural-image-QA.png)

*Figure 13: Neural-image QA (Figure source: [Malinowski et al. (2015)](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Malinowski_Ask_Your_Neurons_ICCV_2015_paper.pdf))*

## D. Attention Mechanism

One potential problem that the traditional encoder-decoder framework faces is that the encoder at times is forced to encode information which might not be fully relevant to the task at hand. The problem arises also if the input is long or very information-rich and selective encoding is not possible. 

For example, the task of text summarization can be cast as a sequence-to-sequence learning problem, where the input is the original text and the output is the condensed version. Intuitively, it is unrealistic to expect a fixed-size vector to encode all information in a piece of text whose length can potentially be very long. Similar problems have also been reported in machine translation ([Bahdanau et al., 2014](https://arxiv.org/abs/1409.0473)).

In tasks such as text summarization and machine translation, certain alignment exists between the input text and the output text, which means that each token generation step is highly related to a certain part of the input text. This intuition inspires the attention mechanism. This mechanism attempts to ease the above problems by allowing the decoder to refer back to the input sequence. Specifically during decoding, in addition to the last hidden state and generated token, the decoder is also conditioned on a "context" vector calculated based on the input hidden state sequence.

[Bahdanau et al. (2014)](https://arxiv.org/abs/1409.0473) first applied the attention mechanism to machine translation, which improved the performance especially for long sequences. In their work, the attention signal over the input hidden state sequence is determined with a multi-layer perceptron by the last hidden state of the decoder. By visualizing the attention signal over the input sequence during each decoding step, a clear alignment between the source and target language can be demonstrated (Figure 14). 

![alt txt](img/bahdanau.jpg)

*Figure 14: Word alignment matrix (Figure source: [Bahdanau et al. (2014)](https://arxiv.org/abs/1409.0473))*

A similar approach was applied to the task of summarization by [Rush et al. (2015)](https://arxiv.org/abs/1509.00685) where each output word in the summary was conditioned on the input sentence through an attention mechanism. The authors performed abstractive summarization which is not very conventional as opposed to extractive summarization, but can be scaled up to large data with minimal linguistic input. 

In image captioning, [Xu et al. (2015)](http://proceedings.mlr.press/v37/xuc15.html) conditioned the LSTM decoder on different parts of the input image during each decoding step. Attention signal was determined by the previous hidden state and CNN features. In ([Vinyals et al., 2015](https://papers.nips.cc/paper/5635-grammar-as-a-foreign-language.pdf)), the authors casted the syntactical parsing problem as a sequence-to-sequence learning task by linearizing the parsing tree. The attention mechanism proved to be more data-efficient in this work. A further step in referring to the input sequence was to directly copy words or sub-sequences of the input onto the output sequence under a certain condition ([Vinyals et al., 2015](https://papers.nips.cc/paper/5866-pointer-networks)), which was useful in tasks such as dialogue generation and text summarization. Copying or generation was chosen at each time step during decoding ([Paulus et al. (2017)](https://arxiv.org/abs/1705.04304)).

In aspect-based sentiment analysis, [Wang et al. (2016)](https://aclanthology.coli.uni-saarland.de/papers/D16-1058/d16-1058) proposed an attention-based solution where they used aspect embeddings to provide additional support during classification (Figure 15). The attention module focused on selective regions of the sentence which affected the aspect to be classified. This can be seen in Figure 16 where, for the aspect *service* in (a), the attention module dynamically focused on the phrase "fastest delivery times" and in (b) with the aspect *food*, it identified multiple key-points across the sentence that included "tasteless" and "too sweet". Recently, [Ma et al. (2018)](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16541) augmented LSTM with a hierarchical attention mechanism consisting of a target-level attention and a sentence-level attention to exploit commonsense knowledge for targeted aspect-based sentiment analysis.

![alt txt](img/aspectattention.png)

*Figure 15: Aspect classification using attention (Figure source: [Wang et al. (2016)](https://aclanthology.coli.uni-saarland.de/papers/D16-1058/d16-1058))*

![alt txt](img/heatmap.png)

*Figure 16: Focus of attention module on the sentence for certain aspects (Figure source: [Wang et al. (2016)](https://aclanthology.coli.uni-saarland.de/papers/D16-1058/d16-1058))*

On the other hand, [Tang et al. (2016)](https://arxiv.org/abs/1605.08900) adopted a solution based on a memory network (also known as MemNet ([Weston et al., 2014](https://arxiv.org/abs/1410.3916))), which employed multiple-hop attention. The multiple attention computation layer on the memory led to improved lookup for most informational regions in the memory and subsequently aided the classification. Their work stands as the state of the art in this domain. 

Given the intuitive applicability of attention modules, they are still being actively investigated by NLP researchers and adopted for an increasing number of applications. 

## E. Parallelized Attention: The Transformer

Both CNNs and RNNs have been crucial in sequence transduction applications involving the encoder-decoder architecture. Attention-based mechanisms, as described above, have further boosted the capabilities of these models. However, one of the bottlenecks suffered by these architectures is the sequential processing at the encoding step. To address this, [Vaswani et al. (2017)](https://arxiv.org/abs/1706.03762) proposed the *Transformer* which dispensed the recurrence and convolutions involved in the encoding step entirely and based models only on attention mechanisms to capture the global relations between input and output. As a result, the overall architecture became more parallelizable and required lesser time to train along with positive results on tasks ranging from translation to parsing.

![alt txt](img/mh_attention.png)

*Figure 17: Multi-head Attention: [Vaswani et al. (2017)](https://arxiv.org/abs/1706.03762)*

The Transformer consists stacked layers in both encoder and decoder components. Each layer has two sub-layers comprising *multi-head attention layer* (Figure 17) followed by a position-wise feed forward network. For set of queries $$Q$$, keys $$K$$ and values $$V$$, the multi-head attention module performs attention $$h$$ times where the computation can be seen as:

$$
\begin{align}
    \text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \text{head}_2, \ldots, \text{head}_h) W^o \\
    \text{where}\;\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)\\
    \text{and}\;\text{Attention}(Q,K,V)= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
\end{align}
$$

here, $$W_i^{[.]}$$ and $$W^o$$ are projection parameters. Incorporating other techniques such as residual connections ([He et al., 2016](https://ieeexplore.ieee.org/document/7780459)), layer normalization ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)), dropouts, positional encodings, and others, the model achieves state-of-the-art results in English-German and English-French translation and constituency parsing.
