---
title: Recursive Neural Networks
number: 5
---

Recursive neural networks represent a natural way to model sequences. Arguably, however, language exhibits a natural recursive structure, where words and sub-phrases combine into phrases in a hierarchical manner. Such structure can be represented by a constituency parsing tree. Thus, tree-structured models have been used to better make use of such syntactic interpretations of sentence structure ([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)). Specifically, in a recursive neural network, the representation of each non-terminal node in a parsing tree is determined by the representations of all its children.

## A. Basic model

In this section, we describe the basic structure of recursive neural networks. As shown in Figure 17 and 18, the network $$g$$ defines a compositional function on the representations of phrases or words ($$b, c$$ or $$a, p_1$$) to compute the representation of a higher-level phrase ($$p_1$$ or $$p_2$$). The representations of all nodes take the same form.

![alt txt](img/recursiveNN2.png)

*Figure 17: Recursive neural networks for phrase-level sentiment classification (Figure source: [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf))*

![alt txt](img/recursiveNN.png)

*Figure 18: Recursive neural networks iteratively form high-level representation from lower-level representations.*

In ([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)), the authors described multiple variations of this model. In its simplest form, $$g$$ is defined as: 

$$
\begin{gathered}
\small
{p_1} = tanh\left(W\left[ {\begin{array}{c}
	b\\
	c \\
	\end{array} } \right]\right), {p_2} = tanh\left(W\left[ {\begin{array}{c}
	a\\
	p_1 \\
	\end{array} } \right]\right)
\end{gathered}
$$

in which the representation for each node is a $$d$$-dimensional vector and $$W\in\mathcal{R}^{D\times{2D}}$$.

Another variation is the MV-RNN ([Socher et al., 2012](https://ai.stanford.edu/~ang/papers/emnlp12-SemanticCompositionalityRecursiveMatrixVectorSpaces.pdf)). The idea is to represent every word and phrase as both a matrix and a vector. When two constituents are combined, the matrix of one is multiplied with the vector of the other:

$$
\begin{gathered}
\small
{p_1} = tanh\left(W\left[ {\begin{array}{c}
	Cb\\
	Bc \\
	\end{array} } \right]\right), {P_1} = tanh\left(W_M\left[ {\begin{array}{c}
	B\\
	C \\
	\end{array} } \right]\right)
\end{gathered}
$$

in which $$b,c,p_1\in\mathcal{R}^{D}$$, $$B,C,P_1\in\mathcal{R}^{D\times{D}}$$, and $$W_M\in\mathcal{R}^{D\times{2D}}$$. Compared to the vanilla form, MV-RNN parameterizes the compositional function with matrices corresponding to the constituents.

The recursive neural tensor network (RNTN) is proposed to introduce more interaction between the input vectors without making the number of parameters exceptionally large like MV-RNN. RNTN is defined by:

$$
\begin{gathered}
\small
{p_1} = tanh\left(\left[ {\begin{array}{c}
	b\\
	c\\
	\end{array} } \right]^{T}V^{[1:D]}\left[ {\begin{array}{c}
	b\\
	c\\
	\end{array} } \right]+
  W\left[ {\begin{array}{c}
	b\\
	c\\
	\end{array} } \right]\right)
\end{gathered}
$$

where $$V\in\mathcal{R}^{2D\times{2D}\times{D}}$$ is a tensor that defines multiple bilinear forms.

## B. Applications

One natural application of recursive neural networks is parsing ([Socher et al., 2011](https://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf)). A scoring function is defined on the phrase representation to calculate the plausibility of that phrase. Beam search is usually applied for searching the best tree. The model is trained with the max-margin objective ([Taskar et al., 2004](https://papers.nips.cc/paper/2397-max-margin-markov-networks.pdf)).

Based on recursive neural networks and the parsing tree, [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)) proposed a phrase-level sentiment analysis framework (Figure 19), where each node in the parsing tree can be assigned a sentiment label.

![alt txt](img/but.png)

*Figure 19: Recursive neural networks applied on a sentence for sentiment classification. Note that "but" plays a crucial role on determining the sentiment of the whole sentence (Figure source: [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf))*

[Socher et al. (2012)](https://ai.stanford.edu/~ang/papers/emnlp12-SemanticCompositionalityRecursiveMatrixVectorSpaces.pdf) classified semantic relationships such as cause-effect or topic-message between nominals in a sentence by building a single compositional semantics for the minimal constituent including both terms. [Bowman et al. (2014)](https://arxiv.org/abs/1406.1827) proposed to classify the logical relationship between sentences with recursive neural networks. The representations for both sentences are fed to another neural network for relationship classification. They show that both vanilla and tensor versions of the recursive unit performed competitively in a textual entailment dataset.

To avoid the gradient vanishing problem, LSTM units have also been applied to tree structures in ([Tai et al., 2015](https://arxiv.org/abs/1503.00075)). The authors showed improved sentence representation over linear LSTM models, as clear improvement in sentiment analysis and sentence relatedness test was observed.