---
title: Convolutional Neural Networks
number: 3
---

Following the popularization of word embeddings and its ability to represent words in a distributed space, the need arose for an effective feature function that extracts higher-level features from constituting words or n-grams. These abstract features would then be used for numerous NLP tasks such as sentiment analysis, summarization, machine translation, and question answering (QA). CNNs turned out to be the natural choice given their effectiveness in computer vision tasks ([Krizhevsky et al., 2012](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf); [Razavian et al., 2014](http://openaccess.thecvf.com/content_cvpr_workshops_2014/W15/papers/Razavian_CNN_Features_Off-the-Shelf_2014_CVPR_paper.pdf); [Jia et al., 2014](https://arxiv.org/abs/1408.5093)). 

![alt txt](img/collobertCNN.png)

*Figure 5: CNN framework used to perform word wise class prediction (Figure source: [Collobert and Weston (2008)](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf))*

The use of CNNs for sentence modeling traces back to [Collobert and Weston (2008)](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf). This work used multi-task learning to output multiple predictions for NLP tasks such as POS tags, chunks, named-entity tags, semantic roles, semantically-similar words and a language model. A look-up table was used to transform each word into a vector of user-defined dimensions. Thus, an input sequence $$\{s_1, s_2, ... s_n\}$$ of $$n$$ words was transformed into a series of vectors $$\{ {\bf w_{s_1}}, {\bf w_{s_2}}, ... {\bf w_{s_n}} \}$$ by applying the look-up table to each of its words (Figure 5). 

This can be thought of as a primitive word embedding method whose weights were learned in the training of the network. In ([Collobert et al., 2011](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)), Collobert extended his work to propose a general CNN-based framework to solve a plethora of NLP tasks. Both these works triggered a huge popularization of CNNs amongst NLP researchers. Given that CNNs had already shown their mettle for computer vision tasks, it was easier for people to believe in their performance. 

CNNs have the ability to extract salient n-gram features from the input sentence to create an informative latent semantic representation of the sentence for downstream tasks. This application was pioneered by [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf), [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062), [Kim (2014)](https://arxiv.org/abs/1408.5882), which led to a huge proliferation of CNN-based networks in the succeeding literature. Below, we describe the working of a simple CNN-based sentence modeling network: 

## A. Basic CNN

### 1. Sentence Modeling
For each sentence, let $${\bf w_{i}} \in \mathcal{R}^d$$ represent the word embedding for the $$i^{th}$$ word in the sentence, where $$d$$ is the dimension of the word embedding. Given that a sentence has $$n$$ words, the sentence can now be represented as an embedding matrix $${\bf W} \in \mathcal{R}^{n \times d}$$. Figure 6 depicts such a sentence as an input to the CNN framework.

![alt txt](img/CNN.jpg)

*Figure 6: CNN modeling on text (Figure source: [Zhang and Wallace (2015)](https://arxiv.org/abs/1510.03820))*

Let $${\bf w_{i:i+j}}$$ refer to the concatenation of vectors $${\bf w_{i}}, {\bf w_{i+1}}, ... {\bf w_{j}}$$. Convolution is performed on this input embedding layer. It involves a *filter* $${\bf k} \in \mathcal{R}^{hd}$$ which is applied to a window of $$h$$ words to produce a new feature. For example, a feature $$c_i$$ is generated using the window of words $${\bf w_{i:i+h-1}}$$ by

$$
c_i = f({\bf w_{i:i+h-1}}.{\bf k}^T + b )
$$

Here, $$b \in \mathcal{R}$$ is the bias term and $$f$$ is a non-linear activation function, for example the hyperbolic tangent. The filter $$k$$ is applied to all possible windows using the same weights to create the feature map.

$$
c = [c_1, c_2, ... , c_{n-h+1}]
$$

In a CNN, a number of convolutional filters, also called kernels (typically hundreds), of different widths slide over the entire word embedding matrix. Each kernel extracts a specific pattern of n-gram. A convolution layer is usually followed by a max-pooling strategy, $$\hat{c} = max\{c\}$$, which subsamples the input typically by applying a max operation on each filter. This strategy has two primary reasons.

Firstly, max pooling provides a fixed-length output which is generally required for classification. Thus, regardless the size of the filters, max pooling always maps the input to a fixed dimension of outputs. Secondly, it reduces the output's dimensionality while keeping the most salient n-gram features across the whole sentence. This is done in a translation invariant manner where each filter is now able to extract a particular feature (e.g., negations) from anywhere in the sentence and add it to the final sentence representation. 

The word embeddings can be initialized randomly or pre-trained on a large unlabeled corpora (as in [Section 2](https://nlpoverview.com/#2)). The latter option is sometimes found beneficial to performance, especially when the amount of labeled data is limited ([Kim, 2014](https://arxiv.org/abs/1408.5882)). This combination of convolution layer followed by max pooling is often stacked to create deep CNN networks. These sequential convolutions help in improved mining of the sentence to grasp a truly abstract representations comprising rich semantic information. The kernels through deeper convolutions cover a larger part of the sentence until finally covering it fully and creating a global summarization of the sentence features.

### 2. Window Approach

The above-mentioned architecture allows for modeling of complete sentences into sentence representations. However, many NLP tasks, such as NER, POS tagging, and SRL, require word-based predictions. To adapt CNNs for such tasks, a window approach is used, which assumes that the tag of a word primarily depends on its neighboring words. For each word, thus, a fixed-size window surrounding itself is assumed and the sub-sentence ranging within the window is considered. A standalone CNN is applied to this sub-sentence as explained earlier and predictions are attributed to the word in the center of the window. Following this approach, [Poria et al. (2016)](http://www.sentic.net/aspect-extraction-for-opinion-mining.pdf) employed a multi-level deep CNN to tag each word in a sentence as a possible aspect or non-aspect. Coupled with a set of linguistic patterns, their ensemble classifier managed to perform well in aspect detection. 

The ultimate goal of word-level classification is generally to assign a sequence of labels to the entire sentence. In such cases, structured prediction techniques such as conditional random field (CRF) are sometimes employed to better capture dependencies between adjacent class labels and finally generate cohesive label sequence giving maximum score to the whole sentence ([Kirillov et al., 2015](https://arxiv.org/pdf/1511.05067v2.pdf)).

To get a larger contextual range, the classic window approach is often coupled with a time-delay neural network (TDNN) ([Waibel et al., 1989](https://ieeexplore.ieee.org/document/21701)). Here, convolutions are performed across all windows throughout the sequence. These convolutions are generally constrained by defining a kernel having a certain width. Thus, while the classic window approach only considers the words in the window around the word to be labeled, TDNN considers all windows of words in the sentence at the same time. At times, TDNN layers are also stacked like CNN architectures to extract local features in lower layers and global features in higher layers ([Collobert et al., 2011](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)).

## B. Applications

In this section, we present some of the crucial works that employed CNNs on NLP tasks to set state-of-the-art benchmarks in their respective times.

[Kim (2014)](https://arxiv.org/abs/1408.5882) explored using the above architecture for a variety of sentence classification tasks, including sentiment, subjectivity and question type classification, showing competitive results. This work was quickly adapted by researchers given its simple yet effective network. After training for a specific task, the randomly initialized convolutional kernels became specific n-gram feature detectors that were useful for that target task (Figure 7) . This simple network, however, had many shortcomings with the CNN's inability to model long distance dependencies standing as the main issue.

![alt txt](img/Positive-negative.jpg)
![alt txt](img/not-too.png)

*Figure 7: Top 7-grams by four learned 7-gram kernels; each kernel is sensitive to a specific kind of 7-gram (Figure Source: [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062))*

This issue was partly handled by [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062), who published a prominent paper where they proposed a dynamic convolutional neural network (DCNN) for semantic modeling of sentences. They proposed dynamic k-max pooling strategy which, given a sequence $${\bf p}$$ selects the $$k$$ most active features. The selection preserved the order of the features but was insensitive to their specific positions (Figure 8). Built on the concept of TDNN, they added this dynamic k-max pooling strategy to create a sentence model. This combination allowed filters with small width to span across a long range within the input sentence, thus accumulating crucial information across the sentence. In the induced subgraph (Figure 8), higher order features had highly variable ranges that could be either short and focused or global and long as the input sentence. They applied their model on multiple tasks, including sentiment prediction and question type classification, achieving significant results. Overall, this work commented on the range of individual kernels while trying to model contextual semantics and proposed a way to extend their reach.

![alt txt](img/DCNN.jpg)

*Figure 8: DCNN subgraph. With dynamic pooling, a filter with small width at the higher layers can relate phrases far apart in the input sentence (Figure Source: [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062))*

Tasks involving sentiment analysis also require effective extraction of aspects along with their sentiment polarities ([Mukherjee and Liu, 2012](http://www.aclweb.org/anthology/P12-1036)). [Ruder et al. (2016)](https://arxiv.org/abs/1609.02748) applied a CNN where in the input they concatenated an aspect vector with the word embeddings to get competitive results. CNN modeling approach varies amongst different length of texts. Such differences were seen in many works like [Johnson and Zhang (2015)](https://papers.nips.cc/paper/5849-semi-supervised-convolutional-neural-networks-for-text-categorization-via-region-embedding), where performance on longer text worked well as opposed to shorter texts. [Wang et al. (2015)](http://www.aclweb.org/anthology/P15-2058) proposed the usage of CNN for modeling representations of short texts, which suffer from the lack of available context and, thus, require extra efforts to create meaningful representations. The authors proposed semantic clustering which introduced multi-scale semantic units to be used as external knowledge for the short texts. CNN was used to combine these units and form the overall representation. In fact, this requirement of high context information can be thought of as a caveat for CNN-based models. NLP tasks involving microtexts using CNN-based methods often require the need of additional information and external knowledge to perform as per expectations. This fact was also observed in ([Poria et al., 2016](https://arxiv.org/abs/1610.08815)), where authors performed sarcasm detection in Twitter texts using a CNN network. Auxiliary support, in the form of pre-trained networks trained on emotion, sentiment and personality datasets was used to achieve state-of-the-art performance.

CNNs have also been extensively used in other tasks. For example, [Denil et al. (2014)](https://arxiv.org/abs/1406.3830) applied DCNN to map meanings of words that constitute a sentence to that of documents for summarization. The DCNN learned convolution filters at both the sentence and document level, hierarchically learning to capture and compose low-level lexical features into high-level semantic concepts. The focal point of this work was the introduction of a novel visualization technique of the learned representations, which provided insights not only in the learning process but also for automatic summarization of texts. 

CNN models are also suitable for certain NLP tasks that require semantic matching beyond classification ([Hu et al., 2014](https://arxiv.org/abs/1503.03244)). A similar model to the above CNN architecture (Figure 6) was explored in ([Shen et al., 2014](http://www.iro.umontreal.ca/~lisa/pointeurs/ir0895-he-2.pdf)) for information retrieval. The CNN was used for projecting queries and documents to a fixed-dimension semantic space, where cosine similarity between the query and documents was used for ranking documents regarding a specific query. The model attempted to extract rich contextual structures in a query or a document by considering a temporal context window in a word sequence. This captured the contextual features at the word n-gram level. The salient word n-grams is then discovered by the convolution and max-pooling layers which are then aggregated to form the overall sentence vector.

In the domain of QA, [Yih et al. (2014)](http://acl2014.org/acl2014/P14-2/pdf/P14-2105.pdf) proposed to measure the semantic similarity between a question and entries in a knowledge base (KB) to determine what supporting fact in the KB to look for when answering a question. To create semantic representations, a CNN similar to the one in Figure 6 was used. Unlike the classification setting, the supervision signal came from positive or negative text pairs (e.g., query-document), instead of class labels. Subsequently, [Dong et al. (2015)](http://www.aclweb.org/anthology/P15-1026) introduced a multi-column CNN (MCCNN) to analyze and understand questions from multiple aspects and create their representations. MCCNN used multiple column networks to extract information from aspects comprising answer types and context from the input questions. By representing entities and relations in the KB with low-dimensional vectors, they used question-answer pairs to train the CNN model so as to rank candidate answers. [Severyn and Moschitti (2016)](https://arxiv.org/abs/1604.01178) also used CNN network to model optimal representations of question and answer sentences. They proposed additional features in the embeddings in the form of relational information given by matching words between the question and answer pair. These parameters were tuned by the network. This simple network was able to produce comparable results to state-of-the-art methods.

CNNs are wired in a way to capture the most important information in a sentence. Traditional max-pooling strategies perform this in a translation invariant form. However, this often misses valuable information present in multiple facts within the sentence. To overcome this loss of information for multiple-event modeling, [Chen et al. (2015)](https://pdfs.semanticscholar.org/ca70/480f908ec60438e91a914c1075b9954e7834.pdf) proposed a modified pooling strategy: dynamic multi-pooling CNN (DMCNN). This strategy used a novel dynamic multi-pooling layer that, as the name suggests, incorporates event triggers and arguments to reserve more crucial information from the pooling layer. 

CNNs inherently provide certain required features like local connectivity, weight sharing, and pooling. This puts forward some degree of invariance which is highly desired in many tasks. Speech recognition also requires such invariance and, thus, [Abdel-Hamid et al. (2014)](https://ieeexplore.ieee.org/document/6857341) used a hybrid CNN-HMM model which provided invariance to frequency shifts along the frequency axis. This variability is often found in speech signals due to speaker differences. They also performed limited weight sharing which led to a smaller number of pooling parameters, resulting in lower computational complexity. [Palaz et al. (2015)](https://ronan.collobert.com/pub/matos/2015_cnnspeech_interspeech.pdf) performed extensive analysis of CNN-based speech recognition systems when given raw speech as input. They showed the ability of CNNs to directly model the relationship between raw input and phones, creating a robust automatic speech recognition system. 

Tasks like machine translation require perseverance of sequential information and long-term dependency. Thus, structurally they are not well suited for CNN networks, which lack these features. Nevertheless, [Tu et al. (2015)](https://arxiv.org/abs/1503.02357) addressed this task by considering both the semantic similarity of the translation pair and their respective contexts. Although this method did not address the sequence perseverance problem, it allowed them to get competitive results amongst other benchmarks.

Overall, CNNs are extremely effective in mining semantic clues in contextual windows. However, they are very data heavy models. They include a large number of trainable parameters which require huge training data. This poses a problem when scarcity of data arises. Another persistent issue with CNNs is their inability to model long-distance contextual information and preserving sequential order in their representations ([Kalchbrenner et al., 2014](http://www.aclweb.org/anthology/P14-1062); [Tu et al., 2015](https://arxiv.org/abs/1503.02357)). Other networks like recursive models (explained below) reveal themselves as better suited for such learning. 
