---
title: Distributed Representation
number: 2
---

Statistical NLP has emerged as the primary option for modeling complex natural language tasks. However, in its beginning, it often used to suffer from the notorious *curse of dimensionality* while learning joint probability functions of language models. This led to the motivation of learning distributed representations of words existing in low-dimensional space ([Bengio et al., 2003](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)). 

## A. Word Embeddings

Distributional vectors or word embeddings (Figure 2) essentially follow the distributional hypothesis, according to which words with similar meanings tend to occur in similar context. Thus, these vectors try to capture the characteristics of the neighbors of a word.
The main advantage of distributional vectors is that they capture similarity between words. Measuring similarity between vectors is possible, using measures such as cosine similarity.
Word embeddings are often used as the first data processing layer in a deep learning model. Typically, word embeddings are pre-trained by optimizing an auxiliary objective in a large unlabeled corpus, such as predicting a word based on its context (Mikolov et al., [2013b](https://arxiv.org/abs/1301.3781), [a](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)), where the learned word vectors can capture general syntactical and semantic information. Thus, these embeddings have proven to be efficient in capturing context similarity, analogies and due to its smaller dimensionality, are fast and efficient in computing core NLP tasks.

![alt txt](img/distributional.png)
*Figure 2: Distributional vectors represented by a $${\bf D}$$-dimensional vector where $${\bf D} << {\bf V}$$, where $${\bf V}$$ is size of Vocabulary. Figure Source: [http://veredshwartz.blogspot.sg](http://veredshwartz.blogspot.sg).*

Over the years, the models that create such embeddings have been shallow neural networks and there has not been need for deep networks to create good embeddings. However, deep learning based NLP models invariably represent their words, phrases and even sentences using these embeddings. This is in fact a major difference between traditional word count based models and deep learning based models. Word embeddings have been responsible for state-of-the-art results in a wide range of NLP tasks ([Bengio and Usunier, 2011](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37180.pdf); [Socher et al., 2011](https://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf); [Turney and Pantel, 2010](https://jair.org/index.php/jair/article/view/10640); [Cambria et al., 2017](http://sentic.net/sentiment-analysis-suitcase.pdf)). 

For example, [Glorot et al. (2011)](http://www.icml-2011.org/papers/342_icmlpaper.pdf) used embeddings along with stacked denoising autoencoders for domain adaptation in sentiment classification and [Hermann and Blunsom (2013)](http://www.aclweb.org/anthology/P13-1088) presented combinatory categorical autoencoders to learn the compositionality of sentence. Their wide usage across the recent literature shows their effectiveness and importance in any deep learning model performing a NLP task. 

Distributed representations (embeddings) are mainly learned through context. During 1990s, several research developments ([Elman, 1991](https://link.springer.com/content/pdf/10.1023/A:1022699029236.pdf)) marked the foundations of research in distributional semantics. A more detailed summary of these early trends is provided in ([Glenberg and Robertson, 2000](https://www.sciencedirect.com/science/article/pii/S0749596X00927141); [Dumais, 2004](https://onlinelibrary.wiley.com/doi/abs/10.1002/aris.1440380105)). Later developments were adaptations of these early works, which led to creation of topic models like latent Dirichlet allocation ([Blei et al., 2003](http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf)) and language models ([Bengio et al., 2003](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)). These works laid out the foundations of representation learning.

In 2003, [Bengio et al. (2003)](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) proposed a neural language model which learned distributed representations for words (Figure 3). Authors argued that these word representations, once compiled into sentence representations using joint probability of word sequences, achieved an exponential number of semantically neighboring sentences. This, in turn, helped in generalization since unseen sentences could now gather higher confidence if word sequences with similar words (in respect to nearby word representation) were already seen. 

![alt txt](img/bengio_Page_1.png)
*Figure 3: Neural Language Model (Figure reproduced from [Bengio et al. (2003)](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)). $$C(i)$$ is the $$i^{th}$$ word embedding.*

[Collobert and Weston (2008)](https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf) was the first work to show the utility of pre-trained word embeddings. The authors proposed a neural network architecture that forms the foundation to many current approaches. The work also establishes word embeddings as a useful tool for NLP tasks. However, the immense popularization of word embeddings was arguably due to ([Mikolov et al., 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)), who proposed the continuous bag-of-words (CBOW) and skip-gram models to efficiently construct high-quality distributed vector representations. Propelling their popularity was the unexpected side effect of the vectors exhibiting compositionality, i.e., adding two word vectors results in a vector that is a semantic composite of the individual words, e.g., 'man' + 'royal' = 'king'. The theoretical justification for this behavior was recently given by [Gittens et al. (2017)](http://www.aclweb.org/anthology/P17-1007), which stated that compositionality is seen only when certain assumptions are held, e.g., the assumption that words need to be uniformly distributed in the embedding space. 

[Pennington et al. (2014)](https://www.aclweb.org/anthology/D14-1162) is another famous word embedding method which is essentially a "count-based" model. Here, the word co-occurrence count matrix is preprocessed by normalizing the counts and log-smoothing them. This matrix is then factorized to get lower dimensional representations which is done by minimizing a "reconstruction loss".

Below, we provide a brief description of the word2vec method proposed by [Mikolov et al., (2013)](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf). 

## B. Word2vec

Word embeddings were revolutionized by Mikolov et al. ([2013b](https://arxiv.org/abs/1301.3781), [a](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)) who proposed the CBOW and skip-gram models. CBOW computes the conditional probability of a target word given the context words surrounding it across a window of size $$k$$. On the other hand, the skip-gram model does the exact opposite of the CBOW model, by predicting the surrounding context words given the central target word. The context words are assumed to be located symmetrically to the target words within a distance equal to the window size in both directions. In unsupervised settings, the word embedding dimension is determined by the accuracy of prediction. As the embedding dimension increases, the accuracy of prediction also increases until it converges at some point, which is considered the optimal embedding dimension as it is the shortest without compromising accuracy.

Let us consider a simplified version of the CBOW model where only one word is considered in the context. This essentially replicates a bigram language model. 

![alt txt](img/CBOW.png)
*Figure 4: Model for CBOW (Figure source: [Rong (2014)](https://arxiv.org/abs/1411.2738))*

As shown in Figure 4 , the CBOW model is a simple fully connected neural network with one hidden layer. The input layer, which takes the one-hot vector of context word has $${\bf V}$$ neurons while the hidden layer has $${\bf N}$$ neurons. The output layer is softmax of all words in the vocabulary. The layers are connected by weight matrix 
$${\bf W} \in \mathcal{R}^{V \times N }$$ and $${\bf W^{'}} \in \mathcal{R}^{N \times V}$$, respectively. Each word from the vocabulary is finally represented as two learned vectors $${\bf v_c}$$ and $${\bf v_w}$$, corresponding to context and target word representations, respectively. Thus, $$k^{th}$$ word in the vocabulary will have

$$
\begin{gathered}
{\bf v_c} = {\bf W_{(k,.)}} \quad and \quad {\bf v_w} = {\bf W^{'}_{(.,k)}}
\end{gathered}
$$

Overall, for any word $$w_i$$ with given context word $$c$$ as input,

$$
\begin{gathered}
 {\bf P(\frac{w_i}{c})} = {\bf y_i} = \frac{e^{u_i}}{\sum_{i=1}^{V} e^{u_i}} \quad where, {\bf u_i} = {\bf v_{w_i}^T.v_c}
\end{gathered}
$$

The parameters $${\bf \theta} = {\bf \{V_w, V_c\}}$$ are learned by defining the objective function as the log-likelihood and finding its gradient as

$$
\begin{gathered}
{\bf l(\theta)} = {\bf \sum_{w \in Vocabulary} log(P(\frac{w}{c}))} \\
{\bf \frac{\partial l(\theta)}{\partial V_w}} = {\bf V_c(1-P(\frac{w}{c}))}
\end{gathered}
$$

In the general CBOW model, all the one-hot vectors of context words are taken as input simultaneously, i.e,

$$
{\bf h} = {\bf W^T(x_1 + x_2 + ... + x_c)}
$$

One limitation of individual word embeddings is their inability to represent phrases ([Mikolov et al., 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)), where the combination of two or more words (e.g., idioms like "hot potato" or named entities such as "Boston Globe") does not represent the combination of meanings of individual words. One solution to this problem, as explored by [Mikolov et al. (2013)](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf), is to identify such phrases based on word co-occurrence and train embeddings for them separately. More recent methods have explored directly learning n-gram embeddings from unlabeled data ([Johnson and Zhang, 2015](https://papers.nips.cc/paper/5849-semi-supervised-convolutional-neural-networks-for-text-categorization-via-region-embedding)). 

Another limitation comes from learning embeddings based only on a small window of surrounding words, sometimes words such as *good* and *bad* share almost the same embedding ([Socher et al., 2011](https://nlp.stanford.edu/pubs/SocherPenningtonHuangNgManning_EMNLP2011.pdf)), which is problematic if used in tasks such as sentiment analysis ([Wang et al., 2015](http://www.aclweb.org/anthology/P15-1130)). At times these embeddings cluster semantically-similar words which have opposing sentiment polarities. This leads the downstream model used for the sentiment analysis task to be unable to identify this contrasting polarities leading to poor performance. [Tang et al. (2014)](http://www.aclweb.org/anthology/P14-1146) addresses this problem by proposing sentiment specific word embedding (SSWE). Authors incorporate the supervised sentiment polarity of text in their loss functions while learning the embeddings.

A general caveat for word embeddings is that they are highly dependent on the applications in which it is used. [Labutov and Lipson (2013)](http://anthology.aclweb.org/P/P13/P13-2087.pdf) proposed task specific embeddings which retrain the word embeddings to align them in the current task space. This is very important as training embeddings from scratch requires large amount of time and resource. [Mikolov et al. (2013)](https://arxiv.org/abs/1301.3781) tried to address this issue by proposing *negative sampling* which is nothing but frequency-based sampling of negative terms while training the word2vec model. 

Traditional word embedding algorithms assign a distinct vector to each word. This makes them unable to account for polysemy. In a recent work, [Upadhyay et al. (2017)](http://www.aclweb.org/anthology/W17-2613) provided an innovative way to address this deficit. The authors leveraged multilingual parallel data to learn multi-sense word embeddings. For example, the English word bank, when translated to French provides two different words: *banque* and *banc* representing financial and geographical meanings, respectively. Such multilingual distributional information helped them in accounting for polysemy.

Table 1 provides a directory of existing frameworks that are frequently used for creating embeddings which are further incorporated into deep learning models.


| Framework       | Language | URL                                                |
|-----------------|----------|----------------------------------------------------|
| S-Space         | Java     | https://github.com/fozziethebeat/S-Space           |
| Semanticvectors | Java     | https://github.com/semanticvectors/semanticvectors |
| Gensim          | Python   | https://radimrehurek.com/gensim/                   |
| Pydsm           | Python   | https://github.com/jimmycallin/pydsm               |
| Dissect         | Python   | http://clic.cimec.unitn.it/composes/toolkit/       |
| FastText        | Python   | https://fasttext.cc/                               |
| Elmo            | Python   | https://tfhub.dev/google/elmo/2                    |
| Glove           | Python   | https://github.com/maciejkula/glove-python         |

*Table 1: Frameworks providing embedding tools and methods.*

## C. Character Embeddings

Word embeddings are able to capture syntactic and semantic information, yet for tasks such as POS-tagging and NER, intra-word morphological and shape information can also be very useful. Generally speaking, building natural language understanding systems at the character level has attracted certain research attention ([Kim et al., 2016](https://arxiv.org/abs/1508.06615); [Dos Santos and Gatti, 2014](http://anthology.aclweb.org/C/C14/C14-1008.pdf); [Santos and Guimaraes, 2015](https://arxiv.org/abs/1505.05008); [Santos and Zadrozny, 2014](http://proceedings.mlr.press/v32/santos14.pdf)). Better results on morphologically rich languages are reported in certain NLP tasks. [Santos and Guimaraes (2015)](https://arxiv.org/abs/1505.05008) applied character-level representations, along with word embeddings for NER, achieving state-of-the-art results in Portuguese and Spanish corpora. [Kim et al. (2016)](https://arxiv.org/abs/1508.06615) showed positive results on building a neural language model using only character embeddings. [Ma et al. (2016)](http://sentic.net/label-embedding-for-zero-shot-named-entity-typing.pdf) exploited several embeddings, including character trigrams, to incorporate prototypical and hierarchical information for learning pre-trained label embeddings in the context of NER.

A common phenomenon for languages with large vocabularies is the unknown word issue or out-of-vocabulary word (OOV) issue. Character embeddings naturally deal with it since each word is considered as no more than a composition of individual letters. In languages where text is not composed of separated words but individual characters and the semantic meaning of words map to its compositional characters (such as Chinese), building systems at the character level is a natural choice to avoid word segmentation ([Chen et al., 2015](https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/11000)). Thus, works employing deep learning applications on such languages tend to prefer character embeddings over word vectors ([Zheng et al., 2013](http://www.aclweb.org/anthology/D13-1061)). For example, [Peng et al. (2017)](http://sentic.net/radical-embeddings-for-chinese-sentiment-analysis.pdf) proved that radical-level processing could greatly improve sentiment classification performance. In particular, the authors proposed two types of Chinese radical-based hierarchical embeddings, which incorporate not only semantics at radical and character level, but also sentiment information. [Bojanowski et al. (2016)](https://arxiv.org/abs/1607.04606) also tried to improve the representation of words by using character-level information in morphologically-rich languages. They approached the skip-gram method by representing words as bag-of-characters n-grams. Their work thus had the effectiveness of the skip-gram model along with addressing some persistent issues of word embeddings. The method was also fast, which allowed training models on large corpora quickly. Popularly known as *FastText*, such a method stands out over previous methods in terms of speed, scalability, and effectiveness. 

Apart from character embeddings, different approaches have been proposed for OOV handling. [Herbelot and Baroni (2017)](https://arxiv.org/abs/1707.06556) provided OOV handling on-the-fly by initializing the unknown words as the sum of the context words and refining these words with a high learning rate. However, their approach is yet to be tested on typical NLP tasks. [Pinter et al. (2017)](https://arxiv.org/abs/1707.06961) provided an interesting approach of training a character-based model to recreate pre-trained embeddings. This allowed them to learn a compositional mapping form character to word embedding, thus tackling the OOV problem.

Despite the ever growing popularity of distributional vectors, recent discussions on their relevance in the long run have cropped up. For example, [Lucy and Gauthier (2017)](https://arxiv.org/abs/1705.11168) has recently tried to evaluate how well the word vectors capture the necessary facets of conceptual meaning. The authors have discovered severe limitations in perceptual understanding of the concepts behind the words, which cannot be inferred from distributional semantics alone. A possible direction for mitigating these deficiencies will be grounded learning, which has been gaining popularity in this research domain.

## D. Contextualized Word Embeddings

The quality of word representations is generally gauged by its ability to encode syntactical information and handle polysemic behavior (or word senses). These properties result in improved semantic word representations. Recent approaches in this area encode such information into its embeddings by leveraging the context. These methods provide deeper networks that calculate word representations as a function of its context. 

Traditional word embedding methods such as Word2Vec and Glove consider all the sentences where a word is present in order to create a global vector representation of that word. However, a word can have completely different senses or meanings in the contexts. For example, lets consider these two sentences - 1) "The *bank* will not be accepting cash on Saturdays" 2) "The river overflowed the *bank*.". The word senses of *bank* are different in these two sentences depending on its context.
Reasonably, one might want two different vector representations of the word *bank* based on its two different word senses. The new class of models adopt this reasoning by diverging from the concept of global word representations and proposing contextual word embeddings instead.

Embedding from Language Model (ELMo) ([Peters et al., 2018](https://arxiv.org/abs/1802.05365)) is one such method that provides deep contextual embeddings. ELMo produces word embeddings for each context where the word is used, thus allowing different representations for varying senses of the same word. Specifically, for $$N$$ different sentences where a word $$w$$ is present, ELMo generates $$N$$ different representations of $$w$$ i.e., $$\bm{w}_1, \bm{w}_2, \dot, \bm{w}_N$$. 

The mechanism of ELMo is based on the representation obtained from a bidirectional language model. A bidirectional language model (biLM) constitutes of two language models (LM) 1) *forward LM* and 2) *backward LM*. A forward LM takes input representation $$\bm{x}_{k}^{LM}$$ for each of the $$k^{th}$$ token and passes it through $$L$$ layers of forward LSTM to get representations $$\overrightarrow { \bm { h } } _ { k , j } ^ { L M }$$ where $$j=1, \ldots ,L$$. Each of these representations, being hidden representations of recurrent neural networks, is context dependent. A forward LM can be seen as a method to model the joint probability of a sequence of tokens:
$$p \left( t _ { 1 } , t _ { 2 } , \ldots , t _ { N } \right) = \prod _ { k = 1 } ^ { N } p \left( t _ { k } | t _ { 1 } , t _ { 2 } , \ldots , t _ { k - 1 } \right)$$. At a timestep $$k-1$$ the forward LM predicts the next token $$t_{k}$$ given the previous observed tokens $$t_1, t_2, ..., t_k$$. This is typically achieved by placing a softmax layer on top of the final LSTM in a forward LM. On the other hand, a backward LM models the same joint probability of the sequence by predicting the previous token given the future tokens: $$p \left( t _ { 1 } , t _ { 2 } , \ldots , t _ { N } \right) = \prod _ { k = 1 } ^ { N } p \left( t _ { k } | t _ { k + 1 } , t _ { k + 2 } , \ldots , t _ { N } \right)$$. In other words, a backward LM is similar to forward LM which processes a sequence with the order being reversed. The training of the biLM model involves modeling the log-likelihood of both the sentence orientations. Finally, hidden representations from both LMs are concatenated to compose the final token vectors ([Mousa et al., 2017](http://www.aclweb.org/anthology/E17-1096)). 

For each token, ELMo extracts the intermediate layer representations from the biLM and performs a linear combination based on the given downstream task. A $$L$$-layer biLM contains $$2L+1$$ set of representations as shown below -

$$
\begin{aligned} R _ { k } & = \left\{ \mathbf { x } _ { k } ^ { L M } , \overrightarrow { \mathbf { h } } _ { k , j } ^ { L M } , \overleftarrow { \mathbf { h } } _ { k , j } ^ { L M } | j = 1 , \ldots , L \right\} \\ & = \left\{ \mathbf { h } _ { k , j } ^ { L M } | j = 0 , \ldots , L \right\} \end{aligned}
$$

Here, $$h_{k,0}^{LM}$$ is the token representation at the lowest level. One can use either character or word embeddings to initialize $$h_{k,0}^{LM}$$. For other values of $$j$$,

$$
h_{k,j}^{LM} = \left[\overrightarrow { \mathbf { h } } _ { k , j } ^ { L M }, \overleftarrow { \mathbf { h } } _ { k , j } ^ { L M }\right] \;\forall j = 1 , \ldots , L. 
$$

ELMo flattens all layers in $R$ in a single vector such that - 

$$
\text{\textbf{ELMo}} _ { k } ^ { t a s k } = E \left( R _ { k } ; \Theta ^ { t a s k } \right) = \gamma ^ { t a s k } \sum _ { j = 0 } ^ { L } s _ { j } ^ { t a s k } \mathbf { h } _ { k , j } ^ { L M }
$$

In the equation above, $$s _ { j } ^ { t a s k }$$ is the softmax-normalized weight vector to combine the representations of different layers. $$\gamma ^ { t a s k }​$$ is a hyperparameter which helps in optimization and task specific scaling of the ELMo representation. ELMo produces varied word representations for the same word in different sentences. According to [Peters et al. (2018)](https://arxiv.org/abs/1705.11168), it is always beneficial to combine ELMo word representations with standard global word representations like Glove and Word2Vec. 

Off-late, there has been a surge of interest in pre-trained language models for myriad of natural language tasks ([Dai et al., 2015](https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning)). Language modeling is chosen as the pre-training objective as it is widely considered to incorporate multiple traits of natural language understanding and generation. A good language model requires learning complex characteristics of language involving syntactical properties and also semantical coherence. Thus, it is believed that unsupervised training on such objectives would infuse better linguistic knowledge into the networks than random initialization. The *generative pre-training* and *discriminative fine-tuning*
procedure is also desirable as the pre-training is unsupervised and does not require any manual labeling.

[Radford et al. (2018)](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) proposed similar pre-trained model, the OpenAI-GPT, by adapting the *Transformer* (see section 4-E). Recently, [Delvin et al. (2018)](https://arxiv.org/abs/1810.04805) proposed BERT which utilizes a transformer network to pre-train a language model for extracting contextual word embeddings. Unlike ELMo and OpenAI-GPT, BERT uses different pre-training tasks for language modeling. In one of the tasks, BERT randomly masks a percentage of words in the sentences and only predicts those masked words. In the other task, BERT predicts the next sentence given a sentence. This task in particular tries to model the relationship among two sentences which is supposedly not captured by traditional bidirectional language models. Consequently, this particular pre-training scheme helps BERT to outperform state-of-the-art techniques by a large margin on key NLP tasks such as QA, Natural Language Inference (NLI) where understanding relation among two sentences is very important. We discuss the impact of these proposed models and the performance achieved by them in section 8-A.

The described approaches for contextual word embeddings promises better quality representations for words. The pre-trained deep language models also provide a headstart for downstream tasks in the form of transfer learning. This approach has been extremely popular in computer vision tasks. Whether there would be similar trends in the NLP community, where researchers and practitioners would prefer such models over traditional variants remains to be seen in the future.
