---
title: Performance of Different Models on Different NLP Tasks
number: 8
---

We summarize the performance of a series of deep learning methods on standard datasets developed in recent years on 7 major NLP topics in Tables 2-7. Our goal is to show the readers common datasets used in the community and state-of-the-art results with different models.

## A. POS tagging

The WSJ-PTB (the Wall Street Journal part of the Penn Treebank Dataset) corpus contains 1.17 million tokens and has been widely used for developing and evaluating POS tagging systems. [Giménez and Marquez (2004)](https://pdfs.semanticscholar.org/f7ac/2682ace56c752c60868312e5eeeef0d96a41.pdf) employed one-against-all SVM based on manually-defined features within a seven-word window, in which some basic n-gram patterns were evaluated to form binary features such as: "previous word is *the*", "two preceding tags are DT NN", etc. One characteristic of the POS tagging problem was the strong dependency between adjacent tags. With a simple left-to-right tagging scheme, this method modeled dependencies between adjacent tags only by feature engineering. In an effort to reduce feature engineering, [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) relied on only word embeddings within the word window by a multi-layer perceptron. Incorporating CRF was proven useful in ([Collobert et al., 2011](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)). 
[Santos and Zadrozny (2014)](http://proceedings.mlr.press/v32/santos14.pdf) concatenated word embeddings with character embeddings to better exploit morphological clues. In ([Santos and Zadrozny, 2014](http://proceedings.mlr.press/v32/santos14.pdf)), the authors did not consider CRF, but since word-level decision was made on a context window, it can be seen that dependencies were modeled implicitly. [Huang et al. (2015)](https://arxiv.org/abs/1508.01991) concatenated word embeddings and manually-designed word-level features and employed bidirectional LSTM to model arbitrarily long context. A series of ablative analysis suggested that bi-directionality and CRF both boosted performance. [Andor et al. (2016)](https://arxiv.org/abs/1603.06042) showed a transition-based approach that produces competitive result with a simple feed-forward neural network. When applied to sequence tagging tasks, DMNs ([Kumar et al., 2015](https://arxiv.org/abs/1506.07285)) essentially allowed for attending over the context multiple times by treating each RNN hidden state as a memory entry, each time focusing on different parts of the context. 

| Paper                                                                                                        | Model                              | WSJ-PTB (per-token accuracy %) |
|--------------------------------------------------------------------------------------------------------------|------------------------------------|--------------------------------|
| [Giménez and Marquez (2004)](https://pdfs.semanticscholar.org/f7ac/2682ace56c752c60868312e5eeeef0d96a41.pdf) | SVM with manual feature pattern    | 97.16                          |
| [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)                 | MLP with word embeddings + CRF     | 97.29                          |
| [Santos and Zadrozny (2014)](http://proceedings.mlr.press/v32/santos14.pdf)                                  | MLP with character+word embeddings | 97.32                          |
| [Huang et al. (2015)](https://arxiv.org/abs/1508.01991)                                                      | LSTM                               | 97.29                          |
| [Huang et al. (2015)](https://arxiv.org/abs/1508.01991)                                                      | Bidirectional LSTM                 | 97.40                          |
| [Huang et al. (2015)](https://arxiv.org/abs/1508.01991)                                                      | LSTM-CRF                           | 97.54                          |
| [Huang et al. (2015)](https://arxiv.org/abs/1508.01991)                                                      | Bidirectional LSTM-CRF             | 97.55                          |
| [Andor et al. (2016)](https://arxiv.org/abs/1603.06042)                                                      | Transition-based neural network    | 97.45                          |
| [Kumar et al. (2015)](https://arxiv.org/abs/1506.07285)                                                      | DMN                                | 97.56                          |

*Table 2: POS tagging*

## B. Parsing

There are two types of parsing: dependency parsing, which connects individual words with their relations, and constituency parsing, which iteratively breaks text into sub-phrases. Transition-based methods are a popular choice since they are linear in the length of the sentence. The parser makes a series of decisions that read words sequentially from a buffer and combine them incrementally into the syntactic structure ([Chen and Manning, 2014](https://aclanthology.coli.uni-saarland.de/papers/D14-1082/d14-1082)). At each time step, the decision is made based on a stack containing available tree nodes, a buffer containing unread words and the obtained set of dependency arcs. [Chen and Manning (2014)](https://aclanthology.coli.uni-saarland.de/papers/D14-1082/d14-1082) modeled the decision making at each time step with a neural network with one hidden layer. The input layer contained embeddings of certain words, POS tags and arc labels, which came from the stack, the buffer and the set of arc labels. 

[Tu et al. (2015)](https://arxiv.org/abs/1503.02357) extended the work of [Chen and Manning (2014)](https://aclanthology.coli.uni-saarland.de/papers/D14-1082/d14-1082) by employing a deeper model with 2 hidden layers. However, both [Tu et al. (2015)](https://arxiv.org/abs/1503.02357) and [Chen and Manning (2014)](https://aclanthology.coli.uni-saarland.de/papers/D14-1082/d14-1082) relied on manual feature selecting from the parser state, and they only took into account a limited number of latest tokens. [Dyer et al. (2015)](https://arxiv.org/abs/1505.08075) proposed stack-LSTMs to model arbitrarily long history. The end pointer of the stack changed position as the stack of tree nodes could be pushed and popped. [Zhou et al. (2017)](https://jair.org/index.php/jair/article/view/11054) integrated beam search and contrastive learning for better optimization.

Transition-based models were applied to constituency parsing as well. [Zhu et al. (2013)](http://www.aclweb.org/anthology/P13-1043.pdf) based each transition action on features such as the POS tags and constituent labels of the top few words of the stack and the buffer. By uniquely representing the parsing tree with a linear sequence of labels, [Vinyals et al. (2015)](https://papers.nips.cc/paper/5635-grammar-as-a-foreign-language.pdf) applied the seq2seq learning method to this problem.

| Paper                                                                                         | Model                                               | WSJ (UAS/LAS) |
|-----------------------------------------------------------------------------------------------|-----------------------------------------------------|---------------|
| [Chen and Manning (2014)](https://aclanthology.coli.uni-saarland.de/papers/D14-1082/d14-1082) | Fully-connected NN with features including POS      | 91.80/89.60   |
| [Weiss et al. (2015)](https://arxiv.org/abs/1506.06158)                                       | Deep fully-connected NN with features including POS | 94.30/92.40   |
| [Dyer et al. (2015)](https://arxiv.org/abs/1505.08075)                                        | Stack-LSTM                                          | 93.10/90.90   |
| [Zhou et al. (2017)](https://jair.org/index.php/jair/article/view/11054)                      | Beam contrastive model                              | 93.31/92.37   |

*Table 3.1: Dependency Parsing (UAS/LAS = Unlabeled/labeled Attachment Score; WSJ = The Wall Street Journal Section of Penn Treebank)*

| Paper                                                                                                     | Model                                      | WSJ (F1 %) |
|-----------------------------------------------------------------------------------------------------------|--------------------------------------------|------------|
| [Petrov et al. (2006)](https://pdfs.semanticscholar.org/d84b/9507ff9687a900fde451f27106d930c1b838.pdf)    | Probabilistic context-free grammars (PCFG) | 91.80      |
| [Socher et al. (2011)](https://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf) | Recursive neural networks                  | 90.29      |
| [Zhu et al. (2013)](http://www.aclweb.org/anthology/P13-1043.pdf)                                         | Feature-based transition parsing           | 91.30      |
| [Vinyals et al. (2015)](https://papers.nips.cc/paper/5635-grammar-as-a-foreign-language.pdf)              | seq2seq learning with LSTM+Attention       | 93.50      |

*Table 3.2: Constituency Parsing*

## C. Named-Entity Recognition

CoNLL 2003 has been a standard English dataset for NER, which concentrates on four types of named entities: people, locations, organizations and miscellaneous entities. NER is one of the NLP problems where lexicons can be very useful. [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) first achieved competitive results with neural structures augmented by gazetteer features. [Chiu and Nichols (2015)](https://www.aclweb.org/anthology/Q16-1026) concatenated lexicon features, character embeddings and word embeddings and fed them as input to a bidirectional LSTM. On the other hand, [Lample et al. (2016)](https://arxiv.org/abs/1603.01360) only relied on character and word embeddings, with pre-training embeddings on large unsupervised corpora, they achieved competitive results without using any lexicon. Similar to POS tagging, CRF also boosted the performance of NER, as demonstrated by the comparison in ([Lample et al., 2016](https://arxiv.org/abs/1603.01360)). Overall, we see that bidirectional LSTM with CRF acts as a strong model for NLP problems related to structured prediction.

[Passos et al. (2014)](http://www.aclweb.org/anthology/W14-1609) proposed to modify skip-gram models to better learn entity-type related word embeddings that can leverage information from relevant lexicons. [Luo et al. (2015)](http://www.aclweb.org/anthology/D15-1104) jointly optimized the entities and the linking of entities to a KB. [Strubell et al. (2017)](https://arxiv.org/abs/1702.02098) proposed to use dilated convolutions, defined over a wider effective input width by skipping over certain inputs at a time, for better parallelization and context modeling. The model showed significant speedup while retaining accuracy.  

| Paper                                                                                        | Model                                     | CoNLL 2003 (F1 %) |
|----------------------------------------------------------------------------------------------|-------------------------------------------|-------------------|
| [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) | MLP with word embeddings+gazetteer        | 89.59             |
| [Passos et al. (2014)](http://www.aclweb.org/anthology/W14-1609)                             | Lexicon Infused Phrase Embeddings         | 90.90             |
| [Chiu and Nichols (2015)](https://www.aclweb.org/anthology/Q16-1026)                         | Bi-LSTM with word+char+lexicon embeddings | 90.77             |
| [Luo et al. (2015)](http://www.aclweb.org/anthology/D15-1104)                                | Semi-CRF jointly trained with linking     | 91.20             |
| [Lample et al. (2016)](https://arxiv.org/abs/1603.01360)                                     | Bi-LSTM with word+char embeddings         | 89.15             |
| [Lample et al. (2016)](https://arxiv.org/abs/1603.01360)                                     | Bi-LSTM-CRF with word+char embeddings     | 90.94             |
| [Strubell et al. (2017)](https://arxiv.org/abs/1702.02098)                                   | Dilated CNN with CRF                      | 90.54             |

*Table 4: Named-Entity Recognition*

## D. Semantic Role Labeling

Semantic role labeling (SRL) aims to discover the predicate-argument structure of each predicate in a sentence. For each target verb (predicate), all constituents in the sentence which take a semantic role of the verb are recognized. Typical semantic arguments include Agent, Patient, Instrument, etc., and also adjuncts such as Locative, Temporal, Manner, Cause, etc. ([Zhou and Xu, 2015](http://www.aclweb.org/anthology/P15-1109)). Table 5 shows the performance of different models on the CoNLL 2005 & 2012 datasets.

Traditional SRL systems consist of several stages: producing a parse tree, identifying which parse tree nodes represent the arguments of a given verb, and finally classifying these nodes to determine the corresponding SRL tags. Each classification process usually entails extracting numerous features and feeding them into statistical models ([Collobert et al., 2011](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)). 

Given a predicate, [Täckström et al. (2015)](https://transacl.org/ojs/index.php/tacl/article/view/465/105) scored a constituent span and its possible role to that predicate with a series of features based on the parse tree. They proposed a dynamic programming algorithm for efficient inference. [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) achieved comparable results with a convolution neural networks augmented by parsing information provided in the form of additional look-up tables. [Zhou and Xu (2015)](http://www.aclweb.org/anthology/P15-1109) proposed to use bidirectional LSTM to model arbitrarily long context, which proved to be successful without any parsing tree information. [He et al. (2017)](https://aclanthology.coli.uni-saarland.de/papers/P17-1044/p17-1044) further extended this work by introducing highway connections ([Srivastava et al., 2015](https://papers.nips.cc/paper/5850-training-very-deep-networks.pdf)), more advanced regularization and ensemble of multiple experts.

| Paper                                                                                        | Model                                       | CoNLL 2005 (F1 %) | CoNLL 2012 (F1 %) |
|----------------------------------------------------------------------------------------------|---------------------------------------------|-------------------|-------------------|
| [Collobert et al. (2011)](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) | CNN with parsing features                   | 76.06             |                   |
| [Täckström et al. (2015)](https://transacl.org/ojs/index.php/tacl/article/view/465/105)      | Manual features with DP for inference       | 78.60             | 79.40             |
| [Zhou and Xu (2015)](http://www.aclweb.org/anthology/P15-1109)                               | Bidirectional LSTM                          | 81.07             | 81.27             |
| [He et al. (2017)](https://aclanthology.coli.uni-saarland.de/papers/P17-1044/p17-1044)       | Bidirectional LSTM with highway connections | 83.20             | 83.40             |

*Table 5: Semantic Role Labeling*

## E. Sentiment Classification

The Stanford Sentiment Treebank (SST) dataset contains sentences taken from the movie review website Rotten Tomatoes. It was proposed by [Pang and Lee (2005)](https://aclanthology.coli.uni-saarland.de/papers/P05-1015/p05-1015) and subsequently extended by [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf). The annotation scheme has inspired a new dataset for sentiment analysis, called CMU-MOSI, where sentiment is studied in a multimodal setup ([Zadeh et al., 2013](http://sentic.net/multimodal-sentiment-intensity-analysis-in-videos.pdf)). 

([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)) and ([Tai et al., 2015](https://arxiv.org/abs/1503.00075)) were both recursive networks that relied on constituency parsing trees. Their difference shows the effectiveness of LSTM over vanilla RNN in modeling sentences. On the other hand, tree-LSTM performed better than linear bidirectional LSTM, implying that tree structures can potentially better capture the syntactical property of natural sentences. [Yu et al. (2017)](https://www.aclweb.org/anthology/D17-1056) proposed to refine pre-trained word embeddings with a sentiment lexicon, observing improved results based on ([Tai et al., 2015](https://arxiv.org/abs/1503.00075)).

[Kim (2014)](https://arxiv.org/abs/1408.5882) and [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062) both used convolutional layers. The model ([Kim, 2014](https://arxiv.org/abs/1408.5882)) was similar to the one in Figure 5, while [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062) constructed the model in a hierarchical manner by interweaving k-max pooling layers with convolutional layers. 

| Paper                                                                        | Model                                  | SST-1 | SST-2 |
|------------------------------------------------------------------------------|----------------------------------------|-------|-------|
| [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf) | Recursive Neural Tensor Network        | 45.7  | 85.4  |
| [Kim (2014)](https://arxiv.org/abs/1408.5882)                                | Multichannel CNN                       | 47.4  | 88.1  |
| [Kalchbrenner et al. (2014)](http://www.aclweb.org/anthology/P14-1062)       | DCNN with k-max pooling                | 48.5  | 86.8  |
| [Tai et al. (2015)](https://arxiv.org/abs/1503.00075)                        | Bidirectional LSTM                     | 48.5  | 87.2  |
| [Le and Mikolov (2014)](https://arxiv.org/abs/1405.4053)                     | Paragraph Vector                       | 48.7  | 87.8  |
| [Tai et al. (2015)](https://arxiv.org/abs/1503.00075)                        | Constituency Tree-LSTM                 | 51.0  | 88.0  |
| [Yu et al. (2017)](https://www.aclweb.org/anthology/D17-1056)                | Tree-LSTM with refined word embeddings | 54.0  | 90.3  |
| [Kumar et al. (2015)](https://arxiv.org/abs/1506.07285)                      | DMN                                    | 52.1  | 88.6  |

*Table 6: Sentiment Classification (SST-1 = Stanford Sentiment Treebank, fine-grained 5 classes [Socher et al. (2013)](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf); SST-2: the binary version of SST-1; Numbers are accuracies (%))*

## F. Machine Translation

The phrase-based SMT framework ([Koehn et al., 2003](http://www.aclweb.org/anthology/N03-1017)) factorized the translation model into the translation probabilities of matching phrases in the source and target sentences. [Cho et al. (2014)](https://arxiv.org/abs/1406.1078) proposed to learn the translation probability of a source phrase to a corresponding target phrase with an RNN encoder-decoder. Such a scheme of scoring phrase pairs improved translation performance. [Sutskever et al. (2014)](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf), on the other hand, re-scored the top 1000 best candidate translations produced by an SMT system with a 4-layer LSTM seq2seq model. Dispensing the traditional SMT system entirely, [Wu et al. (2016)](https://arxiv.org/abs/1609.08144) trained a deep LSTM network with 8 encoder and 8 decoder layers with residual connections as well as attention connections. ([Wu et al., 2016](https://arxiv.org/abs/1609.08144)) then refined the model by using reinforcement learning to directly optimize BLEU scores, but they found that the improvement in BLEU scores by this method did not reflect in human evaluation of translation quality. Recently, [Gehring et al. (2017)](https://arxiv.org/abs/1705.03122) proposed a CNN-based seq2seq learning model for machine translation. The representation for each word in the input is computed by CNN in a parallelized style for the attention mechanism. The decoder state is also determined by CNN with words that are already produced. [Vaswani et al. (2017)](https://arxiv.org/abs/1706.03762) proposed a self-attention-based model and dispensed convolutions and recurrences entirely.

| Paper                                                                                                               | Model                                                   | WMT2014 English2German | WMT2014 English2French |
|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|------------------------|------------------------|
| [Cho et al. (2014)](https://arxiv.org/abs/1406.1078)                                                                | Phrase table with neural features                       |                        | 34.50                  |
| [Sutskever et al. (2014)](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf) | Reranking phrase-based SMT best list with LSTM seq2seq  |                        | 36.50                  |
| [Wu et al. (2016)](https://arxiv.org/abs/1609.08144)                                                                | Residual LSTM seq2seq + Reinforcement learning refining | 26.30                  | 41.16                  |
| [Gehring et al. (2017)](https://arxiv.org/abs/1705.03122)                                                           | seq2seq with CNN                                        | 26.36                  | 41.29                  |
| [Vaswani et al. (2017)](https://arxiv.org/abs/1706.03762)                                                           | Attention mechanism                                     | 28.40                  | 41.00                  |

*Table 7: Machine translation (Numbers are BLEU scores)*

## G. Question Answering

QA problems take many forms. Some rely on large KBs to answer open-domain questions, while others answer a question based on a few sentences or a paragraph (reading comprehension). For the former, we list (see Table 8) several experiments conducted on a large-scale QA dataset introduced by ([Fader et al., 2013](http://www.aclweb.org/anthology/P13-1158)), where 14M commonsense knowledge triples are considered as the KB. Each question can be answered with a single-relation query. For the latter, we consider (see Table 8) (1) the synthetic dataset of *bAbI* ([Weston et al., 2015](https://arxiv.org/abs/1502.05698)), which requires the model to reason over multiple related facts to produce the right answer. It contains 20 synthetic tasks that test a model's ability to retrieve relevant facts and reason over them. Each task focuses on a different skill such as *basic coreference* and *size reasoning*. (2) The Stanford Question Answering Dataset (*SQuAD*) ([Rajpurkar et al., 2016](https://arxiv.org/abs/1606.05250)), consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles. The answer to each question is a segment of text from the corresponding article.

The central problem of learning to answer single-relation queries is to find the single supporting fact in the database. [Fader et al. (2013)](http://www.aclweb.org/anthology/P13-1158) proposed to tackle this problem by learning a lexicon that maps natural language patterns to database concepts (entities, relations, question patterns) based on a question paraphrasing dataset. [Bordes et al. (2014)](https://arxiv.org/abs/1404.4326) embedded both questions and KB triples as dense vectors and scored them with inner product. 

[Weston et al. (2014)](https://arxiv.org/abs/1410.3916) took a similar approach by treating the KB as long-term memory, while casting the problem in the framework of a memory network. On the *bAbI* dataset, [Sukhbaatar et al. (2015)](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf) improved upon the original memory networks model ([Weston et al., 2014](https://arxiv.org/abs/1410.3916)) by making the training procedure agnostic of the actual supporting fact, while [Kumar et al. (2015)](https://arxiv.org/abs/1506.07285) used neural sequence models (GRU) instead of neural bag-of-words models as in ([Sukhbaatar et al., 2015](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf)) and ([Weston et al., 2014](https://arxiv.org/abs/1410.3916)) to embed memories.

For models on the *SQuAD* dataset, the goal is to determine the start point and end point of the answer segment. [Chen et al. (2017)](https://arxiv.org/abs/1704.00051) encoded both the question and the words in context using LSTMs and used a bilinear matrix for calculating the similarity between the two. [Shen et al. (2017)](https://arxiv.org/abs/1609.05284) proposed Reasonet, a model that read a document repeatedly with attention on different parts each time until a satisfying answer is found. [Yu et al., 2018](https://arxiv.org/abs/1804.09541) replaced RNNs with convolution and self-attention for encoding the question and the context with significant speed improvement.

| Paper                                                                                        | Model                              | bAbI (Mean accuracy %) | Farbes (Accuracy %) | SQuAD (EM/F1 %) |
|----------------------------------------------------------------------------------------------|------------------------------------|------------------------|---------------------|-----------------|
| [Fader et al. (2013)](http://www.aclweb.org/anthology/P13-1158)                              | Paraphrase-driven lexicon learning |                        | 0.54                |                 |
| [Bordes et al. (2014)](https://arxiv.org/abs/1404.4326)                                      | Weekly supervised embedding        |                        | 0.73                |                 |
| [Weston et al. (2014)](https://arxiv.org/abs/1410.3916)                                      | Memory networks                    | 93.3                   | 0.93                |                 |
| [Sukhbaatar et al. (2015)](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf) | End-to-end memory networks         | 88.4                   |                     |                 |
| [Kumar et al. (2015)](https://arxiv.org/abs/1506.07285)                                      | DMN                                | 93.6                   |                     |                 |
| [Chen et al. (2017)](https://arxiv.org/abs/1704.00051)                                       | Document Reader                    |                        |                     | 70.0/79.0       |
| [Shen et al. 2016)](https://arxiv.org/abs/1609.05284)                                        | ReasoNet                           |                        |                     | 69.1/78.9       | 
| [Yu et al. (2018)](https://arxiv.org/abs/1804.09541)                                         | QAnet                              |                        |                     | 76.2/84.6       |

*Table 8: Question answering*

## H. Dialogue Systems

Two types of dialogue systems have been developed: generation-based models and retrieval-based models. 

In Table 9, the Twitter Conversation Triple Dataset is typically used for evaluating generation-based dialogue systems, containing 3-turn Twitter conversation instances. One commonly used evaluation metric is BLEU ([Papineni et al., 2002](https://www.aclweb.org/anthology/P02-1040.pdf)), although it is commonly acknowledged that most automatic evaluation metrics are not completely reliable for dialogue evaluation and additional human evaluation is often necessary. [Ritter et al. (2011)](http://www.aclweb.org/anthology/D11-1054) employed the phrase-based statistical machine translation (SMT) framework to "translate" the message to its appropriate response. [Sordoni et al. (2015)](https://arxiv.org/abs/1506.06714) reranked the 1000 best responses produced by SMT with a context-sensitive RNN encoder-decoder framework, observing substantial gains. [Li et al. (2015)](https://arxiv.org/abs/1510.03055)  reported results on replacing the traditional maximum log likelihood training objective with the maximum mutual information training objective, in an effort to produce interesting and diverse responses, both of which are tested on a 4-layer LSTM encoder-decoder framework.

The response retrieval task is defined as selecting the best response from a repository of candidate responses. Such a model can be evaluated by the recall1@$$k$$ metric, where the ground-truth response is mixed with $$k-1$$ random responses. The Ubuntu dialogue dataset was constructed by scraping multi-turn Ubuntu trouble-shooting dialogues from an online chatroom ([Lowe et al., 2015](https://aclanthology.coli.uni-saarland.de/papers/D14-1003/d14-1003)). [Lowe et al. (2015)](https://aclanthology.coli.uni-saarland.de/papers/D14-1003/d14-1003) used LSTMs to encode the message and response, and then inner product of the two sentence embeddings is used to rank candidates. 

[Zhou et al. (2016)](http://www.aclweb.org/anthology/D16-1036) proposed to better exploit the multi-turn nature of human conversation by employing the LSTM encoder on top of sentence-level CNN embeddings, similar to ([Serban et al., 2016](https://arxiv.org/abs/1507.04808)). [Dodge et al. (2015)](https://arxiv.org/abs/1511.06931) cast the problem in the framework of a memory network, where the past conversation was treated as memory and the latest utterance was considered as a "question" to be responded to. The authors showed that using simple neural bag-of-word embedding for sentences can yield competitive results.

| Paper                                                            | Model                                    | Twitter Conversation Triple Dataset (BLEU) | Ubuntu Dialogue Dataset (recall 1@10 %) |
|------------------------------------------------------------------|------------------------------------------|--------------------------------------------|-----------------------------------------|
| [Ritter et al. (2011)](http://www.aclweb.org/anthology/D11-1054) | SMT                                      | 3.60                                       |                                         |
| [Sordoni et al. (2015)](https://arxiv.org/abs/1506.06714)        | SMT+neural reranking                     | 4.44                                       |                                         |
| [Li et al. (2015)](https://arxiv.org/abs/1510.03055)             | LSTM seq2seq                             | 4.51                                       |                                         |
| [Li et al. (2015)](https://arxiv.org/abs/1510.03055)             | LSTM seq2seq with MMI objective          | 5.22                                       |                                         |
| [Lowe et al. (2015)](https://arxiv.org/abs/1506.08909)           | Dual LSTM encoders for semantic matching |                                            | 55.22                                   |
| [Dodge et al. (2015)](https://arxiv.org/abs/1511.06931)          | Memory networks                          |                                            | 63.72                                   |
| [Zhou et al. (2016)](http://www.aclweb.org/anthology/D16-1036)   | Sentence-level CNN-LSTM encoder          |                                            | 66.15                                   |

*Table 9: Dialogue systems*

## I. Contextual Embeddings

In this section, we explore some of the recent results based on contextual embeddings as explained in section 2-D. ELMo has contributed significantly towards the recent advancement of NLP. In various NLP tasks, ELMo outperformed state of the art by significant margin (Table 10). However, latest mode BERT surpass ELMo to establish itself as the state-of-the-art in multiple tasks as summarized in Table 11.

| Task  | Previous SOTA                                            | Previous SOTA Results | Baseline | ELMo + Baseline | Increase (Absolute/Relative) |
|-------|----------------------------------------------------------|-----------------------|----------|-----------------|------------------------------|
| SQuAD | [Liu et al. (2017)](https://arxiv.org/abs/1712.03556)    | 84.4                  | 81.1     | 85.8            | 4.7 / 24.9%                  |
| SNLI  | [Chen et al. (2017)](https://arxiv.org/abs/1609.06038)   | 88.6                  | 88.0     | 88.70 (+- 0.17) | 0.7 / 5.8%                   |
| SRL   | [He et al. (2017)](http://aclweb.org/anthology/P17-1044) | 81.7                  | 81.4     | 84.6            | 3.2 / 17.2%                  |
| Coref | [Lee et al. (2017)](https://arxiv.org/abs/1707.07045)    | 67.2                  | 67.2     | 70.4            | 3.2 / 9.8%                   |
| NER   | [Peters et al. (2017)](https://arxiv.org/abs/1705.00108) | 91.3 (+- 0.19)        | 90.15    | 92.22 (+- 0.10) | 2.06 / 21%                   |
| SST-5 | [McCann et al. (2017)](https://arxiv.org/abs/1708.00107) | 53.7                  | 51.4     | 54.7 (+- 0.5)   | 3.3 / 6.8%                   |

*Table 10: Comparison of ELMo + Baseline with the previous state of the art (SOTA) on various NLP tasks. The table has been adapted from [Peters et al. (2018)](https://arxiv.org/abs/1802.05365). SOTA results have been taken from [Peters et al. (2018)](https://arxiv.org/abs/1802.05365); SQUAD ([Rajpurkar et al., 2016](https://arxiv.org/abs/1606.05250)): QA task; SNLI ([Bowman et al., 2015](https://nlp.stanford.edu/pubs/snli_paper.pdf)): Stanford Natural Language Inference task; SRL ([Zhou and Xu, 2015](http://www.aclweb.org/anthology/P15-1109)): Semantic Role Labelling; Coref ([Pradhan et al., 2012](http://www.aclweb.org/anthology/W12-4501)): Coreference Resolution; NER ([Tjong et al., 2003](http://aclweb.org/anthology/W03-0419)): Named Entity Recognition; SST-5 ([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)): Stanford Sentiment Treebank 5-class classification;*


| Task  | BiLSTM + ELMo + Attn | BERT |
|-------|----------------------|------|
| QNLI  | 79.9                 | 91.1 |
| SST-2 | 90.9                 | 94.9 |
| STS-B | 73.3                 | 86.5 |
| RTE   | 56.8                 | 70.1 |
| SQuAD | 85.8                 | 91.1 |
| NER   | 92.2                 | 92.8 |

*Table 11: QNLI ([Wang et al., 2018](https://arxiv.org/abs/1804.07461)): Question Natural Language Inference task; SST-2 ([Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)): Stanford Sentiment Treebank binary classification; STS-B ([Cer et al., 2017](http://www.aclweb.org/anthology/S17-2001)):  Semantic Textual Similarity Benchmark; RTE ([Bentivogli and Clark, 2009](https://www.semanticscholar.org/paper/The-Sixth-PASCAL-Recognizing-Textual-Entailment-Bentivogli-Clark/9746994d09b5bf6c40bee3693ee8678e191f84b8)): Recognizing Textual Entailment; SQuAD ([Rajpurkar et al., 2016](https://arxiv.org/abs/1606.05250)): QA task; NER ([Tjong et al., 2003](http://aclweb.org/anthology/W03-0419)): Named Entity Recognition.*