---
title: Deep Reinforced Models and Deep Unsupervised Learning
number: 6
---

## A. Reinforcement learning for sequence generation

Reinforcement learning is a method of training an agent to perform discrete actions before obtaining a reward. In NLP, tasks concerning language generation can sometimes be cast as reinforcement learning problems.

In its original formulation, RNN language generators are typically trained by maximizing the likelihood of each token in the ground-truth sequence given the current hidden state and the previous tokens. Termed "teacher forcing", this training scheme provides the real sequence prefix to the generator during each generation (loss evaluation) step. At test time, however, ground-truth tokens are then replaced by a token generated by the model itself. This discrepancy between training and inference, termed "exposure bias" ([Bengio et al., 2015](https://papers.nips.cc/paper/5956-scheduled-sampling-for-sequence-prediction-with-recurrent-neural-networks.pdf); [Ranzato et al., 2015](https://arxiv.org/abs/1511.06732)), can yield errors that can accumulate quickly along the generated sequence.

Another problem with the word-level maximum likelihood strategy, when training auto-regressive language generation models, is that the training objective is different from the test metric. It is unclear how the n-gram overlap based metrics (BLEU, ROUGE) used to evaluate these tasks (machine translation, dialogue systems, etc.) can be optimized with the word-level training strategy. Empirically, dialogue systems trained with word-level maximum likelihood also tend to produce dull and short-sighted responses ([Li et al., 2016](https://arxiv.org/abs/1606.01541)), while text summarization tends to produce incoherent or repetitive summaries ([Paulus et al., 2017](https://arxiv.org/abs/1705.04304)). 

Reinforcement learning offers a prospective to solve the above problems to a certain extent. In order to optimize the non-differentiable evaluation metrics directly, [Ranzato et al. (2015)](https://arxiv.org/abs/1511.06732) applied the REINFORCE algorithm ([Williams, 1992](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)) to train RNN-based models for several sequence generation tasks (e.g., text summarization, machine translation and image captioning), leading to improvements compared to previous supervised learning methods. In such a framework, the generative model (RNN) is viewed as an agent, which interacts with the external environment (the words and the context vector it sees as input at every time step). The parameters of this agent defines a policy, whose execution results in the agent picking an action, which refers to predicting the next word in the sequence at each time step. After taking an action the agent updates its internal state (the hidden units of RNN). Once the agent has reached the end of a sequence, it observes a reward. This reward can be any developer-defined metric tailored to a specific task. For example, [Li et al. (2016)](https://arxiv.org/abs/1606.01541) defined 3 rewards for a generated sentence based on ease of answering, information flow, and semantic coherence. 

There are two well-known shortcomings of reinforcement learning. To make reinforcement learning tractable, it is desired to carefully handle the state and action space (Young et al., [2010](https://hal.archives-ouvertes.fr/hal-00598186), [2013](http://mi.eng.cam.ac.uk/~sjy/papers/ygtw13.pdf)), which in the end may restrict expressive power and learning capacity of the model. Secondly, the need for training the reward functions makes such models hard to design and measure at run time (Su et al., [2011](http://svr-www.eng.cam.ac.uk/~sjy/papers/svgk15.pdf), [2016](https://arxiv.org/abs/1605.07669)).

Another approach for sequence-level supervision is to use the adversarial training technique ([Goodfellow et al., 2014](https://papers.nips.cc/paper/5423-generative-adversarial-nets)), where the training objective for the language generator is to fool another discriminator trained to distinguish generated sequences from real sequences. The generator *G* and the discriminator *D* are trained jointly in a min-max game which ideally leads to *G*, generating sequences indistinguishable from real ones. This approach can be seen as a variation of generative adversarial networks in ([Goodfellow et al., 2014](https://papers.nips.cc/paper/5423-generative-adversarial-nets)), where *G* and *D* are conditioned on certain stimuli (for example, the source image in the task of image captioning). In practice, the above scheme can be realized under the reinforcement learning paradigm with policy gradient. For dialogue systems, the discriminator is analogous to a human Turing tester, who discriminates between human and machine-produced dialogues ([Li et al., 2017](https://arxiv.org/abs/1701.06547)).

## B. Unsupervised sentence representation learning

Similar to word embeddings, distributed representation for sentences can also be learned in an unsupervised fashion. The result of such unsupervised learning are "sentence encoders", which map arbitrary sentences to fixed-size vectors that can capture their semantic and syntactic properties. Usually an auxiliary task has to be defined for the learning process.

Similar to the skip-gram model ([Mikolov et al., 2013](https://arxiv.org/abs/1301.3781)) for learning word embeddings, the skip-thought model ([Kiros et al., 2015](https://papers.nips.cc/paper/5950-skip-thought-vectors.pdf)) was proposed for learning sentence representation, where the auxiliary task was to predict two adjacent sentences (before and after) based on the given sentence. The seq2seq model was employed for this learning task. One LSTM encoded the sentence to a vector (distributed representation). Two other LSTMs decoded such representation to generate the target sequences. Standard seq2seq training process was used. After training, the encoder could be seen as a generic feature extractor (word embeddings were also learned in the same time). 

[Kiros et al. (2015)](https://papers.nips.cc/paper/5950-skip-thought-vectors.pdf) verified the quality of the learned sentence encoder on a range of sentence classification tasks, showing competitive results with a simple linear model based on the static feature vectors. However, the sentence encoder can also be fine-tuned in the supervised learning task as part of the classifier. [Dai and Le (2015)](https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning) investigated the use of the decoder to reconstruct the encoded sentence itself, which resembled an autoencoder ([Rumelhart et al., 1985](https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf)). 

Language modeling could also be used as an auxiliary task when training LSTM encoders, where the supervision signal came from the prediction of the next token. [Dai and Le (2015)](https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning) conducted experiments on initializing LSTM models with learned parameters on a variety of tasks. They showed that pre-training the sentence encoder on a large unsupervised corpus yielded better accuracy than only pre-training word embeddings. Also, predicting the next token turned out to be a worse auxiliary objective than reconstructing the sentence itself, as the LSTM hidden state was only responsible for a rather short-term objective.

## C. Deep generative models

Recent success in generating realistic images has driven a series of efforts on applying deep generative models to text data. The promise of such research is to discover rich structure in natural language while generating realistic sentences from a latent code space. In this section, we review recent research on achieving this goal with variational autoencoders (VAEs) ([Kingma and Welling, 2013](https://arxiv.org/abs/1312.6114)) and generative adversarial networks (GANs) ([Goodfellow et al., 2014](https://papers.nips.cc/paper/5423-generative-adversarial-nets)).

Standard sentence autoencoders, as in the last section, do not impose any constraint on the latent space, as a result, they fail when generating realistic sentences from arbitrary latent representations ([Bowman et al., 2015](https://arxiv.org/abs/1511.06349)). The representations of these sentences may often occupy a small region in the hidden space and most of regions in the hidden space do not necessarily map to a realistic sentence ([Zhang et al., 2016](https://www.semanticscholar.org/paper/Generating-Text-via-Adversarial-Training-Zhang-Gan/fc3fb80cae9c41790da6454b928c9f794361a7c7)). They cannot be used to assign probabilities to sentences or to sample novel sentences ([Bowman et al., 2015](https://arxiv.org/abs/1511.06349)).

The VAE imposes a prior distribution on the hidden code space which makes it possible to draw proper samples from the model. It modifies the autoencoder architecture by replacing the deterministic encoder function with a learned posterior recognition model. The model consists of encoder and generator networks which encode data examples to latent representation and generate samples from the latent space, respectively. It is trained by maximizing a variational lower bound on the log-likelihood of observed data under the generative model. 

[Bowman et al. (2015)](https://arxiv.org/abs/1511.06349) proposed an RNN-based variational autoencoder generative model that incorporated distributed latent representations of entire sentences (Figure 20). Unlike vanilla RNN language models, this model worked from an explicit global sentence representation. Samples from the prior over these sentence representations produced diverse and well-formed sentences.

![alt txt](img/VAE.png)

*Figure 20: RNN-based VAE for sentence generation (Figure source: [Bowman et al. (2015)](https://arxiv.org/abs/1511.06349))*

[Hu et al. (2017)](https://arxiv.org/abs/1703.00955) proposed generating sentences whose attributes are controlled by learning disentangled latent representations with designated semantics. The authors augmented the latent code in the VAE with a set of structured variables, each targeting a salient and independent semantic feature of sentences. The model incorporated VAE and attribute discriminators, in which the VAE component trained the generator to reconstruct real sentences for generating plausible text, while the discriminators forced the generator to produce attributes coherent with the structured code. When trained on a large number of unsupervised sentences and a small number of labeled sentences, [Hu et al. (2017)](https://arxiv.org/abs/1703.00955) showed that the model was able to generate plausible sentences conditioned on two major attributes of English: tense and sentiment.

GAN is another class of generative model composed of two competing networks. A generative neural network decodes latent representation to a data instance, while the discriminative network is simultaneously taught to discriminate between instances from the true data distribution and synthesized instances produced by the generator. GAN does not explicitly represent the true data distribution $$p(x)$$. 

[Zhang et al (2016)](https://www.semanticscholar.org/paper/Generating-Text-via-Adversarial-Training-Zhang-Gan/fc3fb80cae9c41790da6454b928c9f794361a7c7) proposed a framework for employing LSTM and CNN for adversarial training to generate realistic text. The latent code $$z$$ was fed to the LSTM generator at every time step. CNN acted as a binary sentence classifier which discriminated between real data and generated samples. One problem with applying GAN to text is that the gradients from the discriminator cannot properly back-propagate through discrete variables. In ([Zhang et al., 2016](https://www.semanticscholar.org/paper/Generating-Text-via-Adversarial-Training-Zhang-Gan/fc3fb80cae9c41790da6454b928c9f794361a7c7)), this problem was solved by making the word prediction at every time "soft" at the word embedding space. [Yu et al. (2017)](https://arxiv.org/abs/1609.05473) proposed to bypass this problem by modeling the generator as a stochastic policy. The reward signal came from the GAN discriminator judged on a complete sequence, and was passed back to the intermediate state-action steps using Monte Carlo search.

The evaluation of deep generative models has been challenging. For text, it is possible to create oracle training data from a fixed set of grammars and then evaluate generative models based on whether (or how well) the generated samples agree with the predefined grammar ([Rajeswar et al., 2017](https://arxiv.org/abs/1705.10929)). Another strategy is to evaluate BLEU scores of samples on a large amount of unseen test data. The ability to generate similar sentences to unseen real data is considered a measurement of quality ([Yu et al., 2017](https://arxiv.org/abs/1609.05473)). 

