---
title: Memory-Augmented Networks
number: 7
---

The attention mechanism stores a series of hidden vectors of the encoder, which the decoder is allowed to access during the generation of each token. Here, the hidden vectors of the encoder can be seen as entries of the model's "internal memory". Recently, there has been a surge of interest in coupling neural networks with a form of memory, which the model can interact with.

In ([Weston et al., 2014](https://arxiv.org/abs/1410.3916)), the authors proposed memory networks for QA tasks. In synthetic QA, a series of statements (memory entries) were provided to the model as potential supporting facts to the question. The model learned to retrieve one entry at a time from memory based on the question and previously retrieved memory. In large-scale realistic QA, a large set of commonsense knowledge in the form of (subject, relation, object) triples were used as memory. [Sukhbaatar et al. (2015)](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf) extended this work and proposed end-to-end memory networks, where memory entries were retrieved in a "soft" manner with attention mechanism, thus enabling end-to-end training. Multiple rounds (hops) of information retrieval from memory were shown to be essential to good performance and the model was able to retrieve and reason about several supporting facts to answer a specific question (Figure 21). [Sukhbaatar et al. (2015)](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf) also showed a special use of the model for language modeling, where each word in the sentence was seen as a memory entry. With multiple hops, the model yielded results comparable to deep LSTM models.

![alt txt](img/hops.png)

*Figure 21: Multiple supporting facts were retrieved from the memory in order to answer a specific question using an attention mechanism. The first hop uncovered the need for additional hops (Figure source: [Sukhbaatar et al. (2015)](https://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf))*

Furthermore, dynamic memory networks (DMN) ([Kumar et al., 2015](https://arxiv.org/abs/1506.07285)) improved upon previous memory-based models by employing neural network models for input representation, attention, and answer mechanisms. The resulting model was applicable to a wide range of NLP tasks (QA, POS tagging, and sentiment analysis), as every task could be cast to the $$<$$memory, question, answer$$>$$ triple format. [Xiong et al. (2016)](https://arxiv.org/abs/1603.01417) applied the same model to visual QA and proved that the memory module was applicable to visual signals. 