# Pretraining

The goal of this script is to provide a method for creating pretrained Reformer Models to later be used for transfer learning.

I download the data from https://dumps.wikimedia.org/ and extract the data to json objects using https://github.com/attardi/wikiextractor