| Model Name| Model Type (Encoder-Decoder, etc.)   | Pre-train Objective |  Tokenization  | Vocab Size | OOV Handling | Embeddings | Attention | Activations | Parameters | Training | Pre-Train Data | Batch Size|
|   :----: |   :----:   |     :----:   |    :----:   |  :----:   |  :----: |   :----:  |    :----: |    :----:   |    :----:   |:----:   |:----:   |:----:   |
|   <span style="color:blue"> BERT </span> |  <span style="color:blue"> Encoder-Only </span> |  <ul><li> <span style="color:blue"> **Masked Language Modeling (MLM)** </span> :~15% tokens chosen -> 80% replaced with [MASK], 10% random token, 10% left unchanged. A shallow decoder is used to reproduce the original text. </li><li> <span style="color:blue"> **Next Sentence Prediction (NSP)** </span> : Binary classification task, predicts if 2 sequences follow each other in corpus (useful on Q&A, etc.). Sampling is 50% 0,1.  </li></li><li> Training loss is mean of MLM + NSP likelihood </li></ul> | <ul><li> <span style="color:blue"> **Wordpiece Tokenization** </span> : [original paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf), [huggingface explanation](https://huggingface.co/transformers/tokenizer_summary.html#wordpiece) </li><li> **Token break down** :<span style="color:green"> **[CLS] token** </span> (useful for many-to-one fine-tuned tasks such as classification) + <span style="color:green"> **WordPiece tokens** </span> + <span style="color:green"> **[SEP]** </span>  token for each sentence. </li><li> <span style="color:blue">  **MAX 512 Tokens**. </span> </li></ul>  | <span style="color:blue">  30k tokens </span>  |  Greedy decomposition of token into sub-words until it finds tokens in vocabulary.   |    Sum of: **Token embeddings** (WordPiece) + **segment embedding (learned)** + **Absolute position embedding** |    **Scaled Dot-product Self-Attention** (note: advised to pad inputs on right rather than left since positional embeddings are absolute.)   |  <span style="color:blue">  **GeLU** </span> : **Dying ReLU problem** - a node can be stuck @ 0 with negative inputs, stops learning, cannot recover.   |  <ul><li> **BERT base**: 12 layers (transformer blocks), 12 attention heads, hidden size = 768 -> ~110 MM params </li><li> **BERT large**: 24 layers, 16 attention heads, hidden size = 1024 -> ~340 MM params </li><li> Generally the parameter space choices are ```embed_size (E) == hidden_size (H)```, the feed-forward size is ```4H``` and the number of attention heads is ```H/64```. To see the math on how the total number of parameters is calculated, check out this comment on BERT github [here](https://github.com/google-research/bert/issues/656#issuecomment-554718760) </li></ul> | <ul><li> Adam and L_2 weight decay </li><li> Learning rate is warmed up during first 10k steps to peak value of 1e.-4, then linearlly decayed </li><li> Models are pretrained for S= 1MM updates</li><li> No layers frozen  </li><li> Same learning rate throughout </li></ul> | Book Corpus and Wikipedia (~16 GB uncompressed) | 256 batch size, maximum length  512 |
| DistilBERT | Encoder-Only (Distilled version of BERT) | **Triplet loss**: (1) MLM + (2) Distillation + (3) Cosine-Distance (No NSP)| Same as BERT | Same as BERT | Same as BERT | Embeddings are similar to BERT, except the segment embeddings are removed | Same as BERT | Same as BERT | 66M parameters | Same as BERT (I think) | Same as BERT | 4096 batch size |
| ALBERT | Encoder-Only | <ul><li> **Masked Language Model (MLM) loss function (refer to BERT)** </li><li> **Sentence Order Prediction (SOP) loss** </li></ul> | SentencePiece (as opposed to BERT's WordPiece) similar to XLNet | ~30k | Same as XLNet (greedy algorithm) | SentencePiece embeddings | Encoder-only self-attention, but with different prob(masking) | GeLU (same as BERT) | **Albert-base (sharing attention layers)**: 12 layers, hidden_size=768, embed_size=128 --> **64 MM parameters**  | <ul><li>Fine-tuning is task specific (see table 14)</li><li>  **LAMB** optimizer was used w/ LR=0.00176 @ 125k steps </li></ul> | Same as BERT | 4096 batch size |
| RoBERTa | Encoder-Only | **Masked Language Model** objective with dynamic masking (see below) + **No NSP or SOP** (NSP removal was shown to be better)| **Byte-level BPE (like GPT)** | 50k | Same as GPT? | Same as BERT | Same as BERT | Same as BERT | Model parameters are kept fixed: <ul><li> L=12, H=768, A=12 -> 110MM parameters (+~15MM for increase in vocabulary with byte-level BPE) </li></ul>  | They increase the pre-training steps from 100k (BERT) to up to 500k. They have a tweak on ADAM hyper-parameters. | They combine **5** datasets for **160MM GB in text**: <ul><li> Book Corpus + Wikipedia </li><li> CC-news </li><li> OpenWebText </li><li> Stories </li></ul>| ~2k batch size, max sequence length ~ 512 (less sometimes due to sampling technique) |
| BART | Encoder-Decoder (Transformer) | **Re-construction loss**: Usual decoder cross-entropy between decoder and original document (encoder sees corrupted document); although, they look at several variants: <ul><li> **GPT::Language Model** </li><li> **XLNet::Permuted Language Model** </li><li> **BERT: MLM** </li><li> **Multitask MLM** </li><li> **Masked Seq-to-seq** </li><ul> They use **two-stream attention** to compute likelihoods. | Same BPE encoding as GPT-2 | Same as GPT? Or RoBERTa?| Same as GPT? Or RoBERTa?| Same as GPT? Or RoBERTa? | Same as the original Transformer | GeLU | <ul><li> BART contains roughly 10% more parameters than equivalent sized BERT model: 6 encoder layers, 6 decoder layers, embed_size==hidden_size=768.</li><li> For large-scale experiments: 12 encoder, 12 decoder, hidden_size=1024.</li></ul> | <ul><li> 5MM steps. Use 30% token masking, permute sentences. </li><li>There is a different training process for NMT (2-step process)</li></ul>| 160 GB of data similar to *Liu et al 2019* | (for large scale experiments) batch_size=8K|
| T5 | Encoder-Decoder | **BERT-style denoising objective**: Similar to MLM, model is trained to predict missing or corrupted tokens in input. 15% of tokens are randomly sampled and dropped out. (Note: They experimented with many variants) | **SentencePiece** | 32k (across many languages w/ 10:1 English-to-non-English) | Same as BERT | Just token embeddings | Self-attention + encoder-decoder attention (per layer) | ReLU | This study looks at many variants, but the base is similar to BERT_base: <ul><li> 12 blocks (encoder + decoder) </li><li> hidden_size == embed_size = 768 </li><li> FFN_dim=3072 (4*hidden) </li></ul> Utimately, about **twice the size of BERT --> 220MM params**. | <ul><li> **Pre-training:** 2^19 steps for pre-training. </li><li> Use **adaFactor** optimization with **inverse square root** LR scheduler </li><li> **Greedy decoding** at test time </li><li> **Fine-tuning**: 2^18 steps always with same batch_size dimensions, LR=0.001, **5k checkpoints and report results for highest validation performance**. | Common Crawl's C4 data (20 TB) | T=512, batch= 128 with packing such that each batch is approximately **65k tokens** (much smaller than other studies) |
| Adapter-BERT | Encoder-Only | Same as BERT (only fine-tuning is happening in this paper) | Same as BERT | Same as BERT | Same as BERT | Same as BERT | Same as BERT | Same as BERT | pre-trained BERT + Adapter Layers | Fine-tuning procedure: ADAM with LR increased during first 10%, followed by decaying to 0 during last 90%. LR is swept ans well as adapter size. |Same as BERT | Batch size = 32 |
| ByT5 | Encoder-Decoder | Similar to **Span Corruption** (T5's pre-training objective) | Tokenless! (uses UTF-8 bytes) | 256 byte values + 3 special tokens | There is still an OOV token, but is not used | Only 256 token embeddings, no positional embeddings | Self-Attention + Encoder-Decoder Attention | ReLU? | Model sizes were made to match mT5 (small, base, large, XL, XXL); to compensate, increased depth of encoder ("heavy encoder") and dim_model, dim_ffn | All hyper-parameters are the same as mT5, except now: <ul><li> Sequence length: 1024 tokens/bytes </li><li> 1MM steps </li><li> batch_size = 2^30 tokens </li></ul> | Same mC4 as mT5 model |T=1024 tokens, batch_size = 2^30 tokens |
| CLIP | Encoder-Only (2 Transformer-based Encoders: Text + Image) | Cross-Entropy loss (to minimize cosine similarity a la constrastive learning) | For Text Encoder: BPE | For Text Encoder: ~50k | Same as GPT | Multi-modal embeddings combining text,image features | Used in both text and image encoders differently | linear projections to embedding | In base Text Encoder, 63M| Adam optimizer with weight decay, Cosine scheduler,learnable temperature | WebImageText dataset, 400MM (text,image) pairs | 32k |
| DALL-E | Decoder-Only (read about attention) | no pre-training/fine-tuning per se | BPE for text + pixel tokens | text=16,384; image=8,192 | greedy? | token | 3 types of attention: text-to-text (causal), text-to-image, image-to-image attention | GeLU? | Up to 12 BN! |  training broken out into 2 steps: 1. dVAE (gumbel-softmax relaxation) 2. transformer (cross-entropy) with 16-bit precision | Conceptual Captions + proprietary dataset |  per-gpu=8, total_batch=512 |
| Codex | Decoder-Only (GPT) | The usual "causal" GPT decoding problem is presented |  BPE | ~50k + white space tokens | GPT-3 construction | GPT-3 construction| GPT-3 construction | GPT-3 construction | Large model was 12 B parameters| Training was similar as GPT: 175 step linear warm up, cosine learning rate decay. Training lasted 100B tokens using Adam Opimitizer with weight decay. | 54 million open repositories on Github were scraped. After a number of filters, the final dataset was 159 GB.| ?|
