title: GPT-3
category: page
slug: gpt-3
sortorder: 1001
toc: False
sidebartitle: GPT-3
meta: GPT-3 is a trained neural network with 175 billion parameters that allows it to be significantly better at text generation than previous models.


[GPT-3](https://arxiv.org/abs/2005.14165) is a neural network
trained by the [OpenAI](https://openai.com/) organization with 
significantly more parameters than previous generation models. 

There are several variations of GPT-3, which range from 125 to 175 billion
parameters. The different variations allow the model to better respond to 
different types of input, such as a question & answer format, long-form 
writing, human language translations (e.g. English to French). The large
numbers of parameters make GPT-3 significantly better at natural 
language processing and text generation than the prior model, 
[GPT-2](https://openai.com/blog/gpt-2-1-5b-release/), which only had 
1.5 billion parameters.

<img src="/img/logos/openai.jpg" width="100%" alt="OpenAI logo." class="shot rnd">

GPT-3 can only currently be access by an 
[API provided by OpenAI](https://openai.com/blog/openai-api/), which is
in private beta.


## What's so special about GPT-3?
The GPT-3 model can generate texts of up to 50,000 characters, with no 
supervision. It can even generate creative Shakespearean-style fiction
stories in addition to fact-based writing. This is the first time that a
neural network model has been able to generate texts at an acceptable 
quality that makes it difficult, if not impossible, for a typical
person to whether the output was written by a human or GPT-3.


## How does GPT-3 work?
To generate output, GPT-3 has a very large vocabulary, which it can 
combine to generate sentences. These words are sorted 
into different categories (nouns, verbs, adjectives, etc.), and for each 
category, there is a “production rule”, which can be used to generate a 
sentence. The production rules can be modified with different parameters.

A few examples:

* noun + verb = subject + verb
* noun + verb + adjective = subject + verb + adjective
* verb + noun = subject + verb
* noun + verb + noun = subject + verb + noun
* noun + noun = subject + noun
* noun + verb + noun + noun = subject + verb + noun + noun

In addition, GPT-3 is able to understand negations, as well as the use 
of tenses, which allows the model to generate sentences in the past, 
present and future.


## Does GPT-3 matter to Python developers?
GPT-3 is not that useful right now for programmers other than as an
experiment. If you get access to [OpenAI's API](https://openai.com/blog/openai-api/) 
then Python is an easy language to use for interacting with it and
you could use its text generation as inputs into your applications.
Although there have been some initial impressive experiments in 
generating code for
[the layout of the Google homepage](https://twitter.com/sharifshameem/status/1283322990625607681),
[JSX output](https://twitter.com/sharifshameem/status/1282676454690451457),
and [other technical demos](https://twitter.com/__MLT__/status/1287357881675853825),
the 
[model will otherwise not (yet) put any developers out of a job](https://www.youtube.com/watch?v=Yg3C38P5EkA)
who are coding real-world applications.


## How was GPT-3 trained?
At a high level, training the GPT-3 neural network consists of two steps.

The first step requires creating the vocabulary, the different 
categories and the production rules. This is done by feeding 
GPT-3 with books. For each word, the model must predict the category 
to which the word belongs, and then, a production rule must be created.

The second step consists of creating a vocabulary and production rules 
for each category. This is done by feeding the model with sentences. 
For each sentence, the model must predict the category to which each 
word belongs, and then, a production rule must be created.

The result of the training is a vocabulary, and production rules for each 
category.

The model also has a few tricks that allow it to improve its ability to 
generate texts. For example, it is able to guess the beginning of a word
by observing the context of the word. It can also predict the next word 
by looking at the last word of a sentence. It is also able to predict the
length of a sentence.

While those two steps and the related tricks may sound simple in theory,
in practice they require massive amounts of computation. Training
175 billion parameters in mid-2020 cost in the ballpark of 
[$4.6 million dollars](https://lambdalabs.com/blog/demystifying-gpt-3/#:~:text=But%20to%20put%20things%20into,for%20a%20single%20training%20run.),
although some other estimates calculated it could take up
to $12 million depending on how the hardware was provisioned.


## GPT-3 resources
These resources range from broad philosophy of what GPT-3 means
for machine learning to specific technical details for how the model
is trained.

* [The Ultimate Guide to OpenAI's GPT-3 Language Model](https://www.twilio.com/blog/ultimate-guide-openai-gpt-3-language-model)
  is a detailed tutorial on how to use OpenAI's playground user interface,
  what the parameters do, and how to convert what you have done in
  the playground into a Python script that calls their API.

* [OpenAI's GPT-3 Language Model: A Technical Overview](https://lambdalabs.com/blog/demystifying-gpt-3/)
  and
  [GPT-3: A Hitchhiker's Guide](https://lambdalabs.com/blog/gpt-3/)
  are two long-format guides that analyze how GPT-3's technical 
  specifications fit in the larger machine learning ecosystem, quotes by
  researchers on its usage, and some initial resources to get a
  better understanding of what this model is capable of performing.

* [What Is GPT-3: How It Works and Why You Should Care](https://www.twilio.com/blog/what-is-gpt-3)
  presents a high-level accessible overview of GPT-3, how it compares
  to other language models, and resources to learn more.

* [How GPT3 Works - Visualizations and Animations](https://jalammar.github.io/how-gpt3-works-visualizations-animations/)
  contains some wonderful animated visuals to show how the model is
  trained and what happens in various scenarios such as text output
  and code generation.

* [GPT 3 Demo and Explanation](https://www.youtube.com/watch?v=8psgEDhT1MM)
  is a video that gives a brief overview of GPT-3 and shows a bunch
  of live demos for what has so far been created with this technology.

* [Tempering expectations for GPT-3](https://minimaxir.com/2020/07/gpt3-expectations/)
  points out that many of the good examples on social media have been
  cherry picked to impress readers.

* [Why GPT-3 matters](https://leogao.dev/2020/05/29/GPT-3-A-Brief-Summary/)
  compares and contrasts this model with similar models that have been
  developed and tries to give an overview of where each one stands
  with its strengths and weaknesses.


## GPT-3 tutorials
* [Building a Chatbot with OpenAI's GPT-3 engine, Twilio SMS and Python](https://www.twilio.com/blog/openai-gpt-3-chatbot-python-twilio-sms)
  is a step-by-step tutorial for using GPT-3 as a smart backend
  for an SMS-based chatbot powered by the [Twilio API](/twilio.html).

* [Automating my job by using GPT-3 to generate database-ready SQL to answer business questions](https://blog.seekwell.io/gpt3)
  walks through how the author created a bridge to translate between 
  plain English-language questions and 
  [relational database](/databases.html) SQL. The post provides both 
  a story for why someone would want to use GPT-3 for this purpose and
  incremental steps for how the author started and figured out how
  to make it better. In the end it does not quite work in all scenarios
  but the proof of concept is impressive and the story is a fun read.

* [gpt-3-experiments](https://github.com/minimaxir/gpt-3-experiments)
  contains Python code open sourced under the MIT license that
  shows how to interact with the API.

* [Twilio](/twilio.html) put out a series of fun GPT-3 tutorials that show
  the range of creative outputs the model can generate:

    * [Control a Spooky Ghost Writer for Halloween with OpenAI's GPT-3 Engine, Python, and Twilio WhatsApp API](https://www.twilio.com/blog/ghost-writer-spooky-openai-gpt3-python-whatsapp)
    * [Generating Lyrics in the Style of your Favorite Artist with Python, OpenAI's GPT-3 and Twilio SMS](https://www.twilio.com/blog/generating-lyrics-in-the-style-of-your-favorite-artist-with-python-openai-s-gpt-3-and-twilio-sms)
    * [Automated Yugioh Deckbuilding in Python with OpenAI's GPT-3 and Twilio SMS](https://www.twilio.com/blog/building-computer-generated-yugioh-decks-in-python-with-openai-s-gpt-3-and-twilio-sms)
    * [Build a Telephone Chatbot with GPT-3 and Twilio Autopilot](https://www.twilio.com/blog/build-telephone-chatbot-gpt3-twilio-autopilot)

