Chapter 1 of 10

Predict one character

I want the model to continue text such as 1+1= with an answer. It will choose one character, append it to the input and repeat until it decides to stop.

First, try that loop yourself. We have only defined the task at this point. Later chapters create the examples, convert them to numbers and build the network that makes the choices.

Append 2 to 1+1=, then end the answer. Reset and append another character. Notice that every choice changes the text available for the next prediction.

Try the prediction loop

Choose each character yourself. The network has not been built yet.

Starting textYour next character

1+1=

Append your chosen character. Notice how it becomes part of the input for the following prediction.

Start with the source

I use Python and PyTorch. PyTorch supplies arrays, matrix operations and automatic derivatives. I define the tokenizer and network myself, then start with random weights. No pretrained model is loaded.

Set up the project, then follow the code in each chapter. Commands below run from the course folder with the Python environment activated.

course/
  scripts/prepare_data.py   # create addition examples
  tiny_gpt/tokenizer.py     # characters to numbers
  tiny_gpt/model.py         # the network
  tiny_gpt/train.py         # learn the weights
  tiny_gpt/generation.py    # predict new characters

You can run the supplied files as you learn, or write your own versions from the displayed source. For now, the loop only appends a supplied character:

prefix = "1+1="
next_character = "2"  # your choice in this exercise
prefix = prefix + next_character
print(prefix)        # 1+1=2

The next prediction will receive the updated prefix. Chapter 4 builds the function that scores characters. Chapter 5 trains it to make better choices.

Technical details

This exercise appends your chosen character to the input. It does not run a network or calculate an answer. The completed model will perform the same repeated append operation, choosing characters from its own output scores.

Code and training steps