HeyMSK

Build a small language model

Language models behind tools like ChatGPT generate text one token at a time. They learn patterns from training examples, then use those patterns to predict what comes next. A token can be a word, part of a word or a character.

In this tutorial I use addition to build a small model you can run here. A calculator follows an addition rule. My model learns from examples such as 1+1=2 and predicts one character at a time. That means it can also get the answer wrong.

Run 1+1=, then try 1+1. Compare the continuations.

Try the finished model

Runs on your device. First load is about 14 MB.

Improved base

Exact input: "1+1="

Continuation
No output yet
Input + continuation
1+1=

Run to see a prediction.

Model and input details

Printable ASCII is passed verbatim. BOS is added once. All 98 tokens remain available. Context: 32 tokens. Run budget: 16 output tokens, capped by remaining context. EOS ends generation. Unexpected PAD or BOS tokens are visible.

Improved base. Epoch 140. 1.74 MB of weights. The WASM runtime is downloaded on first use.

Start building

Allow 15 to 30 minutes to explore. Installing Python and training locally take extra time.

  1. Predict one character
  2. Prepare examples
  3. Turn text into tokens
  4. Build the network
  5. Train it
  6. Test it
  7. Teach a new format
  8. Check probabilities
  9. Train from feedback
  10. Compare with larger models
Code and training steps