HeyMSK
Chapter 10 of 10

Compare with larger models

My model has 413,440 parameters and a 32-token context. Larger language models also learn weights and predict tokens, but use broader data, larger networks and different tokenizers and training choices.

This arithmetic model cannot hold a conversation. Its small size lets me inspect the same basic training process. Use the source below to build and train it locally.

Try hello with different stages. Open Code and training steps to reproduce the model.

What this small experiment contains
Parameters413,440
Transformer blocks2
Attention heads per block4
Embedding width128
Vocabulary98 tokens
Context32 tokens

More parameters give a model more learned settings. They also increase memory and computation.

Build this step

The pieces are examples, token IDs, a network, a loss, weight updates and a generation loop. Running the build commands creates a model from random parameters using this data pipeline.

The browser uses an ONNX file, which packages the trained network for another runtime. Exporting changes how the model is executed. It does not teach it new arithmetic.

Open the complete build sequence to run the project on your machine. Recorded results and file verification are available if you want to compare your run with mine.

Explore the completed models

Try hello, then an arithmetic prefix. All five saved stages are available now. Compare what they learned and where they fail.

Runs on your device. First load is about 14 MB.

Exact input: "hello"

Continuation
No output yet
Input + continuation
hello

Run to see a prediction.

Model and input details

Printable ASCII is passed verbatim. BOS is added once. All 98 tokens remain available. Context: 32 tokens. Run budget: 16 output tokens, capped by remaining context. EOS ends generation. Unexpected PAD or BOS tokens are visible.

Improved base. Epoch 140. 1.74 MB of weights. The WASM runtime is downloaded on first use.

Technical details

Browser inference uses float32 ONNX graphs in a single-thread WASM worker. Model files are downloadable and checked by SHA-256. The page contains static article text, while the network runs locally after you load it.

Code and training steps