Tokens: The LEGO Bricks of AI

Models Don't Read Words

Plot twist for everything you learned in the last lesson: when we said LLMs predict "the next word," we were simplifying. Models don't actually deal in words. They deal in tokens, the LEGO bricks of language.

Before your prompt ever reaches the model, a piece of software called a tokenizer chops it into chunks. Common words usually survive as a single token. Rarer or longer words get snapped apart into pieces. Then each token is swapped for a number, because deep down, the model is just doing math on lists of numbers.

cat: one token. Short, common, unbothered.
unbelievable: might become un + believ + able. Three bricks.
Supercalifragilistic: a small pile of bricks and a tokenizer sweating.

See It Live

Enough telling. Type anything into this visualizer and watch the tokenizer do its chopping in real time. Try your name, a long word, some emoji, a number like 1234567. The weird cases are the fun ones.

🔤 Token visualizer

Loading tokenizer… 🧮

Things worth trying while you're in there:

A common word vs a made-up word (compare running with flurbling)
Your name: common names are often one token; unusual ones get diced
Numbers: watch long numbers split in unintuitive places
A word with and without a leading space: spacing changes tokenization more than you'd expect

Why Words Split in Surprising Ways

Tokenizers are built by scanning mountains of text and keeping the most frequently occurring chunks as the brick set, typically tens of thousands of them. Frequency, not grammar, decides the cuts. That's why splits can look bizarre: tokenization might split as token + ization, which is almost linguistic, while some perfectly normal word splits somewhere ugly in the middle. The tokenizer never took an English class. It took a statistics class.

Rule of thumb for English: one token is roughly three-quarters of a word, so 1,000 tokens is about 750 words. Handy later, when we talk about context windows and why AI services bill by the token.

The Strawberry Problem

Here's a party trick: ask an LLM how many letter R's are in "strawberry." Models have famously flubbed this, and tokens are the reason.

You see s-t-r-a-w-b-e-r-r-y: ten letters, easy to count. The model may see something like str + aw + berry: three opaque bricks, each just an ID number. The individual letters inside each brick are not directly visible; asking the model to count letters is like asking you to count the studs inside a sealed LEGO piece. The letter-level detail was smoothed away before the model ever saw the text.

This is why a model can write you a genuinely moving poem about strawberries and still fumble counting the R's. It's not stupidity. It's eyesight. Tokens are the only glasses it has.

The same effect explains other odd weak spots: spelling words backwards, precise character-level edits, rhyming under strict letter constraints. Anything requiring letter-by-letter vision fights against the brick-by-brick reality.

Why Should You Care?

Three practical reasons tokens matter beyond trivia night:

Cost. API pricing is per token, in and out. Verbose prompts literally cost more.
Limits. The context window (Lesson 8) is measured in tokens: it's the model's total budget for reading plus writing.
Debugging weirdness. When a model botches spelling, counting, or slicing text, you'll know it's usually a token problem, not a model having a bad day.

Here's the sentence from the strawberry problem, pre-chopped for your inspection:

🔤 Token visualizer

Loading tokenizer… 🧮

Checkpoint

Models read tokens, not letters. Text gets chopped into frequent chunks, which explains costs, limits, and why counting letters is hard.

Token Quiz

📝 Quiz

Question 1 of 4

What is a token, in LLM terms?