Last lesson we said an LLM works in tokens, not words, and that you pay per token and get capped by a token limit. Time to actually look at one. Tokenization is just the step that turns your text into those tokens before the model sees it. Once you can see it happen, a lot of LLM behavior stops feeling like magic.

Trying out tokenization

The cleanest way to get this is to run it. OpenAI's tokenizer lives in a small library called tiktoken (pip install tiktoken).

import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")tokens = enc.encode("Blogging about FastAPI is fun")print(tokens)                              # token ids the model actually readsprint(len(tokens))                         # how many tokensprint([enc.decode([t]) for t in tokens])   # the text piece for each token

Run it and that last line shows the real split, something like ['Blog', 'ging', ' about', ' Fast', 'API', ' is', ' fun']. A few things jump out:

1. "Blogging" is not one token. Common words survive whole, but this one breaks into "Blog" and "ging."

2. The spaces did not vanish. " about" includes its leading space. To the model, a space is part of the token, not a gap between them.

3. "FastAPI" splits into "Fast" and "API" because the model never saw it often enough to earn its own token.

Tokenization quirks -

A token is a chunk the model learned during training, so how text splits can surprise you.

Capitalization changes things: "Blog" and "blog" can be different tokens. A leading space matters too, so "blog" and " blog" are not the same. Rare words, long words, and typos split into more pieces. Numbers get chopped in odd ways, so "2026" might be two tokens. And most non-English text costs noticeably more tokens for the same meaning, sometimes two or three times as many, which quietly raises your bill.

None of this is something to fight. It is just useful to know why a short-looking string sometimes costs more than you expected.

Why are you learning about tokenization?

Two very practical reasons, both from the last lesson.

Because it costs money💸💰🤑. Cost is per token, so counting them tells you what a request will run you before you send it. And the context window is measured in tokens, so if you are stuffing a long blog plus a prompt plus the model's reply into one call, you need to know it fits. len(enc.encode(text)) is your quick budget check.

A teeny-tiny habit that pays off: count tokens before shipping user text to a model, so a giant blog post does not blow past the limit and get silently cut off.

def count_tokens(text: str) -> int:    enc = tiktoken.encoding_for_model("gpt-4o-mini")    return len(enc.encode(text))