We just spent a whole section on FastAPI: routes, request types, background tasks. Now we switch to the other half of this course, the AI part. Before you call an LLM from an endpoint, it pays to know what the thing on the other end is actually doing. It is simpler, and stranger, than it looks.
It predicts the next token
Strip away the noise and a large language model does one thing: given some text, it guesses what comes next. You hand it "This blog is about" and it predicts the most likely next chunk, then the next, then the next, until it decides to stop.
It learned this by reading an enormous pile of text until it got very good at "what usually follows these words." Nobody hand-wrote the rules. It saw enough examples to pick up the patterns on its own.
Tokens, not words
The model does not think in words. It works in tokens, which are pieces of words. "Blogging" might split into "Blog" and "ging." A common short word is one token, a long or rare one can be several.
Two reasons this matters to you:
Cost and limits are counted in tokens, not words. You pay per token, and every model has a token ceiling.
That ceiling is the context window: the total tokens the model can look at in one shot, your input plus its output. Go past it and the oldest text drops off.
Rough rule of thumb: one token is about four characters of English, so 100 tokens is around 75 words.
It is guessing, on purpose
Here is the part that trips people up. The model does not pick one "correct" next token. It holds a probability for every possible token and samples from them. That sampling is why the same prompt can hand you a slightly different answer each time you run it.
A knob called temperature controls how adventurous that sampling gets. Low temperature, it plays safe and takes the obvious next token. High temperature, it wanders more. For a "summarize this blog" endpoint you want it low. For "give me ten blog title ideas" you want it higher.
It has no memory
Every call starts from scratch. The model does not remember your last request. If you want it to know what was said earlier, you send that history back in with the new request. That is exactly why a chat app resends the whole thread each time.
And since it predicts plausible text instead of looking facts up, it can say something wrong with complete confidence. That is a hallucination. Hold onto this one: an LLM is a fluent guesser.