Last post ended at a ceiling. Our chat history gets resent, rebilled, and reprocessed on every turn, and there is a hard limit on how much the model can even accept. That limit is the context window. Today we measure it, hit it, and handle it.

One desk and everything on it

The context window is the model's total token budget for a single call. The system message, the whole chat history, the new user message, and the reply the model is about to write all share that one budget. It varies by model: a small local model might give you 4k tokens, GPT-4o gives 128k, the biggest ones stretch into the millions.

Think of it as desk space. The model can only look at what fits on the desk. Nothing hangs off the edge.

Back in How LLMs Work I said "go past it and the oldest text drops off." Small confession: the model drops nothing. The raw API just refuses your request. The dropping is done by apps like ChatGPT, quietly, before sending. By the end of this post, you will be doing it too.

Watch the history grow

Our chat() function from Messages and Roles appends to messages forever. Let's put a meter on it with tiktoken from the Tokenization post:

import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")def history_tokens():    return sum(len(enc.encode(m["content"])) for m in messages)chat("My name is Sourabh. Say hi.")print(history_tokens())chat("Recommend three Python books.")print(history_tokens())chat("Why those three?")print(history_tokens())

Three things worth noticing here:

1. The number only goes up. Every reply gets appended as assistant and becomes input you pay for on every later request. Turn 50 of a chat carries turns 1 through 49 on its back.

2. The model's own replies are the fast growers. Your questions are one line, its answers are three paragraphs.

3. The real count is a bit higher than ours, since APIs add a few formatting tokens per message. Close enough for a budget check.

Two ways to hit the ceiling

The loud way: send more tokens than the window holds and the API rejects the whole request with an error complaining about context length. No reply, just a 4xx and regret.

The quiet way is sneakier. Input and output share the window, so a huge input leaves almost no room for the reply. The model starts answering and gets cut off mid-sentence, and finish_reason says "length". Same symptom as the max_tokens cap from last post, different cause. 😬

Trimming: the sliding window

The standard fix is to keep the system message, keep the most recent messages, and stop sending the middle:

MAX_MESSAGES = 5def chat(user_text):    messages.append({"role": "user", "content": user_text})    trimmed = [messages[0]] + messages[1:][-MAX_MESSAGES:]    response = requests.post(url, json={"messages": trimmed}, timeout=30)    reply = response.json()["choices"][0]["message"]["content"]    messages.append({"role": "assistant", "content": reply})    return reply

Compare it with the version from Messages and Roles, only one line is new. messages[0] pins the system message so the model never forgets its job, and messages[1:][-MAX_MESSAGES:] sends just the last 10 messages, which is 5 back-and-forth turns. We still store the full history locally, we just stop mailing all of it.

The tradeoff is real: once turn 1 slides out of the window, the model genuinely forgets your name from turn 1. The fancier fix is to summarize old turns into one short message instead of dropping them, but that costs an extra LLM call. Start with the sliding window, upgrade when forgetting becomes a problem.

Back to our Blog

Our /blog/summarize endpoint takes user-submitted content, and users will absolutely paste a novel into it. Remember the count_tokens habit from the Tokenization post? This is where it pays off:

MAX_INPUT_TOKENS = 3000@app.post("/blog/summarize")def summarize_blog(blog: Blog):    if count_tokens(blog.content) > MAX_INPUT_TOKENS:        raise HTTPException(status_code=413, detail="Blog too long to summarize in one call")    payload = {        "messages": [            {"role": "system", "content": "You are an editor for a tech blog. Write summaries in exactly two sentences, plain language, no marketing fluff."},            {"role": "user", "content": blog.content}        ],        "temperature": 1,        "max_tokens": 150,    }    return {"title": blog.title, "summary": call_llm(payload)}

One guard clause, and oversized blogs now get a clear 413 Payload Too Large instead of a confusing failure from the LLM provider. Rejecting early with an honest message beats letting the error surface three layers deep. Your future self, reading logs at midnight, says thanks.