Let me name the destination before we take today's step. Everything since the Vectors post has been building toward one feature: a reader asks our Blog a question, we retrieve the most relevant thing we ever wrote, paste it into the prompt, and the LLM answers from our actual posts instead of its vague memories of the internet. That pattern is RAG, Retrieval Augmented Generation. Retrieve first, generate after. Last post handed us the retrieval engine, an embedding model plus cosine_similarity. But feed that engine whole blog posts and it disappoints. Today is about why, and about the fix with the least glamorous name in GenAI: chunking.

The averaging problem

Picture our "Deploying FastAPI" post: two thousand words on Docker, Nginx, environment variables, and one paragraph complaining about cloud bills. Embedding squashes all of that into a single point on the meaning map. Where does the point land? Somewhere in the middle of those topics. An average.

Now a reader asks "how do I set environment variables?" The question's vector points sharply at env vars. The post's vector points at deployment-in-general. The match comes out lukewarm even though the perfect paragraph is sitting right there, buried mid-post. Long text makes blurry vectors.

And suppose the lukewarm match wins anyway. We then paste two thousand words onto the model's desk (the Context Window post's desk, where every token pays rent) to deliver one useful paragraph. Blurry retrieval and an inflated bill, in one move.

Cut first, embed after

The fix is to stop embedding posts and start embedding pieces of posts. A piece about env vars points at env vars, nothing else. 🔪

def chunk_text(text, chunk_size=200, overlap=50):    chunks = []    start = 0    while start < len(text):        chunks.append(text[start:start + chunk_size])        start += chunk_size - overlap    return chunks

A few decisions are hiding in those eight lines:

1. chunk_size counts characters. Real systems usually count tokens instead (tiktoken from the Tokenization post does the counting), but characters keep the idea visible.

2. The window moves chunk_size - overlap forward each pass, so every chunk re-reads the tail of the one before it. That overlap is not decoration, it is the whole trick. Demo below.

3. No imports, no API calls. Chunking is plain string surgery, done before any model gets involved.

Why overlap earns its keep

Chop three innocent sentences with a tiny window:

text = "Docker packs the app. Nginx sits in front. Env vars hold the secrets."for chunk in chunk_text(text, chunk_size=30, overlap=10):    print(repr(chunk))

'Docker packs the app. Nginx si''. Nginx sits in front. Env var''t. Env vars hold the secrets.'' secrets.'

The first chunk beheads the Nginx sentence mid-word. Without overlap, no chunk would ever contain that sentence whole, and a question about Nginx would find nothing worth matching. With overlap, chunk two rewinds ten characters and catches it intact. Every sentence that gets cut somewhere survives complete somewhere else.

Yes, the last chunk is a sad little stump. Real chunkers merge or drop those. Ours is allowed to be nine characters of comedy.

Smarter knives exist

Fixed-size chunking cuts mid-sentence on purpose and trusts overlap to patch the damage. You can also split on paragraph breaks so chunks follow the author's own structure, or on sentence boundaries, or go full semantic chunking where an embedding model decides where the topic shifts. The tradeoff is always the same: cleaner cuts, more moving parts. Fixed-size with overlap is the default first move in most production systems, so it is the one we keep.

Next post we embed a reader's question, let most_similar fetch the best chunks, and hand them to the LLM with orders to answer from them. Retrieval, then generation. RAG, assembled from parts you already own. 🙂