Roles in LLM API | FASTAPITUTORIAL

Last post ended with a promise: we would look at those roles properly. Time to pay up.

So far every request we sent had exactly one message, from one role: user. But the messages list can hold three kinds of speakers, and knowing who says what is most of the game in controlling an LLM.

Meet the cast

messages = [    {"role": "system", "content": "You are a strict editor for a tech blog."},    {"role": "user", "content": "Can you review my intro paragraph?"},    {"role": "assistant", "content": "Sure, paste it here."},    {"role": "user", "content": "FastAPI is a modern web framework..."}]

system is you talking to the model as its boss. Instructions, persona, rules. The model gives these more weight than anything else in the list.
user is the human. Questions, data, requests.
assistant is the model itself. Wait, why would we send the model its own replies back? 🤔 Hold that thought for two minutes.

Putting system to work

Our /blog/summarize endpoint from last post produced whatever style of summary the model was in the mood for. Let's pin it down:

main.py

@app.post("/blog/summarize")def summarize_blog(blog: Blog):    payload = {        "messages": [            {"role": "system", "content": "You are an editor for a tech blog. Write summaries in exactly two sentences, plain language, no marketing fluff."},            {"role": "user", "content": blog.content}        ]    }    response = requests.post(LLM_URL, json=payload, timeout=30)    if response.status_code != 200:        raise HTTPException(status_code=502, detail="The LLM did not respond, try again")    data = response.json()    return {"title": blog.title, "summary": data["choices"][0]["message"]["content"]}

Notice the split: the rules live in system, the blog content lives in user. Before, we crammed the instruction and the content into one user message. That works, but separating them keeps instructions stable while the data changes per request, and models are trained to treat system instructions as harder to override. Cleaner and safer.

The model remembers nothing

Here is the part that surprises everyone. Run this:

import httpxurl = "https://algoholia.com/api/free-llm/v1/chat/completions"first = {"messages": [{"role": "user", "content": "My name is Po the Panda. Say hi."}]}httpx.post(url, json=first, timeout=30)second = {"messages": [{"role": "user", "content": "What is my name?"}]}response = httpx.post(url, json=second, timeout=30)print(response.json()["choices"][0]["message"]["content"])

The model has no clue who you are. Each API call is a complete stranger walking in fresh. LLM APIs are stateless, just like HTTP itself. ChatGPT only feels like it remembers you because the app quietly resends the entire conversation with every new message.

Faking memory, the honest way

So that is what the assistant role is for. We keep the history ourselves and replay it:

main.py

messages = [{"role": "system", "content": "You are a helpful assistant."}]def chat(user_text):    messages.append({"role": "user", "content": user_text})    response = httpx.post(url, json={"messages": messages}, timeout=30)    reply = response.json()["choices"][0]["message"]["content"]    messages.append({"role": "assistant", "content": reply})    return replyprint(chat("My name is Po the Panda."))print(chat("What is my name?"))

Now it "remembers". Every turn we append the user message, get a reply, append that reply as assistant, and send the whole thing back next time. Memory is not a model feature, it is a list you maintain. 🙂

One thing you may have spotted: this list grows forever, and we know from the tokenization post that every message costs tokens. We will deal with that soon.

Next up, the model's dials, We will learn about some parameters which can help us tweak the models response.

messages = [ {"role": "system", "content": "You are a strict editor for a tech blog."}, {"role": "user", "content": "Can you review my intro paragraph?"}, {"role": "assistant", "content": "Sure, paste it here."}, {"role": "user", "content": "FastAPI is a modern web framework..."}]

@app.post("/blog/summarize")def summarize_blog(blog: Blog): payload = { "messages": [ {"role": "system", "content": "You are an editor for a tech blog. Write summaries in exactly two sentences, plain language, no marketing fluff."}, {"role": "user", "content": blog.content} ] } response = requests.post(LLM_URL, json=payload, timeout=30) if response.status_code != 200: raise HTTPException(status_code=502, detail="The LLM did not respond, try again") data = response.json() return {"title": blog.title, "summary": data["choices"][0]["message"]["content"]}

import httpxurl = "https://algoholia.com/api/free-llm/v1/chat/completions"first = {"messages": [{"role": "user", "content": "My name is Po the Panda. Say hi."}]}httpx.post(url, json=first, timeout=30)second = {"messages": [{"role": "user", "content": "What is my name?"}]}response = httpx.post(url, json=second, timeout=30)print(response.json()["choices"][0]["message"]["content"])

messages = [{"role": "system", "content": "You are a helpful assistant."}]def chat(user_text): messages.append({"role": "user", "content": user_text}) response = httpx.post(url, json={"messages": messages}, timeout=30) reply = response.json()["choices"][0]["message"]["content"] messages.append({"role": "assistant", "content": reply}) return replyprint(chat("My name is Po the Panda."))print(chat("What is my name?"))