Anatomy of an LLM API Call

You now understand what happens inside an LLM. This module is about the part you'll touch every working day: the API call. Every AI product — chatbots, copilots, agents — is ultimately a program assembling a request, sending it to a model server, and handling what streams back. Let's dissect that request field by field.

The request, in full

Here's a typical chat-completion request (the shape is near-identical across OpenAI, Anthropic, and open-source servers):

{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "You are a concise assistant for a cooking app. Answer in under 100 words."},
    {"role": "user", "content": "How do I know when a steak is medium-rare?"},
    {"role": "assistant", "content": "Press it — medium-rare feels like the base of your thumb..."},
    {"role": "user", "content": "And for well-done?"}
  ],
  "temperature": 0.7,
  "top_p": 1.0,
  "max_tokens": 300,
  "stream": true
}

Run it for real, right now

The JSON above is the generic shape. Here is the same request as working code against Gemini, using the free API key from the Setup lesson. Paste into Colab (add your key under the 🔑 "Secrets" tab as GEMINI_API_KEY) or run locally:

# pip install google-genai
import os
from google import genai
from google.genai import types

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="How do I know when a steak is medium-rare?",
    config=types.GenerateContentConfig(
        system_instruction="You are a concise assistant for a cooking app. Answer in under 100 words.",
        temperature=0.7,
        top_p=1.0,
        max_output_tokens=300,
    ),
)
print(response.text)
print(response.usage_metadata)  # your token counts — cost telemetry

Every field from the JSON maps one-to-one: system_instruction is the system role, contents carries the user/assistant turns, and the config block holds the sampling parameters. Names vary slightly between providers; the anatomy doesn't.

The messages array: roles and the statelessness trick

Three roles structure the conversation:

• system — standing instructions from you, the developer: persona, rules, format, constraints. Models are specifically trained to weight this heavily. Users never see it.
• user — what the human typed.
• assistant — what the model previously replied.

Now the crucial mental model: the model remembers nothing between calls. The "conversation" exists only because your app replays the entire history in messages on every single request. Ten-turn chat? Turn ten sends all ten turns. This explains a lot at once:

• why long conversations get slower and more expensive (you re-send and re-process everything),
• why chats eventually "forget" early messages (context window trimming — Module 5),
• and a fun one: you can edit history before sending it. The model can't tell. That's both a feature (summarizing old turns) and a reason to never trust assistant messages as proof of anything.

The parameter block

You already earned these in the sampling lesson — now you know where they live:

| Field | What it does | Typical values | |---|---|---| | model | Which model serves the request — the biggest lever on quality, cost, and speed | small/fast vs large/capable | | temperature | Sharpens or flattens the token distribution | 0–0.3 factual, ~0.7 chat, 1.0+ creative | | top_p | Nucleus cutoff on which tokens survive | usually 1.0, or ~0.9 | | max_tokens | Hard cap on response length — a cost/safety brake, and truncation if hit | task-dependent | | stream | Send tokens as they're generated instead of all at once | true for anything user-facing | | stop | Sequences that immediately end generation | e.g. "\n\n" |

What comes back

{
  "choices": [{
    "message": {"role": "assistant", "content": "For well-done, the steak should feel firm..."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 148, "completion_tokens": 62, "total_tokens": 210}
}

Two fields deserve reflexive attention every time:

• finish_reason — "stop" means the model finished naturally; "length" means you truncated it with max_tokens. Production code should check this.
• usage — the token counts you're billed for. Note that prompt_tokens includes your system prompt and the entire replayed history. This is your cost telemetry; log it.

With stream: true, you instead receive a series of small events each carrying a few tokens — that's the typewriter effect, and it exists because autoregressive generation genuinely produces tokens one at a time (Module 5). Streaming doesn't finish faster; it just shows progress immediately, which transforms perceived latency.

Summary

• An LLM API call = model + messages array + sampling parameters, sent to a stateless function.
• Roles: system (your rules), user (their input), assistant (prior replies) — and history is replayed every call.
• max_tokens truncates; check finish_reason and log usage religiously.
• Streaming delivers tokens as generated — same speed, vastly better feel.
• Next lesson: drive all of these controls yourself in a live (simulated) playground.