Tokens, Context, and Cost

You now know what tokens are and how text becomes them. Here is why your finance team cares: tokens are the billing unit and the capacity unit of every large language model. Every API invoice, every "context length exceeded" error, and every slow response traces back to token counts.

The problem: invisible meters running

Two meters tick every time you call a model, and neither is measured in words or characters.

Meter 1: the context window. A model can only attend to a fixed number of tokens at once — its context window. Everything must fit inside: your system prompt, the conversation history, any documents you pasted, and the response being generated. Windows today range from ~8,000 tokens to 1,000,000+, but whatever the limit, crossing it is a hard failure or silent truncation, not a graceful degradation.

Meter 2: the bill. APIs charge per token, with two different rates:

• Input tokens (everything you send) — cheaper, because the model processes them in one parallel pass.
• Output tokens (everything the model generates) — typically 3-5x more expensive, because generation happens one token at a time, each requiring a full pass through the model.

Prices are quoted per million tokens. A frontier model might charge a few dollars per million input tokens and several times that for output; small models can be 100x cheaper than large ones.

Why prompts cost money

A common surprise: you pay for your question, not just the answer — and in most real applications, input dominates. Consider a chatbot with a 2,000-token system prompt. Every single message the user sends re-transmits that entire prompt, plus the whole conversation history. By turn 20, each exchange might carry 15,000 input tokens to produce a 200-token reply. Multiply by thousands of users, and the system prompt you wrote once is being billed millions of times.

Do the math yourself

Use the calculator below. Try modeling: (a) a one-off question, (b) a chatbot at turn 20, (c) a pipeline that summarizes 500 documents of 3,000 tokens each. Watch which side of the ledger — input or output — dominates in each case.

Estimating cost in code

import tiktoken

INPUT_PRICE = 3.00 / 1_000_000   # dollars per input token
OUTPUT_PRICE = 15.00 / 1_000_000 # dollars per output token

enc = tiktoken.get_encoding("cl100k_base")

def estimate_cost(prompt: str, expected_output_tokens: int) -> float:
    input_tokens = len(enc.encode(prompt))
    return (input_tokens * INPUT_PRICE
            + expected_output_tokens * OUTPUT_PRICE)

prompt = "Summarize this contract:\n" + open("contract.txt").read()
print(f"Estimated: ${estimate_cost(prompt, 500):.4f} per call")
print(f"At 10,000 calls/day: ${estimate_cost(prompt, 500) * 10_000:.2f}/day")

Production systems run exactly this kind of check before expensive batch jobs — a mispriced loop over a million documents is a very real way to burn a month's budget overnight.

Levers that actually cut costs

• Shorter system prompts. Every word you trim is trimmed from every call, forever.
• History management. Summarize or drop old turns instead of resending everything.
• Prompt caching. Many providers charge heavily discounted rates for repeated prompt prefixes — structure prompts so the static part comes first.
• Model routing. Use a small cheap model for easy requests, escalating to the big one only when needed.
• Capping output. Set max output tokens; output is the expensive direction.

Build it yourself

1. Take a real prompt you have used with any AI assistant. Using the calculator, estimate its cost at 1 call, then at 100,000 calls per day for a month.
2. Rewrite that prompt to be 30 percent shorter without losing meaning. Compute the monthly savings.
3. Sketch (on paper) a chatbot that keeps conversations under 4,000 input tokens per turn no matter how long the chat runs. What do you truncate, summarize, or cache?

Summary

• Tokens are both the capacity unit (context windows) and the billing unit (per-million-token pricing) of LLMs.
• Output tokens cost several times more than input tokens because generation is sequential.
• In real apps, input usually dominates: system prompts and chat history are re-billed on every call.
• Cost levers: shorter prompts, history summarization, prompt caching, model routing, and output caps.
• Always estimate before you loop — token math is cheap, surprise invoices are not.