Context Windows
What a Context Window Is
A context window is the maximum sequence of tokens a language model can process in one request. It contains the information supplied to the model and, depending on the serving system, space reserved for the model's response.
Think of it as a finite workbench. System instructions, conversation history, retrieved documents, tool results, and the current request all compete for space. A larger workbench can hold more material, but it does not guarantee that the model will use every item equally well.
The Token Budget
Text is split into tokens, which may be whole words, parts of words, punctuation, or other symbols. The practical budget is:
input tokens + generated tokens <= model or service limit
The exact accounting depends on the model and API. Hidden formatting, message metadata, tool definitions, images, and reserved output capacity may also consume part of the available budget.
What Uses the Window
- System instructions: Rules that define the model's role and behavior
- Conversation history: Earlier user and assistant messages included in the request
- Retrieved context: Documents or passages supplied by a search or retrieval system
- Tool data: Function definitions, tool calls, and returned results
- Current request: The user's latest input
- Output allowance: Capacity needed for the generated answer
Each model call is based on the context sent with that call. A chat application may appear to remember a conversation, but it normally does so by resending selected history rather than by giving the model permanent memory.
When the Limit Is Reached
A request that exceeds the supported window cannot be processed as-is. The surrounding application or model service may reject it, remove older content, shorten retrieved documents, summarize history, or reduce the maximum output length.
These choices change model behavior. Removing early instructions can lose constraints, while aggressive summarization can discard exact details. Good context management therefore decides what to keep, not just what can fit.
Capacity Is Not Usability
The advertised window is a hard capacity limit, not a promise of uniform accuracy across that range. Models can miss relevant details in long inputs, especially when information is surrounded by distractors or placed in less favorable positions. This reduction in effective use is often called context rot.
Practical Design
- Reserve output space before assembling the prompt.
- Retrieve only material that is relevant to the current task.
- Keep critical instructions explicit and easy to locate.
- Summarize history carefully, preserving decisions, constraints, and unresolved questions.
- Test with realistic context lengths instead of relying only on the advertised maximum.