AI, explained

What is a context window?

It is the model's working memory: the total span of text it can read and reason over in a single request, measured in tokens.

Think of the context window as a desk. Everything the model needs for the current task has to fit on that desk at once: the brief, the reference documents, the running conversation, and the space to write its answer. When the desk fills up, older material slides off the edge.

Why it matters in real use

Two practical failures come from ignoring the window. The first is silent truncation. Paste a long contract into an assistant with a small window and the tail of the document simply never reaches the model, so it answers confidently from half the text. The second is conversational amnesia. In a long chat, early instructions can fall out of view, and the assistant starts contradicting things it agreed to twenty messages ago.

Bigger windows are not a free upgrade

Modern models offer very large windows, sometimes hundreds of thousands of tokens. That is genuinely useful, but it has two costs. Every token in the window is processed, so a huge prompt is slower and more expensive per request. And models exhibit a "lost in the middle" effect: information buried in the centre of a very long context is recalled less reliably than material at the start or end. A larger window is a bigger desk, not a better filing system.

What good teams do instead

• Retrieve, do not dump. Rather than pasting an entire knowledge base, use retrieval to pull only the passages relevant to the question. This is the core reason RAG systems stay accurate and cheap.
• Summarise rolling history. In long-running agents, compress older turns into a short summary so the useful signal survives while the token count stays manageable.
• Put critical instructions where they are seen. Keep the essential rules near the top or bottom of the prompt, not buried mid-document.

In production automation, most of the engineering effort is not about buying the biggest window; it is about deciding what deserves to be on the desk for each task. If your AI tools feel forgetful or inconsistent on long inputs, the context strategy is usually the fix, and an AI audit will show where the leaks are.

Frequently asked questions

Related