GlossaryTechnology

Context Window

Term 37 of 175 · Technology

In one sentence

The context window is the maximum amount of information (measured in tokens) that a language model can process at once, adding up the user's question, the instructions, the attached data and the response it generates.

Reviewed by Juan Manuel Garrido

Co-founder of VantegrateLinkedIn

Area: Enterprise AI Agents

Context Window definition

The context window is the limit on the information a large language model (an LLM) can "keep in view" while it generates a response. It is measured in tokens (fragments of words), and it covers everything that goes in and comes out in a single interaction: the system instructions, the user's question, the documents or data you attach, the conversation history and the model's own response.

Think of it as the model's short-term working memory: if something does not fit in that window, the model does not "see" it and cannot use it to reason. When a conversation or a document exceeds that limit, the oldest content gets trimmed and starts to be lost, which explains why an assistant sometimes seems to "forget" what was said at the beginning of a long chat.

In the context of AI agents, the context window is a critical resource: it defines how much of your business information (a manual, a customer history, a catalog) the agent can consider before acting. That is why it works alongside techniques such as RAG, which retrieves only the relevant fragments so the window does not get overloaded.

The context window is one of the most misunderstood concepts in generative AI, and understanding it well prevents the wrong expectations when you implement assistants or agents. It is not the model's "permanent memory" or its general knowledge (that comes from training): it is the temporary space that holds the information for one specific query.

How it is measured and why in tokens

The window is measured not in words or characters, but in tokens. A token is a fragment of text: a common word can take up one or more tokens, and as a rule of thumb Google estimates for its Gemini models that 100 tokens are roughly 60 to 80 English words (Google). By that rule, a 128,000-token window is enough for about 77,000 to 102,000 English words, the equivalent of a short book, and somewhat less in Spanish. The key point is that this budget is shared: every token your instructions and attached data take up is one token less available for the response.

Input and output share the same budget

This is the part that most surprises anyone designing their first assistant. The window is split among three blocks:

System prompt: the fixed instructions that define how the model behaves (tone, rules, format).
User context: the current question, the conversation history and any document or data injected into it (for example, via RAG or tool calling).
Generated response: what the model produces also consumes tokens from the same window.

If you fill almost the entire window with input data, there is little room left to respond, and the response may come out cut off. That is why managing the window is a design job, not something you solve just with a "bigger" model.

Why it matters in a real project

In a B2B deployment, the context window defines how "informed" an agent can be without turning to external systems. A concrete case in Argentina: a consumer goods company builds an AI assistant so its sales force can look up commercial policies, prices and terms by customer. If they try to put the full catalog, the price list and the history of every account inside the window on every query, the cost skyrockets (you pay per token) and latency goes up. The solution is not to keep enlarging the window, but to use RAG to retrieve only the relevant fragments from a vector database and inject them right when they are needed.

Common mistakes

Believing a bigger window is always better. More tokens mean more cost and more latency per query. A large window is a tool, not an end in itself.
Assuming the model "remembers" everything that fits. There is a phenomenon called "lost in the middle": models pay more attention to the beginning and the end of the window than to information placed in the middle of a very long text.
Confusing the context window with persistent memory. When the session closes, that context disappears. For an agent to "remember" across conversations, that information has to be stored and retrieved explicitly (for example, from a CRM or a database).
Filling it with noise. More context is not better context. Injecting irrelevant documents degrades the quality of the response and increases the risk of hallucination.

Context window vs RAG: they do not compete, they complement each other

A common confusion is choosing "between" enlarging the window or using RAG. In practice they work together: the window is the container, and RAG decides what goes inside it.

Aspect	Context window	RAG (retrieval-augmented generation)
What it solves	How much information fits per query	Which information to bring into that window
Nature	Fixed limit of the model	Dynamic retrieval technique
Cost at scale	Grows with every token included	Keeps the window bounded
Knowledge it provides	Only what you inject at that moment	Access to large external corpora
Main risk	Overloading it or cutting off the response	Retrieving fragments that are not very relevant

The rule of thumb: use the window for the immediate context of the task, and RAG to give the model access to a body of knowledge that would never fit in full within any token limit.

Frequently asked questions

FAQs about Context Window

What is the context window in AI?

The context window is the maximum amount of information, measured in tokens, that a language model can process at once. It includes the instructions, the user's question, attached documents, the conversation history and the response the model generates. If something does not fit in that window, the model does not take it into account when reasoning. It works like a temporary working memory: when a conversation or a text exceeds it, the oldest content is trimmed and lost.

What is the context window measured in?

It is measured in tokens, which are fragments of text smaller than a full word. A common word can take up one or more tokens, and as a practical reference Google estimates for its Gemini models that 100 tokens equal about 60 to 80 English words. By that rule, a 128,000-token window is enough for about 77,000 to 102,000 English words, similar to a short book, and somewhat less in Spanish. That token budget is split between the input information and the response the model generates.

Is a larger context window always better?

Not necessarily. A larger window lets you process more information at once, but it also increases the cost (you usually pay per token) and the latency of each response. In addition, models tend to pay less attention to information placed in the middle of very long texts, a phenomenon known as lost in the middle. In many cases it is better to keep the window bounded and use techniques such as RAG to inject only the relevant fragments, instead of loading all the content.

What is the difference between the context window and the model's memory?

The context window is short-term working memory: it exists only during an interaction and disappears when the session ends. The model's general knowledge, by contrast, comes from its training and is fixed. For an agent to remember information across different conversations, the context window is not enough: you have to store that data explicitly in an external system, such as a CRM or a database, and inject it again when it is needed.

How does the context window relate to RAG?

The context window and RAG complement each other instead of competing. The window is the container that defines how much information fits per query, while RAG is the technique that decides which information to bring into that window. Instead of trying to squeeze a huge corpus into the token limit, RAG retrieves only the most relevant fragments from a vector database and injects them at the right moment. That keeps the window bounded, controls cost and gives the model access to knowledge that would never fit in full.

An AI agent that already knows how to do this

We implement AI agents on the CRM you already use, for sales, collections and support. Tell us which process eats your day and we will tell you straight whether an agent solves it.

Let's talk on WhatsApp See an agent in action

Keep exploring