The context window
Everything a model reads to produce an answer — your instructions, the conversation so far, files, tool results — sits in one place called the context window. It is measured in tokens, pieces of words; a token is roughly three-quarters of an English word. Current models hold around a million tokens.
Three facts follow, and they explain most surprises:
- The model has no memory between calls. A chat feels continuous because the whole conversation is sent again every time. Start a new chat and everything from the old one is gone unless you bring it.
- If it isn't in the window, it doesn't exist. Your codebase, your company's rules, yesterday's decision — the model knows none of it until it is put there.
- You pay for all of it, every call. A long conversation costs more per message than a short one, because the whole history is input each time.
The right things, not everything
Prompting was about the brief. Context is the rest: the material the model works from. A big window tempts you to paste in everything. Resist it — irrelevant material costs money, slows the answer, and gives the model more to be distracted by. A stale file or an outdated plan in the window is worse than nothing: the model will use it.
- Include what a new colleague would need to do this task: the relevant code or document, the constraint, an example of what good looks like.
- Leave out what they wouldn't: unrelated files, old attempts, the full log when ten lines matter.
- Say where things are rather than pasting them, when the model has tools to look: "the handlers are in src/api" beats 40 files of handlers.
Three ways to get information in
- Put it in the prompt. Simplest, and right whenever the material is small enough.
- Retrieve it. Keep the material in a searchable store and include only the pieces that match the question. This is RAG (retrieval-augmented generation).
- Let the model fetch it. Give it tools — search, read a file, query a database — and it pulls in what it needs as it works. This is how agents do it.
Integrate shows each of these in code.
Order matters
Put the long, stable material first — instructions, reference documents — and the specific question last. Models answer better when the question follows the material, and it makes caching possible: providers can remember a prefix you send repeatedly and charge about a tenth of the price for it on later calls. Anything that changes (a date, a user name, the question) breaks the cache from that point on, so keep it at the end.
Long sessions go stale
In a long chat or agent session the window fills with everything that happened: dead ends, superseded versions, long tool output. Quality drifts — the model repeats an abandoned approach or forgets an early instruction. The fixes are simple:
- Start fresh when you change tasks. A new session with a three-line summary beats an old one with two hours of history.
- Compact. Agent tools can summarize the session so far and continue from the summary. Do it at a natural stopping point, not mid-problem.
- Write it down. Decisions and state that must survive belong in a file — a plan, a notes file, the project's instruction file — not only in the conversation.
Words to know
Tokens, context window, caching, RAG — these come up on every page from here on. The Glossary has one-line definitions when you need a reminder.