By turn 34 of a long support chat, an assistant can ask the customer for the account number they typed at turn 3 and still be nowhere near the model’s limit. Nothing crashed. The number is still in the window, buried under tens of thousands of tokens of account JSON the assistant fetched over and over.
I think about the context window the way I’d think about memory in a constrained program. You don’t wait for the out-of-memory error to decide who gets how much. Every kind of content gets an allocation, a position and an eviction rule before the first long session runs. “Context engineering” has become the name for this in the past month or so. Most of the work is bookkeeping.
A working budget well under the window
A 200K window, or a 1M one, is the hard limit, and I don’t budget against it. Every token is paid for again on every turn, in money and in latency. Models also attend less reliably to the middle of a very long prompt, so quality sags long before the limit does. I pick a working number and split it. This one is 64,000 tokens.
| Section | Order | Cap | Eviction rule | Cacheable |
|---|---|---|---|---|
| Instructions, tool definitions | 1 | 8,000 | None. Over the cap fails the deploy | Yes |
| Session summary | 2 | 3,000 | Rewritten only at compaction | Yes, between compactions |
| Conversation history | 3 | 24,000 | Oldest turns fold into the summary | Yes, append-only |
| Tool results, as records | 4 | 16,000 | Latest copy per record, then least recently fetched | No |
| Retrieved facts | 5 | 8,000 | Lowest relevance score first | No |
| Current user message | 6 | 5,000 | None. Over the cap is rejected | No |
The numbers are a starting point. What I care about is that each row has all five columns filled in before anyone runs a long session.
Order decides what caches
Both big providers discount a repeated prefix. Anthropic’s prompt caching has you mark breakpoints with cache_control. OpenAI’s applies automatically when the start of a prompt matches a recent one. The rule underneath is the same: the cache covers the prompt from the first token up to the first one that changed.
So I order sections by how often they change. Instructions and tool definitions go first and never get edited mid-session. The summary comes next, because it changes only at compaction. History is append-only. The volatile material goes last: records, retrieved passages, the new message.
Two habits break this quietly. A timestamp or request ID in the system prompt turns every call into a cache miss. Retrieved passages placed above the history do the same to everything below them, because they change on every turn.
Tool results stay out of the history
Tool results are the usual overspend. A lookup returns 3,000 tokens of JSON, the model needed four fields, and the whole payload rides along for the rest of the session. Then the tool gets called again and a second copy joins it.
In this design the history gets one line when a tool returns: “fetched account:4471, see records”. As far as the API is concerned, that line is the tool result. The payload goes into a records section keyed by record ID, where a new copy replaces the old one. If records still exceed the cap, the least recently fetched one goes. Its pointer stays in the history, so the model knows the record exists and can fetch it again.
Compaction with a trigger and a test
The default kind of compaction is an accident: a request fails at the limit and somebody adds code to truncate the oldest messages. I’d design it as a step. When history passes its cap, the oldest turns go to a summarizer and history drops to half the cap in one move. Half, because each compaction rewrites the summary and costs one cache miss. Trimming a single turn on every turn costs a miss on every turn.
The summarizer needs a test before it ships. I’d take recorded long sessions, pick facts that appear only in the early turns (a callback number, say, or a credit the agent promised), compact, and ask the assistant about them. That pass rate gets rerun whenever the summarizer prompt or the model changes.
An assembler that logs what it drops
import logging
from dataclasses import dataclass
log = logging.getLogger("context")
@dataclass
class Item:
section: str
text: str
key: str = "" # record id for tool results, e.g. "account:4471"
rank: float = 0.0 # higher survives longer: turn number or relevance score
# section: (order, token cap, eviction rule, cacheable)
BUDGET = {
"prefix": (1, 8_000, "never", True),
"summary": (2, 3_000, "never", True),
"history": (3, 24_000, "to_summary", True),
"records": (4, 16_000, "lowest_rank", False),
"retrieved": (5, 8_000, "lowest_rank", False),
"user": (6, 5_000, "never", False),
}
def count(text):
return len(text) // 4 + 1 # rough; swap in your provider's token counter
def assemble(items, turn):
out, dropped, to_summarize = [], [], []
for name, (_, cap, rule, cacheable) in sorted(BUDGET.items(), key=lambda kv: kv[1][0]):
section = [i for i in items if i.section == name]
if name == "records": # keep only the latest copy of each record
latest = {}
for i in sorted(section, key=lambda i: i.rank):
if i.key in latest:
dropped.append((name, i.key, "superseded", count(latest[i.key].text)))
latest[i.key] = i
section = list(latest.values())
used = sum(count(i.text) for i in section)
if used > cap and rule == "never":
raise ValueError(f"{name} is {used} tokens, cap is {cap}")
target = cap // 2 if rule == "to_summary" else cap
victims = sorted(section, key=lambda i: i.rank) if used > cap else []
while used > target and victims:
v = victims.pop(0)
section.remove(v)
used -= count(v.text)
if rule == "to_summary":
to_summarize.append(v)
dropped.append((name, v.key or v.text[:40], rule, count(v.text)))
out += [(i, cacheable) for i in section]
for name, what, why, tokens in dropped:
log.info("turn=%d dropped section=%s item=%r reason=%s tokens=%d", turn, name, what, why, tokens)
total = sum(count(i.text) for i, _ in out)
stable = sum(count(i.text) for i, c in out if c)
log.info("turn=%d tokens=%d cacheable_share=%.2f", turn, total, stable / max(total, 1))
return [i.text for i, _ in out], to_summarize
It returns the blocks in order, plus the turns that need folding into the summary. Rank is the turn number for history and records, and the relevance score for retrieved passages. I’d keep the drop log even if I threw the rest away. A budget is hard to tune when nobody can see what got cut.
Calloway Telecom at turn 30
Calloway Telecom is fictional and so are its percentages. The failure is the part I’ve tried to keep true to life. Its support assistant holds up for about thirty turns, then starts repeating questions and losing track of what it promised. A token breakdown of the long sessions shows tool results holding 71% of the window, nearly all of it the same account JSON fetched again on most turns. Only 12% of input tokens are cache hits, because a sliding window trims the oldest message every turn and the prefix keeps moving.
The fix is the table. Tool results get a 25% cap and the latest-copy rule, the history keeps pointers, and compaction runs in batches. Sessions now run past turn 120 with the early-turn facts still answerable, and the cached share of input rises to 64%.
When a long session goes wrong now, the drop log is the first thing Calloway reads, and so far it has pointed at a cap every time.