Context Windows & Memory

The Whiteboard in the Model's Head

Ever had a long chat with an AI and noticed it suddenly forgot your name, your project, or that you told it (twice) that you're allergic to cilantro? You've just met the context window.

Think of it as a whiteboard. Everything the model can "see" during a conversation must fit on that board: your messages, its replies, any documents you pasted, plus the system instructions humming in the background. The board is generous but finite, and when it fills up, something has to be erased.

Crucially, the whiteboard is the model's only working memory. An LLM has no notebook, no filing cabinet, no diary. If it's not on the board right now, it does not exist to the model.

Measured in Tokens, Naturally

The context window is a budget measured in tokens (Lesson 4 flashback). And here's the detail people miss: the budget covers input and output combined. Your entire conversation so far AND the reply being generated all share the same board.

CONTEXT WINDOW (the whiteboard)
+------------------------------------------+
| system instructions        (~500 tokens) |
| your first message         (~120 tokens) |
| model reply                (~450 tokens) |
| that PDF you pasted     (~30,000 tokens) |
| ...40 more messages...  (~15,000 tokens) |
| reply being written NOW  (grows as it    |
|                           streams out)   |
+------------------------------------------+
Everything above must fit inside one token budget.

Modern models have context windows ranging from tens of thousands to over a million tokens. Some can hold entire books, even small codebases, on the board at once. Roomy! But every board has edges.

What Happens When the Board Fills Up

When a conversation outgrows the window, the common strategy is brutal and simple: the oldest content gets dropped (except protected system instructions). The chat effectively slides off the top of the whiteboard, oldest lines first.

Here's the eerie part: the model doesn't notice. It doesn't experience forgetting, because from its perspective the erased content never existed. It just answers based on whatever's currently on the board. That's why long conversations produce that distinctive glitch: the AI cheerfully re-asks your name, re-suggests the idea you rejected an hour ago, or suddenly drops your carefully specified requirements.

Wait, Doesn't My Chat App Remember Me?

Some chat products do seem to recall things across sessions: your preferences, past topics. That's not the model's context window growing infinite. It's the app keeping notes outside the model and quietly pasting relevant bits back onto the whiteboard each session. Clever plumbing, same finite board. The distinction matters: the model itself remains goldfish-brained by design; any long-term "memory" is an engineering feature bolted on around it.

Working With the Window, Not Against It

A few habits that make the whiteboard work for you:

• Start fresh for new topics. A new chat is a clean board: old clutter can't confuse anything, and you're not paying (in tokens or attention) for irrelevant history.
• Re-state what matters. In marathon conversations, periodically summarize key decisions: "Quick recap: we chose option B, the budget is fixed, the tone is playful." You just rewrote the essentials in fresh ink at the safe end of the board.
• Paste selectively. Dumping a 300-page document to ask about one chapter floods the board. Bring the pages that matter.
• Front-load instructions... then repeat them. In very long chats, requirements from an hour ago may be near the erasure zone. Important constraints deserve an encore.

How you ask for long-document help matters too:

Whiteboard Quiz