I’ve done this more than once: pasted a long report into ChatGPT, gone back and forth on it for twenty minutes, then asked a question about something I’d mentioned right at the start — only to get an answer that made it obvious the model had no idea what I was talking about. And, like most people, my first instinct was to type back something along the lines of “I literally told you that ten minutes ago.”
It’s tempting to picture this as an AI having a “senior moment,” misplacing a fact the way you’d misplace your keys. That’s not actually what’s going on. What’s happening underneath is a genuinely fascinating hardware problem, and once you understand it, ChatGPT’s “forgetfulness” stops looking like a bug and starts looking like a fairly reasonable trade-off.
Language models have a hard physical limit on how much text they can actively work with at any one moment. It’s called the context window. Modern models can handle a lot more of it than they used to, but making that window bigger isn’t a free upgrade you tick a box for — every extra page you feed in costs real memory, real computing power, and real electricity on hardware that’s eye-wateringly expensive. This isn’t an oversight. It’s physics, hardware, and economics all showing up to the same meeting at once.
Two Very Different Kinds of “Remembering”
When people say an AI “remembers” or “forgets” something, they’re usually blurring together two completely different mechanisms.
The first is trained knowledge — the deep, permanent patterns the model absorbed while being trained on enormous amounts of text, before you ever typed a word to it. Think of this as a researcher’s actual education: years of study baked into how they think, not something they’re actively flicking through.
The second is conversational context — the temporary working notes that exist only for the length of your current chat. This is closer to the desk that researcher is sitting at right now. The whole university library is the model’s trained knowledge, vast and permanent. The desk, though, only has room for a handful of open books at once — no matter how enormous the library behind it is. Once something falls off the edge of that desk, it’s gone from the current conversation, even though the library itself hasn’t changed at all.
What’s Actually Sitting on the Desk: Tokens
Before the model reads anything you type, it chops your text into small chunks called tokens — a token might be a short whole word, a fragment of a longer one, or even a bit of punctuation. “Forgetfulness,” for instance, might get split into something like “forget” and “fullness.”
To put that in perspective: a short email is roughly 150 tokens. A meaty blog post is somewhere around 1,500. A short novel runs to about 100,000 tokens, and a full technical textbook series can push past a million. The context window is simply the hard ceiling on how many of these tokens the model can hold at once. Some newer models advertise windows in the hundreds of thousands, even millions, of tokens — but being able to accept a book-length document and reading every part of it with equal care are two genuinely different achievements, which is really the whole point of this article.
The Real Bottleneck: What’s Called the KV Cache
Powerful language models don’t run on the kind of chip sitting in your laptop. They run on specialised GPUs — graphics processing units — packed with a type of ultra-fast memory called HBM, sitting right next to the chip’s processing engine, like a workbench built into the machine itself.
The moment you start a long conversation, a hefty chunk of that fast memory gets claimed by something called the KV cache (short for “key-value”).
Here’s why that’s needed. Modern models use an architecture called the Transformer, built around a mechanism called attention, which lets the model look back at earlier parts of the conversation to work out how the word it’s currently generating relates to everything that came before. Take a sentence like: “Sarah placed the silver keys inside the blue wooden drawer. Twenty paragraphs later, she opened it. Where were the keys?” To answer that correctly, the model has to link “it” all the way back to “the blue wooden drawer,” several paragraphs earlier.
Rather than redoing that connective work from scratch for every single new word it writes, the model calculates these relationships once and stores the result in the KV cache. Clever — but the cache grows with the conversation. Roughly speaking, the longer your chat gets, the more layers the model has, and the more detail each layer tracks, the larger that cache balloons, and it eats GPU memory fast.
Why Not Just Bolt On More Memory Chips?
If memory’s the bottleneck, the obvious fix looks simple: cram ten times more memory chips onto every server. Engineers are doing some of that. But three walls get in the way.
Heat is one — fast memory chips run hot, and packing more of them onto a board makes them genuinely difficult to cool. Economics is another — a data centre isn’t running your conversation in isolation, it’s juggling hundreds of thousands of people at once, and if your one long chat is hogging 16 gigabytes of GPU memory, that’s capacity taken away from everyone else. And there’s a bandwidth problem too — physically moving data between memory chips and the processor takes time, so the more memory the system has to sift through, the slower your reply gets.
The Quadratic Trap: Why Long Conversations Get Expensive Fast
Memory is only half the story — the maths involved is the other half, and this is where things get properly steep.
In a standard Transformer, full attention means every token has to compare itself against every other token in the conversation. Double the length of the text, and you don’t double the work — you roughly quadruple it. Picture ten people in a room, each swapping notes with everyone else: that’s 45 conversations, easy enough. Now put ten thousand people in a stadium and ask for the same thing. The number of exchanges doesn’t just grow — it explodes into the millions. That’s essentially what a GPU is doing internally the moment you feed it an entire technical manual instead of a paragraph, which is exactly why response times can noticeably slow down on very long chats.
“Lost in the Middle”: Big Context Isn’t the Same as Perfect Memory
Here’s the part that surprised researchers. Even when a model’s context window is technically large enough to hold an entire book, it doesn’t pay equal attention to every page of it.
Studies on long-context models found something researchers now call the “lost in the middle” effect: information right at the start of a prompt tends to get remembered well — it’s treated a bit like foundational instructions. Information right at the end also tends to stick, since it’s the freshest and most recent. But detail buried somewhere in the middle of a long block of text is far more likely to get dropped, blurred, or simply overlooked.
It’s a bit like handing someone a five-hundred-page textbook five minutes before an exam. The book being physically on the desk doesn’t mean they’ve absorbed page 243 with the same care as the first page or the last.
How Systems Actually Cope With This
To stop everything grinding to a halt under the weight of very long conversations, a few practical strategies get used, often together. Some systems simply drop the oldest messages once the token limit is hit — fine for a short, casual back-and-forth, less fine if you needed that early detail later. Others use rolling summarisation, quietly condensing earlier parts of the conversation down into a short summary as you go, which works well for long brainstorming sessions where the gist matters more than the exact wording.
Then there’s the more elegant option: retrieval-augmented generation, or RAG.
RAG: Searching the Library Instead of Memorising It
Rather than forcing a model to hold an entire library in its active working memory at once, RAG flips the approach — it searches the library first, and only brings back what’s actually relevant.
Say you ask a company assistant, “what was our policy on remote working in 2023?” A system built this way doesn’t load fifty thousand internal documents into the model’s context. Instead, a search step scans the document store in milliseconds, finds the two paragraphs that actually answer the question, and hands just those to the model. The answer comes back accurate, using a tiny fraction of the memory a brute-force approach would need. It’s the difference between memorising an entire library and knowing exactly which shelf to walk to.
So, Is ChatGPT Actually “Forgetful”?
Not in the way it feels. When it seems to lose track of something you said earlier, that detail has usually fallen out of the active context window, been quietly compressed away in a rolling summary, missed by a retrieval step that didn’t flag it as relevant, or simply buried in the middle of a long stretch of text where attention runs thin.
Underneath a perfectly ordinary-looking chat box sits one of the trickier engineering problems in modern computing — every extra bit of text costs real memory chips, real power, real cooling, and a genuinely large amount of maths. An AI that could instantly read hundreds of books and recall every line forever isn’t just a software feature someone hasn’t got round to yet; it’s a challenge bound by hardware, energy grids, and the physical limits of silicon.
Which, oddly, means the more useful goal probably isn’t giving these systems infinite memory at all. It’s teaching them the very human skill of remembering the right thing at the right time — which, if I’m honest, is a skill I could stand to work on myself, laptop or not.