Memory vs. context window vs. RAG

“Just give it memory” is one of the most common requests in AI product work right now, and one of the least specific. In practice, teams reach for one of three quite different tools — a bigger context window, retrieval-augmented generation, or an actual persistent memory layer — and they aren’t interchangeable. Picking the wrong one doesn’t fail loudly. It just quietly under-delivers.

A bigger context window delays forgetting. It doesn’t prevent it.

The context window is how much text a model can see in one call. Making it bigger means a conversation can run longer before anything falls off the end. That’s genuinely useful — but it’s still scoped to one conversation. Close it, start a new one, and the window is empty again, no matter how large it was. A context window is short-term memory with a longer fuse, not memory across sessions.

RAG answers “what’s in this document,” not “what does this student know.”

Retrieval-augmented generation is built to solve a different problem: a model needs facts it wasn’t trained on, or that live in documents too large to paste into a prompt. So you embed the documents, retrieve the relevant chunks at query time, and hand them to the model.

This works well when the missing information is written down somewhere — a manual, a knowledge base, a set of policies. It doesn’t help when the missing information is about a person’s history with your product, because that was never a document to begin with. Retrieving “chapter 4 of the textbook” tells the model nothing about whether this particular student has understood chapter 4 yet.

Memory is a record that outlives the conversation

A real memory layer is neither of the above. It’s a structured, per-user record that’s written to after an interaction and read back before the next one — independent of any single conversation’s length, and not scoped to a single document collection. It answers a different question entirely: not “what does the model currently see” or “what’s in our documents,” but “what do we already know about this specific person.”

That’s also why it’s usually the smallest of the three to implement well and the easiest to get wrong. It doesn’t need a bigger prompt or a bigger vector index — it needs a clear, minimal record of what actually matters (what a student struggles with, what’s worked, what’s changed) and a fast way to read it back. Bolting it onto RAG infrastructure built for documents, or trying to fake it with an ever-growing context window, tends to produce something that’s expensive and still doesn’t quite remember the right things.

The three aren’t competing solutions to the same problem. They’re solutions to three different ones — and most AI tutoring products that “forget” are missing the third, not the first two.