LLMs struggle with long-term memory
1. The Big Idea Behind the Transformer Instead of reading a sentence word-by-word as in an RNN, the Transformer reads the whole sentence in parallel. This alone dramatically speeds up training. But then the natural question would be: How does the model know which words relate to each other if it isRead more
1. The Big Idea Behind the Transformer
Instead of reading a sentence word-by-word as in an RNN, the Transformer reads the whole sentence in parallel. This alone dramatically speeds up training.
But then the natural question would be:
- How does the model know which words relate to each other if it is seeing everything at once?
- This is where self-attention kicks in.
- Self-attention allows the model to dynamically calculate the importance scores of other words in the sequence. For instance, in the sentence:
“The cat which you saw yesterday was sleeping.”
When predicting something about “cat”, the model can learn to pay stronger attention to “was sleeping” than to “yesterday”, because the relationship is more semantically relevant.
Transformers do this kind of reasoning for each word at each layer.
2. How Self-Attention Actually Works (Human Explanation)
Self-attention sounds complex but the intuition is surprisingly simple:
- Think of each token, which includes words, subwords, or other symbols, as a person sitting at a conference table.
Everybody gets an opportunity to “look around the room” to decide:
- To whom should I listen?
- How much should I care about what they say?
- How do their words influence what I will say next?
Self-attention calculates these “listening strengths” mathematically.
3. The Q, K, V Mechanism (Explained in Human Language)
Each token creates three different vectors:
- Query (Q) – What am I looking for?
- Key (K) – what do I contain that others may search for?
- Value.V- what information will I share if someone pays attention to me?
Analogical is as follows:
- Imagine a team meeting.
- Your Query is what you are trying to comprehend, such as “Who has updates relevant to my task?”
- Everyone’s Key represents whether they have something you should focus on (“I handle task X.”)
- Everyone’s Value is the content (“Here’s my update.”)
- It computes compatibility scores between every Query–Key pair.
- These scores determine how much the Query token attends to each other token.
Finally, it creates a weighted combination of the Values, and that becomes the token’s updated representation.
4. Why This Is So Powerful
Self-attention gives each token a global view of the sequence—not a limited window like RNNs.
This enables the model to:
- Capture long-range dependencies
- Understand context more precisely
- Parallelize training efficiently
- Capture meaning in both directions – bidirectional context
And because multiple attention heads run in parallel (multi-head attention), the model learns different kinds of relationships at once for example:
- syntactic structure
- Semantic Similarity
- positional relationships
- co-reference: linking pronouns to nouns
Each head learns, through which to interpret the input in a different lens.
5. Why Transformers Replaced RNNs and LSTMs
- Performance: They simply have better accuracy on almost all NLP tasks.
- Speed: They train on GPUs really well because of parallelism.
- Scalability: Self-attention scales well as models grow from millions to billions of parameters.
Flexibility Transformers are not limited to text anymore, they also power:
- image models
- Speech models
- video understanding
GPT-4o, Gemini 2.0, Claude 3.x-like multimodal systems
agents, code models, scientific models
Transformers are now the universal backbone of modern AI.
6. A Quick Example to Tie It All Together
Consider the sentence:
- “I poured water into the bottle because it was empty.”
- Humans know that “it” refers to “the bottle,” not the water.
Self-attention allows the model to learn this by assigning a high attention weight between “it” and “bottle,” and a low weight between “it” and “water.”
This dynamic relational understanding is exactly why Transformers can perform reasoning, translation, summarization, and even coding.
Summary-Final (Interview-Friendly Version)
A Transformer is a neural network architecture built entirely around the idea of self-attention, which allows each token in a sequence to weigh the importance of every other token. It processes sequences in parallel, making it faster, more scalable, and more accurate than previous models like RNNs and LSTMs.
Self-attention works by generating Query, Key, and Value vectors for each token, computing relevance scores between every pair of tokens, and producing context-rich representations. This ability to model global relationships is the core reason why Transformers have become the foundation of modern AI, powering everything from language models to multimodal systems.
See less
1. LLMs Don’t Have Real Memory Only a Temporary “Work Scratchpad” LLMs do not store facts the way a human brain does. They have no memory database. They don't update their internal knowledge about a conversation. What they do have is: A context window, such as a temporary whiteboard A transient, sliRead more
1. LLMs Don’t Have Real Memory Only a Temporary “Work Scratchpad”
LLMs do not store facts the way a human brain does.
They have no memory database.
They don’t update their internal knowledge about a conversation.
What they do have is:
Think of the context window as the model’s “short-term memory.”
If the model has a 128k-token context window, that means:
It doesn’t have a mechanism for retrieving past information if that information isn’t re-sent.
This is the first major limitation:
2. Transformers Do Not Memorize; They Simply Process Input
Transformers work by using self-attention, which allows tokens (words) to look at other tokens in the input.
But this mechanism is only applied to tokens that exist right now in the prompt.
There is no representation of “past events,” no file cabinet of previous data, and no timeline memory.
LLMs don’t accumulate experience; they only re-interpret whatever text you give them at the moment.
So even if you told the model:
If that information scrolls outside the context window, the LLM has literally no trace it ever existed.
3. They fail to “index” or “prioritize” even within the context.
A rather less obvious, yet vital point:
Instead, they all rely on attention weights to determine relevance.
But attention is imperfect because:
This is why LLMs sometimes contradict themselves or forget earlier rules within the same conversation.
They don’t have durable memory they only simulate memory through pattern matching across the visible input.
4. Training Time Knowledge is Not Memory
Another misconception is that “the model was trained on information, so it should remember it.”
During the training process, a model won’t actually store facts like a database would.
Instead, it compresses patterns into weights that help it predict words.
Limitations of this training-time “knowledge”:
So even if the model has seen a fact during training, it doesn’t “recall” it like a human it just reproduces patterns that look statistically probable.
This is not memory; it’s pattern extrapolation.
5. LLMs Do Not Have Personal Identity or Continuity
Humans remember because we have continuity of self:
Memory turns into the self.
LLMs, on the other hand:
6. Long-term memory requires storage + retrieval + updating LLMs have none of these
For long-term memory of a system, it has to:
LLMs do none of these things natively.
This is why most companies are pairing LLMs with external memory solutions:
These systems compensate for the LLM’s lack of long-term memory.
7. The Bigger the Model, the Worse the Forgetting
Interestingly, as context windows get longer (e.g., 1M tokens), the struggle increases.
Why?
Because in very long contexts:
So even though the context window grows, the model’s ability to effectively use that long window does not scale linearly.
It is like giving someone a 1,000-page book to read in one sitting and expecting them to memorize every detail they can skim it, but not comprehend all of it with equal depth.
8. A Human Analogy Explains It
Impoverished learner with:
No emotional markers No personal identity Inability to learn from experience That is roughly an LLM’s cognitive profile. Brilliant and sophisticated at the moment but without lived continuity.
Final Summary
Interview Ready LLMs struggle with long-term memory because they have no built-in mechanism for storing and retrieving information over time. They rely entirely on a finite context window, which acts as short-term memory, and anything outside that window is instantly forgotten. Even within the window, memory is not explicit it is approximated through self-attention, which becomes less reliable as sequences grow longer. Training does not give them true memory, only statistical patterns, and they cannot update their knowledge during conversation.
To achieve long-term memory, external architectures like vector stores, RAG, or specialized memory modules must be combined with LLMs.
See less