The World of Tokens

• Humans read sentences as words and meanings.
• Consider it like breaking down a sentence into manageable bits, which the AI then knows how to turn into numbers.
• “AI is amazing” might turn into tokens: → [“AI”, “ is”, “ amazing”]
• Or sometimes even smaller: [“A”, “I”, “ is”, “ ama”, “zing”]
• Thus, each token is a small unit of meaning: either a word, part of a word, or even punctuation, depending on how the tokenizer was trained.
• Similarly, LLMs can’t understand sentences until they first convert text into numerical form because AI models only work with numbers, that is, mathematical vectors.

Each token gets a unique ID number, and these numbers are turned into embeddings, or mathematical representations of meaning.

 But There’s a Problem Order Matters!

Let’s say we have two sentences:

• “The dog chased the cat.”
• “The cat chased the dog.”

They use the same words, but the order completely changes the meaning!

A regular bag of tokens doesn’t tell the AI which word came first or last.

That would be like giving somebody pieces of the puzzle and not indicating how to lay them out; they’d never see the picture.

So, how does the AI discern the word order?

An Easy Analogy: Music Notes

Imagine a song.

Each of them, separately, is just a sound.

Now, imagine if you played them out of order the music would make no sense!

Positional encoding is like the sheet music, which tells the AI where each note (token) belongs in the rhythm of the sentence.

Position Selection – How the Model Uses These Positions

Once tokens are labeled with their positions, the model combines both:

• What the word means – token embedding
• Where the word appears – positional encoding

These two signals together permit the AI to:

• Recognize relations between words: “who did what to whom”.
• Predict the next word, based on both meaning and position.

 Why This Is Crucial for Understanding and Creativity

• Without tokenization, the model couldn’t read or understand words.
• Without positional encoding, the model couldn’t understand context or meaning.

Put together, they represent the basis for how LLMs understand and generate human-like language.

In stories,

• they help the AI track who said what and when.
• In poetry or dialogue, they serve to provide rhythm, tone, and even logic.

This is why models like GPT or Gemini can write essays, summarize books, translate languages, and even generate code-because they “see” text as an organized pattern of meaning and order, not just random strings of words.

 How Modern LLMs Improve on This

Earlier models had fixed positional encodings meaning they could handle only limited context (like 512 or 1024 tokens).

But newer models (like GPT-4, Claude 3, Gemini 2.0, etc.) use rotary or relative positional embeddings, which allow them to process tens of thousands of tokens  entire books or multi-page documents while still understanding how each sentence relates to the others.

That’s why you can now paste a 100-page report or a long conversation, and the model still “remembers” what came before.

Bringing It All Together

•  A Simple Story Tokenization is teaching it what words are, like: “These are letters, this is a word, this group means something.”
• Positional encoding teaches it how to follow the order, “This comes first, this comes next, and that’s the conclusion.”
• Now it’s able to read a book, understand the story, and write one back to you-not because it feels emotions.

but because it knows how meaning changes with position and context.

 Final Thoughts

If you think of an LLM as a brain, then:

• Tokenization is like its eyes and ears, how it perceives words and converts them into signals.
• Positional encoding is to the transformer like its sense of time and sequence how it knows what came first, next, and last.

Together, they make language models capable of something almost magical  understanding human thought patterns through math and structure.