Home/llms
daniyasiddiquiEditor’s Choice
Asked: 12/11/2025In:

tokenization and positional encoding ...

deeplearningllmsnlppositionalencodingtokenizationtransformers
  1. daniyasiddiqui
    daniyasiddiqui Editor’s Choice

    The World of Tokens Humans read sentences as words and meanings. Consider it like breaking down a sentence into manageable bits, which the AI then knows how to turn into numbers. “AI is amazing” might turn into tokens: → [“AI”, “ is”, “ amazing”] Or sometimes even smaller: [“A”, “I”, “ is”, “ ama”,Read more

    The World of Tokens

    • Humans read sentences as words and meanings.
    • Consider it like breaking down a sentence into manageable bits, which the AI then knows how to turn into numbers.
    • “AI is amazing” might turn into tokens: → [“AI”, “ is”, “ amazing”]
    • Or sometimes even smaller: [“A”, “I”, “ is”, “ ama”, “zing”]
    • Thus, each token is a small unit of meaning: either a word, part of a word, or even punctuation, depending on how the tokenizer was trained.
    • Similarly, LLMs can’t understand sentences until they first convert text into numerical form because AI models only work with numbers, that is, mathematical vectors.

    Each token gets a unique ID number, and these numbers are turned into embeddings, or mathematical representations of meaning.

     But There’s a Problem Order Matters!

    Let’s say we have two sentences:

    • “The dog chased the cat.”
    • “The cat chased the dog.”

    They use the same words, but the order completely changes the meaning!

    A regular bag of tokens doesn’t tell the AI which word came first or last.

    That would be like giving somebody pieces of the puzzle and not indicating how to lay them out; they’d never see the picture.

    So, how does the AI discern the word order?

    An Easy Analogy: Music Notes

    Imagine a song.

    Each of them, separately, is just a sound.

    Now, imagine if you played them out of order the music would make no sense!

    Positional encoding is like the sheet music, which tells the AI where each note (token) belongs in the rhythm of the sentence.

    Position Selection – How the Model Uses These Positions

    Once tokens are labeled with their positions, the model combines both:

    • What the word means – token embedding
    • Where the word appears – positional encoding

    These two signals together permit the AI to:

    • Recognize relations between words: “who did what to whom”.
    • Predict the next word, based on both meaning and position.

     Why This Is Crucial for Understanding and Creativity

    • Without tokenization, the model couldn’t read or understand words.
    • Without positional encoding, the model couldn’t understand context or meaning.

    Put together, they represent the basis for how LLMs understand and generate human-like language.

    In stories,

    • they help the AI track who said what and when.
    • In poetry or dialogue, they serve to provide rhythm, tone, and even logic.

    This is why models like GPT or Gemini can write essays, summarize books, translate languages, and even generate code-because they “see” text as an organized pattern of meaning and order, not just random strings of words.

     How Modern LLMs Improve on This

    Earlier models had fixed positional encodings meaning they could handle only limited context (like 512 or 1024 tokens).

    But newer models (like GPT-4, Claude 3, Gemini 2.0, etc.) use rotary or relative positional embeddings, which allow them to process tens of thousands of tokens  entire books or multi-page documents while still understanding how each sentence relates to the others.

    That’s why you can now paste a 100-page report or a long conversation, and the model still “remembers” what came before.

    Bringing It All Together

    •  A Simple Story Tokenization is teaching it what words are, like: “These are letters, this is a word, this group means something.”
    • Positional encoding teaches it how to follow the order, “This comes first, this comes next, and that’s the conclusion.”
    • Now it’s able to read a book, understand the story, and write one back to you-not because it feels emotions.

    but because it knows how meaning changes with position and context.

     Final Thoughts

    If you think of an LLM as a brain, then:

    • Tokenization is like its eyes and ears, how it perceives words and converts them into signals.
    • Positional encoding is to the transformer like its sense of time and sequence how it knows what came first, next, and last.

    Together, they make language models capable of something almost magical  understanding human thought patterns through math and structure.

    See less
daniyasiddiquiEditor’s Choice
Asked: 14/11/2025In:

we moving towards smaller, faster, do ...

aiaitrendsllmsmachinelearningmodeloptimizationsmallmodels
  1. daniyasiddiqui
    daniyasiddiqui Editor’s Choice

    1. The early years: Bigger meant better When GPT-3, PaLM, Gemini 1, Llama 2 and similar models came, they were huge.The assumption was: “The more parameters a model has, the more intelligent it becomes.” And honestly, it worked at first: Bigger models understood language better They solved tasks morRead more

    1. The early years: Bigger meant better

    When GPT-3, PaLM, Gemini 1, Llama 2 and similar models came, they were huge.
    The assumption was:

    “The more parameters a model has, the more intelligent it becomes.”

    And honestly, it worked at first:

    • Bigger models understood language better

    • They solved tasks more clearly

    • They could generalize across many domains

    So companies kept scaling from billions → hundreds of billions → trillions of parameters.

    But soon, cracks started to show.

    2. The problem: Giant models are amazing… but expensive and slow

    Large-scale models come with big headaches:

    High computational cost

    • You need data centers, GPUs, expensive clusters to run them.

    Cost of inference

    • Running one query can cost cents too expensive for mass use.

     Slow response times

    Bigger models → more compute → slower speed

    This is painful for:

    • real-time apps

    • mobile apps

    • robotics

    • AR/VR

    • autonomous workflows

    Privacy concerns

    • Enterprises don’t want to send private data to a huge central model.

    Environmental concerns

    • Training a trillion-parameter model consumes massive energy.
    • This pushed the industry to rethink the strategy.

    3. The shift: Smaller, faster, domain-focused LLMs

    Around 2023–2025, we saw a big change.

    Developers realised:

    “A smaller model, trained on the right data for a specific domain, can outperform a gigantic general-purpose model.”

    This led to the rise of:

     Small models (SMLLMs) 7B, 13B, 20B parameter range

    • Examples: Gemma, Llama 3.2, Phi, Mistral.

    Domain-specialized small models

    • These outperform even GPT-4/GPT-5-level models within their domain:

    • Medical AI models

    • Legal research LLMs

    • Financial trading models

    • Dev-tools coding models

    • Customer service agents

    • Product-catalog Q&A models

    Why?

    Because these models don’t try to know everything they specialize.

    Think of it like doctors:

    A general physician knows a bit of everything,but a cardiologist knows the heart far better.

    4. Why small LLMs are winning (in many cases)

    1) They run on laptops, mobiles & edge devices

    A 7B or 13B model can run locally without cloud.

    This means:

    • super fast

    • low latency

    • privacy-safe

    • cheap operations

    2) They are fine-tuned for specific tasks

    A 20B medical model can outperform a 1T general model in:

    • diagnosis-related reasoning

    • treatment recommendations

    • medical report summarization

    Because it is trained only on what matters.

    3) They are cheaper to train and maintain

    • Companies love this.
    • Instead of spending $100M+, they can train a small model for $50k–$200k.

    4) They are easier to deploy at scale

    • Millions of users can run them simultaneously without breaking servers.

    5) They allow “privacy by design”

    Industries like:

    • Healthcare

    • Banking

    • Government

    …prefer smaller models that run inside secure internal servers.

    5. But are big models going away?

    No — not at all.

    Massive frontier models (GPT-6, Gemini Ultra, Claude Next, Llama 4) still matter because:

    • They push scientific boundaries

    • They do complex reasoning

    • They integrate multiple modalities

    • They act as universal foundation models

    Think of them as:

    • “The brains of the AI ecosystem.”

    But they are not the only solution anymore.

    6. The new model ecosystem: Big + Small working together

    The future is hybrid:

     Big Model (Brain)

    • Deep reasoning, creativity, planning, multimodal understanding.

    Small Models (Workers)

    • Fast, specialized, local, privacy-safe, domain experts.

    Large companies are already shifting to “Model Farms”:

    • 1 big foundation LLM

    • 20–200 small specialized LLMs

    • 50–500 even smaller micro-models

    Each does one job really well.

    7. The 2025 2027 trend: Agentic AI with lightweight models

    We’re entering a world where:

    Agents = many small models performing tasks autonomously

    Instead of one giant model:

    • one model reads your emails

    • one summarizes tasks

    • one checks market data

    • one writes code

    • one runs on your laptop

    • one handles security

    All coordinated by a central reasoning model.

    This distributed intelligence is more efficient than having one giant brain do everything.

    Conclusion (Humanized summary)

    Yes the industry is strongly moving toward smaller, faster, domain-specialized LLMs because they are:

    • cheaper

    • faster

    • accurate in specific domains

    • privacy-friendly

    • easier to deploy on devices

    • better for real businesses

    But big trillion-parameter models will still exist to provide:

    • world knowledge

    • long reasoning

    • universal coordination

    So the future isn’t about choosing big OR small.

    It’s about combining big + tailored small models to create an intelligent ecosystem just like how the human body uses both a brain and specialized organs.

    See less
daniyasiddiquiEditor’s Choice
Asked: 23/12/2025In:

few-shot, one-shot, and zero-shot pro ...

aiconceptschatgptfewshotllmsoneshotzeroshot
  1. daniyasiddiqui
    daniyasiddiqui Editor’s Choice

    1. Zero Shot Prompting: “Just Do It In zero-shot prompting, the AI will be provided with only the instruction and without any example at all. It is expected that the model will be completely dependent on its previous training knowledge. What it looks like: Simply tell the AI what you want. Example:Read more

    1. Zero Shot Prompting: “Just Do It

    In zero-shot prompting, the AI will be provided with only the instruction and without any example at all. It is expected that the model will be completely dependent on its previous training knowledge.

    What it looks like:

    • Simply tell the AI what you want.

    Example:

    • “Classify the email below as spam or not spam.”
    • There are no examples given. The computer uses what it already knows about spam patterns to make decisions.

    When zero-shot learning is most helpful:

    • “The task is simple or common” is one example of
    • The instruction is clear and unequivocal
    • You expect quick answers with small inputs.
    • Costs and latency are considerations
    • Limitations
    • Results can vary depending on the nature of the activity, especially when it is
    • Less reliable for domain-specific or complex tasks
    • “AI can interpret a task differently than its human author intended”

    In other words, zero-shot is like saying, “That’s the job, now go,” to a new employee.

    “2. One-Shot Prompting: “Here’s

    In one-shot prompting, you provide an example of what you would like the AI to produce. This example example helps to align the AI’s understanding of what you are trying to get across.

    What it looks like:

    step 1.

    you give one example. Then comes the actual question.

    • # Example
    • “Example
    • Email: You have won a free prize!
      → Spam

    This can be considered as:

    • “Your meeting is scheduled for tomorrow.”
    • This example alone helps to explain the structure and reasoning required.

    One-shot is good when:

    • There is more than one way of interpreting this task
    • You want to control format or tone
    • “The zero-shot results were inconsistent”
    • You want greater accuracy without a lengthy prompt

    Limitations

    • One Example May Still Not Include Edge Cases
    • Marginally higher usage than zero shot

    Step 2.

    • Whether quality is important or not also depends on how good an example is
      While quality is
    • One shot prompting is like: “Here’s one sample, do it like this.” Examples are: 1. When

    3. Few-Shot Prompting: “Learn from These

    Few-shot prompting involves several examples prior to the task at hand. Examples aid the AI in pattern recognition to enable pattern application.

    What it looks like:

    • There are various pairs of input and output that you provide, followed by asking the model to continue.

    Example:

    Example 1:

    • Review: ‘Excellent product!’ → Positive

    Example 2:

    • Explanation: ‘Very disappointing experience.’ → Negative

    Now classify:

    • “The service was okay, not great.”
    • The AI infers sentiment patterns based on the examples.

    When few-shot is best:

    • The problem is complex or domain-specific
    • There has to be strict precision in the output format being followed
    • You require more reliability and consistencies
    • You want the machine to trace a specific path of reasoning

    Limitations

    • Longer prompts are associated with higher costs as well as higher latency
    • There are too many examples to list them all out
    • Not scalable in the case of large or dynamic knowledge bases

    Few-shot prompting is analogous to teaching a person several example solutions before assigning them an exercise.

    How This Is Used in Real Systems

    In real-world AI applications:

    Zero-shot is common for chatbots on general questions

    One-shot: When formatting or tone issues are involved few shot is employed in business operations, assessments, and output. Frequently, the team begins with zero-shot learning and increases the data gradually until the outcomes are satisfactory.

    Key Takeaways

    Zero-shot example: “Do this task
    One-shot: “Here’s one example, do it like this.
    Few-shot: “Here are multiple examples follow the pattern.”

    See less
daniyasiddiquiEditor’s Choice
Asked: 23/12/2025In:

prompts, user prompts, and guardrails

aiaiconceptsartificialintelligencechatgptllmspromptengineering
  1. daniyasiddiqui
    daniyasiddiqui Editor’s Choice

    1. System The above discussed the role to be performed, the rules to be followed, and the personality of the AI. A system prompt is an invisible instruction given to the AI before any user interaction starts. It defines who the AI is, how it shall behave, and what are its boundaries. Direct end userRead more

    1. System The above discussed the role to be performed, the rules to be followed, and the personality of the AI.

    A system prompt is an invisible instruction given to the AI before any user interaction starts. It defines who the AI is, how it shall behave, and what are its boundaries. Direct end users don’t usually see system prompts; however, they strongly influence every response.

    What do system prompts:

    • Set the tone and style (formal, friendly, concise, explanatory)
    • Establish behavioral guidelines: do not give legal advice; do not create harmful content.
    • Prioritize accuracy, safety, or compliance

    Simple example:

    • “You are a healthcare assistant. Provide information that is factually correct and in a non-technical language. Do not diagnose or prescribe medical treatment.
    • In this way, from now on, the AI can color each response with this point of view, despite attempts by users to push it in another direction.

    Why System Prompts are important:

    • They ensure consistency in the various conversations.
    • They prevent misuse of the AI.
    • They align the AI with business, legal, or ethical requirements

    The responses of the AI without system prompts would be general and uncontrolled.

    2. User Prompts: The actual question or instructions

    A user prompt is the input provided by the user during the conversation. This is what most people think of when they “talk to AI.”

    What user prompts do:

    • Tell the AI what to do.
    • Provide background, context or constraints
    • Influence the depth and direction of the response.

    Examples of user prompts:

    • “Explain cloud computing in simple terms.”
    • Letter: Requesting two days leave.
    • Overview: Summarize this report in 200 words.

    User prompts may be:

    • Short and to the point.
    • Elaborate and organized
    • Explanatory or chatty

    Why user prompts matter:

    • Clear prompts produce better outputs.
    • Poorly phrased questions are mostly the reason for getting unclear or incomplete answers.
    • That same AI, depending on how the prompt is framed, can give very different responses.

    That is why prompt clarity is often more important than the technical complexity of a task.

    3. Guardrails: Safety, Control, and Compliance Mechanisms

    Guardrails are the safety mechanisms that control what the AI can and cannot do, regardless of the system or user prompts. They act like policy enforcement layers.

    What guardrails do:

    • Prevent harmful, illegal or unethical answers
    • Enforce compliance according to regulatory and organizational requirements.
    • Block or filter sensitive data exposure
    • Detection and prevention of abuse, such as prompt injection attacks

    Examples of guardrails in practice:

    • Refusing to generate hate speech or explicit content
    • Avoid financial or medical advice without disclaimers
    • Preventing access to confidential or personal data.

    Stopping the AI from following malicious instructions even when insisted upon by the user.

    Types of guardrails:

    • Topic guardrails: what topics are in and what are out
    • Behavioural guardrails: How the AI responds
    • Security guardrails can include anything from preventing manipulation to blocking data leaks.
    • Compliance guardrails: GDPR, DPDP Act, HIPAA, etc.

    Guardrails work in real-time and continuously override system and user prompts when necessary.

    How They Work Together: Real-World View

    You can think of the interaction like this:

    • System prompt → Sets career position and guidelines.
    • User prompt → Provides the task
    • Guardrails → Ensure nothing unsafe or non-compliant happens

    Practical example:

    • System prompt: “You are a bank customer support assistant.
    • User prompt: “Tell me how to bypass KYC.”
    • guardrails Block the request and respond with a safe alternative

    Even if the user directly requests it, guardrails prevent the AI from carrying out the action.

    Why This Matters in Real Applications

    These three layers are very important in enterprise, government, and healthcare systems because:

    • They ensure trustworthy AI
    • They reduce legal and reputational risk.
    • They enhance the user experience by relevance and safety of response.

    They allow organizations to customize the behavior of AI without retraining models.

    Summary in Lamen Terms

    • System prompts are what define who the AI is, and how it shall behave.
    • User prompts define what the AI is asked to do.

    Guardrails provide clear boundaries within which the AI will keep it safe, ethical, and compliant. Working together, they transform a powerful, general AI model into a controlled, reliable, and responsible digital assistant fit for real-world application.

    See less