hallucinations in legal or medical co ...
What LLMs Actually Do At their core, LLMs like GPT-4, GPT-4o, Claude, or Gemini are predictive models. They are shown a sample input prompt and generate what is most likely to come next based on what they learned from their training corpus. They've read billions of words' worth of books, websites, cRead more
What LLMs Actually Do
At their core, LLMs like GPT-4, GPT-4o, Claude, or Gemini are predictive models. They are shown a sample input prompt and generate what is most likely to come next based on what they learned from their training corpus. They’ve read billions of words’ worth of books, websites, codebases, etc., and learned the patterns in language, the logic, and even a little bit of world knowledge.
So yes, basically, they are pattern matchers. It’s not a bad thing. The depth of patterns that they’ve been taught is impressive. They can:
- Solve logic puzzles
- Do chain-of-thought mathematics
- Generate functional code
- Abstract dense legal text
- Argue both sides of a debate
- Even fake emotional tone convincingly
- But is this really “reasoning,” or just very good imitation?
Where They Seem to Reason
If you give an LLM a multi-step problem—like doing math on a word problem or fixing some code—it generally gets it correct. Not only that, it generally describes its process in a logical manner, even invoking formal logic or rule citations
This is very similar to reasoning. And some AI researchers contend:
If an AI system produces useful, reliable output through logic-like operations, whether it “feels” reasoning from the inside out is it even an issue?
- To many, the bottom line is behavior.
- But There Are Limits
- Even though they’re so talented, LLMs:
Have trouble being consistent – They may contradict themselves in lengthy responses.
Can hallucinate – Fabricating facts or logic that “sounds” plausible but isn’t there.
Lack genuine understanding – They lack a world model or internal self-model.
Don’t know when they don’t know – They can convincingly offer drivel.
So while they can fake reasoning pretty convincingly, they have a tendency to get it wrong in subtle but important ways that an actual reasoning system probably wouldn’t.
Middle Ground Emerges
The most advanced reply could be:
- LLMs are not human-like reasoning, but they’re generating emergent reason-like behavior.
Which is to say that:
- The system was never explicitly trained to reason.
- But due to scale and training, reason-like behaviors emerge.
- It’s not mere memorization—it’s abstraction and generalization.
For example:
GPT-4o can reason through new logic puzzles it has never seen before.
By applying means like chain-of-thought prompting or tool use, LLMs can break down issues and tap into external systems of reasoning to extend their own abilities.
Humanizing the Answer
Imagine you’re talking to a very smart parrot that has read every book written and is able to communicate in your language. At first, it seems like they’re just imitating voice. Then the parrot starts to reason, give advice, abstract papers, and even help you debug your program.
Eventually, you’d no longer be asking yourself “Is this mimicry?” but “How far can we go?”
That’s where we are with LLMs. They don’t think the way we do. They don’t feel their way through the world. But their ability to deliver rational outcomes is real enough to be useful—and, too often, better than what an awful lot of humans can muster under pressure.
Final Thought So,
- are LLMs just pattern matchers?
- Yes. But maybe that’s all reasoning has ever been.
If reasoning is something which you are able to do once you’ve seen enough patterns and learned how to use them in a helpful manner. well, maybe LLMs have cracked the surface of it.
We’re not witnessing artificial consciousness—but we’re witnessing artificial cognition. And that’s important.
See less
So, First, What Is an "AI Hallucination"? With artificial intelligence, an "hallucination" is when a model confidently generates information that's false, fabricated, or deceptive, yet sounds entirely reasonable. For example: In the law, the model might cite a bogus court decision. In medicine, it mRead more
So, First, What Is an “AI Hallucination”?
With artificial intelligence, an “hallucination” is when a model confidently generates information that’s false, fabricated, or deceptive, yet sounds entirely reasonable.
For example:
These aren’t typos. These are errors of factual truth, and when it comes to life and liberty, they’re unacceptable.
Why Do LLMs Hallucinate?
LLMs aren’t databases—They don’t “know” things like us.
They generate text by predicting what comes next, based on patterns in the data they’ve been trained on.
So when you ask:
“What are the key points from Smith v. Johnson, 2011?”
If no such case exists, the LLM can:
Create a spurious summary
Make up quotes
Even generate a fake citation
Since it’s not cheating—it’s filling in the blanks based on best guess based on patterns.
In Legal Contexts: The Hazard of Authoritative Ridiculousness
Attorneys rely on precedent, statutes, and accurate citations. But LLMs can:
Make up fictional cases (already occurs in real courtrooms, actually!)
Misquote real legal text
Get jurisdictions confused (e.g., confusing US federal and UK law)
Apply laws out of context
Actual-Life Scenario:
In 2023, a New York attorney employed ChatGPT to write a brief. The AI drew on a set of fake court cases. The judge discovered—and penalized the attorney. It was an international headline and a warning story.
Why did it occur?
In Medical Settings: Even Greater Risks
Think of a model that recommends a drug interaction between two drugs that does not occur—or worse, not recommending one that does. That’s terrible, but more terrible, it’s unsafe.
And Yet.
LLMs can perform some medical tasks:
Abstracting patient records
De-jargonizing jargonese
Generating clinical reports
Helping medical students learn
But these are not decision-making roles.
How Are We Tackling Hallucinations in These Fields?
This is how researchers, developers, and professionals are pushing back:
Human-in-the-loop
Retrieval-Augmented Generation (RAG)
Example: An AI lawyer program using actual Westlaw or LexisNexis material.
Model Fine-Tuning
Prompt Engineering & Chain-of-Thought
Confirmation Layers
Anchoring the Effect
Come on: It is easy to take the word of the AI when it talks as if it has years of experience. Particularly when it saves time, reduces expense, and appears to “know it all.”
That certainty is a double-edged sword.
Think:
So, Where Does That Leave Us?
That is:
Closing Thought
LLMs can do some very impressive things. But not in medicine and law. “Impressive” just isn’t sufficient there.
And they must be demonstrable, safe, andatable as well.
Meanwhile, consider AI to be a very good intern—smart, speedy, and never fatigued…
See lessBut not one you’d have perform surgery on you or present a case before a judge without your close guidance.