Why an AI chatbot invents a fact and says it with total confidence

Ask a chatbot for five papers on a niche topic and you may get five clean citations — author, journal, year, volume, page range — of which two were never written. The tone will be identical for the three that exist and the two that do not. There is no hedge, no flicker, no tell. This is the single most confusing thing about large language models for anyone using one seriously, and the explanation is not that the engineers were careless. It is that the machine you are talking to has no mechanism for the thing you assumed it was doing.

The model is predicting the next token, not looking anything up

A large language model does one operation, repeatedly: given the text so far, produce a probability distribution over what comes next, pick one, append it, and repeat.

The unit it works in is not a word but a token — a subword fragment. "Nanotechnology" might arrive as three or four pieces; a common word is usually one. The model converts each token into a long list of numbers, passes those through a transformer — the architecture introduced in the 2017 paper Attention Is All You Need, whose central trick is letting every token weigh every other token in the context when deciding what it means here — and out the far end comes a score for every token in the vocabulary. Sample one. Do it again.

Training means adjusting billions of internal weights so that this prediction gets less wrong across an enormous body of text. Nothing in that process builds an index. There is no table of facts inside, no row for a journal article that the model consults and finds empty. What there is, is a very finely tuned sense of what text of this kind usually looks like. A plausible citation is a shape — a name, a plausible journal for that subject, a year in the right range — and the model is extremely good at shapes.

So the honest one-line answer to "why did it make that up" is: it did not make anything up, because it was never retrieving anything. It generated text that fit. Sometimes text that fits is also true, because true text was overwhelmingly what it learned from. Sometimes it is not, and nothing in the machinery distinguishes those two cases at the moment of generation.

Fluency and accuracy are produced by the same process, so they arrive with the same confidence. Everything that follows about using these tools well comes from taking that sentence seriously.

Four different failures wearing one name

"Hallucination" is used for at least four things that have different causes and different fixes.

Gaps in what it saw. The training data is vast but not complete, and it stops at a cutoff date. Ask about a small regional institution, an obscure standard, a paper from last month, and the model has thin or no signal — but it still produces its best-fitting continuation rather than a blank.

Plausible-pattern completion. This is the citation case. The model has seen hundreds of thousands of references and learned the grammar of a reference perfectly. Generating one that is well-formed and false is the easy outcome; generating one that is well-formed and real requires the specific memorised string to be strong enough to win.

Sampling. Output is drawn from a distribution, not read off deterministically. The setting usually called temperature controls how much the sampling favours the highest-probability token. Turn it down and output gets more repetitive and more conservative; turn it up and it gets more varied and more inventive — in both senses of that word. Some of what looks like a model's belief is just which sample you happened to get.

Pressure to answer. After pretraining, models are tuned on human preference data — the stage usually labelled instruction tuning and RLHF. Human raters, on average, prefer a helpful, complete answer to a refusal. Optimise for that and you get a system with a mild but real bias toward producing something. A model that hedged on every uncertain question would score worse with raters even where hedging was correct.

Why "just make it say I don't know" is hard

The intuitive fix is to have the model report its own confidence. The difficulty is that the model's internal state and the text it emits are not the same thing.

Interpretability research has repeatedly found signals inside these networks that track whether a statement is true or whether the model is on shaky ground — the information is, to some degree, in there. But the text layer is trained to be fluent and to please, and confident prose is fluent, pleasing prose. So the phrase "I'm certain that" is itself just tokens the model predicted. It is a rendering of a style of certainty, not a readout of an internal probability.

This is also why asking a chatbot "are you sure?" is weaker evidence than it feels. You are not querying a confidence meter; you are adding text to the context that makes apologetic-correction tokens more likely. Models often capitulate and change a correct answer under that pressure. Politeness training and truth-tracking are pulling in different directions.

What actually reduces it

The fixes that work all share a shape: stop asking the model to recall, and give it something to work from.

Retrieval-augmented generation (RAG) is the main one in production systems. Before answering, the system searches a real corpus — your documents, a database, the live web — and puts the retrieved passages into the context. The model then summarises and reasons over text that is actually present rather than reconstructing it from weights. This helps a great deal, and it is why a tool that shows sources is usually more trustworthy than one that does not.

It is not a cure. Retrieval fails when the search returns the wrong passages; the model can still misread or over-generalise what it retrieved; and it can blend a retrieved fact with a remembered one and produce a hybrid that no source supports. A citation displayed next to a claim means the system found that document — not that the document says what the sentence says. Checking that link is work no tool has removed.

Constrained tasks help structurally. There is a large reliability gap between recall tasks ("list the papers") and transform tasks ("here is the paper — summarise its method", "rewrite this paragraph", "convert this table to CSV", "find the inconsistency in these numbers"). Transform tasks put the ground truth in the prompt. Most of the genuinely safe daily uses of these tools are transform tasks.

Consistency checks are cheap and underused. Ask the same factual question in a fresh session two or three times. Answers derived from something solidly learned tend to repeat; fabrications tend to vary, because they were sampled rather than retrieved. Divergence across samples is a usable warning light.

And the oldest one: verify anything that carries a name, a number or a date. A DOI that resolves. An ISSN that matches. A statute that exists. In June 2023 a New York lawyer was sanctioned after filing a brief containing judicial decisions that ChatGPT had produced and that did not exist — the court's frustration was not that a machine erred, but that nobody had opened the cases before filing. That remains the whole lesson.

Why it matters for students and researchers

For anyone studying or publishing, the practical consequence is a division of labour, not a verdict. These models are genuinely strong at reformulating, drafting, translating, explaining unfamiliar notation, generating candidate approaches and criticising text you supply. They are weak precisely where academic work is least forgiving: exact citations, precise numerical claims, and anything at the edge of the literature where training data thins out.

Understanding why that asymmetry exists is what separates using the tool from being used by it — and it is also the layer where the jobs are. Prompting is a surface skill; the durable one is knowing how tokenisation, sampling, fine-tuning and retrieval interact well enough to design a system that fails visibly instead of silently. That is the difference between a demo and something an institution can rely on, and it is what applied AI and machine-learning coursework should actually be teaching.

Frequently asked questions

What is an AI hallucination?

It is output that is fluent, well-formed and false — an invented citation, a plausible but wrong number, a confident description of something that does not exist. It happens because a language model generates text by predicting likely continuations rather than by retrieving verified facts.

Why does ChatGPT make up references?

Because it learned the form of a citation from millions of examples, and producing a well-formed citation is far easier than reproducing one exact memorised string. Author names, journal titles and year ranges that fit the subject are all highly predictable patterns, so the model fills them in.

Does a newer or larger model fix hallucination?

Scale and better training reduce the rate, particularly on well-covered topics, but they do not remove the cause. As long as answers are generated by prediction rather than lookup, a fluent wrong answer stays possible — which is why grounding and verification matter more than model size.

Does retrieval-augmented generation solve it?

It helps substantially by putting real source text in front of the model, and it is why tools that cite sources are generally safer. But retrieval can return the wrong documents, and the model can still misread or overstate what it retrieved. A shown citation proves a document was found, not that it supports the claim.

How can I tell if an AI answer is invented?

Check anything with a name, number or date against a real source; ask the same question in a fresh session and see whether the answer holds; and prefer tasks where you supply the material — summarising, rewriting, comparing — over tasks that ask the model to recall facts from memory.