Hallucinations: why AI makes things up and how to check its work

In June 2025 the High Court in London heard two cases in which legal arguments relied on case law that did not exist. In one, the court’s review found that 18 of the 45 cases cited were fictitious, and many of the others did not say what they were quoted as saying. The judgment, Ayinde v London Borough of Haringey, warned that freely available AI tools “can produce apparently coherent and plausible responses to prompts” that “may turn out to be entirely incorrect.”

Lawyers are not the only people at risk. Anyone who pastes AI output into a report, a client email or a social post can publish a made-up statistic. This guide explains why AI tools invent things and gives you a checking routine that fits into a normal working day.

What a hallucination is

The US National Institute of Standards and Technology uses the term confabulation in its generative AI risk profile (NIST AI 600-1), defined as systems that “generate and confidently present erroneous or false content in response to prompts.” It notes that the same thing is often called hallucination or fabrication.

In practice it shows up in a few recognisable forms:

• invented facts, figures, dates or people;
• citations, links or studies that do not exist;
• real sources credited with quotes they never contained, which the High Court also described;
• summaries of a document you supplied that add or change details;
• confident answers to a question built on a false premise.

The common thread is confidence. The wrong answer reads exactly like a right one.

Why AI makes things up

Language models are trained to predict the next word across huge amounts of text, and that text carries no “true” or “false” labels. OpenAI’s September 2025 explanation points out that regular patterns such as spelling improve as models grow, but arbitrary facts, such as a particular person’s birthday, cannot be worked out from patterns at all. Where the pattern runs out, the model still produces the most plausible continuation.

OpenAI also argues that common ways of grading models make the problem worse, because they “reward guessing over acknowledging uncertainty.” It compares this to a multiple-choice exam: a blank answer scores zero, so a lucky guess beats an honest “I don’t know.” Its recommendation is that evaluations should penalise confident errors more than expressions of uncertainty.

The practical lesson is that errors cluster where the model has little to go on: niche topics, precise numbers, recent events, named sources and anything you ask it to produce when no such thing exists, such as “five studies proving our approach works.”

Reduce errors before they happen

Anthropic’s guidance on reducing hallucinations lists techniques that work with most tools:

1. Allow “I don’t know”. Tell the model it may say it lacks enough information. Anthropic says this alone “can drastically reduce false information.”
2. Restrict it to your sources. Supply the documents and instruct it to use only those, not its general knowledge.
3. Ask for quotes first. For long documents, have it pull out word-for-word quotes before it answers, then work from those.
4. Require a citation per claim. If it cannot find a supporting quote, it should withdraw the claim.
5. Compare runs. Ask the same question several times. Answers that disagree are a warning sign.

Anthropic is clear that these methods reduce hallucinations but do not eliminate them, and that critical information still needs validating.

A checking routine for AI output

NIST suggests comparing outputs against known ground truth, documenting fact-checks and verifying the sources and citations in outputs. Turned into a routine for everyday work, that looks like this:

1. Mark every checkable claim. Highlight names, numbers, dates, quotes, citations and statements of law or policy.
2. Trace each one to a primary source. Go to the original: the regulator, the statute, the company’s own report. The High Court pointed lawyers to sources such as legislation.gov.uk and the National Archives’ case law service. The same principle applies in every field.
3. Open every link. Check that the page exists and says what the output claims.
4. Search for each quote. Find the exact words in the source. If they are not there, treat the quote as invented.
5. Recalculate numbers. Redo any sums, percentages or currency conversions yourself.
6. Record what you checked. A short note of sources and dates protects you if the work is questioned later.

A worked example

An example (illustration only): a junior analyst asks an AI tool to summarise a 60-page market report for a client slide. The summary says the market “grew 14% in 2024, led by online sales” and cites page 12.

Following the routine, she opens page 12. The 14% figure is there, but it is a forecast for 2026, not growth in 2024. The report does mention online sales, as one of three drivers, not the leading one. A third claim in the summary, about a named competitor’s market share, appears nowhere in the report at all.

The check took 15 minutes. She rewrites the slide from the source, notes the page number for each figure, and adds an instruction to her prompt for next time: quote the sentence behind each number, and say so when the report does not cover a point.

Common mistakes

• Asking the same chat “Are you sure?” A model can confirm a wrong answer as readily as a right one. Check the source, not the model’s confidence.
• Trusting a citation because it looks properly formatted. Fabricated references often have realistic titles, authors and page numbers.
• Checking only the first few claims. Errors are often buried in the detail near the end.
• Assuming a newer or paid model has solved the problem. Better models make fewer errors, but none of the guidance cited here says any model is free of them.
• Passing the checking job down without guidance. The High Court said the duty to check applies equally to work produced by trainees or paralegals. Whoever signs off owns the result.

How much checking is enough

Match the effort to the stakes. Brainstorming headlines or rephrasing an email needs little more than a read-through. Anything that will be published, sent to a client, used in a decision about a person, or that involves legal, financial, medical or safety information needs the full routine, every time.

A useful test: if this claim turned out to be false, who would be affected and how badly? If the answer is “a customer, a client or a court”, trace it to the source before it leaves your hands.