Hallucinations: checking what AI tells you

AI tools sometimes invent facts, quotes, references and even court cases, and present them with total confidence. Learning to spot and check these 'hallucinations' protects your grades, your reputation and sometimes your safety.

What a hallucination is

• IBM defines AI hallucinations as outputs that sound plausible but are factually wrong, irrelevant or made up.
• LLMs may invent facts, studies, web addresses (URLs) or details about real people and organisations (IBM).
• The cause is built in: generative models predict likely-sounding output from patterns and aim for plausibility, not correctness (IBM).
• Gaps, errors and biases in training data also become part of what the model produces (IBM).

A real case: the lawyers and the fake judgments

• In Mata v. Avianca in a US federal court in New York, lawyers used ChatGPT for legal research and filed a brief citing court decisions that did not exist, complete with invented quotes (LawSites).
• When the court questioned whether the cases existed, the lawyers kept standing by them. On 22 June 2023, Judge P. Kevin Castel fined the lawyers and their firm 5,000 US dollars.
• They were also ordered to notify their client and each real judge who had been falsely named as the author of the fake decisions.
• If trained professionals can be caught out, a student's essay with invented references can be too.

Red flags to watch for

• Specific references, page numbers, quotes or statistics: these are exactly what models tend to invent.
• Web links: click them. A link that leads nowhere or to an unrelated page is a warning sign.
• Claims about real people, especially private individuals or recent events.
• Very recent news: a model's training data has a cut-off date, so it may describe an old situation as current.
• Answers that perfectly match what you hoped to hear. Ask the same question a different way and see if the answer changes.

A five-minute checking routine

• Ask the tool for its sources, then find each one yourself in a library database, on the publisher's site or on an official website. Do not trust a citation just because it exists in the answer (University of Reading).
• Check any number or date against the original source, not against another AI answer.
• Use lateral reading: open new tabs to see what reliable sites say about the claim (covered in the Critical Thinking module).
• Compare the AI's explanation with your textbook, class notes or reading list (University of Oxford).
• If you cannot confirm a claim, leave it out or clearly mark it as unverified.

When the stakes are high

• For health, legal, money or safety questions, treat AI answers as a starting point for questions to ask a qualified person, never as the final word.
• IBM says human oversight is essential wherever hallucinations could cause significant harm.
• In an emergency in the UAE, call the emergency services rather than asking a chatbot (see the First Aid and Personal Safety modules).
• Remember the responsibility is yours: the University of Reading reminds students that responsibility for the work lies with you, not the tool.