Sign in with Google

How to Fact-Check AI Answers

Four claim types, four checks, and a triage rule so you are not verifying everything equally.

beginner10 min read
ai-toolscritical-thinkingresearchlearning-meta
A model gives you a statistic, a cited paper, and a line of code. Two of the three are right and one is invented, and nothing in the answer marks which. That is the real problem with AI output — not that it is often wrong, but that the wrong parts look exactly like the right parts. What follows is a set of checks calibrated to what a mistake would actually cost, so you spend thirty seconds on claims that do not matter and five minutes on the ones that do.
iWhat you need
Any AI chat tool, a browser for looking things up, and a willingness to be slower for about a week while the habits set. No paid plan required. It helps enormously to have one real task in hand — a decision you are making, something you are writing — because verification practiced on hypothetical questions does not stick.
A language model generates text that is statistically likely to follow your prompt. That is the whole mechanism. It is not looking anything up unless it has been handed a search tool, and even then the sentence it writes around what it found is produced the same way as everything else. Fluency and accuracy are separate axes, and the model is only optimizing one of them.
Which leads to the consequence people find genuinely hard to internalize: confidence carries no information. A fabricated citation arrives in exactly the same measured tone as a real one, because the tone came from the same process that produced the citation. No internal signal leaks out. When you notice yourself thinking "it sounded sure," you have caught yourself reading a cue that does not exist.
!Hedging is not a signal either
Models are trained to sound appropriately uncertain, which makes the hedges stylistic rather than diagnostic. An answer that opens with "I believe" is not measurably more likely to be wrong than one that does not — and a flat declarative is not more likely to be right.
Claims do not all need the same effort. Sort what you were told into four buckets, because each one fails in its own way and each has a specific cheap check.
  1. 1Verifiable facts — dates, names, definitions, who did what. These fail by conflation: two similar people, two similar events, the right fact attached to the wrong subject. Check with one search against a source that would know, and confirm the subject matches, not just the fact.
  2. 2Numbers and statistics. These fail worst, because a plausible magnitude is easy to generate and hard to doubt. Find the primary source — the agency, the study, the filing — and read the number there. Never accept a number from a summary of a summary.
  3. 3Citations and quotes. These fail most spectacularly and get their own section below. The check is opening the source and confirming it says what was claimed.
  4. 4Code and API details. These fail by inventing methods, parameters, and flags that do not exist. Run the code, and read the library's own reference for anything you have not personally used.
✓Suspicion should scale with quotability
A round, striking, repeatable figure — "73% of companies" — is exactly the shape of thing a model produces smoothly whether or not it is real. Roughness in a number is weakly reassuring; polish is weakly alarming.
Citations fail in four distinct ways, and identifying which one you are looking at tells you what to do next.
  1. 1Wholly fabricated: a plausible title, plausible authors, a real journal, a believable year, and nothing behind it. Search the exact title in quotation marks. If nothing comes back, it does not exist.
  2. 2Real paper, wrong findings. The source exists but does not say what was claimed, or says something considerably weaker. This is the most common failure and the hardest to catch, because the first check — does this paper exist — passes cleanly.
  3. 3Real finding, wrong attribution. The result is real but belongs to a different study, a different author, or a different year than the one you were given.
  4. 4A URL that returns a 404 or resolves to something unrelated. Links are generated character by character like everything else; a well-formed URL is not a working one.
The check that actually works is opening the source and locating the specific sentence that supports the claim. Not skimming an abstract for related vocabulary — finding the claim. If you cannot find it, the citation has failed even though the paper is real, and that distinction matters when you go back and ask again.
A model is a poor judge of its own output but a useful assistant in checking it. Four requests genuinely add information rather than just reassurance.
  1. 1Ask for sources you can open, then open them. Not a vague "cite your sources" — ask for links or exact titles you can search, and treat anything you cannot find as a failed claim.
  2. 2Ask what would falsify the claim. "What evidence would show this is wrong?" gives you a concrete thing to look for, and an answer that cannot name one usually signals a claim too vague to be checked.
  3. 3Ask it to separate what it is confident about from what it is inferring. This works better than requesting a confidence score, because it forces the answer into two lists you can treat differently.
  4. 4Ask the same question in a fresh conversation and compare. Recalled facts tend to be stable across runs; fabrications tend to drift. Two different invented citations for one claim is close to a confession.
✗"Are you sure?" is a weak check
It produces either a reversal or a doubling-down, and which one you get depends more on your tone than on the truth. You are measuring agreeableness, not accuracy. To actually test a claim, ask the model to argue the opposite case and judge whether that argument comes out stronger.
Code is the one claim type with a free oracle attached: you do not have to reason about whether a method exists, you can execute it and know in ten seconds. People skip this constantly, because generated code reads as though somebody already tested it.
  1. 1Run it on real input, including an empty case and one edge case you care about. Code that runs is not code that is correct.
  2. 2Look up any method, flag, or parameter you have not used before. Invented parameters are the signature failure here, and they frequently fail silently instead of raising an error.
  3. 3Check the library version the pattern assumes. An approach that was correct three releases ago reads as completely idiomatic and simply will not work.
  4. 4Read what it does when something fails. Generated code assumes the happy path — missing files, empty results, and network errors are routinely unhandled.
You cannot verify everything, and trying to means you stop using the tool at all. Match the effort to the consequence, in four tiers.
  1. 1No check: being wrong costs nothing. Brainstorming, phrasing, explanations you are about to test against something real anyway.
  2. 2One search: you will repeat it to someone, but nothing turns on it. A date, a definition, a name, a rough magnitude.
  3. 3Primary source: it goes into something you publish, send to a client, or decide with. Every number in this tier gets traced to its origin.
  4. 4Full check plus a second opinion: money, health, law, safety, or anything irreversible. Verify independently, then ask a person who knows the domain.
  5. 5The forcing question that picks the tier for you — if this is wrong, who finds out, and when? If the answer is nobody ever, relax. If it is a customer in production, do not.
Every check above is faster and more accurate when you already know something about the subject. Domain knowledge is what produces the flinch — the small "that does not sound right" that tells you where to aim a search. Without it you are checking uniformly and slowly. With it, you check the two claims that matter and skip the rest with a clear conscience.
That is the honest argument against outsourcing your fundamentals to a model. Not that using AI is lazy — but that the ability to catch its errors is built from precisely the knowledge you would be skipping. Learn the basics of the fields you actually depend on and the tool becomes dramatically more useful, because you can finally tell a good answer from a merely fluent one.

Symptom, cause, fix

  1. 1The paper title returns nothing in a search. It was fabricated. Ask for a real source and verify that one instead — do not ask the model to find the paper it invented, because it will hand you a variant.
  2. 2The paper exists but the claim is not in it. The citation was attached to the wrong finding. Treat the claim as unsupported and go looking for its actual source.
  3. 3You caught one error and now distrust the whole answer. Good instinct, wrong response. Recheck the claims that matter and keep the rest; one bad citation does not falsify the other four.
  4. 4Two conversations give you different numbers. At least one was generated rather than recalled, so neither is trustworthy. Go to the primary source.
  5. 5Generated code runs but the output is subtly wrong. It is using an outdated pattern or a misread parameter. Read the reference for every call, then test an input whose correct answer you already know.
  6. 6You are spending more time verifying than the answers save. You are checking everything at the same depth. Triage by cost and stop verifying what does not matter.
  7. 7It confidently contradicts something you know. Say so specifically and supply the correct fact rather than asking whether it is sure — a stated correction is something a model can actually work with.
Verification gets much cheaper when the question was well-posed to begin with, which is what How to Write Better AI Prompts is about — a request that names its constraints produces fewer claims you have to chase. How to Use ChatGPT Effectively and How to Use Claude Effectively cover the tool-specific habits that reduce fabrication in the first place, including when to give a model search access rather than letting it answer from memory. For the code and API bucket, the primary source is documentation, so How to Read Technical Documentation is the companion skill. And because all of this is really applied epistemology, the Philosophy and Critical Thinking roadmap takes the underlying question — how anyone knows anything — considerably further than one guide can.