How to Spot Made-Up Numbers and Quotes in a Claude Report

Hello builders,

You can send a Claude report to a client or a CMO with your name on it once you can spot the numbers and quotes it made up, and four kinds of line tell you where to look. The short report below is the kind Claude drafts every day, I wrote it for my book with one error planted in it, and nothing about it feels wrong on a first read. Give yourself 60 seconds to find the error, then I’ll show you how to catch Claude hallucinations, including the harder kind that contradicts nothing else in the draft.

Quarterly Marketing Spend, Summary for the CMO

Marketing spend for the quarter came to $1.84 million, 6% under the approved budget. Paid search remained the largest line, and event spend rose after the two regional conferences. Agency fees fell for the second quarter running as more content work moved in-house. We recommend holding the paid search budget flat next quarter and moving the saving into content.

Paid search $610,000 · Events $425,000 · Content $318,000 · Agency fees $267,000 · Total $1,620,000

Found it? The opening sentence says $1.84 million, and the lines beneath it add up to $1.62 million. The report contradicts itself in its first line, the one a CMO is most likely to repeat in a meeting, and the two figures sit $220,000 apart.

Asking Claude won’t catch it

The tempting fix is to ask Claude how sure it is. Anthropic’s own exam guide marks that as the wrong answer in its first sample question, and the rationale says why: “Self-reported confidence … is not a reliable accuracy signal.” The Claude help center says the same thing more bluntly:

“Claude can display quotes that may look authoritative or sound convincing, but are not grounded in fact. In other words, Claude can write things that might look correct but are very mistaken.”

“Users should not rely on Claude as a singular source of truth and should carefully scrutinize any high-stakes advice given by Claude.”

So we check the draft against its sources, and we decide how hard to look by what’s riding on it. Anthropic’s model tutorial puts the rule well: “check the result in proportion to what’s riding on it: a quick lookup needs only a glance, while analysis that will shape a decision deserves a closer review.”

Four lines always get checked

Here’s the list I use. Four kinds of claim get checked every time, whatever the stakes.

1. Figures. Totals, percentages, growth rates and counts get recomputed or found in the source.
2. Named sources. Quotes, studies and anything “according to” someone get checked against the original.
3. Anything recent. News, current rules and prices get confirmed, because the model has a training cutoff.
4. Anything a decision rests on. If someone will act on it, we trace it back.

If I only have 10 minutes, I spend them on number 4 first, since that’s the line somebody is about to act on.

Now the harder half, which one Reddit user asked outright: “How DO you catch hallucinations and decide when to fact check something, if they didn’t happen to produce inconsistencies?” An invented figure in a competitor summary won’t disagree with any other line. Say your notes read “Revenue up strongly per its annual letter,” and Claude’s snapshot says the competitor “grew revenue 38% last year.” So where did the 38% come from? Your notes never gave a number, and no other line in the snapshot mentions revenue, so there’s nothing for the figure to clash with. Or a transcript has the CFO saying the company is “reviewing our presence in two regions,” and the briefing quotes the CEO saying “we will exit two regions by year end.” Both errors sit quietly in drafts that read well.

To catch an error that agrees with everything around it, we trace each claim on the list back to where it came from. When Claude drafted from attached material, we ask it for the passages, with a message like this one.

For every figure and every quote in your draft, give me the passage from the attached files it came from, word for word, with the file name.

Then we read those passages ourselves, because a quote can be invented too. When Claude can’t point to a passage, we treat the figure as unsourced and either find it in the source or take it out.

Decide good enough first

The last piece is deciding when a draft is ready to send, which sounds like a judgment call until the criteria are written down before anyone looks at the output. For a weekly client summary email, a sheet reads like this, and an email passes only if every line passes.

• Every figure matches the source report
• The one decision needed is in the first two sentences
• Under 200 words
• Nothing from the internal-only list, such as vendor names and internal codes
• Every action has an owner and a date

We grade 10 real outputs against it and record the pass rate out of 10. Below 10, the criterion that failed most often points at the cause, usually a line of the brief or an out-of-date source. Then hand the sheet and three outputs to a colleague to grade without your verdicts; if you disagree on a line, we rewrite how that line gets checked until two people reach the same answer. Start with the next Claude draft you were about to forward today.

Now go build something this weekend!

John Cook