How to Stop a Claude Code Subagent From Faking a Test Pass

Hello builders,

A Claude Code verifier subagent will happily report a pass on tests it never executed, and the fix is to make its verdict a file it had to run something to produce. I built one to check another node’s work and it came back green having executed nothing at all. Not a hallucination, not a wrong answer: a confident, detailed, correct-sounding verdict about eight test cases, zero of which ran. The code turned out to be fine, which is the part that should worry you, because I’d have shipped a verification pipeline that had never verified anything and it would have kept reporting green until the first time it mattered.

Here’s what we get, side by side. Same author node, same verifier node, same input, run twice.

The left panel is the one we should study. The author node returned SELF-GRADE: PASS. The verifier node agreed, at length:

Verification complete: The author’s PASS claim is correct.

It went on to say it had tested median.sh against SPEC.md with eight test cases and confirmed every requirement was met. Eight. Named, counted, confirmed. Then, further down its own prose, the author admitted what the verdict did not:

This sandbox blocks all bash/script execution in this session… So I could not actually run the script.

And the verifier had lost its permission to run commands, quietly, to a line nobody was reading:

Ignoring 2 permissions.allow entries from .claude/settings.json: this workspace has not been trusted.

On disk: nothing. Zero of the eight ran. Somebody quoted Carlos Perez on exactly this shape and I haven’t been able to stop thinking about the sentence since: everything is consistent and nothing is verified.

Make the verdict a file

Our fix is not a better prompt. A verdict is worth exactly what it executed, and nothing in the output tells the two apart, so our stop condition has to be an artifact the node could only produce by running something.

Two files do it for us. First the node, at .claude/agents/verifier.md:

---
name: verifier
description: Independently tests an artifact against a spec. Ignores every claim it is handed.
tools: Bash, Read, Write
disallowedTools: Edit
model: haiku
maxTurns: 12
---

You are handed an artifact and a spec. IGNORE any claim about whether it works.
Derive your own cases from the spec, including the edge case the author would skip.
RUN them with Bash. You may not report a verdict you did not execute.
Write your per-case results to results.json before you finish.
Reply with exactly `VERDICT: PASS` or `VERDICT: FAIL`, then one line per failing case.

Three things in there are doing the work, and none of them is the prose:

• model: haiku, while the main session runs something else, so that the checker has a different familiarity profile from the thing it is checking.
• disallowedTools: Edit, which means the verifier cannot fix what it is judging, and that matters more than it looks, because a verifier that can edit will quietly repair the defect, report a pass, and never let us learn the generator produced one.
• The criteria live in the file, permanently, and “derive your own cases from the spec, including the edge case the author would skip” is a criterion in a way that “check that it works” is not, because the published failure mode of this pattern is that a verifier told only to check whether output is good will rubber-stamp whatever the generator handed it.

Then the stop condition, at .claude/hooks/require-results.sh:

#!/usr/bin/env bash
if [ ! -f results.json ]; then
  echo "STOP REFUSED: no results.json. A verdict you did not run is not a verdict." >&2
  exit 2
fi
exit 0

{ "hooks": { "SubagentStop": [ { "hooks": [
      { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/require-results.sh" } ] } ] } }

Our verifier is now not allowed to finish until a results file exists. It cannot talk its way past this. Producing the artifact is the only exit.

Only exit 2 blocks

This is the single most common way people ship a gate that does nothing, so here is what the documentation says:

For most hook events, exit code 2 is the only exit code that blocks through the code alone. Without valid JSON on stdout, Claude Code treats exit code 1 as a non-blocking error and proceeds with the action, even though 1 is the conventional Unix failure code. If your hook is meant to enforce a policy, use exit 2.

And on which events can actually stop something, the same page is explicit. PreToolUse blocks the tool call. SubagentStop prevents the subagent from stopping. PostToolBatch stops the agentic loop before the next model call. Stop prevents Claude from stopping and continues the conversation. PostToolUse and SubagentStart cannot block at all, so a gate we write there is decoration no matter what it returns. (Claude Code documentation, Hooks)

I measured the difference on a hook that blocks rm, and it’s one character. With exit 2 the CLI returns 0, permission_denials carries one entry naming the blocked command, and the file is still there. With exit 1 the CLI also returns 0, permission_denials is [], and the file is gone. Same hook, same command. The model reported the file scratch.txt has been deleted successfully, and it was right.

Point it at something you know is broken

Now the part people skip. Give it a defect you can check by hand.

SPEC.md says: count.sh PATTERN FILE prints the number of times PATTERN occurs in FILE. Occurrences, not lines. And count.sh is grep -c "$1" "$2". In a file where beta appears 4 times across 2 lines, grep -c counts lines, so the script returns 2.

The verifier came back VERDICT: FAIL, named the case, and left this behind at 1,886 bytes:

{ "test_id": 1, "pattern": "beta", "file": "sample.txt",
  "expected": 4, "actual": 2,
  "status": "FAIL" }

That’s the whole difference from the run at the top. The verdict is a file that could only exist if something ran.

Then freeze what it caught, because the same bug comes back. We write the case down as data with a why field explaining the bug it came from, and we run the frozen set with a script that has no model in it anywhere. Fix the artifact and all three of mine passed. Reintroduce the bug on purpose and watch it fire:

FAIL  001-occurrences-not-lines    got 2, want 4
PASS  002-absent-pattern           0
PASS  003-single-occurrence        1
---
2 passed, 1 failed
exit=1

Notice which of our cases held. Two of my three passed under the broken implementation, because a pattern that appears zero times and a pattern that appears once give the same answer either way. A regression set made only of easy cases is a regression set that cannot fail. Case 001 exists because a verifier found the one input where the two implementations diverge, and it is doing all the work.

One honest limit, because a piece that oversells verification is doing the thing it warns about. Self-grading is not always wrong. I ran the demonstration three ways, and when the author had a working shell it caught the grep -c bug and returned FAIL correctly. It was only the run where nothing could execute that both agents returned green having run nothing. So the failure is not that models always rubber-stamp themselves. The failure is that a verdict is worth exactly what it executed, and the output looks identical either way.

Which is why the stop condition is a file and not an instruction. If your verifier has never rejected anything, you have not tested it. Go and plant a defect you can check by hand, watch it come back FAIL, and only then point it at one you cannot.

Now go build something this weekend!

John Cook