← All recipes

Opus writes the tests, Sonnet implements: 5 of 5 on the hidden edge cases, but 0 of 5 passed the lint check, at 2.8x the cost

We built the same bill-splitting function 15 times: plain Sonnet 5.5, Sonnet told to write tests first, and Sonnet implementing against a test file Opus 5.5 wrote in a separate session and Sonnet was not allowed to edit. Hidden edge-case test: 4 of 5, 4 of 5 and 5 of 5. But none of the 5 Opus-tests runs passed the project's own npm run check, and they cost $0.219 a run vs $0.079 for plain Sonnet.

Cost per run Each bar is the average cost of the runs, starting at $0.

Other checks: Plain sonnet run 5 missed “ran tests after last edit”; Sonnet tests first run 1 missed “ran tests after last edit”; All 5 Opus tests locked runs missed “npm run check passes”. All 5 Opus-tests runs failed npm run check on a lint rule (money must be rounded with roundCents(), not Math.round/Math.floor). In the plain and tests-first arms, 1 run in each did not run the tests after its last edit.

See every run, check and diff →

A reader (@damkina7) asked: does having a stronger model write the tests first, with the test file locked, stop the agent grading its own homework better than a plain “write tests first” prompt? Our Oct 16 test showed the plain tests-first sentence helps a little. This one adds a third setup where the tests come from a different model and the implementer can’t touch them.

The setup

The same small task as the Oct 16 test: add splitBill(items, people) to a cart module, splitting the taxed total between people. The repo has a lint rule inside npm run check (money must be rounded with the project’s roundCents()), and a hidden test file Claude never saw checks the edge cases: shares add up to total() to the cent, no two shares differ by more than a cent, one person, an empty cart, and invalid people counts.

Three setups, 5 runs each, on Claude Code 2.1.321, every run in a fresh copy of the repo:

  • Plain Sonnet: claude-sonnet-5-5, the one-sentence task and nothing else.
  • Sonnet, tests first: the same prompt plus “Before you implement it, write tests for it, including edge cases, and run them to see them fail. Then implement it until they pass.” (the exact wording from the Oct 16 test).
  • Opus tests, locked, then Sonnet: two separate sessions per run. First claude-opus-5-5 was asked to write tests for the not-yet-existing splitBill into tests/lib/splitbill.test.js, with edits under src/ denied so it couldn’t implement anything. A new Sonnet session then implemented splitBill and was told the test file was locked. A fresh Opus test file was generated for each of the 5 runs, and Opus’s cost is counted in that setup’s cost per run.

How the lock works. chmod alone isn’t a lock, since the agent could change it. We passed Claude Code a deny rule for Edit and Write on that one file during the Sonnet session, and the runner recorded a sha256 hash of the file before the Sonnet step and compared it afterwards (one of the checks). Bash was already limited to npm / node --test / read-only git commands. We tested the lock in a $0.054 smoke run first: Claude Code refused an Edit and a Write of the locked file and a sed -i through Bash, while an edit to another file went through, and the hash was unchanged. Neither model ever saw the hidden test.

Six checks graded each run (the sixth only for the Opus setup): npm run check passes, the hidden edge-case test, a splitBill test was added, generated files untouched, tests run after the last edit, and for the Opus setup the locked file’s hash unchanged. We also applied every saved diff to a clean copy of the repo and re-ran npm run check and the hidden tests. The results matched the runner’s.

What happened

Plain Sonnet Sonnet, tests first Opus tests (locked) + Sonnet
Hidden edge-case test 4 of 5 4 of 5 5 of 5
npm run check passes 5 of 5 5 of 5 0 of 5
Ran tests after last edit 4 of 5 4 of 5 5 of 5
Added a splitBill test 5 of 5 5 of 5 5 of 5
Generated files untouched 5 of 5 5 of 5 5 of 5
Locked file hash unchanged n/a n/a 5 of 5
All checks 3 of 5 3 of 5 0 of 5
Average cost per run $0.079 $0.098 $0.219
Cost range $0.075 to $0.089 $0.086 to $0.110 $0.176 to $0.286
Average time per run 25.8s 35.4s 54.0s
Median time per run 24.0s 37.0s 45.9s
Average turns 16.4 13.4 17.8

For the Opus setup, the average cost splits into $0.172 for writing the tests (Opus) and $0.048 for the implementation (Sonnet). 5 runs per column. Costs as Claude Code reports them. Every run, check and diff: run data.

What we learned

The Opus-written tests did get the edge cases right, 5 of 5 times. The hidden test passed in every run of that setup. The two misses elsewhere, one in plain Sonnet (run 4) and one in tests-first (run 4), were the same bug: the last person absorbed all the rounding leftovers, so shares differed by more than a cent. With 4 of 5 vs 5 of 5, that is a one-run difference. It’s a hint, not proof that independent tests beat the tests-first sentence.

All 5 Opus-tests runs failed the project’s own check. Every implementation used Math.round or Math.floor in cart.js, which the lint step inside npm run check rejects. None of those 5 Sonnet sessions ran npm run check (from the transcripts); they ran the test file and stopped. All 10 plain and tests-first runs did run it and avoided the rule. We can’t say why. Our implementation prompt for this setup said “run the tests” and described the locked test file, and the other two prompts didn’t, so the wording is a possible cause. The locked tests also meant Sonnet’s attention went to passing a given file, not to reading the repo’s conventions.

It cost 2.8x as much as plain Sonnet. $0.219 vs $0.079 a run, and 2.2x the tests-first setup’s $0.098. Opus’s test-writing alone was $0.172 on average, about 78% of the total. It also took about twice as long: a median of 45.9s vs 24.0s.

The lock was never tested in the scored runs. The Sonnet sessions never tried to edit the locked file, so there were 0 denied writes to it in the 5 runs. The denial is demonstrated only in the smoke run, and the hash check passed 5 of 5. Locking costs nothing, but we can’t say it changed anything here.

What failed, and the limits of this test

  • 5 runs per setup is a small sample. 5 of 5 vs 4 of 5 on the hidden test is one run’s difference, and 0 of 5 on npm run check is a count from only one task.
  • The Opus setup bundles two things. It combines a different author for the tests with a lock on them. We didn’t run Opus tests without the lock, or locked Sonnet-written tests, so this can’t tell you what the locking added.
  • The prompts weren’t identical. The Opus setup’s implementation prompt mentions the locked tests and says to run them, which may be why none of its runs ran npm run check.
  • Two runs didn’t re-run tests after their last edit: plain Sonnet run 5 and tests-first run 1.
  • One task, one small repo, one pair of models. Whether Opus-written tests help on a harder function is untested.
  • Smoke runs of 1 Opus-tests run ($0.210) and the lock tests ($0.084 and $0.054) came before the scored runs. The first lock smoke’s results weren’t kept; the second ($0.054) and the one-run Opus smoke are kept with the run data. The whole slot cost $2.33.

The recipe

  1. If you hand tests to another model, also tell the implementer to run the project’s full check (npm run check), not only the tests. Ours skipped the lint step in 5 of 5 runs.
  2. Hidden edge-case tests are the fair way to grade this. Tests the same agent wrote and then passed prove little; the hidden test caught 2 of 10 plain and tests-first runs.
  3. Price the whole pipeline. Writing tests with Opus added $0.172 a run to a $0.048 implementation.
  4. A deny rule on a single file (--disallowedTools "Edit(./path)" "Write(./path)") did stop an edit, a write and a sed -i in our smoke run. chmod doesn’t.

Keep reading

All recipes →
  • Verified

    3 of 3 eachruns kept SAVE10 coverage and caught a 9% mutation; $0.013 a run for Haiku 5.5, $0.117 for Sonnet 5.5

    Test cleanup is the job everyone wants to hand off. Can the cheap model do it without quietly deleting coverage?

    We gave Sonnet 5.5 and Haiku 5.5 the same messy test suite: duplicates, skips, a dead import, and one precise SAVE10 assertion that looks redundant. Both kept the coverage and caught a SAVE10→9% mutation in 3 of 3 runs. Haiku averaged $0.013 a run; Sonnet $0.117, about 9× more.

    • models
    • testing
    • cost
  • Verified

    5 of 5 vs 4 of 5runs passing the hidden tests: Sonnet 5.5 high effort ($0.386 a run) vs Opus 5.5 low effort ($0.461 a run)

    Sonnet on high effort vs Opus on low: Sonnet passed 5 of 5 for 16% less, Opus finished sooner

    We ran the same 3-step CSV export feature in a 164-file repo 5 times with Sonnet 5.5 at --effort high and 5 times with Opus 5.5 at --effort low. Sonnet high passed the hidden tests in 5 of 5 runs for $0.386 a run. Opus low passed 4 of 5 for $0.461, but its median run took 98s vs 134s for Sonnet.

    • models
    • effort
    • cost
  • Verified

    $2.64 vs $3.28average cost per run, Fable 5.2 (Claude Code 2.1.321 rerun) vs Fable 5.1 (both fixed the bug in every scored run)

    Fable 5.2 is 20% cheaper per token. On our bug fix it averaged $2.64 a run, under Fable 5.1's $3.28

    Launch-day test: Claude Fable 5.2, Fable 5.1 and Opus 5.5 on the same planted bug in undici (902 files), same prompt and tools. Fable 5.2 (rerun on Claude Code 2.1.321) fixed the bug in 3 of 3 and averaged $2.64 a run and 2.37M tokens. Fable 5.1 averaged $3.28 and 2.04M; Opus $0.77 and 1.39M. Small samples: 3, 2 and 3 runs.

    • models
    • cost

Compare this test with every other one on the scoreboard.