Opus writes the tests, Sonnet implements: 5 of 5 on the hidden edge cases, but 0 of 5 passed the lint check, at 2.8x the cost
We built the same bill-splitting function 15 times: plain Sonnet 5.5, Sonnet told to write tests first, and Sonnet implementing against a test file Opus 5.5 wrote in a separate session and Sonnet was not allowed to edit. Hidden edge-case test: 4 of 5, 4 of 5 and 5 of 5. But none of the 5 Opus-tests runs passed the project's own npm run check, and they cost $0.219 a run vs $0.079 for plain Sonnet.
Other checks: Plain sonnet run 5 missed “ran tests after last edit”; Sonnet tests first run 1 missed “ran tests after last edit”; All 5 Opus tests locked runs missed “npm run check passes”. All 5 Opus-tests runs failed npm run check on a lint rule (money must be rounded with roundCents(), not Math.round/Math.floor). In the plain and tests-first arms, 1 run in each did not run the tests after its last edit.
A reader (@damkina7) asked: does having a stronger model write the tests first, with the test file locked, stop the agent grading its own homework better than a plain “write tests first” prompt? Our Oct 16 test showed the plain tests-first sentence helps a little. This one adds a third setup where the tests come from a different model and the implementer can’t touch them.
The setup
The same small task as the Oct 16 test: add splitBill(items, people) to a cart module, splitting the taxed total between people. The repo has a lint rule inside npm run check (money must be rounded with the project’s roundCents()), and a hidden test file Claude never saw checks the edge cases: shares add up to total() to the cent, no two shares differ by more than a cent, one person, an empty cart, and invalid people counts.
Three setups, 5 runs each, on Claude Code 2.1.321, every run in a fresh copy of the repo:
- Plain Sonnet:
claude-sonnet-5-5, the one-sentence task and nothing else. - Sonnet, tests first: the same prompt plus “Before you implement it, write tests for it, including edge cases, and run them to see them fail. Then implement it until they pass.” (the exact wording from the Oct 16 test).
- Opus tests, locked, then Sonnet: two separate sessions per run. First
claude-opus-5-5was asked to write tests for the not-yet-existingsplitBillintotests/lib/splitbill.test.js, with edits undersrc/denied so it couldn’t implement anything. A new Sonnet session then implementedsplitBilland was told the test file was locked. A fresh Opus test file was generated for each of the 5 runs, and Opus’s cost is counted in that setup’s cost per run.
How the lock works. chmod alone isn’t a lock, since the agent could change it. We passed Claude Code a deny rule for Edit and Write on that one file during the Sonnet session, and the runner recorded a sha256 hash of the file before the Sonnet step and compared it afterwards (one of the checks). Bash was already limited to npm / node --test / read-only git commands. We tested the lock in a $0.054 smoke run first: Claude Code refused an Edit and a Write of the locked file and a sed -i through Bash, while an edit to another file went through, and the hash was unchanged. Neither model ever saw the hidden test.
Six checks graded each run (the sixth only for the Opus setup): npm run check passes, the hidden edge-case test, a splitBill test was added, generated files untouched, tests run after the last edit, and for the Opus setup the locked file’s hash unchanged. We also applied every saved diff to a clean copy of the repo and re-ran npm run check and the hidden tests. The results matched the runner’s.
What happened
| Plain Sonnet | Sonnet, tests first | Opus tests (locked) + Sonnet | |
|---|---|---|---|
| Hidden edge-case test | 4 of 5 | 4 of 5 | 5 of 5 |
npm run check passes |
5 of 5 | 5 of 5 | 0 of 5 |
| Ran tests after last edit | 4 of 5 | 4 of 5 | 5 of 5 |
| Added a splitBill test | 5 of 5 | 5 of 5 | 5 of 5 |
| Generated files untouched | 5 of 5 | 5 of 5 | 5 of 5 |
| Locked file hash unchanged | n/a | n/a | 5 of 5 |
| All checks | 3 of 5 | 3 of 5 | 0 of 5 |
| Average cost per run | $0.079 | $0.098 | $0.219 |
| Cost range | $0.075 to $0.089 | $0.086 to $0.110 | $0.176 to $0.286 |
| Average time per run | 25.8s | 35.4s | 54.0s |
| Median time per run | 24.0s | 37.0s | 45.9s |
| Average turns | 16.4 | 13.4 | 17.8 |
For the Opus setup, the average cost splits into $0.172 for writing the tests (Opus) and $0.048 for the implementation (Sonnet). 5 runs per column. Costs as Claude Code reports them. Every run, check and diff: run data.
What we learned
The Opus-written tests did get the edge cases right, 5 of 5 times. The hidden test passed in every run of that setup. The two misses elsewhere, one in plain Sonnet (run 4) and one in tests-first (run 4), were the same bug: the last person absorbed all the rounding leftovers, so shares differed by more than a cent. With 4 of 5 vs 5 of 5, that is a one-run difference. It’s a hint, not proof that independent tests beat the tests-first sentence.
All 5 Opus-tests runs failed the project’s own check. Every implementation used Math.round or Math.floor in cart.js, which the lint step inside npm run check rejects. None of those 5 Sonnet sessions ran npm run check (from the transcripts); they ran the test file and stopped. All 10 plain and tests-first runs did run it and avoided the rule. We can’t say why. Our implementation prompt for this setup said “run the tests” and described the locked test file, and the other two prompts didn’t, so the wording is a possible cause. The locked tests also meant Sonnet’s attention went to passing a given file, not to reading the repo’s conventions.
It cost 2.8x as much as plain Sonnet. $0.219 vs $0.079 a run, and 2.2x the tests-first setup’s $0.098. Opus’s test-writing alone was $0.172 on average, about 78% of the total. It also took about twice as long: a median of 45.9s vs 24.0s.
The lock was never tested in the scored runs. The Sonnet sessions never tried to edit the locked file, so there were 0 denied writes to it in the 5 runs. The denial is demonstrated only in the smoke run, and the hash check passed 5 of 5. Locking costs nothing, but we can’t say it changed anything here.
What failed, and the limits of this test
- 5 runs per setup is a small sample. 5 of 5 vs 4 of 5 on the hidden test is one run’s difference, and 0 of 5 on
npm run checkis a count from only one task. - The Opus setup bundles two things. It combines a different author for the tests with a lock on them. We didn’t run Opus tests without the lock, or locked Sonnet-written tests, so this can’t tell you what the locking added.
- The prompts weren’t identical. The Opus setup’s implementation prompt mentions the locked tests and says to run them, which may be why none of its runs ran
npm run check. - Two runs didn’t re-run tests after their last edit: plain Sonnet run 5 and tests-first run 1.
- One task, one small repo, one pair of models. Whether Opus-written tests help on a harder function is untested.
- Smoke runs of 1 Opus-tests run ($0.210) and the lock tests ($0.084 and $0.054) came before the scored runs. The first lock smoke’s results weren’t kept; the second ($0.054) and the one-run Opus smoke are kept with the run data. The whole slot cost $2.33.
The recipe
- If you hand tests to another model, also tell the implementer to run the project’s full check (
npm run check), not only the tests. Ours skipped the lint step in 5 of 5 runs. - Hidden edge-case tests are the fair way to grade this. Tests the same agent wrote and then passed prove little; the hidden test caught 2 of 10 plain and tests-first runs.
- Price the whole pipeline. Writing tests with Opus added $0.172 a run to a $0.048 implementation.
- A deny rule on a single file (
--disallowedTools "Edit(./path)" "Write(./path)") did stop an edit, a write and ased -iin our smoke run.chmoddoesn’t.