Test cleanup is the job everyone wants to hand off. Can the cheap model do it without quietly deleting coverage?
We gave Sonnet 5.5 and Haiku 5.5 the same messy test suite: duplicates, skips, a dead import, and one precise SAVE10 assertion that looks redundant. Both kept the coverage and caught a SAVE10→9% mutation in 3 of 3 runs. Haiku averaged $0.013 a run; Sonnet $0.117, about 9× more.
@kaiargus answered our question about what to hand off to Claude Code with: “the test cleanup. boring, easy to check, and if the agent messes it up nothing broken ships.” The risk with that job is quiet: an agent can delete a test that looks redundant and leave the suite green while coverage is gone. We put Sonnet and Haiku on the same messy suite and graded them on whether a planted edge still caught a one-line bug.
The setup
The fixture is a small Node ESM pricing library with no dependencies: subtotal, discount, applyCoupon (SAVE10 = 10% off, FLAT5 = $5 off capped at the subtotal, empty/whitespace = none, unknown throws Unknown coupon: <trimmed>), and total (coupon then 8% tax), all rounded with roundCents. Production under src/ is correct and lean. The prompt forbids changing it.
The mess is the point. Under tests/ we planted:
- Duplicate soft SAVE10 checks that only assert the total got cheaper (a 9% SAVE10 bug still passes them)
- Copy-pasted FLAT5 fixtures instead of a shared helper
test.skipleftovers and a commented-out old coupon- A dead test that imports a removed
legacyComputeDiscounthelper - No-op asserts (
typeof r === 'number',assert.ok(true)) - Misleading names
- One precise SAVE10 assertion on a cart of
$10.05, where 10% is$1.005and rounds to$1.01. It looks like yet another “SAVE10 works” test. Delete it, and the soft checks stay green under a 9% bug.
The prompt (same for both models):
Clean up this project’s test suite without changing behavior. Deduplicate redundant tests, rename unclear tests, remove dead or skipped tests that add nothing, and extract shared fixtures where it helps. Do not change any files under src/. Run the tests when you’re done and leave them passing.
Two setups, 3 runs each, on Claude Code 2.1.312:
- Sonnet:
claude-sonnet-5-5 - Haiku:
claude-haiku-5-5
Seven checks graded each run afterwards. The two that answer the question: a hidden behavioral suite Claude never saw (discount, coupon, total edges, including the half-cent SAVE10 and the FLAT5 cap), and a mutation check that flips SAVE10 from 10% to 9%, expects npm test to fail, then restores the source. The mutation check only counts if the suite was green before the plant, so a leftover broken import cannot fake a catch. The others: npm test passes, src/ untouched, tests reran after the last edit, changes only under tests/ with no new deps, and a precise SAVE10 amount assertion still present.
What happened
| Sonnet 5.5 | Haiku 5.5 | |
|---|---|---|
| Mutation caught (SAVE10→9%) | 3 of 3 | 3 of 3 |
| Hidden behavioral suite | 3 of 3 | 3 of 3 |
npm test passes |
3 of 3 | 3 of 3 |
src/ untouched |
3 of 3 | 3 of 3 |
| Precise SAVE10 coverage kept | 3 of 3 | 3 of 3 |
Changes only under tests/ |
3 of 3 | 3 of 3 |
| Ran tests after last edit | 3 of 3 | 3 of 3 |
| All 7 checks | 3 of 3 | 3 of 3 |
| Average cost per run | $0.117 | $0.013 |
| Cost range | $0.103 to $0.143 | $0.010 to $0.016 |
| Average tokens per run | 241,646 | 363,247 |
| Average output tokens | 5,659 | 12,808 |
| Average turns | 13.0 | 21.0 |
| Average time | 38.6s | 54.6s |
| Average diff lines (+ and −) | 242 | 210 |
3 runs per column (averages unless marked). Token counts include cached input. Every run, check and diff: run data.
What we learned
Haiku kept the coverage. All 3 Haiku runs deleted the dead import, the skips, and the soft duplicates, and still left a precise SAVE10 amount assertion that failed under the 9% mutation. So did all 3 Sonnet runs. On this fixture, the cheap model did not quietly delete the planted edge.
The price gap was the story. Haiku averaged $0.013 a run; Sonnet $0.117, about 9× more. Haiku used more turns (21 vs 13) and more output tokens (about 2.3×), and took longer (54.6s vs 38.6s), but at Haiku prices the extra work still came in under two cents.
Both cleaned real mess, not just formatting. Diffs averaged a few hundred lines, mostly deletions: shared fixtures, renamed tests, one coupons file instead of a pile of copies. Sonnet’s diffs were a bit larger on average (242 lines vs 210); neither setup edited src/ or added dependencies.
The mutation check is what makes this job gradeable. A green npm test after cleanup is not enough. Soft “cheaper than before” assertions stay green when SAVE10 becomes 9%. The check that answered @kaiargus’s risk was: break SAVE10 on purpose and watch the cleaned suite fail.
What failed, and the limits of this test
- Nothing failed on the scored runs. All 6 passed all 7 checks, so this test cannot tell you which model is safer on a harder cleanup. It only shows that on this suite, Haiku matched Sonnet on coverage kept and cost a lot less.
- The planted edge had to look redundant without announcing itself. Early fixture comments that explained the 9% bug were stripped before scoring so the agent was not being lectured. A messier real suite, or one where the only precise assert is easier to mistake for a duplicate, might still separate the models.
- Three runs a side is a small sample. Use this as a cost signal for a boring, checkable handoff, not as a rate.
The recipe
- When you hand Claude Code a test cleanup, grade it with something the agent never saw: a hidden suite, a mutation, or both.
- Soft asserts (“total got cheaper”) are not enough to protect percentage discounts. Keep at least one exact amount.
- For this kind of cleanup, try Haiku first. Here it matched Sonnet on every check at about one-ninth the cost.
- Forbid edits under
src/in the prompt, and checkgit statusonsrc/afterwards.