← All recipes

Sonnet on high effort vs Opus on low: Sonnet passed 5 of 5 for 16% less, Opus finished sooner

We ran the same 3-step CSV export feature in a 164-file repo 5 times with Sonnet 5.5 at --effort high and 5 times with Opus 5.5 at --effort low. Sonnet high passed the hidden tests in 5 of 5 runs for $0.386 a run. Opus low passed 4 of 5 for $0.461, but its median run took 98s vs 134s for Sonnet.

Cost per run Each bar is the average cost of the runs, starting at $0.

See every run, check and diff →

This is the first test in a series on reasoning effort vs model tier. The question: if you turn a cheaper model up and a stronger model down, which one does the better job, and for how much? Claude Code’s --effort flag sets the level per session (low, medium, high, xhigh, max). We put Sonnet 5.5 on high and Opus 5.5 on low and gave them the same feature.

The setup

The same task as our mixed-models test: fleetdesk, a Node API with 164 files, 16 modules and 18 test files. One feature, three steps, in one continuous session:

  1. Plan (plan mode): a CSV export of a tenant’s invoices at GET /billing/invoices.csv, with exact columns, money formatted “the way the rest of the codebase formats money”, inclusive date filters, sorting, RFC 4180 quoting, a header-only body when nothing matches, and 400/404 rules. “Plan the change first. Don’t edit anything yet.”
  2. Implement: “Go ahead and implement the plan, with tests. Run the tests.”
  3. Follow-ups: an optional currency filter (USD, EUR or GBP, anything else is a 400) and a line in CHANGES.md.

The seeded invoices have traps the hidden tests check: a customer name with a comma, one with double quotes, two invoices on the same date, a 105-cent amount, another tenant’s invoice and one dated the day before the range.

Two setups, 5 runs each, on Claude Code 2.1.321:

  • Sonnet high: claude --model claude-sonnet-5-5 --effort high
  • Opus low: claude --model claude-opus-5-5 --effort low

Nine checks graded every run afterwards. The two that answer the question are hidden tests Claude never saw: 4 for the CSV export and 1 for the currency follow-up. We also re-applied each saved diff to a clean copy of the repo and ran the hidden tests again; the results matched.

What happened

Sonnet 5.5, high effort Opus 5.5, low effort
CSV export hidden tests (4) 5 of 5 4 of 5
Currency follow-up hidden test 5 of 5 5 of 5
All 9 checks 5 of 5 4 of 5
Average cost per run $0.386 $0.461
Cost range $0.319 to $0.537 $0.438 to $0.522
Average time per run 160s 131s
Median time per run 134s 98s
Average turns 36.2 24.2
Average tool calls 33.2 21.2
Average output tokens 19,948 11,723
Average tokens per run (incl. cached input) 680,739 388,748
Average diff lines (+ and −) 169 121

5 runs per column. Costs as Claude Code reports them. Every run, check and diff: run data.

What we learned

Sonnet high did the job every time, for less. It passed every hidden test in 5 of 5 runs at $0.386 a run on average, 16% below Opus low’s $0.461. Opus low did less work, with about 41% fewer output tokens and 12 fewer turns, but Opus’s higher price per token more than made up for that.

Opus low was faster. Its median run took 98s vs 134s for Sonnet high. Opus low wrote less: smaller diffs (121 lines vs 169), fewer tests, and fewer tool calls. Sonnet high read more of the repo and wrote more tests; in 4 of 5 runs it touched 8 or more files, while Opus low touched 5 to 7.

The one miss was a judgment call on line endings. Opus low run 2 ended every CSV line with \r\n. RFC 4180 does say CRLF, and the prompt cited RFC 4180 for quoting, but it also said “end the body with a newline”, and the hidden tests expect \n. Its own tests passed because they expected \r\n too. It’s a real spec-reading miss for a client that splits on \n, but it isn’t a broken export. The other 4 Opus runs used \n.

For reference, Sonnet at its default effort did this same task for less. In our Oct 30 test, plain Sonnet 5.5 passed all 9 checks in 3 of 3 runs at $0.283 a run, with about 15,800 output tokens. High effort here cost 36% more and used about 26% more output tokens. That was Claude Code 2.1.312, not 2.1.321, and only 3 runs, so treat it as a hint, not a result.

What failed, and the limits of this test

  • 5 runs a side is a small sample. 5 of 5 vs 4 of 5 is one run’s difference, and that run failed on line endings. Read this as “both setups mostly got it right; Sonnet high cost less,” not as a pass rate.
  • One slow run per side. Sonnet high run 3 took 309s and Opus low run 4 took 262s. Both spent about 218s on step 2 alone, vs 39 to 62s in the other runs, and the Opus run did it in only 10 turns. The two sides ran at the same time, so this looks like a shared slowdown rather than the model, but the logs can’t prove it. That’s why the table shows the median as well as the average.
  • We didn’t see the effort level in the logs. We passed --effort on every step, and Claude Code’s init event reports per-turn effort as active, but it doesn’t print the level. The behavior fits: Sonnet high used more output tokens than Sonnet at default effort did in October.
  • One task, one repo. A harder task, where Opus’s extra capability matters more, could flip this. The series continues with Haiku high vs Sonnet low and Fable 5.2 high vs Opus low.
  • A smoke run of 1 per side ($0.90) came first and isn’t in the table. It ran on the wrong Node version, so npm test failed for the fixture, not for the model. Its results are kept with the run data.

The recipe

  1. If a task is well specified and checkable, try Sonnet with --effort high before reaching for Opus. Here it matched or beat Opus on low effort and cost 16% less.
  2. If you’re waiting on the result, Opus with --effort low came back about 36s sooner at the median, for 75 cents more per 10 runs.
  3. When the output format matters, say it exactly. “End the body with a newline” next to “RFC 4180” left room for \r\n.
  4. Grade with tests the agent never saw. Opus run 2 wrote tests that matched its own choice and passed.

Keep reading

All recipes →
  • Verified

    $2.64 vs $3.28average cost per run, Fable 5.2 (Claude Code 2.1.321 rerun) vs Fable 5.1 (both fixed the bug in every scored run)

    Fable 5.2 is 20% cheaper per token. On our bug fix it averaged $2.64 a run, under Fable 5.1's $3.28

    Launch-day test: Claude Fable 5.2, Fable 5.1 and Opus 5.5 on the same planted bug in undici (902 files), same prompt and tools. Fable 5.2 (rerun on Claude Code 2.1.321) fixed the bug in 3 of 3 and averaged $2.64 a run and 2.37M tokens. Fable 5.1 averaged $3.28 and 2.04M; Opus $0.77 and 1.39M. Small samples: 3, 2 and 3 runs.

    • models
    • cost
  • Verified

    3 of 3 eachruns kept SAVE10 coverage and caught a 9% mutation; $0.013 a run for Haiku 5.5, $0.117 for Sonnet 5.5

    Test cleanup is the job everyone wants to hand off. Can the cheap model do it without quietly deleting coverage?

    We gave Sonnet 5.5 and Haiku 5.5 the same messy test suite: duplicates, skips, a dead import, and one precise SAVE10 assertion that looks redundant. Both kept the coverage and caught a SAVE10→9% mutation in 3 of 3 runs. Haiku averaged $0.013 a run; Sonnet $0.117, about 9× more.

    • models
    • testing
    • cost
  • Verified

    0 of 15runs that started a subagent, on a 902-file repo (all 15 fixed the bug)

    We gave Opus a 902-file repo and a search-heavy bug. It started 0 subagents in 15 runs

    A follower said our last task was too small for Opus to delegate. So we used undici, Node's HTTP client (902 files, pinned commit), and a bug that needs tracing through 4+ files. Sonnet alone, Opus with subagents off and Opus with default subagents all fixed it in 5 of 5 runs, and none of them started a single subagent. Sonnet cost $0.248 a run; Opus $0.776 with subagents available and $0.951 without.

    • subagents
    • models
    • cost

Compare this test with every other one on the scoreboard.