Haiku was enough
We gave all three models a checkout feature with coupons, tax and edge cases. All nine runs passed every hidden test. Haiku averaged under a cent; Opus cost 25 times more.
Most people pick one model and use it for everything. We wanted to know what you actually get for the price on an ordinary, well-specified coding task.
The setup
The same small cart module as our CLAUDE.md recipe, with the same short CLAUDE.md, so every model knew the house rules. The prompt asks for a real feature with several edge cases (quoted exactly):
Add a checkout(items, { coupon } = {}) function to the cart module. It returns { subtotal, discount, tax, total }. Rules: items with qty 0 or less are ignored. Coupons are case-insensitive and ignore surrounding spaces: ‘SAVE10’ takes 10% off the subtotal, ‘FLAT5’ takes $5 off but never more than the subtotal, an empty or missing coupon means no discount, and any other code throws an Error whose message is ’Unknown coupon: ’ followed by the trimmed code. Tax applies to the subtotal after the discount. subtotal, discount and tax are each rounded to cents, and total is subtotal - discount + tax. Add tests.
After each run we copied in hidden tests Claude never saw, covering every rule in the prompt.
What happened
| Haiku 5.5 | Sonnet 5.5 | Opus 5.5 | |
|---|---|---|---|
| Hidden tests pass (all edge cases) | 3 of 3 | 3 of 3 | 3 of 3 |
npm run check passes |
3 of 3 | 3 of 3 | 3 of 3 |
| Added its own tests | 3 of 3 | 3 of 3 | 3 of 3 |
| Followed the CLAUDE.md rules | 3 of 3 | 3 of 3 | 3 of 3 |
| Average cost per run | $0.009 | $0.106 | $0.221 |
| Average turns | 14.7 | 14.3 | 12 |
| Average time | 28s | 27s | 42s |
Claude Code 2.1.296, 3 runs per model. Costs are as Claude Code reports them, at list prices. Every run, check and diff: run data.
What we learned
On a clear task, all three were equally right. Every run passed every hidden test, kept the generated file untouched, added a changelog line and ran the checks before finishing. Nothing in the results separates the models on quality.
The price gap was huge. Haiku averaged under a cent per run. Sonnet cost about 12 times more, and Opus about twice Sonnet, or 25 times Haiku.
Haiku worked harder for it. It wrote about 50% more output tokens than Sonnet and had more shell commands refused by the permission checks (3 or 4 per run, against 2 to 4 for Sonnet and 1 for Opus). It still finished in about the same time, and the price per token is low enough that the extra work barely shows.
Opus was the most direct, and the slowest. It took the fewest turns and hit the fewest dead ends, but each turn took longer, so it finished about 15 seconds behind the others.
This was an easy test to pass. The prompt spelled out every rule. We haven’t yet tested vague requests, bugs with no clear cause or changes across many files, which is where the bigger models are supposed to earn their price.
The recipe
- Try Haiku first for well-specified tasks. If you can write the rules down as clearly as our prompt, Haiku is likely enough. Switch models in a session with
/model. - Write the spec, not just the request. Our prompt listed every edge case. We think that clarity is what let the cheapest model match the most expensive one, though we haven’t yet run the same task with a vague prompt.
- Keep a test the model didn’t write. Our hidden tests are what made the comparison trustworthy. In your project, that’s your existing test suite plus a review.
- Move up a tier when the task gets fuzzy, not by default. Our working assumption is that bigger models earn their price on ambiguity, not typing. That’s the next thing we’ll test.