A 27B model on two home AMD cards passed 15 of 18. Sonnet passed 18.
We ran Qwen3.8-27B on two Radeon AI PRO R9700s against Claude Sonnet 5.5 and Haiku 5.5 on six auto-graded tasks. Sonnet with thinking passed all 18 runs for $0.30. Qwen with thinking passed 15 for $0 in API, and most of its git patches didn't apply.
This is our first local-model test. The question: can a 27B model running on two AMD cards at home keep up with Claude on everyday developer work?
The setup
Local: Qwen3.8-27B (MXFP4) on vLLM 0.27.1, split across two Radeon AI PRO R9700s (32 GB each) in a 7950X box. vLLM also ran a small DFlash2 draft model for speculative decoding, which speeds up output. Each card was capped at 210 W. We ran it with thinking off and with thinking on.
Cloud: Claude Sonnet 5.5, once with thinking set as low as the API allows and once with its default adaptive thinking, plus Claude Haiku 5.5 with thinking disabled.
The tasks: six tasks, each graded by code, with no human judgment:
- Subtle bug. Fix an interval-merging function against hidden edge-case tests (touching intervals, nested intervals, not mutating the input, 300 random cases).
- Multi-file change as a diff. Add a feature across three files and reply with one unified diff. We ran
git applyon it, then hidden tests. A diff that doesn’t apply is a fail. - Algorithm. Peak concurrent bookings with half-open intervals and tricky ties, checked against brute force, then 200,000 bookings in under 2 seconds.
- Long context. An application log of about 8,000 tokens (8,249 by Qwen’s count, 7,968 by Claude’s) with four questions that have exact answers: a request id, a count, a config key and a host.
- Logic puzzle. Six runners, six clues, one valid finishing order.
- Tool calls as JSON. Schedule a meeting and send a chat message. The answer had to pass a JSON schema and pick the right contact, and the time had to account for the end of daylight saving time on Nov 1.
Every model got the identical prompt text with no system prompt, 3 runs per task. That’s 18 runs per setup, run on Oct 20, 2026.
What happened
| Task | Qwen, thinking off | Qwen, thinking on | Sonnet 5.5, thinking off | Sonnet 5.5, adaptive | Haiku 5.5 |
|---|---|---|---|---|---|
| Subtle bug | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| Multi-file diff | 0 of 3 | 1 of 3 | 1 of 3 | 3 of 3 | 0 of 3 |
| Algorithm | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| Long-context log | 0 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 |
| Logic puzzle | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| Tool-call JSON | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| Total | 12 of 18 | 15 of 18 | 16 of 18 | 18 of 18 | 14 of 18 |
| API cost, all 18 runs | $0 | $0 | $0.217 | $0.302 | $0.010 |
Claude costs are from API usage at list prices. The whole comparison cost $0.53. Qwen has no API cost, but both cards sat at their 210 W cap while generating (a median of 210 W each, about 420 W for the pair). Every run’s numbers and grade: raw results (JSON).
Speed
Qwen started answering fastest. Its median time to first token was 0.18 to 0.19 seconds, against 0.62 to 0.85 for Sonnet and 0.60 for Haiku. Claude’s times include the trip over the internet. Ours went through a tunnel on the home network, so neither is a pure model number.
Qwen’s output speed in this pack is inflated. It measured a median of about 240 tokens per second, against 174 to 187 for Sonnet and 289 for Haiku. But the draft model guesses easy, predictable text well, and code and JSON are predictable. Earlier the same day, on 1,000 tokens of free-form prose, Qwen ran at 109 tokens per second (104 to 115 over 3 runs). That’s the number to expect for writing. We didn’t measure Claude on that prose test.
Thinking cost Qwen time. With thinking on, the visible answer started 4.7 to 22.2 seconds after the request (the median for each task). On the diff task Qwen with thinking took 24.8 seconds in total, Sonnet with adaptive thinking 21.8, and Sonnet without thinking 7.5.
On the long log, Qwen’s first, uncached run took 1.73 seconds to the first token. Sonnet without thinking had a median of 1.45 and Haiku 0.60. Qwen’s later runs reused vLLM’s prompt cache, so we don’t count them.
What failed
Diffs that don’t apply, for almost everyone. 10 of the 15 diff runs failed git apply, mostly because the hunk headers had the wrong line counts. In all 10, the code itself was right: once we recounted or ignored the headers, the hidden tests passed. But a broken patch is what you’d actually get if you pasted it into git apply. Only Sonnet with adaptive thinking wrote valid patches 3 of 3 times. Qwen got the logic right in all 6 of its runs and a patch that applied in 1.
Counting without thinking. With thinking off, Qwen said the log had 8 billing errors in all 3 runs. There were 9. With thinking on, it got all four answers right in all 3 runs. Haiku’s one miss was picking the host with 7 “pool exhausted” lines over the one with 8.
A daylight saving slip. In one of three thinking-on runs, Qwen booked the Nov 3 meeting at UTC-5. Daylight saving time ends on Nov 1, so Chicago is UTC-6 that day, and the meeting would land an hour off.
Three tasks didn’t separate anyone. The subtle bug, the algorithm and the logic puzzle were 3 of 3 for every setup.
Caveats
- 3 runs per task. A one-run difference is within noise. The broken diffs and the thinking-off counting miss repeated, so we trust those.
- Sonnet’s “thinking off” isn’t fully off. The lowest setting the API accepts still let Sonnet think for 2,311 tokens in one diff run. That was the one diff Sonnet passed without adaptive thinking.
- Haiku had thinking disabled. Haiku 5.5 doesn’t accept Sonnet’s lowest setting, so we turned thinking off completely, and it used no thinking tokens.
- The diff task is partly about format. It tests whether a model can write a valid unified diff, not just correct code.
- One local setup. One quantization, one runtime, two cards capped at 210 W, not tuned for speed.
What to take from it
- A home 27B model is close on short, well-defined coding and logic tasks. With thinking on, Qwen matched Claude on the bug fix, the algorithm, the puzzle and the long log, at no API cost.
- Turn thinking on for anything that counts or reads carefully. Thinking took Qwen from 12 to 15 of 18, and it fixed the counting miss on the long log.
- Don’t trust pasted diffs from any model. Let the tool apply edits, or run
git apply --checkbefore you rely on a patch. Only Sonnet with adaptive thinking got this right every time. - Pay for Sonnet when the output has to work the first time. It was the only setup with no misses, and the whole pack cost $0.30.