Paste the error, or describe it? Both found the bug; only pasted errors caught the second crash

When something breaks, you can tell Claude Code what you saw (“GET /notes returns a 500 for one user”) or paste the actual error and stack trace. Pasting takes a few extra seconds of digging through logs. We wanted to see whether it changes the fix.

The setup

The same notes API as our plan mode recipe, with one bug added. A PATCH /notes/:id with "tags": null (what a “clear tags” button might send) saves null instead of an empty list. After that, anything that loops over that note’s tags throws. Three places do: the note list, search (which uses the same formatter as the list) and tag counts, which reads the store directly.

So a complete fix has three parts: stop saving null (the root cause), make the list work again for the user whose note is already broken, and make tag counts work for that note too. The existing tests all pass with the bug in place.

The prompt, the same in every run:

One of our users gets a 500 “internal error” from GET /notes. Everyone else is fine. Fix it.

We ran it 3 times each two ways:

• Describe: the prompt as written.
• Paste the error: the same prompt, followed by “Here’s the error from the server log:” and the real stack trace we got by triggering the bug:

TypeError: note.tags is not iterable
    at noteSummary (file:///app/src/utils/format.js:22:20)
    at Array.map (<anonymous>)
    at GET /notes (file:///app/src/routes/notes.js:5:71)
    at file:///app/src/middleware/errors.js:8:14
    at handle (file:///app/src/server.js:37:51)

Both versions started from the same repo, with edits allowed. Afterwards, three hidden tests Claude never saw checked each part of the fix: clearing tags with null stores [], and a note already saved with tags: null no longer breaks GET /notes and search, or tag counts.

What happened

Claude Code 2.1.302, Sonnet 5.5, 3 runs per column. Every run, check and diff: run data.

What we learned

Both versions found the root cause every time. In all six runs Claude read the route, the notes service and the formatter, spotted input.tags && optionalTags(input) in updateNote, and changed it so null becomes []. All six also added a ?? [] guard to the formatter so the already-broken note stops crashing the list. In a codebase this small, the one-line description was enough to get there.

The stack trace sent Claude straight to the crash. Pasted-error runs opened format.js, the top line of the trace, as their first step. Described runs started with a search for “notes” across the repo and worked inward from the route. Turns and time came out about the same (8.3 turns each, 16.3s against 16.0s), so the head start didn’t make the runs shorter.

The difference was the second crash. The trace names the exact failure, note.tags is not iterable. In 2 of 3 pasted-error runs, Claude searched the whole codebase for other code reading tags, found the same loop in tag counts, and guarded it too. The third pasted-error run and all three described runs fixed only the list. For that user, GET /tags/counts would still return a 500. All three described runs said their guard took care of the note already saved with null; one said “no data cleanup is needed”. Tag counts say otherwise.

It cost about half a cent more. Pasted-error runs averaged $0.049 against $0.044, about 11% more. Most of that came from the two runs that made the extra fix.

Nobody checked their own work. The prompt only said “Fix it”, and Claude took that literally. No run added a regression test. Only one of six ran npm test after its last edit, and every other run said in its final message that it hadn’t run the tests. Two pasted-error runs first tried to apply their edits with a chained sed command, which wasn’t allowed in our setup, and switched to the edit tool.

The described runs looked around more. All three described runs noticed an unrelated bug in the same file (dates come out a month behind) and said they’d left it alone. No pasted-error run mentioned it. Neither column touched it.

Six runs is a small sample, and a 2-of-3 against 0-of-3 split could shift with more runs. Before this batch we ran one more set of six with an earlier version of the checks, where one hidden test bundled the list and tag counts together. Every one of those six runs made the same two-line fix (root cause plus the list guard) and none fixed tag counts, in either column. That set is in the repo with the rest of the run data. We also only tried one bug, in a codebase of about 470 lines, where the root cause sits in a different file from the crash. In a bigger codebase, or with a bug the description doesn’t point toward, we’d expect the trace to matter more. We haven’t tested that yet.

The recipe

1. Paste the real error and stack trace when you have one. It costs a few seconds to copy and about half a cent in tokens, and here it was the only version that found the second place the same bug crashed.
2. Ask it to check every other reader of the broken data. The two runs that caught the second crash did it by searching for everything else that reads tags. You don’t have to hope for that; ask for it.
3. Ask for a regression test and a test run. “Fix it” got fixes with no new tests and, in 5 of 6 runs, no test run after the last edit.
4. A clear description still finds a clear bug. If you can’t get the trace, a one-line symptom found the root cause 3 of 3 times on this codebase. Just check the fix covers every path, not only the one you reported.