Root-cause-first debugging: reproduce, isolate, hypothesize, then verify, instead of guessing at fixes.
What it does
The usual way a bug gets fixed is by changing things until the error stops. That often works, and it often moves the bug somewhere else instead of removing it, so the same failure returns next week with a different stack trace.
With this skill active, the assistant will not touch the code first. It makes the failure happen on demand, shrinks it to the smallest place it still shows up, writes down one cause it can actually test, and then runs an experiment designed to prove the cause wrong. Only once the evidence lands does it change anything, and only one thing.
What you get back is a short written trail: the symptom, the exact command that reproduces it, the theories it killed along the way, the confirmed cause in a sentence, the fix, and a rerun of the original failure showing it gone. A regression test comes with it, checked to make sure it fails on the old code.
You end up with a confirmed root cause, a fix you can explain, and the original failure rerun to prove it is gone.
When to reach for it
Use it for any bug worth understanding: a crash, a failing or flaky test, a wrong number that never errors, or the classic failure that only happens in production. It earns its keep most when a fix has already been attempted and nobody can explain why it helped.
Skip it for a typo you can see in the stack trace, and skip it in the first ten minutes of an outage, where reverting the bad deploy and investigating afterward is the honest order.
---
name: systematic-debugging
description: Finds the root cause before any code changes - reproduce the failure on demand, isolate it to the smallest place it still happens, state one testable cause, prove or kill it with evidence, then fix and verify against the original reproduction. Use for any bug, exception, failing or flaky test, or behavior that is wrong without erroring.
---
# Systematic Debugging
You are the engineer who refuses to change code before understanding the failure. A fix nobody can explain is a guess, and guesses relocate bugs rather than remove them.
## Workflow
The order matters. Each step produces the evidence the next one needs.
1. **Collect the symptom.** Exact observed behavior versus expected, the full error text and stack trace, the environment, and what changed recently.
2. **Reproduce.** Make the failure happen on demand. Write down the command and conditions that trigger it.
3. **Isolate.** Shrink the space until the smallest input, test, or code path still fails.
4. **Hypothesize.** One specific, testable mechanism. Written down before you look for evidence.
5. **Test the hypothesis.** Run an experiment whose result would kill the hypothesis if it is wrong. Let the result decide.
6. **Fix the cause.** The smallest change that removes the mechanism, not the symptom.
7. **Verify.** Rerun the reproduction, run the neighbors, add a regression test.
```text
Debugging checklist
[ ] Full error text and stack trace read, not paraphrased
[ ] Failure reproduced on demand, command written down
[ ] Reproduction shrunk to the smallest failing case
[ ] One hypothesis written before evidence gathering
[ ] Experiment run that could have killed the hypothesis
[ ] Fix addresses the confirmed mechanism, one change
[ ] Original reproduction rerun and now passes
[ ] Neighboring tests run, regression test added
```
## 1. Collect the symptom
Ask for, or go and find:
- The exact error text and full stack trace. A paraphrase loses the line that matters.
- What was expected instead, in one sentence.
- Steps, inputs, user, tenant, environment, browser, and time of day if it seems to matter.
- The last known good state: which commit, release, or day it worked.
- Frequency: always, or roughly how often.
"It doesn't work" is not a symptom. Push once for a specific observation before starting.
## 2. Reproduce
If you cannot make it fail on demand, you cannot prove you fixed it. Everything downstream depends on this step.
- Start from the reported steps exactly. No improvements yet.
- If it does not fail, close the gap between your environment and theirs one variable at a time: data, config, timezone, locale, feature flag, account, browser, node version.
- For intermittent failures, find the knob that changes the odds: a loop of 100 runs, a fixed random seed, a throttled network profile, a cold cache. A reliable 1-in-5 is a reproduction; a shrug is not.
- Save the trigger as a command you can paste. `pytest tests/billing/test_totals.py --randomly-seed=48213` beats "run the suite a few times".
If the failure resists reproduction after a genuine attempt, stop and add observability instead: structured logs at the boundary, a request id through the stack, an error report with the payload attached. Say plainly that you are collecting evidence rather than fixing.
## 3. Isolate
Shrink until the failing surface is small enough to hold in your head.
- **Binary search the code path.** Disable, stub, or short-circuit half the pipeline. Whichever half still fails contains the cause. Repeat.
- **Binary search history.** If it used to work, `git bisect` between the last good and first bad commit. This is the highest-yield tool available and people forget it exists.
- **Shrink the input.** Remove rows, fields, characters until removing anything more makes the failure disappear. What remains is the trigger.
- **Compare working and broken.** Same code, one works. Diff the two: environment variables, data, versions, permissions, order of execution.
- **Read the stack trace properly.** Bottom frames are your code; top frames are where it blew up. The interesting frame is usually the last one you own.
Write down what you ruled out. It stops you re-checking it at hour three.
## 4. Hypothesize
State one mechanism, specific enough to be wrong:
- Good: "The list renders empty because the filter runs before the fetch promise resolves."
- Good: "Both tests mutate the module-level rules dict, so order decides the total."
- Too vague to test: "Something is wrong with the caching."
One hypothesis at a time. If you are holding three, pick the one that is cheapest to kill and start there.
## 5. Test the hypothesis
Design the experiment so a wrong hypothesis actually fails:
- Add a log or breakpoint that prints the value your hypothesis predicts.
- Force the condition: pin the clock, seed the random, run the two tests in the suspect order, set the flag by hand.
- Change one variable per run. Two changes make the result unreadable.
- If the experiment confirms nothing either way, the experiment was wrong. Redesign it before touching the hypothesis.
When a hypothesis dies, write down what its death rules out, then form the next one. Do not stack guesses on top of a dead theory.
## 6. Fix the cause
- Make the smallest change that removes the confirmed mechanism.
- If the real fix is large and the situation is urgent, mitigate deliberately: revert, flag off, or patch the symptom, and say out loud that this is a mitigation and the cause is still there.
- Resist the tidy-up urge. Refactoring in the same change makes the verification ambiguous.
## 7. Verify
- Rerun the exact reproduction command. The failure must be gone.
- For intermittent bugs, rerun enough times to beat the original odds by a wide margin. A 1-in-5 bug needs far more than five clean runs.
- Run the surrounding tests to check you did not break a neighbor.
- Add a regression test that fails on the old code and passes on the new one. Confirm it fails on the old code, otherwise it tests nothing.
## Report format
```text
Symptom: what was observed, in one line
Reproduction: the command or steps that fail reliably
Ruled out: hypotheses killed, and what killed them
Root cause: the confirmed mechanism, one or two sentences
Fix: the change, and why it removes the mechanism
Verification: reproduction result, test results, regression test added
```
## Example
**Symptom.** `test_invoice_totals` passes locally, fails in CI roughly one run in three with `AssertionError: 1200 != 0`.
**Reproduce.** The CI log printed the test-ordering seed. Running `pytest tests/billing --randomly-seed=48213` locally failed on the first try, every time. That turned "sometimes in CI" into a one-line command.
**Isolate.** Ran the first half of the ordered suite, then the second. The failure followed a pair: `test_apply_discount` running before `test_invoice_totals`. Either test alone passed.
**Hypothesis 1 (killed).** The CI database is not reset between tests. Experiment: ran the failing pair against a freshly created database. Still failed. Database state is not involved, which also rules out the fixture teardown.
**Hypothesis 2.** Both tests read a module-level `TAX_RULES` dict, and the discount test mutates it without restoring it.
**Test.** Printed `TAX_RULES` before and after `test_apply_discount`. It gained a `{"region": "EU", "rate": 0}` entry that was never removed, and the totals test matched that zero-rate rule first.
**Fix.** The discount test now takes a fixture that deep-copies the rules and restores them in teardown. Two lines, no production change, because the production path never mutates the dict.
**Verify.** Seed 48213 passes. Twenty random seeds pass. Added `test_apply_discount_leaves_rules_unchanged`, confirmed it fails on the old fixture and passes on the new one.
## Rules
- No shotgun debugging. Changing several things at once and seeing green teaches you nothing about which one mattered.
- Reproduce before fixing, verify after fixing. Both, every time.
- If a fix works and you cannot explain why, you have not found the cause. Keep going or say so explicitly.
- Read the actual error before theorizing. The answer is in the trace more often than anyone admits.
- Never delete or weaken a failing test to make it pass. If the test is genuinely wrong, fix the test as its own change with its own reasoning.
- Timestamp and keep the debugging notes. The second person to hit this bug is usually you, six months later.
## Edge cases
- **Cannot reproduce at all.** Do not guess a fix. Add logging, capture a real payload, and say the investigation is blocked on evidence.
- **Flaky test.** Treat flakiness as the bug. Find the shared state, the real clock, the network call, or the ordering assumption. Quarantining without a cause is a decision to fix it later, so label it as one.
- **Heisenbug.** If it disappears under the debugger or with logging added, suspect timing, buffering, or optimization. Switch to recording evidence that does not change the timing, such as ring-buffer logs flushed after the failure.
- **The bug is in a dependency.** Prove it with a minimal script that uses only the dependency. Then pin, patch, or work around it, and link the upstream issue in a comment.
- **Two bugs at once.** If the evidence contradicts itself, you probably have two mechanisms. Split them: fix the one you can prove, then re-reproduce and start over on the remainder.
- **Production is on fire.** Mitigate first, investigate second, and keep the artifacts you need for the investigation before they roll off: logs, a copy of the payload, the deployed commit.
- **Someone already suggested a fix.** Do not apply it until you can state the mechanism it removes. A plausible fix with no confirmed cause is still a guess.
## Files
- `references/examples.md` - three complete bug hunts, each with the killed hypotheses shown: an intermittent empty list, an off-by-one-day date bug, and a 500 that affected one account. Read it to see how much evidence a step should produce before you move on.
- `references/techniques.md` - the isolation toolkit in detail: `git bisect` on a script, input shrinking, differential comparison, logging that does not lie, and running down flaky tests. Read it when the isolate step stalls.
references/examples.md
# Worked bug hunts
Three investigations end to end, including the hypotheses that turned out to be wrong. The dead ends are the point: they are what a real hunt looks like, and each one narrows the search.
## Contents
- [Hunt 1: the list is empty about one time in five](#hunt-1-the-list-is-empty-about-one-time-in-five)
- [Hunt 2: invoices dated one day early for some customers](#hunt-2-invoices-dated-one-day-early-for-some-customers)
- [Hunt 3: 500 on the orders page for exactly one account](#hunt-3-500-on-the-orders-page-for-exactly-one-account)
- [Patterns across all three](#patterns-across-all-three)
---
## Hunt 1: the list is empty about one time in five
### Symptom
Support reports that the dashboard sometimes shows "No projects yet" for users who definitely have projects. Reloading fixes it. No error in the console, no failed request in the network tab, nothing in the server logs.
### Reproduce
Reported steps did nothing on the first ten reloads. The knob turned out to be network speed: with the browser throttled to "Fast 3G", the empty state appeared on roughly two reloads in three.
```
Reproduction: load /dashboard with network throttled to Fast 3G, hard reload.
Rate: ~65% empty. Unthrottled: ~5%.
```
Throttling did not create the bug. It changed the odds, which is exactly what a reproduction for a timing bug needs to do.
### Isolate
Added a temporary log line at three points: request start, response received, list render.
```
[12:04:41.220] fetch start /api/projects
[12:04:41.223] render list items=0
[12:04:41.901] fetch done items=14
```
Render fires 678ms before the data arrives, and never fires again. That places the failure in the component, not in the API.
### Hypothesis 1 (killed)
*The API returns an empty array on cold cache and the component is correct.*
Experiment: replayed the same request with `curl` twenty times against a cold cache. Every response had 14 items. Killed. The server is not involved, which also rules out the caching layer and the query.
### Hypothesis 2 (confirmed)
*The component renders once on mount with the initial empty state and never re-renders, because the effect that sets the data writes to a variable the render does not depend on.*
Experiment: logged the state setter and the render count.
```
render #1 items=0 loading=false
setData(14) <- called
(no render #2)
```
Confirmed. `loading` was initialized to `false`, so the empty state was shown immediately, and the data was stored on a ref rather than in state, so the update never triggered a render.
### Fix
Initialize `loading` to `true`, hold the fetched rows in state, and render the empty state only when loading has finished and the array is genuinely empty.
```diff
-const rows = useRef([]);
-const [loading, setLoading] = useState(false);
+const [rows, setRows] = useState([]);
+const [loading, setLoading] = useState(true);
```
### Verify
Thirty throttled reloads, zero empty states. Twenty unthrottled, zero. Added a test that mounts the component with a promise that resolves after a tick and asserts the empty state is absent while loading is true. Confirmed the test fails against the old component.
### Report
```text
Symptom: dashboard shows the empty state despite existing projects, ~1 in 5 loads
Reproduction: /dashboard with Fast 3G throttling, hard reload (~65% failure)
Ruled out: API returning empty (20 cold-cache curl calls, all 14 items)
Root cause: empty state rendered while the fetch was still in flight; rows lived on a
ref, so the resolved data never triggered a re-render
Fix: rows moved into state, loading initialized to true, empty state gated on
loading === false
Verification: 30 throttled + 20 unthrottled reloads clean; regression test added and
confirmed failing on the old code
```
---
## Hunt 2: invoices dated one day early for some customers
### Symptom
Finance reports that a handful of December invoices show `2025-11-30` on the PDF while the database row says `2025-12-01`. No error anywhere. Most invoices are fine.
### Reproduce
The affected customers had one thing in common in the export: all of them were in the Americas. Setting the local timezone to `America/Los_Angeles` and regenerating any December invoice reproduced it on the first try.
```
TZ=America/Los_Angeles npm run invoice:render -- --id 88213 # shows 2025-11-30
TZ=Europe/Berlin npm run invoice:render -- --id 88213 # shows 2025-12-01
```
Same row, same code, different output. That is a differential comparison and it names the variable immediately.
### Isolate
The date passes through three layers: a `DATE` column, a serializer, and the template. Printed the value at each boundary.
```
db row : 2025-12-01 (type DATE, no time, no zone)
after serialize: 2025-12-01T00:00:00.000Z
in template : Sun Nov 30 2025 16:00:00 GMT-0800
```
The corruption happens between the serializer and the template.
### Hypothesis 1 (killed)
*The database is storing the wrong day because the writing service runs in a different zone.*
Experiment: queried the row directly with `psql`. It reads `2025-12-01`. Killed, and worth killing early, because a storage bug would have meant a data migration rather than a code change.
### Hypothesis 2 (confirmed)
*A calendar date with no time is being turned into an instant at UTC midnight, and then formatted in the local zone, which is behind UTC and therefore lands on the previous day.*
Experiment: in a node REPL with `TZ=America/Los_Angeles`:
```js
new Date('2025-12-01T00:00:00.000Z').toLocaleDateString() // '11/30/2025'
```
Confirmed in one line. The bug is not in our code so much as in the decision to represent a calendar date as an instant.
### Fix
Keep the date as a plain string through the whole path and format it with an explicit UTC formatter in the template. No `Date` object is constructed for a value that has no time component.
```diff
-const d = new Date(invoice.issued_on);
-return d.toLocaleDateString('en-US');
+return formatCalendarDate(invoice.issued_on); // parses YYYY-MM-DD, no zone math
```
### Verify
Regenerated all 214 December invoices under four zones: `America/Los_Angeles`, `America/New_York`, `UTC`, `Asia/Tokyo`. All show the stored date. Added a test that runs the formatter under a fixed `TZ=America/Los_Angeles` and asserts `2025-12-01` renders as December 1. Confirmed it fails on the old code.
### Report
```text
Symptom: some invoice PDFs show the day before the stored issue date
Reproduction: TZ=America/Los_Angeles npm run invoice:render -- --id 88213
Ruled out: wrong value in the database (verified directly with psql)
Root cause: a calendar date was parsed into a UTC instant and then formatted in the
local zone; any zone behind UTC renders the previous day
Fix: date kept as a YYYY-MM-DD string and formatted without zone conversion
Verification: 214 invoices re-rendered under 4 zones, all correct; zone-pinned
regression test added and confirmed failing on the old code
```
---
## Hunt 3: 500 on the orders page for exactly one account
### Symptom
One customer gets a 500 on `/orders` every time. Every other account works. The error report shows `TypeError: Cannot read properties of null (reading 'toUpperCase')` with a stack ending in `formatMoney`.
### Reproduce
Impersonated the account in staging against a copy of production data. Reproduced on the first load, every load. A reliable, account-specific reproduction is the easiest kind to work with, because the difference is in the data.
### Isolate
`formatMoney` receives `(amount, currency)` and the stack says `currency` is null. Ran the account's orders through the serializer in a script and printed any row with a null currency.
```sql
SELECT id, created_at, currency FROM orders
WHERE account_id = 4471 AND currency IS NULL;
id | created_at | currency
---------+-------------------------+----------
118_402 | 2019-03-14 09:22:11+00 | NULL
```
One row out of 1,206, created in 2019.
### Hypothesis 1 (killed)
*A recent deploy started writing null currencies.*
Experiment: checked for null currencies across all accounts by creation month. Every null predates the 2019 currency migration; there are eleven in total and none from the last six years. Killed. This is old data, not an active writer, so the fix does not need to be shipped in the next ten minutes.
### Hypothesis 2 (confirmed)
*The 2019 migration backfilled `currency` from the account's billing profile, and rows belonging to accounts that had no profile at migration time were left null. The serializer has assumed the column is non-null ever since.*
Experiment: checked the migration script in git history. It joined on `billing_profiles` with an inner join. The eleven null rows all belong to accounts created before billing profiles existed. Confirmed by matching account ids.
### Fix
Two changes, deliberately separate:
1. Code: `formatMoney` treats a null currency as the account's current currency and logs a warning with the order id. One order rendering slightly wrong beats a whole page failing.
2. Data: a backfill for the eleven rows, using the account's earliest known currency, run as its own reviewed migration.
Then the column gets a `NOT NULL` constraint so the assumption is enforced instead of assumed.
### Verify
Impersonated the account: page loads, all 1,206 orders render, the warning fires once for order 118402. After the backfill the warning stops. Query for null currencies returns zero rows, and the constraint is in place. Added a serializer test with a null currency fixture.
### Report
```text
Symptom: /orders returns 500 for account 4471 only; TypeError in formatMoney
Reproduction: impersonate account 4471 in staging, load /orders (fails every time)
Ruled out: a recent writer creating nulls (all 11 nulls predate the 2019 migration)
Root cause: the 2019 currency backfill used an inner join on billing_profiles, so
orders from accounts with no profile kept a null currency; the serializer
has assumed non-null since
Fix: formatMoney falls back to the account currency and warns; 11 rows
backfilled in a separate migration; NOT NULL constraint added
Verification: page loads with all 1,206 orders; null-currency query returns 0 rows;
serializer test with a null fixture added
```
---
## Patterns across all three
1. Every hunt produced a one-line reproduction command before anyone read the suspect code.
2. Every hunt killed at least one plausible hypothesis, and each death shrank the search: not the server, not the database, not a recent deploy.
3. The confirming experiment was tiny in all three cases: three log lines, one REPL expression, one SQL query.
4. The fix came after the mechanism could be stated in a single sentence.
5. Each ended with a regression test that was confirmed to fail on the old code. A regression test that passes both ways is decoration.
references/techniques.md
# Isolation toolkit
What to reach for when the failure reproduces but you cannot see where it lives.
## Contents
- [git bisect, including the automated form](#git-bisect-including-the-automated-form)
- [Binary search inside the code path](#binary-search-inside-the-code-path)
- [Shrinking the input](#shrinking-the-input)
- [Differential comparison](#differential-comparison)
- [Logging that does not lie](#logging-that-does-not-lie)
- [Running down a flaky test](#running-down-a-flaky-test)
- [When the bug hides from you](#when-the-bug-hides-from-you)
## git bisect, including the automated form
The fastest tool available when the code used to work, and the one people forget.
```bash
git bisect start
git bisect bad # current commit fails
git bisect good v2.14.0 # this release worked
# git checks out the midpoint; test it, then:
git bisect good # or: git bisect bad
# repeat until git names the first bad commit
git bisect reset
```
With a scripted check it runs unattended. The script must exit 0 for good and non-zero for bad:
```bash
git bisect start HEAD v2.14.0
git bisect run pytest tests/billing/test_totals.py -q
```
Notes that save time:
- 1,000 commits take about 10 steps. It scales far better than reading diffs.
- If some commits do not build, exit 125 from the script and bisect skips them.
- Bisect finds the commit that introduced the *symptom*. That is often the commit that exposed an older latent bug, so read the diff rather than reverting on sight.
## Binary search inside the code path
When history is not the axis, cut the pipeline instead.
- Comment out, stub, or feature-flag off the back half of the flow. Still failing? The cause is in the front half. Repeat.
- For a chain of transforms, assert the shape of the data between each pair of stages. The first assertion that trips is your boundary.
- For a page that fails to render, delete components from the tree until it renders. The last one removed owns the failure.
Keep a written list of what is currently disabled. Debugging sessions end with mystery breakage far more often from forgotten stubs than from the original bug.
## Shrinking the input
If a specific payload, file, or record triggers it, make it smaller.
1. Cut the input in half. Test the halves.
2. Keep the failing half, cut again.
3. Stop when removing any single field, row, or character makes the failure disappear.
The remainder is a minimal reproduction and usually names the cause outright: one emoji in a name field, one row with a null, one line ending that differs.
For generative bugs, a property-based testing library will shrink the failing case automatically. That is the same technique with a machine doing the cutting.
## Differential comparison
Two environments, one works. The cause is in the difference, so enumerate it rather than theorizing about it.
Check, in rough order of how often each is the answer:
- Data. Different rows, different tenant, different volume.
- Environment variables and config, including the ones with defaults.
- Timezone and locale. `TZ=` and `LC_ALL=` reproduce an astonishing number of "works for me" bugs.
- Dependency versions, including transitive ones. Compare the lockfiles, not the manifests.
- Runtime version and platform. Node 18 versus 22, glibc versus musl, arm64 versus x86.
- Feature flags and permissions for the specific user.
- Clock: real time versus a frozen test clock, and machines with slow disks.
Change one difference at a time until the working side breaks or the broken side works. Either direction identifies the variable.
## Logging that does not lie
- Log values, not milestones. `items=0 loading=false` beats "about to render".
- Include a request or run id in every line so you can follow one path through interleaved output.
- Log at boundaries: entering a function with its arguments, leaving with its return value. Most bugs live at a boundary where two components disagree about the shape of the data.
- Print types alongside values when the language allows it. `"5"` and `5` look the same in a log and behave differently.
- Timestamp with milliseconds. Ordering bugs are invisible at second resolution.
- Remove the temporary lines before committing, or promote them to real structured logs on purpose.
## Running down a flaky test
Flakiness is a bug with a probability attached, not a personality trait.
1. Get the odds up. Run it in a loop until you have a failure rate you can measure: `for i in $(seq 50); do pytest tests/x.py -q || echo "FAIL $i"; done`.
2. Check ordering. Most test frameworks can randomize or pin order. If a seed reproduces it, you have shared state between tests.
3. Suspect, in this order: shared mutable state, real clock or timezone, real network, unawaited async work, and test data that collides.
4. Run the suspect pair alone. Two tests reproducing it is a small enough surface to read directly.
5. Fix the shared thing. Adding a retry or a sleep hides the failure and keeps the cause.
## When the bug hides from you
If it disappears under a debugger, with logging added, or in a slower build, the timing itself is part of the mechanism.
- Record instead of pausing: buffer events in memory and dump them after the failure.
- Reproduce under the conditions that made it appear, not the conditions that make it convenient: same build flags, same concurrency, same throttling.
- Suspect optimization, buffering, and race windows that your instrumentation widens or closes.
- If the failure needs concurrency, force the interleaving deliberately with a lock or a deterministic scheduler rather than hoping to catch it.
Install it
Claude Code (this project)
Unzip the download into .claude/skills/. The folder is already named right.
Claude Code (every project)
Unzip into ~/.claude/skills/ to use it everywhere.
claude.ai
Upload the zip under Settings, then Capabilities, then Skills (paid plans).
How to call it
Just describe the task. The description line inside the skill does the matching. Or call it directly:
/systematic-debugginghelp me find the root cause of this bugthis test only fails sometimesit works locally but breaks in productionI fixed it but I don't understand why it workswalk me through debugging this properly
Works beyond Claude
Skills use the open Agent Skills format. The exact same folder works in OpenAI Codex, Gemini CLI, Cursor, GitHub Copilot and other tools that read the format.
Using a plain chat app? Paste SKILL.md into ChatGPT or Gemini as instructions. You lose auto-triggering, but the method still works.
