systematic-debugging

Systematic Debugging

Random fixes waste hours and create new bugs. The rule is absolute: no fix without root cause investigation first. A symptom fix is a failure even when the symptom disappears.

When this applies

Every technical issue. It applies hardest exactly when skipping it is most tempting: emergencies, "obvious" one-liners, and the third attempt after two fixes failed. Systematic is faster than thrashing; that is the whole point.

The four phases

Complete each phase before the next. No proposing fixes during Phases 1-3.

Phase 1: Root cause investigation

1. Read the error completely. Full stack trace, line numbers, error codes. Errors often contain the answer.
2. Reproduce reliably. Exact steps, every time. Not reproducible yet: gather more data, do not guess.
3. Check what changed. git diff, recent commits, new dependencies, config and environment differences.
4. Instrument multi-component systems. When the failure crosses boundaries (CI to build to signing, API to service to DB), log what enters and exits each component, run once, and let the evidence show which layer breaks. Read references/evidence-gathering.md when the failure spans more than one component or sits deep in a call stack.
5. Trace bad values to their source. Where does the bad value originate? What called this with it? Keep walking up. Fix at the source, never at the layer that happened to crash.

Phase 2: Pattern analysis

Find working code similar to the broken code, in this codebase or the reference implementation. Read the reference completely, not skimmed. List every difference between working and broken, including the ones that "can't matter". Note what config, environment, and dependencies the working version assumes.

Phase 3: Hypothesis and test

State one specific hypothesis: "X is the root cause because Y." Make the smallest change that tests it, one variable at a time. Confirmed: go to Phase 4. Not confirmed: new hypothesis. Never stack a second change on an unconfirmed first. If you genuinely do not understand something, say so and research; do not bluff.

Phase 4: Implementation

1. Write a failing test that reproduces the bug (test-driven-development). No test, no fix.
2. Implement one fix targeting the root cause. No bundled refactoring, no "while I'm here".
3. Verify: new test passes, no other test broke, original symptom gone.

The three-strike architecture rule

Count your failed fixes. At three, stop fixing. Three failures where each fix reveals a new problem elsewhere, or each fix needs "massive refactoring", is not bad luck. It is evidence the pattern itself is wrong. Question the architecture with your human partner before attempting fix four. This is not a failed hypothesis; it is a wrong design.

When there is genuinely no root cause

If full investigation shows the issue is environmental, timing-dependent, or external: document what you ruled out, implement appropriate handling (retry, timeout, clear error), and add logging for next time. But treat this verdict with suspicion. Most "no root cause" conclusions are incomplete investigations.

Good vs bad

Bad: Test fails with a timeout. You double the timeout, it passes, you move on. The race condition ships and pages someone at 2 a.m.

Good: Test fails with a timeout. You instrument the async path, find the completion event fires before the listener attaches, fix the ordering at the source, and write a test that fails on the old ordering.

Verification

Before proposing any fix, write the one-sentence root cause: "X breaks because Y, evidenced by Z." Expect all three parts to be concrete, with Z pointing at output you actually collected this session. If you cannot fill in Z, you are still in Phase 1. Go gather the evidence.

Completion checklist

• [ ] Error read fully; reproduction is reliable
• [ ] Root cause stated with evidence, not inference
• [ ] Working-vs-broken differences listed before hypothesizing
• [ ] One hypothesis tested at a time, minimal change each
• [ ] Failing test written before the fix
• [ ] Fix targets the source, all tests pass, symptom gone
• [ ] Fewer than 3 failed attempts, or architecture discussion happened

Any box unchecked: not done. Fix or say so.

Footguns

• Fix stacking. Applying fix two on top of unconfirmed fix one. Now you cannot tell what worked and you have two changes to unwind. Revert to baseline between hypotheses.
• Partial reference reading. Adapting a pattern from a skimmed reference guarantees you missed the part that matters. Read every line of the reference implementation before comparing.
• Confusing symptom relocation with progress. If the error moved somewhere else, you patched a layer, not the cause. Trace further up.

Red flags

Stop and return to Phase 1 when you hear yourself say:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "It's probably X, let me fix that"
• "I don't fully understand it but this might work"
• "Skip the test, I'll manually verify"
• "One more fix attempt" (after two failures)

These phrases are the failure mode, in any wording. Proposing a fix without a written root cause plus evidence is guessing, whatever it is called.