{“content”:”---\nname: systematic-debugging\ndescription: Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.\nversion: 1.1.0\nauthor: Hermes Agent (adapted from obra/superpowers)\nlicense: MIT\nmetadata:\n hermes:\n tags: [debugging, troubleshooting, problem-solving, root-cause, investigation]\n related_skills: [test-driven-development, writing-plans, subagent-driven-development]\n---\n\n# Systematic Debugging\n\n## Overview\n\nRandom fixes waste time and create new bugs. Quick patches mask underlying issues.\n\nCore principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.\n\nViolating the letter of this process is violating the spirit of debugging.\n\n## The Iron Law\n\n\nNO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST\n\n\nIf you haven’t completed Phase 1, you cannot propose fixes.\n\n## When to Use\n\nUse for ANY technical issue:\n- Test failures\n- Bugs in production\n- Unexpected behavior\n- Performance problems\n- Build failures\n- Integration issues\n\nUse this ESPECIALLY when:\n- Under time pressure (emergencies make guessing tempting)\n- “Just one quick fix” seems obvious\n- You’ve already tried multiple fixes\n- Previous fix didn’t work\n- You don’t fully understand the issue\n\nDon’t skip when:\n- Issue seems simple (simple bugs have root causes too)\n- You’re in a hurry (rushing guarantees rework)\n- Someone wants it fixed NOW (systematic is faster than thrashing)\n\n## The Four Phases\n\nYou MUST complete each phase before proceeding to the next.\n\n---\n\n## Phase 1: Root Cause Investigation\n\nBEFORE attempting ANY fix:\n\n### 1. Read Error Messages Carefully\n\n- Don’t skip past errors or warnings\n- They often contain the exact solution\n- Read stack traces completely\n- Note line numbers, file paths, error codes\n\nAction: Use read_file on the relevant source files. Use search_files to find the error string in the codebase.\n\n### 2. Reproduce Consistently\n\n- Can you trigger it reliably?\n- What are the exact steps?\n- Does it happen every time?\n- If not reproducible → gather more data, don’t guess\n\nAction: Use the terminal tool to run the failing test or trigger the bug:\n\nbash\n# Run specific failing test\npytest tests/test_module.py::test_name -v\n\n# Run with verbose output\npytest tests/test_module.py -v --tb=long\n\n\n### 3. Check Recent Changes\n\n- What changed that could cause this?\n- Git diff, recent commits\n- New dependencies, config changes\n\nAction:\n\nbash\n# Recent commits\ngit log --oneline -10\n\n# Uncommitted changes\ngit diff\n\n# Changes in specific file\ngit log -p --follow src/problematic_file.py | head -100\n\n\n### 4. Gather Evidence in Multi-Component Systems\n\nWHEN system has multiple components (API → service → database, CI → build → deploy):\n\nBEFORE proposing fixes, add diagnostic instrumentation:\n\nFor EACH component boundary:\n- Log what data enters the component\n- Log what data exits the component\n- Verify environment/config propagation\n- Check state at each layer\n\nRun once to gather evidence showing WHERE it breaks.\nTHEN analyze evidence to identify the failing component.\nTHEN investigate that specific component.\n\n### 5. Trace Data Flow\n\nWHEN error is deep in the call stack:\n\n- Where does the bad value originate?\n- What called this function with the bad value?\n- Keep tracing upstream until you find the source\n- Fix at the source, not at the symptom\n\nAction: Use search_files to trace references:\n\npython\n# Find where the function is called\nsearch_files(\"function_name(\", path=\"src/\", file_glob=\"*.py\")\n\n# Find where the variable is set\nsearch_files(\"variable_name\\\\s*=\", path=\"src/\", file_glob=\"*.py\")\n\n\n### Phase 1 Completion Checklist\n\n- [ ] Error messages fully read and understood\n- [ ] Issue reproduced consistently\n- [ ] Recent changes identified and reviewed\n- [ ] Evidence gathered (logs, state, data flow)\n- [ ] Problem isolated to specific component/code\n- [ ] Root cause hypothesis formed\n\nSTOP: Do not proceed to Phase 2 until you understand WHY it’s happening.\n\n---\n\n## Phase 2: Pattern Analysis\n\nFind the pattern before fixing:\n\n### 1. Find Working Examples\n\n- Locate similar working code in the same codebase\n- What works that’s similar to what’s broken?\n\nAction: Use search_files to find comparable patterns:\n\npython\nsearch_files(\"similar_pattern\", path=\"src/\", file_glob=\"*.py\")\n\n\n### 2. Compare Against References\n\n- If implementing a pattern, read the reference implementation COMPLETELY\n- Don’t skim — read every line\n- Understand the pattern fully before applying\n\n### 3. Identify Differences\n\n- What’s different between working and broken?\n- List every difference, however small\n- Don’t assume “that can’t matter”\n\n### 4. Understand Dependencies\n\n- What other components does this need?\n- What settings, config, environment?\n- What assumptions does it make?\n\n---\n\n## Phase 3: Hypothesis and Testing\n\nScientific method:\n\n### 1. Form a Single Hypothesis\n\n- State clearly: “I think X is the root cause because Y”\n- Write it down\n- Be specific, not vague\n\n### 2. Test Minimally\n\n- Make the SMALLEST possible change to test the hypothesis\n- One variable at a time\n- Don’t fix multiple things at once\n\n### 3. Verify Before Continuing\n\n- Did it work? → Phase 4\n- Didn’t work? → Form NEW hypothesis\n- DON’T add more fixes on top\n\n### 4. When You Don’t Know\n\n- Say “I don’t understand X”\n- Don’t pretend to know\n- Ask the user for help\n- Research more\n\n---\n\n## Phase 4: Implementation\n\nFix the root cause, not the symptom:\n\n### 1. Create Failing Test Case\n\n- Simplest possible reproduction\n- Automated test if possible\n- MUST have before fixing\n- Use the test-driven-development skill\n\n### 2. Implement Single Fix\n\n- Address the root cause identified\n- ONE change at a time\n- No “while I’m here” improvements\n- No bundled refactoring\n\n### 3. Verify Fix\n\nbash\n# Run the specific regression test\npytest tests/test_module.py::test_regression -v\n\n# Run full suite — no regressions\npytest tests/ -q\n\n\n### 4. If Fix Doesn’t Work — The Rule of Three\n\n- STOP.\n- Count: How many fixes have you tried?\n- If < 3: Return to Phase 1, re-analyze with new information\n- If ≥ 3: STOP and question the architecture (step 5 below)\n- DON’T attempt Fix #4 without architectural discussion\n\n### 5. If 3+ Fixes Failed: Question Architecture\n\nPattern indicating an architectural problem:\n- Each fix reveals new shared state/coupling in a different place\n- Fixes require “massive refactoring” to implement\n- Each fix creates new symptoms elsewhere\n\nSTOP and question fundamentals:\n- Is this pattern fundamentally sound?\n- Are we “sticking with it through sheer inertia”?\n- Should we refactor the architecture vs. continue fixing symptoms?\n\nDiscuss with the user before attempting more fixes.\n\nThis is NOT a failed hypothesis — this is a wrong architecture.\n\n---\n\n## Red Flags — STOP and Follow Process\n\nIf you catch yourself thinking:\n- “Quick fix for now, investigate later”\n- “Just try changing X and see if it works”\n- “Add multiple changes, run tests”\n- “Skip the test, I’ll manually verify”\n- “It’s probably X, let me fix that”\n- “I don’t fully understand but this might work”\n- “Pattern says X but I’ll adapt it differently”\n- “Here are the main problems: [lists fixes without investigation]“\n- Proposing solutions before tracing data flow\n- “One more fix attempt” (when already tried 2+)\n- Each fix reveals a new problem in a different place\n\nALL of these mean: STOP. Return to Phase 1.\n\nIf 3+ fixes failed: Question the architecture (Phase 4 step 5).\n\n## Common Rationalizations\n\n| Excuse | Reality |\n|--------|---------|\n| “Issue is simple, don’t need process” | Simple issues have root causes too. Process is fast for simple bugs. |\n| “Emergency, no time for process” | Systematic debugging is FASTER than guess-and-check thrashing. |\n| “Just try this first, then investigate” | First fix sets the pattern. Do it right from the start. |\n| “I’ll write test after confirming fix works” | Untested fixes don’t stick. Test first proves it. |\n| “Multiple fixes at once saves time” | Can’t isolate what worked. Causes new bugs. |\n| “Reference too long, I’ll adapt the pattern” | Partial understanding guarantees bugs. Read it completely. |\n| “I see the problem, let me fix it” | Seeing symptoms ≠ understanding root cause. |\n| “One more fix attempt” (after 2+ failures) | 3+ failures = architectural problem. Question the pattern, don’t fix again. |\n\n## Quick Reference\n\n| Phase | Key Activities | Success Criteria |\n|-------|---------------|------------------|\n| 1. Root Cause | Read errors, reproduce, check changes, gather evidence, trace data flow | Understand WHAT and WHY |\n| 2. Pattern | Find working examples, compare, identify differences | Know what’s different |\n| 3. Hypothesis | Form theory, test minimally, one variable at a time | Confirmed or new hypothesis |\n| 4. Implementation | Create regression test, fix root cause, verify | Bug resolved, all tests pass |\n\n## Hermes Agent Integration\n\n### Investigation Tools\n\nUse these Hermes tools during Phase 1:\n\n- search_files — Find error strings, trace function calls, locate patterns\n- read_file — Read source code with line numbers for precise analysis\n- terminal — Run tests, check git history, reproduce bugs\n- web_search/web_extract — Research error messages, library docs\n\n### With delegate_task\n\nFor complex multi-component debugging, dispatch investigation subagents:\n\npython\ndelegate_task(\n goal=\"Investigate why [specific test/behavior] fails\",\n context=\"\"\"\n Follow systematic-debugging skill:\n 1. Read the error message carefully\n 2. Reproduce the issue\n 3. Trace the data flow to find root cause\n 4. Report findings — do NOT fix yet\n\n Error: [paste full error]\n File: [path to failing code]\n Test command: [exact command]\n \"\"\",\n toolsets=['terminal', 'file']\n)\n\n\n### With test-driven-development\n\nWhen fixing bugs:\n1. Write a test that reproduces the bug (RED)\n2. Debug systematically to find root cause\n3. Fix the root cause (GREEN)\n4. The test proves the fix and prevents regression\n\n## Real-World Impact\n\nFrom debugging sessions:\n- Systematic approach: 15-30 minutes to fix\n- Random fixes approach: 2-3 hours of thrashing\n- First-time fix rate: 95% vs 40%\n- New bugs introduced: Near zero vs common\n\nNo shortcuts. No guessing. Systematic always wins.\n”}