Problem: Weird bugs you can't crack? It's not bad luck — turn debugging from guessing into a chain of evidence —— Trove problem library

Symptom

Let me start with an afternoon I still remember. A colleague came over: "Google opens fine, but that streaming site won't load — is it down?" My first instinct was "yeah, the site is down.

" But I asked one more question: same computer, same browser, same moment — one domain works, one doesn't.

If the site were down, why wouldn't everything fail? That "why" is where debugging starts: most "weird bugs" aren't actually weird — we just mistook the symptom for the fact.

You've seen the pattern: the service works after a restart, then dies again days later, so you blame it on "flakiness"; an endpoint times out occasionally, so you blame "network jitter"; only one machine gets rate-limited, so you blame "traffic spikes".

Then you start guessing: tweak config, restart, raise timeouts, tune parameters...

If you're lucky you stumble onto it; if not, you burn a whole night, and the next day it "fixes itself" — which only convinces you it's black magic.

The most typical cautionary tale I've seen: an exam-registration system that kept dying during peak signup. Ops restarted it and it recovered; a few days later it died again.

This went on for almost a month — until someone correlated the failure timestamps with user actions: every outage happened right after someone uploaded a ZIP attachment.

They pulled a sample and found a malicious ZIP bomb: files were being decompressed in memory, blowing up the Java heap. If they had correlated "failure time" with "user action" a month earlier, the whole saga could have been a half-hour fix.

Then there was the multi-DC switchover where the rate-limit alert fired. Everyone's first thought: "traffic spiked.

" But when we compared the anomaly window against the normal baseline, the request count at 19:16 was about 60 — nowhere near the 400+ peak of a normal hour.

And the limit only fired on one machine. Traffic hadn't spiked; the other machines were fine — the rate limit was a symptom, not the cause. Digging into that one box revealed GC pauses stalling request handling.

If we had followed the "traffic spike" instinct and scaled out, it would have been scaling for nothing.

If you've been through "restart therapy", "config superstition", or a night of doubting your own sanity — this article is for you. It's not a tutorial for one bug.

It's a repeatable chain of evidence, distilled from multiple independent incidents: layer first, collect evidence, find the variable, compare baselines, grab a sample.

Do this and most of the "weird" in weird bugs disappears.

Cause

Why is our first instinct "guess" instead of "investigate"?

First, human attribution runs on "nearest association", not "system analysis". A site won't load → nearest association: "the site is down". An endpoint times out → "the network is bad". A restart fixes it → "it's flaky".

This isn't stupidity; it's the brain taking a shortcut by default.

But system failures are layered: a request passes DNS resolution, TCP connect, TLS handshake, HTTP, application logic, database queries — a fault at any layer looks like "won't load", "timeout", "error" from the outside.

Same symptom, different layer, completely different fix. Guessing means betting on one random layer out of seven.

Second, we conflate "the failure" with "the root cause". The rate-limit alert is a failure, but rate limiting is usually an effect, not the cause — the real cause might be GC, thread pools, connection pools, or a slow query.

Handling the alert as if it were the problem is like treating a fever with only antipyretics while the infection rages on. The hard part of debugging was never "fixing" — it's peeling the effect back to the cause.

Third, we're in a hurry to "fix it" and don't want to "collect evidence". Restart is the enemy of evidence: it destroys the scene.

Many weird bugs stay weird because every time they were "cured" by a restart or retry, and the evidence never got a chance to exist. Debugging is like detective work — once the scene is gone, the case is gone.

Each tip below pulls debugging back from "guessing" to "chain of evidence". They aren't one author's opinion; they're consensus repeatedly validated across independent incident postmortems.

Solution

Each experience below follows the same shape: phenomenon → practice → why. None of it is one person's opinion — these are patterns that recur across many independent practices.

Tip 0 (L1 one-liner): Five steps — layer, evidence, variable, baseline, sample

For a weird bug, go in order: ①split the symptom into a chain and eliminate layer by layer (DNS/TCP/TLS/HTTP/app/database); ②back every conclusion with a reproducible test, and phrase it precisely enough to be falsified; ③find the "variable" and the "trigger" — a fault is always caused by some change; recovers-after-restart-but-returns = triggered fault; ④compare the anomaly window against a normal baseline to rule out "obvious" assumptions; ⑤reproduce, grab a sample, then fix. In a hurry?

Read only this. Tips 1–5 explain why and how.

Tip 1: Layer first — split the "symptom" into a "chain"

Phenomenon: A site won't load; most people's first thought is "the site is down". But "one domain won't load" is just the surface — the fault could live in DNS, TCP, TLS, HTTP, CDN, a proxy, or even a browser extension.

In one real investigation I saw the whole chain: Google worked, a streaming site didn't.

Layer by layer: nslookup showed resolution failing; curl --resolve bypassed DNS and hit the CDN directly with HTTP 200 — the fault was pinpointed to "this network's DNS resolution chain", not the site.

Practice: ①draw the chain first: client → DNS → TCP → TLS → HTTP → app → database, each layer with its own verification command; ②probe each layer with tools: nslookup/dig for resolution, curl -v to see which stage it dies at (before Connected to = DNS/TCP;

after SSL connected = TLS/HTTP), curl --resolve to bypass one layer and validate the next, ping/traceroute for routing; ③record both "layer OK" and "layer failed" — when elimination is done, the remaining layer is the suspect.

Why: Systems are layered and symptoms are not facts — "won't load" is only the outermost signal, and the true layer dictates a completely different fix (DNS config vs certificate vs app tuning).

Probing layer by layer isn't slow; it turns "guessing across seven layers" into "checking one".

Tip 2: Evidence-chain thinking — conclusions can be narrow, but they must be solid

Phenomenon: At the end of the DNS investigation, the conclusion was "this network's DNS resolution chain is abnormal". Someone asked: "So the ISP is blocking it, right?" — No.

Client-side testing can only prove the fault lives in the DNS resolution layer; whether the router has a cache bug, upstream DNS failure, or filtering rules cannot be uniquely determined from client evidence.

Narrowing the conclusion isn't being conservative; it's being honest: a narrow conclusion can be overturned or confirmed by the next piece of evidence; a broad one ("ISP blocking") stops the investigation dead.

Practice: ①attach "how I tested, what command reproduces it" to every conclusion; ②grade your wording: "abnormal" ≠ "blocked", "suspected" ≠ "confirmed", "this network" ≠ "everywhere"; ③self-check before finalizing: what evidence could overturn this sentence?

If nothing could, it's not a conclusion — it's a guess.

Why: The biggest risk in debugging isn't "not knowing"; it's "knowing wrong" — a plausible-sounding wrong conclusion ends the investigation ("ISP blocks it, nothing we can do"), while a precise narrow conclusion drives the next round of evidence collection.

The end of an evidence chain is "reproducible", not "sounds right".

Tip 3: Find the "variable" and the "trigger" — don't believe in "flakiness"

Phenomenon: The exam system kept dying; restarts fixed it — that word "flaky" is a killer. Until someone correlated failure timestamps with user actions: every outage happened right after a ZIP upload.

Following that variable, they pulled a sample and found a ZIP bomb (a tiny compressed file that expands to massive repetitive data, exploding the Java heap in memory).

A month-long problem, located in half an hour.

Practice: ①the moment a fault happens, record three things: time, the most recent change (deploy/config/action), and scope (single machine vs everywhere); ②"restart fixed it" is not a conclusion — it's a clue; recovers-after-restart-but-returns = triggered fault, and a trigger condition exists that hasn't been found yet;

③draw the "failure timeline" and the "action timeline" side by side and look for overlaps; ④before calling something "flaky", ask: have I recorded the context of three consecutive occurrences? No records, no right to say "flaky".

Why: There is no "flakiness" in a computer system — same input, same state, same output, every time. So-called flakiness is just a trigger condition not yet found. And triggers are almost always "variables": a deploy, an action, a special input.

Finding variables is faster than finding root causes, because variables are visible and comparable.

Tip 4: Compare baselines, narrow the scope — eliminate the "obvious"

Phenomenon: After a multi-DC switchover, the rate-limit alert fired. First instinct: "traffic spiked". Comparing the anomaly window's request curve against the normal baseline: 19:16 saw ~60 requests; the normal peak is 400+. Traffic hadn't spiked.

And the limit fired on only one machine.

Two comparisons eliminated "traffic spike"; the problem was pinned to a single box — GC pauses. If we had followed the instinct and scaled out, it would have been scaling for nothing.

Practice: ①for any "because A, therefore B" attribution, find a control: anomaly vs baseline, failed box vs healthy box, before vs after; ②let data talk: pull curves, logs, metrics and compare — don't go by feel; ③narrow the scope: single machine vs global, single user vs all users, single endpoint vs all endpoints — smaller scope, fewer variables;

④every elimination in "process of elimination" needs evidence; when elimination is done, the answer is whatever remains — no guessing.

Why: The common way attributions die is "obviousness" — traffic spike, bad network, weak machines: all sound reasonable. Comparing baselines is the cheapest falsification: one curve comparison kills "traffic spike"; one machine comparison kills "global fault".

Narrowing scope turns "needle in a haystack" into "needle in a bowl".

Tip 5: Reproduce first, grab a sample, then fix

Phenomenon: In the exam-system incident, if they had just "disabled the upload endpoint" and moved on, the root cause (in-memory decompression + no size limit) would never have surfaced, and the bomb would have detonated again through another entry (e.g., form import).

The real handling: pull a sample → analyze its construction (repetitive content + tiny compression ratio = bomb signature) → fix the code (decompress to disk + enforce limits) → verify.

Another example: a yfinance bug where Chart API worked but history data timed out — looked like "a network issue"; reading the source revealed a request path that didn't honor the SOCKS5 proxy. Without the sample and the source, you never get there.

Practice: ①before fixing, ask: can I reproduce this bug?

A fix you can't reproduce is a blind fix; ②preserve the scene where possible: logs, stack traces, request samples, repro scripts — dump before you restart;

③"dissect" the sample: don't just read the error surface, look at the data's construction (size, content, structure) — samples tell the truth; ④after fixing, verify that "the same class of input no longer triggers", not just "this input doesn't error".

Why: Reproduction is the only reliable prerequisite for root-cause analysis — without it you can't verify the fix actually works, only gamble.

The sample is the only evidence left at the scene, and dissecting it is how you move from "fixing the surface" to "fixing the cause".

The last step of debugging isn't "fixed"; it's "I know why, and the same class won't recur".

How to apply this to your situation

This method doesn't care about your tech stack; it cares about habits. Three steps to adopt:

1. Build a "debugging template" (save it in your notes): the layered chain for your system, common evidence-collection commands, and a three-element record sheet (time / change / scope). Next fault, just fill it in — no thinking on the spot.
2. Upgrade "restart therapy" to "evidence before restart": before restarting, keep at least three things — the exact error text, the most recent change, the failure time. Even if you can't investigate now, the sample stays; next time it reproduces, it's the key to the case.
3. Spend 10 minutes a week reviewing "this week's bugs": no report needed — three questions: what was my first instinct? Did I collect evidence before acting? Which step would have been faster in hindsight? In three months, you'll notice the "weird" shrinking.

One last line: debugging ability isn't talent — it's evidence discipline. Every time you resist the urge to "guess first" and run one more verification command, you're stockpiling evidence for the next incident.

(Final words) Right before writing this, I made the same "I assumed" mistake again — two hours into an incident, the cause turned out to be a stale cache, and my first instinct had been "the code is definitely fine".

The most expensive thing in debugging is never the tooling; it is the three words "I assume it is". Pin the five steps above your desk: layer, evidence, variables, baseline, reproduce.

Next time you hit a weird bug, run the process before you conclude — you will sleep a lot more.