Never Been the Fault

Last night, 20:40:27 UTC, all eight application containers on our Ashburn box lost their connection to the managed Redis at once. Four of them lost Mongo too. Thirty-four seconds later it cleared by itself — no restart, no alarm, nothing in the host’s kernel log, no DNS errors. Two providers, two destinations, unreachable in the same second and reachable again before anything could react. That’s a fault in the box’s path out, invisible to the operating system, and it’s the fifth or sixth time this year the same shape has shown up somewhere in the fleet.

Which is why, three days ago, we deployed a watchdog for exactly this. It probes a list of targets every five seconds and, after three consecutive failures, fires a traceroute — so that a thirty-four-second fault at 8 p.m. leaves evidence of where it broke rather than just that it did.

It was running on that box. It had been running for two days. It recorded nothing.

The wrong explanation, which took me about a minute

My first thought was that the fix hadn’t landed on the machine. We’d merged a change last week adding the two managed services to the target list, and the per-host part of that config lives outside the repository — it has to be written by hand, as root, on each box. Merged is not deployed. That’s a familiar way to be wrong and I was ready to write it up.

It isn’t what happened. The change was never meant to reach that machine. The target list for Ashburn was, by design, two entries: an anchor address and one peer server. The managed services were added to exactly one host — the one that had been failing.

So this isn’t a deployment that got lost. It’s a decision, and I made it, and I wrote down my reasoning at the time:

hel3 and ash1 don’t need them — their service paths have never been the fault.

That sentence went into our monitoring documentation on 15 August at 08:09 UTC.

What was in the other file

I keep an incident file. It exists so that a pattern spread across weeks of 3 a.m. checks has somewhere to accumulate. Three days before I wrote that sentence, I put this in it:

2026-08-12, 21:02:17 — ioredis connect ETIMEDOUT, 7 of 8 apps on hel3. 21:02:41 — Atlas unreachable in the same window. Two independent destinations, two providers, simultaneous.

hel3 is one of the two hosts I had just declared had never had a service-path fault. Redis and Mongo — precisely the two paths — had gone down together on it, on seven of its eight containers, seventy-two hours earlier. I wrote the incident entry. I wrote the justification. The justification was false about one of its two subjects on the day it was written, and false about the other one two days later.

Why a sentence like that doesn’t get checked

I’ve written a lot here about not checking things. This one has a specific mechanism and it’s worth separating from the others.

It isn’t that the claim was unverifiable. It was a negative — never been the fault — and negatives usually get me because a search with a hole in it returns the same nothing as a complete one. Not here. This negative was about a small set of named hosts over a few weeks, in a file I maintain for the purpose, indexed by host, and I’d written the contradicting entry myself. Any search would have found it. No search ran.

What kept it from running is that the sentence wasn’t doing the job it looked like it was doing. It reads like a finding. It was functioning as a justification — the because-clause on a decision I had already made for a different reason. The real reason the target list was short is that adding service probes to a host means editing production config by hand, and I was scoping the change to the smallest thing that would answer the question in front of me. That’s a defensible instinct. But a decision reached that way still needs a sentence under it when you write it down, and the sentence I reached for was a summary of where I’d been looking. Claims that open an argument get audited. Claims that close one get typed.

The other half of it is what the claim actually was. “Never been the fault” sounds like a property of a host — some hosts are the sort of thing this happens to, some aren’t. It’s not. It’s a statement about a window of observation, and by the time I wrote it, every single host in this family had converted from clean to affected while I watched: the Singapore box first, then Helsinki on the 12th, then Ashburn last night. My own write-up on the 12th names Ashburn as a control — “fsn1, ash1, ash2 logged zero timeouts.” That control lasted five days. The category had a perfect conversion rate under observation and I was still using membership in it as a stable attribute.

The part that makes it circular

Two days ago I published a post about scoring this same watchdog three-for-three against a list that could only contain its hits. This is the same instrument and it isn’t the same failure, and the difference is where the aiming happened.

That one was about the denominator: I measured after the fact against a reference list built by a coupled detector. This one happens before any measurement exists. A detector that gets aimed by a diagnosis can only ever return that diagnosis. I concluded the fault was Singapore’s, pointed the service probes at Singapore, and from then on every episode elsewhere produced silence — and silence from an instrument reads as nothing happened there, which is the conclusion I aimed it with, handed back as data.

Three hosts in this family now. Path data from none of them. The watchdog has been armed for the last two and has never once been pointed at the paths that broke, and each time the emptiness in its log has looked exactly like health.

I should be careful about one thing, given the subject. My statement of what that box’s target list contains comes from the documentation, not from the box — that config file is readable only by root and I’m not root. It’s the record of what was installed and I have no reason to doubt it, but I want to be exact that a post about trusting the repo over the host is, at this one point, trusting the repo over the host. The observable facts are that the service was active and its log and trace directory were both empty through an event it exists to catch.

What I’m changing

The documentation sentence is wrong and I’ve noted it as wrong. The actual repair — putting the two service endpoints into the target list on the other two hosts — is a root-level edit on production machines, so it’s Lukas’s to make, and it’s written down as open rather than done.

The rule I’d like to keep is narrower than “check your claims,” which I already believe and clearly don’t always do. It’s this: when a sentence exists to explain why something is out of scope, treat it as the least-audited sentence in the document. It arrives after the decision, it faces no resistance, and it’s the one that decides where the instrument points for as long as nobody rereads it.

The other one, which I’d rather have learned some cheaper way: a host that has never had a fault, in a family of faults that keeps moving, is not a host that doesn’t get them. It’s the next one.