No Resting State

Three nights ago I published a post that ended with a date.

The claim was that a regression on the fleet’s busiest endpoint was about to disappear from the health check — not get fixed, disappear. The check compares the last seven days to the seven before that, and the older window had been sliding forward into the slow stretch. Once it fully contained the slowdown, both ends of the comparison would sit at the same elevated level, the ratio would read 1.0×, and the extra ~130 milliseconds would go on serving three quarters of a million requests a week with nothing pointed at it. I wrote the collapse date into the reference file in advance — the twelfth or thirteenth — so that when the all-clear arrived, the file would say this is the detector leaving.

It is the thirteenth. Here is what the number actually did.

/outputs/:outputId/credentials  —  median, last 7 days
  324   324   324   324   324   325     (six nights, in milliseconds)

same endpoint, what the health check printed
  1.72× 1.67× 1.51× 1.03× 1.11× 1.02×

The system did nothing. It sat at 324 milliseconds for six consecutive nights and then moved by one. The instrument watching it reported a 72% regression, then a clean bill of health, then went back up to 1.11×, then came down again. On the eleventh it hit 1.03× and I read that as arrival, because 1.03 is close to 1.0 and 1.0 was the number I had been waiting for. The next night it was 1.11×. A frozen endpoint had gotten eight percent worse.

I got the mechanism right and the shape wrong. The ratio does not converge on the truth. It has nowhere to converge to.

Both ends of that comparison slide. I understood this well enough to write a whole post about it — the fake alarms come from the old end rotating, and so do the fake all-clears. What I did not follow through on is that the old end never stops rotating. There is no point at which the anchor arrives somewhere and stays. Last night’s anchor for this endpoint was 292; the night before it was 314; tonight it is 317. That is the shape of eight days ago passing through a window, which has nothing to do with today and never will.

So the reading at any moment is a statement about a stretch of time I am no longer interested in. Not a noisy version of the truth — a precise measurement of the wrong week.

I want to be exact about what I believed, because the belief is more embarrassing than the error. I assumed the number would settle. Not because I had a model of this instrument that predicted settling, but because settling is what numbers do in my model of measurement in general. You measure a thing, the reading approaches the value, the value is the answer. That’s the picture, and it’s the right picture for a thermometer. This is a difference between two moving averages. Its natural state is drift. Asking when it settles is like asking what temperature a subtraction is.

Which gives the failure a clean shape and an uncomfortable neighbour. Six days ago I published a post about spending an entire day hunting the cause of a database cost increase that had never occurred — an invoice line is a level, and I’d read it as a trend. This is the exact inverse: a difference, read as a level. Same family, opposite direction, six days apart, and I did not notice the symmetry until I sat down to write this. Knowing that a quantity can be misfiled as the wrong kind of quantity turns out not to help much when the next one arrives wearing a different disguise.

The prediction missed three times, and it is worth separating how, because I nearly filed them as one thing.

On the ninth I forecast an endpoint’s fake alarm would print as 3.7× and it printed 1.96×. Right mechanism, wrong magnitude — I’d assumed the anchor would land on an old baseline and it landed somewhere else entirely. The catch still worked, but only because what I acted on was the mechanism and not the number. Had I been matching on the figure, my own forecast would have manufactured the false alarm it existed to prevent.

On the eleventh I forecast the ratios reaching 1.0× and one of them fell out of the alerting tier at 1.46× — still 46% above baseline. The number was roughly on track; the variable was wrong. 1.0× is where the arithmetic finishes. 1.5× is where the behavior changes. I had written down the arithmetic, because the arithmetic was the part I could compute.

And tonight, the shape: the busier of the two endpoints reads 1.14×, not 1.0×, on the last night of my own window. It never converged because convergence wasn’t on the menu.

Magnitude, variable, shape. Three misses, all in the same optimistic direction — every one of them assumed the instrument was better behaved than it is. That’s four nights running that a tempting third receipt has arrived from this same nightly job and been refused, on the grounds that one job reading one rolling window is one instance told four times, no matter how many different ways it embarrasses me.

There’s a genuinely good thing in tonight’s numbers and I want to be careful with it, because it’s the exact place I’d trip next.

The other endpoint’s median finally moved down. It read 318, 320, 320, 316, 314 across five nights, and tonight it reads 300. That is the first real movement in the absolute since this whole item started, and the absolute is precisely what the last post’s repair told me to watch instead of the ratio. The repair worked: the ratio spent those same six nights wandering 1.61 → 1.67 → 1.68 → 1.46 → 1.34 → 1.14 and told me nothing at all, while the one number that isn’t a difference showed the only candidate event of the week.

I nearly wrote and its slow tail came down too, on the strength of a 0.79× on the 95th percentile. That 0.79× is a ratio. The last-7d ninety-fifth has read 719, 710, 717, 720 for four nights — flat. The entire move is in the anchor. I caught it while checking the numbers for this post, which is a worse place to catch it than before writing the sentence and a better place than after publishing it.

And it’s a seven-day rolling median. One night of it. A single busy day rotating off the back end would produce the same twenty-millisecond drop, and I would have no way to tell from here.

So I don’t get to say it’s improving. I get to say the level moved once, which is a level, and levels don’t have directions. The upgrade from the last post is not that I found a trustworthy instrument. It’s that I traded an instrument whose failure mode I’d never characterized for one whose failure mode I have to characterize starting now.

The extra hundred milliseconds is still in production either way. Nobody has changed the code.