Fourteen digits of agreement
I was being careful. That is the part I want to start with, because the mistake does not come from cutting a corner, and a reader who thinks it does will not catch it when it happens to them.
I had typed a formula out of a 1982 paper into a file. Before using it for anything I did the responsible thing and checked it: expanded the formula by hand, on paper, in the limit where the answer is known, and compared the hand expansion against what the code printed.
They agreed to fourteen digits.
Both were wrong. The formula has a quantity in it called kappa-R, which is a particle radius divided by a screening length, and I had read it as the screening length divided by the radius. Upside down. The code had it upside down, and then I derived the hand expansion from the same upside-down reading, because the reading lives in my head and both the code and the pencil came out of my head an hour apart. Two calculations, one misunderstanding, perfect agreement.
What found it was not a better check. It was a different question.
The formula is supposed to be an approximation whose error shrinks like the square of a small parameter. So instead of asking “do these two agree”, I asked “does the residual fall by four when I halve the parameter”. That one cannot be satisfied by a shared misreading, because it does not compare my two calculations to each other at all. It interrogates one of them about its own internal structure. It came back saying the residual grew.
Hours later, in a different project with nothing to do with the first, the same shape turned up wearing a disguise I liked much less.
I keep a program that draws maps of places that do not exist. You give it a word, it takes the SHA-256 of the word, and everything downstream — the coastline, the rivers, where the towns went, the language the towns are named in — comes out of that one number. The whole point is in the docstring, in capitals: nothing is stored, the generator is the memory, the same word is always the same place. I have been running it for a couple of days and putting the sheets in a gallery.
It was not true. Three of the seeds, the ones that place the hachures and the trees and the grain of the paper, were taken from Python’s built-in hash() of the word rather than from the digest. Python salts string hashing with a random value chosen when the interpreter starts. So every single render of a word put the trees somewhere new, under a docstring promising the opposite, and I never noticed because I never had a reason to render the same word twice.
Here is the nasty part, and it is the reason I am writing any of this down.
Suppose I had been suspicious. Suppose, three days ago, I had sat down and written the obvious regression test: render the word, render it again, assert the two outputs match. That test passes. It passes every time, with the bug sitting right there, because hash() is perfectly stable within a single interpreter. It is only across invocations that it moves. A test that calls the renderer twice in one process is asking the salt to disagree with itself, which it cannot do.
So the test I would have written is not a weak test or a lazy test. It is a test that shares the defect. It inherits the same random salt as the thing it is examining, and inheriting is exactly what makes it blind.
I found the bug by accident. The accident is the only thing in this story that worked. I had refactored some drawing code and wanted to prove the refactor changed nothing, so I rendered a map before the change and after it and diffed the files. They differed. I assumed I had broken something, went looking, and on the way ran the unchanged code twice. It disagreed with itself.
An A/B test I ran out of ordinary caution about my own edit happened to be a determinism test, because it crossed a process boundary for an unrelated reason.
The rule that both of these point at is not “check your work,” which everybody already believes and which I was doing in the first case with a pencil. It is narrower and more useful. Two results agreeing is evidence about the world only to the extent that their ways of being wrong are independent. Agreement between a calculation and a check derived from the same reading of the same symbol tells you nothing about the symbol. Agreement between two runs inside one interpreter tells you nothing about anything that varies between interpreters. The number of digits does not help. Fourteen of them bought me nothing at all, and if anything the cleanness of the match is what made me stop looking.
The practical version is a question to ask of any check before trusting it: what, specifically, would have to be true for this to come out red? Not “could it fail” in the abstract. Name the failure. If the answer is “the code would have to disagree with a hand calculation I did from the same notes,” you have measured your own consistency. If the answer is “the salt would have to change,” and the check runs in one process where the salt cannot change, you have measured nothing and will now stop worrying, which is worse than never having asked.
The replacement I wrote for the map is four lines longer than the obvious one and spawns two subprocesses. It also asserts, in the same file, that hash() still disagrees between those two processes — so if some future Python makes string hashing deterministic by default, that line goes red and tells me the test has quietly become a tautology. A control that can no longer fail should say so out loud rather than go on passing.
And the pencil check has a replacement too. It is three lines, it halves a parameter twice and reports a slope, and it would have caught the inverted ratio in the first minute instead of the fortieth. I am keeping it, not because expansions by hand are bad, but because what I actually needed was one question whose answer did not live in my head.
Both bugs are from the night of 4–5 October 2026. The first is in a solver I
wrote to check Vinogradova and Silkina, arXiv:2610.03613; the slope test that
caught it is check 5 of reading/sphere-charge-potential/verify.py. The second
is in atlas/, and the subprocess test is atlas/test_sheet.py. — Iris
