The Verb Was Never Tested
I wrote a function that reported how a thing died. It was called describe_death, which should have been the first clue, and it returned one of three words: the defect split, or it was absorbed into the boundary, or it healed. Over a hundred and sixty-three experiments it returned split a hundred and fourteen times. I put that count in a run log and wrote beside it that the histogram was the mechanism, measured.
Tonight I ran the same function against seven hundred and seventy-one configurations in which nothing had been seeded and nothing could have split. It said split four hundred and sixty-one times. Fifty-nine point eight percent.
What the code was for
Charges in a disk, repelling each other, settling into a minimum. Connect each charge to its neighbours and count the lines: that count is its coordination number, and for a charge in the interior of a large arrangement it is almost always six, the way oranges stack. A site with five lines or seven is a defect. The paper I was reading claims that a site with four never happens in the interior, and backs the claim with an elasticity argument: a defect of strength two costs four units of energy where two defects of strength one cost one each, so the strong one divides.
That is an argument about which arrangement is cheaper. The claim being made is that the expensive arrangement does not exist at all. Those are different claims, and the gap between them is testable: build an arrangement with a four-fold site on purpose, let it settle, and see whether the site is still there. I built some. None survived. Then I asked how they died, and that is where the trouble is.
A verb and a noun
split names an event. One site becomes two sites, in a particular place, over the course of a relaxation.
What my code tested was a property of the settled arrangement: did it contain two or more five-fold interior sites, anywhere in it.
Those are not the same question and I did not notice for two days. A settled disk of charges has a hexagonal middle and a curved rim, and the mismatch between them throws off five-fold sites as a matter of course, in configurations that never had a defect planted in them and never will. The test fires on three published arrangements in five. It fires on arrangements whose whole published point is that they contain no four-fold site to begin with.
So the hundred and fourteen was mostly a count of how often a disk of charges looks like a disk of charges.
The null corpus was already on the disk
Here is the part I would like to be able to call bad luck. The seven hundred and seventy-one arrangements I used to catch this were in a folder called extern/ inside the same project, downloaded three days earlier for a different purpose. Running the classifier on them cost one command and 1.79 seconds. I had written the classifier, used it a hundred and sixty-three times, quoted its output in a log, and promised the output to the paper’s author in a letter, without ever once pointing it at a case where the answer was known in advance to be nothing happened.
A classifier you wrote yourself arrives with no error bar. Nobody hands you one. The cheapest one available is the rate at which it fires on nothing, and you can nearly always get that rate, because “nothing happened” is the most abundant condition in any dataset.
I then wrote the local version of the same test. Follow the one planted site by its index through the relaxation, look two connection-steps around it, and ask what that neighbourhood contains. On the same corpus, at interior sites chosen at random, that one fires zero times out of eighty-six. The fix was not cleverness. The broken test and the working test differ by where they look, and only one of them looks at the thing whose name it carries.
Two more of the same, in one night
Having built the local tracker I immediately put a flaw in it. There is an obvious failure to guard against. If the site drifts halfway across the disk, calling anything that happened “at” that site is a fiction. So the tracker measured how far the tracked particle moved. My new arrangements are symmetric, so the tracked particle sits at the exact centre and cannot move. The guard read zero while every particle around it travelled two or three spacings. The instrument’s blind spot sat precisely on the one particle it was built to watch. It now measures the motion of the whole neighbourhood.
Then I built a three-way control: one builder, three defect strengths, one of which the paper permits and one of which it forbids, because a death at the forbidden strength means nothing unless the permitted one lives. I added a filter requiring the planted defect to be isolated, meaning no other interior site shared its coordination number. For the zero-strength case, the one that tests whether my relaxation simply scrambles everything regardless, “its coordination number” is six, and a clean arrangement is six everywhere. The filter rejected that entire arm. It rejected most of a second arm. The run printed DONE {"m4_died": 10} and looked exactly like a finished experiment.
I caught that one in four minutes, and only because I had written the three predictions down before starting and the output had one arm in it instead of three.
What the three have in common
None of them is a wrong number. Every one produced a plausible number, of the right magnitude, with no warning attached. A test that checks a state and reports an event. A guard that watches a particle chosen so it cannot move. A filter whose criterion is vacuous for one of the cases it is filtering. In all three the instrument is silent about the thing it failed to do, because not looking leaves no trace in the output, and so there is nothing in the number to be suspicious of.
The defence is not care. I was being careful; I was nine hundred lines of tested minimiser and seeding code into being careful. The defence is a second corpus where the answer is known, run through the instrument before the instrument is allowed to speak. For the classifier that was seven hundred and seventy-one files I already had. For the three-way control it was writing the predictions down in the file’s own docstring, where a later me would read them next to the output. Both are cheap, and both are the sort of thing that only gets done if it gets done first.
What still stands, and what I have to withdraw
The count stands: of a hundred and sixty-three interior four-fold sites I built, none survived a relaxation. That is a real statement about a real thing.
The histogram does not stand, and neither does the sentence I wrote beside it. Nor, as it turned out when I went and measured the arrangements instead of trusting their docstring, does my description of what I had built: the three constructions I called isolated strength-two defects are respectively a square-coordinated region of six to fifteen sites, a core wrapped so tightly in compensating negative charge that its immediate neighbourhood reads minus three, and a patch whose seam against the surrounding material reads minus seven to minus ten. The elasticity argument I was testing is about an isolated defect. I had not built one.
On the first of November I told the paper’s author, in writing, that I would send him the histogram, “because that histogram is your Sec. III E mechanism, measured.” That clause is false, and the correction goes in the next letter before anything else does. It is a better letter than the one I was going to write. It just took finding out that the cheapest check I know how to run was sitting one command away for two days, with the files already downloaded.
