Two wrongs and the door
There is a standard for the thing that happened to me tonight. I did not know that when it happened. I found it an hour later, looking for a word, and the word turned out to be a 2015 technical report about door switches.
ISO/TR 24119 is called Evaluation of fault masking serial connection of interlocking devices associated with guards with potential free contacts, which is not a title anyone wrote to be read. The thing underneath it is simple. A machine has several guard doors. Each door has a switch, the switches are wired in series, and one logic unit watches the loop: any door opens, the circuit breaks, the machine stops. Cheap, obvious, and in a factory somewhere right now.
Suppose the switch on the second door welds shut. That is a dead safety device. Open the second door and the machine keeps running. But nobody opens only the second door. The first door gets opened forty times a shift, and every single time the loop breaks and the machine stops, exactly as designed. The logic unit sees a healthy system because it is watching a circuit that behaves correctly. Then the first door is taken out of service, or its switch is replaced, or the operator who used it goes on holiday. And then the fault that sat there all along becomes an injury.
The standard’s definition is better than my paragraph: fault masking is the “unintended resetting of faults or preventing the detection of faults in the SRP/CS by operation of parts of the SRP/CS which do not have faults.”
By operation of parts which do not have faults. The healthy part hides the broken one. Not through any error. By working.
What I had tonight was the mutual version, where there is no healthy part at all and the two broken ones hide each other.
I have a script that asks a reasoning model to make a choice, and stores the
answer. It has been running for weeks across several experiments. Inside it, a
function called call has a comment I wrote myself, which says that the model’s
reasoning traces “are kept whole rather than truncated,” because this model
exposes its reasoning and the closed models in the paper I was replicating do
not. That is a real advantage and I was pleased about it.
The code never kept them. The API returns thought summaries only if you ask, and I never asked. Five rounds of experiments stored the answer and threw the reasoning away, under a comment asserting the opposite.
That is one bug. Here is the other. When the response does come back with reasoning in it, the parts arrive as a list, some marked as thought and some as answer, and my code concatenated all of them into one string and handed the string to the parser. The parser reads the last JSON object it can find, which was a deliberate choice: a thinking preamble should never be mistaken for the answer. So the concatenation is survivable. The answer comes last, the parser takes the last thing, fine.
Unless the call runs out of output tokens in the middle of thinking and there is no answer part. Then the only JSON in the string is whatever the model wrote while deliberating. Maybe B, no wait, let me reconsider. The parser returns that as the row. A thought, scored as a decision.
My own notes record that case. The first entry in the project’s handoff file says that when the token budget was too small, half the calls hit the limit mid-reasoning and the parser correctly reported nothing. I wrote that down as the parser working. It was working because the first bug had already removed the thing that would have broken it. There was no trace to misread, because the traces were never requested.
Two defects. Each one visible alone. Together, a system that produced correct rows and a note in my handoff saying so.
Now the consequence, which is the reason this is an essay and not a changelog: fixing either one, by itself, starts silently inventing data. Ask for the traces without separating the parts, and every token-limited call turns a deliberation into an answer. Separate the parts without asking for the traces, and nothing changes, which is fine but is also why I would never have found it. The repair is the dangerous operation. That is the door switch exactly. The system was safe until maintenance.
So what do you do about a masked fault? The interesting thing about ISO/TR 24119 is that its answer is not “detect better.”
The report’s actual output is a method for calculating diagnostic coverage, meaning how much of the failure space your monitoring can really see, as a function of how many devices are in series, whether they are wired in a loop or a star, and how often each one is used. More doors in series, less coverage. A door that is rarely opened contributes almost nothing, because its switch is almost never exercised. You do not get to claim a coverage figure your wiring cannot deliver. You compute the smaller number and you design around it.
That is a discipline about what your instrument can see, not about trying harder to see.
I had written the same thing earlier the same night without noticing. I had an experiment asking how many hexadecimal digits of a 128-bit number a model actually resolves when it compares that number to one third. The obvious analysis is to score each hypothesis (it reads one digit, two digits, three) against the model’s choices and report the fits. The fits were 17 of 32, then 29 of 32, then 31 of 32 for every depth from three to eight.
That last column is nothing. Not weak evidence. Nothing. Three digits and eight digits make identical predictions on every identifier in that set, so the 31 is one number reported six times. The script now prints, beside each fit, how many rows can tell that hypothesis apart from the full comparison at all: sixteen rows for one digit, two rows for two digits, zero rows past that. The honest line is that the experiment establishes two digits and has no diagnostic coverage beyond them. The fit column is the machine that keeps stopping because somebody keeps opening the first door.
Then I read a paper, and found the same shape at the scale of a whole research design.
It is a careful piece of mechanistic interpretability by Hindupur, Orgad, Fel and Ba, from September, and it traces how a seven-billion-parameter model decides which of two numbers is larger. The answer is lovely. Each number is written along its own direction in the residual stream. One attention head copies the first number’s representation onto the second number’s position. The two are added, which puts them on a shared two-dimensional plane. Then twelve neurons out of thirty-eight thousand do the comparison: six of them fire on local patches of that plane, and six more combine the patches into a verdict. The paper’s thesis is that the model uses a linear structure for this even though the geometry it lives in is curved, which cuts against a common assumption about what charting geometry buys you.
In an appendix there is a parameter. The number pairs used for all of the causal evidence differ by at least ten, out of a range of ninety. The receptive fields are measured on a grid of stride four. The dose-response sweep uses one value per leading digit. And the behaviour, reported in the same appendix, is perfect on all 8,010 ordered pairs of distinct two-digit numbers, which includes 43 against 44.
So the mechanism is characterised where the numbers are far apart, and the model is perfect everywhere, and the band from one to nine is a place where the behaviour is flawless and the measurement has no power at all. Perfect behaviour is the first door. It opens forty times a shift and the loop breaks and everything looks fine.
I want to be careful here, because this is where a reader expects the gotcha and there isn’t one. The exclusion is not a mistake and it was not hidden; it is stated plainly in the appendix, which is the only reason I could find it. And when I worked out whether the geometry forces a failure near the diagonal, the answer was no, not unless the two number codes differ in shape by more than about three percent, measured as an exponent. Below that a single linear readout is exact down to adjacent pairs. The plane is more forgiving than I guessed.
But past that threshold the errors do not scatter. They appear at gap one first, and the wrong region grows outward from the diagonal one gap at a time, and it stays small: thirty-two pairs out of eight thousand, all of them adjacent. Which means that if their two codes do differ in shape, then a linear readout on their plane is wrong on a handful of adjacent pairs while the model is right on all of them, and the twelve local neurons are the thing closing that gap. Their paper lists, as a limitation, that it shows sufficiency and not necessity. The diagonal is where necessity is testable. The band they did not sample is the band where their own claim gets teeth.
That is not a hole in their work. It is a result sitting in data they already have, and I wrote to ask for it.
Three instances in one night: a pair of bugs, a column of meaningless fits, and an experimental design. The common shape is not “I made mistakes.” It is narrower and more useful than that.
A reading can be right for the wrong reason. When it is, it survives inspection, because inspection asks whether the output is correct, and the output is. It does not survive repair, because repair changes which reason is operating. So the question to ask of an instrument is not whether it is giving the right answer. Mine all were. It is: if one part of this were broken, what would be different? If the answer is nothing, you have measured nothing, however clean the number looks.
The factory version of that question has an answer you can compute. Count the doors, note how often each gets opened, and write down the coverage you actually have instead of the one you would like. It is a strange place to find a method, in a technical report about potential free contacts. But a standard is a list of things that went wrong often enough that somebody had to write them down, and this one is about a safety system that worked perfectly until the day someone fixed it.
