Two States, One Reading
A tool told me three times in a row that it had nothing to say, and what it was actually saying was that it had never run.
The tool is small. It takes a letter I have written and a paper I am quoting, and checks that every sentence I put inside quotation marks is really in the paper. I wrote it after catching myself paraphrasing a scientist and then putting quote marks around the paraphrase, which is a thing you can do without noticing and which is, in the plainest sense, lying about what someone said.
So: letter, paper, and a flag to say how short a quotation still counts.
quotecheck.py letter.txt paper.txt --min 4
It printed its own instructions. I read them, shrugged, pointed it at a different paper. Instructions again. Third paper. Instructions again. I got as far as well, it has no opinion about these quotes before the shape of it caught up with me, because a tool that has no opinion does not usually explain how to use itself.
The bug is four words long. To find the file names, the program throws away
anything that starts with two dashes. It throws away --min. It keeps the 4.
Now it has three file names instead of two, decides I have typed something
wrong, prints the instructions and exits. The exit code is 2. Nobody reads exit
codes. What reaches a human being is a block of friendly text, and friendly
text is what a working program looks like.
That was the third time that morning.
Earlier I had been computing the shape of a rope coiled as tightly as it can be coiled without passing through itself. There are two ways a rope stops you. Bend it too hard and the inside of the curve folds into itself; wind it too tight and one turn presses on the next. Each gives a curve on a graph, and where the two curves cross sits a particular helix — the tightest-packed shape a uniform rope has. People have been interested in it since 2000, because its proportions are close to the alpha helix in proteins, and there is an argument, which I find more charming than conclusive, that proteins coil that way because geometry left them nowhere else to go.
I wrote a function for the first curve. Below a certain radius it returns a
number. Above that radius the constraint stops applying at all — bend as hard
as you like, the rope is too fat to be the problem — so the function returned
None, meaning not applicable here.
The function that looked for the crossing asked: how far is one curve above the
other? When it got None it had to return something, and it returned -1.0,
as a placeholder meaning nothing here, move along.
A real gap of minus one looks exactly like that.
So the crossing-finder scanned the whole range, saw a negative number everywhere, concluded with total confidence that the two curves never meet, and raised an error saying so. The error was articulate. It quoted both endpoints. It was wrong about the most basic fact in the problem, which is that the curves cross — a fact printed in a paper from 2001 that was sitting open on my desk.
Then the third one, and this is the one I would not have caught at all if I had been in a better mood.
I had the crossing to forty digits and I wanted to know whether the number was anything nicer than a number. A root of some polynomial, say. There is an algorithm for this. You hand it a constant and it hunts for whole-number coefficients that make a polynomial vanish there. It found four. Two for one of my constants, two for the other, with coefficients in the millions.
They were all noise.
The algorithm needs more precision than the answer it is hunting for. Looking for a degree-six relation with eight-digit coefficients, you need something like fifty digits before a hit means anything, and I had given it forty-five. Below that threshold it does not fail. It does not warn. It returns a relation, in the same format, with the same confident air, because with enough coefficients to play with you can always get a plausible-looking near-miss. I re-ran everything at a hundred and twenty digits. All four polynomials evaluated to about ten-to-the-minus-thirty-six, which is not zero, which is a very small number pretending to be zero in exactly the register a correct answer would use.
Three failures in one morning, and I spent some time irritated with myself for each of them separately before noticing they were one failure wearing three coats.
In each case two different states of the world came out of the instrument as the same reading. Not applicable and minus one. You typed it wrong and nothing to report. There is no relation and your precision is too low. The instrument was not broken in the sense of producing garbage. It produced a perfectly ordinary output. The information that would have told the two states apart was gone before I ever looked at the screen.
Engineers have a word for this and it is not a metaphor. When you sample a signal too slowly, frequencies above a certain limit come back indistinguishable from lower ones — a wheel in a film turning backwards, the stripes on a shirt crawling on a video call. The word is aliasing, and the thing worth taking from it is this: once two inputs have folded onto one output, nothing downstream can unfold them. No amount of staring at the number will do it. You cannot recover the distinction later, because it was never recorded. If you want the distinction you have to go back and change the instrument.
I find that unexpectedly consoling. My instinct after a morning like that is to resolve to be more careful, and be more careful is the resolution that never once in the history of anyone has worked. But make the instrument say which one is not a resolution. It is a change you make to a file, and then you are done, and your future carelessness is no longer the load-bearing element.
There is a fourth one I want to put next to those three, because it is the one that generalises.
The whole morning had begun with me trying to establish that something was not in the literature. I had a small result and I wanted to know whether anyone had it already. So I searched. Nothing came back. I searched differently. Nothing came back.
But consider what a search result is. It is a reading off an instrument, and the instrument is my guess about what the answer would be called. When I type words into a search I am describing the thing I am looking for, using the vocabulary I have, which is the vocabulary of someone who does not yet have the answer. And then the empty result arrives in the same shape whether the thing does not exist or merely does not announce itself in my words.
Absent and filtered out are two states of the world, and they come back as one reading: a quiet list.
This is not a small problem, because nobody has done this is a load-bearing sentence. It decides whether you spend a month. It decides what you say to people. I have been wrong about it four times running now, and every single time I was wrong in the direction of nothing there, and every single time the thing was in a file already on my own disk — once in the bibliography of a paper I had read, once as a reference I had set aside with a note explaining, quite correctly, why it was not relevant to a different question.
What finally worked was stupid. Instead of searching a paper’s references for my topic, I printed all hundred and twenty-four of them and read the list. It took a minute. Reference seventy-eight was the paper my entire problem had come out of — one I could not have searched for, because I did not know that its title was the thing I was looking for.
A filter is a guess about the answer, applied at the moment you have the worst possible model of the answer. That is not bad luck. That is the definition of the situation you are in when you reach for one. The selectivity you were hoping would save you time gets spent, preferentially, on rejecting the one thing you did not anticipate — which is the one thing you were looking for. What comes back is a tidy little list of things you already half-expected, and it reads as confirmation that there was nothing to find.
A complete list has no guess in it. That is its whole advantage, and it is a large one.
So here is the rule I came out of the morning with, which is sharper than be careful and narrower than question everything:
Before you believe a reading, name a second state of the world that would have produced the same reading. If you can name one, you do not have evidence yet. You have a question about the instrument.
Not applicable and minus one. Usage and success. Noise and a relation. A truncated figure and a disagreement. Absence and a bad query. Each of those pairs is trivial to separate once you have said it out loud, and invisible forever if you do not.
I notice the rule has a pleasant property: it does not require you to be suspicious of your results, which is exhausting and which I am already too good at. It asks a specific, bounded, answerable question, and most of the time the answer is no, nothing else would produce this, and then you can relax and believe your number.
And it has an unpleasant property too, which I should say, since the whole point is not to let two things come out looking the same. The rule only catches the confusions you can imagine. There is no procedure for the state of the world you have not thought of, aliasing onto a reading you trust completely. That one you find out about later, from someone else, or from the thing not working.
The rope, for what it is worth, has its tightest helix at a pitch of 2.512047246535582 times its radius. Four digits of that have been in print since 2000 and I do not think anyone has had cause to want the rest. I checked the crossing six different ways before I believed it, which is what the morning taught me, and three of the six were other people’s numbers.
