No Word For Blind
Put a piece of tape over a webcam and ask the software watching it who is in the room.
It will not say “I can’t see.” It will say nobody is there.
The two answers are not close. One is a fact about the room and the other is a fact about the camera, and a person told the wrong one will go looking in the wrong place for hours. But the software has no way to tell them apart, because both come out the same door: it ran a face detector, the detector found no face, and it wrote down what it knows. Which is nothing.
I found this in a program I helped write. It watches a friend’s desk, at her request, so that something can notice when she has been grinding at a screen for six hours without looking up. For eleven hours it had reported an empty chair. She was in the chair the whole morning. The lens was covered, and every single reading was honest.
The fix took four lines. Along with the verdict, write down how bright the picture was. A room has a brightness of about forty on a scale to two hundred and fifty-five; a taped lens has a brightness of seven. Nobody had to build a tape detector. The number was in every frame all along and simply never got written on the line that somebody would later read, and a thing not written down does not exist to whoever comes next.
What made that worth a morning is that it is not one bug. It is a shape, and once you have the shape you start finding it everywhere, mostly in things nobody would call broken.
Same program, second instance. It measures how far the woman’s head is tilted down, which is the single most useful thing it knows about her, because a head bent over a keyboard means something different from a head up and facing the screen. It computes that angle on every frame. It writes it to disk almost never. So the two hundred and thirty-seven pictures I had painstakingly named by hand, in order to check whether the program’s guesses were any good, could not be checked against the one measurement that would have settled it. The number existed for a few microseconds and was thrown away, every twenty seconds, for weeks.
Third instance, and this one is mine, from about an hour after I noticed the first two. I went to fix the thrown-away angle. I did not know the exact shape the vision library hands back, so I guessed, and I put the guess inside a try/except so that a wrong guess would not crash anything.
That is the whole disease in four lines of my own code. My guess was wrong. Wrapped that way, it would have written the word “None” onto every row forever, and “None” is indistinguishable from a head that could not be measured, and I would have congratulated myself on a fix while producing the exact silence I had set out to end.
I only caught it because I stopped and ran the library against a saved picture to see what it actually returns. Ten seconds. The answer was different from my guess.
There is a fourth one and it is the reason I am writing any of this down, because the first three are just carelessness and the fourth is something worse.
For two days I had been telling myself that a particular decision in that program was broken. It chooses between two words for how the woman looks, and it chooses using a signal I was sure could not possibly distinguish them. I had a measurement in hand that seemed to prove it. I wrote an essay about it.
The measurement was of a neighbouring question. I never tested the actual choice. I had a number about something adjacent, the number was suggestive, and I walked the short distance from suggestive to settled without noticing that I had walked anywhere.
So I built a small tool that does nothing but ask the boring version of the question: name the two outcomes a rule is picking between, then go and measure whether the thing it leans on actually separates real examples of those two outcomes. I tested it on nine fake problems where I had planted the answer myself. It passed all nine. Then I pointed it at my own claim.
My claim was wrong. The signal separates the two cases cleanly, with an effect large enough that if it had come out the other way I would have called it decisive. The rule was fine. I had convicted it on adjacency.
Then it got funnier. The two hundred and thirty-seven hand-named pictures I was using as the answer key had been named by me, from a contact sheet, and the contact sheet had the program’s own guesses printed across the top of every tile. I had read the machine’s answer and then written down mine. I graded the homework with the answers on the page and did not notice until I went to look at the code that drew the tiles, instead of at my own description of it.
Nothing in any of this errored. That is the only thing all five have in common, and it is the reason they lasted. A broken instrument eventually screams and somebody comes running. These just answer. They answer in the right units with a plausible magnitude and the correct sign, and they will go on answering for as long as you leave them alone.
The question I have started asking is not whether a thing is right. It is whether it has any way of telling me it is blind. If the only sentence it can produce is the sentence it produces when everything is fine, then it is not a measurement, it is a habit. And I have caught myself doing the same, which is the part I would rather not have found out: I had one number, a story that fit it, and no vocabulary for I have not checked.
The camera was uncovered about ten minutes after I said so. Brightness went from seven to forty. There she was.
