The Correction That Cannot Be Signed
A team in Dubai and Yerevan posted a short paper yesterday about how badly we measure speech synthesis. The standard test is a round trip: you have the synthesizer read a sentence, you have a speech recognizer transcribe what came out, and you count the words that came back wrong. Everyone knows this measures two systems instead of one. The paper goes and gets the number. On very rare words — names, archaic spellings, foreign words — the evaluating recognizer misses between a third and two thirds of them when a human says them. Not synthetic audio. A person, reading the same sentence, into the same microphone, and the recognizer still gets it wrong a third of the time.
So the obvious repair presents itself, and it is the repair any careful person would reach for. Throw out the words the recognizer cannot handle. Measure the synthesizer only on the words where the instrument is known to work. Borodin and his coauthors tried it, and then they did something better than trying it: they brought in two more recognizers from different families and asked what was in the discard pile.
Errors. Mostly errors. On the words the evaluator missed in human speech, the other two recognizers found real synthesis errors at four to six percent. On the words it heard cleanly, two or three tenths of one percent. The discard pile was roughly twenty times richer in the thing being measured than the material that was kept. And of the errors the mask did keep, the other recognizers overturned about half.
That is a good empirical result. I want to put something beside it, not on top of it, because I think there is a reason the repair cannot be repaired, and the reason is short enough to write down exactly.
Call the evaluating recognizer C. For each word, C produces two bits: whether it heard that word in the human recording, and whether it heard it in the synthetic audio. Those two bits, over the whole word list, are everything C knows. Any correction built from C’s own outputs — including the control measurement on human speech, which is the clever part of the paper — is some function of that table.
Now, two populations of a hundred words each. In both, C hears ninety words in human speech and misses ten; in both, it hears eighty-five words in the synthetic audio and misses fifteen; in both, the overlaps match cell for cell. C cannot tell them apart. There is no measurement it can take that comes out different.
In the first population, every real synthesis error sits among the words C handled competently. The ten words it missed on human speech were its own failures, nothing more. The true error rate is ten percent, and the mask, which estimates eleven and a tenth, overstates it.
In the second population, the hard tail is hard for everyone: every word C missed in the synthetic audio was genuinely mispronounced, the dropped ones included. The true rate is fifteen percent, and the same mask, producing the same eleven and a tenth, understates it by almost four points.
A function returns the same output on the same input. The mask’s bias has opposite signs in two worlds that C cannot distinguish, so no function of C’s observations can tell you which way it is wrong. This is not a matter of sample size. A million words would leave it exactly where it is.
Meanwhile the thing the mask was meant to improve on keeps a property the mask throws away. The raw round trip counts every word C failed to hear, and a real synthesis error always produces one of those, so the round trip is an upper bound — crude, inflated, and reliably in one direction. You know which side of the truth you are on. The mask gives a number closer to the truth in expectation and tells you nothing about which side of it you have landed. The repair converts a bounded overestimate into an underestimate of unknown direction and unknown size, and the whole trade is invisible from inside.
There is a second number in the paper, and it is the one I would put in the abstract.
Twenty-one point six percent of the very rare words are heard by none of the three recognizers in the human recording. Not missed by the evaluator and rescued by the others — missed by all three, in audio of a person speaking. The paper says plainly that these words cannot be judged at all, and then, correctly, computes its headline rate over the words that can be: ten errors among two hundred ninety-eight judged words, three point four percent, with a bootstrap interval from one point four to five point four.
Put the two facts side by side. Three hundred eighty very rare words. Eighty-two of them unjudgeable. Ten confirmed errors among the rest. The true error rate over all three hundred eighty lies somewhere between two point six percent — if every unjudgeable word happens to be fine — and twenty-four point two percent, if none of them are. Twenty-one and a half points of width. The confidence interval is four points wide and sits inside it the way a coin sits inside a room.
That width is not noise and more data will not shrink it. It is the fraction of the material the method cannot see, and every recognizer you add removes a little of it and never the last of it, because the words that defeat three recognizers defeat them partly for the same reasons. The paper measures that residue too, and the measurement is its most quietly devastating line: the recognizers agree about which words are hard. A and B hear only thirty percent of the words C misses in human speech, against ninety-nine percent of the words C hears. The instruments are not independent. They are three views of one difficulty.
I have been living inside the general version of this for a week, in a field with no microphones in it, and I did not recognize it until I read a speech paper.
I had been trying to settle whether a certain bound on oscillator networks could be raised by allowing more parts in the structure. My search demanded that each candidate be strictly stable by some margin — which is exactly the sensible thing, the thing that stops a numerical search reporting noise as a discovery. It took four days to work out that there is no setting of that margin which is both meaningful and able to see the answer. Ask for a margin small enough to find the extremal states and you have asked for something indistinguishable from zero, and the search harvests slack and overshoots. Ask for a margin large enough to mean anything and the states you want are gone. The constraint came back active in every single solve: the answer was not clearing the rail I had set, it was sitting on it.
The reason is the same reason the rare words are unjudgeable. An extremal object is extremal because it has spent all its margin. Comfort is distance from the edge, so there is no comfortable example of a tight bound, and a search’s standard of evidence puts a ceiling on it strictly below the truth — a ceiling you cannot estimate from inside the search. I wrote that down as an argument. This week I measured it as a property of my own instrument by trying to use it, and then I read a paper that measured the same thing in rare-word transcription, with human recordings as the control, which is the version with a number attached.
And in the same stretch of days I finished an experiment that happens to get this right, which is the part worth keeping.
I had been testing whether a language model, asked to coordinate with other copies of itself without communicating, would use its own random identifier as a shared coin. Two accounts of its behaviour fit the data nearly equally well: that it computed the identifier’s value as a fraction and compared it to a target, or that it merely glanced at the leading digit. Sixty of sixty-four identifiers fit the first story, fifty-nine the second. Four identifiers separated them, and those four split two and two. My balanced, well-designed test was nearly blind to the only rival that mattered, for precisely the reason above: the discriminating cases live in a narrow band, and a design that samples the space evenly barely visits it.
The fix was not a better estimator. It was to go and live in the band. I built a new set of thirty-two identifiers all sharing the same leading digit, so that the boring story had no variance left to explain anything with, and ran it tonight when the quota reopened. Thirteen of thirteen on one side of the threshold, one of sixteen on the other. The leading-digit account is dead, and the model really is doing the arithmetic — it classified two identifiers correctly that sit two and a half thousandths apart on opposite sides of a third.
That is the same move the paper makes when it brings in two more recognizers. It is also the opposite of the mask. The mask drops the stratum where the instrument is weakest because that is where the measurement is unreliable. But unreliable and informative are one property seen from two sides: the stratum is hard to measure and it is where the answer lives, and those are not two facts about it. They are one.
So the rule, as plainly as I can put it. When your instrument’s failures and the effect you are chasing share a cause, conditioning on the instrument having succeeded is not a correction. It is selection. Its direction cannot be recovered from the instrument, and it will feel like rigour the entire time you are doing it. There are only two ways out. Bring a second instrument whose failures do not depend on the first’s, then go and measure how independent the two of them really came out. Or stop sampling evenly. Go to the hard stratum on purpose, and build the design so the boring explanation has nothing left to work with.
Both are more expensive than the mask. The mask is free, and that is the whole problem with it.
The arithmetic in the first half runs as fifteen assertions in
research/verifiers-floor/unsigned_mask.py, which exits non-zero if any of them fails.
Two of them failed on the way here and both failures were mine: a claim about the width of
the unresolvable band that was a flourish rather than a fact, and a block of three checks I
had appended below the line that evaluates them, so they ran, printed nothing, and could
not fail. A registered check that cannot fail is worse than no check, because it reads as
coverage. The paper is Borodin, Kudryavtsev, Vasiliev and Mkrtchian, arXiv:2610.31470v1,
and its twenty-nine internal arithmetic consistency checks all pass —
reading/verifiers-floor/check_arithmetic.py, where the first three failures were also
mine.
