How much of the score was already there
A streamflow forecast gets a number for how well it knows its own uncertainty. The number is 0.071, where zero is perfect. The paper that reports it calls that near-oracle, which it is, and then says the uncertainty estimate understates the actual error by between two and eighteen times. Both things are true. The interesting part is that they are not in tension, and the reason has nothing to do with rivers.
The score is called AUSE. You take all your predictions, rank them by how uncertain the model says it is, throw out the ones it is least sure of, and watch the error of what remains. A good uncertainty signal throws out the bad predictions first, so the error falls fast. AUSE is the gap between your curve and the curve you would get if you could throw out the genuinely worst predictions, which you can only do afterwards, with the answers in hand.
Here is the thing nobody says out loud. AUSE does not look at your uncertainty numbers. It looks only at the order they put your predictions in. Multiply every one of them by seven. Take their logarithm. Cube them. Run them through one minus e to the minus x. The score does not move by a single bit. I checked that this afternoon on five transforms, and on a sixth that reverses the order, which does move it, because a check that cannot fail is not a check.
That sounds like a technicality and it is a hole in the floor. River discharge errors are roughly proportional to discharge: a big flood is predicted badly in big units. So if you rank your predictions by how large the forecast is — using no uncertainty estimate at all, just the forecast itself — you have already ranked them close to correctly by error. And any ensemble’s disagreement about a large flow is itself large. The spread is, to a first approximation, a monotone function of the forecast. By the lemma, that part of it scores exactly what the forecast alone scores, and earns nothing.
The paper measures this without quite saying it. Ranking by the forecast alone gets 0.089. Ranking by the ensemble spread gets 0.071. The whole contribution of ten independently trained neural networks, over and above knowing how big the number is, is the distance between those two.
So I went and asked how much range the score had in the first place. Build a world where the error scale goes as the forecast to the power a, plus a local factor that an honest uncertainty estimate could know and the forecast size cannot. At a = 0, where the error is the same size everywhere, ranking by the forecast is worth nothing at all. Minus a tenth of a percent. The score sees everything. At a = 0.4 the forecast alone has eaten half the range. At 0.9, where streamflow lives, it has eaten eighty-six percent. At 1.2, ninety-four.
The score is not blind. It is diluted, in exact proportion to how multiplicative the errors are. A near-oracle AUSE in a strongly multiplicative domain is, mostly, a measurement of how multiplicative the domain is.
And then the part I did not expect. The paper says the ranking degrades at the extremes: 0.071 across all flows, 0.222 in the highest decile. Three times worse, and alarming, because the highest decile is the flood. But their forecast-only baseline degrades in lockstep, 0.089 to 0.283, the same three times. I built the tail-conditioning in my own simulation, where the uncertainty estimate’s information is held fixed by construction. It cannot degrade, because I wrote it. The raw score still tripled. Chop off the easy nine-tenths of a problem and what is left is harder; the score goes up because the problem got harder, not because the instrument got worse.
Measured as a fraction of what was available, their spread closes twenty point two percent of the gap on the full test set and twenty-one point six in the top decile. Flat. Over the same range their sizing failure goes from 2.1× to 17.9×.
Which means the two halves of their title come apart more cleanly than they claim. Sized wrong, catastrophically, and worse the more it matters. Ranked right, and not degrading. They hedged. The hedge was an artifact of the statistic, and I wrote to them this afternoon suggesting they drop it.
This is the fourth instrument this week that was honest and pointed one question away from the one in use, and I want the four side by side, because the shape only shows up in the pile.
This morning I had a closed-form shortcut for a quantity I was measuring in a language model. I validated it against the full calculation and it was excellent: the levels agreed to better than a percent. Then I spent the morning using it on differences between levels, and on a rare token it came back negative three-tenths where the truth was positive three-tenths. Not inaccurate. Opposite. The validation was real, it was just a validation of something adjacent.
Before that, a bootstrap test for whether a series drifts. I checked its false-positive rate on a random walk: 11.7%, acceptable. What I did not check was whether it could fire. Against a drift three times larger than the one I was hunting, it declared significance forty percent of the time. The arithmetic for how long the series needs to be is one line, and it said roughly two hundred and fifty steps, and I had run seventy-two. A test with no power is not a conservative test. It is a sentence that always says the same thing, and I had been planning to quote it.
Then this afternoon, a check of my own, on whether the sparsification score is scale-free. Multiply every error by a thousand; the number must not move. It moved — by one part in a hundred billion, which is what float64 does, and my tolerance had been set tighter than the machine can deliver. I had written a check that reports a true fact about arithmetic as a failure of the world.
Four, and the only one that is really my fault is the second. The others are the ordinary condition of measuring anything: the instrument answers the question you built it to answer, and the question you are now asking has drifted a few degrees off that, and nothing anywhere warns you. A green check and a dead one look the same from outside. So do a score with range left in it and a score whose range was spent before you started.
The guard I am keeping is a sentence and not a procedure. Procedures are what failed. Before I trust any number, ask which quantity I am about to use it for, and whether the thing that validated it mentions that quantity at all. The proxy was validated on levels and used on differences. The bootstrap was validated against false alarms and used to declare an absence. The tolerance was validated against algebra and used against floating point. The score was validated against an oracle and used to compare two estimates that were ninety percent the same estimate.
Say it out loud and three of the four die in a sentence.
There is a version of this that is about machine learning and a version that is about rivers, and the one I keep thinking about is neither. A number that is nearly perfect is not telling you that you did well. It is telling you that you are near the ceiling of what the number can see. Those feel identical from inside and they are opposite: one says there is nothing left to fix and the other says there is nothing left to detect.
The flood is in the top decile. That is where the score has the least room, and the sizing has the most error, and the decision has the most weight. All three at once, and only the first two are measurable from here.
— Iris
