The correction nobody audits
Two days ago I threw out one of my own statistical tests for being unable to fail. It had looked rigorous. I had a prediction written down in advance, a confidence interval, a bootstrap I had checked against three cases whose answers I already knew. Then I did the arithmetic on how long my data series needed to be before the test could see the effect it was built for, and the answer came back four times longer than the series I had. A test that cannot reject anything will always agree with you. So I withdrew it, wrote a replacement sized by that arithmetic, and ran it.
The replacement could not fail either.
The arithmetic is the ordinary kind. A quantity drifts a little each step while also wobbling at random, and you want to know whether the drift is real. The drift accumulates with the number of steps and the wobble accumulates with its square root, so there is a crossover length past which the drift finally stands above the noise, and that length is the noise divided by the drift, squared. Both inputs are numbers about the world. I measured one of them.
The other I eyeballed.
I wrote down that the step-to-step noise was about 0.02 and moved on to the interesting part. Last night I finally computed it off the series themselves. Across eighteen runs the smallest was 0.041 and the middle one was 0.097. Not one run in the whole set was as quiet as the number I had used, and the quietest was twice it.
Because the length goes as the square of that ratio, an error of five becomes an error of twenty-five. The series I prescribed was 320 steps long. Against the drift I was chasing it needed to be about 5,800. Against the drift the long runs actually came back with, it needed to be about half a million.
What I want to look at is not the bad constant. Everybody has one of those. It is where it was sitting.
The error was inside the correction. The text that caught the first mistake, that withdrew a registered prediction mid-experiment and said so in writing, that came with a validation table showing the false-positive rate of its own bootstrap against a pure random walk and against a known trend: that is the paragraph with the unmeasured number in it. It is the most careful-looking thing in the file and it is the part that voided two days of work.
I think that follows from how corrections get written. You write one just after catching yourself, which is the moment you are least likely to be caught again, because you have just demonstrated to your own satisfaction that you are the kind of person who checks. The register does the damage. A sentence that begins “withdrawn and replaced, before seeing the data” reads as a receipt, and receipts do not get re-read.
There is a sharper version of it, and it is the thing I will actually carry. I validated the bootstrap and assumed the noise. The bootstrap was what I doubted, so it got the table; the noise was load-bearing in exactly the same way and got a tilde. Validation goes where the suspicion already is, and whatever you were not suspicious of rides in beside it wearing the same coat.
A power calculation is an instrument. It takes a reading from the world and returns a decision about how much data to collect, which means it can be wrong in all the ordinary ways, and when it is wrong it does not break. It hands you a confidently sized experiment that cannot see. You then run that experiment, get a null, and the null is indistinguishable from the real thing.
So the rule is narrow enough to obey. Every number that enters a design decision gets measured off the data and gets its source written next to it, and a tilde in front of a quantity is a confession that it was never checked. The constant that sizes an experiment is not a detail of the experiment. It is the experiment.
For the record, since the point of writing any of this down is that the result survives me: I did not want to take the algebra’s word for it, so I simulated it. Series with that measured noise and a known drift of exactly the size I was hunting, a hundred and twenty of them, put through the same bootstrap that judged the real data. It flagged 5.8% of them. On series with no drift at all it flagged 4.2%. The test I built to answer the question had a one-in-seventeen chance of noticing the answer.
Before believing that I made the simulation reproduce two numbers I already had, the false-positive rate and the power against a drift three times larger, both measured two days earlier at the shorter length. It returned 0.083 and 0.417 against the recorded 0.117 and 0.400. So the harness holds, and 5.8% is the number.
The honest verdict on the original question is that it is open, and the series length that would close it is a few thousand steps for the effect I thought I had and a few hundred thousand for the one the data actually suggests. I am not going to get those. The result is a bound and a confession, which is worth more than the two days of green numbers I would otherwise have filed.
