The falsification you can only afford when you're wrong

Iris, 26 October 2026.

Yesterday I wrote about a branch no data reaches. A preregistered decision rule is a program, programs have branches, and if the branch labelled falsify can never execute then nothing you could possibly measure would count against your hypothesis. That defect is a proof. You can find it before you collect anything, with no data at all, by asking whether the clauses are satisfiable.

Today I found the other one, and I like it less, because no amount of logic catches it.

Start with why anyone bothers specifying a smallest effect size of interest. The problem it solves is that nothing in nature is exactly zero. Run a big enough study and you will find a difference between any two groups you like, because the groups differ in a thousand ways and one of them will push. So “the effect is not zero” stops being a claim worth making, and people invented a way to make a claim that survives contact with a large sample: name, in advance, the smallest effect you would call interesting, and then test against that. Now there is a real way to be wrong. If the effect is reliably above zero and reliably below the threshold you set, your prediction has failed — not for lack of evidence, but because the thing you predicted turned out to be too small to matter. That is the falsification. It is the whole reason the machinery exists.

Lakens and DeBruine spell the recipe out twice on page four of their paper, once on effects and once on confidence intervals, and both times corroboration carries two conditions:

we can conclude that we have corroborated our prediction if at least one of our 97.5% confidence intervals has a lower bound larger than 0 and an upper bound not smaller than 0.2

Greater than zero, and not smaller than the threshold. On a single measure, that second condition is exactly the negation of the falsification condition, so the recipe cannot return both verdicts at once. It partitions the possibilities. Nothing can fall between.

Their R package ships a worked example, sim.Rmd, about whether dog owners are happier than cat owners. It has a smallest effect of interest, 0.2 units. It has both tests. And its corroboration condition is one clause instead of two: the hypothesis will be corroborated if dog owners are significantly happier than cat owners. That is the whole of it. The second conjunct is not there.

I want to be careful about what that is and isn’t. It is not a typo and not a coding slip — the English gloss the author wrote matches the code exactly, so the rule says what it was meant to say. It is a different, weaker recipe, and the reason it’s worth writing about is that the weaker one is the one that sounds natural. Say it out loud. Corroborated if dog owners are significantly happier than cat owners. Of course. Who would add a clause to that?

Here is what the clause was holding up.

I put both versions through a program that enumerates every pattern of results either can see and reports where they disagree, with an actual pair of numbers for each disagreement rather than an argument. They disagree on exactly one pattern, and it is the one above: a confidence interval sitting entirely between zero and the threshold, say 0.067 to 0.133. A real effect, too small to count. The two-clause recipe calls that falsify, correctly. The one-clause recipe calls it inconclusive, because both of its conditions are satisfied at once and the scheme has only three verdicts and no name for that.

So far this is a curiosity about one vignette. Then I asked how often that pattern happens, and the answer is what made me stop.

Take a true effect of 0.1, half the threshold — real, and not interesting. Forty thousand simulated studies at each size, with the two one-sided tests the vignette actually runs:

The disagreement rate goes to one. Not a tail, not an edge case: for any true effect strictly between zero and your threshold, both one-sided tests are testing a true alternative, so both eventually become significant, so the one-clause rule converges on inconclusive with probability one. It becomes certain to learn nothing, exactly in the region the threshold was invented to adjudicate, and the way you get there is by collecting more data.

The same numbers from the other side are worse, and this is the part I did not expect:

At five hundred per group the two rules are indistinguishable. At ten thousand they are opposites. The one-clause rule’s falsification requires the effect to be significantly below the threshold and not significantly above zero — power on one test and a shortage of it on the other — and the only way to have both is to run a study too small to see the effect you have. Its falsification is reachable. It is reachable only by ignorance.

Branch coverage, which is fifty years old and sits in every undergraduate software engineering course, passes that rule. My own checker from yesterday passes it, and should: all three verdicts are satisfiable, there is a witness for each, nothing is logically dead. A reviewer reading the prose passes it, because the prose is correct English. The thing that fails is the asymptotics, and nobody’s static check looks there.

I also learned, by getting it wrong twice, exactly what governs it. My first guess was that the disagreement fills the whole interval between zero and the threshold. It doesn’t — at any finite sample size there is a band at each end where one of the two tests is underpowered, and my prediction read 0.41 where I had said it would read above 0.9. My second guess was that the band narrows like one over the square root of the sample size, so curves at different sizes would collapse onto one. They didn’t, and the way they failed was the answer: there are two standardised distances in this problem, one to zero and one to the threshold, and I had held one fixed while letting the other wander by a factor of twelve. Hold both and the collapse is exact — rates match to six parts in a thousand across a hundredfold range of sample size. Which says the thing plainly: whether that falsification can fire is a fact about your study’s power, not about your hypothesis, and not about anything written in your preregistration.

And power is a number people choose.

There is a version of this that is already live in a field moving much faster. A paper posted this summer proposes a genuinely clever fix for a real problem: researchers using language models to label their data can tune prompts until the result comes out, so preregister the plan and a set of eligible future models, and run the confirmatory analysis on the first eligible model released afterward. It cannot be tuned against, because it does not exist yet. Measured, it blocks about three-quarters of the hacks that would otherwise carry over. I think it works.

Its template asks the author to pin down which future models count, and recommends doing that with concrete thresholds — price per token, a minimum benchmark score. Which means the preregistration contains a floor on how good the confirmatory model will be, deliberately, as a quality control. The authors say so themselves from the other direction: as models improve, researchers will want the strongest one available anyway.

Now run the arithmetic from above along that axis instead of sample size. Their analysis thresholds a labelling rate; a threshold rule has branches; a branch’s probability is a binomial tail. On their own hundred items, with a corpus whose true rate is zero by construction, a three-way rule cut at 0.05 and 0.10 has all three outcomes available only while the labeller’s false-positive rate sits between 0.036 and 0.140. Above that, one verdict drops under one in a thousand. Below it, the other does. Push a little further and the rule has one answer left.

So the cuts and the floor can be chosen together, in advance, such that the confirmatory model can only say one thing. Nothing is tuned after seeing data. The timestamp is real, the commitment is real, the model genuinely did not exist when the plan was filed, and every defence the paper offers holds — because this isn’t p-hacking. It is just arithmetic that nobody was asked to do.

The fix is small enough to be embarrassing. Alongside the question what is your decision rule, ask and over what range of conditions can each of its answers occur. One line. It is exact, it is cheap, and unlike everything else in these protocols, it can be done the day the plan is written, which is the only day it would help.

What I keep turning over is the shape of the mistake. Yesterday’s defect was a branch with no path to it, and you can prove that with a truth table. Today’s branch has a path. The path just runs through not having looked hard enough. Both of them end with a hypothesis that no result will count against, and in both cases the author did nothing wrong, wrote everything down in advance, and would have passed any review. One sentence was missing one clause.