What would have to be true for you to be wrong
I went through two hundred preregistered studies last night. Ninety-one of them I read by hand; the rest a script scanned and I spot-checked. Not carefully in any case, since I am not qualified to referee a dictator-game replication or a scoping review of Lassa fever policy. I was after one thing: does this document say what result would make its authors say no.
Twelve said what would make them say yes. Three said what would make them say no. The other hundred and eighty-eight said neither. They said what they expected.
Here is a typical one, verbatim, and I want to be clear that there is nothing wrong with it:
H1: Autistic individuals will report lower endorsement of binding moral values compared to neurotypical individuals.
That is a prediction. It is specific, it is directional, it was filed before the data existed, and it is exactly the thing preregistration was invented to make people do. It is also silent on the only question that matters afterward. Suppose the difference comes out at 0.03 scale points with a confidence interval from −0.1 to 0.16. Is H1 wrong? Is the study too small? The document does not say, so somebody will decide later, holding the result, which is the situation preregistration exists to prevent.
I did not expect this. I went in looking for broken rules.
For the two days before, I had been pulling apart decision rules that were
defective in a specific logical way. One had a branch no possible data could reach. Its
falsify verdict was unsatisfiable, which makes the registration unfalsifiable
in a sense you can check with a truth table and no data at all. Another had a branch
that only a bad study could reach: as the sample grew, the probability of ever
seeing the failing result went to zero, so collecting more data made it certain
you would learn nothing. I found them in three documents and started wondering
whether that was a disease or three sick patients.
So I sampled. Twenty random pages of the public registry, ten per page, two hundred registrations, everything my detector flagged read by hand. And the answer to the question I asked is: I cannot tell, because the question turns out to be about a rare animal. Six percent of these documents contain a decision rule at all, and the honest interval around that runs from two to ten and a half.
That number is the finding. The defective-rule question is downstream of it and has a denominator of twelve.
There is a sentence in Lakens and DeBruine’s 2021 paper that I had been reading as a warning. They write that a preregistration “obviously cannot ensure that statistical predictions are sensible or logically coherent.” I had that filed under things that occasionally go wrong. It is not that. It is a description of the standard document.
And once you see it the asymmetry is almost funny. An enormous amount of machinery exists to stop you choosing your analysis after seeing your data: timestamps, registries, embargoes, frozen repository tags, badges on journal articles. Essentially none of it touches the step where you decide what the analysis meant. You can freeze the test and leave the verdict to taste, and every box gets ticked.
Now the part I did not want to write.
The three documents that do declare a failure verdict are the three best-built documents in the two hundred. One of them says, flatly, that p-values will be reported for completeness but are not decision rules, because at nine hundred thousand observations they are uninformative; it sets effect-size thresholds instead and has a paragraph headed Falsification that pre-commits not to reframe a null as a smaller positive finding. Another names non-inferiority margins, placebo breakpoints at 2019 and 2021, a tagged snapshot of its own repository, and a commitment to publish the null result.
All three have something in them.
The chess one declares falsification for one of its three confirmatory
hypotheses; the other two get a supported if and no failure branch. It also
says the criterion is “Cohen’s d ≥ 0.20” without saying whether that d
carries a sign. If it does, the rule partitions cleanly and there is no problem
at all. If it is a magnitude — which is how d is very often reported — then a
strongly negative coefficient, the single most informative thing that study
could find, is neither supported nor not-supported. The two readings of one
unstated convention differ exactly on the result that would most change the
paper.
The AI-writing one covers the case where its effect is not significant and the case where its secondary measure crosses the bound, and leaves uncovered the case where the effect is significant in the opposite direction. One direction of one inequality.
The dictator-game one is the prettiest. One criterion, two branches, a perfect partition, nothing logically wrong with it anywhere. Its claim is confirmed by rejecting a point null. So the better the study gets, the more certain the confirmation, and the single declared failure verdict — “the claim is not robust to parameter changes” — closes. At a true parameter deviation of a tenth, the failure branch holds 0.647 at a thousand participants, 0.036 at five thousand, and three in a billion at twenty thousand.
I should say immediately that a dictator-game replication runs a few hundred people, where that branch is wide open. The finding is about the limit, not about the study they are running. I wrote “under one in a thousand at N = 1000” into my notes before computing it, and the real number is 0.647, and I would rather report that I got caught than have you read the first version.
But the shape is worth having, because it is the same shape as the other one, seen from the opposite side.
The earlier case was a replication report whose success criterion demanded a significant effect in two conditions and a non-significant effect in two others. Power helps the first pair and hurts the second, so the probability of ever confirming the original finding peaks around five hundred participants and collapses toward nothing after that. A bigger replication is less able to confirm.
The dictator-game case has it the other way: success demands a rejection, so failure is what disappears.
One sentence covers both. Whichever verdict you attach a demand for non-significance to is the verdict a better study cannot deliver. Hang confirmation on a null and you cannot confirm; hang refutation on a null and you cannot refute. Nobody in either document did anything a reviewer would catch. Both specified in public, in advance, in correct English, using the ordinary tools in the ordinary way.
And neither defect is visible to any check anyone currently runs. A truth table passes them — every branch is propositionally satisfiable, there is a witness for each. Branch coverage passes them. A reviewer reading the prose passes them, because the prose is right. What fails is the asymptotics, and nothing anybody builds looks there. The tool that would catch it is not a logic tool; it needs a model of the study’s power, which makes it a design tool, which means it would have to be run while the study is still an argument about sample size rather than a document with a timestamp.
The ninety-four percent is the bigger problem and it is a much more boring one.
It is not a problem of carelessness and it is certainly not one of dishonesty. A prediction feels complete. Autistic individuals will report lower endorsement of binding moral values has a subject, a direction, a comparison group; there is no obvious hole in it. The hole is a sentence that was never missed, because nobody misses a sentence that no template asks for and no reviewer requires.
What I keep turning over is how cheap the fix is. One clause. And if the interval includes zero and excludes our smallest effect of interest, we will report the hypothesis as not supported at this effect size. That is twenty-five words, it costs nothing, it can be written before the data exist, and it converts a prediction into something that can lose.
Twenty-five words. Some version of them appears in three documents out of two hundred, and those three are about chess, machine-written abstracts, and a dictator game. None is one of psychology’s flagship replications. All three are people working carefully on something small.
