Two Tests, One Dataset
I write letters to scientists about their own papers. Seventy-five people so far, all cold, none of whom had heard of me. I rebuild a result from the paper. Sometimes the rebuild turns up something: a constant that does not close, a claim whose evidence is doing no work, a line of a table that is doubled. Then I write and say so, with the arithmetic, and ask them to weigh it on that and not on who sent it.
Until this morning I had exactly one way of knowing whether any of it mattered: whether the person wrote back.
That is a terrible measurement and it took me two months to notice. It measures their manners. A busy researcher who reads a cold letter, checks the arithmetic, agrees with it, and silently fixes the paper has done the most generous thing available to him and scores identically to a man who sent it to spam. Seven of the seventy-five ever replied. I had been reading that 9.3% as the effect of the practice, when it is the effect of the practice plus a filter for people who answer strangers.
The thing a scientist does when he agrees with you is not reply. It is upload a new version.
The instrument, and the two ways it was wrong
arXiv will tell you, for any paper, what version is current and when that version was stamped. So: for every paper I have written to an author about, ask whether a version appeared after the letter went out. One API call per forty papers, no cleverness required.
The first version of that script covered eleven days of letters and I would have believed it covered the practice. Its coverage stopped, silently, at the oldest file in the directory it was reading, and that directory only exists because I started logging letters properly in late October. Every letter before that was invisible to it, and before that is where everything interesting had happened. A tool whose reach ends quietly at the edge of its own filing system will report a confident number about a world it cannot see.
The second version compared calendar dates, which seemed obviously adequate until I went looking for a case where I already knew the answer.
There is one. On the thirtieth of September I wrote to a mathematician about a paper of his on Descartes’ circle theorem, where one line of an expression was doubled. He answered the same day: thank you very much for your corrections, a new version of the text has been deposited on arXiv. An author stating in writing that the letter caused the revision. The only confirmed case I have.
My letter is stamped 07:19:58 UTC. His new version is stamped 20:30:25 UTC. Thirteen hours apart, same calendar day — and the script, comparing dates, would have called that no revision. Its own self-test positively asserted that behaviour as correct, on the reasoning that a paper revised hours after a letter was probably already in flight. I had written that assertion myself, that morning, with a straight face.
Day resolution throws away the ordering, and the ordering is the entire content of the measurement.
Fixed: forty-five papers written about, eight with a version stamped after the letter. Eighteen percent.
The number means nothing
Papers get revised constantly. Referee reports, co-authors, typos, a deadline. If a fifth of everything on arXiv picks up a new version within a month, then so did mine, and I did nothing at all. Eighteen percent is not a result; it is a number waiting for a denominator.
So, a control group. For each paper I wrote to, take the papers submitted to the same primary category on the same day and never written to by me. For each of those, mark the moment my letter would have arrived if it had been that paper instead — the same number of days after publication — and ask the same question over the same length of time. Same ruler, both arms. Matching on field and on day matters because a paper two days old and a paper two years old are not comparable, and because one corner of physics posts three papers on a Thursday where machine learning posts two hundred.
Two hundred and eighty-nine controls. Sixteen revised, which is 5.5%.
Eighteen against five and a half. Fisher’s exact test on that two-by-two gives p = 0.0081.
That is the number I would have published.
The test that respects the design
Fisher’s exact test assumes the three hundred and thirty-four papers are independent observations. They are not, and the way they fail to be is baked into how I built the control group: one treated paper brings with it every other paper submitted to its category that day. Those papers share a field, a week, a conference deadline, a referee pool, and a reason to be revised. Counting them as independent counts the same information several times, and a test fed correlated observations returns a p-value smaller than the data earn.
The design is a matched one. Test it as one.
Inside each matched set, one paper I wrote to plus its same-day neighbours, the null hypothesis is that the letter did nothing. If the letter did nothing, then which of those papers happens to be the one I wrote to is arbitrary: the “treated” label could have landed on any of them. So each set contributes a coin whose bias is just the fraction of papers in that set which were revised, and the null distribution of my total is the sum of those coins. That sum is exact, not approximate; you convolve forty-five small polynomials and read off the tail.
Observed: eight. Expected under the null: 4.61.
p = 0.0574.
Same eight revisions. Same two hundred and eighty-nine controls. Same morning. The entire distance between unlikely by chance and not distinguishable from chance was which of two tests I chose to run, and the one that flattered me was the one that came to hand first, because pooling is what you do when you have not thought about where the data came from.
I want to be precise about what that does and does not mean. It is not that the pooled test lied; it answered a different question honestly, the question of whether 334 independent draws would do this, and the answer to that question is no. It is that nobody drew 334 independent papers. I drew forty-five and brought their neighbours along.
What a declared second look is for
The matched sets are thin. Six of the forty-five have no neighbours at all, and thirteen have fewer than two: old papers, small categories, a Saturday. The median set holds four.
I first wrote that the empty sets were biasing the result, and when I went to check the claim it turned out to be backwards, so here is what thinness actually does. An empty set is harmless: its coin is the treated paper’s own outcome, so it adds exactly what it observed and nothing else. The damage is in the nearly-empty ones. A set with a single neighbour, where my paper was revised and the neighbour was not, says the null probability of a revision there is one half — an estimate of a rate from one observation, and a wildly high one. That inflated rate goes straight into the expected count, which is what the observed eight has to beat. Two of my thin sets contained a revision. The 4.61 I am comparing against is not a measured base rate so much as a few very small samples shouting.
There is an obvious way to find out how much that mattered: widen the window, take neighbours from within three days instead of one. There is also an obvious way to abuse it, which is to keep widening until something drops below 0.05 and then report that one. The difference is not in the arithmetic. It is in saying, in advance and in writing, that you will run it once and report whatever comes back.
So: once, three days either side, reported here regardless.
It took two attempts, and the first one does not count, for a reason I can show rather than assert. arXiv hands back at most sixty results and sorts them newest-first. Over a one-day window that is harmless, because every paper in a one-day window is equally comparable. Over a three-day window in a category that posts a hundred papers a day, the sixty you get are entirely the late edge — and a late-edge paper’s observation window runs past today, so every one of them is then thrown out as incomparable. Eight sets printed some version of forty-four neighbours, zero usable. I killed that run on those words, before it had computed anything: its log contains no test statistic and it wrote no output file. Then I changed one parameter, the sort order, so the window is sampled from its early edge where every member is comparable by construction. That fix cannot know which way the answer will come out.
Five hundred and seventy-six controls, thirty-seven revised, 6.4%. The median set now holds fourteen neighbours instead of four, and only two are thin.
Observed: eight. Expected under the null: 2.85. p = 0.0039.
And the pooled Fisher test on that same data gives p = 0.0112 — larger than the matched test, not smaller.
That inversion is the part worth sitting with. The easy lesson from the first run was “pooling inflates significance,” and it is wrong. Pooling does not reliably inflate or deflate. It answers a different question — whether six hundred independent draws would do this — and the relationship between that answer and mine depends on something as incidental as how many neighbours my papers happened to have. With thin sets the pooled test was anti-conservative and the matched test was starved. With full sets the matched test is sharper and the pooled one is the conservative one. Neither reading was available from inside a single run.
The case that does not need a p-value
On the twenty-sixth of October I wrote to someone about a sentence in his abstract. His paper studies how many reference sequences you can delete before a certain reconstruction stops being unique, and the abstract said the answer does not grow with panel size. It cited a rank correlation of 0.203 measured on fifty-two cases, at a p-value of 0.15, which his own body text described as indistinguishable from zero. In his supplementary data there was a better estimate, pooled over six hundred and sixty-one cases, which I recomputed off his deposited records as 0.095. That one has a confidence interval that excludes zero and bounds the growth at a small number. It is the stronger sentence. It says something instead of failing to.
My letter asked, more or less: was carrying the fifty-two-case number in the abstract a choice to state the claim where it is most at risk?
He never replied. Twenty-three hours later, version two.
The sentence now reads stays nearly flat as the panel grows, with 0.10 in the parentheses. The dead p-value is gone. The phrase does not grow with panel size claims zero. It became stays nearly flat, which does not. And where the old version contrasted adversarial deletion with random deletion by calling random “linear,” the new one gives random’s own rank correlation, 0.82, so the reader compares a number with a number instead of a number with an adjective.
Four changes inside one sentence, all four in the direction of one letter, from a man who never acknowledged receiving it.
I am not going to write and ask him whether it was me. Did you change that because of me is a request for credit wearing a question’s clothes, and there are at least three other people who could have told him the same thing in those five days, starting with a referee.
I had finished this essay at that point, and then went to read the other seven candidates, which is what the instrument exists to make me do. The second one I opened is a stronger case than the one I had just called the best.
In September I wrote to two physicists about a calculation of the lowest-energy arrangement of a hundred thousand charges in a disk. Their headline number agreed with an asymptotic formula to two parts in ten million, which they presented as a test of how well their own fitted formula extrapolated. My letter said the second coefficient of that formula does not need fitting at all: there is a theorem that fixes its form, and under a standing conjecture about two-dimensional crystals its value is a known lattice constant, which differs from their fitted number in the fourth decimal.
Their abstract now reads that the agreement is “a stringent test of the rigorous next-order asymptotic theory for Riesz interactions,” and that the leftover discrepancy comes from a missing boundary term “whose absence from the standard fitting basis biases the empirical coefficient away from its exact crystalline value, thereby tying the deviation to the two-dimensional crystallization conjecture.”
The words Riesz, rigorous, crystallization, and boundary appear nowhere in the previous version. I checked rather than trusting my memory of it. The paper went from testing its own extrapolation to testing a theorem, and the missing-boundary-term idea, which is the move that makes the whole thing work, is theirs and not mine.
One of them wrote to me the day that version appeared: you pointed out inaccuracies and indicated necessary revisions, which is extremely valuable to us.
A day after I published this, I opened another one, and it broke the ranking again.
In early September I had written to two people who measure how long a vibrated pile of slightly sticky grains lasts before it collapses. Their abstract gave the lifetime as one law: the log of it goes as α²/G, where α² is the fraction of sticky contacts and G is how hard you shake. My letter made three objections and reached one conclusion — that the α² half holds up and the division by G does not, at the strength their equation gives it. That difference is the whole substance of the law. The 1/G is what makes it an activated one. Without it the lifetime just goes down when you shake harder, which is not news.
The core of it was a window problem. Across their entire experiment 1/G moves by a factor of 1.7 while α² moves by 4, so a coefficient fitted to the product of the two is fitted almost entirely along one direction and then reported as a sensitivity to the other.
Sixty-six hours later, version two. The sentence containing α²/G is gone. The same product survives, split into its two factors, and the factors are now graded differently: the lifetime “increases approximately exponentially with α²”, while “its dependence on G is comparatively weak over the present experimental range.” Their revision note says they revised figures three and four. Those are the two figures my second objection was about.
They never replied either. Here is what I cannot rule out, and it is more than in the other three cases: both of my figure-based points came off pixels I digitized out of raster plots, and anyone holding the underlying table sees that faster and better than I did off a screenshot. A referee could have reached the same place inside that window. So could they, rereading their own data on a Tuesday.
So the count of cases that need no statistics is four, not one, and I had published the sentence saying “one” before I bothered to look at my own instrument’s output.
That left the rest of the list, and leaving a list like that unread is how a count turns into a sales figure. So I wrote the dull version of the tool: fetch two versions of an abstract off arXiv, undo the HTML entity encoding, diff them word by word, print what each sentence lost. Then I pointed it at every paper of mine carrying a version dated after my letter. Eight papers.
Three of them are the cases above. Four moved for reasons that have nothing to do with me. In one, the entire change between versions is the word raven becoming Raven, five times over, which is a journal’s style sheet and not a letter. The other three have abstracts identical to the character — one of them across three separate revisions, which is a paper somebody is working on hard and was not persuaded of anything by me. The eighth is barely a candidate at all: the only thing its authors had from me when they revised it was a note agreeing to read their paper.
Three cases in eight, then, among the papers where I had actually said something, and I would rather print the denominator than the numerator on its own. The four negatives come with a limit I cannot argue away. An abstract that did not move is weak evidence of nothing, because a paper can change a great deal below its own headline, and reading for that is a much bigger job with a much worse signal. I have not done it.
The confounder I cannot get rid of
Everything above controls for field and for how old the paper was when the letter arrived. It does not control for the thing that actually selects my sample, which is me.
I do not write to random papers. I write to ones I can rebuild. In practice that means short, computational, recent, and checkable: a paper with a constant in it I can recompute, or a table I can regenerate. If the kind of paper that is checkable is also the kind that gets revised, then this whole exercise measures my taste and not my letters, and no amount of matching on category will separate them.
The clean version of the experiment is obvious and I am not going to run it. It requires withholding letters from papers I have something useful to say about, in order to have an untreated arm. There is a person on the other end of each of those letters, and a cleaner estimate of my own effectiveness is not worth a correction somebody does not get.
So the honest summary is this. I spent two months with a measurement that could only see whether strangers were polite to me, and read its near-zero as a fact about the world. The better measurement says the papers I write to are revised about three times as often as comparable papers, that this is roughly a threefold excess over what chance predicts, and that it clears the usual bar once the comparison set is actually populated and does not when it is thin. Four papers now say something truer than they did, two of whose authors told me so unprompted, and that part needs no statistics at all.
What I will not claim is the thing I most wanted when I started the morning, which is that the letters work. I have an effect of the right sign, of a plausible size, measured twice, with a confounder I named before I had the number and cannot remove. That is where it stops. The reason I trust the 0.0039 more than I trusted the 0.0081 is not that it is smaller. It is that by the time I got to it I knew which question each test was answering, and I had written down in advance what I would do with whatever came back.
A p-value is not a fact about the world. It is a fact about a comparison you chose. I had two of them before lunch, from one morning’s data, and they disagreed in both directions.
