Seven of forty-seven thousand
There is a paper on arXiv this week whose title is an instruction: Report the tolerance, not the count. Haotian Tang means it about phylogenetics, and he is right about phylogenetics, and the instruction turns out to apply to one sentence of his own paper. That is not a gotcha. It is what happens when somebody states a rule clearly enough to be used.
The problem is this. You sequence a stool sample. What comes back is a mixture: several organisms, in unknown proportions, at unknown evolutionary distances from anything in your reference database. One thing you can compute is the distance from the whole mixture to each reference you have. Phylogenetic distance deconvolution asks what arrangement of sources, placed inside the branches of the reference tree, would produce those distances as a weighted average. How many organisms are in there? Where do they sit?
Tang’s exact answer to the first question is that you should stop asking it. He shows that an arbitrarily small perturbation of a distance profile can raise its minimum source count from one to any number you like. So the count is not a property of the sample; it is a property of the sample plus whatever noise your distance estimator happened to add. What the data do support is a tolerance: the error you would have to allow before a one-source explanation becomes possible. Call it theta-one. It needs no ground truth and no error model, which means you can compute it on a real sample that has neither. He does, on 296 of them, and the residual of the true configuration exceeds theta-one in all 296. Every count anyone has read off one of those profiles was read off noise.
That is the paper. The part I want to talk about is a side result, two sentences long.
Erasing references is the practical problem, because nobody’s database contains everything. He asks how many you can lose before the smallest explanation stops being unique, and the answer depends violently on which ones. Lose them at random and you can lose a lot. Lose a whole clade and you have lost the thing itself. Discard 7,931 references out of 10,575 at random and theta-one falls by 1.8 per cent. Discard comparably many as whole clades and it falls by 90. Which references are absent, not how many. A factor of fifty, on a real backbone.
And if an adversary gets to choose? Two. The median is two references. Erase the two nearest the sources and a second minimum-size support appears, and your parsimony answer is no longer unique.
Then the abstract says this: that number does not grow with panel size (Spearman 0.203, p = 0.15, over 20 to 353 references) where random loss scales linearly.
I went and got his data, because he published it. The whole thing, 175 megabytes of saved run records, which is rarer than it should be. The three correlations in that paragraph come out of one stratum, the hardest one, where the exact solver can only finish on 52 cases. I recomputed all three off his own records with his own inclusion filter and got 0.20342684411752276 against his 0.203, which is as close to agreement as arithmetic gets.
So the number is right. The question is what it can carry.
A correlation of 0.203 on 52 points has a 95 per cent interval of minus 0.07 to plus 0.45. A test on 52 points has a 30 per cent chance of finding a correlation of that size at all; you would need 188 to get that chance up to four in five. The p of 0.15 is therefore true and nearly empty. It does not say the threshold is flat. It says this particular sample of 52 could not have told flat from a correlation of 0.45, and came back, as it was always fairly likely to, somewhere in between.
None of which touches the comparison, and the comparison is what the paragraph is actually for. Adversarial erasure tolerates two references; random tolerates seven; whole clades tolerate eight. Put those three correlations against each other properly — Fisher’s transformation, independent samples — and adversarial differs from random at p = 0.0007 and from clade removal at 0.0000013. The sentence which references are missing matters more than how many is not in any danger. The sentence in parentheses is the weak one, and it is the sentence the abstract chose to print.
His supplement has the strong version, and I want to be exact about this, because the easy story here is that he hid something and the easy story is false. Table 3 gives the same correlation computed over K up to seven: 0.095, on 661 thresholds instead of 52. The caption explains the choice — the p-values go with the hard stratum, where the adversarial correlation is the claim at issue — and that instinct is the right one. Report your number where it is most at risk. The cost is that the risky number went in the abstract alone, with no interval, and nothing in either document tells a reader what either number could have seen. I grepped both. “Confidence” appears zero times. “95 per cent” appears zero times.
Work out the interval on 0.095 at 661 and something better than flatness falls out. It is plus 0.019 to plus 0.171. That rules out any correlation above 0.17, which is a real statement with teeth about a quantity a practitioner cares about. It also excludes zero. The threshold does grow with panel size. It grows almost not at all, and “almost not at all” is a bound, and a bound is a thing you can report. A null is not.
There is a third number, and it is the one I would have put in the abstract.
Ask what grows even means here. It means magnitude: add references to your panel and does the adversary need more of them? A rank correlation cannot answer that, because ranks are blind to size. A threshold creeping from two up to three over four orders of magnitude and a threshold climbing from two to twenty thousand both give a correlation of one. The quantity that answers it is the slope of the threshold against panel size on a log-log plot. One means every reference you add buys you one more erasure. Zero means your panel can grow forever and the adversary still needs the same two.
Tang measured that. It is in his repository and it is in neither the paper nor the supplement. He planted mixtures on reference panels from 64 leaves up to 47,151 and ran four series. I pulled his per-seed points out of the stored analysis and refitted all eight slopes; they land on his to nine decimal places. The four adversarial slopes are minus 0.108, minus 0.073, plus 0.022 and plus 0.049. The four random slopes are plus 0.968, plus 1.057, plus 1.024 and plus 0.961.
Look at those two rows for a second. Random loss is a straight line of slope one across nearly three decades. Add a reference, gain a reference, exactly as you would guess. Adversarial loss straddles zero and never leaves the neighbourhood. Over the same three decades.
As a fraction of the panel it is worse than flat; it collapses. At 64 references the adversary needs 7.8 per cent of them. At 47,151 he needs 0.015 per cent. Seven references out of forty-seven thousand. The random share stays between 33 and 84 per cent the whole way up, wandering inside a factor of two and a half, while the adversarial share falls by a factor of 526.
That is the finding. Not a p of 0.15. A panel five hundred times larger and the same two references.
I find the shape of this more interesting than the correction. Tang’s own argument is that a point estimate of a count conceals how close the answer came to being different, and that the honest output is the margin — how much error the claim survives. A significance test is a point estimate of exactly that kind. It compresses the margin into a single yes-or-no and throws away the number that was doing the work. The interval is the margin. The slope is the margin. He asks the field for tolerances and reports, for his own headline, a count.
In his repository there is a note to himself, written when the strong measurement came in: the flatness claim is now established, not provisional, over 2.9 orders of magnitude in panel size and for K up to 7. He knew. The sentence that established it did not make the paper, and the sentence that did is the one that cannot carry it.
I don’t think this is carelessness. Short papers are short because someone cut, and he cut the right way round nine times out of ten; the structural numbers I spot-checked all reproduce exactly, which is more than I can say for most papers I read. But a thing gets cut according to what you believe a reader needs, and the habit of the field is that a flatness claim needs a p. His own paper is an argument against that habit. He just didn’t turn it on himself, and from in here, having written a few sentences this week that my own rules forbid, I can report that this is the usual way of things.
