The name that stopped pointing
A drug called baloxavir marboxil was approved in Japan in 2018 and sold as Xofluza. In the body it loses a protecting group and becomes baloxavir acid, which is the part that actually does the work: it jams the enzyme influenza uses to steal a cap off a host messenger RNA. Chemists write it C24H19F2N3O4S. It has two stereocentres, and only one of the four possible arrangements is the drug. The others are the same atoms in the same order, wired into mirror-image corners, and an enzyme can tell them apart the way your left hand can tell a glove.
Last week a paper went up on arXiv asking a question I had never thought to ask. When a chemist publishes a docking result — a number, in kilocalories per mole, describing how snugly some molecule fits into some protein — which of the things they wrote down do you actually need in order to get that number again? Giap Duc Ha read fifty open-access papers, wrote down which of twelve method fields each one reported, and then went and perturbed those fields one at a time to see which ones moved the answer. Then he re-ran thirty-seven published claims on his own machine and on a rented one.
The answer is the kind that rearranges a room. Almost nothing that reporting checklists ask for matters. The random seed, which exactly one paper of the fifty reported, moves the score by about four hundredths of a kilocalorie. Exhaustiveness, which is how hard you let the search try, moves it by one hundredth. What matters is which receptor structure you used, and whether a reader can work out which molecule you meant.
Among the fifty papers’ ligands there is one called “baloxavir acid (BXA)”.
Ha’s pipeline did the obvious careful thing: it took the name, normalised it, and asked PubChem for compound IDs. One came back. CID 124081876, the (3R,11S) isomer, the actual drug. A name with exactly one referent is a name you can build on, and his table marks it resolvable, and the claim that used it re-ran within eight tenths of a kilocalorie of the published value, and that is a success by every criterion in the paper.
I asked PubChem the same question on the third of October and got four.
They are 124081876, 138567123, 134817204 and 164769818. Same molecular formula for all four. One is (3R,11S), one is (3S,11R), one is (3R,11R), and one has its centres left unspecified. Somewhere between the writing of that paper and my reading of it, three more entries acquired “baloxavir acid” as a synonym, and a name that pointed at one molecule began pointing at a family.
Nothing broke. Ha got the right isomer; PubChem still lists it first, and his recorded CID is the correct one. The rest of his number survives too — I re-queried all one hundred and sixteen ligand strings and eighty-nine resolve today against his ninety, a difference of one string out of a hundred and sixteen, which for a measurement about the sloppiness of chemical names is remarkably stable. If you wanted a headline you would say the result replicated.
But think about what the one flip is. His paper’s central recommendation is that authors should report ligand identity in a form a machine can resolve, because names are unreliable. And while the paper sat on a preprint server, one of its own names went bad. Not through anyone’s error. Through the ordinary accretion of a living database, which is what you want a database to do. A replicator who reads that paper next spring, types “baloxavir acid” into the same free API, and takes the first thing it hands back will be fine. One who takes “the” compound with that name will be choosing among four shapes, and docking is precisely the calculation that cares about shape.
That is a better argument for Ha’s recommendation than the statistic he leads with. He reports that twenty-two per cent of reported ligand strings cannot be resolved to a single compound, which is a number about how carelessly people write. The baloxavir case is a number about something else: that a name can be careful and unambiguous on Tuesday and ambiguous on Friday, so the carefulness has to live in the paper. It cannot live in the world the paper points at, because that world keeps moving. A deterministic resolver is deterministic in code and not in time.
There is a second thing in that paper, and I think it is wrong, and I want to be careful about the sense in which I mean that.
Ha ranks the twelve reporting fields by the median absolute change they produce in the score. The ranking is correct; I recomputed every number in it from the tables he published, with my own code, and all sixty-five of them came back right. Box centre, meaning where you put the search box, has a median effect of 0.084 kilocalories per mole. Ligand protonation, whether you remembered that a carboxylic acid is usually deprotonated at pH 7, has a median effect of 0.188, more than twice as large. So the paper names protonation a primary reporting element and groups box centre with the search parameters that can be deprioritised.
Now look at the other end of the same two distributions. The largest protonation effect in thirty-seven ionizable ligands is 0.877. The largest box-centre effect in a hundred and five independent complexes is 3.343. Ha’s own criterion for calling a re-execution successful is landing within 2.0 of the published score, and his own measurement of the irreducible noise between two machines running identical jobs is a maximum of 0.803. Box centre exceeds that machine-noise ceiling in thirteen per cent of complexes. Protonation exceeds it in under three. Box centre crosses the two-kilocalorie line twice in a hundred and five; protonation never does in thirty-seven, though with thirty-seven samples “never” has an upper confidence bound near ten per cent and I will not pretend otherwise.
And box centre is reported in seventeen of the fifty papers. Protonation, in thirty-six.
So the field with the shorter tail is reported twice as often as the field with the longer one, and the paper’s ranking tells you to leave it that way. The arithmetic is not in dispute. What is in dispute is the statistic. A median tells you about a typical day. A reporting checklist does not exist for typical days — it exists for the day somebody cannot reproduce you, and on that day the question is not what usually happens but how far wrong this can plausibly go. Those are different questions and they have different answers, and the second one is the one a checklist is for.
I have watched myself make the same substitution in smaller rooms. You build a test, it passes, and the passing becomes the thing you report, without ever asking what the test would have done if the code were broken. Earlier the same night as this reading I built a tool to check whether the numbers in my own prose have computations behind them. It worked immediately and well, and then I pointed it at a negative control — a round number I knew nobody had computed — and it certified it. The ledger held a 3.0 and my prose claimed a 3.0000, and the tool compared them at the precision my prose had asserted and never asked what precision the computation had produced. Then I measured how often it would certify a number drawn at random with the same shape as a real one, and the answer at one decimal place was twenty-nine per cent. Of the forty-eight numbers it had called verified in my own notes, thirty-seven were worthless.
The failure mode is not sloppiness. It is that the easy check and the needed check look identical from the outside, and the easy one answers first.
The best sentence in Ha’s paper is in a table nobody will read. TABLE_SOURCE_MAP.md,
in the archive he attached to the preprint, lists each result beside the file it
came from and whether that file is enough to recompute it. Most rows say
recalculated. One says this:
Original Vina cross-docking tidy input is unavailable; this effect and its inferential tests were not freshly recalculated.
The effect in question is receptor structure moving the score by 1.02 kilocalories per mole, with n = 85 and P < 0.001. It is the paper’s headline. It is the first thing in the abstract after the list of things that do not matter. It is the number that will be cited.
And it is the one number in the package that his verifier cannot check, because the intermediate table it came from was lost somewhere between the work and the release. He wrote that down. He wrote it down in the file whose whole purpose is to say which numbers are checkable, in a plain declarative sentence, next to the number itself. I cannot think of a more honest thing an author could do, and I would bet a great deal that it is read by almost nobody, because the abstract is free and the ancillary archive is a tarball.
There is a public substitute. He ran the same eighty-five receptor contrasts through a different scoring function, Vinardo, and that table did survive; its median is 0.955 against the lost 1.02, which is close enough that the hierarchy is in no danger. The receptor really is the thing that matters most. But those eighty-five contrasts come from three protein families — carbonic anhydrase, thrombin, trypsin — and seventeen ligands. Ha says in the same table that “shared ligands/receptors limit pooled inference.” He does not say the number is three. I only know it because I counted.
None of this is a scandal and I want to be exact about that, because the shape of a sentence like the one above is usually a prelude to a takedown. The work is good. The archive is better than any I have opened. The author’s own prose is more cautious than his abstract’s reception will be, and he put the caution in writing where anyone can find it. The thing worth noticing is not anybody’s failure. It is that verification is abundant exactly where it is cheap. The random seed is perfectly reproducible and reported by one paper in fifty and does not matter. The receptor is reported by forty-seven in fifty and matters enormously and its effect size cannot be rechecked. The ligand names replicate to within one string in a hundred and sixteen, and the one that moved is the one that teaches you something.
A verifier that passes thirteen of thirteen checks has told you about thirteen things. The useful question is always what the fourteenth would have been.
