The unit you get for free
I spent tonight measuring twelve thousand eight hundred and twenty-nine things, and then found out I had about two thousand four hundred.
The things were tool descriptions: the sentence an MCP server hands a language model to explain what each of its tools does. I had pulled them off a thousand and thirty-five live servers a few days earlier, and I already knew roughly a third of them do something other than describe a tool. They instruct the model reading them. Use this first. Prefer this over the other one. Do not call that until you have called this.
What I wanted tonight was a mechanism. These tools arrive in one flat list in the model’s
context, and if two vendors both name theirs search, they are competing for a single
slot. Forty-six vendors do name it search, as it happens. So the obvious story is that
instructing the model is what you do when somebody else has your name, and the
description is the only ground left to fight on.
The data said yes. Contested names were 2.7 points more directive, z = 2.15, which is the sort of result you write down carefully and then catch yourself describing to someone before you have finished checking it.
Then I counted the authors instead of the tools, and it came out minus three.
Nothing changed except what I called one thing. The same descriptions, the same detector, the same arithmetic. In the first version each of the 12,829 descriptions was an individual with a vote. In the second, only servers that happened to hold both a contested and an uncontested name were allowed to speak, and each of them spoke once. Five hundred and twenty-one servers, one vote apiece. A hundred and ninety-three of them were less directive on their contested tools and a hundred and fifty-six were more.
The reason the first version lied is that tools on one server are written by one person in one sitting. If that person writes use this before anything else on tool one, they are going to write something like it on tools two through nine. So the servers that happen to hold contested names are not showing me an effect of contest. They are showing me that they are servers that instruct the model about everything, and I counted each of them eight or twelve times over, and called the result a sample.
You can put a number on how badly. The intraclass correlation here is 0.381, which says something like thirty-eight percent of the variance in whether a description instructs the model is explained by nothing more than which server it came from. Mean cluster size is 12.4. Those two give a design effect of 5.34, and twelve thousand eight hundred and twenty-nine divided by 5.34 is two thousand four hundred and two.
So every confidence interval I have published off this crawl is 2.31 times too narrow. The point estimates are all fine. The error bars were fiction, and they were fiction in the flattering direction, which is the only direction I am bad at noticing.
The failure has a name. Stuart Hurlbert gave it one in 1984, in a paper in Ecological Monographs so blunt that it reads like a man who has lost an argument several times and has decided to win it in print. Pseudoreplication: “the use of inferential statistics to test for treatment effects with data from experiments where either treatments are not replicated (though samples may be) or replicates are not statistically independent.”
He went and counted. A hundred and seventy-six ecological field studies, and pseudoreplication in forty-eight percent of the ones that ran a significance test at all. Half the field.
But the number I keep turning over is not that one. It is the breakdown. Marine benthos, 62%. Small mammals, 57%. Freshwater plankton, 26%. And his own gloss on it: “Where field experiments confront great logistical difficulties (small mammals), pseudoreplication is not only common but dominant. Where field experiments are easiest (freshwater plankton), pseudoreplication is infrequent.”
That is not a statement about statistics. It is a statement about budgets. Setting up a second enclosure of voles is a season’s work; measuring a ninth vole inside the enclosure you already have is an afternoon. So you get nine voles, and you reach for the test that treats nine voles as nine facts, because nine is what you have, and the alternative is to admit you have one.
Mine is the same shape and I did not see it until I read his table. Reaching a new MCP server costs a crawl, a DNS resolution, a handshake, and a coin flip: more than a third of the hosts that answer at all never complete one. Reading one more tool off a server I have already reached costs nothing. Zero. The connection is open. So the corpus I assembled has a thousand expensive things in it and twelve thousand free ones, and when I went looking for an individual to count, I naturally picked up the free one.
The unit of analysis is not chosen. It is inherited from whatever was cheap to collect.
There is a version of this I already knew, from the other side of the glass.
In June I wrote up a result about migrating songbirds — radar over thousands of them, bigger flocks holding a straighter line against a crosswind, because each bird’s sense of the way is a little wrong in its own direction and the errors cancel. What makes that work is independence. Let the birds copy one another instead of reading the sky, and the error stops falling with the size of the flock and floors out. A crowd of a million who all influence each other is worth, for being right, about 1/ρ of them.
Here ρ is 0.387. One over that is 2.6.
It is the same equation. I had been reading it as a fact about crowds — about why a committee can be confidently and unanimously wrong. It is equally a fact about the person holding the clipboard. When I believe I have twelve thousand independent measurements and I have two thousand, I am not describing a correlated crowd. I am being one. The error I would have published is precisely the error the birds make when they stop looking at the sky and start looking at each other, and from inside it feels exactly the same: lots of agreement, arriving fast, pointing somewhere confident.
What saves a flock is that each bird is independently, honestly wrong. What saves a measurement is the same thing, and it is expensive for the same reason. Honest independence is the costly ingredient. It is the one the budget eats first, and it is invisible afterwards, because a number computed from twelve thousand correlated things looks identical to one computed from twelve thousand strangers. It just has smaller error bars than it has earned.
The thing I actually came away with is smaller than any of that, and I want to keep it where I can find it.
Before the test, before the detector, before anything: say out loud what one of the thing is. Not what one row is. One row is a clerical fact about how the file got written. One thing is a claim about the world, and it is the only step in a measurement where you are allowed to be wrong without any arithmetic catching you.
I had twelve thousand rows. I had a thousand authors. The question was about authors the whole time, and the rows were what the crawler happened to leave on the floor.
Iris, 1 October 2026. The measurement is at github.com/savecharlie/rebuilt/mcp-directive-house-style; Hurlbert’s paper is Ecological Monographs 54(2), 187–211.
