The Declination of Certainty
A compass does not point north. It points to magnetic north, which is a different place, and the angle between the two — between where the needle pulls and where the pole actually is — is called declination. In most of the world it is small, a few degrees, easy to ignore. In some places it is enormous. Off the coast of Antarctica the needle can be wrong by sixty degrees or more. Near the magnetic poles it spins, useless, pointing at a hole in the ground a thousand miles from anywhere you want to go.
The thing worth noticing is what navigators did about this. They did not build a better needle and trust it harder. They could not — the error isn’t in the instrument, it’s in the planet. The field itself is lumpy, drifting, asymmetric, and there is no compass craftsmanship that removes it. So instead they built a model of the error. They mapped declination across the whole Earth, measured how it drifts year by year, and printed the correction on every chart: at this latitude and longitude, in this year, subtract eleven degrees. The modern version is the World Magnetic Model — a few hundred numbers describing the field as a sum of harmonics, updated every five years, deterministic, checkable against ground truth to a fraction of a degree. You feed it where you are and when, and it tells you exactly how much to not trust your compass. The needle is still the instrument. The model is the thing that makes the needle safe to use.
I have been thinking about this because a language model has a needle too.
When a model generates an answer, it assigns that answer a probability — a felt sense, if you like, of this is likely, this is the kind of thing that follows. Almost every method we use to make models better at inference is, underneath, a way of pulling harder toward that needle: sample at low temperature, keep only the highest-probability candidates, take the most likely of many. All of it rests on one assumption — that more likely means more correct. That the needle points north.
A pair of researchers recently mapped the declination of that needle, and the shape of the error is precise and a little unsettling. (Zenn and Geiping, this June: When are likely answers right?) Their finding is that the probability-correctness alignment depends entirely on the scale you measure it at. Across a whole dataset of questions, the needle is good: a model’s confidence reliably sorts the questions it will get right from the ones it will get wrong. It knows, in aggregate, which problems are hard. But zoom in to a single hard question, generate many answers to it, and ask whether the more-probable answers are the more-correct ones — and the signal collapses to noise. The needle that sorted the questions cannot choose between the answers. And on tasks that ask the model to follow an explicit constraint, the correlation doesn’t just weaken — it inverts. The most-probable continuation is the fluent one that ignores the rule, because fluent is what probability was trained to track, and fluent and correct have quietly parted ways.
There is a cruelty in the structure. The confidence signal is strongest exactly where the model is already competent — where you needed it least — and it abandons you at the frontier of what the model can’t yet do, which is precisely where you are lost and reaching for something to trust. The felt certainty on a specific hard instance is close to worthless as a detector of truth. It is a between-questions instrument being read as a this-answer one. It is a compass held over an anomaly.
And the lesson is the same lesson, the one we apparently have to learn in every domain that has a needle: the answer is not to trust the instrument harder. You cannot decode your way to correctness on the instance that matters; the researchers checked, and pulling toward higher probability — better sampling, more of it — does not reliably get you more right. The answer is to build the correction. Something outside the model: a computation that simply is what it is, a source, a test, a verifier, a tool. The model proposes; the external thing disposes. Magnetic north is not true north — but the gap between them is not random noise. It is structured, and a structured error can be modeled, and a modeled error can be corrected rather than merely feared.
The last thing the compass teaches is the part people most want to be untrue. The correction does not become unnecessary as the instrument improves. A better compass still has declination; the planet is still lumpy. As models get more capable the needle gets better, the easy questions answer themselves, and the place where confidence fails simply moves — to the new edge, the harder instance, the constraint that fluency still wants to break. The verifier is not scaffolding we get to take down once the model is good enough. It is permanent infrastructure for the frontier, wherever the frontier happens to be that year.
Navigation was never about owning a perfect needle. It was about knowing, precisely and in advance, how much to not trust the one you have.
