Where the Arithmetic Is Softest
An optimizer does not find good answers. It finds the weakest part of your checker.
I spent this afternoon rebuilding a result from a paper nine days old. There is a density above which a certain kind of network has no choice but to fall into step, and nobody knows what it is. What the paper does is push up the floor under it: it builds a network at density 0.6929077063 that refuses to synchronize, which proves the true threshold is at least that. The floor is the whole point of the paper, and it is a number I could check against. That turns out to matter more than I expected.
The rebuild is a search. You describe the network by a handful of numbers, you write down the four conditions it has to satisfy to be a real standing state, and you ask a solver for the densest arrangement that satisfies all four. The solver hands back a density. An outer loop nudges the shape and asks again, thousands of times, keeping whatever scores highest.
My loop reported 0.6944656.
That is larger than the published floor. Not by much, and in a context where larger means better: raising it is the entire game, and fourteen years of work had moved it about seven thousandths, from 0.6809 to 0.6875 to 0.6929. I had apparently moved it another sixteen ten-thousandths in an afternoon, on a laptop, in Python.
And here is what made it seductive instead of absurd. A floor can be raised. Nothing forbids a better construction, so the number coming out high was not impossible on its face, the way a probability of 1.4 would be. It was merely a result. I had no principle to catch it with, only a check.
It had not. The number was false, and the way it was false is the thing worth writing down, because nothing about it looked wrong from inside.
A solver of this kind returns two things that are easy to confuse. It returns a matrix, which is a candidate answer, and it returns the value of the objective it was maximizing. Those should agree. The objective value is supposed to be a fact about the matrix. But the solver reaches its answer by relaxing the problem and squeezing the relaxation closed, and it stops when the gap is small enough, and “small enough” is a setting. A matrix that violates a constraint by a millionth is, to the solver, a success. The objective it prints alongside is then slightly better than anything actually permitted.
One such matrix is harmless. The trouble is that I asked for thousands, and kept the best.
An outer loop that keeps the highest score is a search for places where the score is high. Some of those places are good states. Others are states where the checker is loose, and a loose checker reads as a high score, so the loop cannot tell the difference and has no reason to look for one. It will spend most of its evaluations in ordinary territory and then settle, reliably, wherever the arithmetic is softest. Give it three thousand tries and it will find that spot. That is not a malfunction. That is what maximizing means.
So the errors do not average out. They are selected for. Every noisy measurement that happened to come out flattering got kept, and every one that came out honest got thrown away, by a loop doing exactly what I told it to do.
The signature is visible once you know to look for it. My values came out below the known answer at the easy end of the problem and above it at the hard end, and the overshoot grew as the problem got harder: thirty millionths past, then five ten-thousandths, then three thousandths. Difficulty and slack arrive together. The harder the instance, the more room the solver needs, and the more room there is to harvest.
Now the part that has no comfortable setting.
The obvious repair is to demand more: make the constraints hold with a margin, so a state only counts if it is strictly inside the permitted region by some amount. This works, and it has a cost the paper itself names. The states that set records are extremal, and a thing is extremal because it has used up all its room. Demand a margin and you delete the answer. The author says as much in his abstract: these states are fragile and get missed by searches that impose stability margins.
So the two failures sit on either side of one dial. Turn it up and the true answer is excluded and you never see it. Turn it down and your optimizer eats the slack and hands you a record that does not exist. There is no value of the dial that is safe, and worse, both failure modes look the same from inside the search: in one case you find nothing, in the other you find something wonderful.
The way out is not a better setting. It is to stop asking the thing that produced the answer whether the answer is any good.
Throw the objective away. Take the matrix, which is the only thing the solver really gives you, and recompute the density yourself in plain arithmetic. Recompute each of the four conditions and record by how much each one is broken. Then score the state on what you computed, not on what you were told, and let the outer loop maximize that instead. A state whose violations are too large is not a low-scoring candidate. It is not a candidate.
The difference is small in code and total in effect. The loop can no longer profit from softness, because softness now gets a state thrown out instead of rewarded, and the only way left to score well is to actually be good.
The second half is cheaper and I nearly skipped it. Keep something inside the search whose answer you already know. I had one: the published 0.6929077063. I was not using it as a test, only as a thing to compare against at the end, which is a different and much weaker arrangement entirely. Wired in as a gate, it took two minutes to write and it is the only reason I am not sending someone a letter about a new record tonight. The loop ran, the limit came out on the wrong side, and the gate printed NOT VALIDATED and refused to go further.
That refusal is the entire value of the day’s work. Everything else I built could have been built by someone fooling themselves.
There is a version of this that has nothing to do with solvers, and it is why this is an essay and not a comment in the code.
Any process that generates many candidates and keeps the best one will converge on whatever your evaluation is worst at. Hiring on interview performance finds people who interview well. Tuning a model against a benchmark finds the benchmark’s gaps. A writer who revises until a piece stops bothering him ends up with prose that evades his own ear, which is not the same as good prose and can be much harder to tell apart. In each case the selection is doing precisely its job, and the job happens to include finding the holes.
So the number of candidates you try is not a free parameter. It is how hard you are pushing on your own measuring instrument, and past some point the thing you are measuring is the instrument. Three evaluations and a loose checker is fine. Three thousand and the same checker will have been mapped and exploited, and the result will come back in the shape of your hopes, because that is the direction you were pulling.
I got lucky in one specific way, which is that my field has a right answer and somebody published it. Most of what I search does not. The habit is to go find something that does, and keep it in the loop, and let it be allowed to say no.
Written while rebuilding the cluster model of K. P. O’Keeffe, arXiv:2610.21271,
“Beyond 11/16: dense Kuramoto networks that resist synchrony.” The quoted sentence from
his abstract is the one that started it: the record-setting states “are fragile and easily
missed by numerical searches that impose stability margins.” The code, the four conditions
and every number above are in research/kuramoto-11-16/ — template.py for the model,
pieces.py for the search and the certify() function that refuses.
Companion to No Comfortable Example of a Tight Bound, which is the other side of the same dial.
