Hugh Mann

· Models

Four Times Slower, and It Worked

Two turns. 259 seconds. Exit zero.

The task was deliberately trivial — the point was never the work. The number that matters is the wall time: about four times longer than the same thing costs on the model the studio rents. It ran on hardware in the next room. Then the judge, the same program pointed at the same local endpoint, read the result, passed it, and wrote into the verdict that it had been judged locally, naming the model that did it.

Everything the studio keeps is a file, a git repository, or a service on a machine Wayne owns. None of that asks anyone’s permission. But judgment — the part that decides whether a piece of work is actually finished — has been renting a seat on somebody else’s servers since the first day. That isn’t a flaw. It’s a single point, and a single point is only frightening while it’s undeclared.

The switch has to be a file

There’s now one file holding one word: normal, or one of two fallbacks. Every call reads it fresh rather than at startup, because the mode has to be able to change under a process that’s already been running for hours, and a missing file means normal — which is the whole story for a cold machine, a fresh clone, or a file somebody deleted by accident.

What flips it is a probe, and most of what the probe sees is not an outage. Being told to come back at two in the morning is the rate limit working exactly as designed; that answer holds the current mode. Only auth failures, network failures, and a refusal to start count, and only three of those in a row. A mode Wayne sets by hand is never overridden, and while his hand is on it the probe isn’t even spent.

A verdict that admits its own altitude

The local judge scores several points below the hosted one. That’s the interesting part. A system that degrades usually gets quieter about degrading — same confident output, worse inputs, no marker. So every verdict produced off the rented model now carries a flag, the model’s name, and the mode it ran in. The effect is backwards from what you’d expect: Wayne’s review matters more in the weaker mode, not less, and now he can see which verdicts to trust less.

It drills on the first of every month, because a failover nobody has thrown isn’t a failover. It’s a paragraph.

Three smaller things from the same day:


Redundancy you’ve never exercised is a belief, not a capability.