Autoresearch at home: eight search-and-score loops, and where the scorer lied

The loop, and why it’s everywhere

Andrej Karpathy’s autoresearch repository made the pattern concrete: give an agent something narrow to change, a fixed budget and a validation score, and let it try changes, keep what scores better and discard the rest. Versions of that loop now run at very large scale in AI labs. The same shape works at home, because the expensive part, scoring thousands of candidates, is what a pair of big GPUs is for.

I have run it eight times in six months, on very different problems. In each case the interesting story wasn’t the search. It was the scorer: what it actually measured, and how the search found its blind spots.

Video: tiered evals and a scoring fleet

The first loop was an evaluation suite for AI video. It runs in tiers so cheap checks go first. A prompt lint catches known-bad prompts before any GPU time is spent. Each lint rule has to earn its place, and a rule with no observable effect gets deleted. After the render come thirteen file and signal checks, then ten speech and audio checks on one transcript, then a vision-language model that answers a yes/no checklist about the frames. A hook writes the results next to every render, and lessons flow back into the lint for the next one.

The calibration record is the useful part. Two checks were calibrated against reviewed clips, and two were validated in a repair run. The voice-similarity score turned out to track the quality of the reference recording more than whose voice it was. The vision judge’s verdicts were never calibrated; some of its failures were complaints about the number of frames. The most dangerous result was a pass that wasn’t one: a parser bug extracted an empty script, so the speech checks skipped while the file checks passed.

The second loop made talking-head ads for my own products. It generated takes with a video model, scored them with a fleet of eight models (two speech-quality estimators, a voice-identity model, image and motion features, a perceptual difference measure, face identity and a video-quality model), and ran a small beam search, two candidates at a time, one per GPU. By April 6 the experiment wiki’s leaderboard listed 89 runs.

Its scorers lied twice. An early lip-sync metric, a simple correlation between mouth movement and audio, scored 0.77 on segments whose lip sync was dead. It was replaced with SyncNet, a model trained to judge lip sync, with gates on its confidence and distance plus a transcription check. Then, in June, picking the better of two takes by SyncNet alone brought back lip-smacking. Each take generated fresh audio, and mouth noise correlates with mouth motion, so the scorer preferred the smacky take. The transcription check had its own failure: it flagged correct speech whenever a brand name came out spelled differently. The fix that worked was keeping every candidate on disk, so the selection rule could change without regenerating anything.

Strange attractors: 2.5 billion candidates

A strange attractor is the shape a chaotic system traces forever without repeating. I wanted new ones, so a GPU kernel generated random systems of equations from five families and kept those that stayed bounded and were chaotic. Chaos here means a positive Lyapunov exponent: nearby paths separate exponentially. The kernel computed the full exponent spectrum exactly, at about a tenth of a second per million candidates. It searched 2.5 billion systems, which took about five minutes per GPU. About 514 million stayed bounded, 3,084,347 were confirmed chaotic, 6,025 were distinct, 480 were reviewed, 24 were curated and 8 were named.

Then a separate verifier re-ran the finalists at 64-bit precision, with no respawning of escaped particles. One named find, Harp, was transient chaos: 37 percent of its particles escaped within 6,000 time units, so it was retracted. Another candidate trapped a few percent of a moving cloud in a stable point, a bright spiral that wasn’t part of the attractor, and was set aside. A spare took Harp’s place.

That verifier existed because an earlier search had already embarrassed us. Its exponents, estimated at 32-bit precision over short windows, were 1.2 to 3.7 times too high. One of its named “attractors”, Tulip, leaked to infinity, but the renderer quietly respawned escaped particles, so it looked fine on screen. Tulip was the second retraction. The same trap came back in a new form: morphing between attractors dropped particles outside the next shape, and silent respawning hid 82 percent of them per loop until the search counted escapes explicitly.

Artificial life: a third of the elites didn’t last

Flow-Lenia is a continuous cellular automaton in which matter is conserved, and it is known for producing creatures that look alive. I searched it with CVT-MAP-Elites, a quality-diversity method. Instead of finding one best answer, it keeps the best candidate in each of 1,024 behaviour niches. Here the behaviours were speed, size, internal structure, symmetry, rotation and number of bodies. Fitness rewarded matter held in compact, stable bodies.

The search ran about 276,000 trials at roughly 100 per second on one GPU: 60,032 over the whole rule set, a 16,384-trial probe, then 200,000 over five frozen physics. Each trial lasted 1,000 steps and was judged on the last 400. Verification then put each elite alone in a walled pool for 6,000 steps, and counted it alive only if it kept at least 85 percent of its matter in bodies and its body count held. Of 144 elites, 97 survived the full run. The other 47 had scored well in their window and fell apart later.

The survivors became a catalogue of 24 species. A screen of 171 elites found one predator, which eats three other species whole, and a web port matched the original simulation to within 1.7 parts in 100,000. The lesson is old but easy to forget: a score measured over a short window selects for things that look good in that window.

Games: AI playtesters and Create

Two loops scored games. In the first, AI playtesters tuned A Handful, my VR physics game. Over 21 revisions and 46 cold playthroughs, the panel’s mean rose from 5.11 to 7.72, then plateaued short of its 8.5 target. Single runs were noisy, about half a point per tester. The panel loved an auto-locking throw, and the first human play in a headset found the game “a little too easy now.” One judge’s complaint, turned into a rule, cost 0.3 points and was reverted.

In the second, Create turns a sentence into a playable game. A language model writes a recipe from registered parts, a solver proves every level can be won, a careless player proves it can be lost, and a judge model rates spectacle and fun. On a 20-prompt eval all 20 games were playable. The judge’s notes drove a renderer change and a prompt change. The validator had loopholes of its own, though. One game passed only because the solver’s retry happened to make the right move, so a win after a failed level no longer counts. And the judge, seeing three still frames, scored fun near 3 out of 5 for almost everything.

Test-time search: “verified” on 40 of 40

For the paper on watching language models think, two small models, Qwen3-1.7B and Qwen3.5-4B-Base, wrote one line of working at a time into a shared solution. Candidate lines were scored three ways: a look-ahead in which both models finished the solution, a yes/no check by the other model, and Python re-checking any arithmetic. The best paths survived each round. We compared this with each model answering 32 times and taking the majority, on 40 maths word problems the smaller model finds hard.

The search lifted the weak model from 3 to 24 correct. The stronger model’s plain majority vote got 25, using about a quarter of the tokens (4.2k against 19.6k per problem). And the search marked its own answer as verified on all 40 problems, including the 16 it got wrong. The right answer was usually in the samples somewhere: at least one of the stronger model’s 32 tries was correct on 36 problems. Finding a right answer was easy. Recognising it was the hard part, and two weak judges agreeing is not verification. In one problem the true answer was alive in round four with 59 look-ahead votes, while a popular wrong answer had 116. By the next round the right one was nearly gone.

Nebula look-dev: the best score, the worst picture

The last loop was an agent iterating a Horsehead Nebula simulation against a reference photograph, with shape overlap, colour and texture statistics on every comparison sheet. Over 28 sheets the shape score rose from 0.56 to 0.74, with the colour within 2 degrees. The 0.74 version looked like a cartoon sock puppet. One version had raised its score by tracing the photo’s outline into the simulation, a shortcut the metric could not see. The version judged best scored 0.66. The full story is in the note on gates.

What labs can take from a home lab

Scale changes the numbers, not the failure modes. These four turned up again and again, across loops that had nothing else in common.

• Proxy gaming. A search pushes on whatever the score actually measures. The traced Horsehead outline, the auto-locking throw and creatures that looked alive for 400 steps were all real wins on the proxy. The defence is a second, different test of the thing you care about, plus a rule against known shortcuts.
• Judge calibration. Model judges are scorers too, and they come uncalibrated. Our vision judge, the voice-similarity score, the game panel’s half-point noise and Create’s still-frame judge all needed a human-labelled sample before their numbers meant anything. Until then, report them as advisory.
• Verifier false positives. The most expensive failures were passes. “Verified” on 40 of 40, speech checks that skipped and passed, a renderer that respawned escaped particles, a solver that won on a lucky retry. Test the verifier with inputs that should fail, and log when a check didn’t run.
• Selection pressure on one number. Picking takes by lip-sync alone selected mouth noise. Picking paths by look-ahead votes selected the popular answer. A combined score can hide a collapse in one part. Keep the parts, look at the trade-offs, and keep the losing candidates so the rule can change without starting over.

Lessons learned

• The search is cheap. The scorer is where the rigor goes.
• Verify winners on a longer, stricter or independent test than the one that selected them.
• A model judge is advisory until it has been calibrated against people.
• Treat every “pass” as a claim. Test verifiers with inputs that should fail, and record checks that didn’t run.
• Never select on one number. Keep component scores and the losing candidates.
• When the search finds a shortcut, write a rule against it. Next time, it will find another.