Tuning a game with panels of AI playtesters

The setup

A Handful has no score. You hold a cloud of dust and can only nudge it: squeeze it, warm it, stir it, throw things. Gravity, heat and orbits do the rest. Each time you understand something by doing it, a light appears in a shared sky. The ending shows that your dust was once someone else’s star.

When I asked for it on October 3, I asked for “a few revs with sonnet/fable/opus testers almost like autoresearch karpathy style.” That meant a loop. Each revision tests one hypothesis. Scripted bots check that the whole game still runs end to end. Then a panel of AI testers plays it cold, with only the in-game instructions, and scores nine things from 1 to 10: whether you know what to do within 30 seconds, controls, movement, physics, the “aha” moments, beauty, sound, the ending, and whether you’d recommend it. A change that lowers the mean gets reverted. The loop stops when the mean holds at 8.5 or above for two revisions.

The panel was three Claude models with different jobs. Sonnet judged clarity and controls, Fable feeling and beauty, Opus physics truth. Each tester was an agent driving a real browser with a mouse, writing up where it got stuck and what delighted it.

The scores

The climb was real: from 5.11 to a plateau around 7.5 to 7.7. But single runs are noisy, about half a point per tester, so revisions 18 to 21 are statistically level. The 8.5 target was never reached, and I’d rather say so than move the target.

The biggest jumps came from removing blockers and from giving the player something to do where they had been watching. A regression check mattered too. Before testing revision 11, the agent re-checked everything that had made revision 5 the best so far. It found that revision 9’s crisper particles had quietly flattened the glowing cloud, and Fable’s beauty score had dropped from 8 to 6. Restoring the glow was part of revision 11’s gain. A loop that only tests the new change will slowly lose the old wins.

What the testers were good at

They found real bugs that unit checks had missed. In the first build, a start screen that had faded to invisible still caught every click, so all three testers were stuck at the door. In revision 9, Sonnet tried a throw about 19 times and the aiming guide never turned green. The cause: the guide compared the throw’s future path with where the target was now, not where it would be. In revision 10, the star picker showed nothing, because it tested the mouse position mid-frame while the camera was updating. In revision 17, a tester’s planet kept swallowing material until it became a gas giant 1,107 times Earth’s mass, stopped counting as “your world,” and stranded the tester in that chapter for about 20 minutes.

They were also good at “does this make sense?” Their notes named the confusing moment, the misleading caption and the light that arrived for nothing. When the gesture help page still said “drag away to aim” after the game had switched to “drag toward your world,” a tester failed every throw and said exactly why.

Where the judges disagreed

Opus graded hardest. It marked down lights you couldn’t fail to earn, and physics details like a star brighter than its own mass allowed. Its scores sat around 6.2 to 6.7 while Sonnet’s sat around 7.4 to 7.8. In revision 6 I acted on Opus’s complaint with a strict rule: you only earned the “squeeze” light after the cloud had bounced back once. It cost 0.3 points, Opus itself faulted a player who squeezed correctly the first time and got nothing, and it was reverted. From revision 9, Opus was kept for the hardest judgment calls only.

Fable asked for the same thing three revisions running: make each light visibly caused. It stayed flat at 6.7 to 6.9 until that was done, then reached 7.0 and later 7.33. The panel was also good for A/B questions. At the ending, Fable compared a version that showed the final reveal as a match cut with one that stated it in a caption. It chose the unsaid version: “let me make the connection myself, which is the game’s whole thesis.”

Where they were wrong, or blind

Too easy. In revision 12 the throw locked onto an exact orbital path that met your planet, with a green arc and a chime. The testers loved it, and Fable called it “the best piece of interaction design in the game.” The score held. Then I played revision 20 on Vision Pro and wrote back that it was “a little too easy now.” A noisy bot confirmed it: the locked throw hit on the first try, every time. Nobody on the panel had said easy.

Sound. The testers ran in headless browsers and couldn’t hear anything. Early on, sound was scored 5, meaning “unknown,” by rule. Later it was scored from a measured audio report: how much energy sat in the bass, the brightness of the mix, which events fired. That beat guessing, but it measured a proxy. My verdict after one listen was “needs better game audio.”

The headset. Testers played with a mouse. The hand gestures were verified only with synthetic hands in an emulator. The real test was my hands.

Finding the cause. From revision 9, every scroll that squeezed the cloud also widened the camera’s field of view, because the page shell applied its own zoom and the game never reset it. Testers reported the symptoms for nine revisions: things looked sparse and distant, and gas giants stretched at the edges of the screen. None found the cause. My lead agent found it by looking at the framing itself instead of reading more reports. Revision 18 pinned the field of view. A measured check of how much of the screen each subject filled then found four moments still under 30 percent, and the next build fixed them.

Harness artifacts that turned out to be real. One tester’s browser driver pressed without hovering first, so a slow-down that triggered on hover never fired for it. That looked like a test artifact, until I remembered that touch players can’t hover either. The fix, a magnetic grab, helped everyone.

Bots for difficulty, panels for sense

After “too easy,” difficulty became a bot’s job. A noisy bot plays with human-sized errors: 34-pixel clicks, a 320-millisecond reaction, aim off by up to 11 degrees and power off by up to 24 percent. An expert bot uses smaller errors. Each beat gets a table across many random seeds, and my target is a median of about three tries for an average player and one or two for an expert.

An exact, zero-noise bot found two bugs of its own along the way. The aiming prediction assumed a perfectly circular orbit, which the giant planets perturb. And a gentle throw was being mistaken for setting a body down. With the late game two to three times harder, the panel scored revision 21 at 7.72, equal to its best. Both testers finished in about 20 minutes, and the new narrator scored 7 and 8 out of 10.

How I’d run it again

• Use AI panels for clarity and bugs, run cold, with one hypothesis per revision.
• Report the noise. With one run per tester, half a point is weather.
• Don’t chase one judge’s taste. Look for asks that repeat across revisions and testers.
• Measure difficulty with noisy and expert bots, never with opinions.
• When testers keep describing a symptom, measure something before changing anything.
• Keep humans for feel, sound and the headset, and get them in earlier than revision 20.