Testing a VR game: our bots knew the answer, a person in a headset didn’t

The game, and who builds it

Orrery puts you on a salvage tug answering a mayday from a station that orbits a black hole. You dock, float through a zero-gravity berth on hand rails, ride a tram, and work your way down through puzzle rooms and fights, with a blade, a shield and a lens that bends space. It is built for PC VR and played over Quest Link.

Cosmo’s agents build it in parallel workstreams: layout, tools, puzzle rooms, combat, characters, audio, lighting and the opening. A separate QA workstream owns the evals. The game logic lives in engine-free C++ cores, so bots can play it thousands of times faster than real time. A nightly job on Beast, my Linux workstation, runs the long versions at 4:30 in the morning, CPU only, and a headless Unreal build checks that the real game still compiles and passes its engine tests.

What the bots tested

By the night of October 9 the suite was serious. Seeded bots play each room with explicit skill profiles: a perfect bot, and an average one that aims within about 9 degrees, misjudges power by up to 18 percent and reacts in 0.55 seconds. Fights are played by three bots at three difficulties across six rooms. Every simulation tick checks invariants, rules that must always hold: no invalid numbers, speeds and health within bounds, no object tunnelling through a wall, never more than a set number of attackers at once. Property-based tests throw random inputs at the tools and shrink any failure to its smallest example.

Some suites are exhaustive. The dungeon’s key logic is solved over all 1,465 reachable states, to prove every one can still reach the boss. The art gallery is walked by three bot personas, a visitor who looks and lingers, a rusher who walks fast and pulls at everything, and a fuzz bot that does random things: 1,056 episodes, 56.9 million invariant checks, zero violations. The opening has 87 layout checks, 187 core checks and five player profiles. The non-player characters have 714,845 core checks and a 467-check validator.

The nightly run before my sessions covered 16 suites, 550,948 bot episodes and 4,182,198,941 invariant checks in 835 seconds. It also ran sanitizers, tools that catch memory errors and undefined behaviour as they happen.

What the bots caught

They earned their keep. A body that started inside a gravity well’s horizon was ejected at huge speed instead of falling in. A single glitched hand-tracking sample could produce a NaN, “not a number”, and poison every well. A loop between two lenses had no top speed and reached 589 metres per second in a minute. Rare speed spikes showed up near the black holes in a golf-style room, about once in a million ticks. The sanitizers caught a strike handler converting a NaN hand position into an integer, which is undefined behaviour. And the Linux build failed on three compiler warnings that the Mac and Windows builds never showed.

The bots also produced design findings, not just bugs. With the brief’s 9-degree aim error, no small socket could be hit beyond 2 metres, so throw rooms needed bigger targets or catch fields. The throw estimator released objects 67 milliseconds after the hand opened, which cost 15 to 17 percent of throw speed, so the proposal was to back-date the release. A third of far grabs needed a re-aim on a shelf of closely spaced orbs.

Testing the tests with planted bugs

A passing eval is only evidence if you have seen it fail. Mutation testing plants a known bug in the code and checks that the evals catch it. The gallery’s evals caught 4 of 4 planted bugs: a frame that wasn’t released when you walked away, two frames live at once, a pause token never released, and a teleport before the screen went black. The characters’ evals caught 3 of 3. The key-logic solver injects three softlocks into its own input as a self-test.

The most useful result was the one that almost failed. The “two frames live at once” bug was caught only after the eval gained a synthetic stress wall. On the real hanging plan, no two frames were close enough to trigger it. The eval had been passing because the level was kind, not because it was checking. Mutation testing found that gap before a player did.

October 10: the bots passed, I didn’t

I played three builds that day: a midday build, a first-run build and a build that added hands and machines. Every one had passed its bots: feature checks 40 of 40, the walk bot 0 stuck, the intro 0 violations, performance under budget. Within minutes I was trapped behind a hatch that opened onto a wall, firing a lens about 30 times with nothing happening, wondering where my health was, and walking into an unfinished room where I got lost.

We counted. The escape rate is the share of issues I found in a session that no agent or eval had reported first. On October 10 it was about 30 of 30. The coordinator’s explanation, in the eval spec it wrote afterwards, is the clearest sentence in this note: “Our evals test whether the game works for a bot that already knows the answer and moves on a scripted path. They do not test whether a person in a headset, knowing nothing, can perceive what to do, do it with real hands, and get a response.”

Each escape became an eval

Every escaped issue now maps to a new eval with an ID, a pass rule, a place it runs and a tier. Gates block a build. Scorecards are tracked per build. Flags are shown but never block. Gates must pass on the target Windows PC with the headset’s real stereo path, not only on the Linux workstation where the bots live.

The last row applies to the agents’ coordinator too. Every clip sent to me now gets a frame sheet checked first: no frames inside geometry, no black stretches, the feature visible and lit, sound present.

A bot that knows nothing

The deepest change is giving bots only what a player has. The knowledge gate requires a machine-readable record that every required action was told or shown before it was needed: a line of dialogue that played, a prompt that was in view for at least a second, or a highlight on the object. The naive player bot acts only on cues it has perceived, lines heard, prompts seen, glints and lit lamps, and otherwise wanders. It has to finish the critical path in at most twice the designer’s time, with no stretch over two minutes without progress.

Combat gets the same treatment. Bots that do the obvious thing must get the obvious result: hold the shield toward the enemy and block most swings, swing wildly and sometimes win, do nothing and lose without getting stuck. In the spec’s words, “a ‘perfect’ bot is not evidence.” Reports change as well. Every agent report must say what was tested on which machine, what wasn’t, and what a first-time player would experience, and “bots pass” may not be the headline. Two new tester roles play every build before I do: a Naive Player with no access to design docs or code, and a Breaker who tries to trap itself, clip through things and ignore instructions.

What still needs a person

Some of it will always need hands and eyes. My smoke-test notes for each build ask questions no bot answers yet. Do the hands sit right in your grip? Does the lever feel real? Is the zero-gravity berth or the tram uncomfortable? Is the music too loud under the narrator? Are the subtitles readable, and is anything too dark? The sustained-pose limit exists because I held a viewing hood to my face and my neck hurt. A bot holding a pose for ten seconds feels nothing.

That is the real job of the escape rate. It doesn’t replace the person in the headset. It measures the evals against that person, session after session, so the automated part gets better at predicting what I will hit. The target is under 20 percent within three builds, with zero escaped blockers: being trapped, a softlock, a crash, or being lost for more than two minutes. After every session, my notes and a telemetry log become new eval cases the same day.

Lessons learned

• Bots that know the answer test the game, not the player. Give at least one bot only what a player can perceive.
• Measure the evals against real play. An escape rate turns “the bots passed, but” into a number that can fall.
• Turn every escaped issue into an eval the same day, with an owner and a tier.
• Plant bugs on purpose. An eval you have never seen fail is a guess.
• Run the gates where players play: the target machine, with real input and real stereo.
• Never let “bots pass” be the headline. Say what wasn’t tested.