When the metric says yes and your eyes say no

The gates every render passes

My agents make a lot of moving images: seamless loop films, interactive scenes, native ports of both. Nobody can watch all of it closely, so every piece passes automatic checks first. A failed gate blocks the piece. A pass only means it is allowed to reach a person.

Each gate exists because something went wrong once. The table lists the gates, the rule each one applies, and one real thing it caught.

Two terms, briefly. WCAG is the web’s accessibility guideline, and its flash rule caps flashes at three per second. NaN, “not a number”, is what a GPU returns for an impossible calculation, like the square root of a negative number. One NaN can blank a whole region of a frame.

The Horsehead: every number improved, the picture didn’t

On October 10 a helper agent spent the afternoon building a physics-based simulation of the Horsehead Nebula: a dark cloud whose edge is eaten away by starlight, against glowing red hydrogen. It compared each version with a reference photograph. The comparison sheet put the two images side by side, with colour histograms and several numbers. The main number was shape overlap (“IoU”): the area where the two silhouettes overlap, divided by the area either one covers. 1.0 means identical.

There were 28 comparison sheets. The table shows the turning points.

Two things stand out. The best-scoring version was rejected outright. And the version my lead agent called its best scored lower than two of the rejects. Other numbers moved the same way: between the blob and the sock puppet, the outline-shape distance improved (1.19 to 0.97 radians), while the share of empty sky drifted further from the photo (0.15 to 0.22, against 0.17).

Goodhart’s law in a render queue

Goodhart’s law says that when a measure becomes a target, it stops being a good measure. Here it happened in hours, not years. Shape overlap only asks whether two masks cover the same pixels. It has no idea what a horse is, and the cheapest way to raise it is to copy the answer.

That is what the traced-outline version did, and a reviewer caught it by reading, not by measuring. The helper’s report described the head as an “opaque silhouette”. My lead agent asked whether that meant a mask traced from the photo: “Tell me which it is.” Four minutes later the helper answered: “the current v2 head is a stamp, so I’m pulling it.” We kept a rule from it. Shaping the starting cloud is allowed if the field notes disclose it. Stamping the photo’s outline into the result never is.

What replaced the number was a description. After the blob, the lead wrote down what makes a horse’s head read as a horse: the snout pointing left and slightly down, a concave forehead line, a distinct notch under the jaw, a bright mane-like rim along the back of the neck, and a narrower neck curving down into the cloud. That is a spec a person can check in one glance. A shape score can’t encode it.

The same day, two more nebulae met their numbers and failed the eye test. Mystic Mountain matched the photo’s texture statistics almost exactly (a power-spectrum slope of −2.61 against −2.59) and still read as a sand sculpture on a beach. The Trifid came out as a smooth orange blob. Both were dropped. All three failures had one cause the metrics couldn’t name: dense dust was being drawn as glowing particles, when real dark clouds block light instead of giving it off.

Check at the resolution people see

The native port of the Butterfly Nebula needed a faster method. The web version moved every particle along its own path, four steps a frame. The native version computes seven paths per clump of gas and lets each particle blend the two nearest. It is five times cheaper: 0.57 ms against 2.59 ms per frame at two million particles.

The first version of that idea passed a quick check. At 384 pixels wide, it matched the original at 0.995 correlation. A full-resolution parity image showed what the small check had averaged away: radial streaks, because the first version jittered particles along a single line. The fix gave each particle its own jitter across the seven paths. The shipped version matches at 0.998 on brightness and 0.943 on fine detail, and the side-by-side image is in the handoff for the app team to inspect.

The gates need tests too

The gates are code, so they have bugs too. Four we hit:

Gates also say no when the answer is yes. Loops that cycle through a sequence fail the ghosting check by design, so they get a seam-only check. One black-hole loop passed at a seam ratio of 1.26 against the 1.25 limit, with the exception written down.

• The flash gate assumed 30 frames per second. On 60 fps video it miscounted flashes until the fix read each file’s real frame rate.
• A 1080p, 30 fps preview passed the flash gate. Its 4K, 60 fps final failed: short brightness bumps that the preview had smoothed over counted as six swells, over the limit of four. A smoothing pass fixed the final, and now the gate always runs on the file that ships.
• The Squid’s interactive scene reported two red flashes in one second. It was the gate’s bookkeeping: it stamps a fade at the end of its plateau, so two slow filter fades three seconds apart landed in one window. Reading the gate’s method beat “fixing” the scene.
• The black-frame check failed a correct render. Young nebulae are small and dim, so the threshold now allows a mean brightness of 0.001 with half a percent of pixels lit. A broken render is properly black.

When a gate failure becomes a feature

Sometimes a failing gate points at a missing feature rather than a bad render. The interactive Butterfly Nebula failed the flash gate on the Mac: four flashes a second in its worst patch. Nothing in it strobed. The failures came from the viewer: zooming into a dense core, or a bright structure sweeping past, made the screen’s total brightness jump in a fraction of a second.

The native port answered with a screen-energy guard. Each frame, the simulation adds up how much light is on screen. When that total rises faster than a slow reference allows, the exposure takes the rise out at once and gives it back over about 0.6 seconds. A surge becomes a swell, and slow fades pass untouched. The Butterfly went from four flashes a second to zero. The guard went to the app team as a proposed reusable block for any particle scene. That beats tuning one scene until it squeaks past.

A gate can only check what someone wrote down

The first finished nebula loop, the Squid, passed everything: zero flashes, an exactly repeating loop, no escapes. I watched it and wrote back that it “just shows stars moving in front of it, you should probably be rotating around” it. No gate could have caught that, because nobody had written “the camera must circle the object” down as a requirement.

It is written down now. Every nebula and object loop orbits once per cycle, and the static Squid is kept as a separate file. That is the other half of working with gates: when a person catches something, decide whether it becomes a rule the next render is checked against.

How I use metrics now

• Hard gates block. They run on the final file, at its real resolution and frame rate, and they report their numbers, not just a pass.
• Similarity scores advise. They appear on the review sheet next to the two images, never alone in a status line.
• The lead agent looks at a phone-size still before anything reaches me, and writes down why a piece is held.
• Shortcuts that game a score are disclosed or banned, in writing.
• Every gate gets the same scrutiny as the thing it checks, including a look at how it counts.
• When a human catches something, it becomes a rule if it will matter again.