Evaluating H3 and LTX video renders

H3: a clean file can still say the wrong thing

An August 15 sweep covered 59 clips from one of my H3 story corpora. Basic integrity checks were clean, while dialogue checks surfaced a very different picture. These are check-level flags on a specific historical corpus, not a general H3 failure rate.

The useful product decision was to put semantic checks after file checks. A video that decodes correctly can still contain unscripted speech, narrate its prompt, or leave out the intended line.

The most dangerous result was SKIP

One parser expected a dialogue marker immediately after a speaker label. When descriptive prose appeared between them, it extracted an empty script. The speech checks skipped the comparison, while the basic file checks still passed. Human review caught the generated narration.

A separate hook issue came from an offline-mode environment variable inherited from the renderer: it could interfere with loading the speech evaluator. The hook now clears that variable before evaluation. These are harness failures, not evidence that the model followed the script.

My acceptance rule is stricter than “no failed gates”: verify that the expected checks actually ran. The current H3 hook runs the deterministic and audio tiers and writes a sidecar; it is not a guarantee that the full visual and specialist suite ran on every clip.

LTX: selecting a higher score made the audio worse

In a June 12 Dr. Grey UGC experiment, selecting the best of two takes by lip-sync confidence brought back lip-smacking that the earlier single-take version did not have. The selected take scored better on correspondence between mouth movement and audio. That did not mean it sounded better.

The LTX path could regenerate audio for each take, so the selector was not comparing the same soundtrack against different visuals. A better correspondence score could accompany a worse listening experience. Even a GOOD quality label did not remove the problem.

The recorded remedy was to keep the candidate takes, reassemble from the cleaner ones, and investigate a separate mouth-noise penalty. That penalty was a proposed durable fix in those notes, not a validated universal solution. The durable lesson is already usable: do not optimize take selection on lip-sync alone.

Evaluate the edit, not just its ingredients

The April LTX cross-version experiments found that per-frame face quality improved while some assembled transitions became much worse on flow-discontinuity measurements. A more elaborate splice and re-encode path was a suspect, not a proven isolated cause.

Last-frame conditioning improved several measured first transitions in that experiment. It did not establish that every later seam was fixed. This is why I retain shot-level, boundary-level, and whole-sequence evidence rather than reducing a film to one average score.

For multi-character stories, the LTX wrapper now runs script preflight by default. Its post-render visual check is advisory; it reports skipped validation when the VLM is unavailable. The separate regeneration tool can retry flagged beats. Those paths should not be described as a universal blocking visual gate.

The evaluators are now public

Both evaluation stacks described above are on GitHub, with their thresholds and the record of how far each can be trusted. video-rsi is the H3-side suite: thirteen deterministic ffmpeg checks, ten speech and audio checks on one whisper transcript plus speaker embeddings, a VLM-as-judge tier, a 22-rule pre-render prompt lint, and the post-render hook that writes the sidecar. Its README states, check by check, whether a threshold was calibrated against human-judged clips, validated in a repair run, left as a heuristic, or kept advisory.

ugc-factory-core is the LTX side: the SyncNet lip-sync and transcription gate (LSE-C at least 2.0, LSE-D at most 9.0, word error rate at most 0.4) with adaptive retry, the script lint, best-of-K selection from disjoint seed banks, the scoring fleet, and the autoresearch runners behind a 22-experiment leaderboard. On one overnight batch the gate passed 11 of 12, 12 of 14, 12 of 14 and 13 of 14 segments across four synthetic personas; the June take-selection lesson above is written up there as a known trap. The personas, voice references, renders and client scripts are not in either repository.

Know what the evaluator cannot tell you

My evaluation records include important qualifications:

• The 26 speech flags and 13 prompt-echo flags may overlap. Adding them does not give a count of unique failed videos.
• ECAPA voice-similarity thresholds and several VLM thresholds were uncalibrated in the recorded sweep; they are diagnostic signals, not identity-accuracy percentages.
• Sampled-frame checks can miss between-frame and temporal artifacts, and stylized character recognition can be unreliable.
• An unavailable evaluator is missing evidence. It should not silently increase the pass rate.
• A finished showcase film may use a different recipe from the historical evaluation corpus. It is not a controlled benchmark sample.

What I optimize now

I care more about time and cost per accepted beat than about the fastest isolated render. A useful loop starts with cheap specification lint, runs the relevant checks, retains failed candidates, and makes a bounded retry or editing decision.

The final step remains watching and listening. Automated evaluation narrows the search and catches repeatable failures; it does not replace taste, pacing, or the judgment that a scene is worth keeping.