Contents
The question
Generative video pipelines make many takes and something has to choose between them. When the choice is automated, the score that does the choosing becomes the product’s taste. I wanted to know whether the scores I was using could be trusted to pick, and what to check when they can’t.
- 01Lint the script
- 02Generate takes from separate seeds
- 03Gate on lip sync + transcript
- 04Rank on more than lip sync
- 05Keep every candidate
How best-of-N went wrong
The talking-head pipeline renders each segment with LTX-2.3, which generates picture and audio together. A best-of-N mode renders several takes from separate seed banks and keeps the winner, ranked by gate status, then SyncNet label, then raw SyncNet confidence (LSE-C).
That ranking has a blind spot. Each take gets a freshly generated audio track, and SyncNet only measures whether mouth movement matches that audio. Mouth noise, such as lip smacks, moves with the mouth, so it can raise the match. In a two-take run, the higher-scoring takes brought back lip smacking that the single-take version didn’t have. Ranking by the GOOD/FAIR label first didn’t fix it: a smacky take can still be labelled GOOD.
The fix was cheap because every candidate stayed on disk: the bad picks were swapped back by reassembling with ffmpeg, with no regeneration. The lasting response is a separate check for mouth noise, now part of the video-rsi suite. A ranking penalty for it in the LTX pipeline is still pending.
The evaluator needs evaluating
Before SyncNet, the gate used a simple correlation between mouth opening and audio energy. It lied. One segment scored 0.770 and another 0.692, and SyncNet rated both POOR; a segment scoring 0.361 was rated FAIR. The gate now relies on SyncNet labels and rejects a POOR label, a transcript word error rate above 0.4, truncation and ghost echo.
| Label | LSE-C (confidence) | LSE-D (distance) |
|---|---|---|
| GOOD | above 5 | below 7 |
| FAIR | above 3 | below 8 |
| MEDIOCRE | above 1.5 | below 10 |
| POOR | anything else | — |
Which faces can talk
Lip sync depends on the reference image more than I expected. Of the AI image generators I tested, only Qwen-Image-2512 produced faces that lip-synced under LTX-2.3. A face from Gemini’s image model blinked and moved but never formed speech. Two characters made with Qwen-Image-2512, Elena and Sophia, each passed the gate on 12 of 14 segments with a synthetic text-to-speech voice, at average LSE-C of about 4.8 and 6.8.
The scripts mattered as much as the faces. These rules came out of failed takes and are now lint checks:
- About seven words per segment at 129 frames; longer lines run out of time.
- Periods stop the speech; ellipses make it run on. Several sentences in one segment freeze the lips.
- Negations leak into speech: a prompt that says “no X” can make the character say “not”.
- Re-encoding a finished segment can break its lip sync; assemble without dissolves.
- Brand names inflate word error rate: “AutoTalent” transcribed as “Auto Talent” scores 50%.
A tiered suite on every render
video-rsi turns those lessons into four tiers, cheapest first. A hook runs it after every render and writes a sidecar file with the gate result; drivers only continue when the gate is clean. Rerolls get three attempts, worst first, and the same failure twice is treated as a recipe bug rather than bad luck.
| Tier | When | Checks | What it catches |
|---|---|---|---|
| −1 · Spec lint | before generating | 22 rules, 9 hard blocks | bad prompts, negations in dialogue, overlong lines |
| 0 · File checks | after render | 13 ffmpeg/numpy checks | corrupt files, A/V drift, black or frozen frames, missing cuts, loudness, flicker, seams |
| 1 · Speech | after render | 10 checks | script coverage, extra speech, prompt echo, language, names, voice, mouth-noise clicks, hiss |
| 2 · VLM judge | after render | 9 verdicts | binary visual questions with a fixed JSON schema |
The mouth-noise check looks for bursts above 4 kHz, shorter than 60 ms, outside speech: three is a warning and eight is a failure. It exists because SyncNet can’t hear them.
The first sweep
The first full sweep ran the suite over 59 clips from an earlier story corpus. Every clip passed every file check. The speech tier told a different story.
- 59 / 59
- passed container, black-frame, freeze, clipping and resolution checks
- 26
- clips failed for extra speech (8 more warned)
- 13
- clips failed for prompt echo (15 more warned)
- ~35%
- of scripted shot cuts missing, per the run notes
The whole corpus took 38 seconds for file checks, 196 seconds for speech and 855 seconds for the VLM judge. One failure was in the harness itself: an offline-mode setting made the speech checks skip silently while file checks still passed, until a human listened.
Limits
- The voice-similarity and VLM thresholds are uncalibrated; a labelled calibration set is the next step.
- Cut detection may miss soft cuts, so the missing-cut share is approximate.
- The sweep covers one historical corpus. These are check-level flags, not a general failure rate for any model.
- The mouth-noise penalty isn’t yet part of take ranking in the LTX pipeline.
What I’d do next
- Label a 100-clip set by hand and calibrate every threshold against it.
- Add the mouth-noise score to take ranking, so best-of-N optimises for what a listener hears.
- Port SyncNet into the suite as a tier, so lip sync and mouth noise are judged side by side.
Sources
Questions about this work? Get in touch.
sipratt@gmail.comHow I write these: numbers link to their source files; drafting is AI-assisted and every result is checked by hand. More on method