When ROUGE goes up and the notes get worse

The question

Ambient scribes are judged on the finished note, but the note inherits every upstream error: a dropped medication name, a line credited to the wrong speaker, a follow-up instruction that never reached the summary. I wanted to know which stage of an open pipeline actually hurts the note, and whether the metrics teams usually report would notice.

scribe-bench is a non-commercial study built on public data to learn the domain. It isn’t a product, and nothing in it is clinical advice.

The setup

Transcription is scored on the 57 recorded mock primary-care consultations in PriMock57. Notes are scored on ACI-Bench encounters, with a 7,092-term primary-care lexicon for medical terms. The writer is Qwen3.8-27B; the fine-tune is one epoch of LoRA on 2,804 open transcript-and-note pairs. Each stage is scored on held-out data before the next stage uses its output.

1. 01Transcribe and label speakers
2. 02Score the transcript
3. 03Write the note
4. 04Verify each claim
5. 05Keep only what holds on held-out

How the scorers work

Overlap metrics such as ROUGE-L compare a draft with a reference note. Beside them sit the dimensions clinicians care about: lexicon term recall and precision, plan-item recall, misattributions per note (a claim credited to the wrong speaker), follow-up recall, and grounding, meaning whether each sentence is supported by the transcript. The judged dimensions use LLM judges.

Every comparison is a paired bootstrap over encounters with a 95% interval, so a difference counts only when its interval excludes zero. The verifier is tested by injecting known errors into otherwise correct notes: negations, changed numbers, swapped drugs, fabricated findings and third-party leaks.

Speech recognition: same accuracy, different terms

Word error rate would rank the usable recognizers as nearly equal. Medical-term miss does not: the two language-model decoders lose far fewer terms than the transducer decoders, and the intervals do not overlap. Running the streaming model offline on the whole recording changes nothing, so this is the decoder, not latency.

Paired over recordings, MOSS-TD minus the streaming model is −3.9 points of term miss [−6.3, −1.7]; Canary-Qwen minus streaming is −2.6 [−4.8, −0.6] at unchanged word error rate. The last row measures one chat model as I prompted it, not the model’s best use. Complaint-tuned hotwords made word error rate slightly worse (+0.18 [+0.03, +0.33]).

The fine-tune that won every overlap metric

One epoch of LoRA lifted ROUGE-L by 8.6 points, the kind of result that gets announced. Judged on the clinician dimensions, the same notes got worse: more claims credited to the wrong speaker, and fewer follow-up instructions carried into the note. Every interval excludes zero.

The fine-tune also erased citations: the training targets carried none, so the tuned writer stopped citing the transcript at all.

An overnight search, and what the gate couldn’t see

An autoresearch loop tried 131 pipeline variants overnight: recognizer, speaker mapping, prompts, citations, a verifier gate and the writer model. A change had to hold on 20 development visits, then beat the best score on 37 held-out visits, then survive a second judge. Five changes were accepted, and ten development gains failed on held-out.

The loop reported +4.2 on held-out visits. Re-run with the writer’s official release weights instead of the community build the search had used, the gain shrank to +1.3 [−3.8, +6.2], and two losses became measurable. Roughly three of the four points belonged to the modified build, not the pipeline.

The search also rejected citations eight times, because its objective had no verifiability term and citations cost about a point of ROUGE-L. Re-scored with grounding, the cited configuration beats the uncited best by +6.1 grounded [+3.2, +8.9]. A score without the trust dimension removes the trust feature.

A verifier, tested on errors we planted

The claim verifier, Gemma 4 26B-A4B, checks each claim in a note against the transcript. It was measured by planting known errors in correct notes on three ACI-Bench test splits.

By type, negations were the hardest to catch (85–93%); fabricated findings and third-party leaks were caught 99–100% of the time. The catch is in the setup: these are synthetic corruptions. How many of a model’s own, natural errors the verifier catches is unmeasured.

From benchmark to a live visit copilot

The same pipeline runs live. The repo includes a copilot that streams the microphone to a self-hosted realtime recognizer and shows, as the visit unfolds, the next question to ask and questions drawn from guideline lookups.

I later built it out into a fuller private prototype that also drafts a cited note and EHR fields, and stops before anything is signed. It has been measured on synthetic visits only, with no real patients:

What the metric missed

ROUGE-L rewards notes that read like the reference. The fine-tune learned the reference style, so overlap rose while attribution and follow-ups slipped and citations vanished. A dashboard tracking overlap alone would have shipped it, and an optimizer scored on overlap removed a trust feature on its own.

The judges have limits too. Two judges rank notes alike (ρ = 0.80) but disagree on the sign of the official-weights difference, and four judges could not agree on who said what: pairwise κ from −0.02 to +0.29, with no note flagged by all four. These evaluations are good for ordering pipelines and catching objective failures, and unsafe as absolute quality claims without clinician calibration.

Limits

• Mock UK primary-care consultations: 57 recordings, cleaner than clinic audio, with terse reference notes and no chart context.
• No clinician has read any note, and the judges are not calibrated against clinicians.
• The search ran on a community-modified writer build; the official-weights re-run is the number to trust.
• Verifier recall is on injected errors only. The live view was evaluated as a replay.
• Fairness groups are small (3–26 encounters). A quarter of reference plan items were never spoken in the visit.

What I’d do next

• Calibrate the judges against clinician-labelled misattributions on a small set of real audio.
• Score completeness against what was actually said, and put a verifiability term in every objective.
• Train against attribution errors directly, then rerun the fine-tune.
• Measure the verifier on models’ own mistakes, not only on planted ones.