Ambient clinical documentation on open weights

A pipeline, not a model

The commercial products in this space are judged on the note. But the note inherits every error made upstream: a dropped medication name in transcription, a line attributed to the doctor that the patient said, a follow-up instruction that never made it into the summary. I built scribe-bench to measure those stages separately, on data anyone can download, with open-weight models on my own hardware.

Everything in the study is a mock or synthetic consultation: PriMock57 (57 recorded mock primary-care visits with transcripts and notes) and ACI-Bench, MTS-Dialog and MedSynth for note generation and fine-tuning. No real patients, no proprietary data, and no claim that the results transfer to a clinic. The point was the evaluation method.

Transcription: the medical-term miss rate is its own number

Seven speech-recognition stacks ran on the 57 PriMock57 consultations, scored on word error rate, the miss rate on medical terms specifically, diarization error where the system produced speaker turns, and the share of words attributed to the wrong speaker. Word error rate alone would rank the top three systems as roughly equal. The medical-term column does not.

The last row is a reminder that a capable model in the wrong mode is a different system: the same family that transcribes competently as a dedicated recognizer produced an 88 percent diarization error when prompted as a chat model.

Note generation: the fine-tune that looked like an improvement

Qwen3.8-27B wrote visit notes from three transcript sources: the human transcript, the raw ASR output, and a corrected ASR output. The gap between the first and the second is the ceiling on what better transcription buys the note.

One epoch of LoRA fine-tuning on open transcript-and-note pairs lifted ROUGE-L by 8.6 points on 120 ACI-Bench encounters, which is the kind of result that gets announced. The clinician-defined scorers told the other half: speaker misattributions doubled to tripled, and recall of follow-up instructions went down. The tuned model had learned the shape of a note faster than the discipline of one.

A second lesson from the same runs: if you want the note to cite the transcript, the citations have to be in the fine-tuning targets. Tuning on uncited notes erased the grounding behaviour the base model had shown.

A claim-level verifier, and what its recall does not mean

The last stage splits a note into claims and checks each against the transcript. On three held-out splits the verifier recalled 98.7 to 99.6 percent of errors that had been injected into notes deliberately.

Injected errors are not natural errors. The number says the verifier catches the classes of mistake we knew to inject; it does not say how many of a model’s own mistakes it catches, which needs a labelled set of real generation errors. That distinction is written next to the number in the repository, because a verifier presented without it becomes a false guarantee.

Speaker attribution had its own check: several LLM judges scored the same 74 notes for who-said-what, and their agreement is measured rather than assumed.

An overnight loop that only keeps what survives the test slice

Configuration search ran as an autoresearch loop in the pattern I use for video: a bounded set of knobs (prompting, ASR variant, correction, role mapping, scaffolding), one change per trial, and acceptance only if the candidate improved on the validation slice and did not regress on the test slice. The queue of trials and the decisions are part of the repository.

The loop is a search over configurations, not training. Its value was disqualifying plausible ideas quickly: prompt variants, transcript correction, a reasoning scaffold and a verifier gate all gained on the development visits and failed on the held-out set.

What this study cannot claim

The boundaries are the same ones I would want to read on anyone else’s evaluation:

• Mock consultations are cleaner than real ones. Real audio has cross-talk, accents, background noise and longer visits.
• The clinician-defined scorers are modelled on published evaluation dimensions, not reviewed by clinicians on this data.
• ROUGE and BERTScore are reported because they are standard, not because they are trusted.
• The verifier’s recall is on injected errors only.
• This is a non-commercial study to understand the problem space. It is not a product, and nothing here is clinical advice.