From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation

Referee: [Results] Results section: the central claim that cross-model disagreement supplies a usable reference-free uncertainty signal for prioritizing review rests on the unverified premise that low-agreement spans contain higher rates of actual errors. All metrics (ICC, quintile enrichment, 2.5% high-risk mass) are computed exclusively against majority-vote pseudo-references; systematic errors shared across models remain invisible, so neither precision nor recall of the signal against real transcription mistakes can be measured.

Authors: We agree that the absence of human-verified ground-truth transcripts means we cannot compute precision or recall of the disagreement signal against actual errors, and that systematic errors common to multiple models would be invisible to our analysis. Our work is explicitly positioned as an exploratory, reference-free heuristic rather than a validated error detector; this is already noted in the abstract with the statement that 'clinical accuracy of flagged regions remains to be established.' In the revised manuscript we have added a dedicated Limitations subsection in the Discussion that explicitly states the reliance on pseudo-references, the inability to measure true error rates, and the consequent need for future studies with human-annotated data. We have also clarified in the Results that the observed enrichment of content disagreements in low-agreement regions is a necessary but not sufficient condition for utility. These changes improve transparency without overstating the current evidence. revision: yes

Referee: [Methods] Methods section: the alignment procedure for multi-model outputs and the precise definition of the majority-strength metric are described only at high level. Without the exact algorithm, tokenization rules, or handling of insertions/deletions, it is impossible to assess whether alignment artifacts inflate or deflate the reported agreement statistics and the content-disagreement enrichment.

Authors: We accept that the original Methods description was insufficiently detailed. In the revised manuscript we have added a new subsection 'Multi-Model Alignment and Agreement Metrics' that provides: (1) the exact alignment algorithm (pairwise Levenshtein dynamic programming followed by progressive multiple alignment with a majority-vote anchor), (2) tokenization rules (word-level tokenization via spaCy with custom handling for medical abbreviations, numbers, and punctuation), and (3) explicit treatment of insertions/deletions (each treated as a distinct aligned position whose agreement count reflects presence/absence across models). We also supply pseudocode, a worked example of a short audio segment, and the precise definition of the majority-strength score (number of models producing an identical token string at the aligned position). These additions allow readers to evaluate potential alignment artifacts and replicate the pipeline. revision: yes