{"id":"5ffb211e-68b1-4b05-b297-170df3821fdf","arxiv_id":"2505.10360","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FactsR decomposes clinical note generation into real-time fact extraction and recursive refinement, reporting improved completeness and conciseness over a few-shot baseline on the 57-encounter Primock57 benchmark, though with a circular clinician-in-the-loop simulation and no statistical testing.","lead":"This paper introduces FactsR, a method that extracts clinical facts in real time during a consultation and then uses those facts to write the final note. It claims the method produces more accurate and concise notes than standard one-shot AI scribes, but the evaluation is limited by a circular simulation of clinician input.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline numbers depend on a private, uncalibrated judge and on a clinician-loop simulation that leaks the gold note; both must be externally validated before the central claim can be accepted.","rationale":"The reader and I agree that the private judge is the weakest link, but I would give equal weight to the self-admitted gold-note leakage in the interaction simulation. Neither issue is an ad hominem or a disagreement with consensus; they are internal to the evidence. The system description is clear and the method is plausible, and Table 1's ordering is directionally consistent, but the effect sizes are small and the absence of any external measurement leaves the central claim unverified. I am not calling for rejection: a conditional acceptance requiring a blinded clinician study, a non-leaking interaction simulation, and release of the evaluation protocol would be a fair disposition. This matches the reader's CONDITIONAL verdict, so no adjustment is needed.","tokens_in":7548,"tokens_out":7144,"duration_ms":72254,"concrete_test":"Have at least two blinded clinicians rate the outputs of all four conditions in Table 1 for completeness, conciseness, and groundedness using the paper's definitions on a random subset of Primock57; compute inter-rater reliability, model–human agreement, and paired bootstrap confidence intervals for T→DT vs F→DF. In a second arm, generate Fhatp from real clinician edits made on transcripts without access to DG (or on a held-out part of Primock57) and compare Dhatp with the gold-leaking version. If the model–human agreement is poor, or the corrected Dhatp advantage disappears, the reported improvements are artifacts of the judge and the circular simulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing problem is the evaluation in Table 1. Every number is produced by one private alignment model (Section 4.2.2), with no calibration against clinician ratings; the paper's only sanity check is that DG aligns with itself, which establishes self-consistency, not clinical validity. The main comparison T→DT vs F→DF is small (0.802 vs 0.814 completeness, 0.851 vs 0.878 conciseness, 0.971 vs 0.922 groundedness), and no per-note variance, confidence intervals, or significance test is reported, so 'significantly more complete and concise' is not supported. Independently, the clinician-in-the-loop rows Fp→Dp and Fhatp→Dhatp are constructed from DG: Fp removes facts not found in DG and Fhatp adds segments taken from DG, so the improvement in those rows is to some extent guaranteed by construction. The paper itself says this 'introduces circularity and limits conclusions,' yet the conclusion uses these rows to claim that clinician-guided refinement improves safety and fidelity. Both issues must be addressed for the central claim to stand.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FactsR, a modular clinician-in-the-loop pipeline for ambient clinical documentation. Instead of generating a SOAP note from the entire transcript in a single pass, FactsR extracts a set of \"Facts\" incrementally from a stream of consultation windows, refines them via an LLM-based evaluator and refinement model, and then generates the final note from the refined facts. The evaluation uses the publicly available Primock57 benchmark and a private alignment model to compute completeness, conciseness, and groundedness for a few-shot baseline (T→DT), FactsR (F→DF), and two simulated clinician-intervention variants (Fp→Dp and Fhatp→Dhatp). The paper claims that FactsR significantly improves completeness and conciseness while maintaining groundedness, and that clinician-guided refinement further improves fidelity. The authors explicitly acknowledge in Section 4.1 that using the gold notes to simulate clinician interaction introduces circularity and limits conclusions.","tokens_in":7823,"tokens_out":5820,"duration_ms":56743,"significance":"If the evaluation could be trusted, FactsR would be a meaningful step toward safer ambient scribe systems: it decomposes clinical summarization into interpretable fact-extraction and refinement steps, puts clinicians in the loop during the consultation, and targets a real patient-safety concern. The paper makes a good-faith attempt to move beyond lexical metrics, uses a public benchmark rather than a proprietary one, and openly discloses the circularity of the clinician-simulation. These strengths are real. However, the quantitative evidence is currently undermined by an uncalibrated private evaluation model and by a clinician-in-the-loop simulation that leaks the gold note; the central comparative claim is therefore not yet established.","major_comments":[{"comment":"The evaluation depends entirely on a private alignment model that has not been calibrated against clinician judgment; the only reported sanity check is that DG aligns with itself, which demonstrates self-consistency rather than clinical validity. Because all three metrics in Table 1 are produced by this judge, the central comparison is only as credible as the judge, and no evidence (e.g., per-note human ratings, precision/recall of the judge against human annotation, or a public model variant) is provided.","section":"Section 4.2.2"},{"comment":"The primary comparison T→DT versus F→DF yields small differences (completeness 0.802 versus 0.814, conciseness 0.851 versus 0.878, groundedness 0.971 versus 0.922), but no per-note distributions, confidence intervals, or paired significance tests are reported, so the conclusion in Section 5 that FactsR produces notes that are \"significantly more complete and concise\" is not statistically supported.","section":"Table 1, Section 5"},{"comment":"The clinician-in-the-loop rows Fp→Dp and Fhatp→Dhatp are constructed directly from the gold note DG: Fhatp adds segments taken from DG, so the large completeness gain to 0.931 is partly guaranteed by construction. The manuscript itself concedes this in Section 4.1, yet the conclusion still cites these rows as evidence that clinician-guided refinement improves safety and fidelity; this evidence needs to be recharacterized or replaced with a non-leaking protocol.","section":"Sections 4.1 and 4.2.2"},{"comment":"The displayed definition Fp = {f ∈ F | f ⊄ DG} selects facts from F that are not contained in DG, which appears to be the opposite of the intended filtering described in the text (\"filtering out irrelevant facts\") and is inconsistent with the improvement reported for Fp→Dp; this definitional inconsistency must be resolved because the clinician-intervention results depend on it.","section":"Section 4.2.2, Fp definition"}],"minor_comments":[{"comment":"The windows W(j) are defined for every token j, but Algorithm 1 and the text state that updates occur every X tokens; please clarify how the incremental update frequency X relates to the formal window set W.","section":"Section 3, Eq. (1)"},{"comment":"Figure 1 is only a placeholder prompt with bracketed text such as \"[Insert example here]\" and \"[Insert transcript here]\"; since few-shot prompting is the baseline, the actual prompt construction and the number of examples used should be described.","section":"Figure 1"},{"comment":"The table caption says model-based rows are Adjusted Completeness, but the text in Section 4.3 refers to those rows simply as completeness; please clarify which quantity is reported in each row to avoid confusion.","section":"Table 1 caption"},{"comment":"The statement that minimal few-shot prompts \"typically yield suboptimal performance\" is anecdotal; please provide references or report the prompt lengths and examples used in the baseline.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a product-oriented technical report from Corti.ai and does not release model parameters or evaluation code, which is consistent with its stated nature but limits independent verification. The central scientific contribution would be considerably stronger if the authors included at least a calibration study against clinician ratings or released the judge predictions for the 57 notes. Given the explicit circularity admission and the private judge, I would not accept the paper in its current form, but the issues are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the system concept: real-time sliding-window fact extraction with self-refinement, then generating the note from the accumulated facts. Putting the clinician in the loop during the encounter, rather than after, is a genuinely different design from the usual post-hoc ambient scribe, and the write-up is clear about how the pieces fit together. That part deserves attention.\n\nWhat doesn't hold up is the evaluation. The central comparison in Table 1—T→DT versus F→DF—shows only small differences: completeness 0.802 vs 0.814, conciseness 0.851 vs 0.878, and groundedness drops from 0.971 to 0.922. No error bars, no significance test, so the claim that FactsR produces notes that are 'significantly more complete and concise' is not supported by those rows. The larger gains come from Fp and Fhatp, and those are built from the gold note DG: Fp removes facts not in DG, Fhatp adds segments taken from DG. The paper admits this circularity in Section 4.1, but the conclusion then uses those very rows to argue clinician-guided refinement improves safety and fidelity. That argument doesn't survive.\n\nOn top of that, all three metrics are produced by a private alignment model tuned to be strict, with the only sanity check being that DG aligns with itself. That checks self-consistency, not clinical validity. Without calibration against clinician ratings, the numbers are best read as illustrating the method, not measuring it.\n\nWhat I'd give credit for: the motivation is sensible, they correctly argue why lexical metrics fail for this task, and they're transparent about the circularity. The design is modular and the clinician-in-the-loop pattern could be valuable. But the evidence doesn't match the strength of the claims.\n\nWho this is for: someone working on ambient scribe evaluation or interactive note generation. It's worth discussing in a reading group because the evaluation pitfalls are instructive. I would not cite the quantitative results, but I'd cite the approach.\n\nMy recommendation: send to peer review with the expectation of major revision—the evaluation needs non-circular simulation, human raters or a public judge, and variance reporting. The system idea is worth refereeing; the current numbers aren't.","headline":"A promising clinician-in-the-loop scribe design undermined by an evaluation that leaks the gold note and relies on an uncalibrated private judge.","tokens_in":8336,"tokens_out":1975,"would_cite":true,"duration_ms":18011,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FactsR produces clinical notes that are more complete and concise than one-shot ambient scribes by extracting and refining clinical facts in real time during the consultation.","keywords":["ambient scribe","clinical documentation","clinician-in-the-loop","real-time fact extraction","self-refinement","model-as-a-judge","Primock57","SOAP notes"],"falsifier":"Have a panel of clinicians independently rate the same Primock57 outputs from the baseline ambient scribe and from FactsR on completeness, conciseness, and groundedness. If the panel's ordering of the two systems does not reproduce the alignment model's ordering, or if the panel finds the baseline notes no less complete or concise than FactsR's, the central claim's evaluation foundation is not supported.","tokens_in":7412,"feed_emoji":"🩺","tokens_out":7659,"duration_ms":67843,"temperature":0.7,"pith_summary":"FactsR claims that clinical notes become better when an AI scribe does not wait until the consultation ends to summarize. It continuously extracts short clinical findings, called Facts, from the running conversation, checks and refines them through an evaluator-and-refiner loop, and only then turns the verified fact list into a structured note. On the Primock57 benchmark the paper reports that this pipeline beats a standard one-prompt ambient scribe on completeness and conciseness while staying close on groundedness. Simulating a clinician who edits the fact list brings the generated note much closer to a physician-written gold note, which is the safety claim: errors can be caught during the visit rather than in a post-hoc proofread. If the paper is right, ambient scribes stop being retrospective transcribers and become an interactive, clinician-verifiable documentation layer.","feed_headline":"Live fact extraction makes AI clinical notes more complete and concise","feed_subtitle":"FactsR refines facts during the visit instead of after, letting clinicians catch errors before the note is written.","key_machinery":"The load-bearing mechanism is the Fact: a short, patient-specific clinical statement such as \"diarrhea five times daily for 10 days,\" extracted from a sliding window of transcript tokens. A draft model proposes candidate facts from the current window together with facts already accumulated; an evaluator checks each candidate against the window; a refinement model rewrites any candidate that fails; and the loop repeats up to a maximum number of refinement steps. Because the transcript is processed in overlapping windows, facts accumulate as the consultation unfolds rather than in one expensive pass, and the clinician sees an editable fact list. The final note is generated by an instruction-tuned model from the verified fact set, not from the raw transcript, so formatting and phrasing are separated from clinical content.","core_discovery":"The paper's central claim, stated in Section 5, is that FactsR produces notes that are significantly more complete and concise than traditional few-shot ambient scribe approaches while maintaining a high level of groundedness. The reported numbers behind that claim are: the baseline ambient scribe ($T \\rightarrow D_T$) scores $0.802$ completeness and $0.851$ conciseness with $0.971$ groundedness; the FactsR pipeline ($F \\rightarrow D_F$) scores $0.814$ and $0.878$ with $0.922$ groundedness; filtering the facts as a clinician would ($F_p \\rightarrow D_p$) gives $0.821$ and $0.946$; and adding clinically meaningful facts missing from the transcript ($\\hat{F}_p \\rightarrow \\hat{D}_p$) gives $0.931$ completeness, $0.948$ conciseness, and $0.914$ groundedness. The interpretation is that one-pass, post-hoc summarization stays close to the transcript but abstracts and filters poorly, whereas decomposing the consultation into a live, editable fact list moves the note toward what a clinician would actually write.","pith_inferences":["As an editorial extension, the fact list is format-independent, so the same verified facts could be reused for billing codes, handoff notes, or patient summaries without regenerating them.","A direct testable extension would deploy FactsR with a visible fact panel in real outpatient visits and measure whether clinicians' live corrections reduce post-visit note editing time, a quantity the paper does not report.","The draft-evaluate-refine loop is a generic self-refinement pattern, so the same architecture could transfer to other high-stakes summarization settings where a wrong omission is expensive, such as legal or safety-critical incident reports."],"forward_implications":["Clinicians can validate documentation during the consultation, so a factual mistake is catchable before the final note is created.","Note generation becomes modular: templates, fact granularity, and evaluation criteria can be adapted per clinic or specialty without rewriting a monolithic prompt.","The live fact stream can feed real-time decision support during the consultation, not just documentation afterward.","Long or complex consultations are handled incrementally, avoiding the degradation that comes from summarizing a long transcript in one pass."],"supporting_citations":[{"why":"Supplies the Primock57 benchmark of 57 primary-care consultations and the physician-written gold notes used for all quantitative comparisons.","marker":"[Papadopoulos Korfiatis et al., 2022]"},{"why":"Makes the case that lexical metrics such as BLEU and ROUGE are unreliable for clinical dialogue summarization, motivating the alignment-based metric design.","marker":"[Fraile Navarro et al., 2025]"},{"why":"Establishes that clinically relevant information is sparse and interwoven with non-clinical dialogue, which motivates real-time fact extraction over one-pass summarization.","marker":"[Joshi et al., 2020]"},{"why":"Provides the evidence cited for model-as-a-judge overlapping with human expert annotations in healthcare, the basis for trusting the alignment model.","marker":"[Arora et al., 2025]"},{"why":"Defines the model-as-a-judge evaluation paradigm that the paper's alignment metrics instantiate.","marker":"[Gao et al., 2024]"}],"fun_headline_variants":["Live fact checks, not post-hoc prompts, sharpen AI clinical notes","Clinician-in-the-loop AI notes: more complete, concise, grounded","FactsR: Recursive fact extraction boosts clinical note accuracy","Real-time fact editing beats one-shot AI note generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the private alignment model's semantic-equivalence scores are a faithful proxy for what a clinician would call complete, concise, and grounded, and the paper does not calibrate that model against human clinician ratings.","fun_headline_variants_meta":{"raw":{"variants":["Live fact checks, not post-hoc prompts, sharpen AI clinical notes","Clinician-in-the-loop AI notes: more complete, concise, grounded","FactsR: Recursive fact extraction boosts clinical note accuracy","Real-time fact editing beats one-shot AI note generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000874,"raw_usage":{"total_tokens":3772,"prompt_tokens":924,"completion_tokens":2848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2776}},"tokens_in":540,"tokens_out":2848,"duration_ms":22733,"temperature":1.0,"reasoning_tokens":2776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:09:13.662274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of clinicians independently rate the same Primock57 outputs from the baseline ambient scribe and from FactsR on completeness, conciseness, and groundedness. If the panel's ordering of the two systems does not reproduce the alignment model's ordering, or if the panel finds the baseline notes no less complete or concise than FactsR's, the central claim's evaluation foundation is not supported.","supporting_citations":[{"cited_title":"(in press): Primock57: A dataset of primary care mock consultations","cited_arxiv_id":null,"evidence_quote":"Supplies the Primock57 benchmark of 57 primary-care consultations and the physician-written gold notes used for all quantitative comparisons."},{"cited_title":"Expert evaluation of large language models for clinical dialogue summarization","cited_arxiv_id":null,"evidence_quote":"Makes the case that lexical metrics such as BLEU and ROUGE are unreliable for clinical dialogue summarization, motivating the alignment-based metric design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that clinically relevant information is sparse and interwoven with non-clinical dialogue, which motivates real-time fact extraction over one-pass summarization."}],"review_version":1}