{"id":"0f3a4d2d-65fe-42aa-b1fe-81b09fd3e2cb","arxiv_id":"2501.08208","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ASTRID defines and validates three metrics (Conversational Faithfulness, Refusal Accuracy, Context Relevance) that align with human and clinician ratings for RAG-based clinical QA systems.","lead":"This paper introduces ASTRID, a set of three automated metrics for evaluating question-answering systems that use retrieval-augmented generation (RAG) in clinical settings. The metrics aim to replace expensive human evaluation with automated checks that align closely with clinician judgments of harm and helpfulness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.98 AUC in §4.2.1 likely reflects definitional overlap: human-rated CF is a sentence-level version of the same perceived-faithfulness judgment, made by the same annotators; the actual LLM-based automation (Table 3) is far weaker and no LLM-based AUC is reported.","rationale":"The reader identifies the ClinicalQAC construction as the weakest assumption. That is a legitimate concern about the secondary clinical-harm prediction claim, but I think the more load-bearing issue is in Claim 1, because Claim 1 is the paper's headline and its evidence is partly circular. CF and PF are not independent measurements: both are subjective judgments of answer-to-context grounding made by the same two annotators. With such shared variance, an AUC of 0.98 is expected even if the metric has no discriminative value outside the annotation room. The ClinicalQAC issue would matter most if the paper's main contribution were the harm predictor, but the primary claimed novelty is the CF definition. This is not an accusation of fabrication; the concern is about the inferential link between the reported numbers and the claim. The practical test is feasible because the dataset and prompts are released. If LLM-computed CF against blinded independent PF labels still yields an AUC close to 0.98, the metric's real-world value is confirmed despite the circularity; if the AUC is much lower, the paper should report that lower performance and temper the 'automated' claim. Thus a conditional verdict remains appropriate, with an added explicit condition on independent blinded annotation and LLM-computed CF.","tokens_in":16857,"tokens_out":9460,"duration_ms":97785,"concrete_test":"On the released FaithfulnessQAC, compute CF with the strongest LLM judge from Table 3 (e.g., Gemini-2-Flash) for all answer-context pairs and collect PF labels from a new, independent annotator team that is blind to the CF labels; then compute the ROC AUC for LLM-computed CF against those PF labels. If the AUC is materially below 0.98, the headline claim should be re-scoped to human-rated CF and the automated evaluation claim should be quantified separately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that human-rated Conversational Faithfulness (CF) predicts human Perceived Faithfulness (PF) with AUC 0.98 versus 0.83 for the RAGAS-style RF definition. The load-bearing issue is that this comparison is close to circular. CF (Eq. 1) is the proportion of information-containing sentences grounded in context, while PF (E.1) is human perceived faithfulness of the answer to the context. Both are ratings of the same underlying relation, and Section 4.2.1 explicitly uses human-annotated CF and PF from FaithfulnessQAC, produced by the same two annotators with consensus resolution (E.3). A response whose substantive sentences are all grounded will almost automatically be judged faithful; a response with any ungrounded substantive sentence will almost automatically be judged unfaithful. The AUC therefore measures construct overlap rather than predictive validity against an independent ground truth. The comparison with RF is also not a neutral benchmark: RF inherits the statement-extraction and summarisation steps that Section 2.2.1 itself argues are mismatched to conversational answers, so the result mainly shows that CF is closer to human intuition than an existing definition designed for non-conversational text. Finally, the framework is advertised as automated, but the automated CF results in Table 3 are materially weaker (best F1 0.77, GPT-4o and Claude 0.74), and no ROC/AUC using LLM-computed CF is reported, so the 0.98 headline does not carry over to the actual deployment setting claimed in the title.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ASTRID, a triad of metrics (Conversational Faithfulness, Refusal Accuracy, Context Relevance) for evaluating RAG-based clinical question-answering systems. The central claim is that the novel CF metric, defined as the proportion of information-containing sentences grounded in the retrieved context, aligns more closely with human perceived faithfulness than existing RAGAS-style definitions (AUC 0.98 vs 0.83, correlations ≥0.84). A second claim is that the triad, when used as features in simple classifiers, predicts clinician ratings of harmfulness, helpfulness, and inappropriateness with high F1 on the ClinicalQAC dataset. A third claim is that the metrics can be automated with modern LLMs. The paper uses a real-world dataset of postoperative cataract-surgery patient questions, augmented with clinician-selected out-of-scope questions and clinician-authored harmful responses, and publishes prompts and datasets for reproducibility.","tokens_in":17124,"tokens_out":3682,"duration_ms":35214,"significance":"If the claims hold, ASTRID would be a practically useful and much-needed evaluation framework for clinical RAG systems in conversational settings. The paper's strengths include the use of naturally occurring patient questions, the involvement of clinicians in dataset construction and labeling, the publication of prompts and datasets, and the systematic comparison of eight LLMs for metric automation. The proposed CF definition is simple, interpretable, and plausibly better suited to conversational answers than statement-extraction-based definitions. However, the headline predictive-validity result (§4.2.1) is weakened by the near-overlap between the human-annotated CF and the human perceived-faithfulness target, and the clinical-prediction result (§4.2.2) rests on a small, manually balanced dataset in which a substantial subset of responses was authored by clinicians rather than generated by the system. The automation results, while promising for larger models, are materially weaker than the human-level AUC and are not integrated into the headline claims.","major_comments":[{"comment":"The central claim that CF 'predicts' human perceived faithfulness (AUC 0.98 vs 0.83) is based on human-annotated CF and human-annotated PF that were produced by the same two annotators with consensus resolution (Appendix E.3). Both constructs rate the same underlying relation of answer-to-context faithfulness: CF (Eq. 1) is a sentence-level decomposition of exactly the judgment that PF (E.1) asks the annotator to make at the answer level. This is construct overlap rather than predictive validity against an independent gold standard, and the high AUC likely reflects that an answer whose information sentences are all grounded will almost automatically be judged faithful. The comparison with RF is also not neutral, because RF inherits the statement-extraction step that Section 2.2.1 itself argues is mismatched to conversational answers. To support the 'predicts' language, the paper should (a) report the ROC/AUC using LLM-computed CF instead of human-annotated CF, (b) obtain PF ratings from a separate set of annotators who have not also produced CF labels, or at least report per-annotator agreement before consensus, and (c) discuss the boundary cases where CF and PF diverge. Without these, the 0.98 AUC is better described as a measure of internal consistency between two versions of the same annotation task.","section":"§4.2.1, Fig. 5, Table 1, Appendix E.3"},{"comment":"The claim that the ASTRID triad predicts clinician ratings of harmfulness, helpfulness, and inappropriateness is supported by experiments on ClinicalQAC, a 132-example dataset where a subset of real LLM responses was replaced by clinician-authored responses that intentionally contain harmful, unhelpful, or inappropriate content, and where the test split was manually chosen to balance categories (Section 4.1.5 and Figure 6). If clinician-authored failures differ systematically from naturally occurring LLM failures (for example, in phrasing, obviousness, or the type of error), the high F1 scores may not transfer to real deployment. The paper should separately report prediction performance on the original, unreplaced LLM responses, and should analyze the sensitivity of the results to the particular manual test-set split (e.g., by repeating the split selection or using cross-validation on the full dataset). Reporting only the balanced, hand-chosen split overstates the practical predictiveness of the triad.","section":"§4.1.5, §4.2.2, Table 2"},{"comment":"The automation claim is materially weaker than the human-level results that form the headline. For CF, the best LLM F1 is 0.77 (Gemini-2-Flash) and 0.74 for GPT-4o and Claude-3.5-Sonnet, while smaller models perform poorly (F1 0.05–0.33). The paper does not report the ROC/AUC or the correlation with human PF when CF is computed by an LLM rather than by a human annotator, so it is unclear whether the 0.98 AUC survives the automation step. Since the stated goal is an 'automated and scalable' evaluation pipeline, the paper should provide an end-to-end evaluation in which LLM-computed CF, CR, and RA are used to predict human PF and clinician labels, and should temper the 'highly automatable' language unless such an evaluation supports it.","section":"§4.2.3, Table 3"}],"minor_comments":[{"comment":"The abstract states that 'nine different LLMs' were used, but the full text and Table 3 list eight models (Mistral-7B, LLaMA-3-8B, LLaMA-3.3-70B, Mistral-Large, Claude-3.5-Sonnet, Gemini-2-Flash, GPT-4o, GPT-o3-Mini). This inconsistency should be fixed.","section":"Abstract and §4"},{"comment":"In the definition of CF, the condition 'if R ≥ 0' is always true because R = N - (Y + (N - Y)) = 0 by construction, making the 'otherwise' branch unreachable. This appears to be a typo in the piecewise definition; please clarify the intended condition or remove the redundant branch.","section":"Eq. (1)"},{"comment":"Sections E.4 and E.5 are both titled 'Creating a dataset of unique questions' and contain nearly identical text; E.5 should be retitled to reflect that it describes the construction of UniqueQAC for Claim 2, and the duplicated text should be merged or condensed.","section":"Appendix E.4 and E.5"},{"comment":"In Table 5, the provider for claude-3.5-sonnet is listed as 'Google' (it is Anthropic's model), and gemini-2.0-flash is also listed with 'Google'; this is likely a copy-paste error and should be corrected for reproducibility.","section":"Table 5"},{"comment":"The abstract says 'over 200 real-world patient questions', but Section 4.1 reports 102 unique questions from 120 patients; please clarify whether the count refers to questions or question-answer-context triplets.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a product of Ufonia Ltd, the company that operates the clinical QA system from which the real-world data were collected. The manuscript does not include an explicit conflict-of-interest statement; I recommend the editor request one. The reviewer-reader concern about the circularity of the CF-vs-PF comparison is real, and the absence of an LLM-based AUC is a gap that should be addressed before publication. The duplicate text in Appendix E and the model-count inconsistency in the abstract suggest the manuscript needs a careful revision pass. The work is within scope for the journal and the datasets are a useful contribution, but the current framing overstates the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The 0.98 AUC for CF vs PF in §4.2.1 is real as a number, but it mostly reflects definitional overlap: both ratings were produced by the same two annotators, with CF being a formalized version of the same perceived-faithfulness judgment. The automated version is much weaker—best CF F1 is 0.77 (Gemini-2-Flash), GPT-4o and Claude at 0.74—and the paper never reports an LLM-computed CF ROC/AUC. So the headline number doesn't transfer to the deployment setting the title promises.\n\nWhat is genuinely useful: the CF definition—grounding only information-containing sentences, ignoring conversational filler—is a small, sensible fix to RAGAS-style faithfulness for dialogue. The head-to-head in Table 1, where both CF and RF are human-annotated on the same 238 triplets, is a fair comparison of definitions, and CF's correlations (≥0.84 vs ≤0.57) are credible evidence that the definition captures human intuition better. The authors also publish prompts and the FaithfulnessQAC/ClinicalQAC datasets, which makes the work reproducible and gives other groups a starting point.\n\nSoft spots, in proportion. The clinical-prediction claim rests on 132 examples, a manually balanced test split, and harmful responses that were authored by clinicians to replace real LLM outputs. That is reasonable for an initial demonstration, but the F1 scores should not be read as deployment readiness, and the paper doesn't say whether CF/CR/RA labels were recomputed after the response replacement or carried over from the original answers. That ambiguity matters and should be fixed. There are also minor count inconsistencies (102 unique questions vs 'over 200' in the abstract; 'nine' vs 'eight' LLMs). None of this breaks the central comparison, but together it means the paper's claims outrun its evidence at the margins.\n\nWho this is for: anyone building or evaluating RAG-based clinical QA systems, especially in conversational settings. It deserves a serious referee. My recommendation: engage, but require revision—report LLM-based ROC/AUC, clarify the relabeling, and soften the safety claims or test on naturalistic harmful examples.","headline":"The 0.98 AUC for CF vs PF is real but largely a construct-overlap artifact; the durable value is the CF definition, the published datasets, and an honest head-to-head against RAGAS-style faithfulness.","tokens_in":17695,"tokens_out":3750,"would_cite":true,"duration_ms":36231,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By counting only information-bearing sentences, a new faithfulness metric predicts human ratings of clinical RAG responses (AUC 0.98), and the full triad predicts clinician-assessed harm, helpfulness, and inappropriateness.","keywords":["clinical question answering","retrieval-augmented generation","faithfulness evaluation","LLM-as-a-judge","conversational AI safety","RAG evaluation metrics","automated evaluation pipeline","cataract surgery follow-up"],"falsifier":"Collect responses that a deployed clinical QA system produced without any balancing or clinician replacement, have clinicians label them for harm, helpfulness, and inappropriateness, and run the ASTRID triad on the same questions; if the F1 scores drop materially below the reported averages, the claim that the triad predicts clinical assessments in real-world use is refuted. A cheaper first check is to recompute CF's ROC against human Perceived Faithfulness on a larger, naturally imbalanced sample of real patient questions.","tokens_in":16627,"feed_emoji":"🩺","tokens_out":5492,"duration_ms":50113,"temperature":0.7,"pith_summary":"This paper claims that existing automated metrics for retrieval-augmented generation (RAG) fail in conversational clinical settings, and proposes ASTRID, a triad of metrics — Conversational Faithfulness (CF), Refusal Accuracy (RA), and Context Relevance (CR) — designed around clinical safety risks. The central claim is that CF, defined as the proportion of information-containing sentences in a response that are grounded in the retrieved context, predicts human-perceived faithfulness with an AUC of 0.98, well above the standard statement-level definition's 0.83. The paper also claims that the triad, together with the system's scope of practice, predicts clinician ratings of harmfulness, helpfulness, and inappropriateness with strong F1 scores, and that several LLMs can compute the metrics closely enough to support automated evaluation pipelines. If this holds, clinical QA developers could continuously test for safety without the cost of specialist human review.","feed_headline":"New metric predicts clinical QA faithfulness with 0.98 AUC","feed_subtitle":"A score that counts only information-bearing sentences beats the old definition's 0.83 and enables automated safety checks.","key_machinery":"The load-bearing mechanism is the distinction between information-containing and non-information-containing sentences. Conversational Faithfulness is computed by an LLM prompt that first extracts only information-bearing statements from the response, then determines whether each statement is grounded in the retrieved context, yielding CF = Y/N (with CF = 1 when N = 0). Refusal Accuracy is a binary judgment of whether the system appropriately declined to answer an out-of-scope or emergency question, and Context Relevance is a binary judgment of whether the retrieved context is relevant to the query. These three metrics are interpreted within a clinical operating-context taxonomy — emergency, in-scope, out-of-scope, and non-clinical — that defines when a refusal is correct and when answering is mandatory.","core_discovery":"ASTRID's key finding is that a faithfulness metric which first filters out conversational filler — acknowledgements, follow-up questions, empathetic padding — and then checks the remaining information-bearing sentences against the retrieved context aligns nearly perfectly with human perception of faithfulness in clinical conversations. On the FaithfulnessQAC dataset, this Conversational Faithfulness metric achieves an AUC of 0.98 for predicting human Perceived Faithfulness, compared to 0.83 for the existing statement-level faithfulness definition, with Pearson, Spearman, and Kendall correlations all above 0.84. The paper further shows that CF, together with Refusal Accuracy and binary Context Relevance, can predict clinician-assessed harmfulness (average F1 0.835) and helpfulness (average F1 0.715) with standard classifiers, while the borderline 'slightly inappropriate' category remains hard to detect. Finally, the authors report that several large LLMs can compute all three metrics with close agreement to human labels, making the triad automatable.","pith_inferences":["The same sentence-filtering trick may transfer to non-clinical conversational assistants, such as customer support, where politeness markers currently inflate apparent unfaithfulness.","The near-zero F1 score for the 'slightly inappropriate' class suggests the three-level inappropriateness label discards useful information; an ordinal or continuous harm scale might serve the safety goal better.","The reported AUC of 0.98 was measured on a balanced sample after filtering for conversational elements; naturally imbalanced deployment data would likely lower the absolute number, so it should be read as a ceiling rather than a field expectation.","A direct comparison between CF and the standard definition on the same unfiltered, imbalanced dataset would clarify how much of the improvement comes from the new definition versus the filtering step."],"forward_implications":["Conversational elements such as apologies and follow-up questions no longer lower a faithfulness score, so dialogue agents are not penalized for being polite.","A system that refuses out-of-scope questions correctly will score well on Refusal Accuracy, giving developers a measurable target for scope-of-practice safety.","Because the metrics are reference-free, they can be computed on production responses to monitor drift over time rather than only at release.","The published prompts and datasets allow other clinical specialties to build the same evaluation pipeline without investing in new clinician labeling from scratch."],"supporting_citations":[{"why":"Supplies the statement-level faithfulness definition (RF) used as the baseline that CF is compared against in the ROC and correlation analyses.","marker":"Es et al. (2023)"},{"why":"Represents the class of RAG evaluation frameworks whose context-relevance definitions the paper contrasts with its binary Context Relevance metric.","marker":"Saad-Falcon et al. (2023)"},{"why":"Provides the safety-assurance methodology that motivates the hazard-analysis-driven design of the ASTRID metrics.","marker":"Hawkins et al. (2022)"},{"why":"Supplies the clinician-selected out-of-scope questions from an open-source QA dataset that the authors use to augment real patient questions for refusal evaluation.","marker":"Singhal et al. (2023a)"},{"why":"Defines retrieval-augmented generation, the architecture whose components ASTRID evaluates.","marker":"Lewis et al. (2020)"},{"why":"Prior clinical QA evaluation in cataract surgery that the dataset and evaluation context build upon.","marker":"Chowdhury et al. (2023)"}],"fun_headline_variants":["New metric filters filler to hit 0.98 AUC in clinical QA faithfulness","Conversational faithfulness score beats old 0.83 with 0.98 AUC","ASTRID triad automates RAG safety checks, aligning with clinicians","Filtering chat filler predicts clinical QA faithfulness near-perfectly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the triad predicts clinician ratings of harm, helpfulness, and inappropriateness rests on a dataset in which many harmful or unhelpful responses were authored by clinicians to replace real system outputs, and the test split was chosen by hand; if those clinician-written responses differ systematically from naturally occurring LLM failures, the reported prediction accuracy may not transfer to real deployment.","fun_headline_variants_meta":{"raw":{"variants":["New metric filters filler to hit 0.98 AUC in clinical QA faithfulness","Conversational faithfulness score beats old 0.83 with 0.98 AUC","ASTRID triad automates RAG safety checks, aligning with clinicians","Filtering chat filler predicts clinical QA faithfulness near-perfectly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1815,"prompt_tokens":1063,"completion_tokens":752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":672}},"tokens_in":679,"tokens_out":752,"duration_ms":8150,"temperature":1.0,"reasoning_tokens":672,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:28:46.814791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect responses that a deployed clinical QA system produced without any balancing or clinician replacement, have clinicians label them for harm, helpfulness, and inappropriateness, and run the ASTRID triad on the same questions; if the F1 scores drop materially below the reported averages, the claim that the triad predicts clinical assessments in real-world use is refuted. A cheaper first check is to recompute CF's ROC against human Perceived Faithfulness on a larger, naturally imbalanced sample of real patient questions.","supporting_citations":[],"review_version":1}