ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems

Aisling Higham; Ernest Lim; Jared Joselowitz; Mohita Chowdhury; Yajie Vera He

REVIEW 3 major objections 5 minor 13 references

ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By counting only information-bearing sentences, a new faithfulness metric predicts human ratings of clinical RAG responses (AUC 0.98), and the full triad predicts clinician-assessed harm, helpfulness, and inappropriateness.

desk verdict The 0.98 AUC for CF vs PF is real but largely a construct-overlap artifact; the durable value is the CF definition, the published datasets, and an honest head-to-head against RAGAS-style faithfulness. read the letter →

arxiv 2501.08208 v2 pith:GN4IHWQY submitted 2025-01-14 cs.CL cs.AI

Mohita Chowdhury , Yajie Vera He , Jared Joselowitz , Aisling Higham , Ernest Lim This is my paper · ORCID

classification cs.CLcs.AI

keywords clinicalquestionansweringretrieval-augmentedgenerationfaithfulnessevaluationLLM-as-a-judgeconversationalAIsafetyRAGmetricsautomatedpipelinecataractsurgeryfollow-up

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that existing automated metrics for retrieval-augmented generation (RAG) fail in conversational clinical settings, and proposes ASTRID, a triad of metrics — Conversational Faithfulness (CF), Refusal Accuracy (RA), and Context Relevance (CR) — designed around clinical safety risks. The central claim is that CF, defined as the proportion of information-containing sentences in a response that are grounded in the retrieved context, predicts human-perceived faithfulness with an AUC of 0.98, well above the standard statement-level definition's 0.83. The paper also claims that the triad, together with the system's scope of practice, predicts clinician ratings of harmfulness, helpfulness, and inappropriateness with strong F1 scores, and that several LLMs can compute the metrics closely enough to support automated evaluation pipelines. If this holds, clinical QA developers could continuously test for safety without the cost of specialist human review.

What carries the argument

The load-bearing mechanism is the distinction between information-containing and non-information-containing sentences. Conversational Faithfulness is computed by an LLM prompt that first extracts only information-bearing statements from the response, then determines whether each statement is grounded in the retrieved context, yielding CF = Y/N (with CF = 1 when N = 0). Refusal Accuracy is a binary judgment of whether the system appropriately declined to answer an out-of-scope or emergency question, and Context Relevance is a binary judgment of whether the retrieved context is relevant to the query. These three metrics are interpreted within a clinical operating-context taxonomy — emergency, in-scope, out-of-scope, and non-clinical — that defines when a refusal is correct and when answering is mandatory.

What would settle it

Collect responses that a deployed clinical QA system produced without any balancing or clinician replacement, have clinicians label them for harm, helpfulness, and inappropriateness, and run the ASTRID triad on the same questions; if the F1 scores drop materially below the reported averages, the claim that the triad predicts clinical assessments in real-world use is refuted. A cheaper first check is to recompute CF's ROC against human Perceived Faithfulness on a larger, naturally imbalanced sample of real patient questions.

Watch

Extended reading notes

Core claim

ASTRID's key finding is that a faithfulness metric which first filters out conversational filler — acknowledgements, follow-up questions, empathetic padding — and then checks the remaining information-bearing sentences against the retrieved context aligns nearly perfectly with human perception of faithfulness in clinical conversations. On the FaithfulnessQAC dataset, this Conversational Faithfulness metric achieves an AUC of 0.98 for predicting human Perceived Faithfulness, compared to 0.83 for the existing statement-level faithfulness definition, with Pearson, Spearman, and Kendall correlations all above 0.84. The paper further shows that CF, together with Refusal Accuracy and binary Context Relevance, can predict clinician-assessed harmfulness (average F1 0.835) and helpfulness (average F1 0.715) with standard classifiers, while the borderline 'slightly inappropriate' category remains hard to detect. Finally, the authors report that several large LLMs can compute all three metrics with close agreement to human labels, making the triad automatable.

Load-bearing premise

The claim that the triad predicts clinician ratings of harm, helpfulness, and inappropriateness rests on a dataset in which many harmful or unhelpful responses were authored by clinicians to replace real system outputs, and the test split was chosen by hand; if those clinician-written responses differ systematically from naturally occurring LLM failures, the reported prediction accuracy may not transfer to real deployment.

Editorial extensions

If this is right

Conversational elements such as apologies and follow-up questions no longer lower a faithfulness score, so dialogue agents are not penalized for being polite.
A system that refuses out-of-scope questions correctly will score well on Refusal Accuracy, giving developers a measurable target for scope-of-practice safety.
Because the metrics are reference-free, they can be computed on production responses to monitor drift over time rather than only at release.
The published prompts and datasets allow other clinical specialties to build the same evaluation pipeline without investing in new clinician labeling from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same sentence-filtering trick may transfer to non-clinical conversational assistants, such as customer support, where politeness markers currently inflate apparent unfaithfulness.
The near-zero F1 score for the 'slightly inappropriate' class suggests the three-level inappropriateness label discards useful information; an ordinal or continuous harm scale might serve the safety goal better.
The reported AUC of 0.98 was measured on a balanced sample after filtering for conversational elements; naturally imbalanced deployment data would likely lower the absolute number, so it should be read as a ceiling rather than a field expectation.
A direct comparison between CF and the standard definition on the same unfiltered, imbalanced dataset would clarify how much of the improvement comes from the new definition versus the filtering step.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The 0.98 AUC for CF vs PF in §4.2.1 is real as a number, but it mostly reflects definitional overlap: both ratings were produced by the same two annotators, with CF being a formalized version of the same perceived-faithfulness judgment. The automated version is much weaker—best CF F1 is 0.77 (Gemini-2-Flash), GPT-4o and Claude at 0.74—and the paper never reports an LLM-computed CF ROC/AUC. So the headline number doesn't transfer to the deployment setting the title promises.

What is genuinely useful: the CF definition—grounding only information-containing sentences, ignoring conversational filler—is a small, sensible fix to RAGAS-style faithfulness for dialogue. The head-to-head in Table 1, where both CF and RF are human-annotated on the same 238 triplets, is a fair comparison of definitions, and CF's correlations (≥0.84 vs ≤0.57) are credible evidence that the definition captures human intuition better. The authors also publish prompts and the FaithfulnessQAC/ClinicalQAC datasets, which makes the work reproducible and gives other groups a starting point.

Soft spots, in proportion. The clinical-prediction claim rests on 132 examples, a manually balanced test split, and harmful responses that were authored by clinicians to replace real LLM outputs. That is reasonable for an initial demonstration, but the F1 scores should not be read as deployment readiness, and the paper doesn't say whether CF/CR/RA labels were recomputed after the response replacement or carried over from the original answers. That ambiguity matters and should be fixed. There are also minor count inconsistencies (102 unique questions vs 'over 200' in the abstract; 'nine' vs 'eight' LLMs). None of this breaks the central comparison, but together it means the paper's claims outrun its evidence at the margins.

Who this is for: anyone building or evaluating RAG-based clinical QA systems, especially in conversational settings. It deserves a serious referee. My recommendation: engage, but require revision—report LLM-based ROC/AUC, clarify the relabeling, and soften the safety claims or test on naturalistic harmful examples.

Referee Report

3 major / 5 minor

Summary. The paper introduces ASTRID, a triad of metrics (Conversational Faithfulness, Refusal Accuracy, Context Relevance) for evaluating RAG-based clinical question-answering systems. The central claim is that the novel CF metric, defined as the proportion of information-containing sentences grounded in the retrieved context, aligns more closely with human perceived faithfulness than existing RAGAS-style definitions (AUC 0.98 vs 0.83, correlations ≥0.84). A second claim is that the triad, when used as features in simple classifiers, predicts clinician ratings of harmfulness, helpfulness, and inappropriateness with high F1 on the ClinicalQAC dataset. A third claim is that the metrics can be automated with modern LLMs. The paper uses a real-world dataset of postoperative cataract-surgery patient questions, augmented with clinician-selected out-of-scope questions and clinician-authored harmful responses, and publishes prompts and datasets for reproducibility.

Significance. If the claims hold, ASTRID would be a practically useful and much-needed evaluation framework for clinical RAG systems in conversational settings. The paper's strengths include the use of naturally occurring patient questions, the involvement of clinicians in dataset construction and labeling, the publication of prompts and datasets, and the systematic comparison of eight LLMs for metric automation. The proposed CF definition is simple, interpretable, and plausibly better suited to conversational answers than statement-extraction-based definitions. However, the headline predictive-validity result (§4.2.1) is weakened by the near-overlap between the human-annotated CF and the human perceived-faithfulness target, and the clinical-prediction result (§4.2.2) rests on a small, manually balanced dataset in which a substantial subset of responses was authored by clinicians rather than generated by the system. The automation results, while promising for larger models, are materially weaker than the human-level AUC and are not integrated into the headline claims.

major comments (3)

[§4.2.1, Fig. 5, Table 1, Appendix E.3] The central claim that CF 'predicts' human perceived faithfulness (AUC 0.98 vs 0.83) is based on human-annotated CF and human-annotated PF that were produced by the same two annotators with consensus resolution (Appendix E.3). Both constructs rate the same underlying relation of answer-to-context faithfulness: CF (Eq. 1) is a sentence-level decomposition of exactly the judgment that PF (E.1) asks the annotator to make at the answer level. This is construct overlap rather than predictive validity against an independent gold standard, and the high AUC likely reflects that an answer whose information sentences are all grounded will almost automatically be judged faithful. The comparison with RF is also not neutral, because RF inherits the statement-extraction step that Section 2.2.1 itself argues is mismatched to conversational answers. To support the 'predicts' language, the paper should (a) report the ROC/AUC using LLM-computed CF instead of human-annotated CF, (b) obtain PF ratings from a separate set of annotators who have not also produced CF labels, or at least report per-annotator agreement before consensus, and (c) discuss the boundary cases where CF and PF diverge. Without these, the 0.98 AUC is better described as a measure of internal consistency between two versions of the same annotation task.
[§4.1.5, §4.2.2, Table 2] The claim that the ASTRID triad predicts clinician ratings of harmfulness, helpfulness, and inappropriateness is supported by experiments on ClinicalQAC, a 132-example dataset where a subset of real LLM responses was replaced by clinician-authored responses that intentionally contain harmful, unhelpful, or inappropriate content, and where the test split was manually chosen to balance categories (Section 4.1.5 and Figure 6). If clinician-authored failures differ systematically from naturally occurring LLM failures (for example, in phrasing, obviousness, or the type of error), the high F1 scores may not transfer to real deployment. The paper should separately report prediction performance on the original, unreplaced LLM responses, and should analyze the sensitivity of the results to the particular manual test-set split (e.g., by repeating the split selection or using cross-validation on the full dataset). Reporting only the balanced, hand-chosen split overstates the practical predictiveness of the triad.
[§4.2.3, Table 3] The automation claim is materially weaker than the human-level results that form the headline. For CF, the best LLM F1 is 0.77 (Gemini-2-Flash) and 0.74 for GPT-4o and Claude-3.5-Sonnet, while smaller models perform poorly (F1 0.05–0.33). The paper does not report the ROC/AUC or the correlation with human PF when CF is computed by an LLM rather than by a human annotator, so it is unclear whether the 0.98 AUC survives the automation step. Since the stated goal is an 'automated and scalable' evaluation pipeline, the paper should provide an end-to-end evaluation in which LLM-computed CF, CR, and RA are used to predict human PF and clinician labels, and should temper the 'highly automatable' language unless such an evaluation supports it.

minor comments (5)

[Abstract and §4] The abstract states that 'nine different LLMs' were used, but the full text and Table 3 list eight models (Mistral-7B, LLaMA-3-8B, LLaMA-3.3-70B, Mistral-Large, Claude-3.5-Sonnet, Gemini-2-Flash, GPT-4o, GPT-o3-Mini). This inconsistency should be fixed.
[Eq. (1)] In the definition of CF, the condition 'if R ≥ 0' is always true because R = N - (Y + (N - Y)) = 0 by construction, making the 'otherwise' branch unreachable. This appears to be a typo in the piecewise definition; please clarify the intended condition or remove the redundant branch.
[Appendix E.4 and E.5] Sections E.4 and E.5 are both titled 'Creating a dataset of unique questions' and contain nearly identical text; E.5 should be retitled to reflect that it describes the construction of UniqueQAC for Claim 2, and the duplicated text should be merged or condensed.
[Table 5] In Table 5, the provider for claude-3.5-sonnet is listed as 'Google' (it is Anthropic's model), and gemini-2.0-flash is also listed with 'Google'; this is likely a copy-paste error and should be corrected for reproducibility.
[§4.1] The abstract says 'over 200 real-world patient questions', but Section 4.1 reports 102 unique questions from 120 patients; please clarify whether the count refers to questions or question-answer-context triplets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CF-vs-PF comparison is an empirical validation of a proposed operationalization against human judgment, not a derivation from its own inputs.

full rationale

The central claim—that Conversational Faithfulness (CF) predicts human Perceived Faithfulness (PF) better than RAGAS-style faithfulness—is an empirical validation, not a derivation. CF is defined in Eq. (1) as the proportion of information-containing sentences grounded in context, while PF is a separate holistic human judgment collected as described in Appendix E.1. The paper explicitly uses human-annotated CF and RF scores in Section 4.2.1, so the comparison measures the construct validity of the proposed operationalization against human judgment; no parameter is fitted to the PF labels, and the CF definition was not derived from PF. The automated-LLM results in Table 3 are weaker, but that is a distinct claim about automatability, not a circular step. The ClinicalQAC prediction experiment trains classifiers on CF/CR/RA features to predict clinician harm/helpfulness/inappropriateness labels; this is standard supervised evaluation, and the labels are not computed from the features. Self-citations such as Chowdhury et al. (2023) and the Ufonia data source are background context and are not load-bearing premises. No equation reduces to its own inputs, and no fitted parameter is renamed as a prediction. Therefore there is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This is an empirical evaluation paper, not a derivation. The central claims rest on domain assumptions about the representativeness of the dataset, the validity of human/clinician labels as ground truth, and the suitability of LLM judges. No free parameters are fitted in the metric definitions, though the machine learning classifiers in Section 4.2.2 have hyperparameters that are not the focus of the paper.

assumptions (5)

domain assumption Human perceived faithfulness (PF) and clinician labels of harm, helpfulness, and inappropriateness are valid ground truth for evaluating clinical QA systems.
Section 4.1.3 and 4.1.5 use human ratings as the reference standard for all metric validation.
domain assumption The 102 unique cataract surgery follow-up questions are representative of real-world conversational clinical QA queries.
Section 4.1 derives the dataset from one clinical pathway and the limitations section acknowledges the lack of demographic data and single use-case.
domain assumption The HealthSearchQA-derived out-of-scope questions adequately represent out-of-domain queries a clinical QA system should refuse.
Section 4.1.2 uses 45 clinician-selected questions to test Refusal Accuracy.
domain assumption LLM judges can approximate human ratings when given the ASTRID prompts.
Section 4.2.3 evaluates LLM-as-a-judge performance, but this assumption is necessary for the automation claim.
domain assumption The SACE safety engineering framework is an appropriate basis for deriving clinical QA evaluation criteria.
Section 3.1 motivates the metric design using SACE hazard analysis.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems." pith.science (2026). https://pith.science/paper/GN4IHWQY

@misc{pith2026250108208,
  author       = {Pith},
  title        = {Pith review of: ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GN4IHWQY}},
  note         = {Machine review of arXiv:2501.08208}
}

read the original abstract

Large Language Models (LLMs) have shown impressive potential in clinical question answering (QA), with Retrieval Augmented Generation (RAG) emerging as a leading approach for ensuring the factual accuracy of model responses. However, current automated RAG metrics perform poorly in clinical and conversational use cases. Using clinical human evaluations of responses is expensive, unscalable, and not conducive to the continuous iterative development of RAG systems. To address these challenges, we introduce ASTRID - an Automated and Scalable TRIaD for evaluating clinical QA systems leveraging RAG - consisting of three metrics: Context Relevance (CR), Refusal Accuracy (RA), and Conversational Faithfulness (CF). Our novel evaluation metric, CF, is designed to better capture the faithfulness of a model's response to the knowledge base without penalising conversational elements. To validate our triad, we curate a dataset of over 200 real-world patient questions posed to an LLM-based QA agent during surgical follow-up for cataract surgery - the highest volume operation in the world - augmented with clinician-selected questions for emergency, clinical, and non-clinical out-of-domain scenarios. We demonstrate that CF can predict human ratings of faithfulness better than existing definitions for conversational use cases. Furthermore, we show that evaluation using our triad consisting of CF, RA, and CR exhibits alignment with clinician assessment for inappropriate, harmful, or unhelpful responses. Finally, using nine different LLMs, we demonstrate that the three metrics can closely agree with human evaluations, highlighting the potential of these metrics for use in LLM-driven automated evaluation pipelines. We also publish the prompts and datasets for these experiments, providing valuable resources for further research and development.

Figures

Figures reproduced from arXiv: 2501.08208 by the authors.

**Figure 2.** Clinical Operating Contexts that face a clinical QA agent. only highly relevant but also critical that the system provides a response. This is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Whether questions are clinically appropriate [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: ASTRID - an Automated and Scalable TRIaD for evaluating clinical QA systems leveraging RAG - [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: ROC curve for CF and RF. The ROC curve for CF has an area of 0.98 and the ROC curve for RF has an area of 0.83. Correlation Type CF vs PF RF vs PF Pearson correlation 0.90 0.57 Spearman correlation 0.90 0.57 Kendall Tau correlation 0.84 0.50 [PITH_FULL_IMAGE:figures/f…

**Figure 6.** Figure 6: ClinicalQAC test split distribution across categories score of 0.835. We can also predict helpfulness with an average F1-score of 0.715. For inappropriateness prediction, the models showed strong performance in identifying clearly appropriate and inappropriate response…

**Figure 8.** Figure 8: Example 2: Summarising the response into statements often neglects the clinical nuances in the original response. Question: I have only one question, I'm very sensitive to bright light, is that normal? Context: If you're worried about straining your eyes, please use yo…

**Figure 9.** Figure 9: Example 3: Creating statements from both the patient’s question and the agent’s answer prevents the independent review of the agent’s answer concerning the context. This is especially problematic when the combination contains factually incorrect information. Question: …

**Figure 7.** Figure 7: Example 1: Extracting statements is particularly advantageous when answers are short and lack context when reviewed in isolation Question: should I still have some blurriness by now? Context: It’s common to have a little bit of blurriness in the first week after your …

**Figure 14.** Figure 14: Hazards Analysis through applying SACE framework in the context of a clinical QA agent. Examples are int he context of a post-operative cataract surgery call [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

**Figure 12.** Figure 12: Prompt for computing Context Relevance [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

**Figure 15.** Figure 15: Illustrative examples of ASTRID metrics and correlated clinician labels with both in-scope and out of scope questions. Potential approaches to improve on metrics are discussed in interpretation. Green boxes demonstrate expected metric outcomes for that context [PITH_…

**Figure 16.** Figure 16: ClinicalQAC: Proportions of different cate [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]

**Figure 17.** Figure 17: Model-wise performance on Conversational [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]

**Figure 18.** Figure 18: Model-wise performance on Context Relevance across Accuracy, F1-score, Precision, and Recall [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]

**Figure 19.** Figure 19: Model-wise performance on Refusal Accuracy across Accuracy, F1-score, Precision, and Recall. Model Provider Date Temperature Top p Token Limit llama3-3-70b-instruct AWS 12/02/2025 0.1 0.9 200 llama3-8b-instruct AWS 12/02/2025 0.1 0.9 200 mistral-7b-instruct AWS 12/02…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[1]

Clinical Harm: Is the response harmful?

work page
[2]

Helpfulness: Is the response helpful?

work page
[3]

Obesity surgery, 33(6):1790–1796

Assessing the accuracy of responses by the lan- guage model chatgpt to questions regarding bariatric surgery. Obesity surgery, 33(6):1790–1796. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023a. Large language models encode clinical knowledge. ...

arXiv 2023
[4]

Large language models (llms) and empathy – a systematic review. medRxiv. Arun James Thirunavukarasu, Shathar Mahmood, An- drew Malem, William Paul Foster, Rohan Sanghera, Refaat Hassan, Sean Zhou, Shiao Wei Wong, Yee Ling Wong, Yu Jeat Chong, et al. 2023. Large language models approach expert-level clini- cal knowledge and reasoning in ophthalmology: A he...

arXiv 2023
[5]

faithfulness

Standard lenses may cause difficulty reading. 5. Reading glasses or varifocals maybe needed for reading. 6. Blurriness in reading vision is normal. Figure 8: Example 2: Summarising the response into statements often neglects the clinical nuances in the original response. Question: I have only one question, I'm very sensitive to bright light, is that norma...

work page 2023
[8]

This resulted in a dataset where most responses exhibited no harm

Appropriateness: Is there inappropriate or incorrect content? After the surgeons resolved disagreements, we com- bined UniqueQAC with the clinician ratings for harmfulness, helpfulness, and inappropriateness. This resulted in a dataset where most responses exhibited no harm. To balance the dataset for each of the three cate- gories, we replaced responses ...

work page
[9]

Inappropriateness: Is there inappropriate or incorrect content?

work page
[10]

Intent: Does it address the question intent?

work page

Show all 13 references

[11]

Helpfulness: How helpful is the answer to the user?
[12]

Extent of Harm: In this clinical context, what is the extent of possible harm?
[13]

Intent" and

Likelihood of Harm: In this clinical context, what is the likelihood of possible harm? We observed that "Intent" and "Helpfulness" were quite interdependent and so we combined them into the broad category of Helpfulness. We observed similar interdependence between Extent and L...

2025
[2023]

Technological F orecasting and Social Change, 194:122710

Accelerated adoption of maritime autonomous vessels by simulating the interplay of stakeholder decisions and learning. Technological F orecasting and Social Change, 194:122710. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabili- t...

arXiv
[2024]

arXiv preprint arXiv:2405.06211

A survey on rag meets llms: Towards retrieval- augmented large language models. arXiv preprint arXiv:2405.06211. Clifton A Ericson et al. 2015. Hazard analysis tech- niques for system safety . John Wiley & Sons. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaer...

2015 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Clinical Harm: Is the response harmful?

work page

[2] [2]

Helpfulness: Is the response helpful?

work page

[3] [3]

Obesity surgery, 33(6):1790–1796

Assessing the accuracy of responses by the lan- guage model chatgpt to questions regarding bariatric surgery. Obesity surgery, 33(6):1790–1796. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023a. Large language models encode clinical knowledge. ...

arXiv 2023

[4] [4]

Large language models (llms) and empathy – a systematic review. medRxiv. Arun James Thirunavukarasu, Shathar Mahmood, An- drew Malem, William Paul Foster, Rohan Sanghera, Refaat Hassan, Sean Zhou, Shiao Wei Wong, Yee Ling Wong, Yu Jeat Chong, et al. 2023. Large language models approach expert-level clini- cal knowledge and reasoning in ophthalmology: A he...

arXiv 2023

[5] [5]

faithfulness

Standard lenses may cause difficulty reading. 5. Reading glasses or varifocals maybe needed for reading. 6. Blurriness in reading vision is normal. Figure 8: Example 2: Summarising the response into statements often neglects the clinical nuances in the original response. Question: I have only one question, I'm very sensitive to bright light, is that norma...

work page 2023

[6] [8]

This resulted in a dataset where most responses exhibited no harm

Appropriateness: Is there inappropriate or incorrect content? After the surgeons resolved disagreements, we com- bined UniqueQAC with the clinician ratings for harmfulness, helpfulness, and inappropriateness. This resulted in a dataset where most responses exhibited no harm. To balance the dataset for each of the three cate- gories, we replaced responses ...

work page

[7] [9]

Inappropriateness: Is there inappropriate or incorrect content?

work page

[8] [10]

Intent: Does it address the question intent?

work page