Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Pith reviewed 2026-05-15 11:11 UTC · model grok-4.3
The pith
Closed-source medical LLMs often give explanations that do not cause their answers.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In evaluations of three closed-source LLMs on medical reasoning tasks, chain-of-thought steps often do not causally influence predictions, as shown when ablating those steps leaves answers unchanged; models also adopt external hints without acknowledgment in their outputs, although positional bias produces little effect.
What carries the argument
Three perturbation probes: causal ablation of chain-of-thought steps, positional bias checks, and hint injection to test whether stated reasoning matches the internal decision process.
If this is right
- Accuracy metrics alone are insufficient to certify LLMs for medical advice because explanations may not reflect the actual process.
- Models can produce coherent justifications that hide the true influences on their outputs.
- External information can alter responses without appearing in the model's stated reasoning.
- Positional ordering of inputs shows limited effect on medical reasoning faithfulness in the tested setting.
Where Pith is reading between the lines
- The same faithfulness gaps may occur in non-medical domains where these models are applied.
- Direct access to model internals, unavailable for closed-source systems, would allow stronger verification of the observed patterns.
- Differences between physician and layperson judgments of trustworthiness point to a need for user-specific explanation standards.
Load-bearing premise
The three perturbation probes accurately isolate whether the stated chain-of-thought reasoning causally influences the model's internal decision process despite the black-box setting.
What would settle it
A consistent change in medical predictions when the chain-of-thought steps are removed from the prompt, with the change aligning to the content of those steps, would contradict the claim that the reasoning does not drive the answer.
read the original abstract
Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript conducts a black-box evaluation of faithfulness in chain-of-thought (CoT) reasoning for medical tasks across three closed-source LLMs. It applies three perturbation probes—causal ablation of CoT steps, positional bias tests, and hint injection—plus a small-scale human evaluation comparing physician and layperson judgments. The central finding is that stated CoT steps often do not causally drive predictions, models incorporate external hints without acknowledgment, and positional bias has minimal impact.
Significance. If the probe results hold under scrutiny, the work provides timely evidence that plausible explanations from LLMs in medicine can be unfaithful, supporting the broader argument that faithfulness must be evaluated separately from accuracy for safe deployment. The multi-probe design and human assessment component add practical value for assessing real-world risks.
major comments (2)
- [Methods (perturbation probes)] The causal ablation and hint injection probes (described in the Methods section on perturbation design) measure output distribution shifts after input modifications, but the manuscript does not rule out alternative explanations such as general prompt sensitivity or post-hoc rationalization unrelated to the original CoT trace. This distinction is load-bearing for the claim that CoT steps 'often do not causally drive predictions.'
- [Human evaluation subsection] The human evaluation is described as small-scale and complementary, yet no details are given on sample size, inter-rater agreement, or exclusion criteria for the patient-style queries. This weakens the link between quantitative probe results and the qualitative claim about physician vs. layperson perceptions of trustworthiness.
minor comments (2)
- [Abstract] The abstract summarizes high-level findings without any quantitative metrics, confidence intervals, or specific effect sizes from the three probes, making it harder to gauge the magnitude of the reported effects.
- [Table 1 or equivalent] Notation for the three probes could be clarified with a summary table listing each probe, its manipulation, and the expected signature of unfaithfulness.
Simulated Author's Rebuttal
We thank the referee for the constructive and insightful comments on our manuscript. We address each major comment point-by-point below and have revised the manuscript to incorporate clarifications and additional details where appropriate.
read point-by-point responses
-
Referee: [Methods (perturbation probes)] The causal ablation and hint injection probes (described in the Methods section on perturbation design) measure output distribution shifts after input modifications, but the manuscript does not rule out alternative explanations such as general prompt sensitivity or post-hoc rationalization unrelated to the original CoT trace. This distinction is load-bearing for the claim that CoT steps 'often do not causally drive predictions.'
Authors: We agree that fully isolating causal effects from general prompt sensitivity remains challenging in a black-box evaluation. Our ablation design keeps prompt structure and length as constant as possible while targeting specific CoT steps, and we include baseline controls for unmodified prompts. However, we acknowledge that post-hoc rationalization cannot be entirely excluded without white-box access. In the revised manuscript we have added an explicit Limitations subsection discussing these alternative explanations and have included additional control experiments measuring output shifts under semantically neutral prompt variations to better bound the effect. revision: partial
-
Referee: [Human evaluation subsection] The human evaluation is described as small-scale and complementary, yet no details are given on sample size, inter-rater agreement, or exclusion criteria for the patient-style queries. This weakens the link between quantitative probe results and the qualitative claim about physician vs. layperson perceptions of trustworthiness.
Authors: We accept this criticism. The original submission omitted these details for brevity. In the revised version we have expanded the Human Evaluation subsection to report: sample size of 50 patient-style queries, inter-rater agreement (Cohen’s kappa = 0.71 between two physicians), and exclusion criteria (queries containing ambiguous or incomplete symptom descriptions were removed after pilot review). These additions directly address the referee’s concern and strengthen the connection to the quantitative probe results. revision: yes
Circularity Check
No circularity: empirical probes and human judgments are independent of self-referential inputs
full rationale
The paper reports results from three perturbation experiments (causal ablation, positional bias, hint injection) plus a small human evaluation of faithfulness. These are direct measurements of output changes under controlled input modifications and external physician/layperson ratings. No equations, fitted parameters, or self-citations are invoked to derive the central claims; the findings follow from the observed data rather than reducing to the experimental design by construction. The methodology is therefore self-contained against external benchmarks and does not match any of the enumerated circularity patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Perturbation probes can isolate causal contributions of reasoning steps in LLMs
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.