{"id":"16f7b95c-305a-4e4d-bf64-f17b55e00b5b","arxiv_id":"2412.15748","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A narrative review arguing that medical LLM evaluations should examine reasoning behaviour, not only accuracy, and proposing two conceptual transparency frameworks.","lead":"This preprint is a review of how medical large language models reason, arguing that most evaluations only check final answers and ignore the reasoning process. It also sketches two frameworks for making medical LLM reasoning visible to clinicians and engineers.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Framework 2's extraction premise in §6.1 assumes CoT, attention weights, and LLM-generated diagnostic rules are faithful reasoning traces, but §5.2's cited CoT-mismatch failure mode directly undermines this; no fidelity argument or validation is provided.","rationale":"After reading the manuscript in good faith, I find the review's survey and taxonomy genuinely useful: the primer (§2), the four-way evaluation taxonomy (§5), and the open-challenges list (§6.2) are clear and broadly accurate. The paper also deserves credit for explicitly citing CoT mismatch (§5.2, [80]), which suggests the authors are aware of trace unfaithfulness; the problem is that they never reconcile this with §6.1. The reader's weakest assumption—that instruction-following artifacts faithfully expose reasoning—is the same concern I identify, so I agree with the reader. My stress test sharpens it: all three extraction channels in Framework 2 are vulnerable, and the paper offers neither an argument nor a citation establishing their fidelity. Attention weights are not causally interpretable without additional assumptions; LLM-extracted rules are unvalidated outputs of the same opaque model; CoT is already admitted to diverge from final answers. Because the claimed equivalence to XAI and the visibility of low-level reasoning operations depend on these traces, the central constructive claim is currently unsupported. However, this is a fixable gap rather than a fundamental flaw: a synthetic fidelity check, or an explicit re-scoping of the frameworks as speculative and hypothesis-generating, would address it. The reader's CONDITIONAL verdict remains appropriate; I do not see grounds to reject the paper outright, since its review contribution stands independently of the framework claim.","tokens_in":23813,"tokens_out":4746,"duration_ms":43403,"concrete_test":"Construct a synthetic diagnostic benchmark with a known gold rule set, for example 500–1000 patient cases generated from a small symptom-to-diagnosis decision tree with hidden confounders. Run the §6.1 neurosymbolic pipeline: have the LLM produce CoT, attention weights, and extracted diagnostic rules; feed the rules to the symbolic solver; then compare (a) extracted rules against gold rules, (b) the CoT's stated diagnosis versus the LLM's final answer, and (c) whether CoT and attention select the gold decision-path features. If rule fidelity falls below a pre-specified threshold (e.g., 80%) or CoT mismatch occurs at a nontrivial rate, the extraction stage is unfaithful and the framework must be explicitly relabeled as speculative or revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central constructive claim—that the §6.1 frameworks make low-level reasoning behaviour visible and that understanding reasoning behaviour is functionally equivalent to XAI (§7.4)—rests on an unvalidated premise: CoT explanations, attention weights, and LLM-extracted diagnostic rules are treated as faithful traces of the model's actual reasoning flow. Section 5.2 itself cites CoT mismatch [80], where a model's written reasoning supports one answer while the model selects a different final option; this is direct evidence that CoT is not a reliable trace. Attention weights are not shown to correspond to the computational steps that produce outputs, and the LLM-extracted diagnostic rules are generated by the same opaque model with no gold-standard validation; if they contain hallucinated or incomplete rules, the symbolic solver's output is a new, unaudited artifact rather than a transparent rendering of the LLM's reasoning. The paper labels the frameworks theoretical, so lack of implementation is not itself the problem; the central gap is that the extraction premise is asserted despite the paper's own cited CoT-mismatch evidence, with no argument for when traces are trustworthy or how unfaithfulness would be detected. The review's survey and taxonomy retain value, but the transparency claim as stated is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey-and-position paper on reasoning behaviour in medical large language models (LLMs). It adapts definitions of reasoning, reasoning outcome, and reasoning behaviour from Mondorf & Plank; proposes a four-way taxonomy of evaluation paradigms (conclusion-based, rationale-based, mechanistic, and interactive); tabulates existing medical reasoning methods; argues that there is a striking lack of process-level evaluation; and proposes two theoretical frameworks intended to expose low-level reasoning behaviour. The first framework uses structured data and interpretable tree-based models, and the second combines an LLM with a symbolic solver, extracting CoT explanations, attention weights, and diagnostic rules as reasoning traces. The paper concludes that understanding reasoning behaviour in medical LLMs is functionally equivalent to explainable AI (XAI) and that this transparency will increase clinician trust and accelerate adoption.","tokens_in":24061,"tokens_out":5011,"duration_ms":44887,"significance":"The paper occupies a useful niche by directing attention to process-level evaluation of medical LLMs, which is genuinely underrepresented relative to accuracy-based benchmarking. Its four-way taxonomy in Table 3 is a plausible organizing device, and the surveys in Table 2 and Section 5 gather otherwise scattered recent work, including clinician-led CoT evaluations and process reward models. The authors should be credited for explicitly citing evidence against their own trace-fidelity assumption—the CoT-mismatch failure mode in Section 5.2—and for clearly labelling the proposed frameworks as theoretical rather than implemented. If the central claims could be supported, the paper would provide a compelling argument for complementing accuracy benchmarks with reasoning-process evaluation. As it stands, however, the constructive claims are asserted rather than demonstrated: the trace-fidelity premise is unvalidated, the 'striking lack' conclusion rests on an undocumented selection of works, and the XAI equivalence in Section 7.4 is definitional.","major_comments":[{"comment":"","section":"§6.1 and §5.2"},{"comment":"","section":"§7.4 and Abstract"},{"comment":"","section":"§4.4 and §7.1"}],"minor_comments":[{"comment":"The abstract contains a duplicated word: 'evaluating reasoning reasoning in medical LLMs' should read 'evaluating reasoning in medical LLMs'.","section":"Abstract"},{"comment":"Section 2.4 ends with the incomplete placeholder text 'ref to other dag papers here', which should be removed or replaced with actual citations.","section":"§2.4"},{"comment":"The opening of Section 5 contains an unresolved figure reference, '[Figure ??]', which should be fixed or deleted.","section":"§5"},{"comment":"Table 2 is difficult to read because cells are densely packed and some entries are not separated clearly; the inconsistent author-label format (e.g., 'Binbin et al' versus 'Emre Kıcıman [51]') should be normalized.","section":"Table 2"},{"comment":"Reference [67] is malformed, listing author names without a clear title boundary and ending with 'Towards conversational diagnostic AI. 2024.'; it should be formatted in the journal's reference style.","section":"References"},{"comment":"The abbreviation 'PRM' is first used in Section 6.1 ('the PRM can be trained on reasoning traces') but is only defined later in Section 6.2.2 as 'Process Reward Models'; the definition should be moved to first use.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The survey and taxonomy in this manuscript are likely to be a useful contribution for the medical LLM community, and the authors have been unusually candid in citing evidence that complicates their own framework. The main risk is overstatement of the transparency claims. I would encourage revision that reframes the proposed frameworks as conjectural, adds an explicit trace-faithfulness caveat, and either documents the literature search or weakens the 'striking lack' conclusion. With those changes, the paper could be publishable; without them, the central claims remain unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take: this is a readable narrative review that imports Mondorf & Plank's reasoning-behaviour framework into the medical-LLM context, and the four-way taxonomy of evaluation paradigms (conclusion, rationale, interactive, mechanistic) is genuinely useful for organizing a messy literature. The primer on CoT/ToT/SFT/RLHF is clear, and the open-challenges section touches the right topics: process reward models, inference-time scaling, CoT monitorability. If you want a starting point for thinking about process-level evaluation of clinical LLMs, it's worth a skim.\n\nBut the paper overclaims in three places, and they matter.\n\nFirst, the 'striking lack of studies' is asserted from an unsystematic selection. There is no search protocol, inclusion criteria, or coverage claim, so the reader cannot judge whether the gap is real or an artifact of what the authors happened to cite. The conclusion should be softened to 'few studies in our sample.'\n\nSecond, Section 7.4's statement that understanding reasoning behaviour is 'functionally equivalent to achieving XAI' is a definitional pivot, not an argument. That claim needs a worked-out case, not a sentence.\n\nThird—and this is the one I would flag in review—Framework 2 in Section 6.1 assumes CoT explanations, attention weights, and LLM-extracted diagnostic rules are faithful traces of the model's reasoning. But Section 5.2 itself cites a CoT-mismatch failure mode where the written reasoning supports one answer and the model picks another. That is direct evidence against the premise. The frameworks are explicitly theoretical, so lack of implementation is not the issue; the problem is that the extraction premise is asserted despite the paper's own cited counter-evidence, with no discussion of when traces are trustworthy or how unfaithfulness would be detected. This undercuts the transparency claims.\n\nMinor stuff: leftover editorial placeholder ('ref to other dag papers here', a 'Figure ??'), and several references post-date the arXiv identifier, which needs a timeline note if this is a revision.\n\nOverall: a solid, useful primer with a sensible taxonomy and a shaky transparency story. A serious referee could push the authors to make the survey systematic, defend the XAI equivalence, and either validate or explicitly label the frameworks as speculative. I would send it to an appropriate venue, but only with major revision.\n\nFor you: maybe worth a reading group if the group works on medical AI evaluation, but don't lean on it as a definitive survey.\n\n— [Your name]","headline":"Useful medical-LLM reasoning primer with a sensible evaluation taxonomy, but the survey's 'striking lack' claim and the trace-fidelity premise of its proposed frameworks need serious repair before the transparency claims hold.","tokens_in":24584,"tokens_out":3925,"would_cite":false,"duration_ms":34121,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This review argues that medical LLMs should be judged by how they reason, not just what they answer, and proposes two frameworks to expose that reasoning.","keywords":["large language models","medical artificial intelligence","reasoning behaviour","explainable AI","chain-of-thought","clinical reasoning","symbolic reasoning","reasoning evaluation"],"falsifier":"Run a medical multiple-choice benchmark, record each model's chain-of-thought before its final answer, and measure how often the trace supports a different option than the one chosen. If CoT mismatch proves frequent and unpredictable from the trace itself, the transparency claims of the proposed frameworks would be undercut, because the extracted reasoning would not be the reasoning the model actually used.","tokens_in":23574,"feed_emoji":"🩺","tokens_out":7683,"duration_ms":59445,"temperature":0.7,"pith_summary":"This review argues that the medical AI field has neglected the reasoning behaviour of large language models, focusing on benchmark accuracy while leaving the models' internal logic invisible. It adapts a definition of reasoning behaviour from general LLM research—the specific flow of logic that leads to a conclusion—and claims that understanding this flow in medical LLMs is functionally equivalent to achieving explainable AI. The authors survey current evaluation approaches, sorting them into conclusion-based, rationale-based, mechanistic, and interactive categories, and report that only a few studies actually assess reasoning. They then propose two theoretical frameworks meant to expose reasoning behaviour in a task-agnostic way. If taken up, the implication is that medical LLM evaluation would include process-level checks, not just final answers.","feed_headline":"Judge medical AI by how it reasons, not just its answers","feed_subtitle":"Review defines reasoning behaviour for medical LLMs and proposes two frameworks that make diagnosis logic inspectable.","key_machinery":"The carrying object is the notion of reasoning behaviour as a distinct entity from reasoning outcome, borrowed and adapted from prior LLM research. Everything else hangs on this distinction: if evaluation shifts from final answers to the flow of logic, then new tools and metrics are needed. The paper's first framework exploits the deterministic structure of tree-based models (such as gradient boosting or random forests) so that model training yields interpretable decision sets. The second framework is an LLM-plus-symbolic pipeline: a medical LLM generates chain-of-thought explanations, attention weights, and candidate diagnostic rules, while a symbolic reasoner applies formal rules to patient data; the extracted traces are combined, and a process reward model can score their logical coherence. This second design is the main mechanism making low-level reasoning visible, and its transparency depends on treating CoT, attention, and extracted rules as faithful traces.","core_discovery":"On the paper's own terms, the central discovery is that reasoning behaviour can be defined, surveyed, and made inspectable in medical LLMs. Reasoning is the process of drawing conclusions from available information; reasoning outcome is the event where reasoning reaches a conclusion; reasoning behaviour is the specific flow of logic that leads to that outcome. The paper's strongest claim is that understanding reasoning behaviour in this setting is functionally equivalent to explainable AI, since both aim to reveal how a model is predisposed toward an outcome. Existing work mostly evaluates conclusions, and the authors find a striking scarcity of process-level evaluation in medicine. To fill the gap, they offer two frameworks: one that standardises input data and uses interpretable tree-based models whose decision sets can be read directly, and a second, more complex design that pairs an LLM (for hypotheses, literature, and diagnostic rules) with a symbolic reasoning system whose inference chains, combined with the LLM's chain-of-thought and attention weights, form an inspectable reasoning trace. The paper also catalogs open challenges, including reasoning-knowledge entanglement, scarce high-quality reasoning traces, reward-model scalability, and the autoregressive limitation that prevents LLMs from backtracking.","pith_inferences":["A testable extension of the second framework is to build it on a public clinical dataset and check whether the symbolic rules it extracts remain stable when patient data are perturbed; stability would be evidence that the rules are real reasoning, not memorised phrasing.","If reasoning transparency is treated as XAI, then medical AI regulation could eventually require trace-level documentation, similar to how clinical documentation records a clinician's differential reasoning.","The same reasoning-behaviour lens could be applied to other high-stakes domains such as legal or financial LLMs, where a trace that contradicts the final decision is equally dangerous.","One implicit prediction is that models trained with explicit trace supervision will show fewer hallucinations; this could be tested by comparing hallucination rates in large reasoning models with and without process reward training."],"forward_implications":["If the paper is right, medical LLM benchmarks should add process-level scoring—coherence, logical validity, completeness of reasoning traces—alongside accuracy.","The LLM-plus-symbolic framework would give clinicians a readable inference chain plus attention and CoT, making disagreements between machine and expert diagnosis inspectable.","Interactive evaluations like sequential diagnosis benchmarks would spread, since they expose reasoning under uncertainty and cost constraints better than static multiple-choice tests.","Process reward models trained on reasoning traces could be folded into RLHF training, so models are rewarded for sound step-by-step logic rather than only final answers.","Because CoT mismatch exists, no single trace type should be trusted alone; the paper's position implies evaluation should combine several paradigms simultaneously."],"supporting_citations":[{"why":"Supplies the definition of reasoning behaviour that the paper adapts to medical LLMs, anchoring the whole review.","marker":"[45]"},{"why":"Documents the chain-of-thought mismatch failure mode, the main caveat limiting the paper's transparency claims.","marker":"[80]"},{"why":"Exemplifies a medical large reasoning model with rubric-based human evaluation of reasoning traces, used as a model for trace assessment.","marker":"[60]"},{"why":"Provides the structured data extraction method that the first proposed framework uses in preprocessing.","marker":"[66]"},{"why":"Introduces an interactive sequential diagnosis benchmark that the paper cites as the strongest interactive evaluation paradigm.","marker":"[3]"},{"why":"Defines chain-of-thought prompting, the technique underlying most reasoning behaviour methods surveyed.","marker":"[58]"},{"why":"Defines tree-of-thought reasoning, one of the structured search alternatives the paper discusses.","marker":"[26]"},{"why":"Shows process supervision outperforms outcome supervision, motivating the process reward model component of the second framework.","marker":"[102]"},{"why":"Supplies the gradient-boosting tree method that the first framework relies on for interpretable decision sets.","marker":"[93]"},{"why":"Supplies the random forest method that the first framework relies on for interpretable decision sets.","marker":"[94]"}],"fun_headline_variants":["Medical AI's reasoning revealed: not just right answers","How to see inside medical LLMs: reasoning made inspectable","Don't trust medical AI's answers—check its reasoning","Inspect medical LLM reasoning, not just outcomes","Medical AI: inspect the reasoning, not just the answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that what a model writes out as its reasoning—its chain-of-thought text, attention weights, and extracted rules—faithfully reflects the internal process that produced the answer; the paper itself cites documented cases in which the written reasoning supports one answer while the model selects a different final option.","fun_headline_variants_meta":{"raw":{"variants":["Medical AI's reasoning revealed: not just right answers","How to see inside medical LLMs: reasoning made inspectable","Don't trust medical AI's answers—check its reasoning","Inspect medical LLM reasoning, not just outcomes","Medical AI: inspect the reasoning, not just the answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2836,"prompt_tokens":979,"completion_tokens":1857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1777}},"tokens_in":595,"tokens_out":1857,"duration_ms":12353,"temperature":1.0,"reasoning_tokens":1777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:07:43.705602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a medical multiple-choice benchmark, record each model's chain-of-thought before its final answer, and measure how often the trace supports a different option than the one chosen. If CoT mismatch proves frequent and unpredictable from the trace itself, the transparency claims of the proposed frameworks would be undercut, because the extracted reasoning would not be the reasoning the model actually used.","supporting_citations":[{"cited_title":"Medical reasoning in llms: an in-depth analysis of deepseek r1","cited_arxiv_id":null,"evidence_quote":"Documents the chain-of-thought mismatch failure mode, the main caveat limiting the paper's transparency claims."},{"cited_title":"A generalist medical language model for disease diagnosis assistance","cited_arxiv_id":null,"evidence_quote":"Exemplifies a medical large reasoning model with rubric-based human evaluation of reasoning traces, used as a model for trace assessment."}],"review_version":1}