pith. sign in

arxiv: 2605.26567 · v1 · pith:7T4EYHXJnew · submitted 2026-05-26 · 💻 cs.AI

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

Yuhao Shen , Lang Cao , Simo Du , Yuqing Wang , Juexiao Zhou , Hao Peng , Yue Guo This is my paper

Pith reviewed 2026-06-29 18:28 UTC · model grok-4.3

classification 💻 cs.AI
keywords clinical practice guidelinesexecutable decision logicmedical large language modelsclinical reasoningcounterfactual QA datapost-trainingphysician evaluation
0
0 comments X

The pith

Converting clinical guidelines into executable decision logic creates training data that improves medical LLMs on reasoning tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that clinical practice guidelines contain procedural decision logic that can be made executable and turned into question-answering data for training. By generating both factual cases and counterfactual variations where patient conditions change, the supervision teaches models when and why decisions shift. A model post-trained on this data, called MedGuideX, outperforms standard approaches on clinical reasoning benchmarks and aligns better with physician judgments on reasoning quality. This matters because it offers a structured way to embed evidence-based rules into AI without relying solely on free-text processing.

Core claim

Post-training a medical LLM on QA data generated from executable clinical decision logic derived from CPG recommendations produces MedGuideX. This model achieves a 10.28% relative improvement in average accuracy across four clinical reasoning benchmarks. It also better recovers clinician-authored reasoning steps and generates rationales that physicians prefer for faithfulness, validity, completeness, and clarity.

What carries the argument

The transformation of CPG recommendations into executable clinical decision logic that generates factual and counterfactual QA data for supervision.

Load-bearing premise

Converting guideline recommendations to executable logic and the resulting QA data maintains the original clinical fidelity without adding errors or losing key decision rules.

What would settle it

A post-trained model showing no accuracy gain or lower physician preference scores on the benchmarks compared to the base model would indicate the pipeline does not deliver the claimed benefits.

Figures

Figures reproduced from arXiv: 2605.26567 by Hao Peng, Juexiao Zhou, Lang Cao, Simo Du, Yue Guo, Yuhao Shen, Yuqing Wang.

Figure 1
Figure 1. Figure 1: Overview of MedGuideX. Top: We transform raw CPGs into executable Python functions f via an intermediate decision tree, then sample clinical variables X and execute f(X) to deterministically label both factual and counterfactual QA instances. Bottom: We post-train an LLM with SFT on the mixed QA and RL on factual QA, where re-executing f provides the reward. This design turns each guideline into an executa… view at source ↗
Figure 2
Figure 2. Figure 2: Training strategy ablation on data composition and training phases. Each group of bars corresponds to a [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Layout of one case page in the blinded physician questionnaire. Each page shows the clinical question [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
read the original abstract

Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, and recommendation rules. However, existing methods often use CPGs as free-text training data or retrieval sources, underutilizing their procedural decision structure. To better exploit this structure, we introduce a guideline-derived training pipeline that transforms CPG recommendations into executable clinical decision logic and uses it to generate factual and counterfactual question-answering data. Theses data teach models both guideline-supported decisions and how decisions change under different patient conditions. Post-training a medical LLM on the generated data yields MedGuideX. Across four clinical reasoning benchmarks, MedGuideX achieves a 10.28% relative improvement in average accuracy. Physician evaluation further shows that MedGuideX better recovers clinician authored reasoning steps and produces physician-preferred rationales in faithfulness, validity, completeness, and clarity. Overall, our results show that executable decision logic from CPGs can be transformed into scalable supervision for building reliable medical LLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces MedGuideX, obtained by converting clinical practice guidelines (CPGs) into executable clinical decision logic, generating factual and counterfactual QA pairs from that logic, and post-training a medical LLM on the resulting data. It reports a 10.28% relative improvement in average accuracy across four clinical reasoning benchmarks plus superior physician ratings on recovery of clinician-authored reasoning steps and on rationale faithfulness, validity, completeness, and clarity.

Significance. If the conversion pipeline preserves guideline fidelity and the empirical gains are reproducible, the work would demonstrate a scalable route for turning the procedural structure of CPGs into supervision signals that improve clinical reasoning beyond free-text or retrieval-based uses of guidelines.

major comments (2)
  1. [Methods / pipeline description] The guideline-derived training pipeline (described in the methods) supplies no quantitative validation—expert audit, inter-rater agreement, or side-by-side fidelity metrics—of the CPG-to-executable-logic conversion step. Because the central claim rests on the generated QA data faithfully reflecting the source guidelines, the absence of such checks leaves open the possibility that reported accuracy gains and physician preferences arise from transformation artifacts rather than internalized decision logic.
  2. [Experiments / results] The experimental section reports a 10.28% relative accuracy improvement and physician preference results but provides no details on baselines, statistical tests, controls, or variance across runs. Without these, it is impossible to determine whether the data support the claim that the executable-logic supervision is responsible for the gains.
minor comments (1)
  1. [Abstract] Abstract contains the typo “Theses data” (should be “These data”).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments correctly identify gaps in explicit validation of the conversion pipeline and in the reporting of experimental controls and statistics. We address each point below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Methods / pipeline description] The guideline-derived training pipeline (described in the methods) supplies no quantitative validation—expert audit, inter-rater agreement, or side-by-side fidelity metrics—of the CPG-to-executable-logic conversion step. Because the central claim rests on the generated QA data faithfully reflecting the source guidelines, the absence of such checks leaves open the possibility that reported accuracy gains and physician preferences arise from transformation artifacts rather than internalized decision logic.

    Authors: We agree that direct quantitative validation of the CPG-to-executable-logic conversion step is missing from the current manuscript. While the physician ratings on final rationales provide indirect support for overall pipeline quality, they do not constitute an audit of the conversion itself. In the revision we will add an expert audit subsection: a random sample of 200 guideline-to-logic conversions will be reviewed by two clinicians for fidelity, with Cohen's kappa reported for inter-rater agreement and a side-by-side comparison table of original guideline text versus generated logic. This material will appear in the Methods and a new supplementary table. revision: yes

  2. Referee: [Experiments / results] The experimental section reports a 10.28% relative accuracy improvement and physician preference results but provides no details on baselines, statistical tests, controls, or variance across runs. Without these, it is impossible to determine whether the data support the claim that the executable-logic supervision is responsible for the gains.

    Authors: We acknowledge that the experimental reporting is insufficiently detailed. The manuscript already compares against several medical LLM baselines, but omits statistical tests, ablation controls, and run-to-run variance. In the revised version we will: (i) list all baselines with absolute and relative scores, (ii) add McNemar's tests and paired t-tests with p-values for the accuracy gains, (iii) include an ablation study removing the counterfactual QA component, and (iv) report mean and standard deviation across three independent fine-tuning runs with different random seeds. These additions will be placed in the Experiments section and a new supplementary table. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical training pipeline is self-contained

full rationale

The paper presents an empirical pipeline that converts CPG text into executable logic, generates factual/counterfactual QA pairs from it, fine-tunes a medical LLM on the resulting data, and measures accuracy gains plus physician preference on external benchmarks. No equations, fitted parameters renamed as predictions, self-definitional steps, or load-bearing self-citations appear in the provided text. The central claim (10.28% relative improvement) is evaluated against independent test sets and human raters rather than reducing to the input data or prior author results by construction. The conversion step itself is a methodological choice whose fidelity is not verified within the paper, but that is a correctness concern, not a circularity reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract.

pith-pipeline@v0.9.1-grok · 5722 in / 1066 out tokens · 41081 ms · 2026-06-29T18:28:29.760102+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    C., Papaioannou, J

    Medalpaca–an open-source collection of medi- cal conversational ai models and training data.arXiv preprint arXiv:2304.08247. Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2025a. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXi...

    work page arXiv 2025

  2. [2]

    InInternational Conference on Learning Representations, volume 2024, pages 39578–39601

    Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others

    2024

  3. [3]

    DeepSeek-V3 Technical Report

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Nikita Mehandru, Niloufar Golchini, David Bamman, Travis Zack, Melanie F Molina, and Ahmed Alaa

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room

    Er-reason: A benchmark dataset for llm-based clinical reasoning in the emergency room.arXiv preprint arXiv:2505.22919. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xi- ang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. 2025. s1: Simple test-time scaling. InProceedings of the 2025 Confer...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov

    Executable counterfactuals: Improving llms’ causal reasoning through code.arXiv preprint arXiv:2510.01539. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, and 1 others. 2022a. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682. Jason Wei, ...

    work page arXiv 2025

  6. [6]

    Qwen3 Technical Report

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. 2024. Towards robust offline reinforcement learning under diverse data corruption. InInternational Conference on Learning Representations, volume 2024, pages 15512–15543. Gwangpyo Yoo and Honguk Woo. 2025. Model risk-...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    For each benchmark instance, we first re- trieve the top-3 guideline recommendations from the validated recommendation corpus using TF- IDF retrieval

    as a train-free decision-tree prompting base- line. For each benchmark instance, we first re- trieve the top-3 guideline recommendations from the validated recommendation corpus using TF- IDF retrieval. We then prompt Qwen3.5-9B to con- struct a compact CPGPrompt-style decision tree as a Python dictionary literal, including the selected recommendation ID,...

  8. [8]

    **Shallow anterior chamber** in the left eye compared to the right

  9. [9]

    **Subretinal fluid** in the left eye, particularly around the optic disc and extending into the macula

  10. [10]

    Normal intraocular pressure bilaterally

  11. [11]

    No anterior chamber inflammation observed during slit-lamp examination

  12. [12]

    The presence of a shallow anterior chamber suggests a structural abnormality, possibly related to the anatomy of the eye

    A history of being diagnosed with V ogt-Koyanagi-Harada (VKH) disease, which typically involves bilateral involvement and uveitis. The presence of a shallow anterior chamber suggests a structural abnormality, possibly related to the anatomy of the eye. However, VKH disease is usually characterized by bilateral symptoms and inflammation, which are not evid...

  13. [13]

    **Right upper quadrant abdominal pain**: Consider causes such as gallbladder disease (e.g., cholecystitis), pancreatitis, peptic ulcer disease, or vascular events like mesenteric ischemia or renal infarction

  14. [14]

    **History of flu-like illness and doxycycline use**: Evaluate for complications related to recent antibiotic use (e.g., drug-induced side effects) or secondary infections

  15. [15]

    **Cyanosis and low oxygen saturation**: Investigate cardiac or pulmonary causes, including pulmonary embolism, chronic lung disease, or congenital heart defects

  16. [16]

    **Acute renal infarction on CT scan**: Explore potential causes such as thromboembolism, vasculitis, or other vascular abnormalities

  17. [17]

    — ### Reasoning Process:

    **Congenital heart disease history**: Assess how underlying structural heart defects (e.g., tricuspid atresia, transposition of the great arteries) contribute to systemic issues like hypoxemia or embolic events. — ### Reasoning Process:

  18. [18]

    These findings suggest a systemic issue involving both cardiovascular and possibly gastrointestinal systems

    **Initial Presentation**: - The patient has severe right upper quadrant abdominal pain, cyanosis, and hypoxemia (oxygen saturation 79%). These findings suggest a systemic issue involving both cardiovascular and possibly gastrointestinal systems. - The absence of fever, nausea, vomiting, or diarrhea makes infectious or inflammatory causes less likely in th...

  19. [19]

    This condition predisposes her to cyanosis due to impaired oxygenation and potential complications like paradoxical emboli

    **Cardiac History**: - The patient has a complex congenital heart defect (tricuspid atresia with transposition of the great arteries). This condition predisposes her to cyanosis due to impaired oxygenation and potential complications like paradoxical emboli. - Cyanosis and hypoxemia could also result from pulmonary hypertension or right-to-left shunting d...

  20. [20]

    In this case, embolism is most likely given the systemic nature of the presentation

    **Renal Infarction**: - The CT scan shows an acute right renal infarction, which is often caused by embolism, thrombosis, or vasculitis. In this case, embolism is most likely given the systemic nature of the presentation. - Emboli can originate from the heart, particularly in patients with congenital heart disease, where intracardiac thrombi may form due ...

  21. [21]

    **Doxycycline Use**: - Doxycycline is not commonly associated with embolic phenomena, so it is unlikely to be directly responsible for the renal infarction. - However, the flu-like illness treated with doxycycline raises the possibility of a preceding viral infection that could have triggered a hypercoagulable state or endothelial damage, increasing the r...

  22. [22]

    Given the patient’s congenital heart disease, paradoxical embolism (via a right-to-left shunt) is a plausible mechanism

    **Systemic Embolism**: - The combination of cyanosis, hypoxemia, and renal infarction strongly suggests a systemic embolic event. Given the patient’s congenital heart disease, paradoxical embolism (via a right-to-left shunt) is a plausible mechanism. - Intracardiac thrombus formation is a common complication in patients with congenital heart defects, espe...

  23. [23]

    - No signs of pancreatitis or peptic ulcer disease based on the clinical presentation

    **Exclusion of Other Causes**: - There is no evidence of infection (no fever, leukocytosis, or elevated CRP). - No signs of pancreatitis or peptic ulcer disease based on the clinical presentation. - The absence of neurological symptoms makes stroke or cerebral embolism less likely. — ### Conclusion: The most likely diagnosis is **intracardiac thrombus for...