Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Converting retrieved documents into a dynamic knowledge graph and judging evidence option-by-option lifts medical multiple-choice accuracy by up to 17.1 percentage points over vanilla RAG and above parametric inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:05 UTC pith:KJMELXT6

load-bearing objection A well-controlled RAG study with a genuinely useful ablation, but headline numbers are provisional: λg is tuned on the test sets and teacher-trace contamination is never tested. the 3 major comments →

arxiv 2607.24838 v1 pith:KJMELXT6 submitted 2026-07-24 cs.IR cs.AI

MedJudgeRAG: Option-Wise Evidence Judgment with Dynamic Knowledge Graphs for Medical MCQA

Seongwon Seo , Seung Hwan Cho , Young-Min Kim This is my paper
classification cs.IR cs.AI
keywords medical multiple-choice QAretrieval-augmented generationdynamic knowledge graphoption-wise evidence judgmentRAG degradationsupervised fine-tuningknowledge utilization strategyevidence verdicts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that retrieval-augmented generation can hurt medical multiple-choice answering not because the retrieved documents are bad, but because the model uses them indiscriminately. MedJudgeRAG trains the model to first condense retrieved documents into a question-specific knowledge graph, then give each answer option an evidence verdict (supported, contradicted, or insufficient), and then choose a strategy: reason from the evidence, eliminate contradicted options, or fall back on parametric knowledge. On two medical benchmarks, with the same backbone and the same retrieved documents as vanilla RAG, the method consistently beats both vanilla RAG and no-retrieval baselines. A secondary claim, supported by ablations, is that the knowledge graph's main value is as training-time supervision: models that skip graph generation at inference usually do as well or better.

Core claim

The central finding is that the RAG-degradation problem in medical MCQA can be fixed without changing retrieval: what matters is the model's internal process for using documents. The paper's pipeline structures retrieved documents into a dynamic knowledge graph of entities and relations tied to source-document indices, requires the model to articulate evidence for each option before assigning a verdict, and makes the subsequent strategy conditional on the verdict pattern. Under same-backbone, same-retrieval comparisons, the largest gains are +17.1 percentage points on MedQA and +11.7 on MedMCQA over vanilla RAG, with consistent improvements over parametric inference. The ablation result refr

What carries the argument

The load-bearing machinery is the three-part generation trace: (1) a dynamic knowledge graph with typed entity and relation tuples plus an evidence field citing document indices; (2) an option-wise judgment block that assigns each option a supported/contradicted/insufficient verdict after citing references; and (3) a decision rule that maps the verdict pattern to grounded, elimination, or parametric reasoning. Training uses supervised fine-tuning on teacher-generated traces with a weighted cross-entropy loss that down-weights graph tokens relative to reasoning tokens; at inference, the model can generate the full trace or skip graph generation. The key design choice is that the graph is not

Load-bearing premise

The load-bearing premise is that the teacher model's reasoning traces are genuinely grounded in the retrieved documents and contain no memorized benchmark answers; the paper excludes the gold answer from the prompt and validates trace format and label agreement, but it never tests whether the teacher has seen the test questions before.

What would settle it

Train the identical pipeline on traces for a set of freshly authored medical MCQA questions that are provably absent from the teacher's training data, then compare against parametric inference on those same questions; if the accuracy advantage over the parametric baseline disappears, the reported gains came largely from distilling memorized answers instead of learning evidence judgment.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RAG can be made reliably beneficial for 7-8B models on medical MCQA without modifying the retriever or the document set; the fix is in how the model is trained to process documents.
  • A model that states evidence before giving a verdict and branches among grounded, elimination, and parametric strategies can recover the accuracy lost by naive document concatenation.
  • Skipping explicit graph generation at inference is often the better operating point, so the extra latency and cost of emitting a knowledge graph are not necessary to realize the gains.
  • The same reasoning structure should transfer to other high-stakes multiple-choice settings where retrieved evidence is noisy and scattered.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the teacher has memorized benchmark answers, part of the student's gain over parametric inference may be distilled answer recall rather than learned evidence judgment; a contamination-controlled replication would separate the two.
  • The three-way verdict scheme can be viewed as an interpretable routing policy; training the same routing directly on correctness rewards could yield further gains, since the paper's own ablations show surface graph quality and answer accuracy decouple.
  • Because implicit decoding wins while sharing the same training-time graph supervision, the graph may be functioning as a regularizer or hidden variable that shapes representations rather than as a literal intermediate; this predicts that graph-conditioned pretraining on unlabeled documents could help even when no graph is emitted.
  • The method's reliance on a fixed, hand-specified schema suggests a testable extension: letting the schema emerge from data or adapting it per question type could improve recall of discriminative evidence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MedJudgeRAG, an SFT framework for medical MCQA. A teacher LM (GPT-5.1) generates structured traces consisting of a dynamic KG extracted from retrieved documents, option-wise evidence verdicts (SUPPORTED/CONTRADICTED/INSUFFICIENT), a decision (grounded/elimination/parametric), and a final answer. A student LM (Mistral-7B or Llama-3-8B) is fine-tuned with a weighted cross-entropy loss that down-weights KG tokens. At inference, the model can explicitly generate the KG before reasoning (Explicit) or omit it (Implicit). Experiments on MedQA and MedMCQA report that MedJudgeRAG outperforms Vanilla RAG and Parametric baselines under identical retrieval, with the largest gain +17.1pp (Llama, MedQA, Implicit); ablations indicate the KG is most useful as training-time supervision rather than as an explicit inference output.

Significance. MedJudgeRAG addresses a real problem: vanilla RAG degrades 7-8B LMs on MedQA/MedMCQA, and the controlled comparison (same backbone, same top-5 retrieval, only the utilization method varies) is the right experimental design. The main result—that the proposed training leads to large gains over vanilla RAG and also exceeds parametric inference—is plausible and, if confirmed, would be practically useful. The paper also contributes a released codebase and generated reasoning traces, plus careful ablations (Explicit vs Implicit decoding, w/o KG, λg sweep) that give the first evidence that the KG mainly acts as training-time supervision. However, the empirical magnitudes are weakened by test-set-based hyperparameter selection, and the supervision provenance is not verified. The machine-checkable parts—schema-aware post-processing and decision-rule adherence in Table 5—are strengths.

major comments (3)
  1. [§3.2, §3.4] Document-grounding is asserted but not verified. Post-processing normalizes entity types, filters relation types, and removes dangling references; it never checks that an entity name/description appears in the cited document, and semantic validation only checks format/consistency/gold agreement. Since GPT-5.1 may have memorized MedQA/MedMCQA, the Table 1 gains could partly come from distilling memorized answers rather than from learned evidence judgment; the w/o KG control keeps the same gold-filtered traces and does not remove this path. Add a grounding check (entity containment in cited docs) and/or a teacher memorization probe (corrupted/empty documents), and report pass rates.
  2. [§4.1, Table 2] The hyperparameter λg is selected on the test/validation sets. The text says 'select the best-performing λg based on average accuracy' computed on MedQA test and MedMCQA validation, with no separate development split. The headline numbers in Table 1 (e.g., +17.1pp) are therefore optimistically biased by test-set selection. Table 2 shows all λg settings still beat Vanilla RAG, so the qualitative claim is robust, but the reported magnitudes should be re-estimated after choosing λg on a held-out validation split, or the paper should transparently disclose this selection procedure and not present the numbers as unbiased.
  3. [§3.4, §5.2, Appendix C.4] The training protocol for Implicit decoding and the w/o KG control is under-specified. The paper says that at λg=0 the model learns to reason conditioned on the gold KG prefix, but the Implicit prompt removes KG construction and asks the model to assume no KG. It is never stated whether Implicit training uses the same full-trace target with a masked KG loss or a reduced target without the KG segment; similarly, how the w/o KG control is trained beyond 'removing all KG-related signals' is not precise. This ambiguity makes the central ablation claim—that the KG is best used as graph-conditioned training-time supervision—hard to evaluate. Specify the exact target sequences, loss masks, and prompts used in each condition.
minor comments (5)
  1. [§4.1] Clarify how 'average accuracy' is computed (macro over benchmarks? over decoding modes?) and why Table 1 uses λg=0.0 globally when Table 2 shows λg=0.1 slightly better for Mistral/MedQA Implicit.
  2. [§5.2] The w/o KG control still outperforms Vanilla RAG; consider reporting its gap to Parametric as well, since the paper's framing is about fixing RAG-induced degradation.
  3. [§4.2 / Table 2] No statistical significance or confidence intervals are reported. Many differences in Table 2 are within a few pp; a McNemar test or repeated-seed standard deviations would strengthen the 'consistent improvements' claim.
  4. [§2.2] The evaluation compares only against Parametric and Vanilla RAG. A sentence justifying why other RAG reasoning methods (Self-RAG, InstructRAG, REANO, MedGraphRAG) are not compared under the same retrieval would help position the contribution.
  5. [Throughout] Minor typos/formatting: missing space after '(1)SUPPORTED' in §3.3; 'n/a' entries in Table 4 are fine but could be explained in the caption; Table 1 footnote could be moved to the text for visibility.

Circularity Check

1 steps flagged

No derivation-level circularity; one test-set-selected hyperparameter mildly inflates the headline numbers.

specific steps
  1. fitted input called prediction [§4.1 Implementation Details; §3.4 Teacher Data Generation; Table 1 caption]
    "We experiment with KG loss weight λg ∈ {0.0,0.1,0.3,0.5,1.0} and select the best-performing λg based on average accuracy. ... We evaluate on the held-out MedQA test split and the MedMCQA validation split. ... AllOursresults use λg = 0.0."

    λg is chosen by maximizing average accuracy on the MedQA test split and MedMCQA validation split that are then used to report Table 1; Table 1's 'Ours' numbers are exactly the λg=0.0 rows of Table 2. The headline result is therefore the argmax of a five-point hyperparameter grid evaluated on the same benchmark splits, so the reported accuracy is in-sample with respect to hyperparameter selection rather than an independent prediction. This is a genuine but mild evaluation-circularity: the central comparison to Vanilla RAG survives across essentially all λg settings in Table 2, so it is not load-bearing for the qualitative claim.

full rationale

The derivation chain itself is self-contained. Eq. (1) is an ordinary weighted cross-entropy over explicit SFT targets; the KG schema, verdict rules, and decision rules are stated; and the evaluation compares against parametric and Vanilla RAG under identical backbones and identical retrieved documents, with a rule-based extractor on external benchmarks. The remaining concerns are data-selection/provenance risks, not derivation-level circularity. The gold-label filter on teacher traces (§3.4) could let a memorizing teacher inject benchmark answers through the SFT targets, and document-groundedness is only prompted, not checked against retrieved documents; this would undermine the evidence-judgment interpretation but does not make the student's output definitionally equal to the teacher's answer. No self-citation chain, uniqueness import, or ansatz-by-citation is load-bearing. Overall circularity is minimal.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The method rests on two invented intermediate representations (dynamic KG tuples and option-wise verdicts) with no external ground truth; the paper validates internal consistency (Table 5) but not verdict correctness. The main free parameter, λg, is chosen on the test labels. The SFT supervision assumes the proprietary teacher traces are faithful and uncontaminated.

free parameters (3)
  • λg (KG loss weight) = 0.0 (selected as best)
    Hyperparameter tuned by maximum average accuracy on the MedQA test set and MedMCQA validation set (§4.1; Table 2). Selection is on the evaluation data, so reported gains are optimistically biased.
  • Training sample size per benchmark = 2750 per benchmark (5500 raw, 3148/3222 after filtering)
    Ad hoc choice with no scaling analysis; the filtered counts differ per backbone (§3.4).
  • QLoRA rank/alpha/dropout = Not reported
    Fine-tuning configuration needed to reproduce the SFT; not specified in the paper.
axioms (4)
  • domain assumption Top-5 Contriever retrieval over PubMed + textbooks supplies question-discriminative evidence for a large share of MedQA and MedMCQA questions.
    The whole evidence-judgment mechanism presumes useful documents are retrieved; the paper never reports retrieval quality, the distribution of the grounded/elimination/parametric decisions, or what fraction of questions rely on retrieval.
  • domain assumption GPT-5.1's teacher traces are faithful and free of benchmark memorization.
    SFT data comes exclusively from GPT-5.1 (§3.4), filtered for correctness but never tested for contamination. If the teacher has seen the test questions, the student inherits memorized answers.
  • domain assumption The UMLS 15 entity types and 15 relation types suffice to express task-relevant medical evidence.
    The KG schema (§3.2, Appendix C.1) restricts all extracted knowledge; no analysis is given for whether this schema loses discriminative information.
  • domain assumption The deterministic three-way decision rule is an appropriate use of verdicts.
    §3.3 fixes grounded/elimination/parametric without comparing to alternative verdict-to-answer mappings; the paper only measures adherence to its own rule (Table 5), not the quality of the verdicts themselves.
invented entities (2)
  • Dynamic knowledge graph tuples (Entity/Relation with Evidence) no independent evidence
    purpose: Structures retrieved documents into a question-specific graph used for evidence judgment.
    No gold-standard KG is used; the graphs are produced by the teacher LM and validated only for schema compliance, so there is no external handle to verify their correctness.
  • Option-wise verdict labels (SUPPORTED/CONTRADICTED/INSUFFICIENT) no independent evidence
    purpose: Intermediate classification that determines which of three reasoning strategies the model uses.
    Verdict accuracy is not measured against ground truth; only adherence to the decision rule (Table 5) and final answer accuracy are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 16971 in / 16733 out tokens · 158213 ms · 2026-08-01T06:05:29.080788+00:00 · methodology

0 comments
read the original abstract

In medical multiple-choice question answering (MCQA), Retrieval-Augmented Generation (RAG) can supplement the domain knowledge of language models (LMs). However, since vanilla RAG indiscriminately utilizes retrieved documents, it can degrade LM performance. To address this, we propose MedJudgeRAG. Our framework represents retrieved documents as a dynamic knowledge graph (KG) composed of entities and relations. For each option, the model judges an evidence verdict from the retrieved documents and the KG. Based on the verdict combination, the model determines a knowledge utilization strategy to reason toward the final answer. These capabilities are trained via supervised fine-tuning using structured reasoning traces generated by a teacher LM. The training employs a weighted cross-entropy loss that differentially weights the KG and reasoning segments. Experiments on two medical MCQA benchmarks demonstrate that MedJudgeRAG consistently outperforms both vanilla RAG and parametric baselines. Furthermore, ablation analysis reveals that the dynamic KG is more effective as graph-conditioned supervision at training time than as an explicit output at inference time. Our code is available at https://github.com/hyu-amllab/medjudgerag, and the generated reasoning traces are released at https://huggingface.co/datasets/youarethewon/medjudgerag.

Figures

Figures reproduced from arXiv: 2607.24838 by Seongwon Seo, Seung Hwan Cho, Young-Min Kim.

Figure 1
Figure 1. Figure 1: Overview of the MedJudgeRAG framework. Top: the teacher LM Mϕ (GPT-5.1) generates structured reasoning traces consisting of a dynamic knowledge graph, option-wise evidence judgments, and a decision. Bottom: the student LM Mθ (Mistral/Llama) is trained via supervised fine-tuning on the teacher’s outputs and produces the same structured output at inference time. Dashed lines indicate that the teacher’s outpu… view at source ↗
Figure 2
Figure 2. Figure 2: Effect of KG loss weight (λg) on KG quality and answer accuracy under Explicit decoding. All results are reported across 4 (backbone × benchmark) combinations and 5 λg values (0.0, 0.1, 0.3, 0.5, 1.0). tion failures to the reasoning segment after <ANALYSIS> delimiter. It also encroaches on the token budget within the maximum output sequence length [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 7 linked inside Pith

  1. [1]

    Nature medicine , volume=

    Toward expert-level medical question answering with large language models , author=. Nature medicine , volume=. 2025 , publisher=

    2025

  2. [2]

    arXiv preprint arXiv:2501.18362 , year=

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding , author=. arXiv preprint arXiv:2501.18362 , year=

    Pith/arXiv arXiv

  3. [3]

    Applied Sciences , volume=

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams , author=. Applied Sciences , volume=. 2021 , publisher=

    2021

  4. [4]

    Conference on health, inference, and learning , pages=

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering , author=. Conference on health, inference, and learning , pages=. 2022 , organization=

    2022

  5. [5]

    Advances in neural information processing systems , volume=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in neural information processing systems , volume=

  6. [6]

    International conference on machine learning , pages=

    Retrieval augmented language model pre-training , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  7. [7]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Active retrieval augmented generation , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  8. [8]

    The Twelfth International Conference on Learning Representations , year=

    Ra-dit: Retrieval-augmented dual instruction tuning , author=. The Twelfth International Conference on Learning Representations , year=

  9. [9]

    The Twelfth International Conference on Learning Representations , year=

    Self-rag: Learning to retrieve, generate, and critique through self-reflection , author=. The Twelfth International Conference on Learning Representations , year=

  10. [10]

    arXiv preprint arXiv:2406.13629 , year=

    InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales , author=. arXiv preprint arXiv:2406.13629 , year=

    Pith/arXiv arXiv

  11. [11]

    IEEE transactions on neural networks and learning systems , volume=

    A survey on knowledge graphs: Representation, acquisition, and applications , author=. IEEE transactions on neural networks and learning systems , volume=. 2021 , publisher=

    2021

  12. [12]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Knowledge graph prompting for multi-document question answering , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  13. [13]

    arXiv preprint arXiv:2404.16130 , year=

    From local to global: A graph rag approach to query-focused summarization , author=. arXiv preprint arXiv:2404.16130 , year=

    Pith/arXiv arXiv

  14. [14]

    Advances in neural information processing systems , volume=

    Hipporag: Neurobiologically inspired long-term memory for large language models , author=. Advances in neural information processing systems , volume=

  15. [15]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    REANO: Optimising retrieval-augmented reader models through knowledge graph generation , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  16. [16]

    arXiv preprint arXiv:2502.10996 , year=

    Ras: Retrieval-and-structuring for knowledge-intensive llm generation , author=. arXiv preprint arXiv:2502.10996 , year=

    arXiv

  17. [17]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Medical graph rag: Evidence-based medical large language model via graph retrieval-augmented generation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  18. [18]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    TAdaRAG: Task Adaptive Retrieval-Augmented Generation via On-the-Fly Knowledge Graph Construction , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  19. [19]

    Findings of the association for computational linguistics: acl 2024 , pages=

    Biomistral: A collection of open-source pretrained large language models for medical domains , author=. Findings of the association for computational linguistics: acl 2024 , pages=

    2024

  20. [20]

    arXiv preprint arXiv:2311.16079 , year=

    Meditron-70b: Scaling medical pretraining for large language models , author=. arXiv preprint arXiv:2311.16079 , year=

    Pith/arXiv arXiv

  21. [21]

    NPJ digital medicine , volume=

    Small language models learn enhanced reasoning skills from medical textbooks , author=. NPJ digital medicine , volume=. 2025 , publisher=

    2025

  22. [22]

    arXiv preprint arXiv:2408.06142 , year=

    Med42-v2: A suite of clinical llms , author=. arXiv preprint arXiv:2408.06142 , year=

    Pith/arXiv arXiv

  23. [23]

    Rationale-guided retrieval augmented generation for medical question answering , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  24. [24]

    Bioinformatics , volume=

    Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models , author=. Bioinformatics , volume=. 2024 , publisher=

    2024

  25. [25]

    Pubmedqa: A dataset for biomedical research question answering , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  26. [26]

    International Conference on Learning Representations , year=

    Measuring Massive Multitask Language Understanding , author=. International Conference on Learning Representations , year=

  27. [27]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  28. [28]

    2025 , url =

    OpenAI , title =. 2025 , url =

    2025

  29. [29]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  30. [30]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  31. [31]

    Benchmarking Retrieval-Augmented Generation for Medicine

    Xiong, Guangzhi and Jin, Qiao and Lu, Zhiyong and Zhang, Aidong. Benchmarking Retrieval-Augmented Generation for Medicine. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.372

    doi:10.18653/v1/2024.findings-acl.372 2024

  32. [32]

    arXiv preprint arXiv:2112.09118 , year=

    Unsupervised dense information retrieval with contrastive learning , author=. arXiv preprint arXiv:2112.09118 , year=

    Pith/arXiv arXiv

  33. [33]

    Advances in neural information processing systems , volume=

    Qlora: Efficient finetuning of quantized llms , author=. Advances in neural information processing systems , volume=