{"id":"637e22e4-5a66-4ca7-ad64-acfc5838575c","arxiv_id":"2607.24838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MedJudgeRAG fine-tunes a medical MCQA model to emit per-option evidence verdicts and choose grounded, elimination, or parametric reasoning, improving over vanilla RAG by up to 17 accuracy points.","lead":"This paper presents MedJudgeRAG, a fine-tuned pipeline that converts retrieved medical documents into a small knowledge graph, judges each answer option as supported, contradicted, or insufficiently evidenced, and then picks a reasoning strategy. On MedQA and MedMCQA it outscores both a no-retrieval model and vanilla retrieval-augmented generation by up to 17 accuracy points, and its ablations suggest the graph mainly improves training rather than inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher-trace contamination is untested: student gains may distill memorized answers rather than learned evidence judgment.","rationale":"The reader's weakest assumption matches the main risk I see. The paper's headline numbers are internally consistent and the ablations are informative, but they all share a single supervision source whose document-groundedness is asserted, not verified. The teacher prompt is strong (§C.1), yet there is no automated check that entities/relations cited in each trace actually occur in the cited documents, and the one-shot example is not enough to guarantee adherence across 5,500 traces. Because the traces are filtered on agreement with the gold label, any leaked answer knowledge in the teacher is inherited by the student; this is not controlled by the w/o KG ablation. I therefore agree with the CONDITIONAL verdict. The proposed permutation test is cheap and would either clear the teacher or quantify how much of the gain is memorization rather than evidence judgment. If the test passes, the central claim is substantially strengthened; if it fails, the claimed RAG-fixing mechanism needs re-examination, but the method may still be a valid distillation approach.","tokens_in":17317,"tokens_out":6251,"duration_ms":71976,"concrete_test":"On a random sample of 200 MedQA test questions, run the teacher (same GPT-5.1 API, same prompt template) twice: once with original options and once with the four options permuted and the gold label remapped accordingly. If the teacher is genuinely grounding its verdicts in the documents, its answer choice should track the content of the correct option under permutation; if it is recalling memorized benchmark instances, it will tend to output the original letter and its accuracy will drop sharply. The concern lands if permutation-robust accuracy is substantially below original-order accuracy (e.g., >10 pp drop) or if SUPPORTED verdicts no longer align with the permuted evidence. If that happens, retrain the student on permutation-filtered traces and check whether the Table 1 gains persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Table 1, §4.2) is that MedJudgeRAG's gains over Vanilla RAG and the parametric baseline come from learning option-wise evidence judgment. The supervision traces are generated by GPT-5.1 with the gold answer excluded from the prompt, then filtered for format, rule consistency, and agreement with the gold label (§3.4). That filtering makes the final trace set answer-correct by construction, but it does not verify that the KG entities, relations, or evidence statements are actually extracted from the retrieved documents. The post-processing in §3.2 checks entity-type normalization, relation-type validity, and dangling references; it does not check whether an entity name or description appears in the cited document. GPT-5.1 is proprietary and was trained on data that almost certainly includes MedQA (2021) and MedMCQA (2022). If the teacher has memorized benchmark instances, the 'document-grounded' instruction in the prompt is not enforced, and the student is trained to reproduce answer labels and plausible post-hoc rationales. The +17.1pp/+11.7pp improvements over Vanilla RAG would then partly reflect distillation of memorized answers, not the proposed evidence-judgment mechanism. The w/o KG control (§5.2) does not remove this path because it keeps the same gold-filtered teacher traces.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MedJudgeRAG, an SFT framework for medical MCQA. A teacher LM (GPT-5.1) generates structured traces consisting of a dynamic KG extracted from retrieved documents, option-wise evidence verdicts (SUPPORTED/CONTRADICTED/INSUFFICIENT), a decision (grounded/elimination/parametric), and a final answer. A student LM (Mistral-7B or Llama-3-8B) is fine-tuned with a weighted cross-entropy loss that down-weights KG tokens. At inference, the model can explicitly generate the KG before reasoning (Explicit) or omit it (Implicit). Experiments on MedQA and MedMCQA report that MedJudgeRAG outperforms Vanilla RAG and Parametric baselines under identical retrieval, with the largest gain +17.1pp (Llama, MedQA, Implicit); ablations indicate the KG is most useful as training-time supervision rather than as an explicit inference output.","tokens_in":17655,"tokens_out":9402,"duration_ms":96562,"significance":"MedJudgeRAG addresses a real problem: vanilla RAG degrades 7-8B LMs on MedQA/MedMCQA, and the controlled comparison (same backbone, same top-5 retrieval, only the utilization method varies) is the right experimental design. The main result—that the proposed training leads to large gains over vanilla RAG and also exceeds parametric inference—is plausible and, if confirmed, would be practically useful. The paper also contributes a released codebase and generated reasoning traces, plus careful ablations (Explicit vs Implicit decoding, w/o KG, λg sweep) that give the first evidence that the KG mainly acts as training-time supervision. However, the empirical magnitudes are weakened by test-set-based hyperparameter selection, and the supervision provenance is not verified. The machine-checkable parts—schema-aware post-processing and decision-rule adherence in Table 5—are strengths.","major_comments":[{"comment":"Document-grounding is asserted but not verified. Post-processing normalizes entity types, filters relation types, and removes dangling references; it never checks that an entity name/description appears in the cited document, and semantic validation only checks format/consistency/gold agreement. Since GPT-5.1 may have memorized MedQA/MedMCQA, the Table 1 gains could partly come from distilling memorized answers rather than from learned evidence judgment; the w/o KG control keeps the same gold-filtered traces and does not remove this path. Add a grounding check (entity containment in cited docs) and/or a teacher memorization probe (corrupted/empty documents), and report pass rates.","section":"§3.2, §3.4"},{"comment":"The hyperparameter λg is selected on the test/validation sets. The text says 'select the best-performing λg based on average accuracy' computed on MedQA test and MedMCQA validation, with no separate development split. The headline numbers in Table 1 (e.g., +17.1pp) are therefore optimistically biased by test-set selection. Table 2 shows all λg settings still beat Vanilla RAG, so the qualitative claim is robust, but the reported magnitudes should be re-estimated after choosing λg on a held-out validation split, or the paper should transparently disclose this selection procedure and not present the numbers as unbiased.","section":"§4.1, Table 2"},{"comment":"The training protocol for Implicit decoding and the w/o KG control is under-specified. The paper says that at λg=0 the model learns to reason conditioned on the gold KG prefix, but the Implicit prompt removes KG construction and asks the model to assume no KG. It is never stated whether Implicit training uses the same full-trace target with a masked KG loss or a reduced target without the KG segment; similarly, how the w/o KG control is trained beyond 'removing all KG-related signals' is not precise. This ambiguity makes the central ablation claim—that the KG is best used as graph-conditioned training-time supervision—hard to evaluate. Specify the exact target sequences, loss masks, and prompts used in each condition.","section":"§3.4, §5.2, Appendix C.4"}],"minor_comments":[{"comment":"Clarify how 'average accuracy' is computed (macro over benchmarks? over decoding modes?) and why Table 1 uses λg=0.0 globally when Table 2 shows λg=0.1 slightly better for Mistral/MedQA Implicit.","section":"§4.1"},{"comment":"The w/o KG control still outperforms Vanilla RAG; consider reporting its gap to Parametric as well, since the paper's framing is about fixing RAG-induced degradation.","section":"§5.2"},{"comment":"No statistical significance or confidence intervals are reported. Many differences in Table 2 are within a few pp; a McNemar test or repeated-seed standard deviations would strengthen the 'consistent improvements' claim.","section":"§4.2 / Table 2"},{"comment":"The evaluation compares only against Parametric and Vanilla RAG. A sentence justifying why other RAG reasoning methods (Self-RAG, InstructRAG, REANO, MedGraphRAG) are not compared under the same retrieval would help position the contribution.","section":"§2.2"},{"comment":"Minor typos/formatting: missing space after '(1)SUPPORTED' in §3.3; 'n/a' entries in Table 4 are fine but could be explained in the caption; Table 1 footnote could be moved to the text for visibility.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The controlled experimental design and the released code/data are genuine strengths. The main risk is teacher-trace contamination: if GPT-5.1 has memorized these benchmarks, the mechanism story is substantially weakened. This is testable (grounding checks, corrupted-document probes) and should be requested. The λg selection issue is also fixable. The paper is otherwise a solid empirical contribution, but I would not accept it without addressing these two points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nBottom line: this is a solid, transparent piece of empirical RAG work, and the controlled comparison is the right design. Under identical backbones and identical top-5 Contriever retrieval, MedJudgeRAG beats vanilla RAG in nearly every setting reported, and the best setting improves over vanilla RAG by double-digit points. The w/o KG ablation is the real find: removing all KG tokens/references from the training target but keeping the reasoning structure still beats vanilla RAG, and the graph-conditioned model wins 18/20 comparisons. That supports their claim that the dynamic KG acts mainly as training-time supervision, not an inference-time output. Code and traces are released, which is more than most IR papers do.\n\nWhat's new is the specific recipe: UMLS-constrained dynamic KG extraction, per-option verdicts (SUPPORTED/CONTRADICTED/INSUFFICIENT), a three-way grounded/elimination/parametric decision rule, and a weighted loss that downweights KG tokens. All components have precedents, but the combination and the ablation are not in the cited work. That is enough to justify reviewer attention.\n\nThe soft spots are real. First, λg is selected on the same test/validation sets — the paper says 'select the best-performing λg based on average accuracy.' They report the full grid in Table 2, so it's transparent, but the headline numbers are maxes over a small grid tuned on the held-out sets. A held-out val split or a correction for multiple comparisons would tighten this. Second, and more serious: the teacher-trace contamination path is untested. GPT-5.1 is proprietary and MedQA/MedMCQA are old benchmarks that are almost certainly in its training data. The prompt excludes the gold answer, but the validation pipeline filters for format, rule consistency, and gold-label agreement — not for whether KG entities/relations actually appear in the cited documents. The prompt instructs document grounding, and the post-processing checks schema, but the stress-test concern stands: a memorizing teacher could produce plausible post-hoc rationales that are answer-correct by construction, and the student would be distilling memorized answers. Because the w/o KG control keeps the same gold-filtered traces, it doesn't remove that path either. That weakens the mechanistic interpretation of the gains, not the comparative result against vanilla RAG. Third, there are no repeated runs or significance tests; single-seed results for 2-6 pp differences are fragile.\n\nWho should read this: anyone working on RAG for medical QA or on structured-reasoning distillation. It deserves a serious referee. My recommendation: send it to peer review, with reviewers asked to (i) probe the teacher for benchmark contamination, (ii) select λg on a separate split, and (iii) add multiple seeds. With those, the empirical claim would be trustworthy; without them, treat the headline numbers as provisional.","headline":"A well-controlled RAG study with a genuinely useful ablation, but headline numbers are provisional: λg is tuned on the test sets and teacher-trace contamination is never tested.","tokens_in":18152,"tokens_out":4445,"would_cite":true,"duration_ms":47275,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Converting retrieved documents into a dynamic knowledge graph and judging evidence option-by-option lifts medical multiple-choice accuracy by up to 17.1 percentage points over vanilla RAG and above parametric inference.","keywords":["medical multiple-choice QA","retrieval-augmented generation","dynamic knowledge graph","option-wise evidence judgment","RAG degradation","supervised fine-tuning","knowledge utilization strategy","evidence verdicts"],"falsifier":"Train the identical pipeline on traces for a set of freshly authored medical MCQA questions that are provably absent from the teacher's training data, then compare against parametric inference on those same questions; if the accuracy advantage over the parametric baseline disappears, the reported gains came largely from distilling memorized answers instead of learning evidence judgment.","tokens_in":17210,"feed_emoji":"🩺","tokens_out":6444,"duration_ms":64983,"temperature":0.7,"pith_summary":"This paper tries to establish that retrieval-augmented generation can hurt medical multiple-choice answering not because the retrieved documents are bad, but because the model uses them indiscriminately. MedJudgeRAG trains the model to first condense retrieved documents into a question-specific knowledge graph, then give each answer option an evidence verdict (supported, contradicted, or insufficient), and then choose a strategy: reason from the evidence, eliminate contradicted options, or fall back on parametric knowledge. On two medical benchmarks, with the same backbone and the same retrieved documents as vanilla RAG, the method consistently beats both vanilla RAG and no-retrieval baselines. A secondary claim, supported by ablations, is that the knowledge graph's main value is as training-time supervision: models that skip graph generation at inference usually do as well or better.","feed_headline":"Judge each option's evidence, gain 17 points in medical QA","feed_subtitle":"Graph-structured evidence and per-option verdicts beat naive RAG and no-retrieval models on medical QA.","key_machinery":"The load-bearing machinery is the three-part generation trace: (1) a dynamic knowledge graph with typed entity and relation tuples plus an evidence field citing document indices; (2) an option-wise judgment block that assigns each option a supported/contradicted/insufficient verdict after citing references; and (3) a decision rule that maps the verdict pattern to grounded, elimination, or parametric reasoning. Training uses supervised fine-tuning on teacher-generated traces with a weighted cross-entropy loss that down-weights graph tokens relative to reasoning tokens; at inference, the model can generate the full trace or skip graph generation. The key design choice is that the graph is not","core_discovery":"The central finding is that the RAG-degradation problem in medical MCQA can be fixed without changing retrieval: what matters is the model's internal process for using documents. The paper's pipeline structures retrieved documents into a dynamic knowledge graph of entities and relations tied to source-document indices, requires the model to articulate evidence for each option before assigning a verdict, and makes the subsequent strategy conditional on the verdict pattern. Under same-backbone, same-retrieval comparisons, the largest gains are +17.1 percentage points on MedQA and +11.7 on MedMCQA over vanilla RAG, with consistent improvements over parametric inference. The ablation result refr","pith_inferences":["If the teacher has memorized benchmark answers, part of the student's gain over parametric inference may be distilled answer recall rather than learned evidence judgment; a contamination-controlled replication would separate the two.","The three-way verdict scheme can be viewed as an interpretable routing policy; training the same routing directly on correctness rewards could yield further gains, since the paper's own ablations show surface graph quality and answer accuracy decouple.","Because implicit decoding wins while sharing the same training-time graph supervision, the graph may be functioning as a regularizer or hidden variable that shapes representations rather than as a literal intermediate; this predicts that graph-conditioned pretraining on unlabeled documents could help even when no graph is emitted.","The method's reliance on a fixed, hand-specified schema suggests a testable extension: letting the schema emerge from data or adapting it per question type could improve recall of discriminative evidence."],"forward_implications":["RAG can be made reliably beneficial for 7-8B models on medical MCQA without modifying the retriever or the document set; the fix is in how the model is trained to process documents.","A model that states evidence before giving a verdict and branches among grounded, elimination, and parametric strategies can recover the accuracy lost by naive document concatenation.","Skipping explicit graph generation at inference is often the better operating point, so the extra latency and cost of emitting a knowledge graph are not necessary to realize the gains.","The same reasoning structure should transfer to other high-stakes multiple-choice settings where retrieved evidence is noisy and scattered."],"fun_headline_variants":["Option-wise evidence judgments beat vanilla RAG by 17 points","Dynamic knowledge graphs for medical MCQA: +17 pts over RAG","Per-option evidence verdicts prevent RAG degradation in QA","Graph-conditioned training, not graph output, improves medical QA","Verifying each option's evidence beats naive RAG in MCQA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the teacher model's reasoning traces are genuinely grounded in the retrieved documents and contain no memorized benchmark answers; the paper excludes the gold answer from the prompt and validates trace format and label agreement, but it never tests whether the teacher has seen the test questions before.","fun_headline_variants_meta":{"raw":{"variants":["Option-wise evidence judgments beat vanilla RAG by 17 points","Dynamic knowledge graphs for medical MCQA: +17 pts over RAG","Per-option evidence verdicts prevent RAG degradation in QA","Graph-conditioned training, not graph output, improves medical QA","Verifying each option's evidence beats naive RAG in MCQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3021,"prompt_tokens":755,"completion_tokens":2266,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2177}},"tokens_in":499,"tokens_out":2266,"duration_ms":16932,"temperature":1.0,"reasoning_tokens":2177,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:05:29.080788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical pipeline on traces for a set of freshly authored medical MCQA questions that are provably absent from the teacher's training data, then compare against parametric inference on those same questions; if the accuracy advantage over the parametric baseline disappears, the reported gains came largely from distilling memorized answers instead of learning evidence judgment.","supporting_citations":[],"review_version":1}