{"id":"034392e4-6415-4e23-86f5-a96ac1e9ce08","arxiv_id":"2505.11887","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A medical evaluation LLM trained with curriculum instruction tuning and iterative knowledge introspection correlates with human judgments better than GPT-4 and other baselines on medical QA responses.","lead":"AutoMedEval is a 13-billion-parameter open-source language model that scores and explains answers from medical chatbots. It is trained to match the judgments of doctors, and in the paper's tests it agrees with human raters more often than GPT-4 and other baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 958 wikidoc test entries share their source with the 9,874 training instructions, and the paper never states or verifies a disjoint split; without that guarantee, the reported Spearman 0.6314 / Accuracy-triple 48.65 advantage over GPT-4 and Gemini could reflect memorization rather than general…","rationale":"The reader's weakest assumption is also the condition that matters most for the central claim. AutoMedEval is fine-tuned on GPT-4 evaluations of 9,874 wikidoc QA tuples (Section 3.1), and the test set draws 958 of its entries from the same Medical Meadow Wikidoc source (Section 4.1). The paper reports no split procedure, no question IDs, and no deduplication, so a reader cannot rule out that some test questions are in the training instruction set. Because the iterative knowledge introspection stage even revises the same training instances (Algorithm 2), an overlapping question could supply the model with a near-reference evaluation, inflating comparative metrics against zero-shot baselines. The inclusion of MedAlpaca and MedLlama2 responses in the test reduces but does not eliminate the problem, since question-level familiarity can still bias scoring. The 172 MedText entries are the only clearly external portion and are not reported separately. I see no internal inconsistency in the training method itself, and the concern is testable; if a verified disjoint split or a stable result on MedText is provided, the main comparison would be convincing. This is the same condition the reader flagged, so I keep the CONDITIONAL verdict and recommend no change.","tokens_in":14482,"tokens_out":7357,"duration_ms":74259,"concrete_test":"Release or reconstruct the exact question identifiers for the 9,874 training entries and the 958 wikidoc test entries, then run exact-string and near-duplicate matching (normalized text plus embedding similarity) across the two sets. Recompute Table 2's Spearman and Accuracy-triple separately on (a) the test entries with no training match and (b) the 172 MedText-only entries. If the AutoMedEval advantage over GPT-4/Gemini survives on the disjoint subset, the concern is resolved; if the advantage shrinks materially there, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs the instruction dataset from 9,874 medical-meadow-wikidoc cases (D = T \\ F1), and Section 3.2 fine-tunes AutoMedEval on GPT-4 and ChatGPT evaluations of those cases, with additional iterative revision on the same instances (Algorithm 2). Section 4.1 then defines the test set as \"mainly includes 958 entries from Medical Meadow Wikidoc dataset\" plus 172 MedText entries. The paper never states that the 958 wikidoc test questions are disjoint from the 9,874 training questions, reports no split procedure, no question IDs, and no deduplication. Because the model is trained on distilled GPT-4 evaluations of exactly this question type, any test question that also appears in training gives AutoMedEval access to a reference evaluation for that question; the reported Spearman 0.6314 and Accuracy-triple 48.65 would then reflect memorization of GPT-4's evaluation style on overlapping questions rather than a general medical evaluation capability. Partial mitigation (the test responses include MedAlpaca and MedLlama2 outputs not seen in training) does not remove the concern, since question-level memorization can still bias scoring of the newly generated responses. The 172 MedText entries are the only clearly external portion and should be reported separately. This is an evidentiary gap, not an observed flaw: with a verified disjoint split the central comparison could well stand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AutoMedEval, a 13B open-source evaluation model for medical question-answering. The authors construct an instruction dataset from roughly 9,874 medical-meadow-wikidoc question-answer pairs, have GPT-4 and ChatGPT produce evaluations with retrieval-augmented knowledge completion chains, and have physicians verify a subset of the resulting evaluations. The model is trained on MedLLaMA-13B using curriculum instruction tuning followed by iterative knowledge introspection, in which incorrect training cases are revised through an AI-doctor collaborative feedback loop. The central empirical claim is that AutoMedEval achieves higher correlation with human judgments than all tested baselines, including GPT-4, on response-level Spearman/Pearson correlations, case-level accuracy metrics, and a double-blind preference experiment.","tokens_in":14757,"tokens_out":2801,"duration_ms":30519,"significance":"If the central comparative claims hold under scrutiny, AutoMedEval would be a useful open-sourced building block for medical LLM evaluation, addressing a real gap left by proprietary judges and general-domain evaluators. The paper has notable strengths: the instruction dataset is physician-verified at scale, the ablations isolate the contributions of the knowledge completion chain, curriculum instruction tuning, and iterative knowledge introspection, and the evaluation is anchored in human annotations with inter-annotator reliability reported. The proposed iterative introspection loop is a practical contribution that could transfer to other domain-specific evaluators. However, the significance is currently conditional on resolving a train/test overlap concern and on reframing the rationale-evaluation metrics, since both bear directly on whether the reported advantage over GPT-4 reflects generalizable evaluation capability or imitation/memorization of GPT-4's evaluation style.","major_comments":[{"comment":"The manuscript never states whether the 958-entry Medical Meadow Wikidoc test set is disjoint from the 9,874 training cases drawn from the same source dataset. Because AutoMedEval is trained on GPT-4 and ChatGPT evaluations of wikidoc question-response tuples, any test question that also appears in the instruction-tuning data gives the model direct access to a reference evaluation for that question, which could inflate the reported Spearman 0.6314 and Accuracy-triple 48.65 through memorization rather than generalizable evaluation competence. Partial mitigation from using unseen response generators (MedAlpaca, MedLlama2) does not remove the concern, since question-level memorization can still bias scoring of newly generated responses. The authors should report the exact split procedure, provide question-level deduplication or IDs, and give the 172 MedText results separately as the only clearly external portion of the test set.","section":"Sections 3.1 and 4.1"},{"comment":"The Rationale Evaluation uses BERTScore and BARTScore with evaluations generated by GPT-4 as the reference. This is circular for AutoMedEval because the model is explicitly distilled from GPT-4's evaluations on the same type of input; high semantic similarity to GPT-4 rationales demonstrates imitation fidelity, not human-aligned quality. The paper should either present these numbers as a distillation-fidelity check rather than a capability comparison, or replace the GPT-4 reference with a human-authored reference rationale set such as the physician-verified evaluations used elsewhere in the paper.","section":"Section 4.3"},{"comment":"The saturation prediction is fitted to only three observed points (Accuracy-triple 44.61, 47.13, 48.65 at iterations 0, 1, 2), yet the paper claims that AutoMedEval will outperform curriculum instruction tuning by 17% and that there will be no further growth after six iterations. Equation (7) fixes a1=1, a2=0.9, a3=0.586, and the exponent constants are not derived from any stated fitting procedure; with three data points and at least two free parameters, the sigmoid model is not identifiable and cannot support the saturation and 52.21% predictions. The claim that iterative introspection stops improving after six iterations should be removed or supported by running more iterations and by reporting model-selection or uncertainty information.","section":"Appendix F and Section 4.5.2"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, including 'evalution' in Equation (1), 'conducte ablation studies' in Section 4.5.1, 'Resposne 3' in Table 7, and 'oftencing' in the same table; a careful proofread is needed.","section":"Throughout"},{"comment":"The loop notation 'for i ← −1 to |D|' and 'for t ← −1 to T' appears to be off-by-one (likely intended to start at 1), and line 5 'd ← −D_i' is not standard pseudocode; this should be cleaned up for reproducibility.","section":"Algorithm 1"},{"comment":"The inter-annotator reliability values (ICC 0.712, Krippendorff's alpha 0.725) are reported as a single pair of numbers, but the manuscript does not state which annotation task (response scoring or evaluation-content scoring) these values refer to; please clarify.","section":"Section 4.2"},{"comment":"The quality classifier is trained on only 400 labeled examples (200 high, 200 low) and evaluated on 100 test cases; the 91% accuracy should be accompanied by confidence intervals or a confusion matrix, and the paper should note the limited scale when relying on this classifier to select 4,788 + 3,823 training instructions.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue for me is the unverified disjointness of the wikidoc training and test questions. If the authors can provide a clean split or show that all 958 test questions are absent from the 9,874 training cases, the main comparison becomes credible despite the remaining presentation issues. If they cannot, the central claim about surpassing GPT-4 is not defensible from the reported experiments. The rationale-evaluation circularity is also fixable by reinterpreting or replacing the reference, but the overlap question must be resolved before I would recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper builds a 13B open medical evaluator that beats GPT-4, Gemini, and PandaLM on human correlation, and the training recipe—retrieval-augmented GPT-4 distillation, curriculum instruction tuning, iterative knowledge introspection—is a real new combination for medical evaluation, not a routine application of known pieces. Second, the headline result rests on a train/test split that the text never actually states. The training set is 9,874 medical-meadow-wikidoc cases; the test set is “mainly” 958 wikidoc entries. The paper never says the two are disjoint, gives no deduplication procedure, and reports no question IDs. Since the model is trained on GPT-4's evaluations of exactly that question type, any overlap would let it exploit reference evaluations during testing. The stress-test note is right: this is an evidentiary gap, not an observed leak, and the advantage could well survive a clean split, but the authors have to demonstrate it.\n\nWhat is good. The human evaluation is serious: five doctors, ICC and Krippendorff's alpha reported, double-blind preference test. The ablations show each component contributes. The authors also include an honest limitations section and an ethics statement. The citation pattern is fine.\n\nSoft spots beyond the split. The BERTScore/BARTScore rationale comparison uses GPT-4 evaluations as the reference, which is circular for a model distilled from GPT-4; this only affects a secondary metric, but it should be reworked or dropped. The saturation prediction in Appendix F fits a sigmoid to three data points and then extrapolates “no further growth after six iterations”—that is overclaiming; call it an illustration. And despite the abstract promising open-source artifacts, I could not find links to the model, dataset, or code; if they want reproducibility credit, they should ship them.\n\nBottom line: fixable, but the missing split guarantee is load-bearing. The paper deserves a serious referee, not a desk reject. I would ask the authors for a verified disjoint split, separate results on the 172 MedText entries, a de-emphasized saturation claim, and artifact links. For now I would not cite it in my own work; I would reread a revised version.","headline":"AutoMedEval is a useful medical evaluator with a genuinely new training combination, but its central claim depends on a train/test disjointness guarantee the paper never states.","tokens_in":15326,"tokens_out":2547,"would_cite":false,"duration_ms":26720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoMedEval, a 13B open-source medical evaluator trained with curriculum instruction tuning and iterative knowledge introspection, claims to match human doctors' judgments more closely than GPT-4 and every baseline, with Spearman 0.6314…","keywords":["automatic evaluation","medical LLM evaluation","curriculum instruction tuning","iterative knowledge introspection","GPT-4 distillation","LLM-as-a-judge","retrieval-augmented generation","human-aligned evaluation"],"falsifier":"Audit the 958-item Wikidoc and 172-item MedText test questions for exact or near-duplicate overlap with the 9,874 training instructions using string matching and embedding similarity; if any test question appears in training, the reported Spearman advantage cannot be trusted. Alternatively, score a newly collected set of clinical QA pairs that was never used in instruction construction and compare AutoMedEval's Spearman correlation with five doctors' judgments against GPT-4's 0.5128; a drop to that level would falsify the generalization claim.","tokens_in":14255,"feed_emoji":"🩺","tokens_out":16982,"duration_ms":152360,"temperature":0.7,"pith_summary":"The paper's aim is to supply an open, automatic judge for medical question-answering models that agrees with human doctors closely enough to make large-scale expert evaluation unnecessary. AutoMedEval is a 13B parameter evaluator built on MedLLaMA and trained on a physician-verified instruction set distilled from GPT-4 and ChatGPT. The training recipe combines a three-stage curriculum (pattern recognition, mixed data, high-quality GPT-4 evaluations) with an iterative knowledge-introspection loop in which incorrect evaluations are revised through AI-doctor collaboration and used to fine-tune the model again. On a 1,130-question test set, the paper reports that AutoMedEval's scores correlate with human judgments at Spearman 0.6314, ahead of GPT-4's 0.5128 and every other baseline, which is the paper's central evidence that automatic evaluation can approach human-level medical judgment.","feed_headline":"A 13B open medical judge outranks GPT-4 at matching doctors","feed_subtitle":"Its curriculum and introspection training align automatic scores with doctor rankings better than Gemini or GPT-4.","key_machinery":"The carrier of the argument is a three-part training and data machinery. The dynamic knowledge completion chain lets the GPT-4 teacher issue a '[Question]' query when uncertain, retrieve relevant passages from a vector database of medical and evaluation manuals, append them to the prompt, and repeat until the evaluation can be completed; this grounds the instruction data in medical evidence. Curriculum instruction tuning then fine-tunes MedLLaMA-13B sequentially on ChatGPT-generated evaluations (pattern recognition), a mixture of ChatGPT and GPT-4 evaluations, and finally high-quality GPT-4 evaluations, so the model learns evaluation patterns before quality criteria. Iterative knowledge introspection closes the loop: for training cases the model still evaluates incorrectly, retrieval-augmented GPT-4 drafts revision suggestions, standard GPT-4 judges them, and a chief physician arbitrates disagreements; the corrected instructions are used to fine-tune the model again, calibrating it to human standards.","core_discovery":"The central claim is that a 13B open-source model can outperform proprietary and general-purpose evaluators at judging medical LLM responses, and that this capability can be obtained with limited high-quality data through the paper's hierarchical training method. AutoMedEval was trained on 9,569 instruction instances constructed by asking GPT-4 to evaluate pairs of responses from ChatDoctor and Baize Healthcare to medical questions, with a dynamic knowledge completion chain that retrieves passages from a medical book and manual vector database whenever the teacher is unsure; two chief physicians then filtered out evaluations failing medical, attribution, or fluency standards. After curriculum instruction tuning on 3,823 ChatGPT-sourced and 4,788 GPT-4-sourced instructions, and two rounds of iterative knowledge introspection with AI-doctor revision suggestions, the model reached Spearman 0.6314, Accuracy2-tuple 74.61%, and Accuracytriple 48.65% on a test set of 958 Wikidoc and 172 MedText questions, all higher than GPT-4 (0.5128, 67.98%, 35.42%) and Gemini (0.5674, 71.68%, 42.63%).","pith_inferences":["Not tested in the paper: if the human-correlation results survive an explicit train/test overlap audit, the pipeline becomes a template for other label-scarce expert domains such as legal advice, radiology, and pharmacovigilance.","Because every training label comes from GPT-4 or ChatGPT, AutoMedEval inherits their blind spots; the paper's own error taxonomy (bias, hallucination, unsupported ratings) suggests physician-written rationales or multi-judge distillation as a next step.","A stress test the paper does not run is out-of-distribution transfer to other specialties or languages; strong transfer would indicate a general medical judging skill rather than memorized Wikidoc response styles."],"forward_implications":["Medical model developers can compare candidate LLMs on open-ended questions without commissioning a full expert panel for every comparison, because AutoMedEval's case-level and pair-level scores track chief physicians' judgments.","Because AutoMedEval is open-source and runs locally, it can be used in privacy-sensitive medical settings where sending patient-style questions to GPT-4 or Gemini is not acceptable.","The curriculum plus introspection recipe can bootstrap a domain evaluator from a small verified seed: AutoMedEval used 5,000 high-quality and 4,000 lower-quality instructions to surpass MedLLaMA tuned on 9,000 high-quality GPT-4 instructions by 38.5 percent relative on Accuracytriple.","Iterative introspection yields steady gains (Accuracytriple 44.61 to 47.13 to 48.65 across rounds), and the paper's saturation model predicts gains stop around six iterations, giving a practical stopping rule for the human-in-the-loop correction step."],"supporting_citations":[{"why":"It supplies the GPT-4 distillation practice used to construct the instruction data and provides the PandaLM baseline AutoMedEval must beat.","marker":"Wang et al., 2023b"},{"why":"MedLLaMA is the 13B base model that AutoMedEval starts from, so the evaluator inherits its medical knowledge.","marker":"Wu et al., 2023"},{"why":"It provides the MedAlpaca model and the medical-meadow-wikidoc corpus that supplies the training and test questions.","marker":"Han et al., 2023"},{"why":"ChatDoctor generates one of the two responses evaluated in each training tuple, making it a core part of the instruction data.","marker":"Li et al., 2023b"},{"why":"Baize Healthcare generates the other response in the training tuples, so the instruction data and the evaluation target depend on it.","marker":"Xu et al., 2023a"},{"why":"It documents GPT-4's performance on medical challenge problems, supporting the paper's argument that proprietary judges are not sufficiently reliable in medicine.","marker":"Nori et al., 2023"},{"why":"G-Eval supplies the prior evidence that LLM judges can align with human judgments, the premise AutoMedEval extends to the medical domain.","marker":"Liu et al., 2023"}],"fun_headline_variants":["13B open judge outranks GPT-4 for medical QA scoring","AutoMedEval: open 13B judges medical answers better than GPT-4","13B open evaluator outperforms GPT-4 on medical judgments","Open 13B judge aligns with doctors, exceeds GPT-4 on QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the unstated assumption that the 958-entry Wikidoc test set and the 9,874-entry Wikidoc training set are disjoint, because if a test question also appeared in training the reported human-correlation advantage could be memorized GPT-4 grading style rather than a general evaluation ability.","fun_headline_variants_meta":{"raw":{"variants":["13B open judge outranks GPT-4 for medical QA scoring","AutoMedEval: open 13B judges medical answers better than GPT-4","13B open evaluator outperforms GPT-4 on medical judgments","Open 13B judge aligns with doctors, exceeds GPT-4 on QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001009,"raw_usage":{"total_tokens":4295,"prompt_tokens":1006,"completion_tokens":3289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3207}},"tokens_in":622,"tokens_out":3289,"duration_ms":19837,"temperature":1.0,"reasoning_tokens":3207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:45:03.352851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the 958-item Wikidoc and 172-item MedText test questions for exact or near-duplicate overlap with the 9,874 training instructions using string matching and embedding similarity; if any test question appears in training, the reported Spearman advantage cannot be trusted. Alternatively, score a newly collected set of clinical QA pairs that was never used in instruction construction and compare AutoMedEval's Spearman correlation with five doctors' judgments against GPT-4's 0.5128; a drop to that level would falsify the generalization claim.","supporting_citations":[],"review_version":1}