{"id":"bceaeab3-66e9-45c3-82af-70ff8d1e056b","arxiv_id":"2506.08349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new three-level medical benchmark shows LLM accuracy falls sharply from factual recall (up to 78%) to full clinical diagnosis (max 19%), with larger models and inference-time scaling helping most at intermediate levels.","lead":"A new benchmark called MultiCogEval measures how well medical AI models handle questions at three cognitive levels: recalling facts, applying knowledge, and solving full diagnostic scenarios. Across 40+ LLMs, performance drops steadily as tasks get harder, with the best model scoring only 19.4 percent on the hardest level.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The full-path metric's assumption that every recorded MIMIC-IV exam is mandatory makes the 19.4% headline and the L2-to-L3 decline artifacts of the gold standard; a necessity-filtered denominator could flip the finding.","rationale":"The paper's main contribution is the multi-cognitive-level evaluation framework and the observation that performance declines as tasks become more complex. The Low-to-Mid decline is relatively secure because Mid-Level tasks are reformulated from the same MedQA/MedMCQA MCQs, share knowledge coverage, and show a large, consistent drop across all model families. The least secure part of the central claim is the Mid-to-High decline, because the full-path metric in Eqs. (2)-(3) assumes the recorded exam set in MIMIC-IV is the gold standard for what a model should order. This assumption is unlikely to hold exactly: inpatient records include routine admission labs, defensive testing, and documentation-driven orders, and they may miss tests that a different but equally competent clinician would have ordered. The paper's own Table 5 shows that end-point diagnosis accuracy is substantially higher than full-path accuracy (e.g., DeepSeek-V3: 53.6% vs. 19.4%), meaning the recall multiplier is solely responsible for the apparent collapse at the High level. If the denominator is over-complete, then the absolute numbers and the size of the decline are inflated. The reader's weakest assumption identifies exactly this issue, and I agree it is the load-bearing concern. A necessity-filtered re-evaluation would directly test whether the headline finding survives a more clinically justified gold standard. Until then, the CONDITIONAL verdict is appropriate: the framework and relative rankings may hold, but the headline 19.4% should not be cited as a calibrated measure of clinical problem-solving ability.","tokens_in":23965,"tokens_out":5334,"duration_ms":57288,"concrete_test":"Have three clinicians independently annotate a random sample of 50 High-Level admission records, classifying each exam item in the record as 'necessary for diagnosis', 'reasonable but optional', or 'not indicated'. Recompute Eq. (2) for DeepSeek-V3, GPT-4o, and Llama3-70B using only 'necessary' items in the denominator. If full-path accuracy rises by more than 10 points (e.g., DeepSeek-V3 from 19.4% toward 30%) or if model rankings change, the 19.4% headline and the L2-to-L3 decline are not robust to the gold-standard assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (3) defines examination recall against the full set of exam items in each MIMIC-IV admission record. This treats every recorded test as a required step of an ideal diagnostic workup. Real EHR records routinely include routine, redundant, or defensive tests, and may omit tests a competent clinician would order. Consequently, a model that reaches the correct diagnosis via a more selective or different workup is penalized: the recall term in Eq. (2) can be arbitrarily low even when the diagnosis is correct. The reported 19.4% full-path accuracy for DeepSeek-V3 is therefore not a clean measure of clinical problem-solving ability; it is an artifact of assuming recorded exams are the unique gold-standard ordering. This concern is load-bearing because the central 'decline' from Mid-Level to High-Level is driven by the multiplicative recall term: Table 5 shows DeepSeek-V3's end-point diagnosis accuracy is 53.6%, which is higher than its Mid-Level score of 42.4% in Table 3. If the recall denominator is over-complete, the headline finding and the absolute 19.4% figure are not robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiCogEval, a three-level evaluation framework for medical LLMs inspired by Bloom's Taxonomy. Level 1 (Preliminary Knowledge Grasp) uses multiple-choice questions from MedQA and MedMCQA; Level 2 (Comprehensive Knowledge Application) reformulates those MCQs into statement validation, multi-step rectification, and answer-existence judgment tasks; Level 3 (Scenario-based Problem Solving) uses a full-path clinical diagnosis task built from MIMIC-IV admission records. The authors evaluate 32 general and 8 medical LLMs from six families and report that normalized accuracy declines monotonically from Level 1 to Level 3, with DeepSeek-V3 achieving 19.4% full-path diagnosis accuracy at the highest level. They also analyze the effects of model size, medical-domain fine-tuning, and inference-time scaling.","tokens_in":24150,"tokens_out":7127,"duration_ms":76528,"significance":"If the measurements are valid, the paper makes a useful contribution. It is the first to compare LLMs across multiple cognitive levels in the medical domain, it releases code and data, and the monotonic decline is observed across all six LLM families, which gives the result a systematic robustness. The knowledge-consistency design between levels is thoughtful, and the fine-grained analyses (per-disease long tail, medical fine-tuning effects, inference-time scaling) are valuable. However, the validity of the High-Level metric is the linchpin of the headline claim, and the current evidence for that metric is weaker than the strength of the claim requires.","major_comments":[{"comment":"The examination recall in Eq. (3) uses the full set of exam items recorded in each MIMIC-IV admission as the denominator, i.e., it treats every recorded test as mandatory for an ideal diagnostic workup. MIMIC-IV records reflect real clinical practice, which includes routine, redundant, or defensive tests, and may omit tests that a competent clinician would order. A model that reaches the correct diagnosis through a more selective or different workup is therefore penalized. This is not a minor scaling issue: Table 5 reports an end-point diagnosis accuracy of 53.6% for DeepSeek-V3, which is well above the Mid-Level score reported in Table 3, so the Level-2-to-Level-3 decline in the headline result is entirely produced by the multiplicative recall term. The authors should recompute the High-Level metric with a necessity-filtered denominator (e.g., a clinician-validated subset of essential tests per case) or provide a sensitivity analysis that shows how the results change when the denominator is restricted or reweighted.","section":"Section 3.3, Eq. (3)"},{"comment":"The clinician validation does not validate the full-path High-Level task as actually administered to the models. According to Section F, clinicians were given only the history of present illness and asked to provide a diagnosis, with the list of allowed test types described but not interactively executed; the model evaluation, by contrast, requires sequential exam ordering, result integration, and a final diagnosis after multiple turns. Moreover, the validation uses only 100 samples (20 per task). This small and simplified validation cannot independently establish that the exam-recall gold standard in Eq. (3) is objective, nor does it speak to the procedural-correctness component that drives the Level-3 scores. The paper should either run the interactive task with clinicians or explicitly state this as a limitation and avoid claims that the validation demonstrates the validity of the proposed benchmark.","section":"Section 4.4 / Section F"},{"comment":"The evaluation of GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1 was performed on approximately 10% of the original dataset, but the paper does not specify how this subset was chosen: whether it is a random sample, whether it is stratified by disease, or whether the same subset is used for all four models. Since the headline 19.4% figure for DeepSeek-V3 is computed on this subset, the authors should document the subset construction, state the exact number of samples, and verify that the reported cross-model comparisons are not skewed by subset composition.","section":"Table 2 footnote / Section D"}],"minor_comments":[{"comment":"The text states that model size plays a more critical role at higher cognitive levels, but Figure 5 and the accompanying discussion indicate that the performance gap between 7B and 70B models narrows again at the High-Level. The wording should be adjusted to be consistent with the data, e.g., 'the gap widens from Low to Mid but narrows again at High.'","section":"Section 4.2 / Figure 5"},{"comment":"For the High-Level task, the random accuracy is set to zero. Since the diagnosis is open-ended but the task is constructed from a limited disease pool of 42 diseases, the assumption of zero random accuracy should be justified or relaxed in a sensitivity analysis.","section":"Section B / Eq. (4)"},{"comment":"The manual verification of GPT-4o-generated statements and distractors is reported as an error rate of less than 5%, but the text does not state whether the verified batch of 100 was used to filter the final dataset or only as a quality check. Please clarify whether any generated items were removed or corrected after verification.","section":"Section A.2"},{"comment":"The prompt says 'You will get all the lab tests results at once' but the later interaction requires the model to list specific tests before receiving results. Please clarify the precise protocol so that readers understand whether the model receives results only for requested items or for an entire category.","section":"Section A.3"},{"comment":"There is a typo in the heading: 'Priciples' should be 'Principles.'","section":"Section 3.1"},{"comment":"The main text says the full-path accuracy is computed as a macro-average across examination types and diseases, but the formula in Eq. (2) does not show this averaging. Please make the metric definition self-contained in the main text.","section":"Section 3.3 / Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid evaluation effort, and the release of code and data is commendable. The central claim, however, depends on the validity of the Level-3 exam-recall denominator. If the authors can rerun the analysis with a necessity-filtered gold standard or a thorough sensitivity analysis, the paper would be substantially stronger. In addition, the 'model size' finding should be phrased more carefully, and the 10%-subset evaluation should be documented precisely. This seems like a fixable major issue rather than a fatal flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper builds a three-tier medical evaluation (factual MCQ, reformulated application tasks, full-path diagnosis) and runs 40 models. The L1→L2 drop is real: reformulating the same MCQs into statement-verification, multi-step rectification, and answer-existence tasks consistently cuts performance across every family. That part holds up. The L3 headline — DeepSeek-V3 at 19.4% full-path accuracy — rests on a metric that treats every exam actually ordered in the MIMIC-IV record as the ground truth. That's the weak joint.\n\nWhat's new: the Mid-Level task types are genuinely new, and the knowledge-coverage alignment across levels is thoughtful. The paper also ships code and data, which matters. The clinician validation is small (100 samples) and for High-Level it reduces the task to a one-shot diagnosis from the history — so it doesn't validate the full interaction.\n\nThe soft spots. Equation (3) defines exam recall with the total recorded exam items in the denominator. MIMIC-IV records include routine and possibly redundant tests; the model is penalized for not ordering everything, even when its diagnosis is correct. Table 5 shows end-point diagnosis at 53.6% for DeepSeek-V3 while full-path is 19.4% — so the L2→L3 decline is largely a product of this recall term. Fixing the denominator (e.g., only tests judged necessary by clinicians) could change the ranking and the absolute numbers. The stress-test's claim that this could flip the finding is overstated if you keep the L1→L2 result separate, but it does undermine the L3 part of the headline. Also, the abstract says model size matters more at higher cognitive levels, but Figure 5 shows the gap narrows at L3; the conclusion softens this but the abstract overstates it. The API models use an undocumented 10% subsample — minor, but should be stated.\n\nWho benefits: model developers and people setting clinical expectations. The benchmark is a real artifact; the L3 metric needs work before its absolute numbers are used. Send it out. A serious referee will ask for a justified gold standard on the exam-recall denominator, a fuller clinician validation, and clearer reporting. Don't desk-reject.","headline":"Useful three-level medical benchmark with a solid L1→L2 drop; the L3 headline number is hostage to a questionable gold standard.","tokens_in":24744,"tokens_out":3620,"would_cite":false,"duration_ms":44246,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current LLMs can recall medical facts, but their ability to plan and reason through a real diagnostic workup drops sharply, with the best model reaching only 19.4% full-path diagnosis accuracy.","keywords":["large language models","medical evaluation","cognitive levels","Bloom's taxonomy","clinical diagnosis","benchmark","MIMIC-IV","scenario-based problem solving"],"falsifier":"Have a panel of experienced clinicians independently order tests for the same patient histories used in the high-level task and compare their orders with the recorded MIMIC-IV orders; if clinicians frequently omit or add different tests, or if model accuracy scored against clinician-approved orders rises far above 19.4%, the paper's central conclusion about scenario-based problem solving would be called into question.","tokens_in":23720,"feed_emoji":"🩺","tokens_out":6580,"duration_ms":72965,"temperature":0.7,"pith_summary":"This paper argues that evaluating LLMs on medical multiple-choice exams alone gives a misleading picture of their clinical ability. It proposes a three-level evaluation framework, inspired by Bloom's Taxonomy: preliminary knowledge grasp (MCQ), comprehensive knowledge application (reformulated tasks such as statement validation and multi-step rectification), and scenario-based problem solving (a full-path diagnosis task in which the model sequentially orders tests and then gives a final diagnosis). Across 32 general and 8 medical models, the paper finds a consistent and large performance decline as the cognitive level rises, with the best model, DeepSeek-V3, reaching only 19.4% full-path diagnosis accuracy. If correct, this means current LLMs have not merely a small gap but a deep one between stored medical facts and the planning and reasoning needed in real clinical settings.","feed_headline":"Best medical LLM hits just 19.4% on full-path diagnosis","feed_subtitle":"Knowledge questions look easy; real diagnosis collapses across all 40 tested models.","key_machinery":"The central object is MultiCogEval, a three-level evaluation framework that keeps medical knowledge coverage roughly constant across levels while varying cognitive demand, and normalizes accuracy by random-guessing baselines so the levels are comparable. At L3 it uses a full-path diagnosis metric that multiplies final-diagnosis correctness by examination recall, where the ground-truth examinations are the ones actually recorded in each MIMIC-IV admission note. This machinery is what turns the abstract claim 'higher cognition is harder for LLMs' into a measurable performance gap.","core_discovery":"The paper claims that current LLMs are strong at recalling medical knowledge but weak at applying it in realistic clinical workflows. Using its MultiCogEval framework, it shows that state-of-the-art models score above 60% normalized accuracy on L1 MCQ tasks, drop roughly 20 points on L2 reformulated application tasks, and fall to about 19.4% on L3 full-path diagnosis, where the model must choose which examinations to order, interpret their results, and produce a final diagnosis. It further reports that larger model size matters more at higher cognitive levels, that medical-domain fine-tuning improves L1 and L2 but not L3, and that inference-time scaling helps across levels, especially on L2. The central discovery is a quantitative, level-by-level map of where LLM medical competence breaks down.","pith_inferences":["Editorial extension: the reported 19.4% depends on treating the tests recorded in historical MIMIC-IV notes as the correct test set; if those notes reflect over-testing or missing tests, the exam-recall term may penalize clinically reasonable strategies and the true problem-solving gap could differ.","Editorial extension: the same three-level design could be applied to other professional domains, such as legal reasoning or engineering troubleshooting, where factual recall and on-the-job planning are similarly separable.","Editorial extension: a direct test of the framework's metric would compare clinician-ordered test sets against the recorded orders; if clinicians themselves disagree substantially with the recorded orders, the high-level metric should be re-anchored.","Editorial extension: the framework measures whether models request the same tests as physicians, not whether they request better, cheaper, or safer tests; a cost- or safety-weighted variant could change the ranking of models at L3."],"forward_implications":["Medical exam-style benchmarks overstate readiness: a model can ace knowledge questions while lacking the planning and information-seeking behavior needed for diagnosis.","Model selection should be task-appropriate: roughly 10B-parameter models may suffice for knowledge QA, while complex clinical tasks require larger models.","Medical-domain fine-tuning should shift its objective from memorized knowledge to high-level abilities such as clinical planning and proactive test ordering.","Inference-time scaling is a promising lever for improving application-level medical reasoning, though it does not by itself close the high-level gap.","Benchmark design for medical AI should include interactive scenario tasks with sequential decisions, not only static QA."],"supporting_citations":[{"why":"Supplies the MedQA MCQs used for Low-Level evaluation and as seeds for the Mid-Level reformulated tasks.","marker":"Jin et al., 2021"},{"why":"Supplies the MedMCQA MCQs for Low- and Mid-Level evaluation.","marker":"Pal et al., 2022"},{"why":"Provides the MIMIC-IV electronic health records from which the High-Level full-path diagnosis task is constructed.","marker":"Johnson et al., 2023"},{"why":"Contributes the full-path clinical diagnosis task design that the High-Level evaluation builds on.","marker":"Hager et al., 2024"},{"why":"Provides the Revised Bloom's Taxonomy that motivates the three cognitive levels.","marker":"Anderson & Krathwohl, 2001"},{"why":"Supplies the annotation tool used to extract a disease pool from previous-level questions so High-Level records align with knowledge coverage.","marker":"Kraljevic et al., 2021"},{"why":"Supplies the terminology mapping used to parse model predictions for examinations and diagnoses.","marker":"Bodenreider, 2004"}],"fun_headline_variants":["Medical LLMs: 60% recall, 19.4% full-path diagnosis","LLMs ace medical facts, flunk clinical diagnosis: 19.4%","Bloom's taxonomy exposes LLM medical reasoning gap: 19.4% on diagnosis","Bigger LLMs help more for complex medical tasks, but top score is 19.4%","LLM medical competence: knowledge is high, diagnosis is 19.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the examinations actually recorded in each historical MIMIC-IV admission are the correct set a model should order, so the exam-recall part of the full-path metric treats deviations from those records as errors; if real notes contain unnecessary, redundant, or missing tests, then the 19.4% figure is not a clean measure of clinical problem-solving.","fun_headline_variants_meta":{"raw":{"variants":["Medical LLMs: 60% recall, 19.4% full-path diagnosis","LLMs ace medical facts, flunk clinical diagnosis: 19.4%","Bloom's taxonomy exposes LLM medical reasoning gap: 19.4% on diagnosis","Bigger LLMs help more for complex medical tasks, but top score is 19.4%","LLM medical competence: knowledge is high, diagnosis is 19.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001331,"raw_usage":{"total_tokens":5374,"prompt_tokens":864,"completion_tokens":4510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":4398}},"tokens_in":480,"tokens_out":4510,"duration_ms":37964,"temperature":1.0,"reasoning_tokens":4398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:32.670492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of experienced clinicians independently order tests for the same patient histories used in the high-level task and compare their orders with the recorded MIMIC-IV orders; if clinicians frequently omit or add different tests, or if model accuracy scored against clinician-approved orders rises far above 19.4%, the paper's central conclusion about scenario-based problem solving would be called into question.","supporting_citations":[{"cited_title":"What disease does this patient have? a large-scale open domain question answering dataset from medical exams","cited_arxiv_id":null,"evidence_quote":"Supplies the MedQA MCQs used for Low-Level evaluation and as seeds for the Mid-Level reformulated tasks."},{"cited_title":"K., and Sankarasubbu, M","cited_arxiv_id":null,"evidence_quote":"Supplies the MedMCQA MCQs for Low- and Mid-Level evaluation."},{"cited_title":"E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T","cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-IV electronic health records from which the High-Level full-path diagnosis task is constructed."},{"cited_title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","cited_arxiv_id":null,"evidence_quote":"Contributes the full-path clinical diagnosis task design that the High-Level evaluation builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Revised Bloom's Taxonomy that motivates the three cognitive levels."},{"cited_title":"A., Roberts, A., et al","cited_arxiv_id":null,"evidence_quote":"Supplies the annotation tool used to extract a disease pool from previous-level questions so High-Level records align with knowledge coverage."},{"cited_title":"The unified medical language system (umls): integrating biomedical terminology","cited_arxiv_id":null,"evidence_quote":"Supplies the terminology mapping used to parse model predictions for examinations and diagnoses."}],"review_version":1}