{"id":"3e9d6350-831c-431d-a129-9d30a179bf11","arxiv_id":"2506.00235","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MedOrch is a modular framework in which LLMs call medical tools to answer clinical questions; its headline results on Alzheimer's, chest X-ray, and VQA benchmarks are weakened by best-of-five scoring.","lead":"MedOrch is a framework that lets a large language model call medical tools such as database queries, image analyzers, and web searches while solving clinical questions. The paper claims state-of-the-art results on Alzheimer's, chest X-ray, and medical VQA benchmarks, but the headline numbers rely on a ground-truth-guided best-of-five-answers evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best@5 oracle scoring, not deployable accuracy, produces the reported 93.26% AD diagnosis; under best@1 MedOrch is below the baseline, so the headline superiority claim is unsupported.","rationale":"The strongest claim is the AD diagnosis result, and it rests entirely on the best@5 oracle. This is the single most load-bearing concern because the abstract and introduction advertise a state-of-the-art advantage that disappears under a deployable metric. The paper itself reports the needed counter-evidence: best@1 accuracy is 88.71%, below the 89.05% baseline; majority@5 is 89.93%, a 0.88-point gain rather than 'over four percentage points.' The AD prediction claim (50.35% best@5) is less vulnerable because best@1 already beats the baseline by ~7.6 points, but the diagnosis claim is the one highlighted in the abstract. The lack of code and confidence intervals compounds the issue but does not replace it. A single re-scoring of the existing trajectories with majority voting would settle whether the reported gain is real. The verdict should remain REJECT: the empirical support for the central claim is inadequate.","tokens_in":18156,"tokens_out":5484,"duration_ms":49446,"concrete_test":"Re-score the 638 ADNI three-class test cases used in Table 1 with the already-collected five MedOrch(o1-mini) trajectories per case, using majority voting instead of best-of-five selection. Compare the resulting accuracy to the 89.05% ConvNext+GPT baseline and to the reported 93.26% best@5. If the majority-vote accuracy is at or below the baseline (as Table 1's majority@5 row suggests), the headline superiority claim is an artifact of the oracle scoring rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 93.26% AD diagnosis accuracy that 'surpasses the state-of-the-art baseline by over four percentage points' (Abstract; Table 1). This number comes from the best@5 protocol defined in §4.2.3: five reasoning trajectories are generated and a case is counted correct if any one of them is correct. That is an oracle selection rule: it assumes someone knows the ground truth when choosing a trajectory, which is not possible in deployment. The paper's own Table 1 shows the fragility of the claim. On AD vs MCI vs NC, ConvNext+GPT achieves 89.05%; MedOrch(o1-mini) achieves 88.71% at best@1 and 89.93% at majority@5. Only the best@5 value (93.26%) gives the claimed >4-point gain. The system's trajectories can disagree (e.g., Figure 2 labels the same patient NC in one trajectory and MCI in another), so best@5 cannot be interpreted as expected diagnostic accuracy. The abstract's headline is therefore not supported by a deployable evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedOrch is a modular, tool-augmented multi-agent framework for medical decision support. It uses a reasoning LLM (GPT-4o or o1-mini) that can invoke general-purpose agents (web search, coding, Text2SQL, RAG) and domain-specific agents (medical image analysis, medical imaging QA, clinical knowledge graph, longitudinal data analysis) through special tool-call tokens, without fine-tuning the core reasoning model. The paper evaluates MedOrch on three tasks: Alzheimer's disease diagnosis and progression prediction on ADNI, chest X-ray classification on MIMIC-CXR, and multimodal visual question answering on EHRXQA. The central empirical claim is that MedOrch achieves 93.26% accuracy for AD diagnosis, surpassing the state-of-the-art baseline by over four percentage points, with smaller but still substantial gains on progression prediction.","tokens_in":18330,"tokens_out":6028,"duration_ms":58110,"significance":"If the reported results were valid, MedOrch would demonstrate that a single orchestration framework can integrate heterogeneous medical tools and multiple reasoning trajectories to match or beat task-specific models without retraining. The architecture is genuinely modular, and the transparency of intermediate tool calls and reasoning traces is a useful contribution. The evaluation, however, rests on an oracle-based best@5 protocol that selects the correct answer among five generated trajectories, which is not a deployable accuracy measure and is not comparable to single-run baselines. The paper also leaves open whether diagnostic labels are accessible to the retrieval agents and whether the fine-tuned image-analysis model is evaluated on patients disjoint from its training set. These issues undermine the headline superiority claims and the validity of the ADNI experiments.","major_comments":[{"comment":"The best@5 protocol defined in §4.2.3 ('the model generates five different reasoning paths, and if any of them produce the correct diagnosis, it is counted as correct') is an oracle selection rule: it assumes access to ground-truth labels when picking among the five trajectories. This is not a deployable accuracy measure, and it is not apples-to-apples with the single-run baseline numbers in Tables 1 and 2. On the three-class AD task, MedOrch(o1-mini) under best@1 is 88.71%, below ConvNext+GPT's 89.05%, and only the best@5 value (93.26%) yields the >4-point margin claimed in the abstract; even majority@5 (89.93%) is only slightly above the baseline. The same best@5 protocol is used for the 50.35% progression number (Table 2) and for EHRXQA (Table 4), so those numbers are also upper bounds rather than expected accuracies. The headline superiority claim is therefore unsupported by a valid evaluation.","section":"§4.2.3, Table 1"},{"comment":"The ADNI experimental design does not report whether the ground-truth diagnosis is accessible to the Text2SQL/retrieval agents. The task in §4.2.1 provides a patient ID and partial information and instructs the system to 'determine what additional information is required' and retrieve it. If the ADNI diagnostic label (e.g., the DX variable) is among the fields the Text2SQL agent can query, the task reduces to a database lookup, and the reported accuracies reflect information retrieval rather than reasoning. The manuscript must state explicitly that diagnostic labels and any direct label-derived features were excluded from the schema exposed to the agents; otherwise the AD diagnosis and progression results are not interpretable as evidence for the framework's reasoning capabilities.","section":"§3.3, §4.2.1"},{"comment":"Appendix A.2 states that a 3D ResNet-50 was fine-tuned on ADNI baseline T1 scans with an 80:15:5 patient-level split, but the manuscript does not specify whether the 638 diagnostic QA examples are disjoint from the fine-tuning training split. If the same patients appear in both, the Medical Image Analysis Agent could have memorized labels, inflating MedOrch's performance. The paper must report the relationship between the fine-tuning split and the evaluation set, and ideally re-evaluate with a held-out cohort that is completely disjoint from all trained components.","section":"Appendix A.2, §4.2.1"}],"minor_comments":[{"comment":"Several entries in Table 1 are missing separators between numbers (e.g., '87.5095.6063.33' and '89.0597.3389.29'), making the table hard to read; add column separators or spacing.","section":"Table 1"},{"comment":"The caption contains an incomplete sentence: 'MedOrch (o1-mini) with achieves a best@1 accuracy of 38.61%' should read 'with best@1', and the sentence should be revised for grammar.","section":"Table 2 caption"},{"comment":"The text says MedOrch 'does not assume access to such structured supervision' and 'operates in a fully zero-shot or few-shot fashion', but Appendix A.1 states that the Text2SQL agent is 'provided with comprehensive schema information'. The relationship between the available schema and the claimed lack of supervision should be clarified.","section":"§4.4.2 vs Appendix A.1"},{"comment":"The abstract reports the 50.35% progression accuracy without noting that this is a best@5 oracle number; the protocol should be disclosed in the abstract or the claim should be restricted to majority@5/best@1, which are substantially lower (39.87% and 38.61%, respectively).","section":"Abstract and Table 2"},{"comment":"The EHRXQA evaluation uses a random subset of 300 instances with no reported seed or confidence intervals, and on the Image+Table subset MedOrch(o1-mini, best@5) at 54.47% is below the ChatGPT+M3AE (BM25) baseline at 60.98%; the text's characterization of these results as 'strong' and 'competitive' should be tempered given the margin and the lack of variance estimates.","section":"§4.4 and Table 4"}],"recommendation":"reject","confidential_remarks":"The modular agent framework and the emphasis on transparent, multi-trajectory reasoning are potentially interesting, but the empirical evaluation as presented does not support the headline claims. The best@5 oracle protocol, the unaddressed possibility of label access through the retrieval agents, and the unspecified overlap between the fine-tuned image model's training split and the QA evaluation set are fundamental validity concerns. If the authors can re-run the ADNI experiments with a proper single-run or majority protocol, exclude diagnostic labels from the agent-accessible schema, and ensure patient-level disjointness of all trained components, a substantially revised version could merit reconsideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the headline number – 93.26% AD diagnosis, 'surpassing SOTA by over four points' – is produced by a best@5 oracle protocol, not a deployable accuracy. The paper defines best@5 as counting a case correct if any of five independently generated trajectories gets it right (Section 4.2.3). That requires ground truth to pick the trajectory, which you don't have at inference. Under best@1, MedOrch with o1-mini gets 88.71%, which is below the ConvNext+GPT baseline at 89.05%. Majority@5 gives 89.93%, a modest gain. So the claimed >4 point advantage is an artifact of the evaluation rule. Second, the paper itself shows, in Figure 2, the same patient being labeled NC in one trajectory and MCI in another. That's actually a good illustration of why best@5 is not diagnostic accuracy – it's an upper bound on how often at least one trajectory is right.\n\nNow, the work itself is a reasonable engineering contribution. The idea of a modular medical agent with a tool registry, transparent reasoning traces, and the ability to swap in task-specific agents (FreeSurfer, a fine-tuned ResNet, M3AE+GPT-4V) is well executed. The paper is clearly written, and the appendix gives useful implementation details, including the manual template engineering that the EHRXQA baselines require – which is a fair point in favor of the agentic approach even where the numbers are lower.\n\nThe soft spots, in order: (1) The best@5 protocol is the load-bearing evaluation for the main claims, and it's invalid for comparing deployed accuracy. This should be re-framed as a 'clinician-in-the-loop' or upper-bound analysis, not a head-to-head accuracy comparison. (2) On EHRXQA, Table 4 shows MedOrch below the BM25 baseline on every subset, yet the abstract says 'competitive' – that's technically true but the framing is generous. (3) No code, prompts, or data splits are released; the ADNI split is only described as 80:15:5 patient-level in the appendix, with no mention of how standard that split is. (4) The paper never explicitly lists best@5 as a limitation, despite the earlier discussion of multiple diagnostic perspectives.\n\nI think the stress-test note is right on target. This is an honest paper about a useful framework, but the central empirical claims as stated are not supported. Who is it for? Researchers working on LLM tool-use and medical reasoning will find the architecture and evaluation discussion valuable, particularly as a cautionary example of oracle scoring. It deserves a serious referee, because the framework is real and the flaws are fixable with a revised evaluation and honest framing. As it stands, the paper should not be accepted with these numbers, but it's not a desk-reject either.","headline":"Best@5 oracle scoring inflates MedOrch's headline AD accuracy; the framework itself is a solid modular tool-use system but the main empirical claims should be re-framed.","tokens_in":18917,"tokens_out":4252,"would_cite":false,"duration_ms":36302,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedOrch, a single modular agent framework, claims 93.26% accuracy on Alzheimer's diagnosis and 50.35% on progression prediction by letting one language model call medical tools in context, with no retraining.","keywords":["medical AI agents","tool-augmented reasoning","large language models","Alzheimer's disease diagnosis","multimodal medical data","in-context tool calling","medical visual question answering","chest X-ray classification"],"falsifier":"Re-run the 638-question ADNI diagnosis evaluation using an answer the system would actually commit to — the first trajectory's answer, a majority vote, or a random draw from the five — instead of best@5, and compare with the reported 93.26%. If committed accuracy collapses toward the 88.71% best@1 level, the headline gain is an artifact of selecting trajectories with knowledge of the ground truth; if majority@5 or another commitment rule preserves most of the gain, the claim that diverse reasoning paths improve real diagnosis survives.","tokens_in":17910,"feed_emoji":"🩺","tokens_out":13936,"duration_ms":116820,"temperature":0.7,"pith_summary":"MedOrch is a medical decision-support framework in which one large language model does the reasoning and decides, mid-answer, which specialized tools to call: database queries over patient records, web searches of clinical guidelines, code execution, brain MRI analysis, and medical question answering over images. The paper's central claim is that this single modular architecture, with no fine-tuning and no task-specific training, matches or beats specialist systems on three distinct applications: Alzheimer's disease diagnosis (93.26% accuracy, more than four points above the best baseline), Alzheimer's progression prediction (50.35% accuracy), chest X-ray classification, and multimodal visual question answering (54.47% on the hardest image-plus-table questions). The headline accuracy numbers come from a best@5 protocol, in which a case counts as correct if any of five independently generated reasoning paths reaches the right answer; the paper reports the protocol openly and also shows single-trajectory numbers. The claim matters because, if true, one adaptable, fully auditable system could replace a collection of rigid black-box models and give clinicians a verifiable reasoning trail for every recommendation.","feed_headline":"Tool-calling agents hit 93% on Alzheimer's diagnosis","feed_subtitle":"The same auditable pipeline reads X-rays, queries patient records, and diagnoses Alzheimer's — without retraining.","key_machinery":"The load-bearing mechanism is the in-context tool-calling protocol. A tool registry, containing semantic descriptions, input/output specifications, and usage examples for every available agent, is injected into the reasoning model's system prompt; during inference the model generates special tool-invocation tokens such as [IMAGE_QUERY], [SQL_QUERY], [WEB_QUERY], and [CODE_QUERY] at exactly the points where it judges that external evidence is needed. Each token triggers a three-step cycle — query routing to the appropriate agent, tool execution, and integration of the returned output back into the reasoning chain — and the loop continues until the model emits a conclusion without further tokens. A second mechanism, multi-trajectory generation, carries the accuracy claims: the same case is solved five times with varied evidence priorities and tool orderings, and the paper reports results under best@1, majority@5, and best@5 scoring, attributing the gain from 88.71% (best@1) to 93.26% (best@5) on three-class Alzheimer's diagnosis to the diversity of these reasoning paths.","core_discovery":"The paper's central discovery is that in-context tool calling, initiated by the reasoning model itself, can turn a general-purpose LLM into a competitive multi-specialty diagnostician without retraining. MedOrch injects a tool registry into the model's context, and during its chain of thought the model emits special invocation tokens such as [IMAGE_QUERY], [SQL_QUERY], and [WEB_QUERY]; the system routes each token to the matching agent, executes the tool, and feeds the result back into the ongoing reasoning until the model concludes. On the ADNI dataset this yields 93.26% accuracy for three-class Alzheimer's diagnosis and 89.50% for the finer four-class task under best@5, 4.21 points above the previous best baseline on the three-class task, and 50.35% accuracy on disease-progression prediction (38.61% with a single trajectory, still above the best baseline). With only the image agent swapped, the same core reaches a macro AUC of 61.2% and macro F1 of 25.5% on MIMIC-CXR, and on EHRXQA it reaches 54.47% on image-plus-table questions without the hand-built SQL templates the baselines depend on. The paper presents these results as evidence that reasoning-driven tool orchestration generalizes across modalities and clinical domains without architectural change.","pith_inferences":["My reading: because best@5 credits a case whenever any one of five paths is right, the reported 93.26% is an upper bound over an invisible committee; a deployed system that must commit to a single answer would land between the 88.71% best@1 and the best@5 figure, and the paper does not report that committed accuracy.","A testable extension: scaling the committee to ten or twenty trajectories on the same 638-question ADNI set would show whether the trajectory-diversity gain saturates, telling clinicians how many parallel analyses are actually worth running.","The framework's generality claim could be probed by porting the same registry-and-reasoning recipe to other multimodal medical benchmarks, such as pathology slides paired with reports or ECG-plus-chart data, where the paper's three tasks leave the space of clinical modalities largely unexplored."],"forward_implications":["If one reasoning core with interchangeable tools covers Alzheimer's assessment, chest X-ray reading, and EHR question answering, then adding new clinical capabilities becomes a matter of registering a new tool, not retraining a model.","The documented rise from 88.71% (single trajectory) to 93.26% (best@5) on AD diagnosis implies that generating several transparent reasoning paths materially improves diagnostic accuracy, which the paper ties to surfacing differential diagnoses for clinicians.","Because every tool call and returned result is logged, the framework provides a full audit trail of each recommendation, which the paper argues addresses the clinical and regulatory demand for explainable medical AI.","On EHRXQA, MedOrch reaches competitive accuracy with no hand-written question-to-SQL templates, suggesting that agentic zero-shot tool use can substitute for months of manual benchmark engineering."],"supporting_citations":[{"why":"Supplies the ADNI longitudinal dataset of demographics, cognitive scores, biomarkers, and MRIs on which the Alzheimer's diagnosis and progression experiments are built.","marker":"[11]"},{"why":"Provides the ConvNext+GPT baseline that MedOrch must beat; the four-point accuracy gain on three-class AD diagnosis is measured against it.","marker":"[43]"},{"why":"M3AE acts both as a baseline on the AD tasks and as the medical image-understanding backend inside MedOrch's image QA agent, so its numbers anchor several comparisons.","marker":"[42]"},{"why":"Supplies the MIMIC-CXR dataset and its 14 thoracic-pathology labels for the chest X-ray classification claim.","marker":"[12]"},{"why":"Supplies the EHRXQA benchmark, the ChatGPT+M3AE baselines, and the hand-built SQL template pipeline that MedOrch aims to replace with zero-shot tool use.","marker":"[13]"},{"why":"FreeSurfer, invoked by the Medical Image Analysis Agent to segment brain MRIs and produce volumetric features used in AD diagnosis.","marker":"[35]"},{"why":"LLM-CXR is the strongest chest X-ray baseline on micro-level metrics; the paper's macro-level advantages and micro-level trade-off are defined against it.","marker":"[48]"}],"fun_headline_variants":["Tool-calling agents hit 93% on Alzheimer's, 61% AUC on X-rays","MedOrch: one agent framework, many tools, top scores on three tasks","Agents that call tools outperform baselines on Alzheimer's and X-rays","MedOrch: tool-augmented reasoning hits 93% on Alzheimer's","Tool-orchestrated LLMs achieve competitive results across medical tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best@5 evaluation is a legitimate measure of performance: the paper counts a case as solved if any one of five generated reasoning paths reaches the correct diagnosis, which presumes the evaluator already knows the right answer when choosing which path to credit; a deployed system gets no such oracle, so the headline accuracy is an optimistic upper bound rather than expected performance.","fun_headline_variants_meta":{"raw":{"variants":["Tool-calling agents hit 93% on Alzheimer's, 61% AUC on X-rays","MedOrch: one agent framework, many tools, top scores on three tasks","Agents that call tools outperform baselines on Alzheimer's and X-rays","MedOrch: tool-augmented reasoning hits 93% on Alzheimer's","Tool-orchestrated LLMs achieve competitive results across medical tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001351,"raw_usage":{"total_tokens":5578,"prompt_tokens":1132,"completion_tokens":4446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":748,"completion_tokens_details":{"reasoning_tokens":4343}},"tokens_in":748,"tokens_out":4446,"duration_ms":30337,"temperature":1.0,"reasoning_tokens":4343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:09:24.904794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 638-question ADNI diagnosis evaluation using an answer the system would actually commit to — the first trajectory's answer, a majority vote, or a random draw from the five — instead of best@5, and compare with the reported 93.26%. If committed accuracy collapses toward the 88.71% best@1 level, the headline gain is an artifact of selecting trajectories with knowledge of the ground truth; if majority@5 or another commitment rule preserves most of the gain, the claim that diverse reasoning paths improve real diagnosis survives.","supporting_citations":[{"cited_title":"Large language models improve alzheimer’s disease diagnosis using multi-modality data","cited_arxiv_id":null,"evidence_quote":"Provides the ConvNext+GPT baseline that MedOrch must beat; the four-point accuracy gain on three-class AD diagnosis is measured against it."},{"cited_title":"Multi-modal masked autoencoders for medical vision-and-language pre-training","cited_arxiv_id":null,"evidence_quote":"M3AE acts both as a baseline on the AD tasks and as the medical image-understanding backend inside MedOrch's image QA agent, so its numbers anchor several comparisons."},{"cited_title":"Freesurfer.Neuroimage, 62(2):774–781, 2012","cited_arxiv_id":null,"evidence_quote":"FreeSurfer, invoked by the Medical Image Analysis Agent to segment brain MRIs and produce volumetric features used in AD diagnosis."}],"review_version":1}