{"id":"f8c7a4c2-17fe-4a50-b67f-43c87dfa96d7","arxiv_id":"2505.23806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A cloud-local hybrid, where the cloud writes subtask prompts offline and a local model executes them on patient data, reached 70-85% staging accuracy, above local baselines and clinicians.","lead":"MedOrchestra splits medical AI work into two parts: a cloud model builds task instructions without seeing patient data, while a hospital-run local model applies them to patient records. In a test on 100 pancreatic cancer radiology reports, the system staged tumors more accurately than three specialist doctors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline superiority claim depends on ground-truth labels inferred from the withheld Conclusion sections, while the paper's own Limitations section admits 20-30% of free-text report bodies lack the supporting detail needed to recover those labels.","rationale":"The reader's conditional verdict is appropriate. The MedOrchestra architecture itself is plausible and the privacy-preserving design is a real contribution, so rejection is not warranted. However, the headline empirical claim is not yet robust: the Conclusion-derived ground truth, the admitted insufficiency of 20-30% of free-text bodies, the small effective sample sizes, and the absence of statistical inference jointly leave the superiority claim exposed. My concern sharpens the reader's weakest assumption by tying it directly to the paper's own Limitations text and to the concrete denominators (47 and 48 after exclusions). The proposed audit would settle whether the GT is recoverable from the body and whether the reported margins survive on a body-sufficient subset. Until then, the correct disposition is conditional acceptance with the evaluation strengthened, not unconditional acceptance of the superiority claim.","tokens_in":11440,"tokens_out":2782,"duration_ms":33711,"concrete_test":"Have two independent board-certified radiologists, blinded to the original Conclusion sections, (1) judge whether each report body alone contains enough information to assign an NCCN stage, and (2) independently stage the full report plus original images. Recompute Table 1 restricted to cases rated body-sufficient by both readers, using a GT agreed by both full-report readers. If MedOrchestra's free-text accuracy on this subset is within sampling error of the best clinician, the 'clear superiority' claim fails; if the full-report GT disagrees with the Conclusion-derived GT, the original evaluation is uninterpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MedOrchestra achieves 'clear superiority' over clinicians rests on a ground-truth construction that may encode information unavailable to every evaluated system. Section 4.1 states that GT staging labels were inferred by retrospectively mapping the original radiologist Conclusion sections into NCCN stages, while only the report body was provided to the model and the three clinicians. The Limitations section then concedes that approximately 20-30% of free-text reports did not contain sufficient supporting detail outside the conclusion section. If the conclusion contains staging-relevant statements that are not recoverable from the body alone, then the GT labels are not fully determined by the input given to any system; measured accuracy then becomes a partial measure of agreement with hidden conclusion content rather than true staging ability. The comparison is also statistically fragile: after excluding indeterminate cases, free-text accuracy is computed on 47 reports and structured-text accuracy on 48 reports, yet no confidence intervals, significance tests, or effect-size estimates are reported. The 70.21% versus 65.96% margin over the surgeon group could shrink or vanish under a valid GT, a stricter body-sufficient subset, or sampling noise. Additionally, the retrospective mapping from conclusions to NCCN stages is reported without inter-annotator reliability for the GT itself, so the stability of the labels is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedOrchestra is a hybrid cloud-local LLM framework for clinical text interpretation. A cloud LLM (Gemini 2.5 Pro) receives a task definition, clinical guidelines, and user preferences; it decomposes the task into subtasks, generates self-contained prompts, creates synthetic test cases, and refines prompts until they pass an 80% validation threshold, all without access to patient data. A local LLM (Gemma 3 27B) then executes the prompts on-site on real reports, and the system applies cloud-generated rule-based logic, with five repeated inferences and majority voting, to produce a final outcome. The authors evaluate the system on NCCN-based pancreatic cancer staging using 100 radiology reports (50 free-text, 50 structured) from a single Korean institution. They report that MedOrchestra achieves 70.21% and 85.42% accuracy on free-text and structured reports, respectively, outperforming local LLM baselines and three board-certified clinicians, with higher Cohen's kappa. The paper also includes a candid Limitations section.","tokens_in":11700,"tokens_out":5909,"duration_ms":60825,"significance":"The architecture is a sensible and potentially valuable contribution: it decouples high-level reasoning from PHI-bearing inference, so no protected data leave the hospital, and it uses synthetic validation to avoid fitting prompts to the test labels. If the evaluation were sound, the result would be practically relevant for privacy-preserving clinical NLP. The authors deserve credit for stating limitations plainly, including the ground-truth ambiguity and the 20–30% of free-text bodies lacking sufficient detail. However, the empirical evidence for the headline 'clear superiority' claim is currently insufficient: the ground-truth labels may encode information from withheld Conclusion sections, the sample sizes are small, and no uncertainty or significance analysis is provided.","major_comments":[{"comment":"Ground-truth labels were inferred by retrospectively mapping the original radiologist Conclusion sections into NCCN stages, while all evaluated systems received only the report body. The Limitations section explicitly admits that in 20–30% of free-text reports the body lacks sufficient supporting detail and the conclusion was necessary to determine the GT. This means the GT is not guaranteed to be recoverable from the input given to any model or clinician; accuracy then partly measures agreement with hidden conclusion content rather than staging ability from the body alone. This is load-bearing for the comparison against clinicians. Please report the number and proportion of the 47 evaluated free-text cases in which the body alone determines the stage, present accuracy on the body-sufficient subset for all systems, and either re-derive GT from bodies alone or justify that conclusions add no information beyond the body.","section":"Section 4.1 and Limitations"},{"comment":"All comparisons are based on 47 free-text and 48 structured reports after excluding indeterminate cases, yet no confidence intervals, significance tests, or effect sizes are reported. The free-text margin between MedOrchestra (70.21%, 33/47) and the surgeon group (65.96%, 31/47) is two cases; a paired analysis could easily change the ranking. Cohen's kappa values are also reported without uncertainty. Please provide exact counts, per-system confidence intervals, and paired significance tests (e.g., McNemar) on the same cases.","section":"Section 5, Tables 1–2"},{"comment":"The evaluation uses majority voting over five runs with the tie broken by selecting the higher clinical stage, described as conservative. Since the manuscript itself reports that MedOrchestra's main error mode is overstaging ambiguous language, this tie-breaking rule may systematically bias accuracy in an unknown direction. Please report the sensitivity of the headline accuracy to the tie-breaking rule and to the number of repeated inferences T, especially since the Limitations section states that the choice of repetitions was not systematically validated.","section":"Section 4.3 and Section 5.3"},{"comment":"The GT mapping procedure is under-specified: no number of annotators, inter-annotator reliability, or adjudication rule is given, and the treatment of indeterminate GT labels is unclear (whether the indeterminate flag was applied to GT labels or only to clinician annotations). Without this, label noise in the GT itself is unquantified and could affect all comparisons.","section":"Section 4.1"}],"minor_comments":[{"comment":"There is a typo: 'polices' should be 'policies', and the affiliation line contains 'Univeristy' instead of 'University'.","section":"Section 1"},{"comment":"Please report the denominators (N=47 and N=48) alongside the percentages so readers can reconstruct exact counts.","section":"Table 1"},{"comment":"The label 'MedEx' is inconsistent with the name 'MedOrchestra' used in the text; please unify, and provide counts for all cells of the 2x2 grid in both panels.","section":"Figure 3"},{"comment":"Algorithm 1 refers to 'TestSet_i' but the text defines synthetic examples as X_syn; please define this notation explicitly.","section":"Algorithm 1"},{"comment":"Several reference entries contain 'and 1 others' instead of complete author lists or consistent 'et al.' usage; please fix.","section":"References"},{"comment":"The phrase 'showing clear superiority across all settings' is stronger than the current statistical evidence supports; please soften it or qualify it with the study's limitations.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The ground-truth leakage issue is the main substantive concern, and the authors' Limitations section actually confirms the risk. I would ask the editor to require the body-sufficiency analysis and statistical intervals before considering publication. The architecture itself is worth publishing if the evidence is reworked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: MedOrchestra is a sensible hybrid design—cloud LLM generates and validates subtask prompts on synthetic cases, local LLM executes on real reports—and the paper includes a rare head-to-head against board-certified clinicians. The empirical claim that it beats those clinicians is not yet solid, and the paper's own limitations section points at the main reason.\n\nWhat's new and good: the separation is clean, the prompt validation loop without touching clinical data is a practical contribution, and the rule-based synthesis keeps the final decision interpretable. The evaluation is more serious than most in this space: three specialist groups, free-text and structured reports, confusion matrices, disagreement analysis. The paper cites the relevant orchestration and clinical NLP literature. The transparency about limitations is genuine; the authors flag the 20-30% of free-text bodies lacking sufficient detail, the lack of multi-round validation, and the manual prompt transfer burden.\n\nThe soft spots are evaluation-side. GT staging labels were constructed by retrospectively mapping the original radiologist Conclusion sections, which were withheld from both the model and the clinicians. If the body does not contain the staging-relevant details for a fifth to a third of the free-text reports, then those GT labels are not recoverable from the input any system saw. That makes the measured accuracies a partial measure of agreement with hidden conclusion content, and it could specifically inflate MedOrchestra's margin over clinicians if the rule-based logic happens to align with how the GT was derived. Second, the comparison rests on 47-48 cases with no confidence intervals, no significance tests, and no inter-annotator reliability for the GT mapping. The 4-point margin over the surgeon group is roughly two cases; that is fragile. There is also some by-construction agreement since the outcome logic encodes the same NCCN guideline used to build GT, though this is not a fitting-to-labels circularity—no clinical data was used in prompt development.\n\nWho is this for? Researchers building privacy-preserving clinical NLP pipelines and anyone thinking about how to evaluate guideline-based tasks. The architecture is reusable and the limitation analysis is honest. It deserves a serious referee; the main revision should be a statistically grounded evaluation and a stricter subset analysis on reports where the body is self-sufficient.\n\nRecommendation: send to peer review, expect major revision on the evaluation.","headline":"Sensible hybrid architecture and a rare clinician benchmark, but the superiority claim needs a stronger evaluation before it lands.","tokens_in":12239,"tokens_out":2405,"would_cite":true,"duration_ms":24490,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedOrchestra shows that a cloud-generated decomposition lets a local language model stage pancreatic cancer above specialist level without seeing patient data.","keywords":["hybrid cloud-local LLM","privacy-preserving clinical NLP","pancreatic cancer staging","radiology report interpretation","task decomposition","synthetic prompt validation","NCCN guidelines"],"falsifier":"Recompute accuracy on the subset of reports whose bodies contain enough staging detail, using ground truth from multidisciplinary consensus rather than the withheld conclusions; if the 20-30% of free-text reports lacking body detail are where MedOrchestra beat the clinicians, the claimed superiority is an artifact of the evaluation design.","tokens_in":11260,"feed_emoji":"🩺","tokens_out":8052,"duration_ms":75202,"temperature":0.7,"pith_summary":"This paper claims that a small local language model can stage pancreatic cancer from radiology reports at the level of board-certified specialists, provided a cloud model first decomposes the task into subtask prompts and a rule-based synthesis scheme without ever seeing patient data. On 100 reports, MedOrchestra reaches 70.21% accuracy on free-text reports and 85.42% on structured reports, beating local-model baselines and three groups of specialists. The significance is practical: clinical AI could gain cloud-level reasoning while keeping sensitive records inside the hospital network.","feed_headline":"Hybrid cloud-local LLM stages pancreatic cancer above specialists","feed_subtitle":"A cloud planner prompts a local model to hit 70% free-text and 85% structured accuracy with patient data kept on-site.","key_machinery":"The key machinery is an orchestrator-executor split with a validation loop. The cloud orchestrator receives only the task description, the clinical guideline, and user preferences; it outputs subtasks, self-contained prompts, synthetic test cases, and rule-based outcome logic. The local executor runs those prompts on real reports and returns subtask outputs, which the rule-based logic combines into a final stage, with majority voting over repeated runs protecting against stochastic errors.","core_discovery":"The paper's central claim is that the limiting factor for local language models in clinical interpretation is not raw capability but task structure. Once a cloud LLM decomposes staging into concrete subtasks, drafts self-contained prompts, validates them on synthetic cases, and fixes rule-based aggregation logic, the local model can execute the reasoning chain and outperform both unaugmented local models and clinicians. On this 100-report pancreatic cancer dataset, that division of labor yields the highest accuracy and the highest agreement with ground-truth staging in both free-text and structured formats.","pith_inferences":["The ground-truth labels are reconstructed from report conclusions, so a sturdier test would use imaging-based or multidisciplinary consensus staging; no such test appears in the paper, and the measured gap over clinicians could shrink. ","The framework implicitly assumes that a cloud model's decomposition and synthesis logic transfer across institutions, so a multi-institution replication would determine whether prompts need site-specific regeneration.","A natural extension is to ablate the validation loop and measure how much of the gain comes from task decomposition rather than from the choice of the cloud model; the paper does not isolate this.","The manual prompt-transfer step, listed as a limitation, could be turned into a versioned and auditable prompt-as-logic artifact, making the system's reasoning traceable in clinical audits."],"forward_implications":["Cloud-generated subtask prompts can be reused for other guideline-based clinical tasks because they are derived from guidelines and synthetic cases rather than patient data.","Institutions with strict data-export rules can still obtain cloud-level reasoning by transferring prompts once, without moving any clinical text off-site.","The large gain from free-text to structured input suggests that standardizing report format may improve machine staging as much as choosing a stronger model.","Handling speculative language such as 'likely' or 'suspicious for' remains the main barrier, since the paper finds it causes overstaging in free-text reports."],"supporting_citations":[{"why":"Demonstrates conversion of mixed-language pancreatic cancer CT reports into NCCN structured templates, the same guideline and disease focus as this evaluation.","marker":"Kim et al., 2025"},{"why":"Introduces decomposed prompting, the modular strategy MedOrchestra adapts by splitting staging into independently promptable subtasks.","marker":"Khot et al., 2023"},{"why":"Shows local LLMs can perform privacy-preserving clinical report review, the deployment premise that motivates keeping inference on-site.","marker":"Vaid et al., 2024"},{"why":"Establishes that LLMs can extract guideline-relevant features from multilingual free-text radiology reports, the same kind of extraction MedOrchestra delegates to its subtask prompts.","marker":"Gu et al., 2024"},{"why":"Provides an earlier conventional-NLP staging pipeline for free-text radiology reports, the baseline the paper's local LLM comparisons extend.","marker":"Nobel et al., 2024"},{"why":"Documents the security and workflow constraints on clinical LLM deployment that the hybrid design is intended to satisfy.","marker":"Dennstädt et al., 2025"}],"fun_headline_variants":["Cloud divides, local conquers in pancreatic staging","Hybrid LLM beats specialists in cancer staging","Cloud planner, local executor: 70% staging accuracy","Local LLM excels when cloud plans the subtasks","Cloud-local split lifts cancer staging above clinicians"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the correct stage for each report, which was reconstructed from the report's conclusion, can actually be determined from the report body alone; the paper admits 20-30% of free-text reports do not contain enough detail outside the conclusion, so if a label is not supported by the body, every system and expert is being measured against information they never saw.","fun_headline_variants_meta":{"raw":{"variants":["Cloud divides, local conquers in pancreatic staging","Hybrid LLM beats specialists in cancer staging","Cloud planner, local executor: 70% staging accuracy","Local LLM excels when cloud plans the subtasks","Cloud-local split lifts cancer staging above clinicians"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1252,"prompt_tokens":882,"completion_tokens":370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":297}},"tokens_in":498,"tokens_out":370,"duration_ms":3932,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:47:35.749997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute accuracy on the subset of reports whose bodies contain enough staging detail, using ground truth from multidisciplinary consensus rather than the withheld conclusions; if the 20-30% of free-text reports lacking body detail are where MedOrchestra beat the clinicians, the claimed superiority is an artifact of the evaluation design.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates conversion of mixed-language pancreatic cancer CT reports into NCCN structured templates, the same guideline and disease focus as this evaluation."},{"cited_title":"Duong, Jeremy Lampert, Paul Kovatch, Robert Freeman, Edgar Argulian, Lincoln Croft, Sotirios Lerakis, Matthew Goldman, Rohan Khera, and Girish N","cited_arxiv_id":null,"evidence_quote":"Shows local LLMs can perform privacy-preserving clinical report review, the deployment premise that motivates keeping inference on-site."}],"review_version":1}