{"id":"335c01a9-3c47-4a82-adf7-15f871101b5d","arxiv_id":"2607.04694","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On MDS-Bench (1,939 raw medical standardization tasks), even Gemini 3 Flash reaches only 48.6% strict end-to-end success at producing source-grounded image-JSON pairs.","lead":"Frontier vision-language models still fail more than half the time when asked to turn messy raw medical dataset folders into clean image-plus-JSON records. The paper builds a 1,939-task benchmark showing that this upstream standardization step—not diagnosis—is a major bottleneck for real-world medical AI.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"E2E bottleneck claim rests on hand-chosen gates (g≥0.85, q≥0.5, f≥0.5) whose sensitivity is unreported; the 48.6% figure may be threshold-driven rather than a robust pipeline failure.","rationale":"The paper cleanly demonstrates a real upstream gap: high SV/SSC versus low SCJ/E2E across nine models, three task types, and multiple modalities (Tables 1–2, Figs. 5–6). That qualitative structure-vs-joint failure is multiply evidenced and does not depend on any single threshold. However, the strongest claim is quantitative (“only 48.6%”). Because E2E is a hard conjunction of three free cutoffs whose sensitivity is never shown, the absolute number that carries the “critical bottleneck” rhetoric is under-supported. The Reader correctly flags public-vs-clinical shift and residual annotation risk (Limitations), but those are external validity issues; the more immediate load-bearing concern is internal: the reported E2E may be an artifact of the chosen gates. A threshold-sensitivity table would settle it. Until then the contribution remains accept-shaped only with the same conditions the Reader already listed (artifact release + threshold reporting + clearer clinical-bound language). Verdict therefore stays CONDITIONAL; no upgrade or downgrade is warranted, only a sharper statement of what must still be shown.","tokens_in":26229,"tokens_out":768,"duration_ms":7320,"concrete_test":"Recompute Table 1 E2E (and SCJ) for Gemini 3 Flash and the next two models under a 3×3 grid of thresholds: gi ∈ {0.7,0.85,0.95}, qi,fi ∈ {0.3,0.5,0.7}. If Gemini E2E rises above 65–70% for any plausible relaxation while the structure-vs-joint gap collapses, the absolute 48.6% bottleneck claim is overstated; if E2E stays <55% across the grid, the claim is robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Abstract; Table 1; §3.4 Eqs. 2–3) is that raw medical data standardization is a critical unsolved bottleneck because even Gemini 3 Flash reaches only 48.6% End-to-End Strict Pass. E2E is defined as the fraction of samples that simultaneously satisfy: valid image-JSON pair, exact source match (si=1), schema validity gi≥0.85, semantic correctness qi≥0.5, and content fidelity fi≥0.5. The paper itself notes that SV is high (80–88%) while joint metrics are low, and that “a minor schema defect should not erase credit,” yet never reports how E2E (or SCJ) moves when the three free thresholds are varied. If relaxing gi to 0.7 or qi/fi to 0.4 lifts Gemini E2E above ~70% while preserving the same qualitative ranking, the absolute “48.6% bottleneck” number is an artifact of the chosen cutoffs rather than a stable measure of pipeline failure. The public-dataset / residual-annotation concern raised by the Reader is real but secondary; the load-bearing internal vulnerability is that the headline quantitative claim is an untested function of three hand-chosen gates.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that medical VLM evaluation has skipped an upstream step: converting raw, heterogeneous clinical dataset folders into source-grounded, VLM-ready image–JSON units. It introduces MDS-Bench (1,939 samples over ~100 public imaging datasets spanning classification, segmentation, and detection; diverse modalities and raw formats) and an eleven-metric protocol that gates field scores on source matching and reports structure, semantic, content, metadata, and joint success, including End-to-End Strict Pass (E2E; Eqs. 2–3). Nine agentic VLMs are evaluated under a staged standardization workflow; Gemini 3 Flash is strongest yet reaches only 48.6% E2E (Table 1), with failures concentrated in content fidelity and joint pipeline success (Table 2, Fig. 6). An inference-strategy ablation (Fig. 7) shows validation-guided complete-candidate selection improves E2E but does not close the gap. The authors conclude that raw medical data standardization is a critical bottleneck for deploying medical VLMs in practice.","tokens_in":26642,"tokens_out":1558,"duration_ms":20971,"significance":"If the empirical picture holds, the work usefully reframes medical multimodal evaluation around a previously under-measured prerequisite rather than only post-curation diagnosis or VQA. Strengths include: a large multi-dataset construction with human verification of source-grounded targets; source matching as a gate for field metrics (methodologically appropriate); equal-weight dataset aggregation; task- and modality-wise breakdowns; error attribution by capability group; and a controlled inference-strategy ablation. The staged agentic task definition and the SCJ/E2E joint metrics make the bottleneck claim falsifiable in a way that schema-only structured-output benchmarks do not. The Limitations section is appropriately candid about public-dataset scope and residual annotation risk. These contributions are of clear interest to medical AI and multimodal agent evaluation, provided the headline quantitative claims are shown to be robust to scoring choices and the agent protocol is fully specified for reproduction.","major_comments":[{"comment":"§3.4, Eqs. (2)–(3), Table 1: The central quantitative claim (48.6% E2E for Gemini 3 Flash; Abstract) is defined by three free gates—gi ≥ 0.85, qi ≥ 0.5, fi ≥ 0.5—plus exact source match. The manuscript motivates gi ≥ 0.85 as avoiding over-penalizing minor schema defects, but does not report sensitivity of E2E (or SCJ) under alternative thresholds (e.g., gi ∈ {0.7, 0.9, 1.0}, qi/fi ∈ {0.4, 0.6}). Because SV is already high (80–88%) while joint scores are low, the absolute “bottleneck” percentage may be partly threshold-driven. A short sensitivity table or curve for the top models is needed so readers can separate robust pipeline failure from cutoff choice; without it, the headline number is under-supported even if the qualitative structure–joint gap remains.","section":"§3.4 Evaluation Protocol; Eqs. (2)–(3); Table 1"},{"comment":"§4.1 Experimental Setup: The evaluation is agentic (file inspection, tools, executable scripts; §3.3), yet the paper does not specify the shared coding-agent environment, available tools, prompt/scaffold text beyond the staged workflow sketch, interaction budget (turns/tool calls), timeout, or whether models may install packages or only use a fixed sandbox. These details are load-bearing for a benchmark whose scores depend on multi-step file recovery and conversion. Please add a reproducible agent protocol (or release the exact harness) so that the 48.6% E2E and model ranking can be re-run under matched conditions.","section":"§4.1 Experimental Setup; §3.3 Staged Reasoning Design"},{"comment":"Abstract / §1 / Limitations: The claim that standardization is a “critical bottleneck … in real practice” is only partially supported by evidence from public research datasets with verified folder layouts. Limitations correctly notes missing PACS/EHR linkage, access control, and incomplete clinical metadata. The abstract and conclusion should be tightened to match the evidence (e.g., bottleneck on heterogeneous public raw imaging archives under the defined schema), or the authors should add a small clinical-style stress subset (messy paths, missing sidecars, multi-study folders) that tests the same metrics. As written, the leap from MDS-Bench to clinical deployment overstates what Table 1 measures.","section":"Abstract; §1 Introduction; Limitations"}],"minor_comments":[{"comment":"Figure 5 heatmaps use hard-to-read glyph encodings in the manuscript text dump; ensure the camera-ready figure has clear numeric annotations or a readable colorbar so per-task differences are inspectable without the appendix tables.","section":"Figure 5"},{"comment":"Figure 6 capability weights (0.15/0.20/0.25/0.10/0.30) are free parameters; state that Wgt is a presentation device and that model ranking under equal metric weights (or leave-one-group-out) is unchanged, or move Wgt to appendix.","section":"Figure 6; §4.2"},{"comment":"Detection has only six datasets (Tables 8–9); the paper already cautions this is indicative—please flag the same caveat in the main-text discussion of Figure 5 so readers do not over-read detection rankings.","section":"Figure 5; Appendix B.2"},{"comment":"Ground-truth construction (§3.2): briefly quantify human verification effort (e.g., fraction of fields corrected from model drafts, inter-annotator check on a subset) to bound residual annotation error mentioned in Limitations.","section":"§3.2 Benchmark Construction; Limitations"},{"comment":"Minor consistency: Abstract says “manually annotate 1,939” while §3.2 describes model-assisted drafts plus human verification—align wording so the construction pipeline is unambiguous.","section":"Abstract; §3.2"},{"comment":"Eq. (15) MSJ uses a geometric mean of metadata components times CF; a one-sentence justification for the geometric mean (vs. product or min) would help readers interpret Table 1’s MSJ column.","section":"Appendix C.6; Eq. (15)"}],"recommendation":"major_revision","confidential_remarks":"The core idea and benchmark are timely and likely of interest; the main risk is overselling a threshold-dependent E2E percentage and an underspecified agent harness. I would not reject on novelty or circularity grounds—the evaluation is external and source-gated. If the authors add threshold sensitivity, full agent protocol/release plans, and tone down “real clinical practice” language, this could become a solid accept after revision. Please ensure data/code release commitments are enforceable for a benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real contribution is the task definition itself. Medical VLM benchmarks start from curated images or QA pairs; this paper measures the step before that—raw dataset folder to source-matched image plus dual JSON—and shows current agentic VLMs are weak on it. MDS-Bench (1,939 samples, ~100 public datasets, classification/segmentation/detection, many modalities and formats) plus the eleven-metric protocol with source matching as a gate is the new piece of work. That is useful and correctly scoped.\n\nWhat they do well: equal-weight averaging across datasets, source-gated field metrics so wrong-source outputs do not get free credit, task/modality breakdowns, error attribution into Structure/Semantic/Content/Metadata/Joint, and a clean inference-strategy ablation (complete-candidate selection helps most). The qualitative pattern is robust: SV 80–88% while SCJ/E2E stay much lower; Content and Joint dominate failures. Gemini 3 Flash leads but still fails more than half the samples under their strict pass. Related-work placement against MultiMedQA, PMC-VQA, GMAI-MMBench, StructEval, Image2Struct, etc. is fair. Circularity is low; ground truth is human-verified against raw sources.\n\nSoft spots, in proportion. The stress-test is right that E2E (gi≥0.85, qi≥0.5, fi≥0.5) is hand-gated and they never report sensitivity; the absolute 48.6% could move. That weakens the headline number, not the existence of a pipeline gap—SCJ and the multi-model, multi-task heatmaps still show the same structure-vs-joint drop. Public imaging folders are not PACS/EHR/access-control environments (they say so in Limitations). Model-assisted draft + human check is standard but residual annotation noise is possible. No code/data release and closed APIs hurt reproducibility. Capability weights in Fig. 6 are arbitrary. None of these erase the core empirical observation.\n\nThis is for people building medical agents or data-prep tooling, and for anyone who treats diagnosis-bench numbers as deployment readiness. It deserves a serious referee. I would bring it to reading group, cite the task framing and the structure/joint gap, and accept for peer review with the usual ask for threshold sweeps and artifact release.","headline":"Solid new upstream benchmark: raw medical folders → source-grounded image+JSON is genuinely missing from medical VLM eval, and the structure-high/joint-low pattern is multiply evidenced even if the exact 48.6% E2E number is threshold-sensitive.","tokens_in":27303,"tokens_out":616,"would_cite":true,"duration_ms":6609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Even the best vision-language models succeed on only about half of raw medical data standardization tasks, exposing a missing first step before diagnosis.","keywords":["vision-language models","medical AI","raw data standardization","MDS-Bench","agentic VLMs","source-grounded evaluation","medical imaging"],"falsifier":"A controlled re-evaluation on private hospital folders (or a public subset with deliberately incomplete metadata) that drives the best model’s end-to-end rate well above or well below 48.6 percent under the same gates would confirm or undermine the bottleneck claim.","tokens_in":27094,"feed_emoji":"🏥","tokens_out":630,"duration_ms":7451,"temperature":0.7,"pith_summary":"Medical vision-language models are usually tested after someone has already cleaned images, labels, and reports into neat pairs. Real clinics do not start there: data arrive as messy folders of volumes, masks, tables, and idiosyncratic directory layouts. This paper defines that missing upstream job—raw medical data standardization—as an agentic task in which a model must find the right sources, convert them into a common image form, and emit source-grounded structured annotations. The authors build MDS-Bench with 1,939 human-verified tasks across roughly one hundred public imaging datasets spanning classification, segmentation, and detection, many modalities, and many file formats. On this benchmark the strongest model still reaches only 48.6 percent strict end-to-end success. The result is meant to show that standardization itself, not only diagnosis after clean inputs, is a major practical bottleneck.","feed_headline":"Best VLMs clear only 48.6% of raw medical data tasks","feed_subtitle":"Messy clinical folders, not clean diagnosis pairs, remain the unsolved first step for medical AI.","key_machinery":"MDS-Bench: 1,939 source-traced standardization tasks scored with an eleven-metric protocol whose strict end-to-end pass requires a valid image-JSON pair, exact source match, schema validity at least 0.85, and semantic and content-fidelity scores at least 0.5.","core_discovery":"Current vision-language models can often emit schema-valid JSON, yet they fail to complete the full pipeline of source identification, visual conversion, annotation alignment, and content-faithful structuring from raw heterogeneous medical folders. On MDS-Bench the best model reaches only 48.6 percent end-to-end strict success, so raw medical data standardization remains unsolved and limits real-world medical AI.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Best VLMs hit only 48.6% on raw medical data standardization","MDS-Bench: top VLM clears just 48.6% of messy clinical folders","Raw medical data stalls VLMs at 48.6% end-to-end success","VLMs fail half of raw heterogeneous medical folder tasks","Gemini 3 Flash reaches 48.6% on medical data standardization"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The claim rests on treating human-verified ground truth from public imaging datasets, scored with the authors’ chosen end-to-end thresholds, as a fair proxy for real clinical raw-data environments with private systems and incomplete metadata.","fun_headline_variants_meta":{"raw":{"variants":["Best VLMs hit only 48.6% on raw medical data standardization","MDS-Bench: top VLM clears just 48.6% of messy clinical folders","Raw medical data stalls VLMs at 48.6% end-to-end success","VLMs fail half of raw heterogeneous medical folder tasks","Gemini 3 Flash reaches 48.6% on medical data standardization"]},"model":"grok-4.5","effort":"low","cost_usd":0.005068,"raw_usage":{"total_tokens":1424,"prompt_tokens":773,"num_sources_used":0,"completion_tokens":103,"cost_in_usd_ticks":50680000,"prompt_tokens_details":{"text_tokens":773,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":548,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":773,"tokens_out":103,"duration_ms":4301,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T15:01:34.386660+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A controlled re-evaluation on private hospital folders (or a public subset with deliberately incomplete metadata) that drives the best model’s end-to-end rate well above or well below 48.6 percent under the same gates would confirm or undermine the bottleneck claim.","supporting_citations":[],"review_version":1}