{"id":"284fde2d-e469-4bda-9d58-43c9785719f2","arxiv_id":"2608.10635","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A medical vision-language model that represents masks as discrete tokens and is trained on a large four-stream corpus reports strong unified results, though its benchmark comparisons are not yet fair or fully held out.","lead":"MedUP turns medical image regions into special language tokens so one vision-language model can answer questions, describe regions, and draw segmentation masks in the same output space. The paper's benchmark evidence is weakened by an unfair protocol comparison and unclear train-test dataset separation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Region-grounded comparison mixes protocols and the benchmark may share datasets with training, so the 'across all tasks' claim is not yet supported.","rationale":"The reader's weakest assumption identifies the same core problem: the benchmark comparison confounds model quality with evaluation protocol, and the train/benchmark split is not demonstrated to be disjoint. Section 3.1 and Section 3.2 make the overlap concern concrete, and Table 4 quantifies the protocol effect as roughly 29-32 EM points, which is large relative to the headline differences. This is not a minor presentation issue: the paper's central empirical claim is 'outperforms native, agentic, and dual-decoder Med-VLMs across all tasks,' and that claim cannot be checked from the reported numbers. The underlying token-interface direction is plausible and the round-trip filtering ablation (Figure 5) is internal evidence, but no code, data, or formal verification is released, and the Seg-CoT contribution is explicitly deferred to future work. For these reasons I keep the reader's REJECT recommendation rather than treating the concern as a wording artifact. A matched-protocol, dataset-disjoint evaluation could change the verdict.","tokens_in":21579,"tokens_out":7996,"duration_ms":78654,"concrete_test":"Obtain or reconstruct the UniMed-Train and UniMed-Bench split lists and compute dataset-name and image-ID overlap. If any overlap exists, re-run the full evaluation on the disjoint subset of benchmark datasets, with every method scored under a single matched protocol (e.g., all models under v1_masks, and additionally all models with token-based grounding where applicable). If MedUP's EM falls to roughly the v1_masks level (≈50) on the disjoint subset, or if baselines close the gap under matched protocols, the 'across all tasks' claim is not supported.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that MedUP natively unifies understanding and perception and outperforms all comparison families rests on a benchmark that does not isolate the proposed interface. In Table 1, MedUP's region-grounded understanding is scored with v2_tokens, while every baseline is scored with v1_masks (footnote to Table 1). Section 4.4/Table 4 shows that this protocol choice alone moves EM from 49.8 to 78.5 for MedUP-Q and to 81.8 for MedUP-H on the same models. The apparent margin over the listed baselines (≤4.4) is therefore largely protocol-driven; although MedUP under v1_masks still beats these baselines, those baselines are not matched in training or interface, so model quality is not isolated. In addition, Section 3.1 states that the two mask-centric training streams are built from '80+1 medical segmentation datasets,' while Section 3.2 describes UniMed-Bench as using 'the same 80 datasets' and calls it held-out, without specifying a dataset-level split. If the benchmark draws images or datasets from the training family, the segmentation and region-understanding numbers reflect distribution overlap, not generalization. Section 4.6 also explicitly postpones Seg-CoT ablations, so the claimed Seg-CoT contribution currently has no quantitative support. Together these confounds make the headline comparison unverifiable as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MedUP, a medical vision-language model that unifies text-guided segmentation, region-grounded understanding, and medical VQA within a single autoregressive framework. The core contribution is UniMedTok, a frozen image-conditioned tokenizer that encodes segmentation masks as two discrete tokens in the LLM vocabulary, allowing masks to be interleaved with text. The authors construct a large training corpus (UniMed-Train, 1.84M instances) and a three-task benchmark (UniMed-Bench), and report that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors. The central claims depend on the validity of the benchmark comparison and the claimed benefits of the mask-token interface and Seg-CoT reasoning.","tokens_in":21886,"tokens_out":5072,"duration_ms":47413,"significance":"The idea of representing medical regions as discrete tokens in the language modeling space is timely and aligns with recent general-domain mask-token methods (e.g., SAMTok, HiMTok). If the reported results were properly controlled, the paper would provide a useful medical-domain instantiation, a large corpus, and a benchmark that could advance unified perception-understanding modeling. The paper also ships a large corpus and explicit two-stage training pipeline. However, the current evaluation contains protocol and data-overlap confounds that prevent the headline claims from being verified. The central contribution is plausible and potentially valuable, but the evidence as presented is not yet convincing.","major_comments":[{"comment":"The main comparison for region-grounded understanding is not apples-to-apples. Table 1 reports MedUP under the v2_tokens protocol while all baselines are evaluated under v1_masks, as stated in the table footnote and in Section 4.1. Table 4 shows that on the same backbones, the protocol alone changes exact match from 49.8 (v1_masks) to 78.5 (MedUP-Q) and 81.8 (MedUP-H), a gap of roughly 30 points. The reported margins over baselines (up to 81.8 vs 4.4) are therefore largely protocol-driven. The claim in the abstract that MedUP 'outperforms native, agentic, and dual-decoder Med-VLMs across all tasks' is not supported by this comparison, because the region-grounded capability is not measured under matched conditions. Please compare MedUP and baselines under both protocols, or clearly restrict the claim to v1_masks, and discuss the protocol effect explicitly.","section":"Table 1 and Table 4"},{"comment":"The held-out status of UniMed-Bench is not established. Section 3.1 states that the mask-centric training streams are built from '80+1 medical segmentation datasets,' while Section 3.2 describes UniMed-Bench as 'built from the same 80 datasets' and calls it held-out. If the benchmark draws images or datasets from the same family used to construct Stage-2 training and to train the Stage-1 tokenizer, then the segmentation and region-grounded results reflect distribution overlap rather than generalization. The paper must specify whether the split is at the dataset level, the image level, or the patient level, and must confirm that no training images appear in UniMed-Bench. Without this clarification, the benchmark numbers cannot be interpreted as measuring generalization.","section":"Sections 3.1 and 3.2"},{"comment":"The contribution of Seg-CoT is not quantitatively supported. The abstract and conclusion state that Seg-CoT improves text-to-mask generation through intermediate reasoning, but Section 4.6 explicitly says 'We leave additional ablations such as round-trip filtering, token budget, and Seg-CoT training effects for future versions once the corresponding experimental evidence is included.' No ablation compares MedUP with and without Seg-CoT. Either provide the missing ablation or remove the claim that Seg-CoT is an empirically validated contribution.","section":"Section 4.6, Abstract, Conclusion"},{"comment":"The role of the tokenizer in bounding segmentation performance should be quantified. The tokenizer is trained on masks from the same dataset family used for Stage-2 training and for evaluation, and it is frozen when decoding all generated mask tokens. The end-to-end segmentation numbers therefore depend on the tokenizer's reconstruction fidelity on exactly those datasets. Reporting the tokenizer's reconstruction Dice/IoU on UniMed-Bench would clarify how much of the reported segmentation performance is attributable to the VLM versus the tokenizer, and would assess the circularity risk that the tokenizer simply memorizes masks from the training family.","section":"Section 2.3 and Appendix C.1"}],"minor_comments":[{"comment":"The aggregation metric for Dice is inconsistent: Table 1 reports macro Dice while Table 3 reports micro Dice. Please define both clearly and indicate which is used for the headline claims.","section":"Tables 1 and 3"},{"comment":"The appendix tables are explicitly selected to show only rows where MedUP achieves the best score. This selection is biased and should be replaced by full results or clearly labeled as illustrative subsets.","section":"Appendix E, Tables 7-10"},{"comment":"Several references (e.g., Aho and Ullman 1972, APA 1983, Chandra et al. 1981, Gusfield 1997) appear unrelated to the medical vision-language content and should be removed or replaced with relevant citations.","section":"References"},{"comment":"The offset of 256 for the second codebook token is mentioned only in the equation. Please add a sentence in the main text explaining that the 512 mask tokens are split into two groups of 256.","section":"Equation (3)"},{"comment":"The Seg-CoT examples show reasoning traces with mask tokens, but the displayed overlays do not show the decoded masks. Including decoded mask visualizations would help the reader verify that the generated tokens correspond to meaningful regions.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is relevant and potentially significant, but the current evaluation has two serious confounds: the protocol mismatch in Table 1/Table 4 and the unclear dataset overlap between UniMed-Train and UniMed-Bench. These are fixable with additional experiments and clarifications, but they must be addressed before the claims of outperforming all baseline families can be taken seriously. The Seg-CoT claim also needs empirical support. I recommend major revision rather than rejection because the architectural idea and corpus are valuable, and the issues are within the scope of a thorough revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read MedUP. The core idea is sound: take the mask-token discretization from SAMTok/HiMTok, adapt it to medical imaging, and train a single autoregressive VLM that handles VQA, text-guided segmentation, and region-grounded understanding in one token space. That is a legitimate contribution, and the paper is honest about its lineage.\n\nWhat is genuinely new is the scale and packaging: a 1.84M-instance four-stream training corpus, a unified benchmark across 80 datasets, and a clean two-stage training recipe. The protocol study in Table 4 is also a good-faith internal comparison; it shows v2_tokens beating v1_masks by roughly 30 EM points on their own models.\n\nThe soft spot is the main comparison. Table 1 scores MedUP under v2_tokens while baselines get v1_masks, and those baselines are not fine-tuned for region-grounded understanding. So the 78.5 EM vs 4.4 EM gap confounds interface with training. Table 4 shows MedUP under v1_masks still reaches 49.8 EM, which is well above the baselines, but those baselines are essentially zero-shot at this task, so the comparison does not isolate model quality. The paper calls UniMed-Bench 'held-out' while saying it uses 'the same 80 datasets' as the training corpus, without specifying the split level. If the split is per-image rather than per-dataset, the segmentation and grounding numbers reflect distribution overlap. Section 4.6 also explicitly postpones Seg-CoT ablations, so one of the three claimed contributions currently has no quantitative support. No code or data is released either.\n\nThese are load-bearing issues, not cosmetic ones. The headline claim of outperforming all families is not verifiable as reported. But the underlying interface is promising, and the corpus and benchmark could become a real resource if the authors rerun the comparison with matched protocols, fine-tuned baselines, and dataset-level held-out splits, and also report Seg-CoT numbers.\n\nI would send this to peer review rather than desk reject. The technical core is coherent, the resource is substantial, and the flaws are fixable, though they require real work. For a reading group it is a useful case study in evaluation pitfalls for grounded VLMs.","headline":"A coherent medical mask-token VLM with a large new corpus, but the main 'outperforms all' comparison mixes protocols and may share training datasets with the benchmark, so the headline claim is not yet supported.","tokens_in":671,"tokens_out":901,"would_cite":false,"duration_ms":36638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masks become discrete tokens in the language model's vocabulary, allowing a single medical vision-language model to answer questions, segment structures, and interpret regions without external decoders.","keywords":["medical vision-language model","UniMedTok","mask tokenization","text-guided segmentation","region-grounded understanding","medical visual question answering","Seg-CoT","vector-quantized mask autoencoder"],"falsifier":"A decisive check would be to list the dataset-level overlap between UniMed-Train and UniMed-Bench, and to train the strongest overlay-based baseline on the same token-based supervision before evaluating it with overlays; if the overlap is substantial, or if the baseline's region-grounded exact match jumps toward 78–82%, the claimed superiority would not be native unification.","tokens_in":21344,"feed_emoji":"🩺","tokens_out":17800,"duration_ms":142273,"temperature":0.7,"pith_summary":"MedUP is designed to show that a medical vision-language model can handle both understanding and perception in one autoregressive network if segmentation masks are turned into short sequences of discrete tokens. The paper introduces UniMedTok, a frozen image-conditioned mask tokenizer that compresses each mask into two 256-entry codebook tokens, and then trains the VLM on a 1.84M-instance corpus mixing medical VQA, text-guided segmentation, region-grounded understanding, and reasoning-augmented segmentation. On the new UniMed-Bench, MedUP reports higher scores than native, agentic, and dual-decoder baseline families on all three tasks while remaining competitive with specialist segmentors that receive oracle visual prompts. The paper's intended lesson is that treating regions as language removes the representation gap created by coordinate strings or external segmentation modules.","feed_headline":"Mask tokens let one medical AI segment, answer, and ground","feed_subtitle":"Discrete mask tokens let one medical model answer, locate, and segment without external decoders.","key_machinery":"The load-bearing mechanism is UniMedTok, a medical mask tokenizer that acts as a native region-language interface. It is an image-conditioned vector-quantized autoencoder with the MT256×2 scheme: two non-shared codebooks of 256 entries each, trained in Stage 1 by mask reconstruction and then frozen. Stage 2 expands the VLM vocabulary with a start token, an end token, and 512 mask code tokens, so the serialization operator $S(M)$ turns any mask into a four-token span. Mask-as-input appends that span to the user prompt, while mask-as-output has the VLM generate the span and the frozen tokenizer decode it back to a dense mask. This mechanism carries the argument because all three tasks are reduced to the same autoregressive next-token objective over text and mask tokens, with no trainable segmentation head.","core_discovery":"The central discovery the authors are trying to establish is that a mask tokenizer can live inside the language model's vocabulary and make perception a form of language modeling. UniMedTok encodes a mask, conditioned on the medical image, into an ordered two-code pair $q=[c_1,c_2]$ from 256-entry codebooks, and with start and end tokens this becomes a four-token text span. The same span can be generated as output for text-guided segmentation or inserted into the prompt for region-grounded understanding, so all tasks collapse into next-token prediction in one shared space. After two-stage training on 1.84M instances, the paper reports that MedUP beats native, agentic, and dual-decoder baselines on all UniMed-Bench tasks and remains competitive with specialist medical segmentors across modalities.","pith_inferences":["Beyond the paper, a decisive next experiment is to train a non-token baseline on the same token-based supervision and evaluate it with overlaid masks; the paper's protocol table suggests the protocol alone can move exact match by about 29 points, so this comparison would separate interface benefit from model quality.","Because both the training corpus and benchmark live in the same 80-dataset family, an external-transfer split on datasets seen in neither stage would test whether the mask-language interface generalizes beyond familiar datasets.","The same mask-token recipe could extend to other dense medical outputs, such as landmark heatmaps, lesion counting, or hierarchical anatomy decomposition, though the paper only demonstrates binary mask prediction.","The paper's limitations section notes the compact two-code representation and offline-only evaluation, so interactive refinement and very small or irregular structures remain open rather than established."],"forward_implications":["A single medical VLM can handle both image-level questions and pixel-precise localization, removing the need for an external segmentation tool or a separate dense decoder for text-guided masking.","Region-grounded understanding is more effective when the region is given as mask tokens rather than as a visual overlay: the paper's protocol study reports exact match rising from 49.8% to 78.5–81.8%.","Round-trip filtering of mask supervision matters: removing low-fidelity mask-token training cases improves text-guided segmentation mean Dice by 7.1 to 9.3 points.","Segmentation-oriented chain-of-thought (Seg-CoT) fits naturally into the same token space, so masks can be produced with an intermediate anatomical and localization rationale within one generation.","Both grounded tasks improve with more Stage-2 training data in the tested scale range, suggesting that larger region-language corpora are a direct route to better grounded Med-VLMs."],"supporting_citations":[{"why":"Establishes the mask-as-two-words token interface that UniMedTok adapts to medical images.","marker":"Zhou et al., 2026"},{"why":"Shows hierarchical mask tokens can be decoded by large multimodal models, motivating the choice of tokenized masks instead of a dense head.","marker":"Wang et al., 2025b"},{"why":"Defines LISA-style reasoning segmentation and supplies the dual-decoder baseline class MedUP must beat.","marker":"Lai et al., 2024"},{"why":"Provides the HuluMed-4B backbone used by the stronger MedUP-H variant.","marker":"Jiang et al., 2025b"},{"why":"Provides the Qwen3-VL-4B backbone used by the MedUP-Q variant.","marker":"Bai et al., 2025"},{"why":"Supplies the chain-of-thought mechanism that Seg-CoT adapts to text-to-mask prediction.","marker":"Wei et al., 2022"},{"why":"Gives the MedSAM specialist segmentor family that the paper claims MedUP remains competitive with.","marker":"Ma et al., 2024"},{"why":"Represents the UniBiomed dual-decoder grounded baseline used in the region-understanding comparisons.","marker":"Wu et al., 2025b"}],"fun_headline_variants":["MedUP: mask tokens make perception a language task","One token space for masks and text: MedUP unifies Med-VLM","Discrete mask tokens in LLM vocab beat external segmentors","Native mask tokens: MedUP segments and reasons without decoders","MedUP collapses segmentation and VQA into next-token prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the benchmark being truly held out from the training corpus and on comparing MedUP with token-based region references against baselines with overlaid masks measuring model quality rather than protocol, especially since the paper's own protocol study moves exact match from 49.8% to 78.5–81.8%.","fun_headline_variants_meta":{"raw":{"variants":["MedUP: mask tokens make perception a language task","One token space for masks and text: MedUP unifies Med-VLM","Discrete mask tokens in LLM vocab beat external segmentors","Native mask tokens: MedUP segments and reasons without decoders","MedUP collapses segmentation and VQA into next-token prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2981,"prompt_tokens":885,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2009}},"tokens_in":501,"tokens_out":2096,"duration_ms":13723,"temperature":1.0,"reasoning_tokens":2009,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:28:39.322994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to list the dataset-level overlap between UniMed-Train and UniMed-Bench, and to train the strongest overlay-based baseline on the same token-based supervision before evaluating it with overlays; if the overlap is substantial, or if the baseline's region-grounded exact match jumps toward 78–82%, the claimed superiority would not be native unification.","supporting_citations":[{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought mechanism that Seg-CoT adapts to text-to-mask prediction."}],"review_version":1}