{"id":"8f7ec5fc-592b-4260-8dff-f82f40d403b2","arxiv_id":"2602.14134","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A standard 4B MLLM, trained with a multi-label loss on vision tokens, directly extracts segmentation and depth maps from vision-token logits without task-specific heads.","lead":"DenseMLLM shows a standard 4B multimodal LLM can do semantic segmentation, depth estimation, and referring segmentation by taking argmax over its own vision-token logits, with no extra decoder. It matters because it suggests dense perception can be a native ability of ordinary vision-language models, simplifying generalist system design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RefCOCO results rely on a two-stage grounding+crop+second-pass pipeline, contradicting the paper's 'zero extra inference' claim and leaving single-pass referring segmentation unvalidated.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption focused on spatial resolution (whether vision tokens after 2x2 merge retain enough detail). I agree that this is a genuine concern, and it is partially supported by Table 5. However, I find a more concrete and internally documented problem in the RefCOCO evaluation: the paper's own appendix describes a two-stage pipeline that contradicts the explicit claim of zero extra inference and no retrieval. This is not a minor wording issue; it directly affects the credibility of a headline benchmark (80.7 cIoU) and the generality of the 'standard MLLM' claim. The concern is load-bearing because if the single-pass variant performs poorly, then for one of the three tasks representing the paper's scope, the method does not work as advertised. The attack is not an ad hominem or a rejection of the overall idea; the ablation study and the segmentation/depth results remain plausible. But the paper must reconcile this inconsistency before the claim can be accepted at face value. My recommendation therefore stays at CONDITIONAL (UNCHANGED from the reader's verdict), with the condition that the authors provide single-pass RefCOCO results or explicitly restate the inference protocol. I do not raise this as grounds for REJECT because the central mechanism—multi-label supervision on vision tokens—is coherent and independently ablated; the RefCOCO issue is a claim-reality gap, not a fundamental unsoundness.","tokens_in":24351,"tokens_out":14083,"duration_ms":142045,"concrete_test":"Run the released DenseMLLM (or ask authors) on RefCOCO-val using only the single-pass polygon protocol in Appendix D.2 (no bounding-box prompt, no crop/resize, no second segmentation pass) and compute cIoU. Compare to the reported 80.7. If the single-pass result is substantially lower (e.g., <75), the two-stage pipeline is responsible for the headline number, and the claims of 'zero extra inference' and 'without retrieval' must be retracted. As a secondary check, inspect the public code repository's default RefCOCO evaluation script to see whether it invokes the two-stage pipeline; if it does, this confirms the dependence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a standard MLLM can perform dense predictions directly from vision tokens, without additional inference steps (Sec. 1, Table 8). This claim is load-bearing for the RefCOCO results in Table 1. The paper reports 80.7 cIoU on RefCOCO-val and states it exceeds UFO 'without the retrieval process.' However, Sec. 4.1 and Appendix D.2 describe a two-stage protocol: first, the model is prompted to produce a bounding box ('Please provide the bounding box coordinate...'), then the image is cropped with 1.2x padding and resized to short edge 1280, and a second prompt ('Segment the core target.') is run on the crop. The mask is then re-inserted into the original image. This is two full MLLM inferences plus image transformations, not zero extra inference. The appendix's single-pass alternative (polygon generation) is described as 'basic' and 'not evaluated.' Thus, for one of the three core dense-prediction tasks, the reported number is not produced by the claimed 'no additions' inference. The central claim is not falsified for segmentation/depth, but the general claim 'without additional inference steps' is false for referring segmentation, and the single-pass capability is untested. The paper should either report the single-pass RefCOCO result or openly state that referring segmentation uses a task-specific two-stage pipeline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DenseMLLM, a 4B-parameter multimodal LLM built from a standard ViT, projector, and LLM backbone. Its main contribution is a multi-label vision-token next-token-prediction loss (NTP-M), which supervises the logits of every vision token against a unified vocabulary containing semantic classes, depth bins, and referring-segmentation labels. At inference, dense predictions are read out directly from those vision-token logits by indexing with text-token IDs, applying argmax, and bilinearly upsampling to image resolution. The authors report strong results on semantic segmentation (54.2 mIoU on ADE20k), depth estimation (87.6 δ1 on DDAD), and referring segmentation (80.7 cIoU on RefCOCO-val), while maintaining competitive general VQA/OCR performance against other 4B MLLMs. The paper claims that this demonstrates a standard MLLM can act as an intrinsic dense predictor without task-specific decoders and without additional inference steps.","tokens_in":24697,"tokens_out":7394,"duration_ms":66974,"significance":"The potential significance is substantial. If the claims hold, the paper shows that a standard MLLM architecture, trained with a simple modification of next-token prediction, can produce dense maps without any specialized decoder, contradicting a widespread assumption in the field. The NTP-M objective is clearly motivated by the multi-label nature of vision tokens, and the internal ablations (Table 3) provide strong evidence that the relevant-negative-sampling component is crucial: performance on ADE20k jumps from 16.7 to 32.7 to 51.2 mIoU across the ablations. The preservation of general VQA and OCR capabilities (Table 2) is also a useful result. The paper provides a detailed four-stage training recipe and identifies a concrete, reproducible mechanism. The main weaknesses are in the evaluation protocol: the RefCOCO numbers come from a two-stage grounding-then-segmentation pipeline, and the depth evaluation uses benchmark-specific prompts and per-dataset quantization ranges. These issues directly affect the central 'zero extra inference' and 'general-purpose' claims, so they must be resolved before the paper can be accepted.","major_comments":[{"comment":"The RefCOCO cIoU numbers in Table 1 are not produced by the single-pass readout claimed in Sec. 1 and Fig. 2c. Appendix D.2 describes the actual protocol: a first MLLM inference answers 'Please provide the bounding box coordinate...', the image is cropped with 1.2× padding and resized to short edge 1280, a second MLLM inference on the crop ('Segment the core target.') yields the mask, and the mask is reinserted into the original image. This is two full model forwards plus image transformations. Table 8 therefore reports 'Ours 0' extra inference only by not counting the RefCOCO procedure, and the Abstract/Sec. 1 statement about 'no additional inference steps' is false for one of the three headline tasks. The single-pass polygon variant is described as 'basic' and 'not evaluated.' Please provide a single-pass RefCOCO result, or explicitly re-scope the claim to a two-stage standard-MLLM pro","section":"§4.1 Evaluation; Appendix D.2; Table 8"},{"comment":"Depth evaluation conditions on dataset identity and per-dataset depth ranges. The prompt says 'from the B dataset' with B = NYUv2/Cityscapes/DDAD, and the text describes different linear quantization ranges per benchmark (0–10 m, 0–80 m, 0.05–120 m). The reported δ1 numbers therefore measure the model with access to the benchmark name and its known depth range, not a prompt-agnostic generalist depth predictor. Please state whether these prompts and quantizations were seen during training, and include at least one cross-dataset evaluation with an open-style prompt (e.g., 'Estimate the depth') to support the claim that a standard MLLM performs general-purpose depth estimation without task-specific customization.","section":"§4.1 Evaluation; Appendix D.3"},{"comment":"The 'zero overhead' framing should account for test-time scaling and post-processing. Eq. (2) up-samples the 1/32-resolution vision-token logits with bilinear interpolation, and Table 5 shows that results continue to improve up to the largest tested scale (NTP-M 45.0 at 1/32 vs. 51.2 at 4/32). The evaluation protocols use 2×–4× input upscaling on several benchmarks and, for ADE20k, a softmax temperature of 0.2 plus DenseCRF. These are standard engineering choices, but they are part of the reported dense-prediction pipeline and should be disclosed in the main-text cost comparison (Table 8), since they add non-negligible compute and are not common to all compared baselines.","section":"§3.1 Eq. (2); Table 5; Appendix D.1"}],"minor_comments":[{"comment":"Typos: 'Qwne2.5-VL' should be 'Qwen2.5-VL'; UniPixel is cited as (Rasheed et al., 2024) but the correct reference is (Liu et al., 2025).","section":"Table 1"},{"comment":"The phrase 'ensuring the indirectness of the framework' is unclear; presumably 'standardness' or 'simplicity' is intended.","section":"Sec. 3.1"},{"comment":"The sentence 'ing our proprietary internal data sources cannot be disclosed' is garbled; please rephrase.","section":"Sec. 4.1"},{"comment":"The baseline naming is inconsistent: Table 3 calls the starting point 'Base (BCE)' while Table 4 calls the equivalent configuration 'NTP-M raw'. Clarify that these denote the same setup.","section":"Tables 3 and 4"},{"comment":"The ADE20k evaluation uses randomly sampled prompts, but no experiment quantifies prompt sensitivity. Reporting mean±std over prompt sets would strengthen the robustness claim.","section":"Appendix D.1"},{"comment":"The label 'No Additions' is in tension with the two-stage RefCOCO protocol described in Appendix D.2. Please align the figure's claim with the actual evaluation.","section":"Fig. 2c"}],"recommendation":"major_revision","confidential_remarks":"The RefCOCO protocol is the deciding issue. If the authors can provide a genuine single-pass referring-segmentation number, even if lower, the central claim becomes much more credible. The depth-prompt conditioning is also important: I would ask for a cross-dataset or prompt-agnostic depth experiment to verify that the performance is not merely benchmark memorization. The core NTP-M method and ablation evidence are solid; the revision is about honest scoping of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — know this: the RefCOCO numbers don't back the zero-extra-inference claim. The paper evaluates referring segmentation with a two-stage pipeline — first grounding to get a bounding box, cropping with padding, resizing to short edge 1280, then a second full MLLM inference with \"Segment the core target\", and filling the mask back. That contradicts Table 8 and the Intro's \"without additional inference steps\" for one of the three core tasks. The appendix even says the single-pass polygon variant is \"basic\" and \"not evaluated.\" This is a load-bearing flaw in the current framing.\n\nWhat's genuinely new is the multi-label vision-token NTP loss with relevant negative sampling. That's a clean idea. The ablation is the strongest part of the paper: on ADE20k from Stage I, BCE gives 16.7, adding independent means gives 32.7, and the relevant-negative sampling jumps to 51.2. The segmentation and depth numbers are competitive with MLLMs that use task-specific decoders. The paper also includes thoughtful analysis of test-time scaling and hyper-parameter robustness.\n\nThe soft spots beyond RefCOCO are real but more minor. Depth prompts include the benchmark name (\"from the NYUv2 dataset\") and per-dataset quantization ranges, so the \"general-purpose\" claim is partly calibration, not just architecture. Test-time upscaling, DenseCRF, softmax temperature, and per-dataset settings are listed in the appendix, but the abstract-level claim hides them. The base LLM is Youtu-LLM, unpublished, and much of the training data is proprietary, so no one can independently confirm the numbers. There are no released code/weights or formal proofs in the paper.\n\nNone of this kills the core method. The idea is coherent, the ablation supports the mechanism, and the paper is honestly written about its limits. But the RefCOCO result as presented is misleading, and the single-pass referring segmentation capability is unvalidated.\n\nThis paper is for researchers working on unifying dense prediction and general VL in one MLLM. It deserves a serious referee — but with a condition: the authors must report the single-pass RefCOCO result or explicitly state that referring segmentation uses a two-stage task-specific pipeline. I would also ask for depth results without benchmark-name prompts and a clear statement of what post-processing is applied. With those revisions, it's a solid contribution — not a breakthrough, but a useful step.","headline":"DenseMLLM has a genuinely new multi-label vision-token loss and a convincing ablation, but the RefCOCO results rely on a two-stage pipeline that contradicts the core zero-extra-inference claim; otherwise worth reviewing.","tokens_in":25215,"tokens_out":3372,"would_cite":true,"duration_ms":31264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard multimodal LLM can deliver dense predictions—segmentation, depth, referring masks—straight from its vision tokens, with no task-specific decoder.","keywords":["multimodal large language models","dense prediction","semantic segmentation","depth estimation","referring expression segmentation","vision token supervision","multi-label loss","next-token prediction"],"falsifier":"Feed the model a synthetic image of alternating 1-pixel vertical stripes on a plain background and ask for segmentation at the model's native input resolution; if the output map cannot reproduce the stripes even after upsampling — while a human or a full-resolution model can — the token grid is the binding resolution limit, and the 'intrinsic dense predictor' claim would need to be retracted to 'upscaled dense predictor.'","tokens_in":24223,"feed_emoji":"🖼️","tokens_out":5574,"duration_ms":49240,"temperature":0.7,"pith_summary":"DenseMLLM claims that a standard multimodal large language model—the usual vision-encoder, projector, and LLM stack—can produce dense predictions such as semantic segmentation, depth maps, and referring-expression masks without any task-specific decoder. The key step is to treat the vision tokens the model already generates as spatial feature maps: index their logits with the vocabulary IDs of the predicted categories, then upsample and take an argmax. To make those vision tokens carry the needed detail, the paper extends next-token prediction from text to vision tokens, supervising each vision token with all the labels present in its patch (multi-label), and handles the large vocabulary with a relevant-negative sampling loss. If correct, dense perception becomes a free capability of a general-purpose MLLM rather than a separate module.","feed_headline":"Standard MLLM outputs masks and depth with no task-specific decoder","feed_subtitle":"One 4B model handles segmentation, depth, and referring masks in a single forward pass.","key_machinery":"The load-bearing component is NTP-M, a multi-label next-token prediction loss for vision tokens. Where standard NTP assigns each token a single vocabulary ID, NTP-M builds a multi-hot target vector per vision token, modelling each vocabulary entry as an independent Bernoulli trial with sigmoid activations. Because the vocabulary is enormous and most entries are irrelevant, the loss averages positives and, for negatives, keeps only the top-k most probable (most confusing) entries per token; this relevant-negative sampling is what makes training stable. On the inference side, the companion mechanism is the 'index-the-logits' extraction: predicted category names supply vocabulary IDs, and the m","core_discovery":"The paper's central discovery is that a standard MLLM, after training with a multi-label version of next-token prediction applied to its vision tokens, is itself an intrinsic dense predictor. At inference, the model first predicts the text labels of the categories present in the image; for each category it looks up the corresponding token IDs in the shared vocabulary, averages the raw vision-token logits across those IDs, reshapes them into a spatial grid, upscales with bilinear interpolation, and takes an argmax over categories to obtain the final mask or depth map. No external decoder, task token, or extra inference pass is involved. The paper reports that this minimalist recipe reaches co","pith_inferences":["If the vision tokens really carry this level of spatial detail, similar multi-label supervision could plausibly extend to video (treating frame tokens as vision tokens) — a direction the paper leaves untested.","The success of simple bilinear upsampling suggests a practical ceiling tied to the 1/32 token grid; a light learned-upsampling layer would likely improve fine details further, at the cost of breaking the 'no decoder' purity.","The paper frames the predictor as 'intrinsic,' but its own ablations show the capability is learned from the multi-label supervision, not present in a plain MLLM; a fairer reading is that standard architectures have the capacity, and the loss unlocks it.","Instance segmentation and panoptic segmentation remain out of reach without an explicit instance-discrimination mechanism, as the paper notes; the multi-label token supervision may need an auxiliary grouping cue for those tasks."],"forward_implications":["A single forward pass of a standard MLLM can replace segmentation heads, depth heads, and mask-retrieval modules for three common dense tasks.","The same training objective transfers to other standard MLLMs: the paper's transfer experiment shows the recipe also improves dense-prediction accuracy when applied to a different off-the-shelf vision-language backbone.","Dense prediction performance scales with input resolution at test time: larger images yield steadily better maps, giving a simple compute-accuracy trade-off without retraining.","Because no task-specific components are added, dense perception can be integrated into general-purpose models without hurting their VQA, OCR, and reasoning scores.","Task-specific fine-tuning on top of the trained model gives further gains, so the model serves as a reusable dense-prediction backbone."],"fun_headline_variants":["Standard LLM sees pixels: dense prediction without extra decoders","DenseMLLM: one pass to masks and depth, no decoder added","Vision tokens alone decode segmentation and depth in MLLM","No extra decoder: standard MLLM does dense prediction natively","Multi-task masks and depth from a vanilla MLLM, no decoder"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 1/32-resolution vision-token grid, after simple bilinear upsampling, retains enough spatial detail to produce competitive pixel-level masks and depth maps; if the grid is too coarse at native resolution, the method's high scores would depend on test-time upscaling rather than the model itself.","fun_headline_variants_meta":{"raw":{"variants":["Standard LLM sees pixels: dense prediction without extra decoders","DenseMLLM: one pass to masks and depth, no decoder added","Vision tokens alone decode segmentation and depth in MLLM","No extra decoder: standard MLLM does dense prediction natively","Multi-task masks and depth from a vanilla MLLM, no decoder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2690,"prompt_tokens":693,"completion_tokens":1997,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":1919}},"tokens_in":437,"tokens_out":1997,"duration_ms":11457,"temperature":1.0,"reasoning_tokens":1919,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:17:40.045645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the model a synthetic image of alternating 1-pixel vertical stripes on a plain background and ask for segmentation at the model's native input resolution; if the output map cannot reproduce the stripes even after upsampling — while a human or a full-resolution model can — the token grid is the binding resolution limit, and the 'intrinsic dense predictor' claim would need to be retracted to 'upscaled dense predictor.'","supporting_citations":[],"review_version":1}