{"id":"986feccf-7c3f-4a19-a963-8fba3b904f74","arxiv_id":"2507.06272","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LIRA improves referring segmentation and reduces hallucination in multimodal LLMs by fusing semantic and pixel features and interleaving local image regions with text descriptions.","lead":"LIRA combines high-level semantic features with pixel-level segmentation features to make multimodal models segment objects more accurately and hallucinate less when describing regions. It also introduces AttrEval, a benchmark for probing whether a model's segmentation token encodes object attributes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ILVC's central benefit is untested under its actual inference condition: Appendix F shows ILVC-at-inference lowers COCO mIoU by 1.7, so the claimed segmentation+comprehension complementarity is not established for the reported benchmarks.","rationale":"The paper is honest enough to include Appendix F, but that appendix exposes the weakest point in the argument. ILVC's benefit is claimed via training-time coupling; at inference the local regions are the model's own imperfect masks. The only controlled test of that regime shows a 1.7-point mIoU drop when ILVC is actually invoked. Because the paper also trains two prompts that let the user suppress ILVC, every headline number is ambiguous unless the prompt choice is disclosed. This does not prove the method is wrong; SEFE may carry most of the segmentation gain, and ILVC may still help hallucination. But the current evidence does not establish the advertised complementarity. The reader's CONDITIONAL verdict is appropriate; the concern is addressable with one ablation, so I would not move to REJECT.","tokens_in":21126,"tokens_out":6752,"duration_ms":74526,"concrete_test":"Run LIRA-8B on RefCOCO val/test and on the hallucination suite (ChairS, POPE, TinyLVLM) under three inference conditions: (a) ILVC disabled by prompt, (b) ILVC enabled with predicted-mask crops, (c) ILVC enabled with GT-mask oracle crops. Also report which prompt was used for every number in Tables 1, 5, and 6. If (b) does not beat (a) on hallucination, or if (b) loses segmentation accuracy on RefCOCO as it does on COCO (58.9 vs 60.6), the ILVC complementarity claim fails under its actual deployment condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core complementarity claim—that ILVC couples local region features with text to reduce hallucination while preserving segmentation—rests on a train/inference mismatch. Section 3.3 trains local crops from ground-truth masks but at inference crops them from model-predicted masks. Appendix F is the only direct evidence under this mismatch and it cuts against the claim: on COCO instance segmentation, the two-prompt model gets 58.9 mIoU with ILVC at inference versus 60.6 without ILVC, while a no-ILVC baseline is 60.0. Thus in a multi-object setting ILVC at inference actively lowers segmentation accuracy. The paper never reports an analogous ILVC-on/off comparison for the RefCOCO results in Table 1 or for the hallucination ablations in Table 6, and it does not state whether the headline numbers use the ILVC or no-ILVC prompt. Without that, the observed gains could be driven by SEFE and by selecting the no-ILVC prompt at inference, with ILVC acting only as a training regularizer. This is a load-bearing gap because the abstract credits ILVC for reduced hallucination and the introduction claims simultaneous strong performance in both tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIRA, a framework that augments InternVL2-based LMMs with two components: SEFE, which fuses a semantic encoder and an OMG-Seg pixel encoder for segmentation, and ILVC, which interleaves local image-region features with textual descriptions using <seg> tokens. The authors report state-of-the-art referring-expression-segmentation results on RefCOCO/RefCOCO+/RefCOCOg, improved grounded conversation generation CIDEr, and claim that co-training with segmentation data causes only a 0.2% drop in comprehension accuracy compared with OMG-LLaVA's 14.3% drop. They also introduce an AttrEval dataset to support the claim that the <seg> token's logits encode object attributes such as color and position.","tokens_in":21458,"tokens_out":4708,"duration_ms":46846,"significance":"If the central claims hold, LIRA would be a useful step toward unified LMMs that segment and comprehend without catastrophic forgetting. The paper's strengths are its broad evaluation (eight comprehension benchmarks, three referring-segmentation splits, gRefCOCO zero-shot, GCG, and three hallucination metrics), its architecture-level ablations of SEFE (Table 4) and ILVC (Table 6), and its candid limitation paragraph acknowledging the low AttrEval accuracy. However, the signature claim that ILVC reduces hallucination while preserving segmentation is currently supported mainly by training-time ablations; the only inference-time controlled experiment (Appendix F) does not support the claim. The introduced AttrEval benchmark is also built from the same RefCOCO data used in training, so the <seg>-token semantic-correlation evidence is vulnerable to circularity. These issues must be resolved before the state-of-the-art claim can be accepted.","major_comments":[{"comment":"Section 3.3 trains ILVC with local crops extracted from ground-truth masks but, at inference, crops are taken from the model's own predicted masks. Appendix F, the only controlled comparison under this mismatch, reports that using ILVC at inference lowers COCO instance-segmentation mIoU from 60.6 to 58.9, while the no-ILVC baseline is 60.0; this does not support the claim that ILVC preserves segmentation while reducing hallucination. The paper does not state whether the headline numbers in Tables 1, 2, and 6 use the ILVC prompt or the no-ILVC prompt, nor does it provide an inference-time ILVC on/off ablation on RefCOCO or on the hallucination benchmarks. Please report those numbers and specify the inference prompt; without them, the central complementarity claim is unverified.","section":"Sec. 3.3, Appendix F"},{"comment":"The evidence for the paper's claimed positive correlation between segmentation accuracy and <seg>-token semantics rests on AttrEval (Section 5 and Appendix D). AttrEval is constructed from RefCOCO, which is part of the instruction-tuning data (Table 8), so the measured correlation may reflect training-data memorization rather than a general property. Please evaluate the same logits-to-attribute mapping on a held-out or external benchmark, or at least verify that no AttrEval instance overlaps with the training set.","section":"Sec. 5, Appendix D"},{"comment":"Tables 1 and 2 report differences of 0.5-2 points as state-of-the-art without error bars or significance tests; on two comprehension benchmarks, MMB-en and MMB-cn, LIRA-8B is actually below the comprehension-only InternVL2-8B baseline (81.1 vs 81.7 and 80.5 vs 81.2). The abstract and Section 4.3 claim state-of-the-art performance in both segmentation and comprehension, which is stronger than the data support. Provide variance estimates and qualify the comparative claims, and note that some baselines (e.g., PSALM with 100 queries, GLaMM with GranD pretraining) use different inference or training budgets.","section":"Tables 1-2, Sec. 4.3"}],"minor_comments":[{"comment":"In Table 8, 'CharQA' should be 'ChartQA' and 'AttriEval' should be 'AttrEval' for consistency; Section 4.3 and Appendix D use both spellings.","section":"Table 8, Appendix D"},{"comment":"The description of the two prompts that control ILVC in Appendix F is omitted; please include them in the implementation details for reproducibility.","section":"Appendix F, Sec. 4.1"},{"comment":"Figure 5 and Appendix C show qualitative hallucination comparisons but no quantitative captioning metric; consider adding CHAIR or a similar metric to support the hallucination-reduction claim.","section":"Fig. 5, Appendix C"},{"comment":"The 'Limitation' paragraph in Section 5 reports Acc1=25.7% on AttrEval; this is low and should be discussed in relation to the claim that the <seg> token logits encode object attributes.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The ILVC inference-time mismatch is the key risk; if the authors can provide the missing ablations (inference prompt choice and ILVC on/off on RefCOCO and hallucination benchmarks), the paper may be acceptable. The AttrEval circularity can be addressed by a disjointness check or external evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a real architecture paper with two separable contributions. SEFE—cross-attention fusion of a semantic encoder and a pixel encoder—is cleanly motivated and its ablation is convincing: +5.7/+3.8 points on understanding/segmentation with the 1.8B backbone, and similar gains with 7B. ILVC is the more interesting but less settled idea: interleave local crops (from masks) and text into the LLM sequence so the model learns region-level descriptions. The <seg>-token logits analysis is a genuinely fresh observation, though it is illustrative, not a mechanism proof. AttrEval is a reasonable first probe for attribute-level hallucination.\n\nSoft spots, in order of importance. First, ILVC has a train/inference mismatch: trained on crops from ground-truth masks, used at inference on crops from predicted masks. The paper's own Appendix F shows that at inference on COCO instance segmentation, ILVC lowers mIoU (58.9 vs 60.6 without ILVC), with the only upside being a 0.6 training-regularizer effect when ILVC is turned off at inference. The paper never reports an ILVC-on/off comparison for the RefCOCO findings or for the hallucination ablations, and it does not state whether the headline numbers use the ILVC prompt. That is a real gap, and the stress-test note is right to call it load-bearing for the complementarity claim. It is not fatal to the paper: SEFE's segmentation gains stand independently, and ILVC's hallucination reduction may still hold as a training-time effect. But the authors need to show it.\n\nSecond, the empirical reporting is a little loose: no error bars, no significance tests, and on MMB-en/MMB-cn the base InternVL2-8B beats LIRA-8B—so \"SOTA\" depends on the benchmark list. Third, AttrEval is built from RefCOCO, the same data LIRA trains on, so the <seg>-semantics claim is partly self-referential. That is fine for a preliminary probe, but it should not be presented as an external validation.\n\nBottom line: this deserves a serious referee and probably a conditional accept after the ILVC-at-inference ablation is added and the benchmark claims are tightened. Worth a reading-group slot if you care about segmentation-capable LMMs.","headline":"A solid LMM-segmentation architecture paper with a convincing SEFE component and a promising but under-tested ILVC idea; the missing ILVC-at-inference ablation is the one thing that should decide revision.","tokens_in":21944,"tokens_out":4310,"would_cite":true,"duration_ms":48233,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LIRA claims that segmentation and comprehension can be complementary in large multimodal models, reporting state-of-the-art on both and reduced hallucination.","keywords":["large multimodal models","referring expression segmentation","hallucination reduction","semantic fusion","local region coupling","grounded conversation generation","attribute inference","segmentation token logits"],"falsifier":"Measure LIRA on RefCOCOg with ILVC fed with predicted masks versus oracle ground-truth masks. If the predicted-mask condition is no better than removing ILVC entirely, the distribution shift is the load-bearing failure; if oracle masks help substantially, the mechanism works only as well as the upfront segmentation does.","tokens_in":20946,"feed_emoji":"🖼️","tokens_out":7321,"duration_ms":74819,"temperature":0.7,"pith_summary":"This paper claims that in large multimodal models, segmentation and comprehension are complementary rather than competing, and that both improve when pixel and semantic features are fused and local regions are tied to text. It proposes LIRA, with a Semantic-Enhanced Feature Extractor (SEFE) that fuses semantic and pixel-level features for precise segmentation, and an Interleaved Local Visual Coupling (ILVC) that crops the segmented region, re-encodes it, and feeds it back into the LLM so descriptions are grounded in visible evidence. The 8B model achieves state-of-the-art on referring segmentation (78.4% on RefCOCOg val) and raised grounded-conversation CIDEr to 38.4, while adding segmentation data to training costs only 0.2% comprehension versus a 14.3% drop for the prior method. The paper also introduces the AttrEval dataset to measure how well a model can infer object attributes, and shows the <seg> token's logits encode attributes like position and color.","feed_headline":"Segmentation and comprehension rise together in LIRA","feed_subtitle":"An 8B model tops RefCOCOg at 78.4% while comprehension drops only 0.2% when segmentation data is added.","key_machinery":"Two mechanisms carry the argument. SEFE (Semantic-Enhanced Feature Extractor) concatenates features from a semantic encoder and a pixel encoder, with a multi-head cross-attention fusion and residual connection, aligning segmentation priors to the LLM's input space. ILVC (Interleaved Local Visual Coupling) builds interleaved sequences of global image, instruction, <seg> token, local region feature, and region description, with the local region cropped from the mask and re-encoded at 448x448; this trains the LLM to condition its output on concrete visual patches. The paper also uses the logits of the <seg> token as an interpretable attribute signal and builds the AttrEval dataset to quantify attribute-inference capability.","core_discovery":"The central claim is that segmentation and comprehension are not a zero-sum trade-off in large multimodal models, provided local visual features are explicitly coupled to text. LIRA trains with a mask-region-text triplet: the region is extracted from the ground-truth mask, re-encoded, and re-entered into the LLM before it generates the region's description, so the model learns to ground words in visible patches. With this, LIRA reports state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg, and the co-training ablation shows only a 0.2% drop in comprehension accuracy when segmentation data is added, in contrast to the 14.3% drop reported for the prior best model. Additionally, the paper introduces AttrEval and finds that the highest-logit tokens in the <seg> embedding correspond to correct object attributes, tying segmentation precision to the semantic content the model already carries.","pith_inferences":["The same <seg>-logit readout used in AttrEval could be turned into an uncertainty signal: when the top attribute logit is low, the model could refrain from committing to a mask or ask for clarification.","The train-GT-mask / infer-predicted-mask gap is testable directly: evaluating LIRA on RefCOCOg with oracle masks versus predicted masks would bound how much of ILVC's benefit survives imperfect segmentation.","The two-prompt with/without-ILVC control on COCO instance segmentation could become a standard robustness check for any mask-conditioned LMM in multi-object scenes.","Re-feeding cropped regions into the LLM is a generic mechanism that could extend to detection, referring grounding, or editing tasks that need region-conditioned text."],"forward_implications":["Adding segmentation data to instruction tuning does not have to degrade comprehension; LIRA's drop is 0.2% on five comprehension benchmarks, against the prior model's 14.3%.","ILVC lowers object hallucination on ChairS, POPE, and TinyLVLM across both 1.8B and 7B backbones.","Grounded conversation generation improves (CIDEr 38.4 on validation), so the mechanism transfers to mask-linked captioning.","The <seg> token's top-logit tokens encode object position and color, making part of the segmentation decision interpretable.","LIRA improves zero-shot generalized referring segmentation on gRefCOCO without being trained on that benchmark."],"supporting_citations":[{"why":"Introduces the embedding-as-mask paradigm and the reasoning-segmentation task that LIRA inherits and compares against.","marker":"[19]"},{"why":"The closest prior model; provides the co-training comprehension drop (14.3%) that LIRA claims to reduce to 0.2%.","marker":"[71]"},{"why":"Supplies the GranDf dataset used for ILVC training in grounded conversation generation.","marker":"[43]"},{"why":"RefCOCO referring-expression dataset used both as training data and evaluation benchmark.","marker":"[68]"},{"why":"RefCOCOg dataset used for evaluating referring segmentation and for extracting ground-truth masks during ILVC training.","marker":"[39]"},{"why":"Provides the pixel encoder and pixel decoder that SEFE and mask decoding are built on.","marker":"[22]"},{"why":"Provides the semantic encoder and LLM backbone that LIRA is built on.","marker":"[8]"},{"why":"The prior grounded conversation generation baseline whose data scale LIRA contrasts with.","marker":"[40]"},{"why":"The baseline whose COCO instance-segmentation setup is reused in the two-prompt error-accumulation control.","marker":"[72]"}],"fun_headline_variants":["LIRA: Segmentation and comprehension no longer a trade-off","LIRA: local region features keep segmentation and comprehension in sync","LIRA solves both segmentation accuracy and hallucinated comprehension","LIRA: co-training segmentation costs just 0.2% comprehension vs 14.3%","LIRA: local interleaved region assistance lifts segmentation and comprehension"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"ILVC is trained using regions cropped from ground-truth masks, but at inference the regions are cropped from the model's own predicted masks, and those imprecise crops feed the LLM features it never saw during training.","fun_headline_variants_meta":{"raw":{"variants":["LIRA: Segmentation and comprehension no longer a trade-off","LIRA: local region features keep segmentation and comprehension in sync","LIRA solves both segmentation accuracy and hallucinated comprehension","LIRA: co-training segmentation costs just 0.2% comprehension vs 14.3%","LIRA: local interleaved region assistance lifts segmentation and comprehension"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3236,"prompt_tokens":945,"completion_tokens":2291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2199}},"tokens_in":561,"tokens_out":2291,"duration_ms":16510,"temperature":1.0,"reasoning_tokens":2199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:18:39.690419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure LIRA on RefCOCOg with ILVC fed with predicted masks versus oracle ground-truth masks. If the predicted-mask condition is no better than removing ILVC entirely, the distribution shift is the load-bearing failure; if oracle masks help substantially, the mechanism works only as well as the upfront segmentation does.","supporting_citations":[{"cited_title":"Lisa: Reasoning segmentation via large language model","cited_arxiv_id":null,"evidence_quote":"Introduces the embedding-as-mask paradigm and the reasoning-segmentation task that LIRA inherits and compares against."},{"cited_title":"Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding","cited_arxiv_id":null,"evidence_quote":"The closest prior model; provides the co-training comprehension drop (14.3%) that LIRA claims to reduce to 0.2%."},{"cited_title":"10 Glamm: Pixel grounding large multimodal model","cited_arxiv_id":null,"evidence_quote":"Supplies the GranDf dataset used for ILVC training in grounded conversation generation."},{"cited_title":"Modeling context in referring expres- sions","cited_arxiv_id":null,"evidence_quote":"RefCOCO referring-expression dataset used both as training data and evaluation benchmark."},{"cited_title":"Psalm: Pixelwise segmentation with large multi-modal model","cited_arxiv_id":null,"evidence_quote":"The baseline whose COCO instance-segmentation setup is reused in the two-prompt error-accumulation control."}],"review_version":1}