{"id":"b973be95-38b8-44f9-9516-c289590ad37d","arxiv_id":"2506.17901","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MMGrounded-PostAlign trains MLLMs to produce a grounded object token or a rejection token plus selective rationales, improving hallucination and VQA benchmarks.","lead":"MMGrounded-PostAlign adds a grounding decoder and a rejection token to multimodal LLMs, training them to anchor answers in image regions and to say 'absent' instead of hallucinating. The paper reports gains on five benchmarks, but leaves training-data and inference details unspecified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing gap is the undisclosed training data: without source, negative-sample construction, and SIMPLE/COMPLEX labeling, the POPE/HaloQuest gains in Table 2 could reflect train/eval overlap rather than the grounding mechanism.","rationale":"The reader's weakest_assumption and my concern coincide. The paper does contain supportive internal evidence: the component ablations in Tables 1 through 3 are directionally consistent, and the design follows prior work on negative data and grounded decoding. But that evidence cannot rescue the central claim while the training distribution is unspecified. Since the same missing disclosure is already the basis of the reader's CONDITIONAL verdict, my stress pass does not move the verdict; it sharpens the required condition: a disjoint-image evaluation plus full data disclosure.","tokens_in":13403,"tokens_out":6575,"duration_ms":79420,"concrete_test":"Disclose the full training recipe, including datasets and image IDs, the negative-sampling rule, the SIMPLE/COMPLEX labeling procedure, the loss weights lambda1/lambda2, and the benchmark answer parser. Then rerun the POPE and HaloQuest evaluations on a strict subset of evaluation images with zero intersection with training images. If the gains persist after removing all overlapping images, the grounding mechanism is supported; if they drop to near baseline, the reported improvements reflect contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1's Training Data Formulation is the only description of the data: 'a diverse multimodal training dataset' with / labels and negative samples. It does not state which image corpus was used, how absent referents were sampled, how queries were labeled simple/complex, how many negatives were included, or how structured outputs are parsed into benchmark answers. This is load-bearing because the headline results—POPE adversarial 78.2 to 82.3 at 7B and 79.2 to 85.6 at 13B in Table 2, and HaloQuest False Premise 2.0 to 33.2 with in Table 1—depend directly on the model learning when to reject an absent object. POPE is built on MSCOCO images, and the paper also evaluates on RefCOCO, whose images come from COCO; without a train/eval disjointness statement, the gains could be inflated by image or template overlap. No code, data, error bars, or inference protocol are provided, so the central claim that grounding suppresses hallucination cannot currently be separated from contamination. This is not an internal inconsistency in the equations; it is a missing external precondition for evaluating every reported improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMGrounded-PostAlign, a post-alignment framework for multimodal large language models that couples a visual grounding module (with and tokens) to a textual grounding mechanism with selective reasoning (/). The method is trained with LoRA on top of LLaVA-1.5-7B/13B and a frozen ViT-H SAM encoder, with losses for negative rejection, selective reasoning, grounding, and language modeling. The reported experiments cover hallucination benchmarks (HaloQuest, POPE), general VQA/reasoning benchmarks (VQAv2, MMBench, MME), and grounding benchmarks (RefCOCO, ReasonSeg), claiming improved hallucination suppression while preserving general capabilities. The main evidence is Table 2 (e.g., POPE adversarial 78.2→82.3 at 7B and 79.2→85.6 at 13B) and Table 1's large HaloQuest gains when the mechanism is enabled.","tokens_in":13637,"tokens_out":3518,"duration_ms":40804,"significance":"If the reported improvements are genuine and reproducible, the framework is a useful contribution: it makes explicit visual grounding and negative rejection into lightweight post-alignment training for MLLMs, and the selective-reasoning ablation in Table 3 is a sensible design comparison. The paper also provides a transparent list of loss terms in Section 3.4, which is a strength. However, the central evaluation currently depends on an undisclosed training dataset and an unspecified inference/parsing protocol, so the headline claims cannot be separated from train/evaluation overlap or formatting artifacts. The contribution is therefore potentially significant but not yet verifiable from the manuscript as written.","major_comments":[{"comment":"The training data is described only as 'a diverse multimodal training dataset' with / labels and negative samples. The paper does not state the image corpus, the number of training instances, the procedure for constructing negative referents (how an absent object is chosen, whether captions/objects are sampled, what ratio of negatives is used), the rule for assigning SIMPLE/COMPLEX labels, or whether the evaluation benchmarks' test images were excluded from training. Because POPE and RefCOCO both use MSCOCO images, and HaloQuest contains false-premise questions, the large gains in Tables 1 and 2 (e.g., HaloQuest False Premise 9.9→33.2 with ; POPE adversarial 78.2→82.3 at 7B) could be inflated by image overlap or template overlap. Please disclose the training data source, the negative-sample construction procedure, and an explicit train/evaluation disjointness statement.","section":"Section 4.1"},{"comment":"The baseline definition is inconsistent across tables. In Table 2, 'Baseline' is defined as the proposed framework with the visual grounding module removed while retaining selective reasoning, but Table 1's first row is described as 'the baseline where the visual grounding module is entirely removed' without stating whether the textual grounding/selective reasoning components are present. This makes it impossible to attribute the improvements to the visual grounding module, the negative rejection mechanism, or the selective reasoning mechanism. Please use one coherent baseline and clearly state which components each row includes.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The inference protocol is not specified: the paper does not explain how the structured output tokens (, , , , rationale, final answer) are parsed into benchmark answers for POPE's yes/no questions, VQAv2's short answers, HaloQuest's free-form answers, or MMBench/MME's multiple-choice prompts. Without this detail, the reported improvements could reflect a better match between the generated format and the benchmark's answer parser rather than improved grounding. Please describe the exact answer extraction and scoring procedure used for each benchmark.","section":"Sections 3.1 and 4.2"},{"comment":"No error bars or significance tests are reported for any of the main results. Several comparisons in Table 2 are within about one point (e.g., VQAv2 79.1→79.9 at 13B, MME 1520.3→1517.4), so without multiple seeds or a statistical test it is unclear which differences are meaningful. Please add variance information or at least state whether results are averaged over multiple runs.","section":"Tables 1–3"},{"comment":"The selective reasoning mechanism depends on SIMPLE/COMPLEX training labels, but the labeling rule is never defined beyond two examples. The paper also does not state how y_reason in Eq. (4) is obtained (manual annotation, automatic heuristic, or model-generated) or how the proportion of simple versus complex queries is chosen. This is load-bearing for Table 3's conclusion that selective reasoning is optimal, because the method's benefit could simply reflect the specific labeling distribution used. Please specify the labeling criteria and distribution.","section":"Section 3.3 and Eq. (4)"}],"minor_comments":[{"comment":"Implementation details omit several hyperparameters needed for reproducibility: the LoRA rank and alpha, the values of λ1 and λ2 in Eq. (5), the total training steps, and the number of negative samples per batch.","section":"Section 4.1"},{"comment":"The MME benchmark is cited through an unrelated survey reference ('A survey of multimodel large language models'); the correct MME paper should be cited.","section":"Reference list, entry [57]"},{"comment":"The attention analysis is described only qualitatively. The paper reports an 89.2% hallucinated-token overlap after removing the image, but does not define how hallucinated tokens are identified in this analysis or how the 'without image' condition is constructed (e.g., blank image, masked image, or text-only prompt). Please clarify the protocol.","section":"Section 4.3 and Figure 4"},{"comment":"The phrase 'zero-shot grounding capabilities' is misleading because the model is fine-tuned on a multimodal grounding training set; 'unseen-category grounding' or 'grounding without task-specific fine-tuning' would be more precise.","section":"Section 4.2, Finding 5"},{"comment":"There are typos such as 'visiual understanding' and the duplicated sentence 'In this section, we introduce MMGrounded-PostAlign...' at the start of Section 3. Please proofread.","section":"Abstract and Section 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the ablations are thoughtfully designed, but the paper currently lacks the data-disclosure and evaluation-detail needed for the claims to be checked. As an editor, I would ask for the training-data description and train/eval disjointness statement, the exact inference parsing, and reproducibility details before considering it for publication. The absence of any code/data release is a particular concern for a benchmarks-driven paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is an integrative fine-tuning method for MLLMs that combines a SAM-based grounding head, a negative rejection token for absent objects, and query-complexity-dependent rationale generation. It reports large gains on POPE and HaloQuest, with the most striking jump on HaloQuest False Premise (2.0 to 33.2 with ). The ablations are clean, and the paper does a good job comparing against BTL-style grounding strategies and against pre- and inter-reasoning. The token-analysis figure (Figure 3) is a nice supporting piece showing that hallucinations persist without image input, which backs the linguistic-prior story.\n\nThe soft spot is the one the stress-test note flags: Section 4.1 describes the training data in one vague sentence. We don't learn the image corpus, how negative samples are built, how queries are labeled SIMPLE/COMPLEX, or how many negatives are used. POPE and RefCOCO both use MSCOCO images; without a disjointness statement, the headline gains could reflect train/eval overlap. This is load-bearing because the whole mechanism is about learning to say for absent objects. There are also no error bars and no code/data release. The good news is the internal logic is sound: the components are known but the combination is new enough to be interesting.\n\nThe paper deserves a serious referee, but the referee should demand the missing details: full training data description, a clear disjointness statement, per-seed variance, and code or at least a detailed protocol. This is not a desk reject; it's a conditional accept after major revision. It will be useful to the hallucination-mitigation community, especially people working on grounding-based approaches. I would not cite it in its current form, but I would read the revised version.","headline":"Useful integrative method for grounding-based hallucination control, but the undisclosed training data makes the headline gains unverifiable as-is.","tokens_in":14202,"tokens_out":2047,"would_cite":false,"duration_ms":23411,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMGrounded-PostAlign claims that forcing an MLLM to ground answers in a visual localization and a selective rationale reduces hallucination, lifting POPE accuracy from 83.3 to 86.6 at 7B scale.","keywords":["multimodal large language models","hallucination mitigation","visual grounding","textual grounding","negative rejection","selective reasoning","post-training alignment","fine-grained visual understanding"],"falsifier":"Retrain the identical pipeline with all POPE, HaloQuest, and MME question-image pairs removed from the training mixture; if the POPE and HaloQuest gains shrink to near baseline, the hallucination suppression comes from benchmark contamination rather than the grounding mechanism. A second check is to remove the branch alone while keeping every other component and measuring HaloQuest's False Premise category, since the paper attributes most of that category's improvement to rejection.","tokens_in":13128,"feed_emoji":"🎯","tokens_out":6211,"duration_ms":63769,"temperature":0.7,"pith_summary":"The paper is trying to establish that a large part of multimodal hallucination is driven by linguistic priors, and that a post-training alignment layer can correct it. The proposed framework, MMGrounded-PostAlign, makes the model produce a visual grounding token that localizes the referred object, a special rejection token when no such object exists, and a textual rationale only when the query needs one. On hallucination benchmarks this raises POPE accuracy from 83.3 to 86.6 at 7B scale and from 85.4 to 88.9 at 13B scale, while VQAv2 and other reasoning scores stay flat or rise. If the claim holds, hallucination suppression does not require a new architecture or losing general reasoning ability; it can be added on top of an existing model with inexpensive LoRA fine-tuning.","feed_headline":"Forcing visual grounding suppresses MLLM hallucinations","feed_subtitle":"A rejection token for absent objects plus selective rationales lifts POPE to 86.6 without hurting VQAv2.","key_machinery":"The load-bearing machinery is the grounding-token interface: the MLLM is trained to emit for the referred object, and its last-layer embedding is fed through an MLP into a multi-task decoder built on a frozen SAM encoder to produce a mask and bounding box. When no referent exists, replaces and bypasses decoding with an empty mask, under a dedicated negative-rejection loss; the query-complexity tokens / similarly route the textual branch under a selective-reasoning loss. These four structured tokens turn grounding into a differentiable training signal that is then added, via LoRA, to the standard language modeling loss. The framework does most of its claimed work through these token-level routing decisions, not through a new pretraining objective.","core_discovery":"The central claim is that anchoring the answer-generation process in explicit multimodal grounding corrects an MLLM's over-reliance on linguistic priors. During decoding, the model first outputs a grounding token; its embedding is passed to a segmentation and bounding box decoder to locate the referent, and when the referent is absent the model must instead emit , which is assigned an empty mask and box. A parallel textual-grounding branch decides, via a learned / classification, whether to generate a rationale before the final answer. The paper reports that these two mechanisms together suppress object hallucinations on POPE and HaloQuest, sharpen attention on image regions, and preserve or slightly improve performance on MME, MMBench, and VQAv2.","pith_inferences":["Beyond the paper: the mechanism is trained on absent referents; a natural test is whether it transfers to attribute or spatial hallucinations, such as a present object with the wrong color, which the paper's loss does not explicitly target.","Beyond the paper: the measured 89.2% overlap of hallucinated tokens between image-present and image-absent decoding could be used as a cheap synthetic-hallucination probe before running full benchmarks.","Beyond the paper: selective reasoning suggests a capacity and compute tradeoff curve; measuring answer quality against rationale length across difficulty buckets would tell whether the learned / router is well calibrated."],"forward_implications":["Hallucination suppression can be layered onto existing MLLMs via LoRA, without replacing the backbone or sacrificing general VQA capability.","When a query names an object that is not in the image, models trained with learn to say that the object does not exist rather than predicting a co-occurring object.","Generating a rationale is not always beneficial; routing simple queries straight to the answer avoids overthinking and keeps the whole process in a single inference pass.","The same grounding interface yields competitive referring-expression comprehension and segmentation, suggesting that grounding can be a byproduct of hallucination correction rather than a separate training goal."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5-7B and LLaVA-1.5-13B backbones on which the framework is built and on which all reported numbers depend.","marker":"[5]"},{"why":"Provides the frozen SAM visual encoder used by the multi-task decoder for masks and boxes.","marker":"[51]"},{"why":"Gives the LoRA mechanism that keeps the MLLM fine-tuning parameter-efficient.","marker":"[50]"},{"why":"Is the HaloQuest benchmark used for the main hallucination ablation in Table 1.","marker":"[53]"},{"why":"Is the POPE benchmark where the headline hallucination-reduction gains are reported in Table 2.","marker":"[54]"},{"why":"Is the VQAv2 benchmark used to show that general answering ability is preserved.","marker":"[55]"},{"why":"Motivates the textual-grounding rationale branch for complex queries.","marker":"[49]"},{"why":"Provides the ReasonSeg benchmark and the LISA baseline used to evaluate the selective-reasoning mechanism in Table 3.","marker":"[28]"}],"fun_headline_variants":["PostAlign: grounded rejections cut MLLM hallucinations","Reject absent objects, ground answers: MLLM fix","Grounding lens corrects MLLM hallucinations","MLLM answers grounded: reject nonexistent objects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training data recipe — how absent referents are turned into negatives and how queries are labeled or — is not disclosed, so the reported gains assume these labels and negatives are created correctly and do not overlap the test benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["PostAlign: grounded rejections cut MLLM hallucinations","Reject absent objects, ground answers: MLLM fix","Grounding lens corrects MLLM hallucinations","MLLM answers grounded: reject nonexistent objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2813,"prompt_tokens":908,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1840}},"tokens_in":524,"tokens_out":1905,"duration_ms":16887,"temperature":1.0,"reasoning_tokens":1840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:03.240297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the identical pipeline with all POPE, HaloQuest, and MME question-image pairs removed from the training mixture; if the POPE and HaloQuest gains shrink to near baseline, the hallucination suppression comes from benchmark contamination rather than the grounding mechanism. A second check is to remove the branch alone while keeping every other component and measuring HaloQuest's False Premise category, since the paper attributes most of that category's improvement to rejection.","supporting_citations":[{"cited_title":"Haloquest: A visual hallucination dataset for advancing multimodal reasoning","cited_arxiv_id":null,"evidence_quote":"Is the HaloQuest benchmark used for the main hallucination ablation in Table 1."}],"review_version":1}