REVIEW 3 major objections 4 minor 1 cited by
MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Multi-image hallucination is a distinct failure mode with measurable drivers, and an attention-rebalancing mechanism reduces it.
desk verdict A plausible first benchmark for multi-image hallucination, but the abstract-only presentation leaves the load-bearing correlation and the method's effectiveness unverified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pieces are the MIHBench task suite and the Dynamic Attention Balancing mechanism. MIHBench specifies three tasks with controlled negative samples and same-object image ratios; these design choices are what make hallucination measurable and what the reported correlations depend on. Dynamic Attention Balancing is the proposed intervention: it reweights attention between the input images (inter-image attention) while keeping the total visual attention share constant, so the model still attends to all images but distributes its cross-image focus differently.
What would settle it
A reader could test whether the reported factors hold up on a benchmark constructed with a different sampling strategy: for instance, if randomizing the position of negative samples eliminates the effect on identity consistency hallucination, or if the correlation between single-image and multi-image tendencies disappears when the task distribution changes, the causal role claimed for those factors would be weakened. Similarly, if Dynamic Attention Balancing shows no gain on a held-out multi-image task suite built from naturally occurring image sets, the generalizability claim would be falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that multi-image object hallucination is a distinct, structured failure mode with its own drivers, and that it can be mitigated by adjusting inter-image attention. The three MIHBench tasks operationalize hallucination as (1) claiming an object exists when it does not, (2) giving an incorrect count of objects across images, and (3) failing to keep object identity consistent across views. The evaluation identifies a progressive relationship between image count and hallucination likelihood, a strong correlation between single-image and multi-image hallucination tendencies, and the influence of same-object image ratio and negative-sample position on identity consistency errors. The proposed Dynamic Attention Balancing mechanism adjusts inter-image attention distributions while preserving the overall visual attention proportion, and experiments across multiple state-of-the-art MLLMs show reduced hallucination occurrence and enhanced semantic integration and reasoning stability.
Load-bearing premise
The results depend on the assumption that the benchmark's task definitions, negative samples, and same-object image ratios reflect real multi-image hallucination behavior rather than quirks of how the benchmark was built.
Editorial extensions
If this is right
- Hallucination research should treat multi-image settings as separate from single-image settings, because error rates scale with image count and with single-image tendencies.
- Benchmark designers can use the three MIHBench tasks as a standard way to measure object existence, count, and identity consistency errors.
- MLLM developers can apply Dynamic Attention Balancing as a lightweight intervention to reduce multi-image hallucinations.
- Identity consistency hallucination can be controlled during evaluation by arranging same-object image ratios and negative-sample positions.
- The reported correlation between single-image and multi-image tendencies suggests that single-image hallucination benchmarks may help predict which models will struggle with multiple images.
Reading between the lines
- The paper does not claim, but its results imply, that single-image hallucination benchmarks could serve as a cheap screening tool before full multi-image evaluation.
- A testable extension of the positional finding would be to check whether simply reordering the same set of images changes a model's answer accuracy in real multi-image applications.
- Because the benchmark controls negative sample placement, it cannot yet tell us how hallucinations behave in open-ended multi-image conversations where no explicit negative is present; extending the tasks to unconstrained prompts would test that boundary.
- If Dynamic Attention Balancing preserves overall visual attention proportion, it may combine with other attention-based mitigation methods, though the paper does not test such combinations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces MIHBench, a benchmark for evaluating object-related hallucinations in multimodal large language models (MLLMs) when processing multiple images. The abstract describes three core tasks: multi-image object existence hallucination, multi-image object count hallucination, and object identity consistency hallucination. It further claims to identify key factors associated with multi-image hallucinations, including a progressive relationship with the number of images, a strong correlation between single-image and multi-image hallucination tendencies, and the influence of same-object image ratios and negative sample positions. The paper then proposes a Dynamic Attention Balancing mechanism and claims that experiments across multiple state-of-the-art MLLMs demonstrate that it reduces hallucination occurrences. However, the abstract provides no quantitative results, no benchmark construction details, no error bars, and no statistical tests, making the central claims unverifiable from the submitted material.
Significance. If the claims hold, MIHBench would address an important gap in hallucination evaluation for multi-image scenarios, and the proposed Dynamic Attention Balancing mechanism could provide a practical mitigation. The task taxonomy (existence, count, identity consistency) is a reasonable decomposition of object-related hallucinations, and the focus on multi-image settings is timely. The abstract also suggests a transferability hypothesis between single-image and multi-image hallucination behavior, which would be useful if established. However, the lack of any supporting evidence in the abstract means that the significance is conditional and cannot be assessed. The paper does not demonstrate the existence of the benchmark, the validity of the measurements, or the effectiveness of the proposed method in this submission.
major comments (3)
- [Abstract, strong correlation sentence] The abstract asserts a strong correlation between single-image hallucination tendencies and multi-image contexts without describing how the multi-image test items are assembled. If multi-image trials are constructed from the same image pools, object annotations, and prompt templates as the single-image trials, then the correlation may partly reflect shared benchmark priors (e.g., object categories that already trigger single-image hallucinations) rather than a distinct multi-image mechanism. The paper must specify the construction protocol, and ideally demonstrate that the correlation holds when multi-image trials are generated independently from single-image benchmarks or when controlling for per-image difficulty and object-category frequency. This point is load-bearing because the correlation motivates the cross-setting transferability assumption behind Dynamic Attention Balancing.
- [Abstract, progressive relationship sentence] The claimed progressive relationship between the number of image inputs and hallucination likelihood is confounded with task difficulty: adding images increases memory load, the number of objects to reason about, and the complexity of cross-image comparisons. Without an experimental design that varies image count while holding image content and question type constant, or statistically controlling for per-image difficulty, the monotonic trend is not established. The abstract should report the design or explicitly acknowledge this confound, and the full paper must include such controls to support the causal claim.
- [Abstract, experiments demonstrate sentence] The abstract provides no quantitative outcomes, no model names, no per-task results, and no statistical significance tests for the Dynamic Attention Balancing mechanism. A claim that the method 'effectively reduces hallucination occurrences' requires, at minimum, hallucination rates before and after application, error bars, and ideally a comparison with existing mitigation baselines. The abstract should include a concise summary of the main quantitative results to allow preliminary assessment of the effectiveness claim.
minor comments (4)
- [Abstract, 'first systematic study'] The abstract claims 'the first systematic study' of multi-image hallucinations but cites no prior work. Please clarify the novelty relative to existing hallucination benchmarks and multi-image reasoning evaluations, or soften the claim.
- [Abstract, Dynamic Attention Balancing] The mechanism name 'Dynamic Attention Balancing' is not defined. A one-sentence description of how inter-image attention distributions are adjusted while preserving overall visual attention would improve the readability of the abstract.
- [General] The abstract does not mention the size of the benchmark, the number of models evaluated, or the evaluation protocol. Adding these details would help place the contribution in context.
- [General] The abstract would benefit from a sentence acknowledging limitations, such as the potential restricted generalizability of the benchmark's construction choices to real-world multi-image tasks.
Circularity Check
No circular derivation is visible in the abstract; benchmark self-evaluation is standard practice, and the correlation claim is a validity concern, not a circularity.
full rationale
This review has access only to the abstract. The paper introduces MIHBench, proposes three evaluation tasks, reports empirical correlations, and proposes a mitigation method evaluated on that benchmark. No equations, fitted parameters, or prior-work citations appear in the abstract, so there is no visible loop in which an output is defined in terms of an input or a fitted value is renamed as a prediction. Evaluating a proposed method on the authors' own benchmark is normal and does not constitute circularity by itself. The reported 'strong correlation between single-image hallucination tendencies and those observed in multi-image contexts' could in principle be influenced by shared benchmark construction, but the abstract does not specify the construction, and no reduction can be exhibited from the available text. That concern belongs to external validity, not circularity. Similarly, the 'progressive relationship between the number of image inputs and the likelihood of hallucination occurrences' is an empirical observation, not a derivation from the tasks' definitions. No self-citation load-bearing argument, uniqueness theorem, or smuggled ansatz is present. Therefore, under the hard rule that circularity claims require quoted evidence of a specific reduction, the honest finding is no significant circularity with score 0.
Assumptions & free parameters
assumptions (1)
- domain assumption The three benchmark tasks (object existence, object count, and identity consistency) capture the important dimensions of multi-image hallucination in MLLMs.
Cite this review
Pith. "Pith review of MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/764XF6CJ
@misc{pith2026250800726,
author = {Pith},
title = {Pith review of: MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/764XF6CJ}},
note = {Machine review of arXiv:2508.00726}
}
read the original abstract
Despite growing interest in hallucination in Multimodal Large Language Models, existing studies primarily focus on single-image settings, leaving hallucination in multi-image scenarios largely unexplored. To address this gap, we conduct the first systematic study of hallucinations in multi-image MLLMs and propose MIHBench, a benchmark specifically tailored for evaluating object-related hallucinations across multiple images. MIHBench comprises three core tasks: Multi-Image Object Existence Hallucination, Multi-Image Object Count Hallucination, and Object Identity Consistency Hallucination, targeting semantic understanding across object existence, quantity reasoning, and cross-view identity consistency. Through extensive evaluation, we identify key factors associated with the occurrence of multi-image hallucinations, including: a progressive relationship between the number of image inputs and the likelihood of hallucination occurrences; a strong correlation between single-image hallucination tendencies and those observed in multi-image contexts; and the influence of same-object image ratios and the positional placement of negative samples within image sequences on the occurrence of object identity consistency hallucination. To address these challenges, we propose a Dynamic Attention Balancing mechanism that adjusts inter-image attention distributions while preserving the overall visual attention proportion. Experiments across multiple state-of-the-art MLLMs demonstrate that our method effectively reduces hallucination occurrences and enhances semantic integration and reasoning stability in multi-image scenarios.
Forward citations
Cited by 1 Pith paper
-
MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation
Credit assignment via LMM pairwise comparisons plus Bradley–Terry rank aggregation and potential-based shaping improves cooperative MARL under sparse rewards and dynamic agent counts.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.