REVIEW 4 major objections 5 minor 1 cited by
LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read LOCUS improves fine-grained perception in multimodal large language models by training them to locate a cropped visual cue inside the full image using an IoU-based reward, with no cropping or zooming needed at inference.
desk verdict LOCUS has a clean proxy-task idea and strong ablations, but the headline gains versus the base model are confounded by a prompt-template mismatch, and no overlap audit or seeds are reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is 'local visual cue search': a crop c = Crop(I, b*) serves as the visual query, and the model must produce the box b* in the full image. The training signal is a verifiable rule-based reward — IoU between the predicted and ground-truth box plus a format-validity term — optimized with group-relative policy optimization. This gives direct spatial feedback, which the ablations show is crucial: supervised fine-tuning on the same data barely helps, while the IoU-rewarded RL transfers to downstream fine-grained perception. Cue size ablation (70% tiny, 30% small) shows that hard, tiny targets drive the transfer, and visual cues outperform text referring expressions.
What would settle it
Check for image-level overlap (exact or near-duplicate) between COCO train2014 and the evaluation images of V*Bench, HR-Bench, and CV-Bench. If a sizable fraction of benchmark images match or closely resemble COCO training images, the +7.8 V*Bench gain could reflect training-set familiarity rather than internalized search. A complementary test: retrain LOCUS on object regions from a non-COCO source and see whether fine-grained gains persist out of distribution.
Extended reading notes
Core claim
The paper's central claim: fine-grained perception failures in multimodal LLMs are largely an evidence-selection problem, not a resolution problem. LOCUS makes local evidence search a verifiable proxy task — predict a crop's bounding box in the full image, rewarded by IoU plus format validity via group-relative policy optimization. The crop is training-only; inference stays standard image-question. Across three backbones, LOCUS reports consistent fine-grained gains (notably +7.8 on V*Bench), and attention-in-box analysis shows more attention on evidence regions, widening in later layers. The paper is explicit that it addresses retrieval, not subtle attribute interpretation failures (Limitati
Load-bearing premise
The gains are attributed to an internalized evidence-search skill, but the paper does not rule out that test benchmarks share images or distribution with the COCO train2014 images used for training; if overlap is substantial, the transfer claim is inflated.
Editorial extensions
If this is right
- If internalized evidence search is the operative mechanism, fine-grained perception can be improved without paying the cost of cropping, zooming, or tool calls at inference.
- The proxy-task design means the same framework can be applied to any backbone with an RL loop, as the paper demonstrates across three model families.
- Improved attention to ground-truth evidence regions under standard full-image inference is consistent with the model having learned to select local cues rather than merely memorizing benchmark answers.
- Answer-level instruction tuning appears insufficient for fine-grained perception; verifiable spatial supervision is what transfers, according to the SFT-vs-RL ablation.
- Because the proxy task is automatically verifiable, training data can be constructed from existing object-region annotations without new human labels.
Reading between the lines
- If the transfer is genuine, this suggests a general recipe: define a verifiable proxy that forces the model to select a specific local region, and that selection skill may transfer to diverse downstream visual question tasks. A natural test would be applying LOCUS-style training on an out-of-domain corpus (e.g., medical or satellite imagery) and measuring whether fine-grained gains persist.
- The 'visual context rot' framing implies a continuum: as resolution rises, context grows and selection becomes harder, so high-resolution inputs alone do not solve fine-grained perception. This predicts that LOCUS's relative gains should increase with input resolution; the HR-4K vs HR-8K gap partially supports this, but a controlled resolution sweep would test it directly.
- The attention-in-box analysis suggests a mechanistic account: cue-localization training reallocates later-layer attention to evidence tokens, where final answers are assembled. If so, LOCUS may be complementary to reasoning-time approaches, and combining internalized cue search with occasional explicit zooming could push fine-grained accuracy further than either alone.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LOCUS, a post-training framework that uses a verifiable proxy task—localizing a cropped visual cue within its full image via an IoU-based GRPO reward—to improve fine-grained perception in MLLMs. The visual cue is used only during training, and at inference the model receives standard full-image question inputs. Experiments across three backbones report gains on fine-grained perception, hallucination, and some reasoning benchmarks, with attention analyses suggesting stronger focus on evidence regions. The central claim is that training-time local visual cue search is internalized, improving evidence selection during ordinary full-image inference.
Significance. If the central claim holds, this is a valuable contribution: it offers a tool-free, automatically verifiable post-training objective that could strengthen fine-grained perception without test-time cropping, zooming, or multi-round search. The paper's experimental design is in several ways careful: the ablation set (RL vs SFT, visual vs text cue, object vs random crop, and cue-size distribution) is well chosen, the IoU reward is externally grounded in COCO annotations, and results are reported on three backbones. The attention analysis is a reasonable attempt to provide mechanistic evidence. However, two confounds—an evaluation-prompt mismatch and possible training/evaluation distribution overlap—prevent the current manuscript from cleanly supporting its headline claim. The strengths are real but the load-bearing evidence needs additional controls before the transfer claim can be accepted.
major comments (4)
- [Appendix B.3 / Table 2] The main Base-vs-LOCUS comparison changes two variables at once: the training method and the evaluation prompt. Appendix B.3 states that checkpoints trained with the reasoning format receive the same <think>/<answer> instruction at evaluation, while models without that format use their standard templates. Thus the base model is compared against a model that is additionally prompted to reason before answering. A chain-of-thought-style prompt alone can improve fine-grained VQA accuracy and also alter attention patterns, so the +7.8 V*Bench gain cannot be attributed solely to internalized visual cue search. The SFT and text-cue ablations (Tables 4-5) also use the think prompt, so they do not supply the missing control. Please report the base model evaluated with the identical <think>/<answer> instruction for all Table 2 metrics, and also report LOCUS without that instruction. This matched-p
- [§4.1 / Table 9 / §4.2] The training corpus is constructed from COCO train2014 object regions, while several downstream fine-grained benchmarks (V*Bench, HR-Bench-4K/8K, CV-Bench, MME-RealWorld-EN) are natural-image benchmarks from a similar distribution. The paper reports no image-level overlap or deduplication analysis. If any benchmark images are exact or near duplicates of COCO images, the reported gains—especially the large V*Bench improvement—could partly reflect training/evaluation overlap rather than internalized evidence search. Please provide an overlap analysis (e.g., exact-match hashing plus nearest-neighbor visual search) between the 99.5K training set and each benchmark, and state whether benchmark images were excluded during training-set construction. Ideally, also include a domain-shifted fine-grained benchmark to test transfer beyond the COCO distribution.
- [Table 2] Several reported deltas are within the range of likely evaluation noise (e.g., +0.0, +0.1, +0.2), yet the paper concludes that LOCUS 'consistently improves' localization-sensitive fine-grained perception. No seeds, error bars, or statistical tests are reported. The headline result rests on a single V*Bench evaluation. Please provide variance over at least three random seeds for the primary backbone on the main benchmarks (V*Bench, HR-8K, POPE), and report the evaluation-protocol variance (greedy vs sampling, vLLM version) for the small-margin claims. Without this, the 'consistent improvement' claim is not quantitatively supported for the smaller deltas.
- [§4.4 / Eq. (7)] The Attention-in-Box (AIB) analysis is used as mechanistic evidence that LOCUS improves evidence selection, but it is computed under the same mismatched prompts as the main evaluation (base without the think instruction, LOCUS with it). The AIB gap in Fig. 4 may therefore reflect the added reasoning instruction rather than the training objective. Additionally, the observation that the Text Cue variant improves downstream accuracy while lowering AIB relative to Base suggests that AIB is not an unambiguous index of evidence selection. Please provide matched-prompt AIB comparisons and a per-sample correlation between AIB change and answer correctness change.
minor comments (5)
- [Tables 2 and 4] The base HR-4K value differs between Table 2 (69.9) and Table 4 (70.1). Similarly, Table 6 reports 'Large cues (>10%)' HR-8K as 66.4 while Table 2 lists the base as 63.8 and LOCUS as 68.4. Please reconcile these numbers or explain the protocol differences.
- [Figure 1 / §1] The figure text 'No crop/no zoom/no external tool at inference' is accurate only if the evaluation prompt is truly unchanged; given Appendix B.3, the extra <think>/<answer> instruction is a prompt change. Please rephrase to 'no visual operations at inference' and clarify in the text that the same reasoning-format instruction is added to evaluation for LOCUS checkpoints.
- [Appendix B.3 / Table 8] The evaluation section says 'decoding uses greedy generation with temperature 0', but Table 8 gives a validation override of T=0.6, p=0.95, n=1. Please clarify which setting was used for the reported benchmark numbers and for the AIB analysis.
- [Limitations] The Limitations section discusses residual perception failures but does not address the two confounds raised in the major comments (prompt mismatch and data overlap). Please add an explicit discussion of these threats to validity and how the additional experiments address them.
- [References / general] Minor formatting issues: 'X AI. 2024. Grok-1.5 vision preview' and 'Y . K. Li' in the DeepSeekMath reference have inconsistent spacing; please run a final reference cleanup.
Circularity Check
No significant circularity: the proxy training objective is externally grounded and downstream gains are empirical transfer, not construction.
full rationale
The paper's claimed derivation is that training on a verifiable local visual cue search proxy (locate a COCO crop in its full image via an IoU reward, Eqs. 1-6) transfers to improved fine-grained perception at standard full-image inference. This derivation is self-contained: the training signal is built from COCO train2014 ground-truth boxes and a rule-based IoU reward, and downstream V*Bench/HR-Bench/CV-Bench accuracy is never used to define or fit the objective. The proxy is independently validated on a held-out set (Table 10), and ablations isolate the RL objective, cue modality, cue size, and cue source under identical data splits. The attention-in-box analysis in §4.4 uses V*Bench boxes only as a post-hoc interpretability diagnostic; it does not feed back into training or evaluation. The few overlapping-author citations (e.g., Liu et al. 2026 on table understanding, Sun et al. 2026 on video reasoning) are contextual and not load-bearing, and no uniqueness theorem or prior LOCUS result is imported to force the choice. The §B.3 evaluation-prompt asymmetry between reasoning-format checkpoints and base models is a potential confound for attributing the +7.8 V*Bench gain, but it is a validity concern, not a circularity: the reported gain is not definitionally equal to the training reward or to any fitted parameter. No load-bearing step reduces by construction to its own input.
Assumptions & free parameters
free parameters (4)
- Reward format weight α =
0.1
- Tiny:small cue sampling ratio =
70%:30%
- Tiny/small cue size thresholds =
Area <1% / 1–5%
- GRPO hyperparameters =
KL 1e-2, LR 1e-6, batch 128, rollouts 8
assumptions (5)
- domain assumption COCO object boxes provide semantically meaningful visual evidence; random crops provide weaker supervision.
- domain assumption IoU reward on crop localization is a valid proxy for evidence-selection ability in full-image VQA.
- domain assumption GRPO with KL regularization preserves base capabilities.
- domain assumption Attention-in-Box measured from the first answer token to image patches reflects evidence use.
- domain assumption Standard benchmark protocols measure the intended capabilities.
Cite this review
Pith. "Pith review of LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/D7GBLTWR
@misc{pith2026260616586,
author = {Pith},
title = {Pith review of: LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/D7GBLTWR}},
note = {Machine review of arXiv:2606.16586}
}
read the original abstract
Multimodal Large Language Models (MLLMs) remain unreliable on fine-grained visual perception, even when high-resolution inputs preserve the necessary local details. We identify this limitation as visual context rot: decisive evidence may exist in the full image, yet fail to be reliably selected and used amid redundant visual context. We propose LOCUS (LOcal visual CUe Search), a training framework that teaches MLLMs to internalize local evidence search through a verifiable proxy task. During training, LOCUS provides a local crop as a visual cue and optimizes the model to recover its spatial support in the full image using an IoU-based reward. The visual cue is used only during training, leaving the standard image-question inference interface unchanged. Experiments across fine-grained perception, hallucination, general understanding, and reasoning benchmarks show that LOCUS improves localization-sensitive visual understanding while preserving broad capabilities. Attention analyses further indicate stronger focus on task-relevant evidence regions, suggesting that training-time visual cue search provides an effective route to internalized fine-grained evidence selection.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 1 Pith paper
-
Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning
A query-conditioned evidence policy that composes budgeted image crops for a frozen MLLM improves accuracy on localizable, resolution-limited visual question answering benchmarks.
Reference graph
Works this paper leans on
-
[4]
How vision becomes language: A layer-wise information-theoretic analysis of multimodal reason- ing.arXiv preprint arXiv:2602.15580. Penghao Wu and Saining Xie. 2024. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 13084– 13094. Penghao Wu, Yushan...
arXiv 2024
-
[5]
Logicvista: Multimodal llm logical reason- ing benchmark in visual contexts.arXiv preprint arXiv:2407.04973. Shuo Yang, Yuwei Niu, Yuyang Liu, Yang Ye, Bin Lin, and Li Yuan. 2026. Look-back: Implicit visual re-focusing in mllm reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 40, pages 11694–11702. En Yu, Kangheng Lin, L...
arXiv 2026
-
[2016]
InEuro- pean conference on computer vision, pages 235–251
A diagram is worth a dozen images. InEuro- pean conference on computer vision, pages 235–251. Springer. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292– 305. Tsun...
arXiv 2023
-
[2024]
Measuring multimodal mathematical reason- ing with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao L...
arXiv 2026
-
[2026]
When thinking hurts: Mitigating visual forget- ting in video reasoning via frame repetition.arXiv preprint arXiv:2603.16256. Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Pei- dian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chen- hong He, Bowen Ye, and 55 others...
arXiv 2025
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.