Pith. sign in

REVIEW 4 major objections 4 minor 192 references

Thinking with Anchors: Grounded and Efficient Document Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 120k-page dataset argues that document understanding should be judged as reasoning over visual anchors, and shows that today's vision-language models still fail at dense document counting, with the best scoring 72.85%.

desk verdict A genuinely useful dataset extension and counting benchmark with a real zero-shot gap, but the headline numbers depend on label-policy ambiguity that is never measured; still worth peer review. read the letter →

arxiv 2608.04424 v1 pith:BYNE6ISE submitted 2026-08-05 cs.CV

Sichen Zhu , Yuchen Zhu , Wenzhuo Xu , Jason Kuen , Wanrong Zhu , Jing Shi , Xuan Shen , Quanyi Wang
show 8 more authors
Yiwei Wang Yujun Cai Bing Shuai Qin Zhang Yongxin Chen Shilong Liu Molei Tao Jiuxiang Gu
This is my paper
classification cs.CV
keywords documentunderstandingvisualanchorsgroundedreasoningchain-of-thoughtdensecountingsemantictagginglayoutanalysisvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ADOPD 2026 is an attempt to turn document page decomposition into a testbed for grounded reasoning. The paper starts from a 120k-page corpus of dense OCR boxes and human-drawn entity polygons, then adds human-cleaned captions, closed-vocabulary semantic tags, and generated chain-of-thought traces in which each counted region is tied to numbered polygon anchors. Its central claim is that boxes, masks, labels, and text blocks should be treated as one shared vocabulary of visual anchors, and that this representation lets a model point to evidence while answering, not just localize. The paper's benchmark evidence is DocCount, 442 dense counting questions over four region classes, where 13 current vision-language models score between 37.56% and 72.85% exact accuracy. The conclusion a sympathetic reader should take is that dense semantic counting on documents is not solved, and that evaluating region semantics separately from geometry exposes long-tail failures that layout benchmarks hide.

What carries the argument

The load-bearing object is the visual anchor: a text block, visual entity, semantic label, bounding box, or polygon mask treated as one shared vocabulary for detection, tagging, and reasoning. On top of it, the construction procedure adds human-cleaned captions, a 30-class tag taxonomy (normalized to 12 classes for evaluation), and Anchor-CoT traces generated by a VLM from simplified polygon masks, where numbered placeholders like [[1]] and [[2]] keep the prose geometrically controlled while the count and instance order remain deterministic functions of verified labels. DocCount is the resulting 442-sample benchmark over four target classes—brand logo, photograph, table, and chart/graph—scored by exact integer accuracy with ten question phrasings. This machinery lets the paper separate perception errors from policy errors: a model must follow the class definition, see the whole page, and enumerate the right instances.

What would settle it

Mask the numbered polygon placeholders in a random sample of Anchor-CoT traces and ask independent annotators to say which page region each reasoning sentence refers to; if the prose alone does not recover the counted instances or yields a different count, the claimed grounding is not real.

Watch

Extended reading notes

Core claim

The core discovery, stated on the paper's own terms, is that document understanding should be organized around visual anchors—document regions that carry geometry, semantic role, and relations to other regions—and that current vision-language models handle this poorly. The dataset builds three capabilities on that abstraction: region-level semantic tagging, unified generation of text and visual entities with coordinates or polygons, and anchor-grounded chain-of-thought reasoning. Evaluations show a genuine domain gap: off-the-shelf detectors and segmenters are near failure on documents, while fine-tuning lifts AP by 50–70 points; zero-shot VLMs overuse common tags and miss the long tail; and on DocCount the best of 13 models reaches only 72.85%, with the task explicitly described as far from solved. The paper therefore argues that document benchmarks should move beyond localization to verify whether a model can use a region as evidence for an answer.

Load-bearing premise

The reasoning text in the Anchor-CoT traces is generated by a VLM without verification of factual grounding—only polygon labels are checked—so the central promise that the dataset teaches anchor-grounded reasoning depends on those traces not hallucinating.

Editorial extensions

If this is right

  • If DocCount is a fair benchmark, any claim of general document reasoning ability should be tested on it, because current best accuracy of 72.85% leaves a quarter of pages wrong.
  • Fine-tuning a grounder on ADOPD 2026 recovers long-tail tags such as background image and color block, whereas zero-shot models almost never emit them, so the dataset provides a way to measure semantic tagging separately from localization.
  • The agentic grouping workflow, which reasons over numbered boxes and applies a geometric guard, closes 49% of the merge-oracle gap on text-box decomposition without retraining, showing that semantic grouping can repair detector over-fragmentation.
  • Unified grounding outputs—boxes and polygons emitted as structured anchors—can be reused and checked by downstream reasoning systems, making answers verifiable rather than free-form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Anchor-CoT traces are faithful, fine-tuning on them should transfer to counting accuracy; a direct test would compare a VLM trained on the grounded traces against one trained on answer-only labels on DocCount.
  • The same anchor vocabulary could be turned into a broader evidence-verification benchmark, for example asking a model to justify a document answer by citing specific polygons and having a deterministic check score the citation.
  • The over-splitting result suggests that detection AP may understate usable performance; a system that merges proposals by reasoning could outperform a better detector at the same task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ADOPD 2026, an enrichment of the ADOPD 2024 corpus with human-cleaned captions, entity-level semantic tags, and generated chain-of-thought (CoT) traces tied to polygon anchors, and it formalizes a 'visual anchor' abstraction. It evaluates three capabilities: Doc2Box/Doc2Mask localization with fine-tuned detectors and an agentic grouping pipeline; 12-class region tagging with zero-shot and fine-tuned VLMs; and DocCount, a dense counting benchmark on 400+ document pages on which 13 VLMs score between 37.56% and 72.85% exact accuracy. The paper's central claims are that document decomposition exhibits a genuine domain gap, that semantic tagging is long-tail difficult, and that dense semantic counting on documents remains unsolved.

Significance. The resource contribution is potentially substantial: 120k pages, roughly 2.5M tagged elements, a released project page and code, and a carefully constructed train/validation/test split. The localization fine-tuning experiments are internally consistent and convincingly demonstrate a real domain gap. The DocCount zero-shot results, if the ground-truth counts are reliable, would provide a useful diagnostic for VLM counting in documents. However, the benchmark's label reliability and the factual grounding of the CoT traces are not established, so the significance of the headline counting claim is currently conditional on additional validation.

major comments (4)
  1. [Sec. 4.3.1-4.3.2, Table 5] The central claim that dense semantic counting is unsolved rests on DocCount ground-truth counts, but the paper never measures the reliability of the classification and grouping decisions that determine each count. Section 4.3.1 filters to the top-1 class, uses a GPT-5.5 verifier and manual missing-label review, yet no inter-annotator agreement is reported and the class definitions leave unresolved cases. In the A.13 examples, spatially contiguous product shots on a shared background are scored as 4 by the ground truth, 6 by Kimi/Qwen, and 4 by Claude for Photograph; the Chart/Graph example shows the legend panel changing the answer between 5 and 6 across models. With 442 samples and no per-class sample sizes or confidence intervals, the ranking between 72.85% and 70.81% is fragile, and the claim that the task 'remains far from solved' may partly reflect label-policy ambiguity. Please report a double-annotated subset with agreement on grouping decisions, per-class counts and confidence intervals, and a sensitivity analysis of model rankings under alternative grouping policies.
  2. [Sec. 3.2, A.12] The Anchor-CoT traces are a core new resource, but their factual grounding is not verified. The text states that only polygon labels are checked by a VLM plus human review; the generator is 'prompted to write natural-language reasoning' and the reasoning text itself is not checked for correctness. Consequently, the dataset's claim to teach 'anchor-grounded reasoning' is not supported: traces could hallucinate region descriptions while the polygon list remains correct. Add a verification step for the reasoning statements, or explicitly scope the contribution to geometrically controlled traces with unverified prose.
  3. [Sec. 4.2, Table 4] The tagging evaluation is based on n=360 (30 regions per class) with no error bars or significance tests. Differences such as Qwen2.5-VL single-image vs dual-image (35.8 vs 36.4) or Gemma4-12B single vs dual (43.9 vs 48.6) are plausibly within noise, and per-class estimates from 30 samples are unstable. The claim that fine-tuning 'is helpful in recovering the rare classes' needs confidence intervals or bootstrap replicates, and the per-class sample sizes should be stated explicitly.
  4. [Table 5 and Sec. 4.3.2] Table 5 reports per-class accuracies but not the per-class sample sizes within DocCount, so the overall accuracy is hard to interpret. If, for example, Photograph samples greatly outnumber Brand Logo samples, the aggregate 72.85% would be dominated by one class. The paper should report the number of documents per class in DocCount and discuss how class imbalance affects the aggregate ranking and the conclusion that the task remains unsolved.
minor comments (4)
  1. [Sec. 4.3.2] The sentence ending 'definition-following, semantic understanding, and dense grounding on documents.n' contains a stray 'n'; the final sentence of the analysis paragraph also appears truncated and should be completed.
  2. [Table 2, Table 5] The number of DocCount samples is given as 442 in Table 2 but as '400+' in the Table 5 caption and in Section 4.3.2; please unify the reported sample count.
  3. [Sec. 3.2 and Table 5] The text says 13 state-of-the-art VLMs are evaluated, but Table 5 appears to list 12 model rows; please check the model count and the table rendering.
  4. [Sec. 4.1.1] The mF1 metric is defined differently for LocateAnything (confidence-free F1@IoU Mean) and for non-VLM detectors (confidence-swept F1 at IoU 0.5); the paper notes this, but the table caption should repeat the caveat so the numbers are not read as directly comparable across columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DocCount scores are zero-shot measurements against human-verified polygon counts; self-citations are provenance and baselines, not load-bearing.

full rationale

The paper's central empirical claims are measured, not derived. DocCount ground-truth counts come from ADOPD 2026 human-labeled polygons filtered by a GPT-5.5 label verifier and manual missing-label review (Sec. 4.3.1). At evaluation time the paper explicitly withholds the anchor supervision: 'target polygon masks, polygon overlays, and target object lists are not included in the evaluation prompt' (Sec. 4.3.2). The 13 model scores in Table 5 are therefore not computed from the construction inputs and are not fitted parameters. The Anchor-CoT traces are generated after the verified labels exist, with the paper stating that 'the count, instance input order, and polygon coordinates remain deterministic functions of the verified clean labels' (Sec. 4.3.1); this is a data-construction property, not a prediction, and those traces are not used to compute the benchmark accuracies. Self-citations to ADOPD 2024 [22], LocateAnything [65], and FLARE [78] are respectively dataset provenance, an evaluated baseline, and background context; none of them supplies the DocCount numbers or the localization/tagging results. Reported localization and tagging experiments similarly use external checkpoints and held-out splits. Concerns about DocCount label ambiguity and about the factual faithfulness of generated CoT traces are benchmark-validity risks, not circular reductions, so they do not raise the circularity score.

Assumptions & free parameters 6 free parameters · 3 assumptions · 2 invented entities

The central claims rest on a small number of hand-set thresholds (merging, split selection), on the assumption that the fixed semantic taxonomies are adequate, and on the unverified correctness of auto-generated reasoning traces. These are the main uncharged dependencies.

free parameters (6)
  • GeoOk merge guard thresholds = x-overlap >= 0.5; fill ratio >= 0.55; union height < 0.9H; union area < 0.6WH; group size <= 20
    Hand-set thresholds in Algorithm 1 (Appendix A.7) that accept or reject detected-box merges; no sensitivity analysis is provided.
  • Detection pool thresholds = confidence >= 0.05; NMS IoU 0.6; cap N=30
    Parameters for collecting the over-recall proposal pool in the agentic grouping experiment (Sec 4.1.2).
  • Split selection weights = 6, 3, 2, 1, 2, -0.25 in Eq (2)
    Hand-chosen weights in the composite score L(S) used to rank candidate train/validation/test splits (Appendix A.1).
  • DocCount density filter = top-1 class with count > 3
    Documents are excluded unless the most frequent target class appears more than three times, which shapes the benchmark difficulty (Sec 4.3.1).
  • Resampling cap for tagging = min(sqrt(Nmax/Nc), 8)
    Oversampling factor used to rebalance rare classes in the tagging experiment (Sec 4.2.1).
  • Logit adjustment tau = 0.5 and 1.0
    Two adjustments explored to recover Visual Motif recall in the closed-set decoding diagnostic (Appendix A.3, Table 13).
assumptions (3)
  • domain assumption The 30-class and 12-class taxonomies capture the semantically relevant roles of document regions.
    Semantic tagging and DocCount assume all useful region types fit a closed vocabulary defined by the authors (Sec 3.1, Appendix A.4).
  • ad hoc to paper GPT-5.5 generated CoT traces are factually grounded in the provided polygon placeholders.
    The CoT generator is assumed to write truthful, region-consistent reasoning; only polygon labels are verified, not the text (Sec 3.2).
  • domain assumption Model-challenge recall at IoU 0.5 is a valid measure of split difficulty.
    Split selection defines challenge as one minus average box/mask recall, which assumes recall-based difficulty aligns with learnability (Appendix A.1).
invented entities (2)
  • Visual anchor
    purpose: Conceptual unit unifying text blocks, entity polygons, semantic tags, boxes, and masks as a shared vocabulary for document reasoning.
    Defined in Sec 1 and used throughout; it is a descriptive abstraction rather than an independently measurable quantity.
  • Anchor-CoT reasoning traces
    purpose: Chain-of-thought reasoning traces grounded to visual anchors, released as supervision for training document reasoning models.
    The traces are generated by GPT-5.5 with no human verification of reasoning correctness; independent grounding beyond the polygon inputs is not established (Sec 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking with Anchors: Grounded and Efficient Document Reasoning." pith.science (2026). https://pith.science/paper/BYNE6ISE

@misc{pith2026260804424,
  author       = {Pith},
  title        = {Pith review of: Thinking with Anchors: Grounded and Efficient Document Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYNE6ISE}},
  note         = {Machine review of arXiv:2608.04424}
}
read the original abstract

Existing document understanding benchmarks have largely focused on locating page elements, yet real-world document intelligence requires models to reason jointly about region semantics, spatial relations, and visual structure. We present ADOPD 2026, a reasoning-oriented extension of ADOPD that turns page decomposition into spatially grounded document understanding. ADOPD 2026 enriches page anchors inherited from ADOPD 2024 dataset with human-cleaned captions, semantic tags, and generated chain-of-thought (CoT) traces grounded to document regions. Instead of treating boxes, masks, and tags as independent supervision signals, we cast text blocks, visual entities, semantic labels, bounding boxes, and polygon masks as a shared vocabulary of visual anchors. This representation supports three connected capabilities. First, region-level semantic tagging asks models to identify document element types from both page context and local appearance, revealing long-tail semantic failures that standard layout benchmarks often hide. Second, unified vision-language grounding generates text regions and visual entities together with coordinates or polygonal outlines, transforming detection and segmentation outputs into structured anchors that can be reused by downstream reasoning systems. Third, current state-of-the-art models still struggle with dense counting tasks evaluated on DocCount, a benchmark derived from ADOPD 2026, highlighting the need for the Thinking-with-Anchors pipeline in document semantic understanding. By connecting page decomposition to verifiable visual-anchor reasoning, ADOPD 2026 provides a task framework that moves document understanding beyond localization toward anchor-grounded document intelligence.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

192 extracted references · 48 canonical work pages

  1. [1]

    Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov

    Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025

    arXiv 2025

  2. [2]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  3. [3]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  4. [4]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean Conference on Computer Vision (ECCV), pp. 213–229. Springer, 2020

    2020

  5. [5]

    SAM 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane ...

    arXiv 2025

  6. [6]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. InCVPR, 2022

    2022

  7. [7]

    Chaudhuri (ed.).Digital Document Processing: Major Directions and Recent Advances

    Bidyut B. Chaudhuri (ed.).Digital Document Processing: Major Directions and Recent Advances. Springer London, 2007. doi: 10.1007/978-1-84628-726-8

    work page doi:10.1007/978-1-84628-726-8 2007

  8. [8]

    Shikra: Unleashing multimodal LLM’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal LLM’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

Show all 192 references
  1. [9]

    Fleet, and Geoffrey Hinton

    Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. InInternational Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=e42KbIw6Wb

    2022

  2. [10]

    Graph-based document structure analysis

    Yufan Chen, Ruiping Liu, Junwei Zheng, Di Wen, Kunyu Peng, Jiaming Zhang, and Rainer Stiefel- hagen. Graph-based document structure analysis. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://proceedings.iclr.cc/paper_files/paper/2...

    2025

  3. [11]

    M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis

    Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. InCVPR, 2023

    2023

  4. [12]

    M 6Doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis

    Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M 6Doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. InProceedings ...

    2023

  5. [13]

    SDAR: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025

    Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and Bowen Zhou. SDAR: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025

  6. [14]

    Diachronic document dataset for semantic layout analysis.arXiv preprint arXiv:2411.10068,

    Thibault Clérice, Juliette Janès, Hugo Scheithauer, Sarah Bénière, Florian Cafiero, Laurent Romary, Simon Gabay, and Benoît Sagot. Diachronic document dataset for semantic layout analysis.arXiv preprint arXiv:2411.10068,

    arXiv

  7. [15]

    Molmo and pixmo: Open weights and open data for state-of- the-art vision-language models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of- the-art vision-language models. In2025 IEEE/CVF Conference on ...

    2025

  8. [17]

    Holocount: A holistic visual counting benchmark for mllms

    Jinhong Deng, Limeng Qiao, and Guanglu Wan. Holocount: A holistic visual counting benchmark for mllms. arXiv preprint arXiv:2607.06420, 2026

    2026 arXiv

  9. [18]

    Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge.International Journal of Computer Vision, 88(2):303–338, 2010

    2010

  10. [19]

    YOLOX: Exceeding YOLO series in 2021.arXiv preprint arXiv:2107.08430, 2021

    Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding YOLO series in 2021.arXiv preprint arXiv:2107.08430, 2021

    2021 arXiv

  11. [20]

    Better & faster large language models via multi-token prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning ...

    2024

  12. [21]

    Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun

    Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. UniDoc: Unified pretraining framework for document understanding. InNeurIPS, pp. 39–50, 2021

    2021

  13. [22]

    ADoPD: A large-scale document page decomposition dataset

    Jiuxiang Gu, Xiangxi Shi, Jason Kuen, Lu Qi, Ruiyi Zhang, Anqi Liu, Ani Nenkova, and Tong Sun. ADoPD: A large-scale document page decomposition dataset. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://openreview.net/forum?id=x1ptaXpOYa

    2024

  14. [23]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InCVPR, 2023

    2023

  15. [24]

    Ahmed Heakl, Youssef Mohamed, Abdullah Sohail, Rania Elbadry, Ahmed Nassar, Peter W. J. Staar, Fahad Shah- bazKhan, ImranRazzak, and SalmanKhan. DocAtlas: Multilingualdocumentunderstandingacross80+languages. arXiv preprint arXiv:2605.12623, 2026. doi: 10.48550/arXiv.2605.12623...

    2026 doi arXiv

  16. [25]

    Drone-based object counting by spatially regularized regional proposal network

    Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu. Drone-based object counting by spatially regularized regional proposal network. In2017 IEEE International Conference on Computer Vision (ICCV), pp. 4165–4173. IEEE, 2017

    2017

  17. [26]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. InACMMM, 2022

    2022

  18. [27]

    Composition loss for counting, density map estimation and localization in dense crowds

    Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds. In European Conference on Computer Vision, pp. 544–559. Springer, 2018

    2018

  19. [28]

    OCR-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-free document understanding transformer. InEuropean Conference on Computer Vision (ECCV), pp. 498–517, 2022. doi: 10.1007/978-3...

    2022 doi

  20. [29]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. InIEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015–4026, 2023

    2023

  21. [30]

    Page segmentation using a convolutional neural network with trainable co-occurrence features

    Joonho Lee, Hideaki Hayashi, Wataru Ohyama, and Seiichi Uchida. Page segmentation using a convolutional neural network with trainable co-occurrence features. InInternational Conference on Document Analysis and Recognition (ICDAR), pp. 1023–1028, 2019. doi: 10.1109/ICDAR.2019.00167

    2019

  22. [32]

    DocBank: A benchmark dataset for document layout analysis

    Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. DocBank: A benchmark dataset for document layout analysis. InProceedings of the 28th International Conference on Computational Linguistics (COLING), pp. 949–960, 2020. doi: 10.18653/v1/2020.col...

    2020 doi

  23. [33]

    Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu

    Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5652–5660, 2021. 13

    2021

  24. [34]

    MonkeyOCR: Document parsing with a structure-recognition-relation triplet paradigm.arXiv preprint arXiv:2506.05218, 2025

    Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang, Xinyu Wang, and Xiang Bai. MonkeyOCR: Document parsing with a structure-recognition-relation triplet paradigm.arXiv preprint arXiv:2506.05218, 2025. doi: 10.48550/arXiv.2506.05218. ...

    2025 doi

  25. [35]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.),Computer Vision - ECCV 2014 - ...

    2014 doi

  26. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  27. [37]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. InECCV, 2024

    2024

  28. [38]

    Unified-IO 2: Scaling autoregressive multimodal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling autoregressive multimodal models with vision, language, audio, and action. InCVPR, 2024

    2024

  29. [39]

    Thinking with visual primitives

    Ruijie Lu, Yiyang Ma, Xiaokang Chen, Lingxiao Luo, Zhiyu Wu, Zizheng Pan, Xingchao Liu, Yutong Lin, Hao Li, Wen Liu, Zhewen Hao, Xi Gao, Shaoheng Nie, Yixuan Wei, Zhenda Xie, Ting Chen, and Gang Zeng. Thinking with visual primitives. Technical report, DeepSeek-AI, 2026

    2026

  30. [40]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2200–2209, 2021

    2021

  31. [41]

    Puneet Mathur, Rajiv Jain, Jiuxiang Gu, Franck Dernoncourt, Dinesh Manocha, and Vlad I. Morariu. Docedit: Language-guided document editing. InAAAI, pp. 1914–1922, 2023. doi: 10.1609/aaai.v37i2.25282

    1914 doi

  32. [42]

    Ajoy Mondal, Peter Lipps, and C. V. Jawahar. IIIT-AR-13K: A new dataset for graphical object detection in documents. InDocument Analysis Systems, volume 12116 ofLecture Notes in Computer Science, pp. 216–230, Cham, 2020. Springer International Publishing. doi: 10.1007/978-3-03...

    2020 doi

  33. [43]

    Iiit-ar-13k: a new dataset for graphical object detection in documents

    Ajoy Mondal, Peter Lipps, and CV Jawahar. Iiit-ar-13k: a new dataset for graphical object detection in documents. InDAS, 2020

    2020

  34. [44]

    IndicDLP: A foundational dataset for multi-lingual and multi-domain document layout parsing

    Oikantik Nath, Sahithi Kukkala, Mitesh Khapra, and Ravi Kiran Sarvadevabhatla. IndicDLP: A foundational dataset for multi-lingual and multi-domain document layout parsing. InDocument Analysis and Recognition – ICDAR 2025, volume 16023 ofLecture Notes in Computer Science, pp. 2...

    2025 doi

  35. [45]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    2025 arXiv

  36. [46]

    A general approach for multi-oriented text line extraction of handwritten documents.IJDAR, 2012

    Nazih Ouwayed and Abdel Belaïd. A general approach for multi-oriented text line extraction of handwritten documents.IJDAR, 2012

    2012

  37. [47]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3147–3157. IEEE, 2023

    2023

  38. [48]

    Continuous document layout analysis: Human-in-the-loop AI-based data curation, database, and evaluation in the domain of public affairs.Information Fusion, 108:102398, 2024

    Alejandro Peña, Aythami Morales, Julian Fierrez, Javier Ortega-Garcia, Iñigo Puente, Jorge Cordova, and Gonzalo Cordova. Continuous document layout analysis: Human-in-the-loop AI-based data curation, database, and evaluation in the domain of public affairs.Information Fusion, ...

    2024

  39. [49]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  40. [51]

    Nassar, and Peter W

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter W. J. Staar. DocLayNet: A large human-annotated dataset for document-layout segmentation. InSIGKDD, pp. 3743–3751, 2022. doi: 10.1145/3534678.3539043

    2022

  41. [52]

    Learning to count everything

    Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3394–3403, June 2021

    2021

  42. [53]

    SAM 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024 arXiv

  43. [54]

    RF-DETR: Neural architecture search for real-time detection transformers

    Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, and Neehar Peri. RF-DETR: Neural architecture search for real-time detection transformers. InInternational Conference on Learning Representations (ICLR),

  44. [55]

    Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method.IEEE transactions on pattern analysis and machine intelligence, 44(5):2594–2609, 2020

    Vishwanath A Sindagi, Rajeev Yasarla, and Vishal M Patel. Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method.IEEE transactions on pattern analysis and machine intelligence, 44(5):2594–2609, 2020

    2020

  45. [56]

    An overview of the tesseract OCR engine

    Ray Smith. An overview of the tesseract OCR engine. InInternational Conference on Document Analysis and Recognition (ICDAR), pp. 629–633, 2007. doi: 10.1109/ICDAR.2007.4376991

    2007

  46. [57]

    ViperGPT: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. InICCV, 2023

    2023

  47. [58]

    Countqa: How well do mllms count in the wild?arXiv preprint arXiv:2508.06585, 2025

    Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. Countqa: How well do mllms count in the wild?arXiv preprint arXiv:2508.06585, 2025

    2025 arXiv

  48. [59]

    Unifying vision, text, and layout for universal document processing

    Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19254–19264, 2023

    2023

  49. [60]

    Kwai keye-vl-2.0 technical report.arXiv preprint arXiv:2606.10651, 2026

    Kwai Keye Team, Bin Wen, Changyi Liu, Chengru Song, Chongling Rao, Guowang Zhang, Han Li, Haonan Fan, Hengrui Ju, Jiankang Chen, et al. Kwai keye-vl-2.0 technical report.arXiv preprint arXiv:2606.10651, 2026

    2026 arXiv

  50. [61]

    YOLOv12: Attention-centric real-time object detectors.arXiv preprint arXiv:2502.12524, 2025

    Yunjie Tian, Qixiang Ye, and David Doermann. YOLOv12: Attention-centric real-time object detectors.arXiv preprint arXiv:2502.12524, 2025

    2025 arXiv

  51. [62]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language- independent named entity recognition. InProceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL, pp. 142–147, 2003

    2003

  52. [63]

    SCAN: Semantic document layout analysis for textual and visual retrieval-augmented generation

    Nobuhiro Ueda, Yuyang Dong, Krisztián Boros, Daiki Ito, Takuya Sera, and Masafumi Oyamada. SCAN: Semantic document layout analysis for textual and visual retrieval-augmented generation. InFindings of the Association for Computational Linguistics: EACL 2026, pp. 1618–1637, Raba...

    2026 doi

  53. [64]

    Nwpu-crowd: A large-scale benchmark for crowd counting and localization.IEEE transactions on pattern analysis and machine intelligence, 43(6):2141–2149, 2020

    Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li. Nwpu-crowd: A large-scale benchmark for crowd counting and localization.IEEE transactions on pattern analysis and machine intelligence, 43(6):2141–2149, 2020

    2020

  54. [65]

    Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.27365, 2026

    Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, Jan Kautz, Lei Zhang, and Zhiding Yu. Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.2...

    2026 arXiv

  55. [66]

    Fast-dLLM v2: Efficient block-diffusion LLM.arXiv preprint arXiv:2509.26328, 2025

    Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dLLM v2: Efficient block-diffusion LLM.arXiv preprint arXiv:2509.26328, 2025

    2025

  56. [67]

    DocGenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models.arXiv preprint arXiv:2406.11633, 2024

    Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang, Chao Xu, Conghui He, Pinlong Cai, Min Dou, Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, and Yu Qiao. DocGen...

    2024 doi

  57. [68]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. InCVPR, 2024. 15

    2024

  58. [69]

    Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  59. [70]

    MM-REACT: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  60. [71]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    2025 arXiv

  61. [72]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. InICLR, 2024

    2024

  62. [73]

    Single-image crowd counting via multi- column convolutional neural network

    Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi- column convolutional neural network. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 589–597, 2016. doi: 10.1109/CVPR.2016.70

    2016 doi

  63. [74]

    Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  64. [75]

    DocLayout-YOLO: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception.arXiv preprint arXiv:2410.12628,

    Zhiyuan Zhao, Hengrui Kang, Bin Wang, and Conghui He. DocLayout-YOLO: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception.arXiv preprint arXiv:2410.12628,

    arXiv

  65. [77]

    PubLayNet: Largest dataset ever for document layout analysis

    Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. PubLayNet: Largest dataset ever for document layout analysis. InInternational Conference on Document Analysis and Recognition (ICDAR), pp. 1015–1022, 2019. doi: 10.1109/ICDAR.2019.00166

    2019

  66. [78]

    URLhttps://arxiv.org/abs/2410.12628

    doi: 10.48550/arXiv.2410.12628. URLhttps://arxiv.org/abs/2410.12628

    doi arXiv

  67. [81]

    FLARE: Diffusion for hybrid language model.arXiv preprint arXiv:2606.01774, 2026

    Yuchen Zhu, Jing Shi, Chongjian Ge, Hao Tan, Yiran Xu, Wanrong Zhu, Jason Kuen, Koustava Goswami, Rajiv Jain, Yongxin Chen, Molei Tao, and Jiuxiang Gu. FLARE: Diffusion for hybrid language model.arXiv preprint arXiv:2606.01774, 2026. 16 120k documents Corpus split Train 80k V ...

    2026 arXiv

  68. [82]

    Icon.Icons include: (a) symbols with specific semantic meaning (weather, restroom, toolbar buttons); (b) small decorative markers (snowflakes, arrows, stars); (c) app logos; (d) QR/barcodes

  69. [83]

    Icons and illustrations co-occurring.Elements inside a highlighted box may be icons or illustrations — inspect each individually

  70. [84]

    icon.A single large, visually striking standalone graphic =prominent pattern

    Prominent pattern vs. icon.A single large, visually striking standalone graphic =prominent pattern. A small semantically specific symbol =icon. 4.Icon vs. color block.Thin arrow-shaped strokes =icon. Visually thick bar-shaped elements =color block. 22

  71. [85]

    Natural background.If an image is simultaneously a natural photographandthe full-bleed page background, label itnatural background

  72. [86]

    Image labels beside photos =image caption

    Titles in newspapers and magazines.Mastheads, article headlines, and section names =title. Image labels beside photos =image caption

  73. [87]

    Header / Footer.Content at the very top or bottom of the page clearly unrelated to the main content (page numbers, chapter names, copyright) =header/footer

  74. [88]

    Plain colored region without border =color block (borderless)

    Background pattern, borderless color block, hyperlink.Grid texture inside an element =background pattern. Plain colored region without border =color block (borderless). Underlined link text =hyperlink

  75. [89]

    Do not split the tag across modes

    Logo with text and graphic combined.Annotate the graphic part asbrand logoin mask mode; annotate the text part asbrand logoin OCR mode. Do not split the tag across modes

  76. [90]

    A single-item list must still be separated

    Mixed pre-annotated OCR boxes.Resolve conflicts: body text + title together→body text; main + sub-title together→title; list item + body text→split into two boxes. A single-item list must still be separated. 11.Line element.A thin arrow whose arrowhead is small and inconspicuo...

  77. [91]

    background pattern vs

    Background image vs. background pattern vs. prominent pattern.Full-bleed white base =background image. Shadow overlay above it =background pattern(foreground). Large graphic poster on top =prominent pattern(foreground)

  78. [92]

    In newspaper layouts, side-column text adjacent to the main article =note; text at the very bottom =footer

    Prominent pattern; newspaper notes and footer.A standalone eye-catching graphic =prominent pattern. In newspaper layouts, side-column text adjacent to the main article =note; text at the very bottom =footer

  79. [93]

    Hyperlinks or emphasis embedded in a large body-text block without their own pre-annotation belong to the body-text box

    Emphasized text and hyperlinks inside body text.Annotate separately if they stand alone. Hyperlinks or emphasis embedded in a large body-text block without their own pre-annotation belong to the body-text box

  80. [94]

    image caption; table title.Contextual explanatory text at the top-left or bottom of an image block =note

    Note vs. image caption; table title.Contextual explanatory text at the top-left or bottom of an image block =note. A centered label pointing directly to an image =image caption. A heading above a table =table title

  81. [95]

    note.A label directly beside or below a photo =image caption

    Image caption in context; magazine header/footer vs. note.A label directly beside or below a photo =image caption. Magazine top/bottom elements: judge by their relationship to the main article content

  82. [96]

    All elements above =foreground

    Background layer determination.Only the element(s) at the absolute bottom layer spanning the full page width or height =background. All elements above =foreground

  83. [97]

    prominent pattern; legend vs

    Decorative pattern vs. prominent pattern; legend vs. chart title.Minor ornamental element with no direct meaning =decorative pattern. Meaningful, conspicuous standalone graphic =prominent pattern. Incon- spicuous small text adjacent to a figure =legend. Prominent label above a...

  84. [98]

    legend.Text directly below a photo pointing to it =image caption

    Image caption vs. legend.Text directly below a photo pointing to it =image caption. Text that serves as a note for multiple figures =legend. 20.List item.Numbered or bulleted list elements that are cohesive and clearly part of a list structure. 23

  85. [99]

    icon (cartoons).Cartoon resembling a recognizable icon-style symbol =icon

    Illustration vs. icon (cartoons).Cartoon resembling a recognizable icon-style symbol =icon. Other cartoons =illustration

  86. [100]

    background pattern (full-bleed non-solid).If the bottommost layer is a non-solid graphic spanning the full width or height, label itbackground image, notbackground pattern

    Background image vs. background pattern (full-bleed non-solid).If the bottommost layer is a non-solid graphic spanning the full width or height, label itbackground image, notbackground pattern

  87. [101]

    There areNnumbered boxes (0..N−1). Group them

    Comprehensive annotation examples.Apply all rules above holistically: identify the bottommost background layer first, then label foreground elements from largest to smallest. A.4.6 Label Refinement The visual mask rules above describe the fine-grained annotation protocol used ...

    2026

  88. [103]

    Below them on the right,[[6]] marks the First in Malaysia badge

    Grounded Object Localization Across the top row,[[1]]encloses the Tourism Malaysia mark,[[2]]encloses the Discovery wordmark, [[3]] encloses the Holiday Rail emblem,[[4]] encloses the KTM wordmark, and[[5]] encloses the Firefly airline logo. Below them on the right,[[6]] marks...

    2020

  89. [104]

    37 Fine-grained Counting [Trigger_Placeholder] How many regions on this page should be counted as Photograph? Original Image Image with Visual Anchors Anchor Thinking Reasoning

    Summation That makes 7 in total. 37 Fine-grained Counting [Trigger_Placeholder] How many regions on this page should be counted as Photograph? Original Image Image with Visual Anchors Anchor Thinking Reasoning

  90. [106]

    Grounded Object Localization From left to right,[[1]]is the large product photograph showing two girls’ dresses,[[2]]is the upper central photograph of a child modeling an outfit,[[3]]is the lower central photograph of another child model, and[[4]]is the right-side product pho...

  91. [107]

    38 Fine-grained Counting [Trigger_Placeholder] Count the Table regions visible on this page

    Summation I count 4. 38 Fine-grained Counting [Trigger_Placeholder] Count the Table regions visible on this page. Original Image Image with Visual Anchors Anchor Thinking Reasoning

  92. [109]

    Grounded Object Localization Scanning downward,[[1]]is the gridded table for early-childhood teaching qualifications,[[2]]is the separate gridded table for primary teaching qualifications,[[3]] is the larger gridded table for secondary teaching qualifications, and[[4]]is the b...

  93. [110]

    Summation The total is 4. 39 Fine-grained Counting [Trigger_Placeholder] What is the total number of Chart / Graph regions in the page image? Original Image Image with Visual Anchors Anchor Thinking Reasoning

  94. [112]

    In the second row,[[4]]is the Marco Rubio panel and[[5]]is the Donald Trump panel; each contains plotted search and news-article trends

    Grounded Object Localization In the top row of small multiples,[[1]]is the Ben Carson time-series panel,[[2]]is the Ted Cruz panel, and[[3]]is the John Kasich panel. In the second row,[[4]]is the Marco Rubio panel and[[5]]is the Donald Trump panel; each contains plotted search...

  95. [113]

    40 Fine-grained Counting [Trigger_Placeholder] What count of Brand Logo regions do you see in this page? Original Image Image with Visual Anchors Anchor Thinking Reasoning

    Summation I count 5 total. 40 Fine-grained Counting [Trigger_Placeholder] What count of Brand Logo regions do you see in this page? Original Image Image with Visual Anchors Anchor Thinking Reasoning

  96. [114]

    Intent Analysis The task is to count every[[T]]region in the document image by following the supplied reading order

  97. [115]

    In the next row,[[3]]is the CINVE 2017 seal and[[4]]is the Mezquita ORG 2017 seal

    Grounded Object Localization In the first awards row,[[1]]is the Los Angeles olive-oil gold-medal seal and[[2]]is the NYIOOC gold-award emblem. In the next row,[[3]]is the CINVE 2017 seal and[[4]]is the Mezquita ORG 2017 seal. The following row contains[[5]], the Zurich Olive ...

    2017

  98. [116]

    Summation That gives me 9. 41 A.13 Examples inDocCountEvaluation Brand Logo Ground Truth7Tagbrand logo How many distinct Brand Logo regions can you find on the page? Original Overlay Kimi K2.5 (reasoning on) predicted 7 correct Looking at the image, I need to identify distinct...

  99. [117]

    Supported by

    **Tourism Malaysia Logo** (top, "Supported by" row): A designed mark with the national flag element and identifying text, functioning as the tourism board’s visual identity

  100. [118]

    D" icon with the

    **Discovery Logo** (top row, next to Tourism Malaysia): The stylized "D" icon with the "Discovery" wordmark, representing the media/brand partner

  101. [119]

    **Circular Partner/Destination Logo** (top row, center): A circular badge with a landscape graphic and text, placed as a distinct sponsoring partner identity

  102. [120]

    **KTM Logo** (top row): The stylized "KTM" lettermark with the signature blue and yellow striped design, identifying the rail service brand

  103. [121]

    firefly" stylized wordmark, and the

    **Firefly Logo** (top right): Consists of the insect icon, the "firefly" stylized wordmark, and the "Beyond Convenience" tagline. Per the definition, a designed mark and its associated brand name appearing together as one unit counts as **one** distinct brand logo

  104. [122]

    FIRST IN MALAYSIA

    **"FIRST IN MALAYSIA" Badge** (right side, diamond-shaped): This is an endorsement badge/seal making a specific claim about the product’s exclusivity. The definition explicitly includes endorsement badges as a type of brand logo

  105. [123]

    FLYRAIL" text is the product/package name presented as a headline, not a distinct corporate visual identity mark. The

    **Visit Truly Asia Malaysia 2020 Logo** (right side, hibiscus and hornbill design): This is the official program/campaign logo for Malaysia’s tourism year, functioning as a visual identity mark for that specific government initiative. The large "FLYRAIL" text is the product/pa...

    2020

  106. [124]

    Supported by TOURISM MALAYSIA

    **Top right corner group:** - “Supported by TOURISM MALAYSIA” — This includes the Tourism Malaysia logo (a stylized flower emblem) with its name. This counts as **one Brand Logo**. - “Discovery” — The orange sun-like icon next to the word “Discovery” is the Discovery Channel b...

  107. [125]

    FIRST IN MALAYSIA

    **Right side, middle section:** - “FIRST IN MALAYSIA” — This is a badge/seal indicating an official certification or program. It has a distinct design (blue diamond shape with white text). Counts as **one Brand Logo** (official program mark). - Below it: “Visit Malaysia 2020” ...

    2020

  108. [126]

    Book now!

    **Bottom right corner:** - “Book now!” with a stylized sail/boat icon — This appears to be a call-to-action button graphic, not a brand logo. Exclude

  109. [127]

    Since we already counted the Firefly logo above, we don’t double-count

    **Airplane in sky:** - The airplane has “firefly” written on its side — but this is part of the aircraft livery, not a separate logo instance. Since we already counted the Firefly logo above, we don’t double-count

  110. [128]

    **Train:** - No visible brand logo on the train itself (just locomotive number “2521”), so no additional logo here

  111. [129]

    3D2N Holiday Package

    **Left side:** - “FLYRAIL” — This is the name of the package, styled in bold orange letters. However, it’s presented as a product/package name, not necessarily a standalone brand logo. Given the context (“3D2N Holiday Package”), it’s likely a marketing title rather than a regi...

  112. [130]

    SPECIAL LAUNCH PROMO 15% OFF Limited time only

    **Circular orange promo badge:** - “SPECIAL LAUNCH PROMO 15% OFF Limited time only.” — This is a promotional graphic, not a brand logo. Exclude

  113. [131]

    **Small airplane on runway:** - No branding visible. Exclude

  114. [132]

    *Terms & Conditions apply

    **Bottom left corner:** - “*Terms & Conditions apply.” — Text only, not a logo. Exclude. Now, summing up the confirmed Brand Logos: - Tourism Malaysia→1 - Discovery→1 - Malaysia Rail→1 - KTM→1 - Firefly→1 - FIRST IN MALAYSIA badge→1 - Visit Malaysia 2020→1 Total = **7** <answe...

    2020

  115. [133]

    Tourism Malaysia logo

  116. [134]

    Firefly logo at top right

  117. [135]

    FlyRail stylized wordmark

  118. [136]

    First in Malaysia

    “First in Malaysia” badge

  119. [137]

    Visit Truly Asia Malaysia 2020 logo

    2020

  120. [138]

    Firefly branding on the aircraft tail

  121. [139]

    Fireflyz.com.my branding on the aircraft fuselage

  122. [140]

    Firefly-style logomark in the bottom-right orange section <answer>11</answer> Claude Sonnet 4.5 (reasoning on) predicted 8 wrong # Detailed Analysis of Brand Logos Let me systematically examine the image to identify all distinct brand logos: ## Top Section - Sponsor Logos (Lef...

  123. [141]

    Supported by TOURISM MALAYSIA

    **Tourism Malaysia** - Official tourism organization logo with "Supported by TOURISM MALAYSIA" text

  124. [142]

    **Recovery** - Orange/yellow logo with arrow design (second sponsor)

  125. [143]

    **Circular green/teal logo** - Appears to be an environmental or certification organization (third sponsor)

  126. [144]

    **KTMB** - Blue and yellow railway company logo (Keretapi Tanah Melayu Berhad)

  127. [145]

    Beyond Convenience

    **firefly** - Orange/yellow airline logo with "Beyond Convenience" tagline ## Main Content Area:

  128. [146]

    **FLYRAIL** - The large orange stylized wordmark serves as the product/service brand identifier for this fly-and-rail package ## Right Side Elements:

  129. [147]

    FIRST IN MALAYSIA

    **"FIRST IN MALAYSIA"** - Diamond-shaped badge/seal that functions as a program identifier and promotional emblem

  130. [148]

    Book now!

    **Visit Terengganu Malaysia 2021** - Floral hibiscus design logo for the tourism campaign ## Excluded Elements: - "Book now!" with arrow design - This is a call-to-action graphic element, not a brand logo - The airplane and train images themselves - These are photographs, not ...

    1970

  131. [149]

    The garment is photographed realistically against a neutral background

    **Region A** – Top left: Shows a red plaid dress with white collar and apron-style front. The garment is photographed realistically against a neutral background. This qualifies as a photograph

  132. [150]

    The image is a realistic product shot — qualifies as a photograph

    **Region B** – Bottom left: Shows a dark blue polka-dot jumper with white blouse underneath, worn by a child (partially visible). The image is a realistic product shot — qualifies as a photograph

  133. [151]

    This is a full-body photo of a real person wearing the item — qualifies as a photograph

    **Region C** – Top middle: Features a girl modeling a denim-style dress with ruffled hem. This is a full-body photo of a real person wearing the item — qualifies as a photograph. 45

  134. [152]

    Again, a realistic photo of a model in the garment — qualifies as a photograph

    **Region D** – Bottom middle: Shows another girl modeling a striped dress with ruffled sleeves. Again, a realistic photo of a model in the garment — qualifies as a photograph

  135. [153]

    Qualifies

    **Region E** – Top right: Displays a two-piece set (top and pants) laid out flat but still presented as a photographic product shot — not illustrated, but photographed. Qualifies

  136. [154]

    Additionally, note that the entire layout includes photographs of actual garments and models — these are all realistic depictions of physical products and people

    **Region F** – Bottom right: Shows a child’s dress with vest-style bodice and floral skirt, again photographed as a product image — qualifies. Additionally, note that the entire layout includes photographs of actual garments and models — these are all realistic depictions of p...

  137. [155]

    Red plaid dress labeled A

  138. [156]

    Jumper-and-blouse set labeled B

  139. [157]

    Child modeling chambray dress labeled C

  140. [158]

    Child modeling striped skirt set labeled D

  141. [159]

    Pants and top set labeled E

  142. [160]

    realistic or naturalistic image content depicting real-world people, objects, places, products

    Dress with vest-effect labeled F Text blocks, labels, and background page areas are not counted. <answer>6</answer> Claude Sonnet 4.5 (reasoning on) predicted 4 correct Looking at this catalog page carefully, I need to identify distinct photographic regions based on the defini...

  143. [161]

    Teaching (Early Childhood Education, ages 0–5)

  144. [162]

    Teaching (Primary, ages 5–13)

  145. [163]

    Teaching (Secondary, ages 13–18)

  146. [164]

    The entire structure functions as a single cohesive table with clear columnar alignment: columns for qualification type, degree name, university/institution, and duration

    Education (non-teaching) Each section consists of multiple rows listing specific degrees, the institutions offering them, and the duration (years). The entire structure functions as a single cohesive table with clear columnar alignment: columns for qualification type, degree n...

  147. [165]

    Teaching — Early Childhood Education

  148. [166]

    Teaching — Secondary

  149. [167]

    Let me examine the page systematically:

    Education — Non-teaching Therefore, the number of visible Table regions is: <answer>4</answer> Claude Sonnet 4.5 (reasoning on) predicted 4 correct Looking at this document about undergraduate qualifications relevant to Education, I need to identify all table regions that disp...

  150. [168]

    Teaching (Early Childhood Education, ages 0-5)

    **First table**: "Teaching (Early Childhood Education, ages 0-5)" - This section contains a structured grid with columns showing qualification names, institutions, and duration. The information is organized in cells with visible borders

  151. [169]

    Teaching (Primary, ages 5-13)

    **Second table**: "Teaching (Primary, ages 5-13)" - This is a separate table below the first one, with the same columnar structure showing Bachelor degrees, institutions, and years

  152. [170]

    Teaching (Secondary, ages 13-18)

    **Third table**: "Teaching (Secondary, ages 13-18)" - Another distinct table section showing secondary education qualifications with institutions and 4-year durations organized in rows and columns

  153. [171]

    Education (non-teaching)

    **Fourth table**: "Education (non-teaching)" - The final table section at the bottom showing non-teaching education qualifications with their respective institutions and durations. Each of these sections is clearly separated and functions as an independent table with: - A left...

  154. [172]

    Articles on Google News

    A **main header section** with text and a small legend/key graphic in the top right corner — this legend shows two lines (red for “Articles on Google News”, black for “Google Searches”) over a simplified x-axis. This is a **small illustrative chart/graph** used as a key or exa...

  155. [173]

    MAX” and “0

    Then there are **five distinct panels**, each showing time-series graphs for a different candidate: - Ben Carson - Ted Cruz - John Kasich - Marco Rubio - Donald Trump Each of these five panels: - Has its own title (candidate name) - Contains a line graph (black = Google Search...

  156. [174]

    <answer>5</answer> Claude Sonnet 4.5 (reasoning on) predicted 6 wrong Let me analyze this image systematically to count all chart/graph regions

    Donald Trump The small graphic in the upper-right appears to function as a legend/key explaining the red area and gray line encodings, rather than a separate independent chart. <answer>5</answer> Claude Sonnet 4.5 (reasoning on) predicted 6 wrong Let me analyze this image syst...

  157. [175]

    ARTICLES ON GOOGLE NEWS

    **Top Right Corner**: There is a small chart labeled "ARTICLES ON GOOGLE NEWS" with "GOOGLE SEARCHES" indicator. This shows two overlaid time series (one appears to be searches, one appears to be articles) - this is **1 chart**

  158. [176]

    each repeated panel is a self-contained visualization with its own data display

    **Main Section - Small Multiples Grid**: The image displays 5 individual time-series charts arranged in a grid, each showing data for a different political candidate: - **Ben Carson** (top row, left) - 1 chart - **Ted Cruz** (top row, center) - 1 chart - **John Kasich** (top r...

  159. [177]

    LOS ANGELES

    L.A. Olive Oil Awards (USA) — Gold Medal→circular gold seal with “LOS ANGELES” and olive branch→1 Brand Logo

  160. [178]

    CINVE 2017

    CINVE Awards (Spain) — Gold Medal→black and gold emblem with “CINVE 2017”→1 Brand Logo

    2017

  161. [179]

    OLIVE OIL AWARD ZURICH 2017

    Zurich Olive Oil Awards (Switzerland) — Gold Medal→white and gold circular emblem with “OLIVE OIL AWARD ZURICH 2017”→1 Brand Logo

    2017

  162. [180]

    ATHENA CONTEST 2017

    Athena Contest (Greece) — Gold Medal→dark circular emblem with “ATHENA CONTEST 2017”→1 Brand Logo

    2017

  163. [181]

    AVPA PARIS

    AVPA Paris (France) — Gourmet Gold Medal→golden circular emblem with star design and “AVPA PARIS”→1 Brand Logo

  164. [182]

    DOMINA IOOC Gold Medal 2017

    DOMINA Olive Oil Competition (Italy) — Gold Medal→golden circular emblem with “DOMINA IOOC Gold Medal 2017”→1 Brand Logo Right column (6 entries):

    2017

  165. [183]

    GOLD AWARD 2017

    NYIOOC 2017 (USA) — Gold Medal→black and gold laurel wreath emblem with “GOLD AWARD 2017”→1 Brand Logo

    2017

  166. [184]

    ORO 2017

    Mezquita Awards (Spain) — Gold Medal→greenish-gold circular emblem with “ORO 2017” and “MEZQUITA”→1 Brand Logo

    2017

  167. [185]

    TERRAOLIVO ISRAEL

    TerraOlivo 2017 (Israel) — Grand Prestige Gold→circular emblem with olive oil pitcher and “TERRAOLIVO ISRAEL”→1 Brand Logo

    2017

  168. [186]

    Les Olivalies Argente

    Les Olivalies (France) — Silver Medal→rectangular certificate-style badge with “Les Olivalies Argente”→1 Brand Logo (functions as official award seal)

  169. [187]

    Expo Azeite IOOC 2017

    Expo Azeite Brazil (Brazil) — Silver Medal→rectangular certificate-style badge with “Expo Azeite IOOC 2017”→1 Brand Logo

    2017

  170. [188]

    None are generic decorative icons — they all serve as official seals or badges for those programs

    Japan Olive (Japan) — Silver Medal→vertical silver badge with “SILVER” and Japanese characters→1 Brand Logo All 12 emblems are distinct visual identity marks associated with specific award competitions or organizations. None are generic decorative icons — they all serve as off...

  171. [189]

    Olive Oil Awards** - Circular gold seal/badge

    **L.A. Olive Oil Awards** - Circular gold seal/badge

  172. [190]

    CINVE 2017

    **CINVE Awards** - Black circular medal with "CINVE 2017" branding

    2017

  173. [191]

    **Zurich Olive Oil Awards** - Circular medal with olive branch design and "GOLD" label

  174. [192]

    **Athena Contest** - Circular black and gold medal badge

  175. [193]

    **AVPA Paris** - Circular gold medal with decorative sunburst design

  176. [194]

    IOOC Gold Medal

    **DOMINA Olive Oil Competition** - Circular gold medal with "IOOC Gold Medal" branding **Right Column:**

  177. [195]

    GOLD AWARD 2017

    **NYIOOC 2017** - Circular black badge with "GOLD AWARD 2017" text

    2017

  178. [196]

    **Mezquita Awards** - Circular gold medal with "ORO" designation

  179. [197]

    **TerraOlivo 2017** - Circular olive-green colored badge

    2017

  180. [198]

    **Les Olivalies** - Certificate/document displaying the award brand identity

  181. [199]

    **Expo Azeite Brazil** - Certificate/document displaying the award brand identity

  182. [200]

    Whether displayed as a medal, badge, seal, or certificate, each serves as the brand logo identifying that particular award program

    **Japan Olive** - Rectangular badge/seal showing the award branding Each of these represents a distinct visual identity mark for a different olive oil competition or award organization. Whether displayed as a medal, badge, seal, or certificate, each serves as the brand logo id...

  183. [2024]

    URLhttps://arxiv.org/abs/2411.10068

    doi: 10.48550/arXiv.2411.10068. URLhttps://arxiv.org/abs/2411.10068

    doi arXiv

  184. [2026]

    URLhttps://openreview.net/forum?id=qHm5GePxTh

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.