Pith. sign in

REVIEW 4 major objections 4 minor 66 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

MLLMs Can See Objects But Can't Reason About Counts

2026-07-08 06:10 UTC pith:AJML3EYM

load-bearing objection Useful benchmark with a well-designed taxonomy; the perception-vs-reasoning narrative is defensible but overstated for some models because it ignores error compounding. the 4 major comments →

arxiv 2607.06420 v1 pith:AJML3EYM submitted 2026-07-07 cs.CV

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

Jinhong Deng , Limeng Qiao , Guanglu Wan This is my paper
classification cs.CV
keywords visual countingmultimodal large language modelsbenchmarkanalytical reasoningrobustness testingnumerical hallucinationset-based reasoninglinguistic prior
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

HoloCount tests whether multimodal large language models can count objects reliably when the task goes beyond simple enumeration. The authors construct a three-level taxonomy: semantic counting (basic and attribute-filtered enumeration), analytical counting (spatial reasoning, set operations, comparisons), and robustness testing (high density, occlusion, linguistic bias, null targets). Across 20 fine-grained subsets and 2,480 QA pairs, they evaluate over 20 state-of-the-art models and find a steep performance cliff: models that score 85-95% on basic object counting drop to 20-50% when the task requires composing counts with logical operations, resisting linguistic priors, or handling dense scenes. The paper's central claim is that the bottleneck for visual counting in MLLMs is not visual perception but symbolic reasoning, arithmetic composition, and the ability to override language-driven priors when they conflict with visual evidence. The benchmark is designed to isolate these failure modes so that a single accuracy score can be decomposed into specific cognitive deficits.

Core claim

The paper discovers a systematic dissociation between perception and reasoning in multimodal counting. Models can see and enumerate objects at high accuracy, but performance collapses when counting requires set-based logic (aggregation, exclusion, comparison), spatial grounding (coordinate-specified regions, relative orientation), or resistance to adversarial priors (linguistic expectations, null targets, visual distractors). Two additional findings stand out. First, on null-target tasks (counting objects that are absent), larger and more capable models paradoxically perform worse than smaller ones, because optimization for detailed scene description makes them reluctant to output zero. This

What carries the argument

HoloCount benchmark: a three-level hierarchical taxonomy (Semantic Counting, Analytical Counting, Robustness Testing) with 20 fine-grained subsets and 2,480 QA pairs spanning 1,481 visual concepts, designed to decouple visual perception failures from reasoning failures and robustness failures.

Load-bearing premise

The ground-truth annotations are correct and the QA pairs are unambiguous. The paper uses one model to generate QA pairs for several subsets and then evaluates that same model on those subsets, without reporting inter-annotator agreement or addressing this overlap.

What would settle it

If a model were shown to perform equally well on analytical and robustness tasks as on atomic counting when given chain-of-thought prompting or explicit counting tools, the dissociation claim would be weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the perception-reasoning dissociation is real, then scaling model size alone will not close the counting gap; architectures need explicit counting and set-operation primitives.
  • The inverse relationship between model scale and null-target accuracy suggests that safety-aligned or detail-oriented training may increase hallucination in specific counting contexts.
  • Thinking mode consistently improves counting by 10-15 points across model scales, implying that step-by-step reasoning compensates for missing counting primitives.
  • The high-density counting failure (most models below 10% accuracy) indicates that current vision encoders lose fine-grained spatial resolution under crowding, a hardware-level constraint.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The overlap between the QA generation model (Gemini-3-Flash-Preview) and the evaluated models could inflate scores for subsets where that model both generated and answered questions, potentially understating the true performance gap.
  • The synthetic high-density subset (symbols on blank canvas) may not transfer to real-world dense scenes, so the high-density failure could be either better or worse in practice.
  • If thinking mode improves counting by 10-15 points, then models may already have latent counting ability that is inaccessible without explicit chain-of-thought prompting, suggesting an architectural bottleneck in output routing rather than capability.
  • The linguistic prior conflict results imply that models encode world knowledge in a way that overrides visual input, which has implications beyond counting for any task where prior expectations conflict with image evidence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces HoloCount, a visual counting benchmark for MLLMs structured around a three-level taxonomy: Semantic Counting, Analytical Counting, and Robustness Testing. The benchmark comprises 2,480 QA pairs across 20 fine-grained subsets spanning 1,481 visual concepts. The authors evaluate over 20 open-source and proprietary MLLMs under zero-shot, no-chain-of-thought conditions using exact match accuracy. The central empirical finding is a systematic performance degradation from basic perception (85-95% on atomic counting) to analytical reasoning and robustness tasks (20-50% on high-density, linguistic prior conflict, and set-based reasoning), which the authors interpret as evidence that the primary bottleneck is symbolic reasoning and arithmetic composition rather than visual perception.

Significance. The paper's main contribution is the diagnostic taxonomy itself, which is more fine-grained than prior counting benchmarks (Table 1). The evaluation of 20+ models under a unified protocol is valuable, and the high-density subset analysis (Table 4) reporting MAE and relative accuracy at 5%/10% thresholds is a useful diagnostic. The observation of the 'inverse performance paradox' on null-target prompting (§4.2) is an interesting and falsifiable finding. The dataset is publicly available, which supports reproducibility.

major comments (4)
  1. §4.2, 'Logical Execution vs. Baseline Perception Disconnect': The central causal claim that 'the primary bottleneck is not visual perception per se but the symbolic reasoning and arithmetic composition required after objects are perceived' is not adequately supported. Several analytical subsets (e.g., Differential Comparison, Joint-Set Aggregation) require two independent perception steps plus an arithmetic operation. Under a simple error-compounding null model with per-step perception accuracy p, expected accuracy is approximately p². For Qwen3.5-397B-A17B (atomic 92.9%), this predicts ~86.3% on Differential Comparison, but observed is 74.0% — a 12-point gap suggesting genuine reasoning failure. However, for Qwen3.5-27B (atomic 90.7%, predicted ~82.3%, observed 76.0%), the gap is only ~6 points. For InternVL3.5-38B (atomic 79.7%, predicted ~63.5%, observed 35.0%), the gap is ~28 points.
  2. §4.2 (continued): The picture is heterogeneous across models: for some, error compounding explains a large fraction of the drop; for others, it explains only a fraction. The paper does not perform this decomposition, yet makes a blanket causal claim about reasoning being the bottleneck. The authors should either (a) add an error-compounding analysis (e.g., comparing observed analytical accuracy against p² predictions derived from each model's atomic accuracy) and revise the causal claim to be model-specific, or (b) soften the claim to acknowledge that perception error compounding is a confound that is not separated from genuine reasoning failure.
  3. §3.3.1, §3.3.2, Appendix B.1: Gemini-3-Flash-Preview is used to generate QA pairs for the attribute-based semantic counting subsets and the analytical counting subsets (Figs. 9-10). The same model is then evaluated on the full benchmark including those subsets (Table 3: Gemini-3-Flash-Preview, 74.8% average). This creates a partial circularity. The paper states that human annotators verified all QA pairs, but no inter-annotator agreement is reported, and the potential advantage to the generation model is not discussed. The authors should (a) report inter-annotator agreement for the human verification stage, and (b) explicitly acknowledge the overlap between the generation model and evaluated models, noting whether Gemini-3-Flash-Preview's performance on the subsets it generated is comparable to or higher than its performance on subsets it did not generate.
  4. §3.3.3, High-Density Enumeration: This subset uses synthetically generated symbols on a blank canvas. The paper acknowledges this was done because real high-density benchmarks (e.g., ShanghaiTech) were 'ambiguous and difficult to verify.' However, synthetic symbols on a blank canvas lack the visual complexity of real dense scenes (overlapping textures, varying scales, cluttered backgrounds). The paper should explicitly acknowledge this as a limitation and note that results on this subset may not transfer to real-world dense counting. Currently, the subset is presented alongside real-image subsets without this caveat.
minor comments (4)
  1. Table 2: Several subsets have small sample sizes (78-102 samples). Per-subset accuracy comparisons based on ~80 samples have large confidence intervals (roughly ±10 percentage points). This should be noted when interpreting per-subset results, especially for Coordinate-Prompt Region Grounding (n=78) and Relative Canonical Orientation (n=84).
  2. §4.2, Table 3: Gemini-3.1-Pro-Preview scores 9.0% on Coordinate-Prompt Region Grounding, which is anomalously low compared to its other analytical scores (81-86%). This is likely a formatting or parsing issue rather than a genuine capability gap. The authors should investigate whether the model's coordinate-prompt responses are being correctly parsed.
  3. Figure 1 caption: The model leaderboard in the figure lists 'Qwen3.5-397B-A17B' with 76.9, but the font and layout make it hard to read. Consider using a separate table for the leaderboard rather than embedding it in the taxonomy figure.
  4. §3.3.3, Partial Object Occlusion: The text states 'IoU > 0.85' for heavy occlusion. This is ambiguous — IoU between what and what? If this refers to the overlap ratio of the object with other objects, this should be clarified.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The comments are well-taken and we will address each in a revised manuscript. Below we respond point by point.

read point-by-point responses

  1. Referee: §4.2 causal claim not adequately supported; error-compounding (p²) analysis missing; gap varies across models.

    Authors: The referee is correct. Our blanket causal claim that 'the primary bottleneck is not visual perception per se but symbolic reasoning and arithmetic composition' is not adequately supported without an error-compounding decomposition. The referee's own back-of-envelope calculations make this vivid: for some models (e.g., Qwen3.5-27B), the p² prediction from atomic accuracy accounts for a large fraction of the observed drop on Differential Comparison, while for others (e.g., InternVL3.5-38B), the residual gap is substantial. We agree that this heterogeneity undermines a single causal narrative. In the revision, we will add an error-compounding analysis: for each model and each analytical subset requiring two perception steps plus an arithmetic operation (Differential Comparison, Joint-Set Aggregation, Complementary Exclusion), we will compute the predicted accuracy under a simple p² null model (using each model's atomic counting accuracy as p) and report the residual gap. We will then revise the causal claim to be model-specific: for models where the residual is small, we will state that error compounding largely explains the drop; for models where the residual is large, we will identify genuine reasoning failure. The original blanket statement will be removed or substantially softened. revision: yes

  2. Referee: §4.2 (continued): perform the decomposition or soften the claim.

    Authors: This is the same point as above and we accept it fully. We will implement option (a): add the p² error-compounding analysis and revise the claim to be model-specific. We will also add a paragraph in §4.2 explicitly acknowledging perception error compounding as a confound that is not fully separated from genuine reasoning failure, and discuss the implications of the heterogeneous residuals across models. revision: yes

  3. Referee: Circular evaluation: Gemini-3-Flash-Preview generates QA pairs and is also evaluated; no inter-annotator agreement reported.

    Authors: This is a fair concern. We will address both sub-points. (a) Inter-annotator agreement: we will report inter-annotator agreement for the human verification stage. In our annotation process, a subset of samples was double-annotated by two independent annotators; we will compute and report Cohen's kappa (or percentage agreement) for this subset in the revision. (b) Generation-evaluation overlap: we will explicitly acknowledge in the manuscript that Gemini-3-Flash-Preview was used to generate QA pairs for the attribute-based semantic counting subsets and the analytical counting subsets, and that the same model is evaluated on the full benchmark. We will also add a comparative analysis: Gemini-3-Flash-Preview's average accuracy on the subsets it generated (Chromatic, Material, Scale, Action & State, General Semantic, and the four set-based analytical subsets) versus subsets it did not generate. From Table 3, Gemini-3-Flash-Preview scores 90.1/77.2/85.0/82.4/86.1/77.5 on the generated semantic subsets and 78.0/77.0/80.2 on the generated analytical subsets, with an overall average of 74.8% — which is comparable to other top models (e.g., Qwen3.5-397B-A17B at 76.9%) and does not show a dramatic advantage on the subsets it generated. We will present this comparison explicitly and note that while the partial circularity is a limitation, the human verification step and the absence of a pronounced performance advantage mitigate the concern. We will also add this as a stated limitation. revision: yes

  4. Referee: High-Density Enumeration uses synthetic symbols on blank canvas; lacks real-world visual complexity; should be acknowledged as a limitation.

    Authors: We agree. The synthetic high-density subset was designed to ensure unambiguous ground-truth counts, but the referee is correct that it lacks the visual complexity of real dense scenes (overlapping textures, varying scales, cluttered backgrounds). We will add an explicit limitation paragraph in §3.3.3 noting that results on this subset may not transfer to real-world dense counting scenarios, and that the synthetic design isolates the challenge of enumerating many items without the confound of visual clutter. We will also note this caveat in the discussion of Table 4 results in §4.2, clarifying that the catastrophic performance on this subset reflects difficulty with high cardinality in a controlled setting, and that real-world dense counting may present additional challenges. In future work (Appendix D), we will mention extending the high-density subset to real images as a priority. revision: yes

Circularity Check

1 steps flagged

Minor self-reference in data generation: Gemini-3-Flash-Preview generates QA pairs for some subsets and is also evaluated on the full benchmark, but human verification breaks the circularity chain.

specific steps
  1. fitted input called prediction [§3.3.1 (Attribution subsets), §3.3.2 (Analytical Counting), Appendix B.1 (Fig. 9-10)]
    "To construct attribute-based counting QA pairs, we prompt a strong MLLM, Gemini-3-Flash-Preview, to generate candidate questions and answers conditioned on each attribute type. We then employ another model as a verifier to check the correctness of the generated QA pairs, filtering out inconsistent or invalid ones. Finally, we conduct manual verification by human annotators to ensure high quality and reliability of the final QA dataset."

    Gemini-3-Flash-Preview is used to generate QA pairs for the Chromatic, Material, Scale, Action & State, General Semantic, and Analytical Counting subsets (Appendix B.1 confirms this LLM-assisted generation paradigm). The same model is then evaluated on the full HoloCount benchmark including those subsets (Table 3: Gemini-3-Flash-Preview scores 74.8% overall, 90.1% on Chromatic, 77.0% on Aggr., etc.). This creates a partial overlap where the evaluated model has potentially seen the data distribution during generation. However, the paper explicitly states that human annotators conduct manual verification of all generated QA pairs, and the ground-truth counts are independently verified. The generation model produces candidate QA pairs that are filtered and corrected by humans, so the final评估上

full rationale

The circularity concern is real but minor. Gemini-3-Flash-Preview generates QA pairs for some subsets and is then evaluated on the full benchmark. However, the paper's data pipeline includes explicit human verification of all QA pairs (§3.3: 'Each data sample undergoes manual review by expert annotators'), and a separate verifier model checks generated QA pairs before human review. The ground-truth answers are human-verified, not model-generated. This breaks the circularity chain: the evaluated model's outputs are compared against human-verified ground truth, not against its own generation. The concern is about data distribution familiarity, not about the prediction being equivalent to the input by construction. The paper's central claims about performance degradation from perception to reasoning are based on externally verified ground truth across 20+ models, most of which had no role in data generation. Score 2 reflects this minor, non-load-bearing self-reference.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No new entities, particles, or forces are introduced. This is a benchmark paper.

axioms (4)
  • domain assumption Exact match accuracy is the appropriate metric for evaluating counting competence
    §4.1: 'we report exact match accuracy as the percentage of samples where the predicted count exactly matches the ground-truth count.' This penalizes off-by-one errors equally with large errors, which may not reflect practical counting utility.
  • domain assumption The three-tier taxonomy (semantic, analytical, robustness) comprehensively covers the relevant dimensions of visual counting
    §3.2: The taxonomy is presented as comprehensive but does not include temporal/dynamic counting, grounded localization, or world-knowledge-dependent counting, which the paper itself acknowledges in Appendix D as future work.
  • domain assumption Human verification ensures ground-truth correctness for all 2,480 QA pairs
    §3.3: 'Each data sample undergoes manual review by expert annotators.' No inter-annotator agreement, annotation time, or quality control metrics are reported to validate this claim.
  • ad hoc to paper Synthetic symbol generation on blank canvas adequately represents high-density counting scenarios
    §3.3.3: 'we developed a script to generate synthetic symbols on a blank canvas, mimicking dense object distributions.' This substitutes real dense scenes with synthetic ones without validation that results transfer to real-world density.

pith-pipeline@v1.1.0-glm · 23690 in / 4076 out tokens · 253617 ms · 2026-07-08T06:10:51.951911+00:00 · methodology

0 comments
read the original abstract

Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.

Figures

Figures reproduced from arXiv: 2607.06420 by Guanglu Wan, Jinhong Deng, Limeng Qiao.

Figure 1
Figure 1. Figure 1: Taxonomy overview of the HoloCount benchmark. The dataset features three taxonomy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example tasks from the HoloCount benchmark, which comprises 20 fine-grained subtasks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The general data curation pipeline of the HoloCount benchmark. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of Qwen3.5 models in thinking versus non-thinking (instruct) mode on [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of ground-truth count values in HoloCount. The main plot shows the frequency [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Word cloud of visual concepts in HoloCount. The benchmark covers 1,481 unique visual [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt used for visual concept extraction via GPT-3.5-turbo. The system prompt instructs [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Object scale distribution in the small-scale enumeration subset (101 samples, 995 boxes). [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt used for generating attribute-based conditional counting QA pairs. The model [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt used for generating reasoning-based counting QA pairs. The model is instructed [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative error examples of Gemini-3.1-Pro-Preview on [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative error examples of Gemini-3.1-Pro-Preview on [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative error examples of Gemini-3.1-Pro-Preview on [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 66 canonical work pages · 9 internal anchors

  1. [1]

    Almustafa Abed, Belhassen Akrout, and Ikram Amous. Convolutional neural network for head segmenta- tion and counting in crowded retail environment using top-view depth images.Arabian Journal for Science and Engineering, 49(3):3735–3749, 2024

    work page 2024

  2. [2]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084, 2019

    work page 2019

  3. [3]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Qwen3-VL Technical Report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025

    work page 2025

  6. [6]

    Mvi-bench: A comprehensive benchmark for evaluating robustness to misleading visual inputs in lvlms, 2026

    Huiyi Chen, Jiawei Peng, Dehai Min, Changchang Sun, Kaijie Chen, Yan Yan, Xu Yang, and Lu Cheng. Mvi-bench: A comprehensive benchmark for evaluating robustness to misleading visual inputs in lvlms, 2026

    work page 2026

  7. [7]

    Spatialrgpt: Grounded spatial reasoning in vision-language models.Advances in Neural Information Processing Systems, 37:135062–135093, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models.Advances in Neural Information Processing Systems, 37:135062–135093, 2024

    work page 2024

  8. [8]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 91–104, 2025

    work page 2025

  9. [9]

    Deep learning in crowd counting: A survey.CAAI Trans

    Lijia Deng, Qinghua Zhou, Shuihua Wang, Juan Manuel Górriz, and Yudong Zhang. Deep learning in crowd counting: A survey.CAAI Trans. on Intell. Tech., 9(5):1043–1077, 2024

    work page 2024

  10. [10]

    Lvis: A dataset for large vocabulary instance segmentation, 2019

    Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation, 2019

    work page 2019

  11. [11]

    Drone-based object counting by spatially regularized regional proposal network

    Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu. Drone-based object counting by spatially regularized regional proposal network. InProceedings of the IEEE international conference on computer vision, pages 4145–4153, 2017. 11

    work page 2017

  12. [12]

    Step3-vl-10b technical report.arXiv preprint arXiv:2601.09668, 2026

    Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, Hangyu Guo, Haoran Lv, Hongyu Zhou, Jia Wang, Jian Zhou, Jianjian Sun, et al. Step3-vl-10b technical report.arXiv preprint arXiv:2601.09668, 2026

    work page arXiv 2026

  13. [13]

    Kevin Zhou, and Jie Chen

    Zhongyi Huang, Yao Ding, Guoli Song, Lin Wang, Ruizhe Geng, Hongliang He, Shan Du, Xia Liu, Yonghong Tian, Yongsheng Liang, S. Kevin Zhou, and Jie Chen. BCData: A large-scale dataset and benchmark for cell detection and counting. pages 289–298. Springer, 2020

    work page 2020

  14. [14]

    Composition loss for counting, density map estimation and localization in dense crowds

    Haroon Idrees, Muhmmad Taber, Khter Ceez, Dong Dng, Ali Mdnni, and Mubarak Lng. Composition loss for counting, density map estimation and localization in dense crowds. InProceedings of the European Conference on Computer Vision (ECCV), pages 532–546, 2018

    work page 2018

  15. [15]

    Vlind-bench: Measuring language priors in large vision-language models, 2025

    Kang il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Measuring language priors in large vision-language models, 2025

    work page 2025

  16. [16]

    T-rex2: Towards generic object detection via text-visual prompt synergy, 2024

    Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy, 2024

    work page 2024

  17. [17]

    Learning To Count Objects in Images

    Victor Lempitsky and Andrew Zisserman. Learning To Count Objects in Images. InAdv. Neural Inform. Process. Syst., pages 1324–1332, 2010

    work page 2010

  18. [18]

    Count anything at any granularity, 2026

    Chang Liu, Haoning Wu, and Weidi Xie. Count anything at any granularity, 2026

    work page 2026

  19. [19]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    work page 2023

  20. [20]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean conference on computer vision, pages 38–55. Springer, 2024

    work page 2024

  21. [21]

    Moment-video: Diagnosing temporal fidelity of video mllms on momentary visual events, 2026

    Xiaolin Liu, Yilun Zhu, Xiangyu Zhao, Xuehui Wang, Yan Li, Xin Li, Haoyu Cao, Xing Sun, Shaofeng Zhang, Xu Yang, Zhihang Zhong, and Xue Yang. Moment-video: Diagnosing temporal fidelity of video mllms on momentary visual events, 2026

    work page 2026

  22. [22]

    Understanding language prior of lvlms by contrasting chain-of-embedding.arXiv preprint arXiv:2509.23050, 2025

    Lin Long, Changdae Oh, Seongheon Park, and Sharon Li. Understanding language prior of lvlms by contrasting chain-of-embedding.arXiv preprint arXiv:2509.23050, 2025

    work page arXiv 2025

  23. [23]

    Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for mllms, 2025

    Lidong Lu, Guo Chen, Zhiqi Li, Yicheng Liu, and Tong Lu. Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for mllms, 2025

    work page 2025

  24. [24]

    Deep learning for site safety: Real-time detection of personal protective equipment.Automation in construction, 112:103085, 2020

    Nipun D Nath, Amir H Behzadan, and Stephanie G Paal. Deep learning for site safety: Real-time detection of personal protective equipment.Automation in construction, 112:103085, 2020

    work page 2020

  25. [25]

    Can current ai models count what we mean, not what they see? a benchmark and systematic evaluation, 2025

    Gia Khanh Nguyen, Yifeng Huang, and Minh Hoai. Can current ai models count what we mean, not what they see? a benchmark and systematic evaluation, 2025

    work page 2025

  26. [26]

    Few-shot object counting and detection

    Thanh Nguyen, Chau Pham, Khoi Nguyen, and Minh Hoai. Few-shot object counting and detection. In European Conference on Computer Vision, pages 348–365. Springer, 2022

    work page 2022

  27. [27]

    Small object detection: A comprehensive survey on challenges, techniques and real-world applications.Intelligent Systems with Applications, 27:200561, 2025

    Mahya Nikouei, Bita Baroutian, Shahabedin Nabavi, Fateme Taraghi, Atefe Aghaei, Ayoob Sajedi, and Mohsen Ebrahimi Moghaddam. Small object detection: A comprehensive survey on challenges, techniques and real-world applications.Intelligent Systems with Applications, 27:200561, 2025

    work page 2025

  28. [28]

    Enhancing object detection in smart video surveillance: A survey of occlusion-handling approaches.Electronics, 13(3):541, 2024

    Zainab Ouardirhi, Sidi Ahmed Mahmoudi, and Mostapha Zbakh. Enhancing object detection in smart video surveillance: A survey of occlusion-handling approaches.Electronics, 13(3):541, 2024

    work page 2024

  29. [29]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. InProceedings of the IEEE/CVF international conference on computer vision, pages 3170–3180, 2023

    work page 2023

  30. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763, 2021

    work page 2021

  31. [31]

    Learning To Count Everything

    Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning To Count Everything. InIEEE Conf. Comput. Vis. Pattern Recog., pages 3394–3403, 2021

    work page 2021

  32. [32]

    Unicbench: Unified counting benchmark for mllm, 2026

    Chenggang Rong, Tao Han, Zhiyuan Zhao, Yaowu Fan, Jia Wan, Song Guo, Yuan Yuan, and Junyu Gao. Unicbench: Unified counting benchmark for mllm, 2026. 12

    work page 2026

  33. [33]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    work page 2019

  34. [34]

    Sindagi, Rajeev Yasarla, and Vishal M

    Vishwanath A. Sindagi, Rajeev Yasarla, and Vishal M. Patel. JHU-CROWD++: Large-scale crowd counting dataset and a benchmark method.IEEE Trans. Pattern Anal. Mach. Intell., 44(5):2594–2609, 2022

    work page 2022

  35. [35]

    CountQA: How Well Do MLLMs Count in the Wild?

    Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. Countqa: How well do mllms count in the wild?arXiv preprint arXiv:2508.06585, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Gemma 3 technical report, 2025

    Gemma Team. Gemma 3 technical report, 2025

    work page 2025

  37. [37]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025

    InternVL Team. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025

    work page 2025

  38. [38]

    Kimi k2.5: Visual agentic intelligence, 2026

    Kimi Team. Kimi k2.5: Visual agentic intelligence, 2026

    work page 2026

  39. [39]

    Kimi-VL Technical Report

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    Kwai Keye-VL Technical Report

    Kwai Keye Team, Biao Yang, Bin Wen, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai keye-vl technical report.arXiv preprint arXiv:2507.01949, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    Qwen3.5: Accelerating productivity with native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026

    work page 2026

  42. [42]

    Ec-bench: Enumeration and counting benchmark for ultra-long videos, 2026

    Fumihiko Tsuchiya, Taiki Miyanishi, Mahiro Ukai, Nakamasa Inoue, Shuhei Kurita, Yusuke Iwasawa, and Yutaka Matsuo. Ec-bench: Enumeration and counting benchmark for ultra-long videos, 2026

    work page 2026

  43. [43]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Nwpu-crowd: A large-scale benchmark for crowd counting and localization.IEEE transactions on pattern analysis and machine intelligence, 43(6):2141–2149, 2020

    Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li. Nwpu-crowd: A large-scale benchmark for crowd counting and localization.IEEE transactions on pattern analysis and machine intelligence, 43(6):2141–2149, 2020

    work page 2020

  45. [45]

    A review of further directions for artificial intelligence, machine learning, and deep learning in smart logistics.Sustainability, 12(9):3760, 2020

    Manuel Woschank, Erwin Rauch, and Helmut Zsifkovits. A review of further directions for artificial intelligence, machine learning, and deep learning in smart logistics.Sustainability, 12(9):3760, 2020

    work page 2020

  46. [46]

    LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models

    Zongyu Wu, Yuwei Niu, Hongcheng Gao, Minhua Lin, Zhiwei Zhang, Zhifang Zhang, Qi Shi, Yilong Wang, Sike Fu, Junjie Xu, et al. Lanp: Rethinking the impact of language priors in large vision-language models.arXiv preprint arXiv:2502.12359, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [47]

    Towards visual grounding: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    work page 2025

  48. [48]

    Learning to separate: Detecting heavily-occluded objects in urban scenes

    Chenhongyi Yang, Vitaly Ablavsky, Kaihong Wang, Qi Feng, and Margrit Betke. Learning to separate: Detecting heavily-occluded objects in urban scenes. InEuropean conference on computer vision, pages 530–546. Springer, 2020

    work page 2020

  49. [49]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    work page 2023

  50. [50]

    Learning to Count Objects in Natural Images for Visual Question Answering

    Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Learning to count objects in natural images for visual question answering.arXiv preprint arXiv:1802.05766, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  51. [51]

    MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?arXiv preprint arXiv:2408.13257, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    yellow flower

    Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 589–597, 2016. 13 A Benchmark Statistics In this section, we conduct a thorough statistical analysis of our benchmark to provide a...

    work page 2016

  53. [53]

    Return ONLY the visual concept noun phrases, separated by commas if multiple

    work page

  54. [54]

    chair” not “chairs

    Use singular form (e.g., “chair” not “chairs”, “tile” not “tiles”)

    work page

  55. [55]

    yellow flower

    Include modifiers that distinguish the concept (e.g., “yellow flower”, “action card”, “wooden chair”) but remove spatial/location phrases (e.g., “on the wall”, “in the foreground”)

    work page

  56. [56]

    How many more X than Y?

    For comparison questions (e.g., “How many more X than Y?”), extract all compared concepts

    work page

  57. [57]

    Do NOT include articles (a, an, the) or quantities

    work page

  58. [58]

    Output ONLY the concept(s), nothing else. Few-shot Examples Q:How many tiles are on the wall with the shower? A:tile Q:How many chairs and tables are there? A:chair, table Q:How many more action cards are there than cat cards? A:action card, cat card Q:How many yellow flowers are there? A:yellow flower Q:Ignore the tubs in the back. How many tubs are ther...

    work page

  59. [59]

    How many red cars?

    Capability: All questions must beconditional_countingwith anattributeconstraint (what to count), not spatial (where to count). Use exactly one attribute type per question. Attribute sub_capability and priority when multiple apply: • color: object color, hair/skin/appearance (e.g. “How many red cars?”, “How many people with blonde hair?”). Use for any colo...

    work page

  60. [60]

    Count must be correct for this image

    Content: Questions must be answerable by counting visible objects in the image that satisfy the attribute. Count must be correct for this image. Vary the attribute types across the list (include at least color and one other type if the image allows). Figure 9: Prompt used for generating attribute-based conditional counting QA pairs. The model receives thi...

    work page

  61. [61]

    How many more red cars than blue cars?

    comparison: Compare quantities between two or more categories (who has more/fewer, or by how many). Examples: “How many more red cars than blue cars?”, “Which color has the most balls?” 2.category_kind: Ask “how many kinds/types” (count of distinct categories, NOT instance count). Examples: “How many different fruits are there?”, “How many types of vehicl...

    work page

  62. [62]

    How many objects are both red and large?

    other_reasoning: Other complex reasoning: multi-step analysis, joint attributes, cross-region counting. Examples: “How many objects are both red and large?” Response Format: Output one JSON object per line: {“question”: “<English counting question>”, “count”: “<number or short text>”, “sub_capability”: “<one of: comparison, category_kind, part_whole, aggr...

    work page

  63. [63]

    No external knowledge

    All questions must be answerable by analyzing visible objects in the image. No external knowledge

    work page

  64. [64]

    Cover at least 3 different sub_capabilities across the set; vary question types

    work page

  65. [65]

    Ensure counts and comparisons are correct for this specific image

    work page

  66. [66]

    How many objects satisfy condition A but not condition B, and are larger than the average size of category C?

    Avoid trivial direct-count questions. Focus on questions that require comparison, aggregation, exclusion, or multi-step reasoning. Figure 10: Prompt used for generating reasoning-based counting QA pairs. The model is instructed to produce diverse question types covering comparison, aggregation, exclusion, and category-level reasoning. Multi-step Compositi...

    work page