Pith. sign in

REVIEW 2 major objections 5 minor 63 references

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

T0 review · 2 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VLMs often already know the correct object count inside their activations; the failure is a misaligned readout that simple probes can detect and selectively correct.

desk verdict Solid multi-probe extension of Sun et al. to VLM counting: count is often present but misaligned, with causal steering and selective re-prompt gains up to 15.6 pp that hold on CountBench. read the letter →

arxiv 2607.09544 v1 pith:KHIB5STM submitted 2026-07-10 cs.CV cs.LG

Ahmed Oumar El-Shangiti , Abzal Nurgazy , Hilal AlQuabeh , Nikolai Rozanov , Kentaro Inui This is my paper
classification cs.CVcs.LG
keywords vision-languagemodelsobjectcountingactivationprobingSVCCAsteeringself-correctionerrordetectionrepresentationalalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models still fail at basic object counting even when they handle harder multimodal tasks. This paper asks whether the correct count is simply missing from the model’s internal representations or whether the count is present but fails to reach the final verbal answer. Across four models and five datasets, nonlinear probes trained on intermediate activations recover the ground-truth count and flag counting errors with high reliability, even on trials where the model’s own output is wrong. Weight-space SVCCA shows that the subspace aligned with the true count and the subspace aligned with the model’s verbalized count are only partially shared and often nearly orthogonal for nonlinear probes. Strengthening the count-aligned direction by activation steering improves accuracy, and a practical detector-guided self-correction loop that re-prompts only when an internal error detector fires raises counting accuracy by as much as 15.6 absolute points with no parameter updates. The work therefore treats counting failures as a readout-alignment problem rather than a pure knowledge gap, and turns that diagnosis into an inference-time fix.

What carries the argument

Multi-probe activation analysis (ground-truth count probe, output-count probe, and binary error detector) together with weight-space SVCCA comparison of their readout subspaces and detector-gated re-prompting at the best detector layer.

What would settle it

If error detectors trained on the same activations fail to predict counting mistakes better than chance on held-out real images, or if steering along the recovered count direction never improves accuracy above random-direction controls across models and datasets, the claim that the count is present but misaligned would be falsified.

Watch

Extended reading notes

Core claim

Nonlinear probes on intermediate VLM activations can recover the true object count and detect counting errors even when the model utters the wrong number; the ground-truth and output readout subspaces are partially shared but misaligned, and strengthening the count direction (by steering or selective re-prompting) improves counting performance without any weight updates.

Load-bearing premise

High probe accuracy and successful steering at the single best layer chosen by held-out F1 truly reflect causally relevant count representations rather than dataset correlations or post-hoc layer selection.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that VLM counting failures often reflect a misalignment between internal count representations and verbalized outputs rather than absent knowledge. Across four VLMs and five datasets (four synthetic regimes plus CountBench), nonlinear probes on intermediate activations recover ground-truth counts and detect errors even when the model is wrong. SVCCA on probe weight matrices shows that ground-truth and output readout subspaces are only partially shared and often misaligned, especially for nonlinear probes. Causal activation steering along the ground-truth probe direction improves accuracy on synthetic data for InternVL2-1B (with random-direction controls), and a detector-guided selective re-prompting method yields up to +15.6 absolute percentage points without parameter updates, outperforming budget-matched random intervention and correlating with detector F1.

Significance. If the result holds, the work supplies both a practical inference-time correction tool for a known VLM weakness and a mechanistic account of the internal-knowledge vs. output gap. Strengths include multi-model and multi-dataset evaluation, three random seeds, random-steering and Random-K/Always-Reprompt baselines, explicit correlation between detector quality and correction gain, and extension of the multi-probe arithmetic framework of Sun et al. (2025) to VLMs. The combination of probing, subspace comparison, causal intervention, and selective correction is a coherent contribution for the field.

major comments (2)
  1. Section 4 and Figure 4 report the only causal steering experiment, and only for InternVL2-1B on the four synthetic datasets. The central claim that recovered directions are causally relevant count subspaces (not merely predictive or dataset-specific) therefore rests on a narrow intervention. The manuscript should either extend steering to at least one larger model and/or CountBench, or substantially qualify the causal claim and treat steering as supporting evidence for InternVL2-1B only.
  2. Section 3.3 selects the best layer l* by held-out F1 for each probe/detector, and all subsequent SVCCA, steering, and correction results use that post-hoc choice. Combined with Appendix C’s own admission that vision-encoder error detectors can exploit scene-complexity confounds, this leaves open whether high probe F1 and the Table 1 gains partly reflect selection of layers where count co-varies with other visual statistics. A fixed-layer or early-stopped protocol (or an ablation showing gains persist under non-oracle layer choice) is needed to make the “count is there but misaligned” claim robust.
minor comments (5)
  1. Abstract and §1 state “up to 15.6% absolute percentage points”; Table 1 correctly uses percentage points. Standardize the wording.
  2. Figure 1 caption and pipeline text are dense; a short legend distinguishing the three probe targets (gt / output / error) would help.
  3. τ is fixed at 0.5 (§5) with no sensitivity analysis; a brief threshold sweep or ROC-style note would strengthen the method section.
  4. Limitations (§7) correctly flag the 1–9 range, model size, and single-model steering; these should be echoed more clearly when interpreting Table 1 gains for Qwen models, which are already strong.
  5. Typographical inconsistencies (e.g., “V o et al.”, mixed “15.6%” vs “15.6 absolute percentage points”) should be cleaned in a revision.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: probes, SVCCA, steering, and detector-guided gains are empirical measurements against held-out ground truth, not reductions of outputs to fitted inputs by construction.

full rationale

The paper's chain is an empirical analysis pipeline, not a derivation that closes on its own definitions. Probes (circular/linear/logistic/MLP) are trained with cross-entropy or smooth-L1 on ground-truth counts y, model outputs ŷ, or binary error e = I[ŷ ≠ y] using activations from a train split; performance is reported as held-out F1 (Fig. 2, Sec. 3.3–3.4). SVCCA compares the principal singular-vector subspaces of independently trained weight matrices W_gt and W_out (Eqs. 4–8, Sec. 3.2, Fig. 3); the low scores for nonlinear probes are an observed geometric fact, not forced by the definition of either probe. Causal steering adds a scaled probe-derived direction d = (w_y − w_ŷ)/‖…‖ at the best layer and measures accuracy change versus an unsteered baseline and a random-direction control (Eq. 1, Sec. 4, Fig. 4); improvement is an experimental outcome, not an algebraic identity. Detector-guided re-prompting thresholds the error-probe score s and re-queries only when s ≥ τ, then reports absolute accuracy lift versus Always-Reprompt and Random-K baselines on the same ground-truth labels (Eqs. 2–3, Table 1, Sec. 5). Layer selection by validation F1 and the narrow scope of the steering experiment (InternVL2-1B, synthetic data) are methodological limitations that affect causal generality, but they do not make any reported number equal to a fitted constant by construction. The sole self-citations (extension of Sun et al. 2025 multi-probe setup; related numeracy geometry work) supply methodological precedent and are not uniqueness theorems that forbid alternatives or close the loop on the counting results. Against external ground-truth benchmarks the pipeline is self-contained; score remains near zero.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The work is empirical and rests on standard probing assumptions plus a handful of free choices (best-layer selection, steering strength α, detector threshold τ=0.5, probe architectures). No new physical or mathematical entities are postulated; the 'misaligned subspaces' are observed, not invented. Free parameters are those fitted or selected on validation data that affect the reported gains.

free parameters (4)
  • best layer l*
    Chosen as argmax_l F1 on held-out validation for each probe/detector; all subsequent steering and correction use this selected layer.
  • steering strength α
    Swept over [5,65]; reported gains depend on operating inside the 'effective steering zone'.
  • error-detector threshold τ
    Fixed at 0.5 for the main self-correction results; controls the intervention budget K.
  • probe architecture hyperparameters
    MLP hidden size 512, circular 2-D projection, logistic per-class weights, training for 2000 epochs with weighted CE; these choices determine reported F1 and downstream gains.
assumptions (3)
  • domain assumption Linear or mildly nonlinear probes can recover information that is linearly (or near-linearly) encoded in frozen activations without the probe itself solving the task from scratch.
    Standard probing assumption (Alain & Bengio 2017; Hewitt & Liang 2019); invoked throughout Section 3 and used to interpret high F1 as evidence of internal count encoding.
  • ad hoc to paper SVCCA on probe weight matrices (rather than activations) measures geometric alignment of supervision-specific readout subspaces.
    Authors note they are the first to apply SVCCA this way; the interpretation that low weight-SVCCA implies misaligned count vs. output directions rests on this methodological choice (Section 3.2).
  • domain assumption Greedy decoding and zero-shot prompting produce representative counting behavior for the four evaluated VLMs.
    All behavioral baselines and corrections are obtained under this protocol (Section 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs." pith.science (2026). https://pith.science/paper/KHIB5STM

@misc{pith2026260709544,
  author       = {Pith},
  title        = {Pith review of: The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHIB5STM}},
  note         = {Machine review of arXiv:2607.09544}
}
read the original abstract

Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal knowledge or a gap between internal representations and verbalized outputs. Training simple probes on activations from four VLMs across five counting datasets reveals that nonlinear probes can reliably detect counting errors, suggesting that VLMs often encode the correct count even when they output the wrong answer. SVCCA analysis shows that probes trained on ground-truth counts and probes trained on model outputs occupy a partially shared activation subspace but read out along misaligned directions. We further validate our findings using a causal steering intervention, proving that strengthening the direction of count-identified probes does improve model counting performance. Motivated by this result, we propose a detector-guided self-correction method that selectively re-prompts the model only when an internal error detector predicts failure. This simple inference-time intervention improves counting accuracy by up to 15.6 absolute percentage points, without any parameter updates. Our results establish activation-based error probing as both a practical tool for improving VLM counting and a mechanistic lens on the gap between internal knowledge and model outputs.

Figures

Figures reproduced from arXiv: 2607.09544 by the authors.

Figure 1
Figure 1. Overview of the activation-based probing, intervention, and self-correction framework. The pipeline [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Layer-wise probing and detector performance [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Weight-space SVCCA analysis of gt_probe versus output_probe. (Left) Mean SVCCA by probe family across the full 4-model, 4-dataset, 3-seed sweep. MLP and Circular probes lie in the near-orthogonal regime, while Linear probes show substantially higher alignment. (Right) SVCCA versus best-layer gap |lgt − lout|. Larger layer disagreement tends to coincide with lower alignment, but low SVCCA persists even at small layer… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Causal steering results for InternVL2-1B on four synthetic counting datasets. Green shading marks the effective steering zone where probe-derived steering exceeds the baseline; pink shading marks the destructive zone where steering falls below the baseline. rather than…
Figure 5
Figure 5. Figure 5: Detector F1 vs. correction gain (∆Acc). Each point is one run (model × dataset × detector × seed). Spearman ρ = 0.803. The system prompt used for all passes is: System Prompt You are a helpful assistant. Output only the final count in the format <count>. B.1 Experiment…
Figure 6
Figure 6. Figure 6: Layer-wise probing and detector performance [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 8
Figure 8. Figure 8: Layer-wise probing and detector performance [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 10
Figure 10. Figure 10: Layer-wise probing and detector perfor￾mance across model depth. Top: ground-truth probe F1 by layer. Middle: output-supervised probe F1 by layer. Bottom: error-detector F1 by layer. Curves are aggregated across models, and the three random seeds. CountBench dataset f…
Figure 11
Figure 11. Figure 11: Detector F1 vs. correction gain (∆Acc). Each point is one run. The dataset is CountBench Dataset Model MLP Log-S Circ-J Circ-S DC-DS InternVL2-1B +5.27 +6.41 +5.13 +4.42 InternVL2-4B +15.10 +15.95 +15.81 +15.53 Qwen3-VL-2B +0.57 +0.43 +0.57 +0.57 Qwen3-VL-8B +4.13 +5.…
Figure 12
Figure 12. Figure 12: Layer-wise SVCCA similarity scores for various Vision Language Models and Datasets. High scores [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Weight-space SVCCA analysis of gt_probe versus output_probe. (Left) Mean SVCCA by probe family across the full 4-model, 3-seed sweep. MLP and Circular probes lie in the near-orthogonal regime, while Linear probes show substantially higher alignment. (Right) SVCCA vers…
Figure 14
Figure 14. Figure 14: Activation-space SVCCA analysis of gt_probe versus output_probe. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: This figure illustrates performance aggre [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 17
Figure 17. Figure 17: This figure illustrates performance aggre [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 19
Figure 19. Figure 19: This figure illustrates performance aggre [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 21
Figure 21. Figure 21: This figure illustrates performance aggre [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 2 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

Show all 63 references
  1. [9]

    2019 , doi =

    Acharya, Manoj and Kafle, Kushal and Kanan, Christopher , booktitle =. 2019 , doi =

    2019

  2. [10]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Counting Everyday Objects in Everyday Scenes , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  3. [11]

    Teaching

    Paiss, Roni and Ephrat, Ariel and Tov, Omer and Zada, Shiran and Mosseri, Inbar and Irani, Michal and Dekel, Tali , booktitle =. Teaching

  4. [12]

    2023 , eprint=

    Teaching CLIP to Count to Ten , author=. 2023 , eprint=

    2023

  5. [13]

    Computational Linguistics , volume =

    Probing Classifiers: Promises, Shortcomings, and Advances , author =. Computational Linguistics , volume =. 2022 , publisher =

    2022

  6. [14]

    2019 , address =

    Tenney, Ian and Das, Dipanjan and Pavlick, Ellie , booktitle =. 2019 , address =

    2019

  7. [15]

    Proceedings of the 2019 Conference of the North

    A Structural Probe for Finding Syntax in Word Representations , author =. Proceedings of the 2019 Conference of the North. 2019 , address =

    2019

  8. [16]

    Locating and Editing Factual Associations in

    Meng, Kevin and Bau, David and Andonian, Alex and Belinkov, Yonatan , booktitle =. Locating and Editing Factual Associations in. 2022 , publisher =

    2022

  9. [17]

    2017 , publisher =

    Raghu, Maithra and Gilmer, Justin and Yosinski, Jason and Sohl-Dickstein, Jascha , booktitle =. 2017 , publisher =

    2017

  10. [18]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Similarity of Neural Network Representations Revisited , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , volume =

    2019

  11. [19]

    Advances in Neural Information Processing Systems , volume =

    Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , author =. Advances in Neural Information Processing Systems , volume =. 2023 , publisher =

    2023

  12. [20]

    Advances in Neural Information Processing Systems , volume =

    Self-Refine: Iterative Refinement with Self-Feedback , author =. Advances in Neural Information Processing Systems , volume =. 2023 , publisher =

    2023

  13. [21]

    Advances in Neural Information Processing Systems , volume =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =. 2023 , publisher =

    2023

  14. [22]

    2025 , eprint=

    Probing for Arithmetic Errors in Language Models , author=. 2025 , eprint=

    2025

  15. [23]

    Do Large Language Models Know What They Don

    Cao, Ruidi and others , journal =. Do Large Language Models Know What They Don. 2024 , note =

    2024

  16. [24]

    arXiv preprint , year=

    Probing the Difficulty Perception Mechanism of Large Language Models , author=. arXiv preprint , year=

  17. [25]

    arXiv preprint , year=

    Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification , author=. arXiv preprint , year=

  18. [26]

    arXiv preprint , year=

    Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention , author=. arXiv preprint , year=

  19. [27]

    The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces

    El-Shangiti, Ahmed Oumar and Hiraoka, Tatsuya and AlQuabeh, Hilal and Heinzerling, Benjamin and Inui, Kentaro. The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces. Proceedings of the 2025 Conference of the Nations of the Americas...

    2025 doi

  20. [28]

    Monotonic Representation of Numeric Attributes in Language Models

    Heinzerling, Benjamin and Inui, Kentaro. Monotonic Representation of Numeric Attributes in Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024. doi:10.18653/v1/2024.acl-short.18

    2024 doi

  21. [29]

    2025 , eprint=

    Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting , author=. 2025 , eprint=

    2025

  22. [30]

    2017 , eprint=

    Adam: A Method for Stochastic Optimization , author=. 2017 , eprint=

    2017

  23. [31]

    International Conference on Learning Representations (ICLR) Workshop , year =

    Understanding Intermediate Layers Using Linear Classifier Probes , author =. International Conference on Learning Representations (ICLR) Workshop , year =

  24. [32]

    Advances in Neural Information Processing Systems , volume =

    Insights on Representational Similarity in Neural Networks with Canonical Correlation , author =. Advances in Neural Information Processing Systems , volume =. 2018 , publisher =

    2018

  25. [33]

    Proceedings of the 40th International Conference on Machine Learning , series =

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , publisher =

    2023

  26. [34]

    Advances in Neural Information Processing Systems , volume =

    Visual Instruction Tuning , author =. Advances in Neural Information Processing Systems , volume =

  27. [35]

    Evaluating Object Hallucination in Large Vision-Language Models: The

    Li, Yifan and Du, Yifan and Zhou, Kun and Wang, Jinpeng and Zhao, Wayne Xin and others , journal =. Evaluating Object Hallucination in Large Vision-Language Models: The

  28. [36]

    Fu, Chaoyou and Dai, Yujie and Luo, Yinpeng and Li, Liang and Ren, Shuhuai and Zhang, Runpeng and Wang, Zihan and Zhou, Chenyang and Shen, Yadong and Zhang, Meng and others , journal =

  29. [37]

    Liu, Yang and Du, Wenhai and Zhang, Kai and Li, Xiaoxin and Hu, Jinghao and Liu, Xiang and Zhou, Zicheng and He, Yuan and Qiu, Zejun and others , journal =

  30. [38]

    Yu, Wenhao and Chen, Yiming and Wu, Xiang and He, Zhen and Liu, Yuhang and Zhao, Xu and others , journal =

  31. [39]

    Yue, Xi and Ni, Yuan and Zhang, Kai and Zheng, Tong and Liu, Yang and Gao, Wenxuan and others , journal =

  32. [40]

    arXiv preprint arXiv:2207.05221 , year =

    Language Models (Mostly) Know What They Know , author =. arXiv preprint arXiv:2207.05221 , year =

    arXiv

  33. [41]

    arXiv preprint arXiv:2212.03827 , year =

    Discovering Latent Knowledge in Language Models Without Supervision , author =. arXiv preprint arXiv:2212.03827 , year =

    arXiv

  34. [42]

    International Conference on Learning Representations , year =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. International Conference on Learning Representations , year =

  35. [43]

    arXiv preprint arXiv:2305.10601 , year =

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author =. arXiv preprint arXiv:2305.10601 , year =

    arXiv

  36. [44]

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =

  37. [45]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  38. [46]

    2025 , eprint=

    Qwen3-VL Technical Report , author=. 2025 , eprint=

    2025

  39. [47]

    Advances in Neural Information Processing Systems , volume =

    PyTorch: An Imperative Style, High-Performance Deep Learning Library , author =. Advances in Neural Information Processing Systems , volume =

  40. [48]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Transformers: State-of-the-Art Natural Language Processing , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    2020

  41. [49]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , year =

    The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , year =. doi:10.18653/v1/2025.emnlp-main.1495 , url =

    2025 doi

  42. [50]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

    VisNumBench: Evaluating Number Sense of Multimodal Large Language Models , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

  43. [51]

    2025 , eprint=

    [De|Re]constructing VLMs' Reasoning in Counting , author=. 2025 , eprint=

    2025

  44. [52]

    2025 , eprint=

    Understanding Counting Mechanisms in Large Language and Vision-Language Models , author=. 2025 , eprint=

    2025

  45. [53]

    2025 , eprint=

    Vision Language Models are Biased , author=. 2025 , eprint=

    2025

  46. [54]

    2025 , eprint=

    Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions , author=. 2025 , eprint=

    2025

  47. [55]

    2024 , eprint=

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks , author=. 2024 , eprint=

    2024

  48. [56]

    2024 , eprint=

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts , author=. 2024 , eprint=

    2024

  49. [57]

    2023 , eprint=

    Visual Instruction Tuning , author=. 2023 , eprint=

    2023

  50. [58]

    Golub , journal =

    Åke Björck and Gene H. Golub , journal =. Numerical Methods for Computing Angles Between Linear Subspaces , urldate =

  51. [59]

    2022 , eprint=

    Toy Models of Superposition , author=. 2022 , eprint=

    2022

  52. [60]

    2025 , eprint=

    Inside-Out: Hidden Factual Knowledge in LLMs , author=. 2025 , eprint=

    2025

  53. [61]

    2025 , eprint=

    LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations , author=. 2025 , eprint=

    2025

  54. [62]

    S tate A ct: Enhancing LLM Base Agents via Self-prompting and State-tracking

    Rozanov, Nikolai and Rei, Marek. S tate A ct: Enhancing LLM Base Agents via Self-prompting and State-tracking. Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025). 2025. doi:10.18653/v1/2025.realm-1.27

    2025 doi

  55. [63]

    2019 , eprint=

    Designing and Interpreting Probes with Control Tasks , author=. 2019 , eprint=

    2019

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.