Pith. sign in

REVIEW 5 major objections 4 minor 85 references

Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that state-of-the-art vision-language models, including reasoning-heavy models, fail at atomic 2D geometric perception tasks that are trivially easy for adults, and it introduces a 13,188-question benchmark, AVSD, that…

desk verdict Useful dataset and taxonomy, but the headline claim runs ahead of the numbers and the showcase figure contains an answer-key error that needs a full audit before the benchmark can be trusted. read the letter →

arxiv 2505.20021 v1 pith:53X3API7 submitted 2025-05-26 cs.CV cs.AI

Hyunsik Chae , Seungwoo Yoon , Jaden Park , Chloe Yewon Chun , Yongin Cho , Mu Cai , Yong Jae Lee , Ernest K. Ryu This is my paper
classification cs.CVcs.AI
keywords atomicvisualskillsvision-languagemodelsgeometricperception2DEuclideangeometrybenchmarkdatasetskilldecompositionstylerobustnessversusreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper decomposes visual comprehension in 2D Euclidean geometry into 36 atomic visual skills—skills like recognizing tangency, parallel lines, angles, and connectedness—and argues that every complex geometry diagram is perceived through combinations of these units. To test this, the authors build the Atomic Visual Skills Dataset (AVSD), with 13,188 problems split across handcrafted, synthetic textbook-style, and style-augmented images, each question aimed at one skill. Benchmarking current open and commercial vision-language models, they find that accuracy tops out near 72 percent for the best model and drops to 30–45 percent for most open models, despite the problems being trivially easy for adults. The paper takes this as evidence that state-of-the-art VLMs lack reliable atomic geometric perception, and that this deficit—not reasoning—is a bottleneck for geometry understanding. It further shows that chain-of-thought prompting barely helps and that fine-tuning on composite diagrams does not fix the deficit, while training on atomic tasks does.

What carries the argument

The load-bearing object is AVSD plus its companion composite-perception test, ν-geometry. AVSD organizes perception into 36 named atomic skills—angle, tangency, parallel, reflection, connectedness, OCR, and so on—with each multiple-choice question designed to isolate one skill, using diverse phrasings and labels to reduce accidental overlap. The dataset has three parts: handcrafted problems for depth and contamination-free testing, procedurally generated textbook-style problems for scale, and style-augmented problems built by passing synthetic diagrams through a conditional diffusion pipeline conditioned on edge maps, to test robustness. ν-geometry plays a scaffolding role: it tests composite perception with sentences describing diagram construction, showing that models already fail when several skills must be integrated, before the atomic decomposition is applied.

What would settle it

Create a matched AVSD-style probe in which the same geometric content is posed with minimal text—for example, asking the model to click a point or choose among unlabeled image crops—and compare accuracy. If models at the top of AVSD jump to near-perfect on these minimal-text versions, the original failures were largely linguistic rather than deficits in atomic visual perception.

Watch

Extended reading notes

Core claim

The central claim is that current vision-language models, including high-capability reasoning models, cannot reliably perform the basic 2D geometric perception skills that adults perform instantly. The empirical case is built on AVSD: across 36 skills, the best proprietary model reaches 72 percent overall, o3 reaches 65 percent, and most open-weight models score between 30 and 45 percent, far below near-perfect adult performance. The authors also report systematic patterns: models perform best on OCR, absolute position, and shape, worst on tangency, parallel, and angle; models agree with each other on which skills are hard; chain-of-thought prompting does not improve accuracy; style changes cause large drops; and geometry-specialized fine-tuned models are no better than general models of the same size. A final training experiment shows that fine-tuning a 13B model on synthetic atomic-skill problems raises its AVSD accuracy, while fine-tuning on composite geometry diagrams does not.

Load-bearing premise

The benchmark assumes that the 36 skills are genuinely atomic and that each question isolates exactly one skill, so a low score means the model lacks that perceptual skill; complete isolation is admittedly impossible, and the text-heavy questions could fail for language reasons.

Editorial extensions

If this is right

  • Benchmark results on composite geometry tasks can no longer be read as pure reasoning scores, because a large share of the error may be upstream atomic perception.
  • Purpose-built atomic perception data, not just more composite geometry problems, should be a component of VLM training pipelines.
  • Chain-of-thought and test-time reasoning are not a substitute for perception, since CoT did not help on AVSD and reasoning budgets are unlikely to recover the deficit.
  • Style robustness must be evaluated separately, since the consistent drop from synthetic to style-augmented subsets shows that apparent geometry ability depends on image surface style.
  • Fine-tuning on atomic skills transfers to handcrafted held-out problems, suggesting these skills are learnable rather than an inherent architectural ceiling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If correct, per-skill accuracy profiles could serve as a diagnostic: models that fail tangency and parallel specifically may share a common deficiency in relational edge-and-circle geometry, which could be probed with even simpler stimuli.
  • The atomicity assumption could be checked by constructing question pairs that differ only in wording while keeping the visual content identical; if accuracy shifts with phrasing, part of the failure is linguistic rather than perceptual.
  • The same decomposition could be ported to other diagram domains such as tables, charts, and scientific figures, where the authors note related but distinct skill sets, and per-skill profiles there might explain known table and chart failures.
  • The paper's pretraining hypothesis yields a measurable prediction: scaling atomic perception data in pretraining should improve not only AVSD but also downstream geometry-reasoning benchmarks, a test the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces the Atomic Visual Skills Dataset (AVSD), a benchmark of 13,188 problems intended to isolate 36 allegedly atomic 2D geometric perception skills, split into a handcrafted set (AVSD-h), a procedurally generated set (AVSD-s), and a style-augmented set (AVSD-c). The authors benchmark 11 open and proprietary VLMs, reporting overall accuracies between 30% and 72%, a performance drop from AVSD-s to AVSD-c, limited benefit from chain-of-thought prompting, and a fine-tuning experiment in which training on AVSD-s-train improves AVSD-h performance. The paper also introduces a smaller composite-perception dataset, ν-geometry, with two-alternative tasks. The central claim is that current VLMs are 'incapable of such atomic visual skills' despite the tasks being 'trivial for adult humans.'

Significance. If the benchmark is valid, it would be a useful resource for evaluating geometric perception separately from geometric reasoning, and the style-robustness subdataset is a plausible diagnostic tool. The authors ship code and data, provide a handcrafted set of 5,163 new problems, and report a broad model comparison including chain-of-thought and reasoning-budget variations. These are real contributions. However, the paper's headline interpretation is currently stronger than its evidence: the reported numbers support 'models struggle' but not 'incapable,' and the validity of the benchmark as a measure of atomic perception is threatened by at least one concrete item defect and by the absence of a full-dataset human or chance baseline. The findings are potentially significant for the VLM evaluation community, but the manuscript needs validation and recalibration before the strong claims can be accepted.

major comments (5)
  1. [Figure 1] The Connectedness example is internally inconsistent: the question asks the model to choose from 'pink, orange, yellow, green, blue, purple, black', but the ground truth is 'brown', which is not among the options. A model answering 'All of them' may be detecting that the correct answer is absent from the choice set rather than failing a visual connectedness skill. Because AVSD-s and AVSD-c are procedurally generated from templated questions and answer keys (Section 4.2, Appendix C.2), this class of error could be systemic. The verification described in Section 5 (30 images per skill, 99% human solvability) checks that humans can solve the problems but does not check whether the ground truth is entailed by the question and present in the option list. The authors should audit every item for choice-set consistency, report the fraction of invalid items per subdataset, and re-run all evaluations after excluding invalid items. Without this audit, low accuracy on AVSD cannot be interpreted as a perceptual deficiency.
  2. [Table 3] The comparison between AVSD-s and AVSD-c in Table 3 is confounded by skill-set mismatch: Table 2(d) states that AVSD-c has only 35 skills because the Color skill was removed, while AVSD-s has 36 skills. The overall accuracy gap between AVSD-s and AVSD-c therefore mixes the effect of style augmentation with the effect of excluding one skill. The authors should recompute the style-robustness gap using only the 35 common skills, and report per-skill AVSD-c results without Color for all models.
  3. [Section 1] The Introduction claims that current VLMs are 'incapable of such atomic visual skills,' but Table 3 reports overall accuracies of 0.72 for Gemini 2.5 Pro, 0.65 for o3, and 0.59 for o1. These numbers support the abstract's wording that models 'struggle' with the tasks, but they contradict 'incapable.' This is not a minor wording issue; the paper's central contribution is the strength of the failure claim. The authors should either revise the strong claim to match the measured accuracies or provide a threshold argument for why accuracies in the 60-72% range should be characterized as incapability.
  4. [Section 5.2] The fine-tuning experiment is presented as evidence that training on atomic tasks generalizes out-of-distribution to the handcrafted AVSD-h set. However, Appendix C.2 states that the AVSD-s task types were 'mostly aimed to depict some questions in the handcrafted dataset (AVSD-h).' This means the AVSD-s-train data are intentionally designed to mimic AVSD-h, so the observed improvement on AVSD-h may reflect learning the same task templates rather than acquiring a general atomic perception skill. To support the OOD claim, the authors should partition AVSD-h by task type and evaluate on task types that have no corresponding AVSD-s template, or otherwise quantify the template overlap between the two sets.
  5. [Section 5] The manuscript's support for the claim that AVSD problems are 'trivial for adult humans' is the authors' own 99% accuracy on 30 images per skill (1080 problems total). This is not a human accuracy baseline on the full dataset, and the paper does not report the random-chance accuracy of AVSD items, which varies because questions have different formats (true/false, multiple choice, list answers). Without a chance baseline or an independent human accuracy estimate on the full test set, the reported model accuracies cannot be calibrated against 'trivial for adult humans.' I recommend adding a human accuracy study (even on a random subset) and reporting chance-level performance by item type.
minor comments (4)
  1. [Appendix F.2] The text refers to 'Table F' as the complete evaluation results, but the actual table in the manuscript is numbered Table 12. Please correct the cross-reference.
  2. [Table 11] The prompt text in Table 11 contains the typo 'Rememeber' for 'Remember'; please fix it in the final version.
  3. [Section 4.1] The caption of Figure 3 states 'We provide a total of 5,163 new handcrafted problems,' which matches Table 2(a). It would help readers if the figure also indicated which skills have particularly few easy/medium/hard problems, since the difficulty distribution appears uneven across skills.
  4. [Table 8] For the open-weight models, the table lists model families and sizes but not specific checkpoint versions or release dates; adding the exact HuggingFace identifiers would improve reproducibility of the benchmark results.

Circularity Check

1 steps flagged · score 4.0 of 10

Main benchmark evaluation is external and not circular; however, the fine-tuning experiment's out-of-distribution claim is weakened by construction because the A VSD-s-train task types were designed to depict A VSD-h questions.

  1. self definitional [Section 5.2 ('Fine-tuning on atomic visual skills') and Appendix C.2 ('Further details on A VSD-s')]
    "Next, we fine-tuned LLaVA-Next-13B on atomic perception tasks using the A VSD-s-train dataset ... This training led to clear improvements, particularly on the handcrafted A VSD-h dataset, demonstrating a certain degree of out-of-distribution (OOD) generalization. ... We mostly aimed for each synthetic task to depict some questions in the handcrafted dataset (A VSD-h)."

    The A VSD-s-train data is the training distribution, and A VSD-h is the held-out set used to claim OOD generalization. The authors state that the synthetic task types were 'mostly aimed' at depicting A VSD-h questions, so the measured A VSD-h improvement is at least partly in-distribution by construction: the training generator was designed to mimic the test set. The claim of 'a certain degree of out-of-distribution generalization' therefore reduces to evaluating on data that the training set was built to resemble. This is a secondary experiment; the benchmark's central finding that VLMs struggle on AVSD does not depend on it.

full rationale

This is a dataset-and-benchmark paper, so there is no formal derivation chain to walk. The central claim that state-of-the-art VLMs score 0.30-0.72 on AVSD is evaluated on external models and does not reduce to the paper's own definitions. The manually defined 'atomic skills' list is admittedly not rigorous, and the Figure 1 Connectedness example has an answer-key inconsistency (ground truth 'brown' is absent from the listed choices), but these are construct-validity and data-quality threats, not circular-derivation threats. The one genuine circular step is the fine-tuning experiment: because A VSD-s-train task types were designed to depict A VSD-h questions, reporting A VSD-h gains as 'out-of-distribution generalization' is partly by construction. Since this affects a secondary claim while the main benchmark evaluation is externally grounded, the score is moderate rather than high.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

No numeric parameters are fitted to make a derivation work. The ledger records the conceptual assumptions that the benchmark claims depend on: atomicity of the 36 skills, correctness of ground truth, and reliability of an LLM judge. The only invented conceptual entity is the 'atomic visual skill' taxonomy, which lacks an independent falsifiable characterization.

assumptions (3)
  • domain assumption The 36 skills are atomic and each AVSD question isolates a single skill.
    Section 4 defines atomicity with non-rigorous criteria and says 'complete isolation is impossible'; low per-skill accuracy is interpreted as a failed atomic skill only under this assumption.
  • domain assumption AVSD ground-truth answers are correct and unambiguous.
    Only a sample of 1,080 synthetic images was author-verified at 99 percent, and A VSD-h difficulty was judged by the authors and three evaluators, with no external human-accuracy measurement.
  • domain assumption The GPT-4o mini extraction and scoring pipeline does not bias results.
    The pipeline was validated on 150 problems with 149 agreements, but the full benchmark and per-skill tables rely on this LLM judge, and list-valued answers are sensitive to the judge's consistency rules.
invented entities (1)
  • Atomic visual skills (36-skill taxonomy)
    purpose: Provides the unit of evaluation and the paper's framing that complex perception decomposes into indivisible primitives.
    The taxonomy is introduced by the authors for this paper, with no independent falsifiable characterization of atomicity; the paper itself states the definition is not fully rigorous.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models." pith.science (2026). https://pith.science/paper/53X3API7

@misc{pith2026250520021,
  author       = {Pith},
  title        = {Pith review of: Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53X3API7}},
  note         = {Machine review of arXiv:2505.20021}
}
read the original abstract

Recent Vision-Language Models (VLMs) have demonstrated impressive multimodal comprehension and reasoning capabilities, yet they often struggle with trivially simple visual tasks. In this work, we focus on the domain of basic 2D Euclidean geometry and systematically categorize the fundamental, indivisible visual perception skills, which we refer to as atomic visual skills. We then introduce the Atomic Visual Skills Dataset (AVSD) for evaluating VLMs on the atomic visual skills. Using AVSD, we benchmark state-of-the-art VLMs and find that they struggle with these tasks, despite being trivial for adult humans. Our findings highlight the need for purpose-built datasets to train and evaluate VLMs on atomic, rather than composite, visual perception tasks.

Figures

Figures reproduced from arXiv: 2505.20021 by the authors.

Figure 1
Figure 1. Examples of AVSD problems and responses by o3 model. Other state-of-the-art models [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of ν-geometry. These tasks test composite geometric perception but do not require any mathematical reasoning. They demonstrate that the state-of-the-art VLMs struggle with geometric perception, even before they get to geometric reasoning. and MathVision [69] evaluated intensive VLMs from open-source to commercial models. Such benchmarks eventually emphasize that current VLMs are not good at geometry problem… view at source ↗
Figure 3
Figure 3. List of 36 atomic visual skills and the number of easy, medium, and hard problems for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Statistics of GPT-4o response on the same question with different styles. This example [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: AVSD-c consists of synthetically generated images with diverse styles imbued with [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Evaluation results on AVSD-h. +CoT implies the performance of the model on the right with chain-of-thought (CoT) prompting [31]. The area ratios of each colored section are aligned with the actual ratio of problem counts. Further details are provided in Appendix F.2 an…
Figure 7
Figure 7. Figure 7: Evaluation results on AVSD-h across the skills. The skills are listed in the tendency of [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Our pipeline for filtering geometric distortions caused by ControlNet transformations. We [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]
Figure 9
Figure 9. Figure 9: An example of before and after preprocessing our data, under the processors of Qwen2.5-VL [PITH_FULL_IMAGE:figures/full_fig_p034_9.png]
Figure 10
Figure 10. Figure 10: Responses of GPT-4o and Gemini 2.5 Flash on a problem from AVSD with and without [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: Examples of ν-geometry with fewer steps (1-3), and the responses from o3, Gemini 2.5 Flash, and Math-LLaVA. 62 [PITH_FULL_IMAGE:figures/full_fig_p062_11.png]
Figure 12
Figure 12. Figure 12: Examples of ν-geometry with more steps (4-6), and the responses from o3, Gemini 2.5 Flash, and Math-LLaVA. 63 [PITH_FULL_IMAGE:figures/full_fig_p063_12.png]
Figure 13
Figure 13. Figure 13: Examples of AVSD-h, and the responses from o3, GPT-4o, and Gemini 2.5 Flash. [PITH_FULL_IMAGE:figures/full_fig_p064_13.png]
Figure 14
Figure 14. Figure 14: Examples of AVSD-s, and the responses from o3, GPT-4o, and Gemini 2.5 Flash. [PITH_FULL_IMAGE:figures/full_fig_p065_14.png]
Figure 15
Figure 15. Figure 15: Examples of AVSD-c, and the responses from o3, GPT-4o, and Gemini 2.5 Flash. [PITH_FULL_IMAGE:figures/full_fig_p066_15.png]
Figure 16
Figure 16. Figure 16: Examples of before and after the style transformation via ControlNet. [PITH_FULL_IMAGE:figures/full_fig_p069_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 64 canonical work pages

  1. [1]

    Abdin, J

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V . Chaudhary, D. Chen, D. Chen, W. Chen, Y .-C. Chen, Y .-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V . Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswa...

    arXiv 2024

  2. [2]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y . Li. Physics of language models: Part 3.2, knowledge manipulation. International Conference on Learning Representations, 2025

    2025

  3. [3]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual question answering.International Conference on Computer Vision, 2015

    2015

  4. [4]

    Arora and A

    S. Arora and A. Goyal. A theory for emergence of complex skills in language models. arXiv:2307.15936, 2023

    arXiv 2023

  5. [5]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models.arXiv:2108.07732, 2021

    arXiv 2021

  6. [6]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-VL technical report. arXiv:2502.13923, 2025

    arXiv 2025

  7. [7]

    a is b" fail to learn

    L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: LLMs trained on "a is b" fail to learn "b is a".International Conference on Learning Representations, 2024

    2024

  8. [8]

    Bordes, R

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Mañas, Z. Lin, A. Mahmoud, B. Jayaraman, et al. An introduction to vision-language modeling.arXiv:2405.17247, 2024

    arXiv 2024

Show all 85 references
  1. [9]

    J. Canny. A computational approach to edge detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6):679–698, 1986

    1986

  2. [10]

    Cao and J

    J. Cao and J. Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding.International Conference on Computational Linguistics, 2022

    2022

  3. [11]

    J. Chen, T. Li, J. Qin, P. Lu, L. Lin, C. Chen, and X. Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression.Empirical Methods in Natural Language Processing, 2022

    2022

  4. [12]

    L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Lin, and F. Zhao. Are we on the right way for evaluating large vision-language models?Neural Information Processing Systems, 2024

    2024

  5. [13]

    Deepmind

    G. Deepmind. Gemini Flash, Apr. 2025

    2025

  6. [14]

    Deepmind

    G. Deepmind. Gemini Pro, Apr. 2025

    2025

  7. [15]

    Dhariwal and A

    P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis.Nueral Information Processing Systems, 2021. 10

    2021

  8. [16]

    X. Fu, Y . Hu, B. Li, Y . Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. BLINK: Multimodal large language models can see but not perceive.European Conference on Computer Vision, 2024

    2024

  9. [17]

    J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y . Wang, L. Hong, J. Han, H. Xu, Z. Li, and L. Kong. G-LLaV A: Solving geometric problems with multi-modal large language model.International Conference on Learning Representations, 2025

    2025

  10. [18]

    Golovneva, Z

    O. Golovneva, Z. Allen-Zhu, J. Weston, and S. Sukhbaatar. Reverse training to nurse the reversal curse.Conference on Language Modeling, 2024

    2024

  11. [19]

    Goyal, T

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in VQA matter: Elevating the role of image understanding in visual question answering.Computer Vision and Pattern Recognition, 2017

    2017

  12. [20]

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence.arXiv:2401.14196, 2024

    2024 arXiv

  13. [21]

    Gurari, Q

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people.Computer Vision and Pattern Recognition, 2018

    2018

  14. [22]

    Hanna, O

    M. Hanna, O. Liu, and A. Variengien. How GPT-2 computes greater-than?: Interpreting mathematical abilities in a pre-trained language model.Neural Information Processing Systems, 2023

    2023

  15. [23]

    T. He, D. Doshi, A. Das, and A. Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks.Neural Information Processing Systems, 2024

    2024

  16. [24]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Nueral Information Processing Systems, 2020

    2020

  17. [25]

    Hsieh, J

    C.-Y . Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.arXiv:2306.14610, 2023

    2023 arXiv

  18. [26]

    J. Hsu, J. Wu, and N. Goodman. Geoclidean: Few-shot generalization in euclidean geometry. Nueral Information Processing Systems, 2022

    2022

  19. [27]

    Flux.1-dev controlnet

    InstantX. Flux.1-dev controlnet. https://huggingface.co/InstantX/FLUX. 1-dev-Controlnet-Canny, 2024

    2024

  20. [28]

    Kafle, B

    K. Kafle, B. Price, S. Cohen, and C. Kanan. DVQA: Understanding data visualizations via question answering.Computer Vision and Pattern Recognition, 2018

    2018

  21. [29]

    Kazemi, H

    M. Kazemi, H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning.International Conference on Machine Learning, 2023

    2023

  22. [30]

    Kembhavi, M

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images.arXiv:1603.07396, 2016

    2016 arXiv

  23. [31]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large language models are zero-shot reasoners.Nueral Information Processing Systems, 2022

    2022

  24. [32]

    B. F. Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  25. [33]

    Lake and M

    B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.International Conference on Machine Learning, 2018

    2018

  26. [34]

    N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos. Teaching arithmetic to small transformers.International Conference on Learning Representations, 2024. 11

    2024

  27. [35]

    Lewis, N

    M. Lewis, N. V . Nayak, P. Yu, Q. Yu, J. Merullo, S. H. Bach, and E. Pavlick. Does CLIP bind concepts? Probing compositionality in large image models.the Association for Computational Linguistics, 2024

    2024

  28. [36]

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu, and C. Li. LLaV A-OneVision: Easy visual task transfer.Transactions on Machine Learning Research, 2025

    2025

  29. [37]

    Z. Lin, X. Chen, D. Pathak, P. Zhang, and D. Ramanan. Revisiting the role of language priors in vision-language models.arXiv:2306.01879, 2024

    2024 arXiv

  30. [38]

    Lin and K

    Z. Lin and K. Lee. Dual operating modes of in-context learning.International Conference on Machine Learning, 2024

    2024

  31. [39]

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee. LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, 2024

    2024

  32. [40]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Neural Information Processing Systems, 2023

    2023

  33. [41]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. International Conference on Learning Representations, 2024

    2024

  34. [42]

    Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna. CREPE: Can vision-language foundation models reason compositionally?Computer Vision and Pattern Recognition, 2023

    2023

  35. [43]

    Masry, X

    A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning.Annual Meeting of the Association for Computational Linguistics, 2022

    2022

  36. [44]

    Methani, P

    N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar. PlotQA: Reasoning over scientific plots. Conference on Applications of Computer Vision, 2020

    2020

  37. [45]

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?Empirical Methods in Natural Language Processing, 2022

    2022

  38. [46]

    S. O. nón, J. Ainslie, V . Cvicek, and Z. Fisher. Making transformers solve compositional tasks. arXiv:2108.04378, 2021

    2021 arXiv

  39. [47]

    Okawa, E

    M. Okawa, E. S. Lubana, R. Dick, and H. Tanaka. Compositional abilities emerge multiplica- tively: Exploring diffusion models on a synthetic task.Nueral Information Processing Systems, 2023

    2023

  40. [48]

    GPT-4 technical report.arXiv:2303.08774, 2024

    OpenAI. GPT-4 technical report.arXiv:2303.08774, 2024

    2024 arXiv

  41. [49]

    GPT-4o system card, August 2024

    OpenAI. GPT-4o system card, August 2024

    2024

  42. [50]

    o3 system card, Apr

    OpenAI. o3 system card, Apr. 2025

    2025

  43. [51]

    o1 system card, Dec

    OpenAI. o1 system card, Dec. 2024

    2024

  44. [52]

    Ovadia, M

    O. Ovadia, M. Brief, M. Mishaeli, and O. Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms.Empirical Methods in Natural Language Processing, 2024

    2024

  45. [53]

    Paiss, A

    R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel. Teaching CLIP to count to ten.International Conference on Computer Vision, 2023

    2023

  46. [54]

    Press, M

    O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrow- ing the compositionality gap in language models.Empirical Methods in Natural Language Processing, 2023

    2023

  47. [55]

    Rahmanzadehgervi, L

    P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen. Vision-language models are blind.Asian Conference on Computer Vision, 2024. 12

    2024

  48. [56]

    Ramesh, E

    R. Ramesh, E. S. Lubana, M. Khona, R. P. Dick, and H. Tanaka. Compositional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks.International Conference on Machine Learning, 2024

    2024

  49. [57]

    Roberts, K

    J. Roberts, K. Han, and S. Albanie. GRAB: A challenging graph analysis benchmark for large multimodal models.arXiv:2408.11817, 2024

    2024

  50. [58]

    Roberts, K

    J. Roberts, K. Han, N. Houlsby, and S. Albanie. SciFIBench: Benchmarking large multimodal models for scientific figure interpretation.Nueral Information Processing Systems (Datasets and Benchmarks Track), 2024

    2024

  51. [59]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models.Computer Vision and Pattern Recognition, 2022

    2022

  52. [60]

    Rozière, J

    B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Ferrer, et al. Code llama: Open foundation models for code.arXiv:2308.12950, 2023

    2023 arXiv

  53. [61]

    J. Shen, Y . Yuan, S. Mirzoyan, M. Zhang, and C. Wang. Measuring vision-language STEM skills of neural models.International Conference on Learning Representations, 2024

    2024

  54. [62]

    W. Shi, Z. Hu, Y . Bin, J. Liu, Y . Yang, S.-K. Ng, L. Bing, and R. K.-W. Lee. Math-LLaV A: Bootstrapping mathematical reasoning for multimodal large language models.Empirical Methods in Natural Language Processing, 2024

    2024

  55. [63]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models.International Conference on Learning Representations, 2021

    2021

  56. [64]

    J. Song, Z. Xu, and Y . Zhong. Out-of-distribution generalization via composition: A lens through induction heads in transformers.Proceedings of the National Academy of Sciences, 2025

    2025

  57. [65]

    Thrush, R

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality.Computer Vision and Pattern Recognition, 2022

    2022

  58. [66]

    S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y . LeCun, and S. Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs.Nueral Information Processing Systems, 2024

    2024

  59. [67]

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs.Computer Vision and Pattern Recognition, 2024

    2024

  60. [68]

    T. H. Trinh, Y . Wu, Q. V . Le, H. He, and T. Luong. Solving olympiad geometry without human demonstrations.Nature, 625:476–482, 2024

    2024

  61. [69]

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with MATH-Vision dataset.Nueral Information Processing Systems, 2024

    2024

  62. [70]

    X. Wang, Z. Hu, P. Lu, Y . Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y . Sun, and W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models.International Conference on Machine Learning, 2024

    2024

  63. [71]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models.Nueral Information Processing Systems, 2024

    2024

  64. [72]

    Z. Xu, Z. Shi, and Y . Liang. Do large language models have compositional ability? an investigation into limitations and scalability.ICLR Workshop on Mathematical and Empirical Understanding of F oundation Models, 2024

    2024

  65. [73]

    Yuksekgonul, F

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision-language models behave like bags-of-words, and what to do about it?International Conference on Learning Representations, 2023. 13

    2023

  66. [74]

    Zhang, O

    J. Zhang, O. Liu, T. Yu, J. Hu, and W. Neiswanger. Euclid: Supercharging multimodal LLMs with synthetic high-fidelity visual descriptions.arXiv:2412.08737, 2024

    2024 arXiv

  67. [75]

    Zhang, A

    L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models.International Conference on Computer Vision, 2023

    2023

  68. [76]

    Zhang, D

    R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y . Qiao, P. Gao, and H. Li. MATHVERSE: Does your multi-modal LLM truly see the diagrams in visual math problems?European Conference on Computer Vision, 2024

    2024

  69. [77]

    H. Zhao, S. Kaur, D. Yu, A. Goyal, and S. Arora. Can models learn skill composition from examples?ICML Workshop on LLMs and Cognition, 2024

    2024

  70. [78]

    T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin. VL-Checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.arXiv:2207.00221, 2022

    2022 arXiv

  71. [79]

    Zheng, X

    M. Zheng, X. Feng, Q. Si, Q. She, Z. Lin, W. Jiang, and W. Wang. Multimodal table under- standing.the Association for Computational Linguistics, 2024

    2024

  72. [80]

    positioned in between,

    Y . Zong, T. Yu, R. Chavhan, B. Zhao, and T. Hospedales. Fool your (vision and) language model with embarrassingly simple permutations.International Conference on Machine Learning, 2024. 14 A Nu-geometry By the nature of geometry problems with visual inputs, accurate visual un...

    2024

  73. [81]

    Easy problems can be solved in a blink of an eye [16]

  74. [82]

    Medium problems can be solved in 10 seconds

  75. [83]

    Diagram on {BACKGROUND}

    Hard problems can be solved within a minute, and can be instantly verified when the answer is given. Each problem was written by one person and independently evaluated by three other people. A problem was accepted if all three evaluators were able to solve it and if the three ...

  76. [84]

    For instance, we forced the distance between any point or text to not be too close and indistinguishable

    During generation, we adjusted the parameters of the plotting code and set conditions to certify visibility of the images. For instance, we forced the distance between any point or text to not be too close and indistinguishable

  77. [85]

    true” or “false

    The automatic generation code chooses one of the predefined tasks, then generates a question, an answer, and an image based on the corresponding templates assigned to the task. The authors double-checked the templates to ensure that the generated result could be reasonable. Da...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.