REVIEW
Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams
Computer Vision and Pattern Recognition · Submitted 6 Nov 2026 · 70 references
Pith's verdict: A well-built, unusually honest benchmark paper: the complexity-twin instrument and the answer-size law are real contributions, while the class comparison is weaker than the headline suggests and the artifacts are promised, not shipped.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Counting errors in multimodal LLMs on diagrams are set mainly by the size of the answer rather than by the counting problem's complexity class; with code and an exact transcription, counting succeeds and reading large drawings becomes…
The reading
What carries the argument
The central object is the complexity twin: a pair of counting functions on one image whose members sit on different rungs of the counting-complexity ladder $\mathrm{TC}^0 \subseteq \mathrm{L} \subseteq \mathrm{GapL} \subseteq \#P$, for instance the Euler characteristic computed by a local bit-quad formula versus the component count needing global connectivity, or spanning trees and perfect matchings computed as determinants versus all matchings and simple paths, which are $\#P$-complete. Because the twin holds the pixels fixed and changes only the question, a performance gap between members is attributable to the computation rather than to the drawing, and an attribution protocol — a forced transcription, a text-only control, and re-solving the transcription embedded in generated code — labels each error perceptual or computational. A cost bound for correct programs (a determinant in $O(n^3)$, frontier dynamic programming exponential only in grid width) shows the classes are asymptotic, so on the bounded-width grids tested both twin members are polynomial-time, and the paper states that the empirical class comparison is therefore informative mainly about answer size.
Extended reading notes
Core claim
On the same pixels one can ask for the number of spanning trees (a Laplacian determinant, in GapL) or the number of simple paths from corner to corner ($\#P$-complete), and three such twin families are built: binary images asking for the Euler characteristic versus component count, grid drawings asking for determinants versus $\#P$ counts, and Eulerian drawings with arrowheads on or off. Over 15 configurations of nine models, most Gemini models transcribe small drawings exactly (85% of scored calls, mean edge F1 0.994) and then fail at the arithmetic: for the primary model the odds of an exact answer fall 2.38-fold per decade of the true answer, while at equal answer size the $\#P$ member is harder only by about one decade of $m_{50}$ ($10^{3.8}$ for GapL against $10^{2.9}$ for $\#P$), a term that disappears if either spanning trees or simple paths is dropped from the regression. The conclusion is a two-regime answer to the title question: in chain of thought, accuracy is set mainly by answer magnitude, not by the question's complexity class, and naming the relevant theorem does not help; given code and an exact transcription, counting is solved even on $10\times 10$ grids, and reading large drawings becomes the failure, with single views missing 2.8% of horizontal edges against 0.8% of vertical ones.
Load-bearing premise
The load-bearing premise is that a counting problem's asymptotic complexity class predicts error rates on the small finite drawings actually tested, even though on bounded-width grids both twin members are polynomial-time algorithms and answer size covaries with instance size, so the class comparison mainly measures answer size — a limitation the paper itself states in Section 3.2 and repeats in its limitations paragraph.
Editorial extensions
If this is right
- Benchmarks that grade visual reasoning by complexity class will largely measure answer size unless magnitude is matched; the paper's pre-registered magnitude-matched twins show at most a small class advantage.
- Tenfold more thinking tokens buy roughly three decades of answer size ($m_{50}$ grows by about 3.2 decades per tenfold token increase), so test-time compute scales with the size of the count, not with the availability of a polynomial-time theorem.
- Stating the relevant theorem (Kirchhoff, Kasteleyn, BEST) changes chain-of-thought accuracy by at most a few points, so the failure is not ignorance of the polynomial-time algorithm.
- Code written from an exact transcription solves counting on nearly all extra-large pairs, including $10\times 10$ grids with answers past $10^{37}$, which fixes the regime where chain of thought solves nothing.
- A second, decorrelated reading of a rotated copy with per-pair crop checks reads 86/96 fresh large grids exactly versus 57/96 for a repeated upright reading, so the horizontal-edge blind spot is largely repairable by multi-view verification.
Reading between the lines
- The per-decade loss of accuracy suggests the models behave like enumerators whose cost grows with the answer rather than like algebra engines; a targeted probe holding the drawing fixed while varying the answer string (for example asking for the count in a different base) could test that mechanism directly, something the paper does not run.
- The blind spot is demonstrated only on rendered line drawings; transferring the two-view rotated protocol to photographs and occluded natural scenes, which the paper lists as untested, would show whether the anisotropy is a general property of the visual encoder.
- Because declaring the correct algorithm class in code prompts is 96% accurate while chain of thought never implements a determinant unprompted, an intervention that makes the model write out the algorithm before answering is a plausible cheap lever for recovering accuracy, consistent with but beyond the paper's Code condition.
- The two-view crop-check recipe is a general repair strategy for any task whose failures are systematic per view, such as OCR or document parsing; the paper demonstrates it only for lattice diagram transcription.
Original abstract
When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting? Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two. We pair counting questions about the same drawing that sit in different counting-complexity classes, such as its spanning trees (a determinant) and its simple paths (#P-complete), and use a transcription probe and a text-only control to separate reading from computing. We test Gemini, Gemma and Qwen3-VL models; for the Gemini models, the answer depends on how the model computes. In chain of thought, most of them read small drawings correctly and then fail at counting, at a rate set far more by the size of the answer than by its complexity class (for the primary model, the odds of an exact answer fall 2.38-fold per decade of the answer). Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck: a single view misses horizontal edges, and a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting? Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two. We pair counting questions about the same drawing that sit in different counting-complexity classes, such as its spanning trees (a determinant) and its simple paths (#P-complete), and use a transcription probe and a text-only control to separate reading from computing. We test Gemini, Gemma and Qwen3-VL models; for the Gemini models, the answer depends on how the model computes. In chain of thought, most of them read small drawings correctly and then fail at counting, at a rate set far more by the size of the answer than by its complexity class (for the primary model, the odds of an exact answer fall 2.38-fold per decade of the answer). Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck: a single view misses horizontal edges, and a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting? | background 0.77 | 0.04 | 0.03 | unclear 0.97 |
| s1 | Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two. | gap 0.73 | 0.04 | 0.03 | asserted 1.00 |
| s2 | We pair counting questions about the same drawing that sit in different counting-complexity classes, such as its spanning trees (a determinant) and its simple paths (#P-complete), and use a transcription probe and a text-only control to separate reading from computing. | method 0.95 | 0.92 | 0.06 | asserted 0.94 |
| s3 | We test Gemini, Gemma and Qwen3-VL models; for the Gemini models, the answer depends on how the model computes. | result 0.54 | 0.87 | 0.10 | asserted 0.93 |
| s4 | In chain of thought, most of them read small drawings correctly and then fail at counting, at a rate set far more by the size of the answer than by its complexity class (for the primary model, the odds of an exact answer fall 2.38-fold per decade of the answer). | result 0.99 | 0.94 | 0.51 | asserted 1.00 |
| s5 | Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck: a single view misses horizontal edges, and a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated. | result 0.99 | 0.94 | 0.27 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:2611.08998 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.