Pith. sign in

REVIEW 3 major objections 3 minor 29 references

Tree-based reasoning loses to zero-shot prompting in every VLM test

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-04 20:24 UTC pith:LMJAOPU2

load-bearing objection Useful negative result for specific hand-built tree pipelines, but the 'consistently underperforms' claim overreaches—knowledge verification is circular and one tree per dataset can't support the generalization. the 3 major comments →

arxiv 2509.09732 v1 pith:LMJAOPU2 submitted 2025-09-10 cs.CV

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

classification cs.CV
keywords vision-language modelstree-based reasoningzero-shot classificationdecision treesGTSRBCIFAR-10hierarchical classificationinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether decomposing image classification into a chain of simple, interpretable visual questions—a decision tree—helps modern vision-language models classify more accurately than simply asking the model directly. Across two datasets (GTSRB with 43 traffic-sign classes and CIFAR-10) and three VLMs, the answer is no: every tree-based variant is less accurate than a plain zero-shot prompt, sometimes by more than 13 percentage points. The paper also shows GPT-4o can answer 98.2% of the tree's own questions correctly when told the correct class, yet cannot convert that knowledge into correct classifications when traversing the tree without the label. The finding matters because structured reasoning is often assumed to improve both accuracy and transparency; this paper supplies systematic evidence that, for current VLMs, the structure can cost accuracy.

Core claim

The central discovery is a negative result: explicit tree-structured decomposition does not improve VLM visual classification and, in these experiments, consistently makes it worse. The authors built one hand-refined decision tree per dataset, traversed it depth-first with node-level prompts, and compared it against zero-shot prompting with and without class descriptions and image captions. On GTSRB the best tree-based result was 52.05% accuracy (GPT-4o) versus 65.78% zero-shot; on CIFAR-10 the gap reached about 14 percentage points, with zero-shot beating tree-based on all classes. The failure is not due to ignorance of the tree's content: a knowledge-verification setup in which GPT-4o is t

What carries the argument

The machinery is a dataset-specific decision tree used as a prompt scaffold: each internal node asks a simple visual or semantic question about the image, the model picks one of the listed branch answers, and traversal continues depth-first to a leaf whose class label is the prediction. Variations include appending the full decision history, adding LLM-generated class descriptions and an image caption, and a separate knowledge-verification run where the ground-truth class is revealed while the model answers each node question. The tree itself is generated by an LLM and then hand-refined by a human, with max depth 16 and 65 nodes for GTSRB and depth 5 with 19 nodes for CIFAR-10.

Load-bearing premise

The central claim generalizes from a single hand-refined decision tree per dataset to 'tree-based reasoning' as a whole; because the paper acknowledges that alternative question orderings change accuracy, a differently designed tree could in principle flip the result.

What would settle it

Build several structurally different decision trees for GTSRB (e.g., shallow and wide, visual-primitive-first, or LLM-searched trees) and run the same VLMs on them; if any such tree consistently matches or beats the zero-shot baseline on held-out classes, the paper's blanket conclusion that tree-based reasoning underperforms would be empirically refuted. A narrower check: re-run the knowledge-verification test without revealing the ground-truth class and compare answer-path accuracy to classification accuracy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the claim holds, adding tree-based reasoning to a VLM classification pipeline is not a free accuracy boost; simpler direct prompting should remain the default unless a specific interpretability requirement overrides accuracy.
  • Error propagation through upper-level nodes is a dominant failure mode; a wrong answer to an early visual question compromises the entire path, so structured systems should minimize depth and front-load reliable atomic visual checks.
  • The strong knowledge-verification performance (98.2%) combined with weak tree traversal means VLM gains in semantic knowledge do not automatically compound into compositional classification gains.
  • LLM-generated class descriptions and image captions can nudge accuracy upward in both settings for larger VLMs, suggesting enrichment of prompts helps more than restructuring the decision process.
  • Tree-based approaches need a clear reporting standard for per-class and per-layer accuracy; since adjacent misclassifications like "pedestrian" versus "pedestrian with child" are systematic, hierarchy design choices can silently bias results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The negative result may be specific to the depth-first, hand-built tree format used here; a shallower, wider tree with visual-primitive-first questions, or a tree that lets the model skip uncertain branches, could plausibly recover some of the gap. The paper itself notes alternative question orderings yield accuracy variations, which implies the conclusion is conditional on tree design.
  • The 98.2% knowledge-verification number might overestimate compositional ability because the ground-truth label is given; a cleaner test would ask the model to answer node questions without revealing the class, then compare the resulting answer chains against the true class.
  • A natural extension is a hybrid classifier that uses the zero-shot prediction as a prior or as one input to the tree, which the paper lists as future work; if such a hybrid beat both baselines, it would reframe the negative result as a failure of pure tree traversal rather than of structured knowledge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes an interpretable classification framework that decomposes visual classification using hand-constructed decision trees, and evaluates it against standard zero-shot prompting on GTSRB and CIFAR-10 with GPT-4o, LLaMA-3.2 11B, and Qwen-VL. The authors report that tree-based reasoning underperforms zero-shot classification for all models and both datasets, despite a separate knowledge-verification experiment in which GPT-4o answers tree-path questions with 98.2% accuracy when the ground-truth class label is provided. They also test variants with question history and LLM-generated class/visual descriptions. The paper concludes that the performance gap reflects limitations in compositional visual reasoning rather than a lack of elementary visual knowledge, and offers design recommendations such as shallow trees and visual-first question ordering.

Significance. If the broad claim were established, the paper would be a useful empirical counterpoint to recent proposals that hierarchical or tree-structured reasoning improves VLM classification. The authors are honest about several failure modes and provide code, multiple models, two datasets, and multiple prompt variants, which is a reasonable starting point. The main value is the negative result for the specific tree pipelines tested, together with an error analysis that identifies path dependence and structural tree flaws. However, the paper's central generalization goes beyond what the experimental design can support, and the knowledge-verification protocol measures something different from unprompted visual attribute recognition. These issues are fixable by reframing the claims and adding or acknowledging the missing control conditions.

major comments (3)
  1. [Tree Structure / Accuracy Comparison] The central claim that 'tree-based reasoning consistently underperforms' is supported by exactly one hand-refined decision tree per dataset: GTSRB depth 16 with 65 nodes and CIFAR-10 depth 5 with 19 nodes. Tree design is a large free choice—node questions, ordering, branching granularity, and phrasing all affect accuracy. The paper's own Discussion concedes that 'alternative question orderings yield accuracy variations' and documents a structural deficiency (Layer 10 'Pedestrian?' vs. Layer 11 'Pedestrian with child?') that systematically misroutes a whole subclass. With no alternative trees, no tree-construction ablations, and no variance over tree instances, the evidence supports only the statement that this particular tree pipeline underperforms zero-shot prompting, not that tree-based reasoning in general does. The title, abstract, and Accuracy Comparison section should be narrowed a
  2. [Decision Tree Knowledge Verification] The 98.2% knowledge verification result is obtained by giving the model the ground-truth class label and then asking the tree-path questions that lead to that label. This protocol measures whether the model can answer textual attribute questions about a named class; it does not measure whether the model can visually recognize those attributes from the image, because the class is provided in advance. Therefore the conclusion in the Conclusion section—that VLMs possess strong understanding of elementary visual attributes but fail to integrate this knowledge in the tree—is not supported. The gap between verification and classification may stem from the tree structure, prompt sensitivity, or the fact that the verification task is a different, easier task. The authors should either run an image-only verification protocol without the label or explicitly rephrase the finding as 'the model can a
  3. [Accuracy Comparison / Figure 3] The accuracy comparison reports point estimates without confidence intervals or significance testing. The zero-shot baseline uses 10 prompt variations to characterize variance, but tree-based results are reported as single runs per model/temperature condition, and Figure 3 shows no error bars. Given the Discussion's admission of path dependence and the known sensitivity of LLMs to phrasing, the reported 13-point gap on GTSRB could be within the range of tree-structure variation. At minimum, the paper should report variance across repeated runs and across prompt phrasings at each node, or explicitly state that the comparison is for one fixed tree.
minor comments (3)
  1. [Conclusion] The Conclusion states that the knowledge verification confirms '90.7% perfect accuracy across GTSRB classes,' while the Abstract and the Decision Tree Knowledge Verification section report an overall average of 98.2% and 39/43 classes with 100% accuracy. The 90.7% figure is probably 39/43, but the wording is confusing and should be clarified.
  2. [Figure 3] Figure 3 is dense and the legend is small; it is difficult to see the difference between 'Tree-based' and 'Tree + History' for several models. Consider separating datasets into two panels with larger fonts.
  3. [Introduction / Contributions] The first contribution describes 'a comprehensive empirical assessment ... under diverse settings, which ensures the generalization of the experiments.' Since only two datasets and one tree per dataset are used, this overstates the scope. Please soften to 'initial empirical assessment' or add more datasets and tree variants.

Circularity Check

1 steps flagged

Central accuracy comparison is empirical and non-circular; one knowledge-verification protocol is self-definitional, but secondary to the main result.

specific steps
  1. self definitional [Methodology: Decision Tree Knowledge Verification; Conclusion]
    "In this setup, the model is given the ground truth class label and prompted to sequentially answer each question along the corresponding path in the tree that leads to this ground truth class. ... knowledge verification confirms VLMs possess a strong understanding of elementary visual attributes ... but their inability to reliably use this decomposed knowledge within the reasoning tree to arrive at the correct final classification provides a key explanation for the observed performance degradation."

    The knowledge metric is defined as the fraction of branch questions answered correctly when the true class label and the tree path to that label are supplied. High accuracy (98.2%) is therefore constructed to reflect consistency with an already-provided answer, not independent visual attribute recognition. Using this score to conclude that the model 'possesses knowledge' that 'fails to translate' into classification makes the knowledge/application gap an artifact of protocol design: the knowledge test reveals the label, while the classification task does not. The explanation is thus nearly tautological, though it is not needed for the core empirical accuracy comparison.

full rationale

The paper's central claim—that tree-based reasoning underperforms zero-shot prompting on GTSRB and CIFAR-10—is an empirical comparison of measured accuracies across three VLMs. No parameter is fitted and then renamed a prediction; no uniqueness theorem or prior self-citation is invoked to forbid alternatives; and no equation equates the output to the input. The tree structures are constructed independently of the evaluation outcome (LLM-generated then human-refined), and the zero-shot baseline is an external comparison. Therefore the main result is self-contained and not circular. The one circular-adjacent element is the Decision Tree Knowledge Verification protocol: the VLM is given the ground-truth class and the exact path, so the 98.2% 'understanding' score is by construction a measure of answering questions when the answer is already supplied; using it to explain a 'knowledge-to-application' gap is a definitional artifact rather than an empirical discovery. This inflates the paper's secondary interpretation but does not affect the measured accuracy ranking. Separately, the generalization from one hand-refined tree per dataset to 'tree-based reasoning consistently underperforms' is a sampling/external-validity limitation, not circularity: the paper itself concedes that alternative question orderings yield accuracy variations and documents a structural flaw (Layer 10 vs Layer 11 pedestrian nodes), so the strength of the universal claim is not fully supported, but that is a correctness risk, not a circular derivation. Overall circularity is low.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

This empirical paper introduces no new theoretical entities. Its free parameters are experimental design choices: the specific trees, the chosen baseline prompt, and the generated descriptions and captions. The axioms are domain assumptions about task representativeness and measurement validity, including a questionable verification protocol that provides the ground-truth label.

free parameters (5)
  • GTSRB decision tree structure = LLM-generated then human-refined, max depth 16, 65 nodes
    One tree per dataset is used; no alternative trees are tested, so the negative result may be specific to this structure.
  • CIFAR-10 decision tree structure = LLM-generated then human-refined, max depth 5, 19 nodes
    Single tree design; representativeness of tree-based reasoning is assumed rather than established.
  • Representative zero-shot prompt = selected as average-performing among 10 hand-written prompts
    Final comparisons use a single chosen prompt, not the full distribution of prompts, so the reported gaps may depend on that selection.
  • Class descriptions = generated by Deepseek
    The description text is a design choice that is not controlled across models and may influence results unevenly.
  • Image captions = generated by the tested model itself
    Captions generated by the model may contain the class label, making description-based gains a potential confound.
axioms (5)
  • domain assumption GTSRB and CIFAR-10 are representative of fine-grained and coarse-grained visual classification.
    Generalization to all such tasks is asserted but not established.
  • ad hoc to paper The single hand-refined tree per dataset is a representative instantiation of tree-based reasoning.
    The central negative claim generalizes beyond one tree design per dataset.
  • ad hoc to paper Answering tree questions with the ground-truth class label provided measures visual understanding of class attributes.
    The protocol gives away the class, so 98.2% accuracy may reflect semantic knowledge rather than visual perception.
  • domain assumption Three runs at two temperature settings are enough to conclude stability.
    No error bars or statistical tests are reported to support the claim of low variance.
  • domain assumption LLM-generated descriptions and image captions improve semantic alignment without leaking the class label.
    Captions are generated by the tested model and may contain the class name, which weakens the interpretation of any gain.

pith-pipeline@v1.3.0-alltime-deepseek · 9283 in / 10552 out tokens · 99149 ms · 2026-08-04T20:24:26.383184+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs." pith.science (2026). https://pith.science/paper/LMJAOPU2

@misc{pith2026250909732,
  author       = {Pith},
  title        = {Pith review of: Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMJAOPU2}},
  note         = {Machine review of arXiv:2509.09732}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.

Figures

Figures reproduced from arXiv: 2509.09732 by Daniel Kondermann, Gerrit Gro{\ss}mann, Islam Mesabah, Peter Neigel, Raj Bhalwankar, Sary Elmansoury, Sebastian J. Vollmer.

Figure 1
Figure 1. Figure 1: Experimental framework setup illustrating both zero-shot classification and hierarchical tree-based reasoning. It also [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of zero-shot classification accuracy [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean accuracy (%) comparison across prompting strategies, models, datasets, and temperature settings. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 19 linked inside Pith

  1. [1]

    Alhamoud, K.; Alshammari, S.; Tian, Y.; Li, G.; Torr, P.; Kim, Y.; and Ghassemi, M. 2025. Vision-Language Models Do Not Understand Negation. arXiv:2501.09425

  2. [2]

    M.; Ali, H.; and Sarfraz, S

    Anis, A. M.; Ali, H.; and Sarfraz, S. 2025. On the Limitations of Vision-Language Models in Understanding Image Transforms. arXiv:2503.09837

  3. [3]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

  4. [4]

    Besta, M.; Memedi, F.; Zhang, Z.; Gerstenberger, R.; Piao, G.; Blach, N.; Nyczyk, P.; Copik, M.; Kwaśniewski, G.; Müller, J.; Gianinazzi, L.; Kubicek, A.; Niewiadomski, H.; O'Mahony, A.; Mutlu, O.; and Hoefler, T. 2025. Demystifying Chains, Trees, and Graphs of Thoughts. arXiv:2401.14295

  5. [5]

    Cao, Q.; Chen, Y.; Lu, L.; Sun, H.; Zeng, Z.; Yang, X.; and Zhang, D. 2025. Generalized domain prompt learning for accessible scientific vision-language models. Nexus, 2(2): 100069

  6. [6]

    Cho, E.; Kim, J.; and Kim, H. J. 2023. Distribution-Aware Prompt Tuning for Vision-Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 22004--22013

  7. [7]

    Ding, T.; Li, W.; Miao, Z.; and Pfister, H. 2025. Tree of Attributes Prompt Learning for Vision-Language Models. arXiv:2410.11201

  8. [8]

    Fatemi, B.; Halcrow, J.; and Perozzi, B. 2023. Talk like a Graph: Encoding Graphs for Large Language Models. arXiv:2310.04560

  9. [9]

    F.; Zhu, D.; Cai, J.; Rezatofighi, H.; and Elhoseiny, M

    Gou, C.; Felemban, A.; Khan, F. F.; Zhu, D.; Cai, J.; Rezatofighi, H.; and Elhoseiny, M. 2024. How Well Can Vision Language Models See Image Details? arXiv:2408.03940

  10. [10]

    He, M.; Shen, Y.; Zhang, W.; Tan, Z.; and Lu, W. 2024. Advancing Process Verification for Large Language Models via Tree-Based Preference Learning. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2086--2099. Miami, Florida, USA: Association for Computational Linguistics

  11. [11]

    Jha, A. 2024. In the Era of Prompt Learning with Vision-Language Models. arXiv:2411.04892

  12. [12]

    Kim, J.; and Ji, H. 2024. Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 6187--6207. Miami, Florida, USA: Association for Computational Linguistics

  13. [13]

    Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario

  14. [14]

    Li, Z.; Wu, X.; Du, H.; Liu, F.; Nghiem, H.; and Shi, G. 2025. A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges. arXiv:2501.02189

  15. [15]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 a . Visual Instruction Tuning. arXiv:2304.08485

  16. [16]

    F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P

    Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023 b . Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172

  17. [17]

    OpenAI. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08774

  18. [18]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

  19. [19]

    V.; and Maji, S

    Saha, O.; Horn, G. V.; and Maji, S. 2024. Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. arXiv:2401.02460

  20. [20]

    Schwonberg, M.; Werner, C.; Gottschalk, H.; and Meyer, C. 2024. A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models. CoRR, abs/2411.16407

  21. [21]

    Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32: 323--332

  22. [22]

    Wang, Z.; Yu, S.; Stengel-Eskin, E.; Yoon, J.; Cheng, F.; Bertasius, G.; and Bansal, M. 2024. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

  23. [23]

    Wei, C. 2024. Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models. arXiv:2405.11301

  24. [24]

    Xu, Z.; Zhu, Y.; Deng, T.; Mittal, A.; Chen, Y.; Wang, M.; Favaro, P.; Tighe, J.; and Modolo, D. 2024. Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity. arXiv:2306.16048

  25. [25]

    Yellinek, N.; Karlinsky, L.; and Giryes, R. 2025. 3VL: Using Trees to Improve Vision-Language Models' Interpretability. arXiv:2312.17345

  26. [26]

    Zhang, R.; Zhang, B.; Li, Y.; Zhang, H.; Sun, Z.; Gan, Z.; Yang, Y.; Pang, R.; and Yang, Y. 2024 a . Improve Vision Language Model Chain-of-thought Reasoning. arXiv:2410.16198

  27. [27]

    Zhang, Y.; Unell, A.; Wang, X.; Ghosh, D.; Su, Y.; Schmidt, L.; and Yeung-Levy, S. 2024 b . Why are Visually-Grounded Language Models Bad at Image Classification? arXiv:2405.18415

  28. [28]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  29. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...