REVIEW 3 major objections 3 minor 29 references
Tree-based reasoning loses to zero-shot prompting in every VLM test
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 20:24 UTC pith:LMJAOPU2
load-bearing objection Useful negative result for specific hand-built tree pipelines, but the 'consistently underperforms' claim overreaches—knowledge verification is circular and one tree per dataset can't support the generalization. the 3 major comments →
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is a negative result: explicit tree-structured decomposition does not improve VLM visual classification and, in these experiments, consistently makes it worse. The authors built one hand-refined decision tree per dataset, traversed it depth-first with node-level prompts, and compared it against zero-shot prompting with and without class descriptions and image captions. On GTSRB the best tree-based result was 52.05% accuracy (GPT-4o) versus 65.78% zero-shot; on CIFAR-10 the gap reached about 14 percentage points, with zero-shot beating tree-based on all classes. The failure is not due to ignorance of the tree's content: a knowledge-verification setup in which GPT-4o is t
What carries the argument
The machinery is a dataset-specific decision tree used as a prompt scaffold: each internal node asks a simple visual or semantic question about the image, the model picks one of the listed branch answers, and traversal continues depth-first to a leaf whose class label is the prediction. Variations include appending the full decision history, adding LLM-generated class descriptions and an image caption, and a separate knowledge-verification run where the ground-truth class is revealed while the model answers each node question. The tree itself is generated by an LLM and then hand-refined by a human, with max depth 16 and 65 nodes for GTSRB and depth 5 with 19 nodes for CIFAR-10.
Load-bearing premise
The central claim generalizes from a single hand-refined decision tree per dataset to 'tree-based reasoning' as a whole; because the paper acknowledges that alternative question orderings change accuracy, a differently designed tree could in principle flip the result.
What would settle it
Build several structurally different decision trees for GTSRB (e.g., shallow and wide, visual-primitive-first, or LLM-searched trees) and run the same VLMs on them; if any such tree consistently matches or beats the zero-shot baseline on held-out classes, the paper's blanket conclusion that tree-based reasoning underperforms would be empirically refuted. A narrower check: re-run the knowledge-verification test without revealing the ground-truth class and compare answer-path accuracy to classification accuracy.
If this is right
- If the claim holds, adding tree-based reasoning to a VLM classification pipeline is not a free accuracy boost; simpler direct prompting should remain the default unless a specific interpretability requirement overrides accuracy.
- Error propagation through upper-level nodes is a dominant failure mode; a wrong answer to an early visual question compromises the entire path, so structured systems should minimize depth and front-load reliable atomic visual checks.
- The strong knowledge-verification performance (98.2%) combined with weak tree traversal means VLM gains in semantic knowledge do not automatically compound into compositional classification gains.
- LLM-generated class descriptions and image captions can nudge accuracy upward in both settings for larger VLMs, suggesting enrichment of prompts helps more than restructuring the decision process.
- Tree-based approaches need a clear reporting standard for per-class and per-layer accuracy; since adjacent misclassifications like "pedestrian" versus "pedestrian with child" are systematic, hierarchy design choices can silently bias results.
Where Pith is reading between the lines
- The negative result may be specific to the depth-first, hand-built tree format used here; a shallower, wider tree with visual-primitive-first questions, or a tree that lets the model skip uncertain branches, could plausibly recover some of the gap. The paper itself notes alternative question orderings yield accuracy variations, which implies the conclusion is conditional on tree design.
- The 98.2% knowledge-verification number might overestimate compositional ability because the ground-truth label is given; a cleaner test would ask the model to answer node questions without revealing the class, then compare the resulting answer chains against the true class.
- A natural extension is a hybrid classifier that uses the zero-shot prediction as a prior or as one input to the tree, which the paper lists as future work; if such a hybrid beat both baselines, it would reframe the negative result as a failure of pure tree traversal rather than of structured knowledge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an interpretable classification framework that decomposes visual classification using hand-constructed decision trees, and evaluates it against standard zero-shot prompting on GTSRB and CIFAR-10 with GPT-4o, LLaMA-3.2 11B, and Qwen-VL. The authors report that tree-based reasoning underperforms zero-shot classification for all models and both datasets, despite a separate knowledge-verification experiment in which GPT-4o answers tree-path questions with 98.2% accuracy when the ground-truth class label is provided. They also test variants with question history and LLM-generated class/visual descriptions. The paper concludes that the performance gap reflects limitations in compositional visual reasoning rather than a lack of elementary visual knowledge, and offers design recommendations such as shallow trees and visual-first question ordering.
Significance. If the broad claim were established, the paper would be a useful empirical counterpoint to recent proposals that hierarchical or tree-structured reasoning improves VLM classification. The authors are honest about several failure modes and provide code, multiple models, two datasets, and multiple prompt variants, which is a reasonable starting point. The main value is the negative result for the specific tree pipelines tested, together with an error analysis that identifies path dependence and structural tree flaws. However, the paper's central generalization goes beyond what the experimental design can support, and the knowledge-verification protocol measures something different from unprompted visual attribute recognition. These issues are fixable by reframing the claims and adding or acknowledging the missing control conditions.
major comments (3)
- [Tree Structure / Accuracy Comparison] The central claim that 'tree-based reasoning consistently underperforms' is supported by exactly one hand-refined decision tree per dataset: GTSRB depth 16 with 65 nodes and CIFAR-10 depth 5 with 19 nodes. Tree design is a large free choice—node questions, ordering, branching granularity, and phrasing all affect accuracy. The paper's own Discussion concedes that 'alternative question orderings yield accuracy variations' and documents a structural deficiency (Layer 10 'Pedestrian?' vs. Layer 11 'Pedestrian with child?') that systematically misroutes a whole subclass. With no alternative trees, no tree-construction ablations, and no variance over tree instances, the evidence supports only the statement that this particular tree pipeline underperforms zero-shot prompting, not that tree-based reasoning in general does. The title, abstract, and Accuracy Comparison section should be narrowed a
- [Decision Tree Knowledge Verification] The 98.2% knowledge verification result is obtained by giving the model the ground-truth class label and then asking the tree-path questions that lead to that label. This protocol measures whether the model can answer textual attribute questions about a named class; it does not measure whether the model can visually recognize those attributes from the image, because the class is provided in advance. Therefore the conclusion in the Conclusion section—that VLMs possess strong understanding of elementary visual attributes but fail to integrate this knowledge in the tree—is not supported. The gap between verification and classification may stem from the tree structure, prompt sensitivity, or the fact that the verification task is a different, easier task. The authors should either run an image-only verification protocol without the label or explicitly rephrase the finding as 'the model can a
- [Accuracy Comparison / Figure 3] The accuracy comparison reports point estimates without confidence intervals or significance testing. The zero-shot baseline uses 10 prompt variations to characterize variance, but tree-based results are reported as single runs per model/temperature condition, and Figure 3 shows no error bars. Given the Discussion's admission of path dependence and the known sensitivity of LLMs to phrasing, the reported 13-point gap on GTSRB could be within the range of tree-structure variation. At minimum, the paper should report variance across repeated runs and across prompt phrasings at each node, or explicitly state that the comparison is for one fixed tree.
minor comments (3)
- [Conclusion] The Conclusion states that the knowledge verification confirms '90.7% perfect accuracy across GTSRB classes,' while the Abstract and the Decision Tree Knowledge Verification section report an overall average of 98.2% and 39/43 classes with 100% accuracy. The 90.7% figure is probably 39/43, but the wording is confusing and should be clarified.
- [Figure 3] Figure 3 is dense and the legend is small; it is difficult to see the difference between 'Tree-based' and 'Tree + History' for several models. Consider separating datasets into two panels with larger fonts.
- [Introduction / Contributions] The first contribution describes 'a comprehensive empirical assessment ... under diverse settings, which ensures the generalization of the experiments.' Since only two datasets and one tree per dataset are used, this overstates the scope. Please soften to 'initial empirical assessment' or add more datasets and tree variants.
Circularity Check
Central accuracy comparison is empirical and non-circular; one knowledge-verification protocol is self-definitional, but secondary to the main result.
specific steps
-
self definitional
[Methodology: Decision Tree Knowledge Verification; Conclusion]
"In this setup, the model is given the ground truth class label and prompted to sequentially answer each question along the corresponding path in the tree that leads to this ground truth class. ... knowledge verification confirms VLMs possess a strong understanding of elementary visual attributes ... but their inability to reliably use this decomposed knowledge within the reasoning tree to arrive at the correct final classification provides a key explanation for the observed performance degradation."
The knowledge metric is defined as the fraction of branch questions answered correctly when the true class label and the tree path to that label are supplied. High accuracy (98.2%) is therefore constructed to reflect consistency with an already-provided answer, not independent visual attribute recognition. Using this score to conclude that the model 'possesses knowledge' that 'fails to translate' into classification makes the knowledge/application gap an artifact of protocol design: the knowledge test reveals the label, while the classification task does not. The explanation is thus nearly tautological, though it is not needed for the core empirical accuracy comparison.
full rationale
The paper's central claim—that tree-based reasoning underperforms zero-shot prompting on GTSRB and CIFAR-10—is an empirical comparison of measured accuracies across three VLMs. No parameter is fitted and then renamed a prediction; no uniqueness theorem or prior self-citation is invoked to forbid alternatives; and no equation equates the output to the input. The tree structures are constructed independently of the evaluation outcome (LLM-generated then human-refined), and the zero-shot baseline is an external comparison. Therefore the main result is self-contained and not circular. The one circular-adjacent element is the Decision Tree Knowledge Verification protocol: the VLM is given the ground-truth class and the exact path, so the 98.2% 'understanding' score is by construction a measure of answering questions when the answer is already supplied; using it to explain a 'knowledge-to-application' gap is a definitional artifact rather than an empirical discovery. This inflates the paper's secondary interpretation but does not affect the measured accuracy ranking. Separately, the generalization from one hand-refined tree per dataset to 'tree-based reasoning consistently underperforms' is a sampling/external-validity limitation, not circularity: the paper itself concedes that alternative question orderings yield accuracy variations and documents a structural flaw (Layer 10 vs Layer 11 pedestrian nodes), so the strength of the universal claim is not fully supported, but that is a correctness risk, not a circular derivation. Overall circularity is low.
Axiom & Free-Parameter Ledger
free parameters (5)
- GTSRB decision tree structure =
LLM-generated then human-refined, max depth 16, 65 nodes
- CIFAR-10 decision tree structure =
LLM-generated then human-refined, max depth 5, 19 nodes
- Representative zero-shot prompt =
selected as average-performing among 10 hand-written prompts
- Class descriptions =
generated by Deepseek
- Image captions =
generated by the tested model itself
axioms (5)
- domain assumption GTSRB and CIFAR-10 are representative of fine-grained and coarse-grained visual classification.
- ad hoc to paper The single hand-refined tree per dataset is a representative instantiation of tree-based reasoning.
- ad hoc to paper Answering tree questions with the ground-truth class label provided measures visual understanding of class attributes.
- domain assumption Three runs at two temperature settings are enough to conclude stability.
- domain assumption LLM-generated descriptions and image captions improve semantic alignment without leaking the class label.
Cite this review
Pith. "Pith review of Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs." pith.science (2026). https://pith.science/paper/LMJAOPU2
@misc{pith2026250909732,
author = {Pith},
title = {Pith review of: Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/LMJAOPU2}},
note = {Machine review of arXiv:2509.09732}
}
read the original abstract
Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.
Figures
Reference graph
Works this paper leans on
-
[1]
Alhamoud, K.; Alshammari, S.; Tian, Y.; Li, G.; Torr, P.; Kim, Y.; and Ghassemi, M. 2025. Vision-Language Models Do Not Understand Negation. arXiv:2501.09425
Pith/arXiv arXiv 2025
-
[2]
Anis, A. M.; Ali, H.; and Sarfraz, S. 2025. On the Limitations of Vision-Language Models in Understanding Image Transforms. arXiv:2503.09837
Pith/arXiv arXiv 2025
-
[3]
Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966
Pith/arXiv arXiv 2023
-
[4]
Besta, M.; Memedi, F.; Zhang, Z.; Gerstenberger, R.; Piao, G.; Blach, N.; Nyczyk, P.; Copik, M.; Kwaśniewski, G.; Müller, J.; Gianinazzi, L.; Kubicek, A.; Niewiadomski, H.; O'Mahony, A.; Mutlu, O.; and Hoefler, T. 2025. Demystifying Chains, Trees, and Graphs of Thoughts. arXiv:2401.14295
arXiv 2025
-
[5]
Cao, Q.; Chen, Y.; Lu, L.; Sun, H.; Zeng, Z.; Yang, X.; and Zhang, D. 2025. Generalized domain prompt learning for accessible scientific vision-language models. Nexus, 2(2): 100069
2025
-
[6]
Cho, E.; Kim, J.; and Kim, H. J. 2023. Distribution-Aware Prompt Tuning for Vision-Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 22004--22013
2023
-
[7]
Ding, T.; Li, W.; Miao, Z.; and Pfister, H. 2025. Tree of Attributes Prompt Learning for Vision-Language Models. arXiv:2410.11201
Pith/arXiv arXiv 2025
-
[8]
Fatemi, B.; Halcrow, J.; and Perozzi, B. 2023. Talk like a Graph: Encoding Graphs for Large Language Models. arXiv:2310.04560
Pith/arXiv arXiv 2023
-
[9]
F.; Zhu, D.; Cai, J.; Rezatofighi, H.; and Elhoseiny, M
Gou, C.; Felemban, A.; Khan, F. F.; Zhu, D.; Cai, J.; Rezatofighi, H.; and Elhoseiny, M. 2024. How Well Can Vision Language Models See Image Details? arXiv:2408.03940
Pith/arXiv arXiv 2024
-
[10]
He, M.; Shen, Y.; Zhang, W.; Tan, Z.; and Lu, W. 2024. Advancing Process Verification for Large Language Models via Tree-Based Preference Learning. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2086--2099. Miami, Florida, USA: Association for Computational Linguistics
2024
-
[11]
Jha, A. 2024. In the Era of Prompt Learning with Vision-Language Models. arXiv:2411.04892
Pith/arXiv arXiv 2024
-
[12]
Kim, J.; and Ji, H. 2024. Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 6187--6207. Miami, Florida, USA: Association for Computational Linguistics
2024
-
[13]
Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario
2009
-
[14]
Li, Z.; Wu, X.; Du, H.; Liu, F.; Nghiem, H.; and Shi, G. 2025. A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges. arXiv:2501.02189
Pith/arXiv arXiv 2025
-
[15]
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 a . Visual Instruction Tuning. arXiv:2304.08485
Pith/arXiv arXiv 2023
-
[16]
F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P
Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023 b . Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172
Pith/arXiv arXiv 2023
-
[17]
OpenAI. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08774
Pith/arXiv arXiv 2023
-
[18]
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020
Pith/arXiv arXiv 2021
-
[19]
Saha, O.; Horn, G. V.; and Maji, S. 2024. Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. arXiv:2401.02460
Pith/arXiv arXiv 2024
-
[20]
Schwonberg, M.; Werner, C.; Gottschalk, H.; and Meyer, C. 2024. A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models. CoRR, abs/2411.16407
Pith/arXiv arXiv 2024
-
[21]
Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32: 323--332
2012
-
[22]
Wang, Z.; Yu, S.; Stengel-Eskin, E.; Yoon, J.; Cheng, F.; Bertasius, G.; and Bansal, M. 2024. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
2024
-
[23]
Wei, C. 2024. Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models. arXiv:2405.11301
Pith/arXiv arXiv 2024
-
[24]
Xu, Z.; Zhu, Y.; Deng, T.; Mittal, A.; Chen, Y.; Wang, M.; Favaro, P.; Tighe, J.; and Modolo, D. 2024. Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity. arXiv:2306.16048
Pith/arXiv arXiv 2024
-
[25]
Yellinek, N.; Karlinsky, L.; and Giryes, R. 2025. 3VL: Using Trees to Improve Vision-Language Models' Interpretability. arXiv:2312.17345
Pith/arXiv arXiv 2025
-
[26]
Zhang, R.; Zhang, B.; Li, Y.; Zhang, H.; Sun, Z.; Gan, Z.; Yang, Y.; Pang, R.; and Yang, Y. 2024 a . Improve Vision Language Model Chain-of-thought Reasoning. arXiv:2410.16198
Pith/arXiv arXiv 2024
-
[27]
Zhang, Y.; Unell, A.; Wang, X.; Ghosh, D.; Su, Y.; Schmidt, L.; and Yeung-Levy, S. 2024 b . Why are Visually-Grounded Language Models Bad at Image Classification? arXiv:2405.18415
Pith/arXiv arXiv 2024
-
[28]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.