Pith. sign in

REVIEW 3 major objections 6 minor 104 references

Language Model as Visual Explainer

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a pixel-trained vision model's internal representations can be parsed into a tree of human-readable visual attributes, assembled by a large language model and grounded in the model's own embedding space, with no…

desk verdict Novel and useful pipeline, but the faithfulness evidence is circular, so the central claim does not hold as written. read the letter →

arxiv 2412.07802 v1 pith:7UEJE4HL submitted 2024-12-08 cs.CV

Xingyi Yang , Xinchao Wang This is my paper · ORCID
classification cs.CV
keywords interpretabilityvisualexplanationtree-structuredlargelanguagemodelsvisionprototypeembeddingsmodelcalibrationexplainableAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that any trained image classifier can be explained as a tree of human-readable visual attributes, without needing any explanation training data. A large language model proposes the attribute hierarchy, each attribute is grounded in example images retrieved from a text-to-image API, and the tree is refined by checking which nodes the classifier's own training embeddings actually visit. At test time, a query image's embedding is routed through the tree, and the root-to-leaf path becomes a sample-specific rationale. If correct, this gives people readable, model-specific explanations that are plausible, faithful to the model's internal logic, and stable under small input changes. The same tree can also be used to fine-tune the classifier, with the paper reporting that this calibration improves accuracy and out-of-distribution robustness.

What carries the argument

The load-bearing object is the visual embedding tree: each node stores one textual attribute plus a support set of prototype embeddings obtained by retrieving or generating images for that attribute and passing them through the vision model. Routing uses the point-to-set distance $D(q, P) = \inf\{d(q,p) \mid p \in P\}$ with $d(q,p) = -\log\left(1 + \frac{1}{\|q-p\|^2}\right)$ to assign training embeddings to nodes; the visitation counts $C_{v^*} = \sum_j \mathbf{1}\{v^* = \arg\min_v D(q_j, P_v)\}$ drive pruning and growing. This object lets the explainer stay entirely outside the model, yet adapt its vocabulary to the model's actual behavior.

What would settle it

Take a classifier whose training images always have a particular region (for example, dog ears) blacked out, so it cannot have learned ear features, and run LVX on it: if the explained tree still routes dog images to 'ears' nodes as top-k explanations, the routing is dominated by the LLM template rather than by the model's actual features, and faithfulness fails.

Watch

Extended reading notes

Core claim

The central claim is that the feature space of a vision model trained only on pixels can be partitioned by an LLM-defined hierarchy of visual attributes, with each node carrying both a textual description and a set of prototype embeddings derived from images retrieved for that description. The tree is not static: the LLM builds an initial version, then the model's own training embeddings are counted against each node via a point-to-set nearest-neighbor distance, infrequently visited nodes are pruned, frequently visited nodes are grown by querying the LLM for finer-grained attributes, and shared nodes across classes are refined with contrastive questions. A test sample is explained by routing its embedding through this refined tree to the top-k nearest nodes, merging the paths into a per-prediction explanatory tree. The paper also claims that using these trees as pseudo-labels in a hierarchical multi-label contrastive loss improves both interpretability and predictive performance, including on out-of-distribution benchmarks.

Load-bearing premise

The method assumes that the images gathered for each attribute, filtered at a fixed similarity threshold, are accurate and disentangled examples of that attribute in the explained model's embedding space, and that the nearest-neighbor distance from a query embedding to a node's support images faithfully reports which attribute the model is actually using.

Editorial extensions

If this is right

  • Explanations are produced at test time with only a few embedding-distance computations per node, so they need no LLM call and run quickly.
  • The same constructed tree can be reused across different architectures, since support images are pooled and can be shared between models.
  • Tree refinement automatically removes attributes the model does not recognize and adds finer attributes where it does, so the explanation reflects the model's real capabilities rather than generic common sense.
  • Calibrating the classifier with the tree as a hierarchical pseudo-label improves both in-domain accuracy and out-of-distribution robustness, according to the reported experiments.
  • The framework extends to multi-label settings such as chest X-ray diagnosis, where per-finding subtrees are attached to a single root.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the faithfulness claim holds, the same retrieval-and-routing machinery could explain other embedding-based behaviors, such as retrieval, zero-shot classification, or captioning, by swapping the category root for any embeddable concept.
  • A direct test would replace the LLM's attribute tree with a random but equally structured tree; comparing explanation quality would isolate how much of LVX's value comes from commonsense structure versus the embedding-space refinement procedure itself.
  • The paper itself notes that explanations are built from the final embedding layer only, so the tree describes the last layer's view of the object; a fuller account of the network's decision would require extending the same routing idea to intermediate feature hierarchies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LVX, a training-free method that explains an image classifier by constructing an attribute tree per class. An LLM generates hierarchical attributes; a text-to-image API (Bing/Stable Diffusion) and CLIP filtering provide support images; these are embedded through the explained model to form prototype sets. Training samples are routed to nearest prototype nodes using the point-to-set distance of Eq. (1) and Eq. (2), and the tree is pruned and grown iteratively. At test time, top-k nearest nodes form an explanatory path. The same trees are used to fine-tune the model with a hierarchical contrastive loss in Eq. (3). The paper introduces H-CIFAR10/100 and H-ImageNet tree annotations and evaluates plausibility (MCS/TK), faithfulness (MSCD), and stability, reporting gains over baselines and accuracy/OOD improvements after calibration.

Significance. If the central claims held, this would be a useful contribution: the task formulation is novel, the explanations are human-readable, no explainer training is required, and the method is demonstrated across many architectures, including self-supervised models and a medical imaging application. The calibration use of explanations is also interesting. The paper ships a substantial amount of experimental material and openly discusses limitations of the support-image collection in Appendix D.3. However, the current evaluation does not establish the central faithfulness claim, and the plausibility benchmark has a component-overlap concern. With independent validation, the method could be a meaningful contribution to post-hoc interpretability.

major comments (3)
  1. [Section 4.1, Eq. (1), Eq. (2), Algorithm 2] MSCD is the same point-to-set distance D(q_j, P_v) that LVX explicitly minimizes during refinement and routing. Section 4.2 states that tree refinement 'explicitly minimizes MSCD,' and Algorithm 2 returns the subtree minimizing the sum of D. Therefore the lower MSCD values in Table 3 are expected by construction and do not independently verify that the tree nodes reflect the classifier's actual decision process. The faithfulness claim requires a metric that is not the method's own optimization objective, for example measuring how model predictions change when the attributed visual property is removed or corrupted.
  2. [Appendix D.1, Section 4.1, Figure 5] The ground-truth trees for H-CIFAR10/100 and H-ImageNet are created with ChatGPT-generated attribute structures and CLIP-based attribute verification, with human review. LVX uses the same LLM and CLIP components to generate trees and filter support images. The MCS/TK gains in Figure 5 may therefore reflect agreement between overlapping LLM/CLIP components rather than independently verified human plausibility. The human review step mitigates but does not remove this overlap; the paper should quantify it, for example by comparing against a human-only annotation set or by ablating the shared ChatGPT/CLIP components from the evaluation.
  3. [Section 4.1, Appendix D.2 and D.3, Eq. (1)] Support prototypes are retrieved from Bing or Stable Diffusion and accepted with a CLIP cosine threshold of 0.5, and Appendix D.3 explicitly acknowledges false-positive and entangled attributes, such as a 'dog with a long tail' image that also contains the head and legs. Since D(q,P) is a minimum over these prototypes, low MSCD can be achieved by routing to retrieval artifacts rather than to attributes the explained model genuinely recognizes. The faithfulness claim needs validation that the support prototypes are disentangled and aligned with the explained model's feature space, for instance by checking prototype separation in that space or by drawing support images from the model's own training distribution.
minor comments (6)
  1. [Section 3.1, Eq. (1)] The text describes D as 'resilient to outliers,' but an infimum over a finite prototype set is not robust; a single outlier prototype can attract all queries. Please clarify or replace with a robust distance.
  2. [Section 3.1, Eq. (1)] The inline definition d(q,p) = -log(1 + 1/||q-p||^2)^2 has a misplaced superscript and is inconsistent with the footnote implementation; please align the two definitions.
  3. [Section 3.1] The word 'growed' should be 'grown.'
  4. [Algorithms 1 and 2] The notation f = ∘h in Algorithm 1 and f = g · h in Algorithm 2 should be written as f = g ∘ h for consistency with Section 2.
  5. [Section 3.3] The statement that 'the support set isn't used in model training' is ambiguous because Eq. (3) uses the parse tree T_j produced by routing with the support prototypes; please clarify that only the tree labels, not the support images, are used.
  6. [Table 3] The MSCD values are negative and reported as 'smaller the better'; please state the range and explain that it is a negative log-distance, and describe what value would correspond to perfect alignment.

Circularity Check

2 steps flagged · score 6.0 of 10

Faithfulness evidence is circular: Table 3's MSCD measures the exact point-to-set distance that LVX pruning, growing, and routing are designed to minimize, so the claimed faithfulness improvement reduces to the method's own objective.

  1. fitted input called prediction [Section 4.1 (Faithfulness metric), Section 4.2 (Faithfulness Results), Eq. (1)-(2), Algorithm 1-2]
    "Faithfulness states that the explanations should reflect the inner working of the model. We introduce Model-induced Sample-Concept Distance (MSCD) to evaluate this, calculated as the average of point-to-set distances 1/Nv P v∈V D(qj, Pv) between all test samples and tree nodes... Thanks to the incorporation of tree refinement that explicitly minimizes MSCD, our LVX method consistently surpasses benchmarks, demonstrating lowest MSCD values."

    The objective minimized by the method is identical to the metric reported as faithfulness. Eq. (1) defines D(q,P)=min_p ||q-p||; Eq. (2) assigns training points by argmin_v D(qj,Pv); Algorithm 1 prunes and grows nodes according to those assignments; Algorithm 2 routes test points by the same D(q,P_v); and MSCD is the average of that same D over test samples and tree nodes. Therefore a low MSCD is a direct consequence of optimizing the tree to be close to the CLIP-retrieved support prototypes, not an independent demonstration that the explained model genuinely recognizes the textual attributes. The paper's own phrasing—'tree refinement that explicitly minimizes MSCD'—confirms that the faithfulness result is fitted by construction rather than externally verified.

  2. other [Appendix D.1 (Annotation Creation), Section 4.1 (Plausibility metrics)]
    "The creation of test annotations for three datasets involved a semi-automated approach implemented in two distinct steps. This process was a collaborative effort between human annotators, a language model (ChatGPT), and CLIP [53]... we utilized ChatGPT 4 to generate initial attribute trees for each class with the category name... we employed an ensemble of predictions from multiple CLIP [53] models."

    The H-CIFAR/H-ImageNet ground-truth trees used for plausibility MCS/TK are generated with ChatGPT and CLIP, the same components that produce LVX's predicted trees: LVX's attributes come from an LLM (Section 3.1) and its support images are filtered by CLIP at threshold 0.5 (Section 4.1). Thus a substantial portion of the measured MCS/TK overlap is shared-generator agreement rather than evidence that the explained vision model uses those attributes. Human annotators review and refine the trees, which prevents full identity, but the plausibility benchmark is still partially self-referential because its ground truth and the method's output share ChatGPT/CLIP ancestry.

full rationale

The derivation chain is not circular everywhere: stability is evaluated under external input perturbations, calibration accuracy is judged on held-out labels, and the OOD results use independent benchmark sets. However, the central faithfulness claim reduces by construction. MSCD is defined as the average point-to-set distance D over test samples and tree nodes, while Eq. (2), Algorithm 1, and Algorithm 2 use exactly that D to prune, grow, and route. Section 4.2 explicitly says tree refinement 'explicitly minimizes MSCD,' so Table 3 reports the value of the method's own optimization objective, not an independent check that the reported attributes are what the model recognizes. The support prototypes are not grounded in the explained model either: they are accepted by CLIP at threshold 0.5, and Appendix D.3 concedes false positives with entangled patterns such as a 'dog with a long tail' image that also includes head and legs. The plausibility annotations are also semi-automatically generated by ChatGPT and CLIP before human review, creating partial self-comparison. Because the central faithfulness evidence is a fitted alignment with the method's own objective, the score is 6 rather than 0; because stability, calibration accuracy, and human-reviewed plausibility retain independent components, the score is not higher.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the reliability of LLM commonsense, text-to-image retrieval, and CLIP filtering, plus free parameters that are set without careful ablation. No new physical entities are introduced.

free parameters (5)
  • CLIP filtering threshold = 0.5
    Used to discard retrieved/generated images that do not match the attribute text; chosen without systematic search (Appendix D.2).
  • Number of support images per node K = 5 to 30
    Number of prototype images per attribute; range chosen ad hoc (Section 3.1).
  • Tree refinement iterations t_max = 5
    Number of prune/grow rounds; set to 5 without ablation (Section 4.1).
  • Calibration loss weight lambda = 0.1
    Weight of hierarchical contrastive loss in Equation 3; one value used (Section 4.1).
  • Tree kernel decay factor = 0.5
    Decaying factor in TK metric; chosen for evaluation, affects plausibility scores (Appendix J.1).
assumptions (6)
  • domain assumption LLM-generated attribute trees are accurate commonsense descriptions of visual categories
    Used in Section 3.1 to construct initial trees; the paper acknowledges in Appendix K that the LLM may produce inaccurate information.
  • domain assumption Text-to-image APIs (Bing, Stable Diffusion) plus CLIP filtering yield images that isolate the target attribute
    Central to building prototype embeddings; Appendix D.3 documents false positives, OOD samples, and bias in this process.
  • domain assumption Nearest-neighbor routing in the vision model's embedding space reflects attribute recognition
    Used in tree refinement (Section 3.1) and test routing (Section 3.2); no validation that the model actually recognizes the attribute.
  • domain assumption The distance metric D(q,P) = min_p ||q-p|| is a valid measure of semantic alignment
    Defined in Equation 1 and used throughout; the log transform in implementation is an ad hoc choice.
  • domain assumption Training-set visit counts indicate which attributes the model recognizes
    Used to prune and grow nodes in Section 3.1; assumes frequency equals capability.
  • domain assumption Semi-automated ground truth with ChatGPT, CLIP, and human verification is a valid gold standard
    H-* benchmark creation in Appendix D.1; biases plausibility scores toward LLM/CLIP-generated concepts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Model as Visual Explainer." pith.science (2026). https://pith.science/paper/7UEJE4HL

@misc{pith2026241207802,
  author       = {Pith},
  title        = {Pith review of: Language Model as Visual Explainer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UEJE4HL}},
  note         = {Machine review of arXiv:2412.07802}
}
read the original abstract

In this paper, we present Language Model as Visual Explainer LVX, a systematic approach for interpreting the internal workings of vision models using a tree-structured linguistic explanation, without the need for model training. Central to our strategy is the collaboration between vision models and LLM to craft explanations. On one hand, the LLM is harnessed to delineate hierarchical visual attributes, while concurrently, a text-to-image API retrieves images that are most aligned with these textual concepts. By mapping the collected texts and images to the vision model's embedding space, we construct a hierarchy-structured visual embedding tree. This tree is dynamically pruned and grown by querying the LLM using language templates, tailoring the explanation to the model. Such a scheme allows us to seamlessly incorporate new attributes while eliminating undesired concepts based on the model's representations. When applied to testing samples, our method provides human-understandable explanations in the form of attribute-laden trees. Beyond explanation, we retrained the vision model by calibrating it on the generated concept hierarchy, allowing the model to incorporate the refined knowledge of visual attributes. To access the effectiveness of our approach, we introduce new benchmarks and conduct rigorous evaluations, demonstrating its plausibility, faithfulness, and stability.

Figures

Figures reproduced from arXiv: 2412.07802 by the authors.

Figure 1
Figure 1. General workflow of LVX. (Left) A toy example that LLM interacts with vision model to examine its capability. (Mid) It combines vision, language, and visual-language APIs to create a parse tree for each visual model. (Right) In testing, embeddings navigate this tree, and the traversed path provides a personalized explanation for the model’s prediction. A key challenge is that vision models, trained solely on pixel d… view at source ↗
Figure 2
Figure 2. The illustration of visual explanatory tree parsing. Each input sample is interpreted as a parse tree to represent the model’s logical process. Let us consider the trained vision model f, de￾fined as a function f : X → Y, where X rep￾resents the input image space and Y denotes the output label space. In this study, our focus lies on the classification task, where f = g ◦ h is decomposed into a feature extractor g an… view at source ↗
Figure 3
Figure 3. Crafting text-image pairs for visual concepts. Through in-context prompting, we extract [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Tree refinement by traversing the embedding tree and querying the LLM model. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Plausibility comparison on three visual tree parsing benchmarks. We plot the mean±std across all networks architectures. For both scores, higher values indicate better performance. facilitate reasoning for visual predictions. To address this, we developed annotations f…
Figure 6
Figure 6. Figure 6: Explanation visualization for ViT-B on ImageNet-1K. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 5
Figure 5. Figure 5: The calibration process not only improved model performance, but also led to more precise [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: Software tool interface for parse tree annotation. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Statistics of the support dataset sets of (Row1) CIFAR10, (Row2) CIFAR100 and (Row3) Im [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Explanation performance comparison on Chestx-ray14 dataset. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Explanation examples for the chest xray diagnosis task. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

104 extracted references · 41 canonical work pages

  1. [1]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4190–4197, Online, July 2020. Association for Computational Linguistics

    2020

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    2022

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bi´nk...

    2022

  4. [4]

    Learning to compose neural networks for question answering

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705, 2016

    arXiv 2016

  5. [5]

    Neural module networks

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48, 2016

    2016

  6. [6]

    A survey on tree edit distance and related problems

    Philip Bille. A survey on tree edit distance and related problems. Theoretical computer science, 337(1- 3):217–239, 2005

    2005

  7. [7]

    e-snli: Natural language inference with natural language explanations

    Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31, 2018

    2018

  8. [8]

    Unsuper- vised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. 2020

    2020

Show all 104 references
  1. [9]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  2. [10]

    This looks like that: deep learning for interpretable image recognition

    Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019

    2019

  3. [11]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings ...

    2020

  4. [12]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020

    2003 arXiv

  5. [13]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021

    2021 arXiv

  6. [14]

    Self-born wiring for neural trees

    Ying Chen, Feng Mao, Jie Song, Xinchao Wang, Huiqiong Wang, and Mingli Song. Self-born wiring for neural trees. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5047–5056, 2021

    2021

  7. [15]

    Whatever next? predictive brains, situated agents, and the future of cognitive science

    Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and brain sciences, 36(3):181–204, 2013

    2013

  8. [16]

    Nearest neighbor pattern classification

    Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21–27, 1967

    1967

  9. [17]

    Extracting tree-structured representations of trained networks

    Mark Craven and Jude Shavlik. Extracting tree-structured representations of trained networks. Advances in neural information processing systems, 8, 1995

    1995

  10. [18]

    Improved regularization of convolutional neural networks with cutout

    Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. 11

    2017 arXiv

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  12. [20]

    wordnet: WordNet Interface, 2023

    Ingo Feinerer and Kurt Hornik. wordnet: WordNet Interface, 2023. R package version 0.1-16

    2023

  13. [21]

    Distilling a neural network into a soft decision tree

    Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784, 2017

    2017 arXiv

  14. [22]

    Interpreting CLIP’s image representation via text-based decomposition

    Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP’s image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019

    2019

  16. [24]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural i...

    2020

  17. [25]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559, 2022

    2022 arXiv

  18. [26]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  19. [27]

    Generating visual explanations

    Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 3–19. Sp...

    2016

  20. [28]

    Learning to reason: End-to-end module networks for visual question answering

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804–813, 2017

    2017

  21. [29]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  22. [30]

    Prototype optimization for nearest-neighbor classification

    Yea-Shuan Huang, Cheng-Chin Chiang, Jun-Wei Shieh, and Eric Grimson. Prototype optimization for nearest-neighbor classification. Pattern Recognition, 35(6):1237–1245, 2002

    2002

  23. [31]

    Discovering states and transformations in image collections

    Phillip Isola, Joseph J Lim, and Edward H Adelson. Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383–1391, 2015

    2015

  24. [32]

    Hierarchical mixtures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994

    1994

  25. [33]

    On the approximability of the maximum common subgraph problem

    Viggo Kann. On the approximability of the maximum common subgraph problem. In STACS, volume 92, pages 377–388. Citeseer, 1992

    1992

  26. [34]

    Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10233–10243, 2022

    Monish Keswani, Sriranjani Ramakrishnan, Nishant Reddy, and Vineeth N Balasubramanian. Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10233–10243, 2022

    2022

  27. [35]

    Textual explanations for self-driving vehicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV) , pages 563–578, 2018

    2018

  28. [36]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  29. [37]

    Learning vector quantization

    Teuvo Kohonen and Teuvo Kohonen. Learning vector quantization. Self-organizing maps, pages 175–189, 1995

    1995

  30. [38]

    Deep neural decision forests

    Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, pages 1467–1475, 2015. 12

    2015

  31. [39]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009

    2009

  32. [40]

    Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions

    Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  33. [41]

    Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions

    Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. InProceedings of the European Conference on Computer Vision (ECCV), pages 552–567, 2018

    2018

  34. [42]

    A systematic investigation of commonsense knowledge in large language models

    Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d’Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro...

    2022

  35. [43]

    Taskmatrix

    Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434, 2023

    2023 arXiv

  36. [44]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  37. [45]

    What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021

    2021 arXiv

  38. [46]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017

    2017

  39. [47]

    Doubly right object recognition: A why prompt for visual rationales

    Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl V ondrick. Doubly right object recognition: A why prompt for visual rationales. arXiv preprint arXiv:2212.06202, 2022

    2022 arXiv

  40. [48]

    Visual classification via description from large language models.arXiv preprint arXiv:2210.07183, 2022

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models.arXiv preprint arXiv:2210.07183, 2022

    2022 arXiv

  41. [49]

    Neural prototype trees for interpretable fine-grained im- age recognition

    Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained im- age recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14933–14943, 2021

    2021

  42. [50]

    Coco attributes: Attributes for people, animals, and objects

    Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 85–100. Springer, 2016

    2016

  43. [51]

    Xplainer: From x-ray observations to explainable zero-shot diagnosis

    Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Petra Jiraskova, Rickmer Braren, and Nassir Navab. Xplainer: From x-ray observations to explainable zero-shot diagnosis. arXiv preprint arXiv:2303.13391, 2023

    2023 arXiv

  44. [52]

    Learning to predict visual attributes in the wild

    Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13018–13028, June 2021

    2021

  45. [53]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  46. [54]

    Maximum common subgraph isomorphism algorithms for the matching of chemical structures

    John W Raymond and Peter Willett. Maximum common subgraph isomorphism algorithms for the matching of chemical structures. Journal of computer-aided molecular design, 16:521–533, 2002

    2002

  47. [55]

    why should I trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, p...

    2016

  48. [56]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    2021

  49. [57]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 13

    2015

  50. [58]

    Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification

    Dawid Rymarczyk, Łukasz Struski, Jacek Tabor, and Bartosz Zieli´nski. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1420–1430, 2021

    2021

  51. [59]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018

    2018

  52. [60]

    Rule extraction from neural networks via decision tree induction

    Makoto Sato and Hiroshi Tsukimoto. Rule extraction from neural networks via decision tree induction. In IJCNN’01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), volume 3, pages 1870–1875. IEEE, 2001

    2001

  53. [61]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023

    2023 arXiv

  54. [62]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017

    2017

  55. [63]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023

    2023 arXiv

  56. [64]

    Learning important features through propagat- ing activation differences

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In International conference on machine learning, pages 3145–3153. PMLR, 2017

    2017

  57. [65]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013

    2013 arXiv

  58. [66]

    Very deep convolutional networks for large-scale image recogni- tion

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  59. [67]

    James Murdoch, and Bin Yu

    Chandan Singh, W. James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In International Conference on Learning Representations, 2019

    2019

  60. [68]

    Smoothgrad: removing noise by adding noise

    Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017

    2017 arXiv

  61. [69]

    Prototypical networks for few-shot learning

    Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017

    2017

  62. [70]

    Neural trees-using neural nets in a tree classifier structure

    J-E Stromberg, Jalel Zrida, and Alf Isaksson. Neural trees-using neural nets in a tree classifier structure. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, pages 137–140. IEEE Computer Society, 1991

    1991

  63. [71]

    Tree sequence kernel for natural language

    Jun Sun, Min Zhang, and Chew Lim Tan. Tree sequence kernel for natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 921–926, 2011

    2011

  64. [72]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

    2015

  65. [73]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016

    2016

  66. [74]

    Visual correspondence-based explanations improve ai robustness and human-ai team accuracy

    Mohammad Reza Taesiri, Giang Nguyen, and Anh Nguyen. Visual correspondence-based explanations improve ai robustness and human-ai team accuracy. Advances in Neural Information Processing Systems, 35:34287–34301, 2022

    2022

  67. [75]

    Adaptive neural trees

    Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning, pages 6166–6175. PMLR, 2019

    2019

  68. [76]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011

    2011

  69. [77]

    Nbdt: neural-backed decision trees

    Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: neural-backed decision trees. arXiv preprint arXiv:2004.00221, 2020. 14

    2004 arXiv

  70. [78]

    Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on co...

    2017

  71. [79]

    A tree-based decoder for neural machine translation

    Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. A tree-based decoder for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4772–4777, 2018

    2018

  72. [80]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022

    2022 arXiv

  73. [81]

    Zero-shot learning—a compre- hensive evaluation of the good, the bad and the ugly

    Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning—a compre- hensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251–2265, 2018

    2018

  74. [82]

    Attribute prototype network for zero-shot learning

    Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. Advances in Neural Information Processing Systems, 33:21969–21980, 2020

    2020

  75. [83]

    Factorizing knowledge in neural networks

    Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. InComputer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV, pages 73–91. Springer, 2022

    2022

  76. [84]

    Deep model reassembly

    Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 35:25739–25753, 2022

    2022

  77. [85]

    Deep neural decision trees

    Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arXiv:1806.06988, 2018

    2018 arXiv

  78. [86]

    Language in a bottle: Language model guided concept bottlenecks for interpretable image classification

    Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  79. [87]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  80. [88]

    Representer point selection for explaining deep neural networks

    Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018

    2018

  81. [89]

    Neural- symbolic vqa: Disentangling reasoning from vision and language understanding

    Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018

    2018

  82. [90]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014

    2014

  83. [91]

    Socratic models: Composing zero-shot multimodal reasoning with language

    Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022

    2022 arXiv

  84. [92]

    Use all the labels: A hierarchical multi-label contrastive learning framework

    Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16660–16669, 2022

    2022

  85. [93]

    Diagnosing and rectifying vision models using language

    Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In International Conference on Learning Representations (ICLR), 2023

    2023

  86. [94]

    Evolutionary design of neural network tree-integration of decision tree, neural network and ga

    Qiangfu Zhao. Evolutionary design of neural network tree-integration of decision tree, neural network and ga. In Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No. 01TH8546), volume 1, pages 240–244. IEEE, 2001

    2001

  87. [95]

    Evaluating commonsense in pre-trained language models

    Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. Evaluating commonsense in pre-trained language models. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 9733–9740, 2020. 15

    2020

  88. [96]

    Deepred–rule extraction from deep neural networks

    Jan Ruben Zilke, Eneldo Loza Mencía, and Frederik Janssen. Deepred–rule extraction from deep neural networks. In Discovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19–21, 2016, Proceedings 19, pages 457–473. Springer, 2016. A Appendix / supplementa...

    2016

  89. [97]

    In this first step, we utilized ChatGPT 4 to generate initial attribute trees for each class with the category name, detailed Section 3.1

    Concept Tree Creation. In this first step, we utilized ChatGPT 4 to generate initial attribute trees for each class with the category name, detailed Section 3.1

  90. [98]

    To determine whether an attribute was present or absent in an image, we employed an ensemble of predictions from multiple CLIP [53] models 5

    Attribute Verification. To determine whether an attribute was present or absent in an image, we employed an ensemble of predictions from multiple CLIP [53] models 5. We filtered the top-5 attributes predicted by CLIP and sought human judgments to verify their correctness. To s...

  91. [99]

    crane” could represent either a construction machine or a bird, leading to ambiguity. Furthermore, an image described as “a dog with a long tail

    Manual Verification. In this phase, annotators examined the accuracy of existing attributes and introduced new relevant ones to enrich the concept trees. Subsequently, human annota- tors conducted a thorough review, refinement, and systematic organization of the attribute tree...

  92. [100]

    If both T and T ′ are leaf nodes, then θ(T, T′) = 1if the labels of T and T ′ are the same, and 0 otherwise

  93. [101]

    If either T or T ′ is a leaf node, then θ(T, T′) = 0

  94. [102]

    , Tn} be the child subtrees of T , and {T ′ 1, T′ 2,

    Otherwise, let {T1, T2, . . . , Tn} be the child subtrees of T , and {T ′ 1, T′ 2, . . . , T′ n′} be the child subtrees of T ′. – If the labels of r and r′ are the same, then θ(T, T′) is the sum of the products of θ(Ti, T′ j) for all combinations of i and j, where i ranges fro...

  95. [103]

    · 1 +T ED(T ′ 1, T ′ 2), which represents the cost of deleting the root r1 and recursively computing TED between T ′ 1 and T ′ 2

    + T ED(children of r1, children of r2), where T ED(children of r1, children of r2) is the TED computed recur- sively between the children of r1 and r2. · 1 +T ED(T ′ 1, T ′ 2), which represents the cost of deleting the root r1 and recursively computing TED between T ′ 1 and T ...

  96. [104]

    · 1 +T ED(T ′ 1, T2), which represents the cost of deleting the root r1 and recursively computing TED between T ′ 1 and T2

    + T ED(children of r1, children of r2), where T ED(children of r1, children of r2) is the TED computed recur- sively between the children of r1 and r2. · 1 +T ED(T ′ 1, T2), which represents the cost of deleting the root r1 and recursively computing TED between T ′ 1 and T2. ·...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.