{"id":"ca20f5f9-08a2-4316-b0db-d21308ad4ec5","arxiv_id":"2412.07802","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LVX builds LLM-generated attribute trees to explain any trained image classifier without training the explainer, but its faithfulness metric is directly optimized by the method.","lead":"LVX explains image classifiers with tree-shaped language descriptions by asking a language model to list visual attributes, finding images for each attribute, and routing test images to the closest attribute. The paper also uses these trees to fine-tune the model, reporting small accuracy gains and better out-of-distribution performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Faithfulness evidence is circular: Table 3's MSCD measures the exact point-to-set distance that LVX pruning/growing and routing are designed to minimize, and the support images defining those distances are vetted only by CLIP, so low MSCD does not show the model genuinely recognizes the reported…","rationale":"The reader's rejection is well supported. The strongest claim in the abstract is that LVX 'demonstrates plausibility, faithfulness, and stability'; the paper's support for the faithfulness part is the MSCD metric, but Eq. 1 is both the engine of tree refinement and the test-time router, so the Table 3 comparison is close to an evaluation of the method against itself. A control with random support images would reveal whether the low MSCD derives from attribute-specific retrieval or simply from having class-local prototypes. In addition, the support set is filtered only by CLIP at threshold 0.5 (D.2), and the manuscript's own limitations section (D.3) admits that retrieved images contain false positives and entangled patterns; without a check that retained images are disentangled exemplars in the explained model's embedding, the explanation could name attributes that appear in the retrieval image but play no role in the classifier's decision. The plausibility benchmark uses the same ChatGPT+CLIP pipeline to build ground-truth trees, so the reported MCS/TK improvements partly reflect shared component biases. The method has merits: it is training-free for explanation and produces structured, inspectable outputs, and the calibration results are reported with concrete numbers. Those merits, however, do not establish the central faithfulness claim as written. I therefore agree with the reader's REJECT verdict; no adjustment is needed.","tokens_in":26031,"tokens_out":7273,"duration_ms":77303,"concrete_test":"On CIFAR-10 with a frozen ResNet-18, construct the LVX trees as described, then build a control in which every attribute node keeps its textual label but its support set is replaced by K images sampled uniformly from that class's training set (no attribute-specific retrieval), leaving the tree structure, pruning/growing counts, and routing otherwise identical. Recompute the Table 3 MSCD values on the test set. If the control's MSCD is equal to or better than LVX's, the reported faithfulness gain is an artifact of prototype proximity rather than evidence that the textual attributes match the model's internal representations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the generated attribute-laden trees are faithful explanations of what the explained vision model actually recognizes. That claim rests on the point-to-set distance D(q, P) = min_p ||q-p|| (Eq. 1), which is used in Section 3.1 both to count training-set visits for pruning/growing and in Section 3.2 to route test samples to the top-k nearest nodes. The faithfulness score introduced in Section 4.1, MSCD, is the average of the same D over tree nodes, so the 'lowest MSCD' in Table 3 reports that LVX is close to the support prototypes that the method itself was optimized to be close to; it does not independently verify that those prototypes correspond to attributes the classifier actually uses. The support prototypes are not grounded in the explained model either: images are retrieved via Bing or Stable Diffusion and accepted with a CLIP cosine threshold of 0.5 (Section 4.1, Appendix D.2), and Appendix D.3 concedes that such images contain false positives and entangled patterns, for example a 'dog with a long tail' image that also contains the head and legs. Because CLIP, not the explained model, decides what each attribute looks like, low MSCD can be achieved by routing to retrieval artifacts. The plausibility evaluations do not repair this: the H-* ground-truth trees are generated by ChatGPT and CLIP and then human-reviewed (Appendix D.1), so MCS/TK primarily measure agreement among LLM/CLIP components shared with LVX rather than with an independent standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LVX, a training-free method that explains an image classifier by constructing an attribute tree per class. An LLM generates hierarchical attributes; a text-to-image API (Bing/Stable Diffusion) and CLIP filtering provide support images; these are embedded through the explained model to form prototype sets. Training samples are routed to nearest prototype nodes using the point-to-set distance of Eq. (1) and Eq. (2), and the tree is pruned and grown iteratively. At test time, top-k nearest nodes form an explanatory path. The same trees are used to fine-tune the model with a hierarchical contrastive loss in Eq. (3). The paper introduces H-CIFAR10/100 and H-ImageNet tree annotations and evaluates plausibility (MCS/TK), faithfulness (MSCD), and stability, reporting gains over baselines and accuracy/OOD improvements after calibration.","tokens_in":26385,"tokens_out":5773,"duration_ms":58059,"significance":"If the central claims held, this would be a useful contribution: the task formulation is novel, the explanations are human-readable, no explainer training is required, and the method is demonstrated across many architectures, including self-supervised models and a medical imaging application. The calibration use of explanations is also interesting. The paper ships a substantial amount of experimental material and openly discusses limitations of the support-image collection in Appendix D.3. However, the current evaluation does not establish the central faithfulness claim, and the plausibility benchmark has a component-overlap concern. With independent validation, the method could be a meaningful contribution to post-hoc interpretability.","major_comments":[{"comment":"MSCD is the same point-to-set distance D(q_j, P_v) that LVX explicitly minimizes during refinement and routing. Section 4.2 states that tree refinement 'explicitly minimizes MSCD,' and Algorithm 2 returns the subtree minimizing the sum of D. Therefore the lower MSCD values in Table 3 are expected by construction and do not independently verify that the tree nodes reflect the classifier's actual decision process. The faithfulness claim requires a metric that is not the method's own optimization objective, for example measuring how model predictions change when the attributed visual property is removed or corrupted.","section":"Section 4.1, Eq. (1), Eq. (2), Algorithm 2"},{"comment":"The ground-truth trees for H-CIFAR10/100 and H-ImageNet are created with ChatGPT-generated attribute structures and CLIP-based attribute verification, with human review. LVX uses the same LLM and CLIP components to generate trees and filter support images. The MCS/TK gains in Figure 5 may therefore reflect agreement between overlapping LLM/CLIP components rather than independently verified human plausibility. The human review step mitigates but does not remove this overlap; the paper should quantify it, for example by comparing against a human-only annotation set or by ablating the shared ChatGPT/CLIP components from the evaluation.","section":"Appendix D.1, Section 4.1, Figure 5"},{"comment":"Support prototypes are retrieved from Bing or Stable Diffusion and accepted with a CLIP cosine threshold of 0.5, and Appendix D.3 explicitly acknowledges false-positive and entangled attributes, such as a 'dog with a long tail' image that also contains the head and legs. Since D(q,P) is a minimum over these prototypes, low MSCD can be achieved by routing to retrieval artifacts rather than to attributes the explained model genuinely recognizes. The faithfulness claim needs validation that the support prototypes are disentangled and aligned with the explained model's feature space, for instance by checking prototype separation in that space or by drawing support images from the model's own training distribution.","section":"Section 4.1, Appendix D.2 and D.3, Eq. (1)"}],"minor_comments":[{"comment":"The text describes D as 'resilient to outliers,' but an infimum over a finite prototype set is not robust; a single outlier prototype can attract all queries. Please clarify or replace with a robust distance.","section":"Section 3.1, Eq. (1)"},{"comment":"The inline definition d(q,p) = -log(1 + 1/||q-p||^2)^2 has a misplaced superscript and is inconsistent with the footnote implementation; please align the two definitions.","section":"Section 3.1, Eq. (1)"},{"comment":"The word 'growed' should be 'grown.'","section":"Section 3.1"},{"comment":"The notation f = ∘h in Algorithm 1 and f = g · h in Algorithm 2 should be written as f = g ∘ h for consistency with Section 2.","section":"Algorithms 1 and 2"},{"comment":"The statement that 'the support set isn't used in model training' is ambiguous because Eq. (3) uses the parse tree T_j produced by routing with the support prototypes; please clarify that only the tree labels, not the support images, are used.","section":"Section 3.3"},{"comment":"The MSCD values are negative and reported as 'smaller the better'; please state the range and explain that it is a negative log-distance, and describe what value would correspond to perfect alignment.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The evaluation circularity is serious and affects the abstract's core claims of faithfulness and plausibility, but the proposed pipeline is a plausible research direction and the issues are addressable with additional experiments. I support major revision rather than rejection: the manuscript needs an independent faithfulness evaluation and a clearer separation between LLM/CLIP-generated ground truth and the method's own components."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the method is genuinely new and reasonably engineered: an LLM generates hierarchical attribute trees, Bing/Stable Diffusion supplies support images, CLIP filters them, and the explained model's embedding space turns them into a prototype tree. The tree is pruned and grown using training-set routing, test explanations are produced without training the explainer, and there are new benchmarks (H-CIFAR-10/100, H-ImageNet) and a user study. That is a real contribution. The pseudo-code is detailed, and the appendix is unusually honest about the support set's false positives and entangled images.\n\nSecond, the faithfulness evaluation does not survive contact with the method. The MSCD metric is the average point-to-set distance between test embeddings and tree-node prototypes—exactly the quantity the refinement procedure minimizes when pruning and growing, and exactly the quantity used to route test samples. The paper says refinement \"explicitly minimizes MSCD\". So Table 3 shows the method is close to prototypes it was built to be close to, not that those prototypes correspond to attributes the classifier actually uses. That is circular. The support prototypes are vetted by CLIP, not by the explained model, and the appendix concedes retrieval artifacts. Low MSCD can be achieved by routing to those artifacts. The plausibility benchmark is also generated with ChatGPT and CLIP—the same components as the method. Human review adds some independence, but the tree labels are still largely LLM/CLIP products. The user study is the most independent evidence, but it tests preference, not faithfulness.\n\nWhere the paper is solid: the pipeline is clear, the ablations are informative, the calibration results are modest but honestly reported, and the X-ray extension is a useful stress test. The method may well be a practical tool for debugging and transparency—but the paper's central claim of faithful, model-grounded explanations is overclaimed. The faithfulness claim should be downgraded to something like \"stable and plausible explanations that track model geometry in the sense of nearest prototypes.\" The plausibility benchmarks need an independent ground truth, or at least a human-only annotation subset. A sensitivity analysis for the CLIP threshold would help.\n\nWho is this for? People building post-hoc explanation tools for vision models, and anyone studying LLM-guided interpretability. It deserves a serious referee, but the referee should demand a non-circular faithfulness evaluation before publication. I would treat the current version as a strong workshop paper with an overclaimed central result, and I would send it to peer review with the expectation of major revision.","headline":"Novel and useful pipeline, but the faithfulness evidence is circular, so the central claim does not hold as written.","tokens_in":26852,"tokens_out":2871,"would_cite":false,"duration_ms":31532,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pixel-trained vision model's internal representations can be parsed into a tree of human-readable visual attributes, assembled by a large language model and grounded in the model's own embedding space, with no…","keywords":["interpretability","visual explanation","tree-structured explanation","large language models","vision models","prototype embeddings","model calibration","explainable AI"],"falsifier":"Take a classifier whose training images always have a particular region (for example, dog ears) blacked out, so it cannot have learned ear features, and run LVX on it: if the explained tree still routes dog images to 'ears' nodes as top-k explanations, the routing is dominated by the LLM template rather than by the model's actual features, and faithfulness fails.","tokens_in":25840,"feed_emoji":"🌳","tokens_out":5516,"duration_ms":54422,"temperature":0.7,"pith_summary":"The paper is trying to show that any trained image classifier can be explained as a tree of human-readable visual attributes, without needing any explanation training data. A large language model proposes the attribute hierarchy, each attribute is grounded in example images retrieved from a text-to-image API, and the tree is refined by checking which nodes the classifier's own training embeddings actually visit. At test time, a query image's embedding is routed through the tree, and the root-to-leaf path becomes a sample-specific rationale. If correct, this gives people readable, model-specific explanations that are plausible, faithful to the model's internal logic, and stable under small input changes. The same tree can also be used to fine-tune the classifier, with the paper reporting that this calibration improves accuracy and out-of-distribution robustness.","feed_headline":"Explaining an image classifier as a tree of visible attributes","feed_subtitle":"A language model drafts the attribute tree; the classifier's own embeddings pick which branches it actually uses.","key_machinery":"The load-bearing object is the visual embedding tree: each node stores one textual attribute plus a support set of prototype embeddings obtained by retrieving or generating images for that attribute and passing them through the vision model. Routing uses the point-to-set distance $D(q, P) = \\inf\\{d(q,p) \\mid p \\in P\\}$ with $d(q,p) = -\\log\\left(1 + \\frac{1}{\\|q-p\\|^2}\\right)$ to assign training embeddings to nodes; the visitation counts $C_{v^*} = \\sum_j \\mathbf{1}\\{v^* = \\arg\\min_v D(q_j, P_v)\\}$ drive pruning and growing. This object lets the explainer stay entirely outside the model, yet adapt its vocabulary to the model's actual behavior.","core_discovery":"The central claim is that the feature space of a vision model trained only on pixels can be partitioned by an LLM-defined hierarchy of visual attributes, with each node carrying both a textual description and a set of prototype embeddings derived from images retrieved for that description. The tree is not static: the LLM builds an initial version, then the model's own training embeddings are counted against each node via a point-to-set nearest-neighbor distance, infrequently visited nodes are pruned, frequently visited nodes are grown by querying the LLM for finer-grained attributes, and shared nodes across classes are refined with contrastive questions. A test sample is explained by routing its embedding through this refined tree to the top-k nearest nodes, merging the paths into a per-prediction explanatory tree. The paper also claims that using these trees as pseudo-labels in a hierarchical multi-label contrastive loss improves both interpretability and predictive performance, including on out-of-distribution benchmarks.","pith_inferences":["If the faithfulness claim holds, the same retrieval-and-routing machinery could explain other embedding-based behaviors, such as retrieval, zero-shot classification, or captioning, by swapping the category root for any embeddable concept.","A direct test would replace the LLM's attribute tree with a random but equally structured tree; comparing explanation quality would isolate how much of LVX's value comes from commonsense structure versus the embedding-space refinement procedure itself.","The paper itself notes that explanations are built from the final embedding layer only, so the tree describes the last layer's view of the object; a fuller account of the network's decision would require extending the same routing idea to intermediate feature hierarchies."],"forward_implications":["Explanations are produced at test time with only a few embedding-distance computations per node, so they need no LLM call and run quickly.","The same constructed tree can be reused across different architectures, since support images are pooled and can be shared between models.","Tree refinement automatically removes attributes the model does not recognize and adds finer attributes where it does, so the explanation reflects the model's real capabilities rather than generic common sense.","Calibrating the classifier with the tree as a hierarchical pseudo-label improves both in-domain accuracy and out-of-distribution robustness, according to the reported experiments.","The framework extends to multi-label settings such as chest X-ray diagnosis, where per-finding subtrees are attached to a single root."],"supporting_citations":[{"why":"Provides the in-context prompting method used to elicit structured attribute trees from the large language model.","marker":"[45]"},{"why":"Supplies the point-to-set prototype distance and the similarity form used for routing and tree refinement.","marker":"[10, 58]"},{"why":"Supplies the hierarchical multi-label contrastive loss used to calibrate the vision model on the generated parse trees.","marker":"[92]"},{"why":"Supplies the contrastive vision-language model used to filter retrieved support images by their alignment to attribute text.","marker":"[53]"},{"why":"Supplies the text-to-image diffusion model used as one source of support images for grounding attributes.","marker":"[56]"},{"why":"Supplies the tree-structured RNN decoder baseline (TrDec) that LVX is compared against for explanation plausibility and stability.","marker":"[79]"},{"why":"Supplies the neural-backed decision tree baseline used for classification-performance comparison with explainable methods.","marker":"[77]"},{"why":"Supplies the DR-CIFAR and DR-ImageNet rationale datasets that motivate the hierarchical benchmarks and baseline comparisons.","marker":"[47]"}],"fun_headline_variants":["LLM-built attribute trees explain and improve vision models","Vision model explained by LLM-grown tree of visible attributes","Tree of visual attributes, LLM-drafted, embedding-pruned","LLM grows and prunes an attribute tree to mirror a vision model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the images gathered for each attribute, filtered at a fixed similarity threshold, are accurate and disentangled examples of that attribute in the explained model's embedding space, and that the nearest-neighbor distance from a query embedding to a node's support images faithfully reports which attribute the model is actually using.","fun_headline_variants_meta":{"raw":{"variants":["LLM-built attribute trees explain and improve vision models","Vision model explained by LLM-grown tree of visible attributes","Tree of visual attributes, LLM-drafted, embedding-pruned","LLM grows and prunes an attribute tree to mirror a vision model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001189,"raw_usage":{"total_tokens":4903,"prompt_tokens":938,"completion_tokens":3965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":3894}},"tokens_in":554,"tokens_out":3965,"duration_ms":25952,"temperature":1.0,"reasoning_tokens":3894,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:02:31.852686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a classifier whose training images always have a particular region (for example, dog ears) blacked out, so it cannot have learned ear features, and run LVX on it: if the explained tree still routes dog images to 'ears' nodes as top-k explanations, the routing is dominated by the LLM template rather than by the model's actual features, and faithfulness fails.","supporting_citations":[{"cited_title":"Use all the labels: A hierarchical multi-label contrastive learning framework","cited_arxiv_id":null,"evidence_quote":"Supplies the hierarchical multi-label contrastive loss used to calibrate the vision model on the generated parse trees."},{"cited_title":"A tree-based decoder for neural machine translation","cited_arxiv_id":null,"evidence_quote":"Supplies the tree-structured RNN decoder baseline (TrDec) that LVX is compared against for explanation plausibility and stability."}],"review_version":1}