{"id":"fc829139-0638-48cc-a61a-7a2607446330","arxiv_id":"2502.01048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A compilation of prior peer-reviewed methods arguing that explaining vision models needs concept extraction and human alignment, not just saliency maps.","lead":"This doctoral thesis gathers and connects the author's published work on explaining deep vision models, from saliency maps to concept discovery. It argues that pixel-level attributions only show where a model looks, and that real understanding requires concept-based methods and human-aligned training.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CRAFT's concept extraction is treated as revealing the model's causal vocabulary, but the provided evidence supports correlation and human plausibility, not causal faithfulness; this gap underdetermines the thesis's 'what' claim.","rationale":"The reader's weakest assumption identifies precisely the point on which the thesis's central narrative turns: CRAFT's concepts must be the features the model actually uses, not merely low-rank factorizations that correlate with human labels. I agree that this is the most load-bearing concern. The thesis argues from the insufficiency of attribution methods (they only give 'where') to the necessity of concept-based methods (to get 'what'), and the existence of a faithful 'what' is the essential premise that makes the second half of the argument work. The provided evidence for CRAFT—human agreement and fidelity-style perturbation tests—establishes that the extracted concepts are meaningful to humans and sensitive to input changes, but not that they correspond to the model's causal computation. A low-rank NMF can discover structure in the activation distribution that is driven by dataset regularities rather than by the model's decision function, and importance scores computed via perturbations can reflect correlation without causation. This is an internal validity gap: it does not require an external consensus, only that the thesis's own claims about CRAFT be backed by evidence of causal faithfulness. Since the reader already conditions the verdict on stronger external validation, my analysis supports keeping the verdict at CONDITIONAL (i.e., unchanged). A concrete synthetic-intervention test would settle the concern: if CRAFT's top concepts fail to predict the effects of counterfactual activation interventions, the 'what' claim is unsupported. I found no other concern that is more load-bearing than this one; the first premise about attribution methods is well-supported by the human studies in Chapter 2, and the alignment work in Chapter 3 is presented as an alternative hypothesis rather than as a necessary component of the main claim.","tokens_in":50704,"tokens_out":3285,"duration_ms":38141,"concrete_test":"Create a synthetic dataset with known independent generative factors (e.g., shape, color, texture) and train a small CNN. Apply CRAFT to extract concepts from a frozen trained model. For each extracted concept in the top-10 by importance, intervene by setting the activation component along that concept direction to a counterfactual value (zero or a value drawn from another image) and record the change in the model's output probability. Compare the ranking of these intervention effects to CRAFT's importance ranking using Spearman rank correlation. Additionally, run the same CRAFT extraction on an identical but randomly initialized model; if the extracted concepts still align with human-identifiable generative factors, the extraction is capturing dataset statistics rather than model-specific computation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that attribution methods only show 'where' a model looks, while CRAFT provides the 'what' by extracting concepts from activation spaces. The load-bearing step is Section 4.2's treatment of non-negative matrix factorization (NMF) of activations as the model's internal vocabulary, with importance estimated via Sobol indices (Appendix C.1.4). What evidence would establish that these concepts are the causal features driving predictions? The human experiments in Appendix C.1.5 and fidelity experiments in C.1.6 show that extracted concepts align with human judgments and with input perturbations, but that is plausibility/correlation, not causal faithfulness. NMF is a low-rank reconstruction of the activation distribution; it can recover components that correlate with human labels without being the features the model's decision function actually depends on. If the concepts are not causal, then CRAFT's explanations are convenient narratives rather than faithful accounts, and the thesis's proposed shift from 'where' to 'what' is not justified. This is a correctness risk for the main argument, not a disagreement with consensus: the internal evidence does not rule out the alternative that CRAFT re-describes the data rather than the model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This doctoral thesis compiles a series of peer-reviewed contributions in explainable computer vision and integrates them into a single narrative. Chapter 2 develops attribution methods and evaluation tools: a stability-based metric (MeGe/ReCo), a Sobol-indices black-box attribution method accelerated with quasi-Monte Carlo sampling, the EVA method with verified perturbation guarantees, and a human-centered meta-predictor evaluation of attribution usefulness. The central negative finding is that attribution methods reveal only where a model looks, not what it perceives, and are therefore of limited use in complex scenarios. Two remedies are then explored: aligning models with human explanations, including training with human rationales and 1-Lipschitz constrained networks (Chapter 3), and concept-based explainability, concretized by CRAFT, a unified extraction/importance framework, MACO feature visualization, and an interactive demo (Chapter 4). The thesis concludes with a discussion of future directions.","tokens_in":50987,"tokens_out":4071,"duration_ms":50200,"significance":"If the central claim is established, the thesis would provide a well-articulated argument for shifting explainability research from pixel-level attribution to concept-level and human-aligned methods. The individual methods have already passed peer review and are benchmarked on ImageNet, Pascal VOC, COCO, FRSign, and human studies. Concrete strengths include the reproducibility orientation: code is released via Xplique, CRAFT, and project repositories, and several derivations (Sobol indices estimators, EVA's verified perturbation analysis, the dictionary-learning unification of concept extraction) are technically grounded. The thesis also makes an honest attempt to evaluate methods with actual users rather than relying only on proxy metrics. The main risk is that the most load-bearing interpretive step, namely that CRAFT's extracted concepts are the causal features underlying model decisions, is supported by correlation, human plausibility, and input-perturbation fidelity rather than by causal intervention evidence; this gap directly affects the 'what' claim that motivates the proposed shift.","major_comments":[{"comment":"The central 'what' claim rests on treating the non-negative matrix factorization of activation vectors as the model's internal vocabulary, with importance estimated by Sobol indices. The evidence in C.1.5 and C.1.6 demonstrates that extracted concepts align with human judgments and with input perturbations, but that is plausibility and correlation, not causal faithfulness. NMF is a low-rank reconstruction of the activation distribution, and it can recover components that correlate with human labels without being the features on which the decision function actually depends. The thesis should either add a direct intervention test, such as ablating or perturbing concept directions and measuring the resulting prediction change against a control dictionary matched for reconstruction error, or explicitly moderate the claim that CRAFT explains 'what' the model perceives.","section":"§4.2 and Appendix C.1.4–C.1.6"},{"comment":"The negative conclusion that attribution methods are of little use in complex scenarios is load-bearing because it motivates both subsequent hypotheses. The manuscript states that real users were tested but does not, in the provided text, give enough detail on participant numbers, task design, statistical power, or confidence intervals to rule out the possibility that the null result is an artifact of the specific visualization or interface. For a claim of this strength, the human study should be reported with the same rigor as the other experiments, including a precise definition of what counts as 'useful' and a comparison against a baseline explanation format.","section":"§2.6 and Appendix A.5"},{"comment":"The alignment hypothesis is examined in part through 1-Lipschitz networks, but Section 3.3 is only a brief outline in the provided manuscript. The conclusion that robustness improves human-aligned explanations is therefore established only for this constrained function class and depends on an optimal-transport perspective that is not presented in enough detail to verify. Either the section should be expanded to include the full derivation and experiments, or the conclusion should be narrowed so that the thesis does not imply a general alignment benefit across standard architectures.","section":"§3.3 and Chapter 3 conclusion"}],"minor_comments":[{"comment":"The convolution definition says 'where N denotes the convolution operation,' but N is already a notation used for the set of integers and does not match the convolution symbol used elsewhere in the formula; this is a typographical or notation error that should be corrected.","section":"§1.1.2, Definition 1.1.6"},{"comment":"The reference to 'Agarwal et al.' near the end of the stability discussion is incomplete; the authors and year should be supplied.","section":"§2.1.4"},{"comment":"The text refers to 'Chapter Chapter 2 and Chapter Chapter 4'; the duplicated word and the awkward phrasing should be cleaned up.","section":"§1.3"},{"comment":"The title advertises 'Large Vision Models,' but the experiments are conducted on ResNet-50, VGG-16, EfficientNet, MobileNet, and ViT-S/32, which are not 'large' by current standards. The title should either be qualified or the scope should be stated more precisely.","section":"Title and Chapter 4"},{"comment":"Several chapters read as summaries or reprints of the underlying conference papers rather than as a fully integrated monograph; adding explicit signposts about which content is reproduced verbatim and which is new synthesis would help the reader evaluate the incremental contribution.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised by the reader is substantiated by the manuscript itself: the CRAFT evaluation in C.1.5 and C.1.6 measures alignment with human judgments and perturbation responses, not causal influence on the model's decisions. This is the single most important issue to resolve before the thesis's global narrative can be accepted as stated. A revision that adds a causal intervention experiment or softens the 'what' claim would be sufficient to make the argument defensible. The manuscript is a compilation of already peer-reviewed works, so the individual technical contributions are sound; the burden is on the new synthesis and the interpretive claims, not on the reprinted methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a doctoral thesis that re-presents the author's peer-reviewed XAI papers (Sobol, EVA, CRAFT, MACO, harmonization) inside a narrative arc: attribution methods tell you where a model looks, concept methods tell you what it sees, and aligning models with human reasoning improves both. As a compilation of already-reviewed work, it introduces no new equations, experiments, or methods; the novelty is in the framing and in the unified dictionary-learning view of concept extraction (Section 4.4), which was itself published at NeurIPS 2023. What the thesis does well is lay out the full stack—from black-box attribution with formal guarantees (EVA) to concept extraction and feature visualization—backed by open-source code and human experiments. That is real, reproducible evidence, and the individual methods are solid.\n\nThe soft spot is the load-bearing claim that CRAFT's concepts are the model's actual vocabulary, rather than plausible summaries. NMF on activations produces factors that correlate with human labels and pass fidelity-style tests, but the thesis does not show that the model's decision function causally depends on these factors. The human and fidelity experiments in Appendices C.1.5–C.1.6 support correlation and plausibility; they do not rule out the alternative that CRAFT re-describes the data manifold rather than the model's computation. That gap is real, and the thesis would be stronger if it framed the 'what' claim as an interpretation with supporting evidence instead of an established fact. The 'first' claims (first formal-guarantee attribution method, etc.) are also asserted without a systematic literature review; some are likely true, but the thesis does not establish priority. Chapter 3.3 is an outline rather than a full section, so the 1-Lipschitz alignment argument is underdeveloped in this arXiv version. To his credit, the author does list limitations in Appendix C.1.1, but the main text does not carry those caveats into the central argument.\n\nWho is this for: graduate students entering XAI and researchers wanting a single entry point to this line of work. It is not a new research contribution, but it is a genuinely useful synthesis. If submitted for peer review as a survey or thesis, it deserves a serious referee; the author should be asked to soften the causal claims, substantiate the priority claims, and complete the outline chapter. I would not cite the thesis itself in my own papers—I would cite the underlying peer-reviewed articles—but I would bring it to a reading group as a case study in how a research program gets framed.","headline":"A well-packaged compilation of already-published XAI methods whose central 'what vs where' claim overstates CRAFT's causal faithfulness; a useful synthesis, not new research.","tokens_in":51504,"tokens_out":3169,"would_cite":false,"duration_ms":30868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis argues that attribution methods only reveal where a vision model looks, and that real explainability requires concept-level tools (CRAFT, MACO) and human-aligned training.","keywords":["explainable AI","attribution methods","concept-based explainability","Sobol indices","feature visualization","1-Lipschitz networks","vision models","CRAFT"],"falsifier":"Train a model on a dataset where the true decision rule is known and controllable, for example a clean classifier that relies on a single shape feature; if CRAFT's concept importance ranking does not put that feature first, or if removing the top concept directions from the activations leaves the model's predictions unchanged, then the concepts are not faithful.","tokens_in":50478,"feed_emoji":"👁️","tokens_out":6911,"duration_ms":61583,"temperature":0.7,"pith_summary":"This thesis argues that standard attribution methods—saliency maps and their relatives—answer only where a vision model looks, not what it perceives, and that this is why they prove inadequate in real-world, complex scenarios. To support this, the author develops a family of explainability tools: a stability-based metric for judging explanations, a Sobol-index attribution method that runs twice as fast as previous black-box approaches, and EVA, an attribution method with formal guarantees via verified perturbation analysis. User studies then show that even the best attribution methods offer little help beyond simple bias detection. The thesis concludes that explainability should move toward concept-based methods that extract the model's internal vocabulary (CRAFT and MACO) and toward training routines that align models with human reasoning, which also improve accuracy.","feed_headline":"Attribution maps say where, not what — concepts fill the gap","feed_subtitle":"Explainability must move from pixel heatmaps to concepts the model uses, and to training aligned with human reasoning.","key_machinery":"The central object is the concept vector in the activation space of the network, with CRAFT applying non-negative matrix factorization of activations to obtain a dictionary of concept vectors that are then scored by their contribution to predictions. Around this sit: Sobol indices with quasi-Monte Carlo sampling for efficient black-box attribution; verified perturbation analysis for EVA's formal guarantees; and the 1-Lipschitz constraint that bounds gradients and makes explanations stable and consistent. The unification claim treats the extraction phase as dictionary learning and the importance phase as attribution applied to concepts.","core_discovery":"The central claim is that the \"where\" information provided by attribution methods is insufficient to understand a deep vision model: in complex scenarios, knowing which pixels a model attends to does not tell you which features or concepts trigger the decision. The thesis therefore establishes that explainability must answer \"what\" the model perceives, and develops CRAFT, a method that automatically extracts the concepts used by a model from its activation space and estimates their importance, together with MACO, a feature-visualization method that renders those concepts. It also shows that aligning models with human explanations—either by imitation or by constraining the model to 1-Lipschitz functions—improves both accuracy and explanation quality. A unifying analysis reveals that concept extraction is a dictionary-learning problem and that concept importance estimation is, implicitly, an attribution problem.","pith_inferences":["If concept vectors are a faithful vocabulary of the model, the same machinery could be used for model editing: shifting activation directions to remove a concept, such as a spurious cue, and measuring the effect on behavior.","The where-versus-what distinction suggests a two-stage explanation pipeline—localize with attribution, then identify the concept at that location—which the thesis itself only partially implements through CRAFT combined with MACO.","Because the importance stage is shown to be a form of attribution, concept-based methods may inherit known attribution failure modes, such as sensitivity to baseline choice; stress-testing CRAFT's importance scores under different baselines would be a natural next experiment.","The alignment results raise the possibility that alignment with human explanations correlates with robustness; a direct test would be measuring adversarial robustness across harmonized and standard architectures."],"forward_implications":["If attribution methods provide only \"where\" information, then safety-critical deployments of explainability should not rely on heatmaps alone; concept-level explanations are needed.","Concept extraction and concept importance estimation form one coherent framework, so progress in attribution methods transfers directly to concept-based explainability.","Training with human-aligned objectives can increase both accuracy and explanation stability, suggesting that interpretability need not come at the cost of performance.","1-Lipschitz neural networks provide provably bounded gradients, which translates into more generalizable and consistent explanations.","Quasi-Monte Carlo Sobol indices yield black-box explanations that are twice as fast as RISE and competitive with white-box methods."],"supporting_citations":[{"why":"Introduces concept activation vectors, the idea that model behavior can be expressed through human-understandable concepts; CRAFT builds on this.","marker":"Kim et al. [2018]"},{"why":"Shows concepts can be extracted automatically from a trained model's activations; direct precursor to CRAFT.","marker":"Ghorbani et al. [2019]"},{"why":"RISE provides the black-box attribution benchmark and the deletion/insertion fidelity metrics used to evaluate and critique attribution methods.","marker":"Petsiuk et al. [2018]"},{"why":"Integrated Gradients is an axiomatic gradient-based baseline that the proposed Sobol method matches or beats while remaining black-box.","marker":"Sundararajan et al. [2017]"},{"why":"Demonstrates that several attribution methods are insensitive to model randomization, motivating the need for rigorous evaluation and underpinning the thesis's critique.","marker":"Adebayo et al. [2018]"},{"why":"Introduces meaningful perturbation and the causal view of attribution that underlies black-box methods like Sobol.","marker":"Fong and Vedaldi [2017b]"},{"why":"Supplies the algorithmic stability theory that grounds the proposed MeGe and ReCo metrics for explanation quality.","marker":"Bousquet and Elisseeff [2002]"},{"why":"Documents shortcut learning in vision models, motivating the alignment hypothesis that models should be trained to mimic human explanations.","marker":"Geirhos et al. [2020b]"},{"why":"Provides the Deel-Lip library for training 1-Lipschitz networks, used to show that constrained models yield more consistent explanations.","marker":"Serrurier et al. [2020]"}],"fun_headline_variants":["Beyond saliency: teaching vision models to say what they see","From where to what: explainability’s next step","CRAFT reveals concepts, not just pixels, in deep vision","Why heatmaps aren’t enough: models need concept reports","Aligning models with human reasoning for true explainability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that the concept vectors extracted by factoring activations correspond to the features the model actually uses to make its decisions, rather than being merely low-rank directions that happen to correlate with human labels.","fun_headline_variants_meta":{"raw":{"variants":["Beyond saliency: teaching vision models to say what they see","From where to what: explainability’s next step","CRAFT reveals concepts, not just pixels, in deep vision","Why heatmaps aren’t enough: models need concept reports","Aligning models with human reasoning for true explainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1578,"prompt_tokens":914,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":581}},"tokens_in":530,"tokens_out":664,"duration_ms":6773,"temperature":1.0,"reasoning_tokens":581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:45:45.062873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on a dataset where the true decision rule is known and controllable, for example a clean classifier that relies on a single shape feature; if CRAFT's concept importance ranking does not put that feature first, or if removing the top concept directions from the activations leaves the model's predictions unchanged, then the concepts are not faithful.","supporting_citations":[{"cited_title":"Rise: Randomized input sampling for explanation of black-box models","cited_arxiv_id":null,"evidence_quote":"RISE provides the black-box attribution benchmark and the deletion/insertion fidelity metrics used to evaluate and critique attribution methods."}],"review_version":1}