{"id":"8a0f095f-ca78-4dfb-a914-3fcb5871d7d3","arxiv_id":"2501.11096","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A reproduction of Wang and Wang's class-contrastive explanation method confirms the main quantitative claims, fails to reproduce the heatmap visualizations, and extends the method to XGradCAM, FullGrad, and Vision Transformers.","lead":"A reproducibility study of a NeurIPS explainable-AI paper confirms the main quantitative findings but reports that the visual heatmaps could not be reproduced and that the original paper's equations contained errors. The study releases working code and shows the contrastive method generalizes to XGradCAM, FullGrad, and Vision Transformers.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's proof that softmax-backprop equals the Eq. 1/2 weighted contrast is arithmetically false; since §3.2 builds the weighted variants from softmax-backprop, the reproduction may be evaluating a different method than the original.","rationale":"The reader's weakest assumption was that the original method was faithfully reconstructed from incomplete documentation. The false algebra in Eq. 4 is a concrete, checkable instance of that risk: the method implemented in §3.2 is not mathematically the method defined by Eqs. 1-2 unless α_s is redefined as p_s. This is an internal inconsistency in the paper's own equations, not a disagreement with external consensus. It does not automatically invalidate the reproduction of Claim 1, because Figure 1 uses an explicit gradient-based weighted contrast, but it undermines the paper's assertion that backprop-from-softmax is a faithful implementation and therefore the Table 1 and XGradCAM/ViT generalization results. The reader already flagged Eq. 4 in the rationale but did not make it the load-bearing concern, hence partial agreement. The released code and author contact are genuine strengths, but they do not repair the algebra. The CONDITIONAL verdict stands; the condition should require either correcting Eq. 4 or explicitly redefining α_s and re-running the affected experiments.","tokens_in":10941,"tokens_out":10677,"duration_ms":104114,"concrete_test":"Numerically test Eq. 4 on the released VGG-16/CUB-200 model: for a set of images, compute W_t = φ_t - Σ_{s≠t}α_s φ_s (Eqs. 1-2) and S_t = (1/p_t)Σ_s (∂p_t/∂y_s)φ_s, using the same φ_s maps. If the cosine similarity of W_t and S_t is not 1, the equivalence fails. Then re-run the blurring/masking protocol with explicit W_t for GradCAM and XGradCAM; if the wtd. columns in Table 1 change materially, especially for p_t≈0.5 samples, the reproduction's 'similar results' are evidence about softmax-gradient contrast, not the original weighted contrast.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1, Eq. 4 asserts that back-propagating from the softmax neuron p_t gives a map proportional to the original weighted contrast. Under the paper's own Eq. 2, α_s = exp(y_s)/Σ_{k≠t}exp(y_k) = p_s/(1-p_t). The Jacobian is ∂p_t/∂y_s = p_t(δ_{ts} - p_s), so softmax-backprop (for a linear explanation method) produces S_t = p_t φ_t - p_t Σ_{s≠t} p_s φ_s = p_t φ_t - p_t(1-p_t)Σ_{s≠t} α_s φ_s. The stated weighted contrast is W_t = φ_t - Σ_{s≠t} α_s φ_s. S_t is a scalar multiple of W_t only if p_t = p_t(1-p_t), i.e. p_t=0; in general the target and non-target coefficients are scaled by different factors (p_t versus p_t(1-p_t)). Thus the ∝ step is false. Since §3.2 says the weighted GradCAM, Linear Approximation, and XGradCAM variants were generated by back-propagating from p_t, the 'wtd.' columns in Table 1 and the generalization claims are not about the Eq. 1/2 method as written. If the original method instead used α_s = p_s, then Eq. 2 misstates the original. Either way, the central reproducibility claim is exposed to the risk the reader identified: the thing being compared may not be the published method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a reproducibility study of Wang & Wang (2022), 'Why Not Other Classes?': Towards Class-Contrastive Back-Propagation Explanations. The authors re-implement the original experiments on ImageNet and CUB-200, verify Claims 1 and 2 quantitatively, and extend the method to XGradCAM, FullGrad, and Vision Transformers (ViT). They report that the original paper's quantitative findings are broadly reproduced, while the heatmap visualizations differ substantially; they attribute this to undocumented visualization details and an erroneous equation in the original paper. The paper releases an open-source repository containing all code used for the study.","tokens_in":11231,"tokens_out":8109,"duration_ms":77255,"significance":"The study is a valuable contribution to the reproducibility literature: it provides an independent implementation, identifies concrete errors and omissions in the original paper (wrong perturbation limit, incorrect sign perturbation equations, missing fine-tuning details), and tests generalization of the contrastive back-propagation idea beyond the original scope. The code is released under an open license, which facilitates further verification. I specifically checked the key theoretical step in Eq. (4): the chain-rule expansion of the softmax Jacobian gives ∂p_t/∂y_t = p_t(1-p_t) and ∂p_t/∂y_s = -p_t p_s for s≠t, which yields a common factor p_t(1-p_t) multiplying the weighted contrast in Eq. (1); the claimed proportionality is therefore exact, contrary to a concern raised in the review process. The main caveat is that the reproduction relies on author communication and an unofficial repository for crucial hyperparameters, which introduces uncertainty about whether the reconstructed method exactly matches the published one; the authors acknowledge this limitation transparently.","major_comments":[{"comment":"The blurring/masking protocol uses equal-area perturbations, which differs from the original paper's protocol. While the authors justify this change to avoid bias, the quantitative comparison to the original Table 1 is therefore not a direct reproduction. The paper should either also report results under the original protocol or provide a sensitivity analysis to show how this deviation affects the 'similar results' conclusion.","section":"Section 3.2.2, Table 1"},{"comment":"The generalization claim is only partially successful. For ViT with GradCAM, the authors report 'qualitatively worse results' and 'nonsense' explanations, with only rare good cases; positive evidence comes mainly from gradient-weighted attention rollout. The abstract's statement that 'generalization seems to be generally good' is therefore overstated and should be revised to reflect the method- and architecture-dependent success.","section":"Sections 3.4 and 3.5"},{"comment":"The paper is transparent about the missing details in the original paper, but it should also discuss the sensitivity of the reproduced results to the assumptions made, such as the choice of VGG-16 with batch normalization from the unofficial repository, the exact optimizer settings, and the corrected epsilon. A brief statement about which results are robust to these choices would strengthen the reproducibility verdict.","section":"Section 4.1"}],"minor_comments":[{"comment":"The phrase 'GradCAM and and Gradient-weighted attention rollout' contains a duplicated 'and'.","section":"Section 2.2"},{"comment":"The word 'calculateing' should be 'calculating'.","section":"Appendix A"},{"comment":"The notation for the softmax probability is inconsistent ('pt' and 'pT'); please use a uniform notation.","section":"Throughout"},{"comment":"The x-axis label is unclear; specify whether 'logit value' refers to the raw logit or the input to the softmax.","section":"Figure 6"},{"comment":"The phrase 'the only difference being the visualization of heatmaps' is too strong given that the quantitative results are described as 'similar but not identical'; suggest rewording to 'the most salient difference'.","section":"Abstract"}],"recommendation":"minor_revision","confidential_remarks":"The stress-test concern about Eq. (4) does not hold: the proportionality is exact with a factor p_t(1-p_t). The paper's honest reporting of limitations and its release of code are strengths. The minor revisions requested concern qualifications of the generalization claim and directness of the quantitative comparison; none of these issues undermines the central conclusion that the original paper's quantitative findings are reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a careful, honest reproduction study of Wang & Wang's contrastive explanation paper. The authors reimplement the method, confirm the quantitative trends of the original's perturbation and blurring/masking experiments, and add extensions to XGradCAM, FullGrad, and Vision Transformers with released code. They also identify genuine errors in the original: the wrong perturbation limit (epsilon = 3e-3 vs 1e-3) and an incorrect clamping/indexing equation. That alone is useful.\n\nThe extensions are modest but real. The ViT results are mixed: contrastive GradCAM mostly fails, while the attention rollout variant works better—so the generalization claim is honest rather than oversold. FullGrad needed normalization to produce a contrastive map, which they flag.\n\nSoft spots: the tables report means without error bars or significance, so the reader can't tell whether the differences between original and weighted are meaningful. The headline 'similar results' sits uneasily with the heatmap mismatch; they attribute it to unspecified visualization details, but they don't quantify the discrepancy or reproduce the original's exact visualization. The reconstruction depended on contacting the authors for the epsilon and using an unofficial repo for VGG-16 details; that is a caveat, not a flaw, and they're transparent about it.\n\nOne note from the stress-test: the concern about Eq. 4 does not survive re-derivation. The softmax Jacobian gives p_t(δ_ts - p_s), and summing over all s gives p_t φ_t - p_t Σ_s p_s φ_s = p_t(1-p_t)[φ_t - Σ_{s≠t} (p_s/(1-p_t)) φ_s], exactly proportional to the weighted contrast with the α_s defined in Eq. 2. So the softmax-backprop implementation is faithful to the stated method.\n\nOverall, this is a genuine contribution to reproducibility, not a takedown. The paper deserves a serious referee. I'd send it to a venue that values replication studies, with requests for variance information and a more careful treatment of the visualization mismatch.\n\nRecommendation: accept for peer review.","headline":"A solid, transparent reproduction study that fixes real errors in the original and ships code; the Eq. 4 objection in the stress-test is a false alarm.","tokens_in":11790,"tokens_out":4418,"would_cite":true,"duration_ms":39134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This reproducibility study re-implements Wang & Wang's class-contrastive back-propagation explanations and finds the quantitative results hold — perturbation experiments and blurring/masking tables match — while the published heatmaps…","keywords":["class-contrastive explanations","back-propagation explanations","weighted contrast","softmax back-propagation","explainable AI","Vision Transformers","reproducibility study","GradCAM"],"falsifier":"Obtain the original authors' fine-tuned VGG-16 weights and their exact heatmap normalization, interpolation, and overlay code; if the reproduced weighted-contrast heatmaps then match the published figures, the mismatch is an artifact of this review's assumptions, and if they still differ, the published visualizations are not reproducible from the paper's own description.","tokens_in":10733,"feed_emoji":"🎯","tokens_out":6396,"duration_ms":60681,"temperature":0.7,"pith_summary":"This paper re-implements the class-contrastive back-propagation explanations proposed by Wang & Wang (2022) and checks three claims: that perturbing pixels selected by weighted contrast tracks softmax probability and accuracy, that the method highlights decisive regions when several classes compete, and that it transfers to other back-propagation methods. The author's reproductions match the original paper's quantitative results on the tested models and datasets, and the method generalizes to XGradCAM, FullGrad, and Vision Transformers with modest modifications. The one unresolved piece is the heatmap visualization, which could not be made to look like the published figures because the original paper lacks code, fine-tuning details, and contains an erroneous perturbation equation. The value of the review is a corrected recipe, an open-source implementation, and a concrete diagnosis of where the original paper's reproducibility fails.","feed_headline":"Contrastive explanations reproduce, except heatmaps","feed_subtitle":"Re-running Wang & Wang's softmax back-prop method confirms its quantitative claims and extends it to ViTs.","key_machinery":"The load-bearing object is the weighted-contrast identity: for any back-propagation explanation $\\varphi^t$ computed from logits, the contrastive explanation is $\\varphi^t_{\\text{weighted}} = \\varphi^t - \\sum_{s\\neq t} \\alpha_s \\varphi^s$ where $\\alpha_s = \\exp(y_s)/\\sum_{k\\neq t} \\exp(y_k)$. For gradient-based methods this equals, up to a scale, the gradient of the softmax output $p_t$ with respect to the explained layer. The paper makes the identity operative by back-propagating from the $p_t$ neuron, and uses corrected clamped update equations for the perturbation experiments. This identity is what lets the method transfer across architectures and explanation methods.","core_discovery":"The central discovery, stated on the paper's own terms, is that the weighted-contrast explanation $\\varphi^t_{\\text{weighted}} = \\varphi^t - \\sum_{s\\neq t} \\alpha_s \\varphi^s$, with $\\alpha_s$ the softmax weight over non-target logits, is realized for gradient-based methods simply by back-propagating from the target softmax neuron $p_t$ rather than the logit $y_t$. This reproduction confirms that the perturbation and blurring/masking results of the original paper hold for the reconstructed setup, and that the recipe extends to XGradCAM, FullGrad, and Vision Transformers when ReLU is removed or normalization is applied. The heatmaps, however, do not reproduce, and the paper attributes that gap to missing documentation, an unstated epsilon of $3\\times 10^{-3}$ (not $10^{-3}$), and errors in the clamped gradient-sign update in the original paper.","pith_inferences":["The paper's observation that explanation weight grows with logit strength suggests that normalizing each class's explanation before the weighted subtraction would sharpen contrast on near-tie images; this is a testable extension the authors themselves gesture at.","Because the collapse of the weighted method for $p_2 < 0.1$ is explained by softmax gradient vanishing, contrastive methods are most informative exactly where the model is indecisive — a regime that misclassification debugging targets, so the boundary is a feature rather than a bug.","The same softmax-backprop trick may also apply to generative or language models whenever a probability distribution over mutually exclusive outputs replaces the logit, though the paper does not test this."],"forward_implications":["Weighted contrast can be implemented by switching the back-propagation target from logit $y_t$ to softmax output $p_t$, making the method a one-line change for many existing explanation libraries.","The recipe carries over to ViTs and to XGradCAM and FullGrad, so class-contrastive explanations are not restricted to the original CNN/GradCAM setting.","The corrected perturbation equations and $\\epsilon = 3\\times 10^{-3}$ are needed to reproduce the original paper's Figure 3; using the printed $\\epsilon = 10^{-3}$ changes the curves.","The author's open-source repository supplies the missing fine-tuning and implementation details, so future reproductions can be checked against deterministic code rather than reconstructed from prose."],"supporting_citations":[{"why":"The original paper under review; supplies the weighted-contrast method, claims, and figures this work reproduces.","marker":"Wang & Wang, 2022"},{"why":"Defines GradCAM, the core explanation method whose softmax-backprop variant is tested; also noted as precedent for back-propagating from the softmax output.","marker":"Selvaraju et al., 2019"},{"why":"Specifies the VGG-16 architecture used in the activation-space and blurring/masking experiments.","marker":"Simonyan & Zisserman, 2014"},{"why":"Provides the CUB-200 dataset on which VGG-16 is fine-tuned and the contrastive visualizations are evaluated.","marker":"Wah et al., 2011"},{"why":"Introduces Vision Transformers and attention rollout, the generalization target for the ViT experiments.","marker":"Dosovitskiy et al., 2020"},{"why":"Defines XGradCAM, one of the alternative back-propagation methods adapted to weighted contrast.","marker":"Fu et al., 2020"},{"why":"Defines FullGrad, the other adapted method whose linear post-processing is modified to allow negative saliency.","marker":"Srinivas & Fleuret, 2019"},{"why":"Provides gradient-weighted attention rollout for ViTs, used to make class-specific explanations spatially coherent in the ViT tests.","marker":"Gildenblat, 2020"}],"fun_headline_variants":["Contrastive XAI reproduces, heatmaps don't","Softmax backprop contrastive method yields same numbers, not heatmaps","Repro confirms contrastive explanations work, heatmaps fail","Class-contrastive backprop method: results hold, visualizations don't"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reproduction assumes the reconstructed method — corrected epsilon and equations, VGG-16 with batch normalization, and the chosen fine-tuning hyperparameters — is what the original authors actually ran, since the original paper provides no code or training details.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive XAI reproduces, heatmaps don't","Softmax backprop contrastive method yields same numbers, not heatmaps","Repro confirms contrastive explanations work, heatmaps fail","Class-contrastive backprop method: results hold, visualizations don't"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1898,"prompt_tokens":931,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":891}},"tokens_in":547,"tokens_out":967,"duration_ms":9868,"temperature":1.0,"reasoning_tokens":891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:38:59.864038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain the original authors' fine-tuned VGG-16 weights and their exact heatmap normalization, interpolation, and overlay code; if the reproduced weighted-contrast heatmaps then match the published figures, the mismatch is an artifact of this review's assumptions, and if they still differ, the published visualizations are not reproducible from the paper's own description.","supporting_citations":[{"cited_title":"Why Not Other Classes? : Towards Class-Contrastive Back-Propagation Explanations","cited_arxiv_id":null,"evidence_quote":"The original paper under review; supplies the weighted-contrast method, claims, and figures this work reproduces."},{"cited_title":"The Caltech-UCSD Birds-200-2011 Dataset","cited_arxiv_id":null,"evidence_quote":"Provides the CUB-200 dataset on which VGG-16 is fine-tuned and the contrastive visualizations are evaluated."},{"cited_title":"Full-gradient representation for neural network visualization","cited_arxiv_id":null,"evidence_quote":"Defines FullGrad, the other adapted method whose linear post-processing is modified to allow negative saliency."},{"cited_title":"Exploring Explainability for Vision Transformers , December 2020","cited_arxiv_id":null,"evidence_quote":"Provides gradient-weighted attention rollout for ViTs, used to make class-specific explanations spatially coherent in the ViT tests."}],"review_version":1}