{"id":"aab5e03b-0f46-43db-80c1-07b61c0106fc","arxiv_id":"2608.12935","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DECAF decomposes paired perturbation responses into endpoint-aligned evidence, endpoint-opposed contradiction, and endpoint-null fragility, and its components track independently measured model behavior better than raw magnitude on controlled and natural-image benchmarks.","lead":"This paper introduces DECAF, a method that separates a model's reaction to a changed input into evidence, contradiction, and fragility, depending on how each step of the change relates to the final effect. The authors report that these three parts carry behavioral meaning that raw response size hides, and that short DECAF trajectories match or beat standard attribution baselines at lower compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.4% ImageNet-9 role-agreement may rest on an endpoint confound: YE/YC labels and DECAF's E/C share the same endpoint contrast d, and matching Abs does not control |d|.","rationale":"The paper's central contribution is a decomposition that claims to assign semantic roles observable in paired perturbation responses. The strongest empirical support for this claim is the 96.4% role-agreement result on ImageNet-9, which the reader correctly identifies as potentially circular. My reading of Section 6 and Appendix F confirms the mechanism: the behavioral indicators YE and YC are endpoint-defined, and DECAF orients E and C from the same endpoint contrast d. Because the magnitude-matching procedure controls only Abs, not M = |d|, the matched pairs may differ substantially in endpoint effect size, allowing the endpoint to drive the apparent agreement. This is a clear correctness risk for the headline number, not merely a stylistic issue. At the same time, the paper has substantial independent support: the controlled 3D Shapes experiments use behavioral targets measured from accuracy drops, label swap rates, and held-out interventions that do not reuse DECAF's construction, and the Covertype audit similarly measures realized behavior from held-out predictions. Those results support the semantic meaning of E, C, and F, even if they do not rescue the specific 96.4 versus 35.0 comparison. The reader's CONDITIONAL verdict is therefore appropriate: the method is plausible and broadly validated, but the ImageNet-9 headline needs an additional endpoint-magnitude control or a clear disclosure of the confound. I would not reject the paper, but I would require the additional analysis before relying on the headline claim.","tokens_in":27674,"tokens_out":5429,"duration_ms":58777,"concrete_test":"In the matched-pair analysis of Section 6.2, additionally match or stratify by endpoint magnitude M = |d| (e.g., require relative |d| difference <= 5% as well, or bin by M-quintiles) and recompute role-agreement; also evaluate an endpoint-only role predictor that assigns E if d > epsilon, C if d < -epsilon, F otherwise, on the same matched pairs. If DECAF's 0.964 drops toward the Abs baseline once |d| is controlled, or if the endpoint-only predictor matches ~0.96, the headline claim is an endpoint-magnitude artifact rather than evidence of trajectory semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 6.2 the behavioral indicators are defined directly from endpoint predictions (Appendix F.4): YE marks a same-to-random background replacement that changes the predicted class or drops true-class probability by at least 0.20, and YC marks a switch to the new background's class under Same-Next. DECAF orients E and C with the same endpoint contrast d = q(x+) - q(x-) (Section 3, Eq. 1). The matched-pair construction controls Abs = E + C + F, the path-integrated magnitude, not the endpoint magnitude M = |d|. Since YE/YC are endpoint-defined and E/C inherit the sign and magnitude of d, matched Abs bins can still contain wide variation in |d|; the 'largest component' can then track endpoint information rather than trajectory semantics. Section 6.1 already shows endpoint magnitude alone reaches AUROC 0.960 for the evidence behavior, so endpoint information nearly saturates that label. The controlled experiments in Section 5 and Appendix I are genuinely independent and support the components, but they do not validate the absolute 96.4 versus 35.0 headline, which requires ruling out this confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DECAF, a post-hoc decomposition of paired perturbation responses into evidence E, contradiction C, and fragility F. The decomposition routes the signed stage response r(t) by its relation to the final contrast d = q(x+)-q(x-): aligned mass to E, opposed mass to C, and endpoint-null response to F, with conservation Abs = E+C+F. The authors prove uniqueness under endpoint-relative axioms (Theorem 1), show that the decomposition strictly refines magnitude (Theorem 2), and validate the components in controlled 3D Shapes and Covertype experiments, on ImageNet-9, and on external attribution benchmarks. The headline empirical claim is that after matching ordinary response magnitude, the largest DECAF component agrees with an independently observed behavior in 96.4% of ImageNet-9 cases versus 35.0% for magnitude alone.","tokens_in":27918,"tokens_out":8363,"duration_ms":84290,"significance":"The theoretical results are elementary and largely definitional, but the paper is appropriately modest about this: Theorem 1 is a uniqueness statement within an explicitly endpoint-relative semantics, and the paper's value lies in the behavioral validation. The controlled 3D Shapes and Covertype studies are genuinely careful: they use held-out behavioral targets defined without DECAF, they treat the model or training trajectory as the statistical unit, and they report bootstrap intervals. The correlations between C and label-swap rate (0.961) and between F and endpoint-null prediction change (0.974) are strong evidence that the components carry the intended semantics in settings where the ground truth is controlled. The forward-only, black-box-friendly implementation and the honest discussion of endpoint-versus-trajectory value (Section 7 and Appendix H) are also strengths. The main open question is whether the ImageNet-9 headline result is contaminated by endpoint information shared between the behavioral labels and the DECAF orientation.","major_comments":[{"comment":"The headline 96.4% versus 35.0% role-agreement does not yet rule out an endpoint confound. The behavioral indicators YE and YC are defined directly from endpoint predictions in Appendix F.4: YE marks a same-to-random background replacement that changes the predicted class or drops true-class probability by at least 0.20, and YC marks a switch to the new background's class under Same-Next. DECAF orients E and C with the same endpoint contrast d = q(x+)-q(x-) (Section 3), and the matched-pair construction in Section 6.2 controls Abs = E+C+F, not |d|. Section 6.1 reports AUROC 0.960 for endpoint magnitude on the evidence behavior, so endpoint information nearly saturates that label. I therefore cannot accept the 96.4% figure as evidence for trajectory semantics until |d| is controlled. Please repeat the matched-pair analysis with |d| additionally matched (for example, within 5% or within narrow strata), and report the role-agreement accuracy of an endpoint-only rule that uses sign(d) and |d|. If the gap persists under this control, the confound is resolved.","section":"Section 6.2 / Appendix F.4"},{"comment":"The role-agreement metric needs a chance baseline under the overlap rule. Because YE, YC, and YF are allowed to overlap and credit is divided among tied largest coordinates, a case with multiple active behaviors can be scored correct regardless of which DECAF coordinate is largest. The reported 0.964 could overstate semantic agreement if many matched cases have two or three active indicators. Please report the distribution of active-indicator counts among the 8,289 matched comparisons, the expected agreement of a random role assignment under the same overlap and tie rule, and, ideally, the agreement restricted to cases with exactly one active indicator.","section":"Section 6.2 / Appendix F.5"},{"comment":"The trajectory gain over endpoint-only M is negative on ImageNet-1k for DECAF-3 and DECAF-5 and only +0.007 for DECAF-9, as shown in Table 5 and Appendix H.3. The paper discusses this limitation clearly, but the abstract's phrase 'short forward-only DECAF trajectories outperform the tested general-purpose attribution baselines' should be qualified in the abstract itself, since the endpoint-only reference also outperforms those baselines on ImageNet-1k and since the trajectory gain is negative for two of the three DECAF variants on that benchmark.","section":"Section 7 / Table 5"}],"minor_comments":[{"comment":"The sentence 'The controlled experiment in Section 5.See Appendix B for the proof' is missing a predicate or a period; it should be rewritten as a complete sentence.","section":"Section 5 (Proposition 1 paragraph)"},{"comment":"The title block contains 'OFEVIDENCE' and 'ANDFRAGILITY' without spaces; the typography should be fixed.","section":"Title block"},{"comment":"The sentence 'Figure 10 contains the panel moved from the main text' is an editing artifact and should be replaced with a normal reference to the figure.","section":"Appendix F.7"},{"comment":"The two Covert et al. entries (2021a and 2021b) have identical titles and should either be merged or distinguished with a note, since they appear to be the same work.","section":"References"},{"comment":"Table 1 reports model calls per image for ImageNet-9, while Table 11 reports rows per image for ImageNet-1k; the main text should state explicitly that the difference arises from the number of factors (one background pair versus sixteen patches) to avoid apparent inconsistency.","section":"Tables 1 and 11"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually honest about its limitations, including the 11-pair endpoint-matched audit in Appendix I.6 and the endpoint-versus-trajectory ablation in Appendix H. The controlled experiments are strong enough that the decomposition itself is of interest even if the ImageNet-9 headline is adjusted. The main risk is the 96.4% versus 35.0% claim: if the authors add the |d| control and the overlap-aware chance baseline, I would expect the paper to be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: DECAF is a simple endpoint-relative decomposition — route each stage's signed response by its relation to the final contrast, with a null branch for endpoint-inactive pairs — and the authors back it with an unusually broad set of experiments. The controlled validation is legitimate and careful. The headline ImageNet-9 number, 96.4% versus 35.0%, has a real confound and should be read with caution.\n\nWhat's new: the specific decomposition is new as a package, though it is the Jordan decomposition of the endpoint-oriented response plus a null branch (they acknowledge this in Appendix B.1). The contribution is the semantics: evidence, contradiction, and fragility as endpoint-relative roles, and the claim that these roles track independently measured behavior. The 3D Shapes experiments are well designed: shortcut reliance via wall-color reversal correlates 0.936 with the evidence margin, contradiction tracks the label-swap rate at 0.961, and fragility tracks prediction-change rate on endpoint-null pairs. The Covertype transfer across 135 models is thorough and honestly reports that some training treatments fail to induce the intended mechanism. The ImageNet-9 audit is broad — 72 models — and includes careful splits, baseline comparisons, and even a FunnyBirds implementation check.\n\nSoft spots: the 96.4% role-agreement in Section 6.2 is the main one. The behavioral indicators YE and YC are defined directly from endpoint prediction changes (Appendix F.4), and DECAF orients E and C using the same endpoint contrast d. Matching Abs does not control |d|, and Section 6.1 shows endpoint magnitude alone reaches 0.960 AUROC for the evidence behavior. So the matched-pair comparison may largely re-discover endpoint information rather than trajectory semantics. The controlled experiments are not circular, but they do not validate the absolute 96.4 versus 35.0 claim. The authors should disclose this confound, stratify by |d| or require endpoint-matched pairs, and add confidence intervals. The theory is correct but definitional; the uniqueness theorem holds within the chosen axioms, which the paper says. The patch reveal uses linear interpolation in pixel space, so fragility may include off-manifold artifacts; that is a minor concern because the controlled fragility experiments are clean.\n\nBottom line: this paper deserves a serious referee. The decomposition is cheap and model-agnostic, and the controlled evidence is genuinely informative. The ImageNet-9 headline needs reframing or stronger endpoint controls. XAI researchers, especially those working on shortcut detection and path sensitivity, will get value from the method and from the evaluation pitfalls it exposes. I would accept it for peer review and expect major revision on Section 6.2.","headline":"DECAF is a cheap, endpoint-relative response decomposition with serious controlled validation; the ImageNet-9 role-agreement headline is confounded by shared endpoint information and needs reframing.","tokens_in":28413,"tokens_out":4657,"would_cite":true,"duration_ms":42175,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that perturbation response magnitude alone is ambiguous, and that DECAF resolves the ambiguity by routing each stage response into endpoint-aligned evidence, endpoint-opposed contradiction, or endpoint-null fragility…","keywords":["explainable AI","perturbation-based explanation","counterfactual explanation","paired reveal","response decomposition","evidence","contradiction","fragility"],"falsifier":"Re-run the ImageNet-9 magnitude-matched audit with behavioral indicators constructed entirely from held-out corruption families that are never part of the DECAF reveal path and never share the endpoint contrast $d$; if the largest-component role agreement drops toward the magnitude-only level while ordinary magnitude is still matched, the claimed independence would be falsified. A second decisive test: build a synthetic model where the endpoint contrast is chosen at random independently of intermediate responses; DECAF would then route responses into $E$ and $C$ purely by orientation, and role agreement with externally defined behavior should fall to chance if the components carry no independent information.","tokens_in":27405,"feed_emoji":"🧩","tokens_out":7425,"duration_ms":65268,"temperature":0.7,"pith_summary":"The paper argues that a perturbation response's magnitude is ambiguous: the same numerical reaction can support the final factual–counterfactual difference, oppose it, or appear only along the reveal path and vanish at the endpoint. DECAF resolves this by using the endpoint contrast $d=q(x^+)-q(x^-)$ as a semantic reference and routing each stage response $r(t)$ into endpoint-aligned evidence $E$, endpoint-opposed contradiction $C$, and endpoint-null fragility $F$, with the exact conservation $\\mathrm{Abs}=E+C+F$. The decomposition is lossless and unique under three operational axioms, and it adds no model queries beyond the paired trajectory. The paper's empirical claim is that these roles track behavior measured independently of DECAF: evidence follows consequential reliance, contradiction follows effect reversal, and fragility follows endpoint-null sensitivity, as demonstrated in a 72-model ImageNet-9 audit where the largest component agrees with observed behavior in 96.4% of magnitude-matched cases versus 35.0% for magnitude alone. A sympathetic reader would care because this offers a way to say not just how much a model reacts but what kind of response the reaction is, without retraining or gradients.","feed_headline":"Same reaction size hides three different model behaviors","feed_subtitle":"In a 72-model audit, the largest DECAF component matches observed behavior in 96.4% of magnitude-matched cases.","key_machinery":"The central object is the paired reveal together with the endpoint-relative routing rule. A paired reveal starts both inputs at a common uninformative state and produces matched trajectories $(x^+(t),x^-(t))$, so the signed response $r(t)=q(x^+(t))-q(x^-(t))$ is observed as information becomes available; the fully observed contrast $d=r(1)$ is then used only as a gate and orientation. Routing $(e,c,f)=(a(sr)_+, a(sr)_-, (1-a)|r|)$ preserves ordinary magnitude exactly, and Theorem 1 proves this is the unique triple satisfying conservation, endpoint gating, and directional support. The mechanism works because it separates the trajectory's relation to the endpoint from its size: aligned and opposed mass are distinguished only on active pairs, while endpoint-null trajectory sensitivity is quarantined as fragility. No gradients, parameters, or internal states are needed, so the decomposition applies to any black-box score.","core_discovery":"On the paper's own terms, DECAF assigns observable semantic roles to paired perturbation responses. Given a factual input and a counterfactual input, the final contrast $d=r(1)$ determines a gate $a=\\mathbf{1}_{|d|\\ge\\epsilon}$ and an orientation $s=\\operatorname{sign}(d)$; each intermediate signed response is routed as $(e,c,f)=(a(sr)_+, a(sr)_-, (1-a)|r|)$. Evidence is response mass aligned with the final effect, contradiction is mass that opposes it, and fragility is response on pairs whose final effect is negligible under the chosen threshold. Theorem 1 shows this routing is the unique splitting satisfying conservation, endpoint gating, and directional support; Theorem 2 shows the profile strictly refines ordinary magnitude, with the same $\\mathrm{Abs}=m$ arising from $(m,0,0)$, $(0,m,0)$, or $(0,0,m)$. The paper further claims the components track independently measured model behavior: on controlled 3D Shapes tasks the evidence margin correlates 0.936 with shortcut-reversal vulnerability, contradiction tracks the pairwise label-swap rate with 0.961, and fragility tracks endpoint-null prediction-change rate; on ImageNet-9, after matching ordinary magnitude within 5%, the largest DECAF component matches an independently defined behavioral indicator in 96.4% of comparisons versus 35.0% for magnitude alone.","pith_inferences":["Editorial inference: because the gate and orientation are derived from the same endpoint contrast used to define the ImageNet-9 evidence and contradiction labels, part of the 96.4% role-agreement may reflect endpoint information shared between predictor and label; a cleaner test would define all behavioral indicators from held-out perturbations that never enter the DECAF path.","Editorial inference: the contradiction channel gives model and data auditors a numeric handle on effect reversal versus attenuation, something ordinary magnitude and signed net response cannot separate; this could be used to compare models on whether a learned effect is genuinely reversed under context shift.","Editorial inference: because fragility is path-relative by construction, the decomposition suggests a diagnostic practice of reporting $(E,C,F)$ under several reveal paths rather than a single number, using the path-stability of $E$ and $C$ as an indicator of robust reliance.","Editorial inference: the method's forward-only, gradient-free nature means it could be lifted to models with discrete outputs or remote score APIs; a testable extension would verify whether the components still track behavioral measures when $q$ is a margin or logit rather than a probability."],"forward_implications":["If $\\mathrm{Abs}$ is replaced by $(E,C,F)$, any decision rule based on ordinary magnitude is still implementable, while the reverse is impossible in general; magnitude-matched comparisons can therefore reveal semantic differences that magnitude alone erases.","When the evaluation target repeats the same intervention that defines the endpoint, endpoint-only magnitude already carries most attribution signal; short DECAF trajectories add their clearest value when rankings must transfer to a different intervention.","Changing only the reveal path from blend to nested-patch raises ordinary response by about 80% without increasing evidence, so path-sensitivity is concentrated in fragility and contradiction rather than in the endpoint-aligned component.","On forward-only attribution benchmarks, short DECAF trajectories outperform the tested general-purpose gradient or sampling baselines, and on a 1B-scale DINOv2 model a short trajectory matches IG-32 with 4.75× lower wall time and 2.36× lower peak memory.","The decomposition provides an observable endpoint-relative semantics, not a claim about latent causes of trajectory responses; fragility marks response on endpoint-null pairs without diagnosing off-manifold artifacts, boundary uncertainty, or calibration effects."],"supporting_citations":[{"why":"Defines perturbation-based explanation by measuring prediction changes under altered inputs, the object class DECAF refines.","marker":"Fong & Vedaldi, 2017"},{"why":"Supplies the ImageNet-9 background variants used to build natural factual–counterfactual pairs in Section 6.","marker":"Xiao et al., 2021"},{"why":"Provides the 3D Shapes generative-factor environment used for exact counterfactual interventions in the controlled validation.","marker":"Burgess et al., 2018"},{"why":"Supplies Integrated Gradients, the strong gradient-based baseline that DECAF matches at lower compute on DINOv2.","marker":"Sundararajan et al., 2017"},{"why":"Supplies RISE, the sampling baseline used in the external attribution comparisons.","marker":"Petsiuk et al., 2018"},{"why":"Supplies FunnyBirds and its part-based evaluation protocols used in the external attribution benchmark.","marker":"Hesse et al., 2023"},{"why":"Supplies the ImageNet IDSDS benchmark and published IG scores used for implementation checks and the ImageNet-1k comparison.","marker":"Hesse et al., 2024"},{"why":"Provides the comparison-of-experiments notion underlying Theorem 2's strict-refinement claim.","marker":"Blackwell, 1953"}],"fun_headline_variants":["DECAF: One reaction size, three semantic roles","Magnitude alone matches behavior 35% of time, DECAF 96.4%","Perturbation response decomposed into evidence, contradiction, fragility","Same reaction size hides evidence, contradiction, fragility","DECAF splits response magnitude into meaningful components"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the behavioral indicators used to validate DECAF are genuinely independent of DECAF's construction: on ImageNet-9 the evidence and contradiction labels are defined from the same endpoint contrast DECAF uses for gating and orientation, so the agreed role may partly reflect shared endpoint information.","fun_headline_variants_meta":{"raw":{"variants":["DECAF: One reaction size, three semantic roles","Magnitude alone matches behavior 35% of time, DECAF 96.4%","Perturbation response decomposed into evidence, contradiction, fragility","Same reaction size hides evidence, contradiction, fragility","DECAF splits response magnitude into meaningful components"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2207,"prompt_tokens":1123,"completion_tokens":1084,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":1001}},"tokens_in":739,"tokens_out":1084,"duration_ms":10586,"temperature":1.0,"reasoning_tokens":1001,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:46.787219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ImageNet-9 magnitude-matched audit with behavioral indicators constructed entirely from held-out corruption families that are never part of the DECAF reveal path and never share the endpoint contrast $d$; if the largest-component role agreement drops toward the magnitude-only level while ordinary magnitude is still matched, the claimed independence would be falsified. A second decisive test: build a synthetic model where the endpoint contrast is chosen at random independently of intermediate responses; DECAF would then route responses into $E$ and $C$ purely by orientation, and role agreement with externally defined behavior should fall to chance if the components carry no independent information.","supporting_citations":[],"review_version":1}