Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model's behavior can be traced to the specific training stage that caused it — pretraining, fine-tuning, or alignment — by estimating what would have changed if that stage had been skipped, without retraining.

desk verdict The stage-level attribution framing is genuinely new, but the deep-model validation changes the counterfactual the paper claims to estimate. read the letter →

arxiv 2506.00175 v5 pith:FC2UJ2DT submitted 2025-05-30 cs.LG cs.AI

Shichang Zhang , Hongzhe Du , Jiaqi W. Ma , Himabindu Lakkaraju This is my paper
classification cs.LGcs.AI MSC 68T0762D20
keywords accountabilityattributioncounterfactualstageeffectstrainingdynamicsinfluencefunctionsspuriouscorrelationdetectionmodelauditingpotentialoutcomesTaylorapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that accountability in multi-stage AI development can be quantified: for any stage — pretraining, fine-tuning, or alignment — one can estimate how the final model's behavior would have differed if that stage's updates had never happened, without retraining. The proposed estimator, the Accountability Attribution Score (AA-Score), is a first-order Taylor approximation that propagates the effect of skipping each update step through the observed training trajectory, incorporating learning-rate schedules, momentum, and weight decay. The paper validates the score against true counterfactuals obtained by retraining with the stage skipped, reporting average correlation 0.9456 on MNIST and roughly 0.55–0.98 on image, text, and medical-imaging tasks. If correct, the method gives auditors a practical tool to localize where spurious correlations, mislabeled-data damage, or backdoor signals entered a model and to target interventions at the responsible stage.

What carries the argument

The load-bearing object is the one-step propagator matrix $M_k = \begin{pmatrix} I - \eta_k(H_k + \lambda I) & -\eta_k \mu I \\ H_k + \lambda I & \mu I \end{pmatrix}$, the linearization of the SGD-with-momentum-and-weight-decay update that carries the effect of a skipped step from step $k$ to step $k+1$, where $H_k$ is the loss Hessian on batch $B_k$, $\eta_k$ the learning rate, $\mu$ the momentum factor, and $\lambda$ the weight decay. This matrix drives the estimator chain: the exact state effect of skipping step $t$ is $(-\eta_t v_{t+1},\, v_{t+1} - v_t)$, and propagating it to the final time through products of these matrices yields $E_t$, so the AA-Score $\hat{\tau}_{K,S} = \sum_{t_i \in S} \nabla_\theta \gamma(x, \theta_K)^\top E_{t_i}$ is the first-order estimate of the counterfactual change in model behavior from removing stage $S$. The fact that $E_t$ is independent of the test input is what makes the method practical: one logged training run supports attribution queries on arbitrary inputs and performance functions.

What would settle it

Train a deep network with aggressive early learning (for example, ResNet-18 on CIFAR-10 with a high learning rate and no warmup), use AA-Score to attribute the first stage of training, then retrain from scratch with that stage skipped and compare every estimated stage effect against the true counterfactual effect. If the correlation between estimates and ground truth collapses, or signs flip, for early stages where the error bound's exponential term $e^{2\eta\Lambda(K-\min(S))}$ is large, the first-order premise fails where the theorem says it should; if the correlation holds there, the bound is loose and the method works more broadly than its own analysis claims.

Watch

Extended reading notes

Core claim

The central claim is that the causal effect of a training stage on final model behavior — the counterfactual change in model output on an input $x$ if stage $S$ had been skipped — is well approximated by the AA-Score $\hat{\tau}_{K,S} = \sum_{t_i \in S} \nabla_\theta \gamma(x, \theta_K)^\top E_{t_i}$, a dot product between the final-model gradient of a performance function $\gamma$ and a stage vector $E_t$ assembled from the observed optimizer updates. Each $E_t$ estimates the displacement of the final parameters caused by skipping one training step, obtained by propagating that step's exact state effect — the observed parameter and velocity changes at that step — through the remaining training dynamics via linearized propagator matrices that include Hessians, momentum, and weight decay; by the linearity of the first-order expansion, a stage's total effect is the sum of its steps' effects. Because $E_t$ does not depend on the test input or the performance function, it is computed once during training and reused for any downstream attribution query. The paper reports that these scores track true retraining counterfactuals closely and that skipping stages with high positive scores for a confounding feature measurably reduces the model's reliance on that feature.

Load-bearing premise

The estimate is only as good as the assumption that the counterfactual training path — what would have happened with a stage skipped — stays close to the path the model actually took, so that a first-order Taylor expansion computed along the observed trajectory remains accurate; the paper's own error bound grows exponentially with the number of remaining training steps and the optimizer's instability, which is why it identifies early-stage attribution in chaotic regimes as the place where the method is most likely to fail.

Editorial extensions

If this is right

  • Auditors could locate the development stage responsible for a specific failure or harmful behavior and quantify its contribution without retraining the model.
  • Spurious correlations can be traced to particular stages, and dropping those stages measurably reduces the model's reliance on the confounder, as shown by decreased confounder log-likelihood on CelebA and CivilComments.
  • The reusable $E_t$ representation means a single logged training run can answer arbitrary 'what if this stage had not happened' queries on any test input or performance metric.
  • AA-Score reflects optimizer choices: lower learning rate or momentum reduces the attributed effect of a stage, and much higher weight decay reduces the signal's prominence, consistent with how those parameters shape updates.
  • Backdoor-poisoned batches can be surfaced by their positive attribution to target-label predictions, though the signal weakens when poisoning is fully dispersed across all batches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's exponential error bound implies a reliability ordering it does not operationalize: audit conclusions about late, stable, moderate-learning-rate stages deserve far more weight than conclusions about early high-learning-rate stages, and a production tool could attach that confidence flag to each score.
  • The framework is stage-agnostic, so a natural untested extension is attributing instruction-tuning and alignment behaviors — sycophancy, refusal styles, jailbreak susceptibility — to the SFT or RLHF stage of a language model.
  • A quantitative prediction follows from the machinery itself: attribution error should grow exponentially as the attributed stage moves earlier in training, which the paper's own retraining protocol could test without any new methodology.
  • The dispersed-backdoor result marks a granularity limit: batch-level stage attribution cannot resolve signals deliberately spread thin across every batch, which implicitly defines when data-point-level attribution methods remain necessary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a counterfactual framework for attributing a trained model's final behavior to the stages of its development (pretraining, fine-tuning, alignment, etc.). The proposed AA-Score estimator approximates the effect of skipping a training stage without retraining, using a first-order Taylor expansion around the observed optimization trajectory. The estimator incorporates learning-rate schedules, momentum, and weight decay through Hessian-based propagator matrices. The authors validate AA-Score by correlating it with retraining-based counterfactuals on MNIST, CelebA, CivilComments, and chest X-ray datasets, and demonstrate applications to spurious-correlation detection and backdoor-attack auditing.

Significance. The problem is timely and practically important: stage-level accountability is a natural question for multi-stage ML pipelines, and the paper proposes a parameter-free, reusable estimator that requires no retraining. The single-step derivation in Section 4.2 is clean, and the MNIST insertion/mislabel experiments, which do use the formally defined in-place counterfactual, provide meaningful evidence that the estimator captures stage effects in small models. If the open concerns about the counterfactual definition and the error bound are resolved, this would be a useful tool for model auditing and debugging.

major comments (3)
  1. [§4.1 vs. §E.1] The counterfactual used for validation in the multi-stage experiments is not the counterfactual defined in Section 4.1. In §4.1, skipping a stage sets ξ_{k+1}(0_S)=ξ_k(0_S) for k∈S, so the momentum buffer from before the skipped stage carries forward unchanged. In contrast, the retraining protocol in §E.1 resumes from the checkpoint before the skipped stage 'with the momentum buffer and learning-rate scheduler reinitialized.' This is a different intervention: it resets the optimizer state and learning-rate schedule that the formal counterfactual would preserve. Consequently, the high correlations in Table 2, Table 5, and the Shift columns of Table 1 compare AA-Score against a ground truth that is not the estimand defined in eq. (5). The only valid tests of the formal estimand are the MNIST insertion and mislabel experiments, which skip steps in-place within the same training run. The authors should either run in-place counterfactuals for the deep model experiments or explicitly redefine the estimand to match the reinitialization protocol and adjust the derivation accordingly.
  2. [Theorem 4.1 / Appendix B] The proof of Theorem 4.1 defines a 'stage linearized displacement' ˆδ_k in eq. (24) with the recurrence ˆδ_{k+1}=ˆδ_k+u_k for k∈S and ˆδ_{k+1}=M_kˆδ_k for k∉S. This object does not coincide with the estimator in Estimator 4.3, which computes each single-step effect u_{t_i} and propagates it through all subsequent steps, including later steps in S, via the product of propagator matrices. For a two-step stage S={1,2}, the estimator gives M_2 u_1 + u_2, while the recurrence in eq. (24) gives u_1 + u_2 (without the M_2 factor on u_1). The error bound in eq. (11) therefore bounds the error of a different estimator, not the one proposed. The authors should supply a proof that applies to Estimator 4.3 or modify the estimator to match the recurrence and state the consequences explicitly.
  3. [Estimator 4.3 / §A.2] The additivity of stage effects as a sum of single-step effects relies on dropping higher-order cross terms in the multivariate Taylor expansion of §A.2. These cross terms are not analyzed anywhere in the paper. Even for purely linear update maps, the sum of single-step effects differs from the exact counterfactual effect of skipping multiple steps; for a simple one-dimensional linear system the discrepancy involves products of the skipped update magnitudes (e.g., (a_1−1)(a_2−1) times the state). The paper should either include these cross terms in the error bound, prove that the cross terms are negligible under the stated assumptions, or provide empirical evidence (e.g., by comparing the sum of single-step estimates against a full multi-step in-place retraining) that the omission is harmless in the tested regimes.
minor comments (5)
  1. [Tables 1, 2, 5] Report standard errors or repeated-run variance for the correlation coefficients; as written, the numbers are point estimates from single seeds, so the reader cannot assess the stability of the validation.
  2. [§5.2 (Figure 3)] In Figure 3 (g-i), the legend labels such as 'lr=0.001 lr=0.0001' are ambiguous; clarify that the first value applies to stage 1 and the second to stage 2, and consider a more explicit visual separation of the two stages.
  3. [§5.2] The sentence introducing the distribution-shift retraining protocol refers to 'full distribution-shift stages' but the actual protocol is only described in Section E.1; add a pointer to E.1 in the main text.
  4. [Throughout] The paper uses 'AA-Score' both for the stage-level causal effect and for the estimator of that effect; suggest reserving AA-Score for the estimator and using τ_{K,S} or 'estimated stage effect' for the quantity being estimated.
  5. [§5.5 / §E.6] The backdoor experiment plots are only described in the appendix; include at least one example figure or a quantitative summary (e.g., AUC separation) in the main text, since this is a claimed practical application.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the AA-Score estimator is derived from the SGD update equations via Taylor expansion with no fitted parameters, and its validation uses independently computed retraining counterfactuals.

full rationale

The paper's central derivation chain (Estimators 4.1-4.3 and Theorem 4.1) does not reduce to its inputs. Estimator 4.1 starts from the exact state difference at the intervened step (eq. 6), then propagates it with the Jacobian M_k of the explicit SGD-with-momentum-and-weight-decay update (eqs. 1-4), so the estimator is a first-principles Taylor approximation along the observed trajectory. There are no fitted parameters, no calibration to validation data, and no quantity is defined in terms of the target effect it is supposed to predict. Estimator 4.3 sums single-step effects via a multivariate Taylor expansion, which is a standard first-order decomposition rather than a definitional identity. Theorem 4.1 derives an error bound from Lipschitz and local-stability assumptions; it does not assume the conclusion. The validation compares AA-Score against retraining counterfactuals computed independently of the estimator. For the MNIST insertion/mislabel settings, the retraining protocol skips the same in-place update steps, matching the formal counterfactual. For the multi-stage experiments, the retraining protocol reinitializes momentum and the learning-rate scheduler (Sec. E.1), which differs from the formal counterfactual defined in Sec. 4.1; that is a correctness/experimental-validity concern, not circularity, because the estimator is not fitted to those retraining outputs. The paper's self-citations (e.g., Deng et al. 2025b, Ley et al. 2024) appear only in related-work lists and are not load-bearing for the derivation. The fact that both the estimator and the ground-truth counterfactual are built from the same SGD update rules reflects consistency, not circularity. No step in the claimed derivation chain is equivalent to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on first-order Taylor validity and on the availability of the observed trajectory. No numerical constants are fitted; the method uses observed updates and derivatives. The key fragility is the closeness assumption behind Theorem 4.1.

assumptions (4)
  • domain assumption The loss Hessian is L-Lipschitz and the update maps are locally stable (||Dψ_k(ξ_k)|| ≤ e^{η_k Λ}) on the region between observed and counterfactual trajectories.
    Invoked in Theorem 4.1 and used to derive eq. (11); if false, the error bound does not hold and the estimator can diverge.
  • domain assumption First-order Taylor expansion of the state and performance function around the observed trajectory is accurate enough.
    Used in Estimators 4.1 to 4.3 and Appendix A; equivalently the counterfactual trajectory stays close to observed. The degree of violation is not measured in the deep-model experiments.
  • domain assumption The full training trajectory, gradients, optimizer states, and hyperparameters are recorded.
    Needed to compute E_t and M_k; the paper notes in Section 6 that without it, extra approximations are required.
  • domain assumption Potential outcomes are well-defined for stage interventions, with consistency and no unmodeled interference between stages.
    The framework treats stage skipping as a well-defined counterfactual; validation via retraining operationalizes this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems." pith.science (2026). https://pith.science/paper/FC2UJ2DT

@misc{pith2026250600175,
  author       = {Pith},
  title        = {Pith review of: Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FC2UJ2DT}},
  note         = {Machine review of arXiv:2506.00175}
}
read the original abstract

Modern AI systems are typically developed through multiple stages-pretraining, fine-tuning rounds, and subsequent adaptation or alignment, where each stage builds on the previous ones and updates the model in distinct ways. This raises a critical question of accountability: when a deployed model succeeds or fails, which stage is responsible, and to what extent? We pose the accountability attribution problem for tracing model behavior back to specific stages of the model development process. To address this challenge, we propose a general framework that answers counterfactual questions about stage effects: how would the model's behavior have changed if the updates from a particular stage had not occurred? Within this framework, we introduce estimators that efficiently quantify stage effects without retraining the model, accounting for both the data and key aspects of model optimization dynamics, including learning rate schedules, momentum, and weight decay. We demonstrate that our approach successfully quantifies the accountability of each stage to the model's behavior. Based on the attribution results, our method can identify and remove spurious correlations learned during image classification and text toxicity detection tasks that were developed across multiple stages. Our approach provides a practical tool for model analysis and represents a significant step toward more accountable AI development.

Figures

Figures reproduced from arXiv: 2506.00175 by the authors.

Figure 1
Figure 1. Illustration of the accountability attribution problem for a generative AI model developed in three stages (e.g., pre￾training, fine-tuning, and alignment). The accountability for the output “Paris” is attributed to the three stages. & Ranzato, 2017; Kornblith et al., 2019; Raghu et al., 2019; He et al., 2022; Chen et al., 2020; Radford et al., 2019; Ouyang et al., 2022; Hu et al., 2022). Each stage builds on the pr… view at source ↗
Figure 2
Figure 2. Illustration of the accountability attribution problem for a stage. A model is developed in N stages, each comprising a sequence of parameter updates. The goal is to estimate the causal effect of a stage, e.g. Stage 2, on the final model behavior. The diagram shows the actual development process (top) and a counterfactual process (bottom) where Stage 2 had not occurred. The accountability of Stage 2 for predicting a… view at source ↗
Figure 3
Figure 3. Performance effect on MNIST. Each bar shows the AA-Score estimation (τˆK,t) for an update step t, which can be aggregated to stage effects. A positive τˆK,t indicates that the stage leads to a higher log-likelihood, i.e., the stage is beneficial. (a) Accurately detect an influential stage of an inserted data point. (b) Capture a stage processing mislabeled data, demonstrating their negative effect on the test perfor… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance effect estimation (left) and retraining likeli￾hood with confounding features as the label (right) on CIVILCOM￾MENTS. There are 8 positive and 2 negative stages. We show the likelihood decreases when skipping the top 3 positive stages and increases when ski…
Figure 5
Figure 5. Figure 5: The effect of inserting a test digit ’4’ during training on the model’s ability to classify four different digits. (a) is the same case as [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: AA-Score results for a BadNets backdoor attack on CIFAR-10 with ResNet18 following BackdoorBench. Poisoned data are inserted during training in dispersed batches. Each batch contains either regular examples (blue) or poisoned examples (red). Batches with poisoned examp…
Figure 7
Figure 7. Figure 7: AA-Score results for a BadNets backdoor attack on CIFAR-10 with ResNet18 following BackdoorBench. Poisoned data are fully dispersed during training, so each batch contains both poisoned and regular examples (all batches shown in red). Batch contributions can be either …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages

  1. [1]

    =ξ k(T= 1). Estimator for effect on the state:For any target time k≥t+ 1 , the first-order Taylor expansion of ξk(ϵ) around ϵ= 1 (the observed path) is1: ξk(ϵ)≈ξ k(1) + ∂ξk(ϵ) ∂ϵ ϵ=1 (ϵ−1)(14) Getξ k(0)with the approximation and plug it into the effect on the state: wk,t =ξ k(1)−ξ k(0)≈ξ k(1)− ξk(1)− ∂ξk(ϵ) ∂ϵ ϵ=1 = ∂ξk(ϵ) ∂ϵ ϵ=1 (15) Let ˆwk,t = ∂ξk(ϵ) ∂...

    work page 2017

  2. [2]

    The storage forP ((t+1)→K) itself isO(p 2)

    if Hessian-vector products are used efficiently within the matrix multiplication. The storage forP ((t+1)→K) itself isO(p 2). Structured approximations for scalingThe full propagation couples all p parameters and is therefore difficult to scale directly. A common heuristic is to restrict the computation to the parameters of each layer l (with dimension pl...

    work page 2023

  3. [4]

    URL https://arxiv.org/abs/2002. 02497. Cole, S. R. and Frangakis, C. E. The consistency state- ment in causal inference: a definition or an assumption? Epidemiology, 20(1):3–5,

    work page 2002

  4. [8]

    doi: 10.18653/v1/2024.acl-long.834

    Association for Computational Linguis- tics. doi: 10.18653/v1/2024.acl-long.834. URL https: //aclanthology.org/2024.acl-long.834/. Ley, D., Srinivas, S., Zhang, S., Rusak, G., and Lakkaraju, H. Generalized group data attribution.arXiv preprint arXiv:2410.09940,

    arXiv 2024

  5. [9]

    Causal Responsibility Attribution for Human-AI Collaboration

    Qi, Y ., Schölkopf, B., and Jin, Z. Causal responsibility attribution for human-ai collaboration.arXiv preprint arXiv:2411.03275,

    work page Pith review arXiv

  6. [13]

    Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771,

    Wolf, T., Debut, L., Sanh, V ., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771,

    arXiv 1910

  7. [16]

    to O(K|B| P l p2 l ). If a model contains one or a few very large layers, those layers can be further partitioned into blocks, reducing the corresponding term from p2 l to P b p2 l,b, where pl,b is the parameter dimension of block b within layer l. This block-wise variant trades off attribution fidelity for lower memory and runtime by dropping the interac...

    work page 2017

  8. [2005]

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G

    doi: 10.1198/016214504000001880. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learn- ing. InInternational conference on machine learning, pp. 1139–1147. PMLR,

    doi:10.1198/016214504000001880

Show all 16 references
  1. [2009]

    T., Ley, D., Dai, Q., Huang, B., Huang, J., et al

    Deng, J., Hu, Y ., Hu, P., Li, T.-W., Liu, S., Wang, J. T., Ley, D., Dai, Q., Huang, B., Huang, J., et al. A survey of data attribution: Methods, applications, and evaluation in the era of generative ai.SSRN Electronic Journal, 2025a. doi: 10.2139/ssrn.5451054. URL https://ssr...

    doi

  2. [2013]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Riv- ière, M., et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    arXiv

  3. [2017]

    URL https://arxiv

    doi: 10.1109/CVPR.2017.243. URL https://arxiv. org/abs/1608.06993. Ilyas, A., Park, S. M., Engstrom, L., Leclerc, G., and Madry, A. Datamodels: Understanding predictions with data and data with predictions. InProceedings of the 39th Interna- tional Conference on Machine Learni...

    2017 arXiv

  4. [2019]

    Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    arXiv

  5. [2020]

    Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G

    doi: 10.1016/j.media.2020.101797. Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners.Advances in neural information processing systems, 33:22243–22255,

    2020

  6. [2023]

    Nuanced metrics for measuring unintended bias with real data for text classification

    Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. InCompanion pro- ceedings of the 2019 world wide web conference, pp. 491–500,

    2019

  7. [2024]

    net/forum?id=3NaqGg92KZ

    URL https://openreview. net/forum?id=3NaqGg92KZ. Barshan, E., Brunet, M.-E., and Dziugaite, G. K. Relatif: Identifying explanatory training samples via relative in- fluence. InInternational Conference on Artificial Intelli- gence and Statistics, pp. 1899–1909. PMLR,

    1909

  8. [2025]

    Wang, X., Peng, Y ., Lu, L., Lu, Z., Bagheri, M., and Sum- mers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.