Pith. sign in

REVIEW 4 major objections 3 minor 10 references

Data Shapley is optimizer-dependent, and this paper's Adam-aware closed-form estimator matches ground-truth marginal contributions (R>0.99) at about 95% of standard training speed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 06:05 UTC pith:EEOHNNGP

load-bearing objection Useful computational trick and a real observation about optimizer dependence, but the main theorem as printed cannot be a per-sample Shapley value — the formula has no z in it. the 4 major comments →

arxiv 2602.00329 v4 pith:EEOHNNGP submitted 2026-01-30 cs.LG cs.AI

In-Run Data Shapley for Adam Optimizer

Meng Ding , Zeqing Zhang , Di Wang , Lijie Hu This is my paper
classification cs.LG cs.AI
keywords data attributionData ShapleyAdam optimizerIn-Run Shapleyadaptive optimizationdata pruninginfluence functionsLinearized Ghost Approximation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that data Shapley values are not intrinsic to data points but depend on the optimizer, and that the standard in-run approximation built for SGD fails under Adam (Pearson R≈0.11). It then derives an Adam-aware closed-form estimator by redefining each step's utility under a fixed-state assumption and Taylor-expanding Adam's variance scaling. The resulting scores match ground-truth one-step Adam utility changes almost perfectly (R>0.99) while a linearized ghost trick computes them without per-sample gradients, keeping throughput at about 95% of plain training. If right, this makes data attribution practical for modern Adam-trained models and shows that attribution methods must be matched to the optimizer.

Core claim

The central claim is that data value is optimizer-dependent: the same sample's contribution differs under SGD vs Adam, and SGD-based in-run proxies diverge from true marginal contributions under Adam (Pearson R≈0.11). To address this, the paper proposes Adam-Aware In-Run Data Shapley. By applying a first-order Taylor expansion to each step's local utility and treating the optimizer state as fixed within the step, it derives a closed-form per-step Shapley value: φ_z(U) ≈ Σ_t −η_t ∇ℓ(w_t, z^val) · m_t / (√v_t + ε). This restores additivity, allowing Shapley's linearity axiom to sum per-step values. The step value is a dot-product between the validation gradient and Adam's normalized momentum,

What carries the argument

The central object is the closed-form per-step Adam-aware Shapley value φ_z(U^(t)_(1)) = −η_t ∇ℓ(w_t, z^val) · m_t/(√v_t+ε), which expresses each step's contribution as a dot-product between the validation gradient and Adam's normalized momentum. Two mechanisms carry the argument: (1) the fixed-state assumption — treating the optimizer history (m_{t-1}, v_{t-1}) as constant within each step — restores linearity and additivity; (2) the Linearized Ghost Approximation, which Taylor-expands the variance scaling 1/(√v_t+ε) around the history term and drops O(g_t^2) terms, so the Adam update becomes a linear combination of the current gradient and historical moments. This allows all pairwise gradi

Load-bearing premise

The method assumes that the perturbation u = m̂/(√v̂+ε) stays small enough that dropping O(g_t^2) terms in the Taylor expansion of Adam's variance scaling leaves attribution scores unchanged.

What would settle it

Run the ghost approximation on a configuration where u remains large — high learning rate, very small batch, or early training with big gradients — and compare ghost-computed scores against exact one-step Adam utility changes. The paper's own worst-case stress test (first 10 steps, 50 samples) already shows a drop to Pearson r≈0.75; a setting that makes u large for many steps and drives the correlation well below the reported R>0.99 would falsify the fixed-state linearization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • SGD-based in-run attribution should not be used to judge data contribution for Adam-trained models; its correlation with true marginal contributions is only about R≈0.11.
  • With the optimizer correctly accounted for, Adam-aware scores track ground-truth one-step utility changes at R>0.99 across a wide range of learning rates (10^-7 to 10^-3).
  • The Linearized Ghost Approximation keeps attribution overhead at roughly 95% of standard training throughput with identical peak memory, making in-run valuation feasible for billion-parameter models.
  • Data pruning with Adam-aware scores beats random pruning on SST-2, while SGD-based scores under the same protocol degrade sharply, so pruning decisions should be optimizer-matched.
  • Semantic source identification: Adam-aware scores rank the true training source of paraphrases and similar-topic queries far better than SGD-based proxies, indicating the method captures optimizer-mediated semantic influence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The fixed-state linearization is load-bearing: if the perturbation u = m̂/(√v̂+ε) is not small — e.g., under very large learning rates, tiny batches, or extreme gradient variance — the dropped O(g_t^2) terms could dominate and the closed-form would no longer reflect Adam's actual updates. A direct test is measuring fidelity in such regimes.
  • The same linearization strategy could extend to other adaptive optimizers (RMSProp, AdamW with decoupled decay, Adafactor) by constructing their own ghost vectors, though each requires a different preconditioner linearization.
  • If data value is genuinely optimizer-dependent, dataset curation (pruning, acquisition) should be performed with the target optimizer in mind; a dataset selected for SGD may be suboptimal for Adam and vice versa.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes an extension of In-Run Data Shapley to the Adam optimizer. It claims that under a first-order approximation the per-step Shapley value has a closed form, Eq. (4), and introduces a “Linearized Ghost Approximation” for efficient computation. The authors report near-perfect fidelity (R > 0.99) to ground-truth marginal contributions and show that data attribution is strongly optimizer-dependent (R ≈ 0.11 between SGD- and Adam-based values). They further demonstrate downstream benefits in semantic source identification and data pruning.

Significance. If the central claim were sound, the paper would make a valuable contribution: a scalable, optimizer-aware data attribution method that could replace SGD-only in-run Shapley in modern training pipelines. The problem is well motivated, and the paper includes several useful empirical elements: controlled semantic perturbation protocols, a pruning study, and attention to computational efficiency. However, the core theoretical derivation is internally inconsistent: the stated closed-form estimator is not per-sample, and the proof does not establish a Shapley marginal contribution. Because the method’s validity depends on this derivation, the contribution is not established.

major comments (4)
  1. [§4.2, Theorem 4.1 and Eq. (4)] The closed form φ_z(U^(t)_(1)) = −η_t ∇ℓ(w_t, z_val)·m_t/(√v_t+ε) contains no dependence on the training sample z. Thus it assigns the same score to every data point at step t, so it cannot serve as a per-sample Shapley value. The proof in Appendix A.1 substitutes the global Adam update for the marginal contribution U(S∪{z})−U(S), but under the fixed-state assumption the update is independent of S and z; this is not a valid Shapley marginal contribution unless one redefines the utility in a way that makes the marginal contribution constant, in which case the Shapley value is also constant and carries no attribution information.
  2. [Appendix A.2, Eq. (8)] The paper’s own detailed expansion contains a sample-specific Linear Gradient Term, −η_t ∇ℓ(w_t,z_val)·[C2_m/A_t ⊙ ∇ℓ(w_t,z)]. This term is missing from Theorem 4.1. Hence the theorem and the efficient computation are mutually inconsistent: the estimator used in the experiments (and in the Linearized Ghost method) is not the one stated as the main theoretical result. The proof of Theorem 4.1 therefore does not support the actual method.
  3. [§5.3, Figure 2b] The fidelity experiment uses “explicit one-step Adam updates” as the ground truth. That ground truth is precisely the first-order Taylor quantity that the proposed approximation linearizes, so the reported R > 0.99 largely reflects self-consistency of the linearization rather than agreement with true marginal contributions or with the retraining-based Data Shapley. The only retraining-based comparison (Figure 1) uses a different quantity and does not evaluate the proposed estimator.
  4. [Appendix D, stress test] The empirical justification of the small-perturbation assumption is narrow: it is demonstrated on DistilGPT2, first 10 steps, and 50 samples. Even in this conservative test, the worst-case Pearson correlation is only r = 0.7502, considerably below the “near-perfect fidelity” (R > 0.99) claimed in the abstract and conclusion. This indicates that the linearization can have substantial error in regimes that are not negligible, and the paper does not provide evidence that the method remains accurate throughout a full training run.
minor comments (3)
  1. [§4.1, Figure 1] The reported correlation R ≈ 0.0579 between SGD- and Adam-based Shapley values is extremely low; given the variance of TMC estimates, it would be useful to report confidence intervals and the number of permutations used. This would clarify whether the near-zero correlation reflects true optimizer dependence or noise.
  2. [§5.1.1, Table 1] The table reports average ranks but not standard errors or the number of trials. Since the claim of semantic source identification relies on these rankings, error bars or significance tests would increase confidence.
  3. [§4.3, Linearized Ghost Approximation] The notation is sometimes ambiguous: A_t(z) is defined as √v_{t-1}(z)+ε but later used as a constant vector. The relationship between the per-sample state and the global state should be clarified.

Circularity Check

2 steps flagged

Theorem 4.1's closed form is independent of the training sample—it is the global Adam update, not a per-sample Shapley value—and the R>0.99 fidelity check compares the approximation to its own unlinearized target.

specific steps
  1. self definitional [Section 4.2, Theorem 4.1 (Eq. 4); proof in Appendix A.1]
    "Note that the model update satisfies w_t = w_{t−1} − η_t · m_t/(√v_t+ε), therefore, we have the marginal contribution of z for any S⊆ B_t\z is U^(t)_(2)(S∪z)−U^(t)_(2)(S) = −η_t ∇ℓ(w_t, z^(val)) · m_t/(√v_t+ε)."

    A Shapley marginal contribution [U(S∪{z})−U(S)] must depend on which sample z is added and on the coalition S. The proof instead substitutes the global Adam update w_t − w_{t−1}, which depends on neither z nor S, so Theorem 4.1 assigns the same scalar to every training point at step t. This is not a derivation of φ_z but a redefinition of the marginal contribution as the full-batch update, so the closed form reduces to its own input by construction. The paper's own Appendix Eq. (7) contains the missing per-sample term −η_t ∇ℓ(w_t,z^(val))·[C2_m/A_t ⊙ ∇ℓ(w_t,z)], confirming that the stated theorem cannot reproduce sample-specific marginal contributions.

  2. other [Section 5.3, Fidelity Comparison; Abstract/Conclusion]
    "we evaluate the fidelity of the proposed Adam-aware closed-form approximation against the reference marginal utility of training samples measured via explicit one-step Adam updates ... achieves near-perfect fidelity to ground-truth marginal contributions (R > 0.99)."

    The 'reference marginal utility' is the exact unlinearized one-step Adam utility change—the very function whose first-order Taylor expansion defines the estimator in Theorem 4.1/Appendix A.2. Correlating the estimator with the exact value of the function it was built to approximate is a self-consistency check of the Taylor truncation, not a check against independent ground-truth Shapley values. Therefore the advertised R>0.99 does not validate the central attribution claim; it only confirms that an expansion tracks its own parent function. The only external benchmark (TMC in Section 4.1) tests optimizer dependence, not this estimator.

full rationale

The central derivation is circular in the strongest sense: the proof of Theorem 4.1 asserts that the marginal contribution of any sample z to any coalition S equals the global Adam update, which contains no z-dependence. Thus the claimed per-sample closed form is, by construction, a constant over samples at each step. The Appendix's Eq. (7) reveals the intended per-sample gradient term, but it is absent from the theorem as stated, making the headline formula self-definitional rather than derived. The fidelity experiment is also a self-consistency check: the 'ground truth' is the exact one-step Adam utility change, i.e., the unlinearized counterpart of the quantity the method approximates. Correlating a Taylor expansion with its own parent function does not establish fidelity to retraining-based Shapley values. There is no load-bearing self-citation chain; the paper's external benchmarks—TMC optimizer-dependence, data pruning on SST-2, and semantic source identification—are independent and non-circular. Because those parts have genuine content but the central closed-form estimator and headline fidelity claim reduce by construction, the score is 7 rather than 10.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical entities. The load-bearing assumptions are the fixed-state redefinition of utility, the validity of the Taylor expansion/truncation, and the identification of one-step marginal utility with the Shapley value. All three are introduced ad hoc to make the derivation work, and the last is circular.

axioms (3)
  • ad hoc to paper The utility of a subset S under Adam can be defined with moment states fixed to the global history (fixed-state assumption), making the update linear in the subset gradient sum.
    Introduced in Section 4.2 and Appendix A.2 to restore additivity; this is the key redefinition that makes Shapley computation tractable but is not the actual Adam training dynamics.
  • domain assumption The first-order Taylor expansion of the validation loss and the truncation of O(g^2) terms in the Adam update are valid in the regime of interest.
    Appendix A.2, Eq. 8; Appendix D provides empirical justification but only for a specific setup (DistilGPT2, first 10 steps, 50 samples).
  • ad hoc to paper The per-sample 'marginal contribution' measured by a one-step Adam update is the appropriate ground truth for the per-step Shapley value.
    Section 5.3 uses this as reference; it holds only under linearity, which is the very assumption the method imposes.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Run Data Shapley for Adam Optimizer." pith.science (2026). https://pith.science/paper/EEOHNNGP

@misc{pith2026260200329,
  author       = {Pith},
  title        = {Pith review of: In-Run Data Shapley for Adam Optimizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEOHNNGP}},
  note         = {Machine review of arXiv:2602.00329}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard. While recent "In-Run" methods bypass the prohibitive cost of retraining by estimating contributions dynamically, they heavily rely on the linear structure of Stochastic Gradient Descent (SGD) and fail to capture the complex dynamics of adaptive optimizers like Adam. In this work, we demonstrate that data attribution is inherently optimizer-dependent: we show that SGD-based proxies diverge significantly from true contributions under Adam (Pearson $R \approx 0.11$), rendering them ineffective for modern training pipelines. To bridge this gap, we propose Adam-Aware In-Run Data Shapley. We derive a closed-form approximation that restores additivity by redefining utility under a fixed-state assumption and enable scalable computation via a novel Linearized Ghost Approximation. This technique linearizes the variance-dependent scaling term, allowing us to compute pairwise gradient dot-products without materializing per-sample gradients. Extensive experiments show that our method achieves near-perfect fidelity to ground-truth marginal contributions ($R > 0.99$) while retaining $\sim$95\% of standard training throughput. Furthermore, our Adam-aware attribution significantly outperforms SGD-based baselines in data attribution downstream tasks.

Figures

Figures reproduced from arXiv: 2602.00329 by Di Wang, Lijie Hu, Meng Ding, Zeqing Zhang.

Figure 1
Figure 1. Figure 1: Comparison between SGD-based and Adam-based data Shapley values (Pearson [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Optimizer dependence and fidelity of data attribution. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fidelity across learning rates. Pearson cor￾relation between predicted scores and ground-truth util￾ity changes across different learning rates η. Adam￾aware approximation maintains high fidelity (R > 0.96) across the board, while the SGD-proxy exhibits unsta￾ble and lower correlation. Optimizer Dependency of Data Value. Figure 2a compares attribution scores pro￾duced by independent Adam and SGD training r… view at source ↗
Figure 4
Figure 4. Figure 4: Data pruning effect: validation loss versus training steps. We compare the full dataset, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical validity of the ghost approximation. (Left) Evolution of the effective perturba￾tion magnitude u, showing that optimization rapidly enters and remains in a small-perturbation regime. (Right) Worst-case fidelity test conducted during the first 10 optimization steps, where u is largest. Even under these early and high-magnitude conditions, the ghost approximation maintains strong agreement with the… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

10 extracted references · 8 linked inside Pith

  1. [5]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Pith/arXiv arXiv

  2. [6]

    A multilinear sampling algorithm to estimate shapley values

    Published as a paper at 3rd DATA-FM Workshop @ ICLR 2026, Brazil Ramin Okhrati and Aldo Lipani. A multilinear sampling algorithm to estimate shapley values. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7992–7999. IEEE,

    2026

  3. [7]

    Attributing data for sharpness-aware minimization.CoRR, abs/2507.04059, 2025a

    Chenyang Ren, Yifan Jia, Huanyi Xie, Zhaobin Xu, Tianxing Wei, Liangyu Wang, Lijie Hu, and Di Wang. Attributing data for sharpness-aware minimization.CoRR, abs/2507.04059, 2025a. Chenyang Ren, Huanyi Xie, Shu Yang, Meng Ding, Lijie Hu, and Di Wang. Evaluating data influence in meta learning.CoRR, abs/2501.15963, 2025b. Herbert Robbins and Sutton Monro. A ...

    Pith/arXiv arXiv

  4. [1951]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Pith/arXiv arXiv 1910

  5. [2013]

    Data banzhaf: A robust data valuation framework for machine learning

    Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. InInternational Conference on Artificial Intelligence and Statistics, pp. 6388–6421. PMLR, 2023a. Jiachen T Wang and Ruoxi Jia. A note on” towards efficient data valuation based on the shapley value”.arXiv preprint arXiv:2302.11431, 2023b. Jiachen T. Wang, ...

    Pith/arXiv arXiv

  6. [2017]

    Beta shapley: a unified and noise-reduced data valuation framework for machine learning.arXiv preprint arXiv:2110.14049,

    Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning.arXiv preprint arXiv:2110.14049,

    Pith/arXiv arXiv

  7. [2019]

    Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    Pith/arXiv arXiv

  8. [2021]

    partial_same

    Published as a paper at 3rd DATA-FM Workshop @ ICLR 2026, Brazil A IN-RUNDATASHAPLEY UNDERADAM A.1 PROOF OFTHEOREM4.1 Theorem A.1(Restatement of Theorem 4.1).In-Run Data Shapley via Adam, considering the first-order approximation, has a closed-form ϕz(U)≈ T−1X t=0 ϕz U (t) (1) ,(5) where ϕz U (t) (1) =−η t∇ℓ wt, z(val) · mt √vt +ϵ , t= 0, . . . , T−1. Pro...

    2026

  9. [2022]

    Towards efficient data valuation based on the shapley value

    Ruoxi Jia, Daiwei Dao, Boxin Wang, and et al. Towards efficient data valuation based on the shapley value. InAISTATS, 2019a. Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve G ¨urel, Bo Li, Ce Zhang, Costas J. Spanos, and Dawn Xiaodong Song. Efficient task-specific data valuation for nearest neigh- bor algorithms.ArXiv, abs/1908.08619, 20...

    Pith/arXiv arXiv 1908

  10. [2025]

    Ferenc Ill´es and P´eter Ker´enyi

    URLhttps://arxiv.org/abs/2411.11667. Ferenc Ill´es and P´eter Ker´enyi. Estimation of the shapley value by ergodic sampling.arXiv preprint arXiv:1906.05224,

    Pith/arXiv arXiv 1906

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.