Pith. sign in

REVIEW 5 major objections 4 minor 47 references

Daunce: Data Attribution through Uncertainty Estimation

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-example loss covariance across K perturbed models estimates the classical influence function, giving scalable attribution that works for large language models and black-box APIs.

desk verdict DAUNCE is a simple, empirically strong attribution method whose central theorem is unsupported as proven; worth engaging for the black-box results. read the letter →

arxiv 2505.23223 v1 pith:QGQUEQ3K submitted 2025-05-29 cs.LG

Xingyuan Pan , Chenlu Ye , Joseph Melkonian , Jiaqi W. Ma , Tong Zhang This is my paper
classification cs.LG
keywords trainingdataattributioninfluencefunctionsuncertaintyestimationcovarianceblack-boxaccesslargelanguagemodelslineardatamodelingscoreperturbation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes DAUNCE, a training data attribution method that estimates how much each training example influences a model's behavior on a test point by fine-tuning K perturbed copies of the model and computing the covariance of per-example losses across those copies. The central claim, formalized in Theorem 1, is that this covariance is an approximately unbiased estimator of the classical influence function, without ever forming the Hessian. If true, DAUNCE provides a scalable route to influence-style attribution that needs only loss values, making it applicable to large models and even black-box APIs. The paper demonstrates improved linear datamodeling scores over TRAK, LoGra, and EKFAC on vision tasks, stronger subset-removal results on LLM fine-tuning, and the first attribution on proprietary OpenAI GPT models.

What carries the argument

The central object is the empirical covariance of per-example losses across K perturbed models, I(x_i,x_j) = (1/(K-1)) Σ_k (L_i(θ_k) - \bar L_i)(L_j(θ_k) - \bar L_j). The perturbation injects a random first-order term into the fine-tuning objective, and the algebra of this term turns the covariance into a Monte Carlo estimate of a quadratic form in the inverse Hessian, which is exactly the influence-function structure. Subsampling and LoRA are used to keep the K training runs cheap at LLM scale.

What would settle it

On a small model where the Hessian and the gradient outer product are known to differ (for example a two-layer ReLU network), compute DAUNCE scores and compare their ranking with exact leave-one-out influence values obtained by retraining; if the Spearman correlation is comparable to random projection baselines, the stated equivalence is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the covariance of per-example losses across K models fine-tuned on randomly subsampled, linearly perturbed objectives estimates the influence function (1/n) ∇L_i(θ0)^T H(θ0)^{-1} ∇L_j(θ0). Theorem 1 proves this in expectation under a Taylor approximation, and the practical upshot is that attribution reduces to training K cheap models and computing pairwise covariances, with no Hessian inversion and no gradient projection. This even works when only API access to loss values is available, which the paper exploits to achieve the first data attribution on proprietary LLMs.

Load-bearing premise

The theorem assumes the average outer product of per-example gradients equals the Hessian, which is exactly true only for special statistical models and only approximately true for deep networks; if that identity fails for a given model, the covariance no longer equals the influence function.

Editorial extensions

If this is right

  • Attribution becomes a drop-in tool for LLM interpretability: any user with API access to a fine-tunable model can rank training examples' influence on a query's output.
  • Because covariance estimation avoids second-order matrices, DAUNCE scales to models where Hessian-based methods are infeasible, as demonstrated on 7B-8B LLMs.
  • The fitted exponential scaling law with K means a modest number of perturbed models (around 100-200) is enough for near-peak performance, making the method cheap in practice.
  • The method can operate in strict black-box settings by replacing fine-tuning with prompt optimization, so attribution is possible even when model weights are never exposed.
  • The correlation variant behaves like influence with unit-normalized gradients, which mitigates the effect of outlier training examples with large gradient magnitudes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the covariance estimator is a randomized, second-order-free probe of the Hessian, so any perturbation scheme whose induced parameter shifts have covariance proportional to H^{-1} (for example, noise on the loss or dropout-like stochasticity) should yield the same attribution, generalizing the method beyond the specific uniform perturbation.
  • The black-box result suggests that a model owner with query-only access can test data influence, ownership, or memorization (e.g., whether a specific example caused a refusal), with auditing and legal implications the paper does not explore.
  • Testable extension: apply DAUNCE to detect backdoor triggers at scale with precision/recall metrics; the paper's case study shows qualitative retrieval of injected examples but does not report quantitative detection accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces DAUNCE, a training-data attribution method that fine-tunes K models on randomly subsampled, randomly perturbed objectives and uses the covariance (or correlation) of per-example losses across those K models as the attribution score. The authors claim a formal connection between this loss covariance and the influence function (Theorem 1, Section 3.2), report higher Linear Datamodeling Scores than TRAK, LoGra, and EKFAC on CIFAR-10/ResNet-9, present LLM-scale subset-removal results on MATH and IFEval against LoGra, and demonstrate black-box attribution on OpenAI GPT models through backdoor training-data retrieval. The empirical protocol is clearly described, and the black-box proprietary-LLM experiments are a novel contribution, but the theoretical framing has a load-bearing gap in the proof of Theorem 1.

Significance. If the covariance-influence equivalence were established, DAUNCE would offer a scalable, Hessian-free, and even black-box-compatible attribution method, which would be a meaningful advance for LLM interpretability and data debugging. The reported LDS improvements over existing baselines and the first demonstration of attribution on proprietary LLMs are valuable empirical contributions. However, the central theorem currently rests on an unstated and generally false identity between the empirical gradient outer product and the Hessian; with a corrected statement the method may still be practically useful, but the paper's theoretical motivation needs substantial revision. The manuscript does not provide code or machine-checked proofs, so reproducibility currently depends on the experimental appendices.

major comments (5)
  1. [Appendix A.2 / Theorem 1] The proof of Theorem 1 collapses (1/n) Σ_i ∇L_i(θ0)∇L_i(θ0)^T into H(θ0). This identity is not stated and is false for general losses and finite samples. Without it, the derivation yields E[Δθ_k Δθ_k^T] = H(θ0)^{-1} G(θ0) H(θ0)^{-1}/n with G(θ0) = (1/n) Σ_i ∇L_i(θ0)∇L_i(θ0)^T, not (1/n) H(θ0)^{-1}. Consequently Theorem 1, as stated, does not establish the claimed equivalence between the loss covariance and the influence function. Please either state and empirically verify the condition under which G(θ0) ≈ H(θ0) (e.g., an exponential-family or MLE setting), or reframe Theorem 1 as estimating a whitened gradient covariance and adjust Sections 3.2, 3.3, and A.3 accordingly.
  2. [Section 3.2 / Algorithm 1] The text explicitly says 'we omit the Subsample step in the theoretical analysis for notational simplicity.' This means Theorem 1 does not cover the actual procedure in Algorithm 1, which trains each perturbed model on a random subset D_k. With subsampling, the expected outer product of Δθ_k acquires a different normalization and the theorem's constant factor (1/n) no longer matches the algorithm's behavior. Please either extend the proof to the subsampled procedure or state clearly that Theorem 1 applies only to an idealized unsampled version and discuss the gap.
  3. [Appendix A.2] The proof states that σ_k := 2ξ_k − 1 has zero mean and variance 1. Since ξ_k ~ Uniform(0,1), Var(σ_k) = 1/3, not 1. This changes the constant factor in the displayed expectation and in the 'approximately unbiased' claim of Theorem 1 by a factor of 3. Rankings are unaffected by a uniform scaling, but the mathematical statement should be corrected.
  4. [Section 4.2 / Appendix C] The IFEval removal intervals [5,000, 7,500, 10,000, 12,500, 15,000, 17,500] were chosen after observing that differences between DAUNCE and LoGra only become significant after removing at least 5,000 examples. This post hoc selection of the evaluation range weakens the claim that DAUNCE 'achieves overall stronger performance' on IFEval. Please report results over a prespecified or complete range of removal sizes, or clearly label the reported range as exploratory and discuss the selection bias.
  5. [Section 4.2 / Figure 2] The LLM subset-removal curves in Figure 2 are shown without error bars or per-seed variability, despite the text in Appendix C stating that results are averaged across 3 random seeds, and TRAK is not compared at LLM scale because no implementation was available. The claim of consistent LLM-scale improvement over existing methods is therefore supported only against LoGra and without uncertainty quantification. Please add error bars or per-seed points and state the TRAK limitation explicitly in the main text.
minor comments (4)
  1. [Section 3.2] In Theorem 1, the notation L_i(θ0)^T H(θ0)^{-1} L_i(θ0) uses L_i to denote a gradient, but L_i is defined as a scalar loss; please write ∇L_i(θ0)^T H(θ0)^{-1} ∇L_i(θ0) consistently in the theorem statement and proof.
  2. [Section 3.1 / Equation (5)] The notation p(x|θ) = softmax(g(θ, x)) is unusual because softmax is over class logits, not over the input x; please clarify that p denotes the model's predictive probability for the label associated with x, and define the chain-rule step more precisely.
  3. [Figure 1 / Section 4.1] The LDS bar charts in Figure 1 do not include error bars or confidence intervals; since LDS is reported as an average over 10,000 validation examples, please provide a measure of variability across validation examples or random seeds.
  4. [Section 6] The statement that the exponential model 'precisely characterizes' the scaling behavior in Figure 6 is too strong without reporting a goodness-of-fit measure or comparing against alternative curves; please temper the claim or add fit diagnostics.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DAUNCE's estimator is independently defined and evaluated on external counterfactuals; the main gaps are mathematical rigor, not input-output circularity.

full rationale

I walked the derivation chain from the perturbed objective (Section 3.1, Eq. 5) through Theorem 1 (Section 3.2) and the appendices, and found no step in which an output is identical to an input by construction, no fitted parameter is renamed as a prediction, and no load-bearing result is imported solely from the authors' prior work. The attribution score is the sample covariance of per-example losses across K independently perturbed models (Eq. 6); this quantity is defined without reference to the Hessian or to influence functions. Theorem 1 is an attempted proof that this covariance approximately equals the influence function, not an assumption of the desired conclusion. The self-citations to uncertainty-estimation work (e.g., [11], [12], [21]) are motivational and provide background techniques; they do not carry the proof of Theorem 1, which is self-contained in Appendices A.1-A.2. Evaluation is against external counterfactuals (LDS on held-out validation examples, subset removal, backdoor retrieval), not against quantities the method was tuned to reproduce. The paper does contain real rigor concerns: the proof in Appendix A.2 replaces the empirical gradient outer product (1/n)Σ∇L_i∇L_i^T with the Hessian H(θ0), an identity that is not true for general losses and finite samples; the noise σ_k=2ξ_k-1 is said to have variance 1 but actually has variance 1/3; and the theorem explicitly omits the Subsamplestep used in Algorithm 1. These are mathematical gaps, not circularity: they affect whether Theorem 1 is proved, but they do not make the estimator's output depend on the target it is supposed to predict. For that reason, the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The theory leans on an unstated Hessian or gradient-covariance equivalence and on the assumption of tiny Δθ_k. The empirical evaluation has no fitted attribution parameters, but it includes hand-chosen hyperparameters, a post hoc evaluation range, and a fitted scaling law.

free parameters (5)
  • Subset ratio r = 0.3 (CIFAR); 512 out of 5000 examples per subset (black-box)
    Controls how much data each perturbed model sees; value is chosen by hand and affects the variance of the covariance estimate.
  • Number of perturbed models K = 200 (CIFAR), 100 (LLM), 50 (black-box)
    Chosen as a compute budget; Figure 6 shows LDS improves with K to a plateau, so K is a tunable accuracy-cost tradeoff.
  • Learning rates = Grid-searched over [1e-1, 3e-2, 1e-2, 3e-3, 1e-3] for CIFAR and [1e-3, 3e-4, 1e-4] for LLM perturbed training
    Selected per task and variant; learning rate affects the step taken during perturbed training and hence the magnitude of Δθ_k.
  • Perturbation distribution = ξ ~ Uniform(0,1), with Var(2ξ-1) treated as 1 in the proof
    The proof uses variance 1 but Uniform(0,1) gives variance 1/3; this is a constant factor in attribution scores, so rankings are unaffected, but the theorem's constant is off.
  • Scaling-law fit = a=-0.16, b=0.085, c=0.16
    Fitted to the LDS-versus-K data in Section 6 and presented as a curve that 'precisely characterizes' the trend; it is not used to make predictions.
assumptions (4)
  • domain assumption 1/n Σ_i ∇L_i(θ0)∇L_i(θ0)^T ≈ H(θ0)
    Needed in Appendix A.2 to turn E[ΔθΔθ^T] into H^{-1}/n; this is the weakest spot in the theory and is not stated in the main text.
  • domain assumption θ0 is near a local optimum so first-order and second-order Taylor expansions are valid
    Used throughout Section 3.1-3.2 to express Δθ_k and loss changes linearly; if perturbed training moves far from θ0, the covariance is no longer an influence measure.
  • domain assumption Subsampling can be omitted from the theoretical analysis without changing the estimator's structure
    Stated before Theorem 1; in practice each model is trained on a random 30% subset, which changes the effective Hessian and the gradient population.
  • ad hoc to paper Dropping the first-order gradient term in black-box training changes Δθ_k negligibly
    Justified only by one gradient-norm plot (Figure 3), not by a bound; this assumption enables the black-box variants of DAUNCE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Daunce: Data Attribution through Uncertainty Estimation." pith.science (2026). https://pith.science/paper/QGQUEQ3K

@misc{pith2026250523223,
  author       = {Pith},
  title        = {Pith review of: Daunce: Data Attribution through Uncertainty Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGQUEQ3K}},
  note         = {Machine review of arXiv:2505.23223}
}
read the original abstract

Training data attribution (TDA) methods aim to identify which training examples influence a model's predictions on specific test data most. By quantifying these influences, TDA supports critical applications such as data debugging, curation, and valuation. Gradient-based TDA methods rely on gradients and second-order information, limiting their applicability at scale. While recent random projection-based methods improve scalability, they often suffer from degraded attribution accuracy. Motivated by connections between uncertainty and influence functions, we introduce Daunce - a simple yet effective data attribution approach through uncertainty estimation. Our method operates by fine-tuning a collection of perturbed models and computing the covariance of per-example losses across these models as the attribution score. Daunce is scalable to large language models (LLMs) and achieves more accurate attribution compared to existing TDA methods. We validate Daunce on tasks ranging from vision tasks to LLM fine-tuning, and further demonstrate its compatibility with black-box model access. Applied to OpenAI's GPT models, our method achieves, to our knowledge, the first instance of data attribution on proprietary LLMs.

Figures

Figures reproduced from arXiv: 2505.23223 by the authors.

Figure 1
Figure 1. LDS results for our method variants and baselines. (a) Comparison in the white-box setting. (b) Results under [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Most Influential Subset Removal results on MATH and IFEval benchmarks, comparing [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Batch-wise gradient norms during training. We [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example queries and their top retrieved influential training examples in the black-box setting. Backdoor [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Example query image and top influential training images (positive and negative) identified by [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: LDS results of DAUNCE as a function of the number of perturbed models K. The black dashed line shows the fitted exponential scaling curve. The Scaling Law of DAUNCE We investigate how the performance of DAUNCE scales with the number of per￾turbed models K by plotting L…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 18 canonical work pages

  1. [1]

    Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    arXiv 2023

  2. [2]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InInternational conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  3. [3]

    Resolving training biases via influence-based data relabeling

    Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence-based data relabeling. InInternational Conference on Learning Representations, 2021

    work page 2021

  4. [4]

    FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

    Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging.arXiv preprint arXiv:2012.15781, 2020

    work page Pith review arXiv 2012

  5. [5]

    G-dig: Towards gradient- based diverse and high-quality instruction data selection for machine translation.arXiv preprint arXiv:2405.12915, 2024

    Xingyuan Pan, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, and Shanbo Cheng. G-dig: Towards gradient- based diverse and high-quality instruction data selection for machine translation.arXiv preprint arXiv:2405.12915, 2024

    arXiv 2024

  6. [6]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    arXiv 2024

  7. [7]

    Influence selection for active learning

    Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. InProceedings of the IEEE/CVF international conference on computer vision, pages 9274–9283, 2021. 10

    work page 2021

  8. [8]

    What is your data worth to gpt? llm-scale data valuation with influence functions.arXiv preprint arXiv:2405.13954, 2024

    Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? llm-scale data valuation with influence functions.arXiv preprint arXiv:2405.13954, 2024

    arXiv 2024

Show all 47 references
  1. [9]

    On the accuracy of influence functions for measuring group effects.Advances in neural information processing systems, 32, 2019

    Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects.Advances in neural information processing systems, 32, 2019

    2019

  2. [10]

    Confidence interval estimation by bootstrap method for uncertainty quantification using random sampling method.Journal of Nuclear Science and Technology, 52 (7-8):993–999, 2015

    Tomohiro Endo, Tomoaki Watanabe, and Akio Yamamoto. Confidence interval estimation by bootstrap method for uncertainty quantification using random sampling method.Journal of Nuclear Science and Technology, 52 (7-8):993–999, 2015

    2015

  3. [11]

    Optimal sample selection through uncertainty estimation and its application in deep learning.arXiv preprint arXiv:2309.02476, 2023

    Yong Lin, Chen Liu, Chenlu Ye, Qing Lian, Yuan Yao, and Tong Zhang. Optimal sample selection through uncertainty estimation and its application in deep learning.arXiv preprint arXiv:2309.02476, 2023

    2023 arXiv

  4. [12]

    Corruption-robust offline reinforcement learning with general function approximation.Advances in Neural Information Processing Systems, 36:36208–36221, 2023

    Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation.Advances in Neural Information Processing Systems, 36:36208–36221, 2023

    2023

  5. [13]

    Training data influence analysis and estimation: A survey.Machine Learning, 113(5):2351–2403, 2024

    Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey.Machine Learning, 113(5):2351–2403, 2024

    2024

  6. [14]

    Training data attribution via approximate unrolled differentiation.arXiv preprint arXiv:2405.12186, 2024

    Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentiation.arXiv preprint arXiv:2405.12186, 2024

    2024 arXiv

  7. [15]

    Trak: Attributing model behavior at scale.arXiv preprint arXiv:2303.14186, 2023

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale.arXiv preprint arXiv:2303.14186, 2023

    2023 arXiv

  8. [16]

    What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems, 33:2881–2891, 2020

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems, 33:2881–2891, 2020

    2020

  9. [17]

    Datamodels: Predicting predictions from training data.arXiv preprint arXiv:2202.00622, 2022

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data.arXiv preprint arXiv:2202.00622, 2022

    2022 arXiv

  10. [18]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. InInternational conference on machine learning, pages 2242–2251. PMLR, 2019

    2019

  11. [19]

    Data banzhaf: A robust data valuation framework for machine learning

    Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388–6421. PMLR, 2023

    2023

  12. [20]

    Fast rates in pool-based batch active learning.Journal of Machine Learning Research, 25(262):1–42, 2024

    Claudio Gentile, Zhilei Wang, and Tong Zhang. Fast rates in pool-based batch active learning.Journal of Machine Learning Research, 25(262):1–42, 2024

    2024

  13. [21]

    Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes

    Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. InInternational Conference on Machine Learning, pages 39834–39863. PMLR, 2023

    2023

  14. [22]

    A new active labeling method for deep learning

    Dan Wang and Yi Shang. A new active labeling method for deep learning. In2014 International joint conference on neural networks (IJCNN), pages 112–119. IEEE, 2014

    2014

  15. [23]

    Leveraging importance weights in subset selection.arXiv preprint arXiv:2301.12052, 2023

    Gui Citovsky, Giulia DeSalvo, Sanjiv Kumar, Srikumar Ramalingam, Afshin Rostamizadeh, and Yunjuan Wang. Leveraging importance weights in subset selection.arXiv preprint arXiv:2301.12052, 2023

    2023 arXiv

  16. [24]

    Reducing labeling effort for structured prediction tasks

    Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. InAAAI, volume 5, pages 746–751, 2005

    2005

  17. [25]

    Deep batch active learning by diverse, uncertain gradient lower bounds.arXiv preprint arXiv:1906.03671, 2019

    Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds.arXiv preprint arXiv:1906.03671, 2019

    1906 arXiv

  18. [26]

    Towards robust model-based reinforcement learning against adversarial corruption.arXiv preprint arXiv:2402.08991, 2024

    Chenlu Ye, Jiafan He, Quanquan Gu, and Tong Zhang. Towards robust model-based reinforcement learning against adversarial corruption.arXiv preprint arXiv:2402.08991, 2024

    2024 arXiv

  19. [27]

    Bootstrap standard error estimates for linear regression.Journal of the American Statistical Association, 100(471):970–979, 2005

    Sílvia Gonçalves and Halbert White. Bootstrap standard error estimates for linear regression.Journal of the American Statistical Association, 100(471):970–979, 2005

    2005

  20. [28]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Ininternational conference on machine learning, pages 1050–1059. PMLR, 2016

    2016

  21. [29]

    Limitations of the empirical fisher approximation for natural gradient descent.Advances in neural information processing systems, 32, 2019

    Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent.Advances in neural information processing systems, 32, 2019

    2019

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  23. [31]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 11

    2016

  24. [32]

    Relatif: Identifying explanatory training samples via relative influence

    Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. InInternational Conference on Artificial Intelligence and Statistics, pages 1899–

  25. [33]

    Scalable influence and fact tracing for large language model pretraining

    Tyler A Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable influence and fact tracing for large language model pretraining. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [34]

    Lora: Low-rank adaptation of large language models.ICLR, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 2022

    2022

  27. [35]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  28. [36]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath...

    2024

  29. [37]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  30. [38]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  31. [39]

    Self-play with execution feedback: Improving instruction-following capabilities of large language models.arXiv preprint arXiv:2406.13542, 2024

    Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models.arXiv preprint arXiv:2406.13542, 2024

    2024 arXiv

  32. [40]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  33. [41]

    Black-box prompt learning for pre-trained language models.Trans

    Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang. Black-box prompt learning for pre-trained language models.Trans. Mach. Learn. Res., 2023

    2023

  34. [42]

    Black-box tuning for language- model-as-a-service

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language- model-as-a-service. InInternational Conference on Machine Learning, pages 20841–20855. PMLR, 2022

    2022

  35. [43]

    CombLM: Adapting black-box language models through small fine-tuned models

    Aitor Ormazabal, Mikel Artetxe, and Eneko Agirre. CombLM: Adapting black-box language models through small fine-tuned models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2961–2974. Association for Computational Linguistics, D...

    2023

  36. [44]

    Estimating training data influence by tracing gradient descent.Advances in Neural Information Processing Systems, 33:19920–19930, 2020

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent.Advances in Neural Information Processing Systems, 33:19920–19930, 2020

    2020

  37. [45]

    Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models.arXiv preprint arXiv:2305.14710, 2023

    Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models.arXiv preprint arXiv:2305.14710, 2023

    2023 arXiv

  38. [46]

    Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models.arXiv preprint arXiv:2408.12798, 2024

    Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models.arXiv preprint arXiv:2408.12798, 2024

    2024 arXiv

  39. [47]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. 12 Appendices A Theoretical Analysis of DAUNCE A.1 Motivation Given an estimator θ0 ...

    2009 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.