Small-to-Large Generalization: Data Influences Models Consistently Across Scale

Alaa Khaddaj; Aleksander Madry; Logan Engstrom

REVIEW 4 major objections 6 minor 13 references

Small-to-Large Generalization: Data Influences Models Consistently Across Scale

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small models predict how data changes move large language models

desk verdict Solid empirical study of proxy-to-large correlation in LMs, but the 'large-scale' claim rests on a single 760M reference and no seed variance. read the letter →

arxiv 2505.16260 v1 pith:3TNI45ZJ submitted 2025-05-22 cs.LG

Alaa Khaddaj , Logan Engstrom , Aleksander Madry This is my paper

classification cs.LG MSC 68T07

keywords small-to-largegeneralizationproxymodelsdatainfluencelanguagemodelscalingattributiondatasetselectionTRAKcross-scalecorrelation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Choice of training data affects small and large language models in nearly the same way, even when the small model costs 175–370× less to train. Across ten training-data distributions and six test sets, the loss ordering of a 57M-parameter proxy tracks that of a 760M reference on nearly all tasks, and a 40M proxy stays well-correlated except on SQuAD and TriviaQA. The relationship holds per sample on many tasks and survives even when the proxy performs at random-guessing level. This makes small models a legitimate substitute for large ones in data attribution and dataset selection, giving practitioners a large compute discount.

What carries the argument

The central object is the cross-distribution loss correlation between a small proxy and a larger reference model: the coefficient of determination ($R^2$) between proxy and reference average losses across training-data distributions, computed per test set, and the analogous per-sample $R^2$. That quantity, not proxy accuracy, is what the paper argues carries the transfer: a proxy is useful when its loss ordering across data choices matches the reference's. The same transferability is operationalized in the downstream applications through linear datamodels of the form $\hat{f}_z(D) = \sum_{i\in D} \tau(z)_i$, whose weights are estimated with the TRAK influence-function method at proxy scale and then used to attribute or select data for the large model. The compute gap between proxy and reference (175×–370×) is set by training each model on Chinchilla-optimal token counts, so the paper separates model size from training compute.

What would settle it

Train the same ten training-data distributions at a 7B- or 13B-parameter scale and check whether the loss ordering of the 40M proxy still matches; if the correlation drops substantially relative to the 760M reference, the scale-transfer claim would fail at the scales it is meant to inform.

Watch

Extended reading notes

Core claim

The central claim is that, for language models, the effect of changing the training-data distribution on a model's losses is largely scale-invariant: small and large models rank the training distributions in the same order. Concretely, after training models from 40M to 760M parameters on ten distributions (six natural sources and four selection-induced), the authors find that proxy losses correlate with reference losses at $R^2$ around 0.9 on four of six test tasks even at a 175× compute gap, with the small models trained on Chinchilla-optimal token counts. The correlation degrades only at the 370× gap (40M parameters) and only on SQuAD and TriviaQA. The same transferability powers two downstream applications: datamodels (TRAK) computed from a proxy that is 1000× cheaper predict the reference's loss on ImageNet with LDS within 10% of the reference's own datamodels, and DSDM dataset selection using a 125M proxy yields a 760M model that matches selection with the 760M reference on SQuAD and LAMBADA. The paper's summary statement is explicitly qualified: proxy reliability is task-dependent and breaks down at the very smallest scales.

Load-bearing premise

The 760M-parameter reference model is taken to be representative of the large-scale models practitioners actually care about, so the correlation pattern observed between 40M and 760M parameters is assumed to continue to much larger models.

Editorial extensions

If this is right

Data attribution for a 760M model can be done with a model that uses 1000× less compute, losing at most about 10% LDS on ImageNet relative to attributing with the reference model itself.
Dataset selection for a 760M model with DSDM is as effective using a 125M proxy as using the 760M model, with no accuracy drop until the proxy is 4× cheaper in training compute.
Small proxies whose accuracy on a task is at or below random guessing can still have losses that highly correlate with the large reference model on that task, so correlation is not tied to proxy competence on the task.
The reliability of a proxy depends on the test distribution: per-sample correlation is high on HellaSwag and weak on SQuAD, so proxy validity must be checked per task rather than assumed globally.
The correlation degrades as the compute gap grows, with the 40M proxy (370× gap) the point of failure on SQuAD and TriviaQA, implying a practical floor on proxy size for given tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If loss ordering transfers across scale as reported, data decisions such as deduplication, curriculum order, and mixture ratios could be screened with small models before spending large-model compute; the paper does not run this experiment but its transferability result makes it plausible.
The task-dependence of correlation suggests 'data influence' is a direction in model-behavior space rather than a single scalar; practitioners should validate proxy fidelity per downstream task, and methods that adapt proxy scale per task could be more efficient than a single global proxy.
A natural test of the mechanism is architecture transfer: the same experiment repeated with non-transformer or mixture-of-experts models would show whether the pattern is a property of data-driven learning rather than of transformer loss geometry.
Because each scale is trained with Chinchilla-optimal tokens, the observed correlation confounds parameter count with training compute; an experiment that fixes parameter count and varies training tokens, or vice versa, would separate the two contributions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper is a carefully run empirical study on whether small proxy LMs can stand in for larger ones when you change the training data. The headline result: loss on a fixed test set, measured across 10 training distributions, is highly correlated between a 40-125M proxy and a 760M reference, even when the proxy trains with 175-370x less compute. The correlation holds on most test tasks, though not strongly on SQuAD and TriviaQA for the smallest proxies. There is also a useful practical finding: for data attribution in vision, proxies up to ~1000x smaller give LDS within 10% of the full model, and for dataset selection in LMs, a 350M proxy (about 4x cheaper) still selects data that beats random selection.

What's genuinely new is the direct measurement of this correlation across compute gaps in LMs, plus the tradeoff curves for attribution and selection. Prior work (Vyas et al., Schaeffer et al.) suggested behavior may be consistent across scale or emergent abilities are a mirage, but the data-influence angle across training distributions is largely untested. The paper is also honest about its limitations: it acknowledges the LDS values are small, that better estimators could change the tradeoff, and that the compute estimates are lower bounds.

The soft spots are real but not fatal. First, the 'large-scale' reference is a single 760M model. The title and abstract generalize to large-scale models, but there is no evidence the correlation holds at 1B-100B. That is a genuine extrapolation. Second, the language correlations are computed over 10 distributions with one training run each; there are no error bars or seed variance. The vision experiments train 8 models and show confidence intervals, but the main LM result doesn't. Third, no code or data is released, which makes reproducibility harder. Fourth, the dataset selection 'threshold' is based on only three proxy sizes, so the 'does not drop until 4x' claim is coarse. None of these is a load-bearing contradiction, but they temper the strength of the claim.

Who should read it: anyone doing data attribution, dataset selection, or using proxies to study data effects in LLMs. It's a useful reference for what we can and cannot conclude from small models today. The paper deserves a serious referee: the experiments are within compute constraints, the measurements are direct, and the limitations are addressable. I'd ask the authors to add seed variance, qualify the title-level claim, and release code/data.

Referee Report

4 major / 6 minor

Summary. This paper studies whether changes in the training-data distribution affect small and large language models in the same way. The authors train MPT-family LMs (40M-760M parameters) on 10 training distributions, measure their losses on 6 test sets, and report high R^2 between proxy and 760M-reference losses across distributions (Figures 1-3). They then examine two proxy-model applications: TRAK-based data attribution for ImageNet/CIFAR ResNets (Figures 6-7) and DSDM-based dataset selection for an MPT-760M language model (Figure 8). The main claim is that small proxies are generally faithful substitutes for larger models, with some task- and scale-dependent breakdowns, and the paper recommends small proxies as a practical tool.

Significance. If the result is robust, it is practically significant: it would justify the common but previously unquantified practice of using small proxy models for data attribution and dataset selection at larger scale. The paper's strengths are that it is a direct measurement rather than a derivation, it includes per-sample and counterfactual analyses, and the downstream applications benchmark proxy-based scores against independently trained large models. I do not see circularity in the downstream evaluations, since the final accuracy of the large model is measured on a fresh training run. However, the load-bearing limitation is that the "large" reference is a single 760M model, and all cross-scale correlations are computed over only 10 training distributions with no seed variance or confidence intervals.

major comments (4)

[Section 2.1 and title/abstract] The title-level claim that data influences models "across scale" is supported only by correlations between 40M-125M proxies and a single 760M reference model. The reference scale is explicitly chosen for compute-budget reasons (Section 2.1), so the paper provides no evidence that the correlation extends to the 1B-100B regime that its practical recommendation targets. This is load-bearing because the contribution is the validation of small-to-large transfer. I do not insist on full-scale training, but the paper should either qualify the title-level claim or add a validation at one larger scale, for example by evaluating a publicly available 1B+ checkpoint on the same training-distribution manipulations.
[Section 2.2, Figures 1 and 2] All cross-scale correlations are computed over only 10 training distributions, with no error bars, no multiple-seed variance, and no leave-one-distribution-out sensitivity. Moreover, several of the distributions are not independent samples: DSDM, Bot-DsDM, DSIR, and Classifier (Section B.1.1) are algorithm-induced subsets of C4, and the DSDM/Bot-DsDM outcomes are reused from datamodel-based selection. This non-independence can inflate the apparent cross-scale agreement. Please report bootstrap confidence intervals and repeat the analysis after excluding the selection-induced distributions, so the reader can see how much of the central correlation rests on those points.
[Section 2.3, Figure 3] The claim that proxy models "highly correlate with large-scale models even at random-guessing accuracy" relies on HellaSwag and COPA, but COPA is not listed among the six test distributions in Section 2.1 or in the downstream-dataset list of Section B.1.1. Additionally, the x-axis of Figure 3 is described only as "normalized to show improvement over outputting a random guess," without giving the precise normalization. Without a consistent task list and a clear definition of the normalized accuracy, this specific claim cannot be checked.
[Section 3.2, Figure 8 and Conclusion] The conclusion that "orders-of-magnitude smaller proxy models can be as effective" is not supported by the language-model dataset-selection experiment, where the proxy sizes range only from 125M to 760M parameters (at most a 6x parameter gap and roughly a 4x compute gap). The vision attribution experiment does probe large compute gaps, but the language selection evidence does not. Please either align the concluding claim with the measured proxy-size range or add a smaller-proxy condition to the dataset-selection experiment.

minor comments (6)

[Section 5] The conclusion contains a typo: "the the choice of training data distribution." Section 4 also contains "approachs" for "approaches."
[Figure 1 caption] The caption says the small model trains with "175× less compare" instead of "175× less compute," and the same figure is described in the main text as a 57M proxy while Section B.1.1 specifies MPT-80M.
[Section 2.1 and Appendix B.1.1] The main text lists TriviaQA among the six test datasets, but the list in Appendix B.1.1 omits TriviaQA; the task lists should be harmonized.
[Section 2.2 and Appendix B.1.3] The main text refers to a 40M proxy with a 370x compute gap, while Appendix B.1.3 uses an MPT-37M model with 840M tokens; the parameter counts and token counts should be made consistent between the main text and the appendix.
[Figure 4 caption] The caption states the proxy model is 57M parameters and trains with "around 175× the compute of the 760M reference model," which reads as the wrong direction; it should say "175× less compute."
[Appendix A.3.2] There is a typo in the text "the quality of the attribution scores" written as "thq quality."

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central correlation claim is a direct measurement, and downstream proxy-model evaluations are benchmarked against external reference-model outcomes.

full rationale

The paper's central claim—that model losses across training data distributions correlate between 40M–760M proxies and a 760M reference—is an empirical measurement reported in Section 2.2 (Figures 1 and 2), not a derivation from fitted parameters; the correlation could have been low and is therefore not true by construction. The downstream data-attribution claim (Section 3.1) computes TRAK attribution scores for proxies of various sizes and evaluates them with the Linear Datamodeling Score against the actual losses of the largest reference model (Eq. 2), an external benchmark that does not assume the proxy scores are correct. Likewise, dataset selection (Section 3.2) trains proxy datamodels, selects data with DSDM, retrains the 760M reference on that selected data, and compares its accuracy to a randomly selected baseline (Figure 8); no fitted proxy quantity is renamed as a prediction. Self-citations to TRAK [PGI+23], DSDM [EFM24], and datamodels [IPE+22] appear as tooling in these pipelines, but their outputs are validated against actual model behavior rather than being used as evidence for the cross-scale correlation itself. The acknowledged limitations—the single 760M reference scale (Section 2.1) and low absolute LDS values (Section 3.1.2)—concern external validity and predictive strength, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper's central correlation finding is an empirical measurement and rests on domain assumptions about representativeness of the model class, the loss metric, the training distributions, and the validity of TRAK/DSDM in the downstream applications. No invented entities or fitted free parameters are introduced.

assumptions (5)

domain assumption The 760M MPT model is representative of the large-model class of interest; observed correlations extrapolate to larger scales.
Invoked in Section 2.1 where 760M is chosen as reference due to academic compute budget; generalized in Section 5 to 'large-scale models'.
domain assumption Loss on downstream test sets is a sufficient measure of model behavior for data influence.
All correlations in Section 2.2 are computed on losses; results are framed as 'model behavior'.
domain assumption The ten training distributions are representative of the space of data distribution changes.
Section 2.1 lists 6 natural and 4 selection-induced distributions; the general claim relies on this sample.
domain assumption TRAK's linearized gradient-based datamodel approximation is valid for the studied models.
Appendix A.2 and Section 3.1 use TRAK for all attribution scores; the downstream results inherit this assumption.
domain assumption Chinchilla-optimal token-to-parameter ratios are the correct operating point for comparing models across scale.
Section 2.1 and B.1.1 set training tokens by Chinchilla ratios, which determines the compute gap.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Small-to-Large Generalization: Data Influences Models Consistently Across Scale." pith.science (2026). https://pith.science/paper/3TNI45ZJ

@misc{pith2026250516260,
  author       = {Pith},
  title        = {Pith review of: Small-to-Large Generalization: Data Influences Models Consistently Across Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TNI45ZJ}},
  note         = {Machine review of arXiv:2505.16260}
}

read the original abstract

Choice of training data distribution greatly influences model behavior. Yet, in large-scale settings, precisely characterizing how changes in training data affects predictions is often difficult due to model training costs. Current practice is to instead extrapolate from scaled down, inexpensive-to-train proxy models. However, changes in data do not influence smaller and larger models identically. Therefore, understanding how choice of data affects large-scale models raises the question: how does training data distribution influence model behavior across compute scale? We find that small- and large-scale language model predictions (generally) do highly correlate across choice of training data. Equipped with these findings, we characterize how proxy scale affects effectiveness in two downstream proxy model applications: data attribution and dataset selection.

Figures

Figures reproduced from arXiv: 2505.16260 by the authors.

**Figure 1.** Proxy-model test loss highly correlates with large-model test loss across choice of training data distribution, even across a large gap in scale. Above, we plot the losses of a small-scale proxy (57M parameters) compared to that of the reference model (760M parameters). Here, the small scale model trains with 175× less compare than the reference model. Each column represents model loss on a different test distribu… view at source ↗

**Figure 2.** Correlation between large- and small-scale model predictions is consistently high, even across large gaps (orders of magnitude) in training compute scale. We plot small- to large-scale correlation against small-scale proxy model compute. There is also large variation across choice of test set: correlation is consistently high on four of six tasks, while losses on SQuAD and TriviaQA correlate less. 2.3 Intriguing pro… view at source ↗

**Figure 3.** Proxy models can be highly predictive of large-scale model predictions even when predicting as well as randomly on a given test set. We plot small- to large-scale loss correlation against small-scale proxy model accuracy on the given task, normalized to show improvement over outputting a random guess (in absolute accuracy). On a number of test sets, proxy models perform no better than random guessing, but still high… view at source ↗

Figures from the paper (20 more)

**Figure 4.** Figure 4: Proxy model predictions can highly correlate with those of the reference model on individual test samples. We visualize loss on individual samples for each scale model across varying training datasets. The proxy model here is 57M parameters, training with around 175× t…

**Figure 5.** Figure 5: The correlation between large- and small-scale model losses on individual samples is highly dependent on the test distribution. We show a histogram of the correlation between large model and proxy model predictions on individual test samples for the test distribution i…

**Figure 6.** Figure 6: In both plots, the x-axis represents the amount of compute required to get the attribution scores of a given model, compared to the large model, and the y-axis represents how well the attribution scores of a given model size can predict the output of the largest model …

**Figure 7.** Figure 7: Most helpful (left) and most detrimental (right) examples for the outputs of models of different sizes are similar. The most helpful and most detrimental examples for the given target example (center) are shown according to each model size (row). We observe a large ove…

**Figure 8.** Figure 8: In both plots, the x-axis represents the amount of compute required to train a given proxy model (relative to training the large model) and the y-axis represents the accuracy on (a) LAMBADA [PKL+16] and (b) SQuAD [RZL+16] of a large model trained on a subset of the MPT…

**Figure 12.** Figure 12: Small models are reliable proxies of large models. In all plots, the x-axis represents the loss achieved by a small MPT model of 37 million parameters trained on 0.84B tokens and the y-axis represents the loss achieved by a larger MPT model of 760 million parameters t…

**Figure 13.** Figure 13: Small models are reliable proxies of large models. We plot the coefficient of determination (R 2 ) between the losses of the small and large models for all examples. B.1.4 How Correlation Changes with Compute We observe a large correlation between the losses achieved …

**Figure 14.** Figure 14: Small models are reliable proxies of large models. In all plots, the x-axis represents the margin of the small ResNet-18- 1 4 model and the y-axis represents the margin of the larger ResNet-18-2 model. Each plot corresponds to a different test example, and within each…

**Figure 17.** Figure 17: Performance of our models on CIFAR-10, CIFAR-100 [Kri09] and ImageNet [KSH12] for different widths. C.1.4 TRAK details In this setup, we train 8 independent models RN-k models for each multiplicative factor k. We then pass the model checkpoints {20 . . . 30} for CIFAR…

**Figure 18.** Figure 18: Random SQuAD samples [RZL+16]. Context is normal text, and the continuation label is hightlighted. 1. Context: In 1854 at Ballarat there was an armed rebellion against the government of Victoria by miners protesting against mining taxes (the "Eureka Stockade"). This …

**Figure 19.** Figure 19: Random LAMBADA samples [PKL+16]. Context is normal text, and the continuation label is hightlighted. C.2.2 Models In this setup, we consider three MPT models presented in [Mos23b] 18. Our three models are of sizes 125M, 350M and 760M parameters respectively. We presen…

**Figure 21.** Figure 21: Performance of our three compute-optimal MPT models [Mos23b; HBM+22]. C.2.4 TRAK details In this setup, the computational requirement is much higher. For that reason, we only train three different models of each size on different random subsets of the training dataset…

**Figure 22.** Figure 22: Most helpful and detrimental examples for the outputs of models of different sizes are similar. We observe a large overlap between the examples that are most helpful (and most detrimental) for the models predictions on the target example. Target Example Whippet Whippe…

**Figure 23.** Figure 23: Most helpful and detrimental examples for the outputs of models of different sizes are similar. We observe a large overlap between the examples that are most helpful (and most detrimental) for the models predictions on the target example. 35 [PITH_FULL_IMAGE:figures/…

**Figure 24.** Figure 24: Random samples of (a) the most helpful and (b) most detrimental examples on SQuAD [RZL+16] according to each of our MPT models. The samples are truncated to 512 characters. "\n" denotes a newline. More examples in Appendix D.1.2. 36 [PITH_FULL_IMAGE:figures/full_fig_…

**Figure 25.** Figure 25: Random samples of (a) the most helpful and (b) most detrimental examples on SQuAD [RZL+16] according to each model. The figure shows a 512-character slice from the training example. "\n" denotes a newline. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]

**Figure 26.** Figure 26: Random samples of (a) the most helpful and (b) most detrimental examples on LAMBADA [PKL+16] according to each model. The figure shows a 512-character slice from the training example. "\n" denotes a newline. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_26.png]

**Figure 27.** Figure 27: The x-axis represents the amount of compute required to get the attribution scores of a given model, compared to the large model, and the y-axis represents how well the attribution scores of a given model size can predict the output of the largest model on CIFAR-100 […

**Figure 28.** Figure 28: Each heatmap represents the Spearman rank correlation [Spe04] between the attribution scores of every pair of models. The rank correlation is computed using (a) the CIFAR-10 attribution scores and (b) the CIFAR-100 scores [Kri09]. Language setup. In the language setup…

**Figure 29.** Figure 29: The heatmap represents the Spearman rank correlation [Spe04] between the attribution scores of every pair of models. The rank correlation is computed using LAMBADA [PKL+16] (left) and SQuAD [RZL+16] (right) attribution scores. 40 [PITH_FULL_IMAGE:figures/full_fig_p04…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[1]

The cost to compute the gradients for the training set is 6 pD = 6p · T · n

work page
[2]

When dealing with a target dataset with ntest examples, this cost is 6p · T · ntest

The cost to compute the gradients for the target example is 6 p. When dealing with a target dataset with ntest examples, this cost is 6p · T · ntest

work page
[3]

The cost to randomly project the gradients of the training examples is n · k · (2p − 1)

work page
[4]

The cost to randomly project the gradients of the test examples is ntest · k · (2p − 1)

work page
[5]

The product Φ⊤Φ requires k2 · (2n − 1) FLOPS

work page
[6]

The inverse operation (Φ⊤Φ)−1 costs around k3 FLOPS

work page
[7]

The product (Φ⊤Φ)−1Φ⊤ costs n · k · (2k − 1) FLOPS

work page
[8]

The final product ϕ(z)⊤(Φ⊤Φ)−1Φ⊤ costs n · (2k − 1) FLOPS for a single target example z, and ntest · n · (2k − 1) for attributing over ntest target examples. The total attribution cost is then the sum of the above terms: Cattrib = (6pT + 4k2 + 2k · p − 2k + 2k · ntest − ntest ) · n (28) + 6p · T · ntest + k · (2p − 1) · ntest − k2 + k3 (29) ≈ (6p · T + 4k...

work page 2019

Show all 13 references

[10]

Sierra Sky Park was the first aviation community to be built[citation needed] and there are now numerous such communities across the United States and around the world

Context: Formed in 1946, Sierra Sky Park Airport is a residential airport community born of a unique agreement in transportation law to allow personal aircraft and automobiles to share certain roads. Sierra Sky Park was the first aviation community to be built[citation needed]...

1946
[11]

Context: The Newcastle Beer Festival, organized by CAMRA, takes place in April. In May, Newcastle and Gateshead host the Evolution Festival, a music festival held on the Newcastle and Gateshead Quaysides over the Spring bank holiday, with performances by acts from the world of...
[12]

Eureka Stockade

Context: In 1854 at Ballarat there was an armed rebellion against the government of Victoria by miners protesting against mining taxes (the "Eureka Stockade"). This was crushed by British troops, but the discontents prompted colonial authorities to reform the administration (p...
[13]

multiplier effect

Context: Within southern California are two major cities, Los Angeles and San Diego, as well as three of the country’s largest metropolitan areas. With a population of 3,792,621, Los Angeles is the most populous city in California and the second most populous in the United Sta...

2021
[2023]

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

URL: https://www.mosaicml.com/blog/mpt-7b. [Mos23b] MosaicML. LLM Foundry. 2023. URL: https://www.github.com/mosaicml/llm-foundry. [MUP+23] Max Marion, Ahmet Ustun, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. “When Less is More: Investigating Data Pruning for P...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

The cost to compute the gradients for the training set is 6 pD = 6p · T · n

work page

[2] [2]

When dealing with a target dataset with ntest examples, this cost is 6p · T · ntest

The cost to compute the gradients for the target example is 6 p. When dealing with a target dataset with ntest examples, this cost is 6p · T · ntest

work page

[3] [3]

The cost to randomly project the gradients of the training examples is n · k · (2p − 1)

work page

[4] [4]

The cost to randomly project the gradients of the test examples is ntest · k · (2p − 1)

work page

[5] [5]

The product Φ⊤Φ requires k2 · (2n − 1) FLOPS

work page

[6] [6]

The inverse operation (Φ⊤Φ)−1 costs around k3 FLOPS

work page

[7] [7]

The product (Φ⊤Φ)−1Φ⊤ costs n · k · (2k − 1) FLOPS

work page

[8] [8]

The final product ϕ(z)⊤(Φ⊤Φ)−1Φ⊤ costs n · (2k − 1) FLOPS for a single target example z, and ntest · n · (2k − 1) for attributing over ntest target examples. The total attribution cost is then the sum of the above terms: Cattrib = (6pT + 4k2 + 2k · p − 2k + 2k · ntest − ntest ) · n (28) + 6p · T · ntest + k · (2p − 1) · ntest − k2 + k3 (29) ≈ (6p · T + 4k...

work page 2019