Pith. sign in

REVIEW 5 major objections 7 minor 51 references

Approximating Language Model Training Data from Weights

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A gradient-based method recovers a usable substitute for a language model's hidden fine-tuning data using only its base and final weights.

desk verdict A genuinely new data-recovery formulation with solid empirical gains, but the core selection signal rests on an unproven linear-trajectory assumption that the synthetic checkpoints do not actually test. read the letter →

arxiv 2506.15553 v1 pith:DU4W7MVV submitted 2025-06-18 cs.CL

John X. Morris , Junjie Oscar Yin , Woojeong Kim , Vitaly Shmatikov , Alexander M. Rush This is my paper
classification cs.CL
keywords trainingdatarecoverymodelweightsselectiongradientmatchingopen-weightsmodelsfine-tuningapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to prove that the hidden fine-tuning data of an open-weights language model can be approximated from the weights alone, provided an attacker knows the optimizer and has a large public unlabeled corpus. Its method, SELECT, picks a small subset of documents whose summed gradients align with the direction between the base and fine-tuned checkpoints, then shows that a fresh model trained on those documents approaches the original model's performance. On AG News, selected data reaches 80% accuracy versus 65% for random selection and 88% for an expert model trained on the true data. On MSMARCO-style supervised fine-tuning, selected data lowers perplexity from 3.3 to 2.3, approaching the expert model's 2.0. This matters because many popular releases are open-weights but closed-data, so the weights themselves may leak enough signal to reconstruct a usable substitute training set.

What carries the argument

The load-bearing object is the model-difference vector $\theta_f - \theta_0$ in weight space, together with per-example last-layer gradients of the base model. The method selects documents whose gradients, when summed, maximize projection onto this difference; the objective is submodular, so greedy selection inherits a near-optimal guarantee, and Johnson-Lindenstrauss random projections store the gradients in a few thousand dimensions. Linear interpolation between $\theta_0$ and $\theta_f$ creates synthetic checkpoints that stand in for the missing intermediate checkpoints used in trajectory-matching distillation.

What would settle it

Run a fully logged fine-tuning job with Adam over several epochs, record the true per-step update directions, and compare per-example gradient alignment at $\theta_0$ against the actual trajectory; if the true training documents show no higher alignment than random documents from the seed set, the central premise is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the vector from the base checkpoint $\theta_0$ to the fine-tuned checkpoint $\theta_f$ acts as a usable surrogate for the optimization trajectory, even though only the endpoints are known. SELECT scores every candidate document by the alignment of its last-layer gradient with this model difference, then greedily builds a batch whose summed gradient points along the difference, treating the objective as submodular and using linear interpolation to create synthetic intermediate checkpoints. The selected documents need not overlap the true training set: they transfer the task well enough that a newly trained model comes close to the released model on both classification and supervised fine-tuning.

Load-bearing premise

The load-bearing premise is that the direction from the base model to the fine-tuned model matches the direction of the first SGD-like update; if real Adam training over many epochs bends away from that direction, the selection signal fades.

Editorial extensions

If this is right

  • A released fine-tuned model, even with its training data withheld, carries enough information to construct a substitute dataset that retrains a different model instance to near-original accuracy.
  • Gradient-based selection beats random, top-k, balanced top-k, and perplexity-based baselines across AG News, DBPedia, IMDB, and supervised fine-tuning on LLAMA-3.2 models.
  • The selection signal strengthens as the public seed corpus grows and as the true fine-tuning data becomes a larger fraction of the candidate pool, indicating that the method can locate leaked data.
  • Last-layer gradients projected to a few thousand dimensions are sufficient for the signal, keeping the method feasible at the scale of millions of candidate documents.
  • The identity of the optimizer matters: SGD and Adam provide stronger selection signals than AdamW, whose weight decay partially erases the weight-level signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the gradient-alignment score could serve as a data-provenance audit, flagging whether a candidate public dataset is consistent with a released fine-tune before any retraining is run.
  • We infer that the method's dependence on the seed distribution is a testable boundary: the paper's seed-transfer table suggests cross-domain seeds still beat random, and a systematic study across far-apart domains would show where SELECT degrades.
  • We infer that if the alignment assumption holds for larger models and longer fine-tuning runs, SELECT could become a practical tool for auditing open-weight releases; if it fails, the method would collapse to random selection, which is the observable failure mode.
  • We infer that the same endpoint-difference signal may extend to preference tuning or instruction tuning, since those also produce a fine-tuned checkpoint from a base checkpoint, but the paper does not test that setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper formalizes the problem of approximating the fine-tuning data of a language model from two checkpoints (base and fine-tuned) plus knowledge of the optimizer, and proposes SELECT, which scores documents in a public unlabeled corpus by the projection of their per-example gradients onto the parameter difference θf−θ0, pseudolabels the seed corpus with the final model, and greedily selects a subset. The authors evaluate on GPT-2-medium classification (AG News, DBPedia, and, depending on the table, IMDB or 20-Newsgroup) and Llama-3.2 SFT, reporting that SELECT improves AG News accuracy from 65.6% (random) to 80.0% versus 88.0% for the expert model, and reduces MSMARCO SFT perplexity from 3.3 to 2.3. They also introduce lexical and optimal-transport metrics, a scaling analysis of selected-set size, a leakage analysis, and ablations of projection dimension, seed distribution, and optimizer choice.

Significance. The problem is timely and the empirical finding is potentially important: if selecting data by gradient alignment with the model difference recovers useful fine-tuning data at this level, it bears on privacy and transparency of open-weight releases. The paper's strengths include an open-source implementation, evaluation against several non-trivial baselines (random, top-k variants, perplexity-based selection), and a consistent pattern of gains across classification and SFT settings. The claimed effect sizes are large and, if reproducible, would justify publication. However, the current manuscript overstates the theoretical basis of the method: the trajectory-matching rationale is not supported by the synthetic-checkpoint construction, and the selection objective/algorithm discrepancy undermines the explanation for why SELECT outperforms top-k. These issues require revision before the empirical claim can be accepted at face value.

major comments (5)
  1. [4.1] The synthetic checkpoints defined by θ̂_j = (j/P)θ0 + (1−j/P)θf satisfy θf − θ̂_j = (j/P)(θf − θ0). Substituting into the selection objective shows that every term is a positive scalar multiple of the same direction θf − θ0; the multi-checkpoint sum therefore contains no directional information about the optimization path and cannot address the stated concern that gradients 'may point in a different direction later on' during training. The method may still be useful as a weighted evaluation of gradients at interpolated points, but the paper should not describe it as trajectory matching, and the underlying alignment assumption is not tested by the experiments.
  2. [4] The batch objective F(B) = Σ_{x∈B} ∇ℓ(x;θ0)·(θf−θ0) is modular, not submodular: the marginal gain of adding x is exactly the constant score ∇ℓ(x;θ0)·(θf−θ0), independent of the current batch. Therefore the submodularity-based justification and the claim that top-k underperforms because it ignores batch-level interactions are inconsistent with the stated objective. Algorithm 1 appears to implement a different selection rule (the role of the update Ĝ ← Ĝ + broadcast(Ĝ_i*) is unexplained and, as written, adds a constant to all scores), so the mechanism by which SELECT avoids redundancy is not specified. The authors should state the precise objective, prove or replace the submodularity claim, and ensure the algorithm matches the objective.
  3. [5] The evaluation protocol violates the threat model: Section 3.1 says no validation data is available, but Section 5 says the retrained model is 'take[n] ... according to a validation set' after 100 epochs. Under the stated threat model an adversary could not choose that checkpoint without labels. The reported numbers may therefore overstate what the method achieves under the paper's own assumptions; either the threat model must include a validation set (and the method's selection should not use it), or the authors should report a no-validation checkpoint rule.
  4. [5 / Table 1] The main classification experiments are described inconsistently: Section 5 says the classification datasets are AG News, DBPedia, and 20-Newsgroup, while Table 1 reports columns for IMDB, and Table 3 also lists Newsgroup and Rotten Tomatoes. Because the central empirical claim depends on the exact benchmarks, this ambiguity must be resolved before the results can be assessed.
  5. [4 / 5] The central hypothesis that the initial gradient direction is aligned with θf−θ0 is only validated indirectly through downstream task performance. No experiment measures, for the actual Adam-trained models, the angle between intermediate optimization steps and θf−θ0, nor does any control compare SELECT against a random direction or against gradients from a different checkpoint. Such a diagnostic would establish whether the method depends on the specific direction θf−θ0 or on a weaker property of the interpolated gradients.
minor comments (7)
  1. [Abstract] The sentence 'our method is able to locate a small subset of public Web documents can be used to train a model' is missing a relative pronoun and should read 'documents that can be used'.
  2. [6] The sentence 'In previous experiments we only considered selecting 10K datapoints' should read 'selecting 1K datapoints from a seed set of 10K'.
  3. [6] The sentence 'the seed set (MSMARCO) is already highly similar to the true finetuning set (Wikipedia documents from Natural Questions)' appears to reverse the roles: the final SFT model is trained on MSMARCO and the seed set is Natural Questions; please correct.
  4. [Algorithm 1] The symbol θ̂_t is not defined in the algorithm or text, the input η is unused, and 'broadcast' is not explained.
  5. [Table 1] The Expert row gives only six numbers for what should be nine columns (Vocab, OTD, Acc, Loss for three datasets); please reformat so each column is identifiable.
  6. [4.3] The Johnson-Lindenstrauss argument requires that the target direction θf−θ0 also be projected with the same random projection; this is not stated.
  7. [4.1] The formula interpolates from θf (j=0) to θ0 (j=P), which is the reverse of the text's 'interpolating between the initial and final model'; clarify the indexing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is validated by held-out retraining on standard benchmarks; the interpolated-checkpoint degeneracy is a robustness concern, not a circular derivation.

full rationale

The paper's core claim is that SELECT selects public unlabeled documents which, when labeled by the final model and used to retrain a fresh model, approximate the original fine-tuned model's accuracy/perplexity. This claim is evaluated self-containedly: the true fine-tuning set is held out, the seed corpus is disjoint Wikipedia/NQ text, no test labels are used in selection, and performance is measured by retraining a fresh model on the selected data with standard test sets. The only self-referential ingredient is autolabeling the seed corpus with the victim final model's predictions, which is part of the threat model and does not force the result (random autolabeled selection is markedly worse). The synthetic checkpoints in Sec. 4.1 are indeed degenerate directionally: substituting θ̂j=(j/P)θ0+(1−j/P)θf gives θf−θ̂j=(j/P)(θf−θ0), so every target is a positive scalar multiple of the single model-difference direction and the multi-checkpoint objective cannot capture trajectory curvature. However, this is a heuristic limitation and robustness concern about whether the alignment assumption holds on curved Adam trajectories, not a circular reduction of the output to the input; the downstream gain is an independent empirical result. Self-citations (Morris et al. 2023/2025, Yin & Rush 2025) are background or baseline references and are not load-bearing. The paper even notes an omitted optimizer ablation to be added later, which is a completeness issue, not circularity. Therefore no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The ledger shows the method introduces no new physical or mathematical entities, but relies on several domain assumptions about gradient geometry that are empirically validated only indirectly. The main free parameters are the projection dimension, the synthetic checkpoint count, the selection budget, and the seed distribution.

free parameters (4)
  • Projection dimension d = 4096
    The low-dimensional gradient storage dimension is chosen as 4096 for all experiments; Section 7 and Figure 5 show performance varies with this choice, and 4096 is justified only by noting FLOPs grow quadratically with dimension.
  • Number of synthetic checkpoints P = Not explicitly stated in the text (referenced as synthetic checkpoint interpolation in Section 4.1)
    The number of interpolated checkpoints P is a free parameter of the method; the paper does not report its exact value or an ablation over it.
  • Number of selected datapoints M = 1000 (with scaling experiments from 100 to 2500)
    The main results select 1K datapoints from 10K seed; M is a user-chosen budget that affects all results and is not derived from first principles.
  • Seed set distribution = Natural Questions Wikipedia passages (with cross-seed experiments)
    The choice of seed corpus is a free design choice; the paper shows results vary substantially across seed distributions (Table 4), so the main results depend on this choice.
assumptions (3)
  • domain assumption The initial gradient from the base model points in the direction of the final model's parameter change (θf − θ0).
    This is the core hypothesis of Section 4 ('We hypothesize that the initial update in stochastic gradient descent... should be aligned with the direction leading toward the final model'). It is stated as a hypothesis, not proved, and is load-bearing for the selection objective.
  • standard math The batch selection objective is submodular, so greedy selection is near-optimal.
    Invoked in Section 4 ('the batch search objective is submodular because it exhibits the diminishing returns property') without proof or verification for the actual gradient-sum objective.
  • standard math Johnson-Lindenstrauss projections preserve inner products sufficiently well for the gradient selection signal.
    Used in Section 4.3 to justify reducing gradient dimension; the lemma is standard, but its applicability at d=4096 for these specific gradients is empirical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Approximating Language Model Training Data from Weights." pith.science (2026). https://pith.science/paper/DU4W7MVV

@misc{pith2026250615553,
  author       = {Pith},
  title        = {Pith review of: Approximating Language Model Training Data from Weights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DU4W7MVV}},
  note         = {Machine review of arXiv:2506.15553}
}
read the original abstract

Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics. We develop a gradient-based approach that selects the highest-matching data from a large public text corpus and show its effectiveness at recovering useful data given only weights of the original and finetuned models. Even when none of the true training data is known, our method is able to locate a small subset of public Web documents can be used to train a model to close to the original model performance given models trained for both classification and supervised-finetuning. On the AG News classification task, our method improves performance from 65% (using randomly selected data) to 80%, approaching the expert benchmark of 88%. When applied to a model trained with SFT on MSMARCO web documents, our method reduces perplexity from 3.3 to 2.3, compared to an expert LLAMA model's perplexity of 2.0.

Figures

Figures reproduced from arXiv: 2506.15553 by the authors.

Figure 1
Figure 1. SELECT Method Overview. We approximate training data from initial and final language model checkpoints. Given an unsupervised seed dataset, we first use the final model θ f to assign a label to each instance. To compute gradients efficiently, we leverage random projections under the Johnson-Lindenstrauss lemma, which guarantees that gradient projection preserves inner-product information. Finally, we perform greedy … view at source ↗
Figure 3
Figure 3. Accuracy (y-axis, left) and OT dis￾tance (y-axis, right) vs the percent of seed data that is leaked from the true dataset (x￾axis). more difficult to compress into fewer samples (since we are selecting 1000 from 10, 000 datapoints). In our SFT experiments ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Dataset Evaluation Metrics Ablation. 3 In addition to storage size, the total number of FLOPs grows quadratically with the dimensionality of the projection. Thus, we opt to use a size of 4, 096 for all experiments. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Performance of SELECT (blue) vs. random selection (red, dashed) on selecting [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 15 canonical work pages

  1. [1]

    Dbpedia: A nucleus for a web of open data, 2007

    S \"o ren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data, 2007. URL https://link.springer.com/chapter/10.1007/978-3-540-76298-0_52

    doi:10.1007/978-3-540-76298-0_52 2007

  2. [2]

    Ms marco: A human generated machine reading comprehension dataset, 2018

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL https://arxiv.org/abs/1611.09268

    arXiv 2018

  3. [3]

    Reconstructing Training Data with Informed Adversaries

    Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries, 2022. URL https://arxiv.org/abs/2201.04845

    work page Pith review arXiv 2022

  4. [4]

    Coresets via bilevel optimization for continual learning and streaming

    Zal \'a n Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33: 0 14879--14890, 2020

    work page 2020

  5. [5]

    Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses

    Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin, and Michal Irani. Deconstructing data reconstruction: Multiclass, weight decay and general losses, 2023. URL https://arxiv.org/abs/2307.01827

    work page Pith review arXiv 2023

  6. [6]

    Extracting training data from large language models, 2021

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021. URL https://arxiv.org/abs/2012.07805

    arXiv 2021

  7. [7]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022

    2022

  8. [8]

    Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr

    Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of a production language model, 2024. URL https://arxiv.org/abs/2403.06634

    arXiv 2024

Show all 51 references
  1. [9]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories, 2022. URL https://arxiv.org/abs/2203.11932

    2022 arXiv

  2. [10]

    Super-samples from kernel herding

    Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint arXiv:1203.3472, 2012

    2012 arXiv

  3. [11]

    Scaling up dataset distillation to imagenet-1k with constant memory, 2023

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory, 2023. URL https://arxiv.org/abs/2211.10586

    2023 arXiv

  4. [12]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013

    2013

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  6. [14]

    Dsdm: Model-aware dataset selection with datamodels, 2024

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024. URL https://arxiv.org/abs/2401.12926

    2024 arXiv

  7. [15]

    Automatic document selection for efficient encoder pretraining

    Yukun Feng, Patrick Xia, Benjamin Van Durme, and Jo \ a o Sedoc. Automatic document selection for efficient encoder pretraining. arXiv preprint arXiv:2210.10951, 2022

    2022 arXiv

  8. [16]

    Logits of api-protected llms leak proprietary information, 2024

    Matthew Finlayson, Xiang Ren, and Swabha Swayamdipta. Logits of api-protected llms leak proprietary information, 2024. URL https://arxiv.org/abs/2403.09539

    2024 arXiv

  9. [17]

    Towards lossless dataset distillation via difficulty-aligned trajectory matching, 2024

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching, 2024. URL https://arxiv.org/abs/2310.05773

    2024 arXiv

  10. [18]

    Reconstructing training data from trained neural networks, 2022

    Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks, 2022. URL https://arxiv.org/abs/2206.07758

    2022 arXiv

  11. [19]

    Can we infer confidential properties of training data from llms?, 2025

    Penguin Huang, Chhavi Yadav, Ruihan Wu, and Kamalika Chaudhuri. Can we infer confidential properties of training data from llms?, 2025. URL https://arxiv.org/abs/2506.10364

    2025

  12. [20]

    D-optimality for regression designs: a review

    RC St John and Norman R Draper. D-optimality for regression designs: a review. Technometrics, 17 0 (1): 0 15--23, 1975

    1975

  13. [21]

    Johnson and Joram Lindenstrauss

    William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a Hilbert space. In Contemporary Mathematics, volume 26, pp.\ 189--206. American Mathematical Society, 1984. doi:10.1090/conm/026/737400

    1984 doi

  14. [22]

    Grad-match: Gradient matching based data subset selection for efficient deep model training

    Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp.\ 5464--5474. PMLR, 2021 a

    2021

  15. [23]

    Glister: Generalization based data subset selection for efficient and robust learning

    Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 8110--8118, 2021 b

    2021

  16. [24]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  17. [25]

    From word embeddings to document distances

    Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp.\ 957--966. PMLR, 2015

    2015

  18. [26]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  19. [27]

    Making large language models better data creators

    Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W White, and Sujay Kumar Jauhar. Making large language models better data creators. arXiv preprint arXiv:2310.20111, 2023

    2023 arXiv

  20. [28]

    Large language models can be strong differentially private learners, 2022

    Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners, 2022. URL https://arxiv.org/abs/2110.05679

    2022 arXiv

  21. [29]

    Efficient model development through fine-tuning transfer, 2025

    Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, and Tu Vu. Efficient model development through fine-tuning transfer, 2025. URL https://arxiv.org/abs/2503.20110

    2025

  22. [30]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  23. [31]

    Dilm: Distilling dataset into language model for text-level dataset distillation, 2024

    Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, and Manabu Okumura. Dilm: Distilling dataset into language model for text-level dataset distillation, 2024. URL https://arxiv.org/abs/2404.00264

    2024 arXiv

  24. [32]

    The llama 3 herd of models, 2024

    Meta-AI. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  25. [33]

    Coresets for robust training of deep neural networks against noisy labels

    Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33: 0 11465--11477, 2020

    2020

  26. [34]

    Twenty Newsgroups

    Tom Mitchell. Twenty Newsgroups . UCI Machine Learning Repository, 1997. DOI : https://doi.org/10.24432/C5C323

    1997 doi

  27. [35]

    Morris, Wenting Zhao, Justin T

    John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. Language model inversion, 2023. URL https://arxiv.org/abs/2311.13647

    2023 arXiv

  28. [36]

    Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G

    John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize?, 2025. URL https://arxiv.org/abs/2505.24832

    2025 arXiv

  29. [37]

    Feder Cooper, Daphne Ippolito, Christopher A

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023. URL https://arxiv.o...

    2023 arXiv

  30. [38]

    G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions— I . Mathematical Programming, 14 0 (1): 0 265--294, 12 1978. doi:10.1007/BF01588971. URL https://doi.org/10.1007/BF01588971

    1978 doi

  31. [39]

    Synthetic text generation for training large language models via gradient matching, 2025

    Dang Nguyen, Zeman Li, Mohammadhossein Bateni, Vahab Mirrokni, Meisam Razaviyayn, and Baharan Mirzasoleiman. Synthetic text generation for training large language models via gradient matching, 2025. URL https://arxiv.org/abs/2502.17607

    2025 arXiv

  32. [40]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 0 19920--19930, 2020

    2020

  33. [41]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  34. [42]

    Training data reconstruction: Privacy due to uncertainty?, 2024

    Christina Runkel, Kanchana Vaishnavi Gandikota, Jonas Geiping, Carola-Bibiane Schönlieb, and Michael Moeller. Training data reconstruction: Privacy due to uncertainty?, 2024. URL https://arxiv.org/abs/2412.08544

    2024 arXiv

  35. [43]

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020. URL https://arxiv.org/abs/1811.10959

    2020 arXiv

  36. [44]

    Less: Selecting influential data for targeted instruction tuning, 2024

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024. URL https://arxiv.org/abs/2402.04333

    2024 arXiv

  37. [45]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023

    2023

  38. [46]

    Junjie Oscar Yin and Alexander M. Rush. Compute-constrained data selection, 2025. URL https://arxiv.org/abs/2410.16208

    2025 arXiv

  39. [47]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective, 2024

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective, 2024. URL https://arxiv.org/abs/2306.13092

    2024 arXiv

  40. [48]

    Character-level convolutional networks for text classification, 2016

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2016. URL https://arxiv.org/abs/1509.01626

    2016 arXiv

  41. [49]

    Dataset condensation with gradient matching, 2021

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, 2021. URL https://arxiv.org/abs/2006.05929

    2021 arXiv

  42. [50]

    Dataset distillation using neural feature regression

    Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 35: 0 9813--9827, 2022

    2022

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.