Pith. sign in

REVIEW 4 major objections 7 minor 51 references

LAMDAS claims that a likelihood-ratio score from a small domain prefix selects data that outperforms full-data training and nine baselines, with only two forward passes per candidate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Clever and efficient likelihood-ratio data selection via a learned domain prefix, but threshold tuning on the test benchmarks inflates the reported gains. the 4 major comments →

arxiv 2509.06524 v1 pith:WSHQC3KD submitted 2025-09-08 cs.CL

LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection

Jian Wu , Hang Yu , Bingchang Liu , Wenjie Yang , Peng Di , Jianguo Li , Yue Zhang This is my paper
classification cs.CL
keywords data selectionlikelihood ratioprefix tuningone-class classificationdomain adaptationcontinual pre-trainingsupervised fine-tuningcode and math reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the problem of selecting training data for a specific domain can be solved by asking the language model itself to act as a one-class classifier: learn a soft prefix that maximizes the likelihood of a small reference set, then score each candidate by the likelihood ratio p(y|C)/p(y). Candidates whose likelihood rises when the prefix is present—score above 1—are kept. The paper reports that models continually pre-trained or fine-tuned on LAMDAS-selected subsets outperform models trained on the full data, and also outperform nine existing selection baselines, while requiring only two model forward passes per candidate. A sympathetic reader would care because it promises accurate domain adaptation with a fraction of the data and modest compute, without hand-crafted features or per-candidate optimization.

Core claim

LAMDAS's central claim is that a likelihood-ratio score computed from a pretrained LLM, without updating the LLM's weights, is a sufficient and efficient relevance signal for domain-specific data selection. Formally, the paper learns a domain prefix C by maximizing log p(x|C) over a small reference set, then scores a candidate y as s(y)=p(y|C)/p(y) and keeps it when s(y)>1. By the Neyman-Pearson lemma, this score is the optimal discriminant between the reference-conditioned and unconditional distributions. Experiments on Qwen2.5 models show, for example, a 24.7% average improvement over full-data continual pre-training for the 7B model using 15% of the data, and a 19.6% average improvement o

What carries the argument

The domain prefix—a learnable soft prefix of fixed length (about 30 tokens) tuned only on the reference data—is the object that carries the argument. It condenses the reference set into a representation that interacts with each candidate through the LLM's cross-attention, and the likelihood ratio s(y)=p(y|C)/p(y) against the prefix-free likelihood turns the LLM into a one-class classifier. The score's optimality rests on the Neyman-Pearson lemma, which is what justifies the threshold of 1.

Load-bearing premise

The learned domain prefix faithfully represents the target domain, so the likelihood ratio p(y|C)/p(y) is a reliable relevance score; if the small reference set is noisy, skewed, or unrepresentative, the prefix cannot separate useful candidates from useless ones.

What would settle it

Build a reference set from one narrow slice of a domain, such as only Python function-definition problems, and a candidate pool spanning many code categories; then train on the LAMDAS-selected subset and evaluate on held-out tasks from the other categories. If the selected-subset model scores below a random subset of equal size on those held-out tasks, the likelihood-ratio score fails to capture the domain as a whole.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Using roughly 15% of candidate tokens for CPT and 30–40% for SFT, models trained on LAMDAS-selected data outperform full-data training on code benchmarks, with gains persisting across model sizes.
  • Selection requires only two forward passes per candidate, so switching domains means retraining a short prefix rather than retraining the LLM.
  • Because the LLM's weights are untouched, small LLMs (0.5B parameters) can serve as the implicit classifier, further reducing selection cost.
  • The same likelihood-ratio score can be combined with quality, diversity, or novelty filters in a pipeline, since it targets domain relevance rather than data quality alone.
  • The Neyman-Pearson justification gives the score a principled decision threshold, making the method nearly hyperparameter-free beyond prefix length and model choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The likelihood-ratio framing suggests a direct test of the domain-representation assumption: if the reference set is a mixture of two coherent subdomains, the learned prefix should behave like a mixture model, and the score should rank pooled candidates by their subdomain membership probabilities.
  • When the reference set is noisy, skewed, or multi-modal, the prefix may collapse onto one mode, causing LAMDAS to systematically filter out samples from other modes; the paper does not explore this failure regime.
  • The same implicit-classifier idea could be extended to multiple reference domains, producing a soft routing score rather than a binary keep/drop decision and turning data selection into multi-way classification.
  • The reported gains over full-data training may partly reflect the well-known risk of catastrophic forgetting when continually pretraining on pure code; a longer-horizon comparison measuring retention of general NLP ability would clarify how much of the gain is selection rather than forgetting avoidance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes LAMDAS, a data-selection method for domain-specific continual pretraining (CPT) and supervised fine-tuning (SFT). A soft prefix is tuned on a small reference dataset (Eq. 1); each candidate sample y is scored by the likelihood ratio s(y) = p(y|C)/p(y) under a pretrained LLM with and without this prefix, and samples above threshold τ are selected. The paper reports that LAMDAS outperforms nine baselines and full-data training across code and math benchmarks, with average relative gains up to 24.7% over the Full baseline for CPT on Qwen2.5-7B and 19.6% for SFT code on Qwen2.5-7B-Instruct, while being substantially faster than direct-optimization baselines. Experiments cover model sizes 0.5B, 1.5B, 7B, and 32B, with ablations on τ, prefix length, classifier size, and classifier type.

Significance. If the empirical claims hold, LAMDAS is a simple, efficient, and reasonably general recipe for domain-specific data selection: it needs only a small reference set and uses the pretrained LLM itself as a scorer, avoiding gradient computation and extra classifier training. The paper includes a broad set of comparisons, a concrete efficiency analysis, and ablations over several design choices. Its strengths are the simplicity of the method and the breadth of the evaluation across CPT/SFT, code/math, and four model sizes. However, the theoretical framing (Neyman-Pearson optimality, mutual-information maximization) is asserted rather than established, and the empirical evaluation is weakened by the use of the test benchmarks to select the threshold τ. With a clean held-out evaluation and a more measured theoretical presentation, this would be a useful contribution to the data-selection literature.

major comments (4)
  1. [§4.4 / Fig. 4] The threshold τ is tuned on the same evaluation benchmarks that are later used to report the headline numbers. Fig. 4 varies τ ∈ {0.8, 0.9, 1.0, 1.1, 1.2} over MATH500, AIME, GSM8K, HE, HE+, MBPP, and MBPP+ for all four model sizes and then selects τ = 1.0 as 'optimal'. Tables 2 and 7 then report results with this threshold as LAMDAS's performance. Because τ directly controls retention ratios (≈50%, 40%, 30%, 10%, 5%), the reported numbers are the best among several runs, not an unbiased estimate, and no baseline receives a comparable test-set tuning. This can inflate the claimed improvements. The authors should choose τ on a held-out validation split, or report results for all thresholds for all methods; the main comparisons should be re-run accordingly.
  2. [§3.1.2, Proposition 3.1] The Neyman-Pearson claim is not supported. Optimal likelihood-ratio testing requires two simple hypotheses with known class-conditional densities. Here p(y|C) and p(y) are both text probabilities produced by the same LLM, with and without a learned prefix; they are not the conditional densities of 'in-domain' vs 'out-of-domain' candidate data. The proposition is therefore an analogy or heuristic, not a theorem. Additionally, Eq. (2) treats p(C) as a prior over the domain, but C is a point-estimated prefix parameter vector, so p(C) is not a well-defined prior. The statement in §3.1.1 that Eq. (1) 'maximizes the mutual information between the reference data and C' is also unproven: maximizing log p(x_{1:N}|C) over C does not by itself maximize MI between the text and a parameter vector. Please either add the missing assumptions/derivation or explicitly present these as motivations rather t
  3. [Table 3 / Appendix A.5] The efficiency comparison omits the cost of learning the domain prefix. Table 3 lists LAMDAS's time complexity as 2×C_forward, but Appendix A.5 says prefix tuning runs for 10 epochs over the reference set, consuming about one hour on a single A100 GPU. This is a fixed training overhead before any candidate is scored. For a fair complexity accounting, this prefix-training cost should be included in the comparison, or justified as negligible with an amortized calculation. Without this, Figure 1's 'most favorable trade-off' claim is overstated relative to methods that do not train on the reference set.
  4. [§4.1, Table 1] The 'Full' baseline is not a meaningful upper bound in the CPT setting: the paper itself reports that Random beats Full (e.g., 29.6 vs 28.7 average for Qwen2.5-7B) and attributes this to catastrophic forgetting. Since the paper's headline is 'LAMDAS outperforms full-data training,' the comparison with Full is confounded by training budget. The paper should consistently frame the result as 'LAMDAS outperforms Random and other selection methods at the same budget,' and report a matched-budget control (same token count, random selection) as the primary baseline, rather than emphasizing the Full comparison.
minor comments (7)
  1. [Tables 1, 2, 7] All results are single runs without standard errors or multiple seeds. Given the relatively small benchmark sizes (e.g., AIME has 30 problems), percentage differences of a few points can be within noise. Reporting seeds, repeats, or confidence intervals would substantially strengthen the empirical claims.
  2. [Abstract / §1] The abstract says 'nine SOTA baselines,' while the introduction says seven baselines for CPT and nine for SFT. Make the count consistent and specify which baselines are used where.
  3. [Table 2] Typo: 'Qwen2.5 serious models' should be 'Qwen2.5 series models.' Similar typos appear in Table 7 and elsewhere.
  4. [Table 3 / Table 8] Units are inconsistent: Table 3 reports 'Run Time s/100' and Table 8 reports 'Speed s/k'. Use a single convention, e.g., seconds per 100 samples, and clarify whether the times include model loading.
  5. [References] Engstrom et al. are cited as both 2024a and 2024b for what appears to be the same DsDm paper. Please check and merge duplicate entries.
  6. [§1 / Footnote a] The footnote says 'Code will be available upon publication.' For a method whose main value is practical data selection, code release is important; please make the code and configuration public, or at least provide detailed pseudocode and hyperparameter settings.
  7. [Figure 1] The caption sentence is duplicated ('suggests a more favorable trade-off' appears twice). Also, the regression line has no error or stability analysis, so the perpendicular-distance ranking should be treated as illustrative.

Circularity Check

1 steps flagged

Reported gains are partly tuned: the selection threshold τ is chosen on the evaluation benchmarks later used for the headline results.

specific steps
  1. fitted input called prediction [Section 4.4 / Appendix C.1 / Figure 4]
    "As shown in Figure 4, a high threshold over-filters valuable examples, reducing diversity and harming generalization, while a low threshold introduces noisy data, degrading performance. An optimal threshold of 1.0 balances quality and diversity, leading to peak performance across tasks regardless of the underlying model for SFT."

    Figure 4 tunes τ over 0.8, 0.9, 1.0, 1.1, 1.2 on exactly the benchmarks used in the final tables (MATH500, AIME, GSM8K, HE, HE+, MBPP, MBPP+), with retention ratios ranging from about 50% down to 5%. Section 4.4 then declares τ=1.0 optimal, and the headline results in Tables 2 and 7 are produced with that value. The reported performance is therefore the best of five test-set-conditioned configurations, not an unbiased prediction of the method's behavior. τ is a fitted input selected on the evaluation data, and the 'peak performance' claim is partly an artifact of that fit. No baseline is granted equivalent test-set threshold tuning, inflating the relative gains.

full rationale

The core derivation is not circular. The selection score s(y_j)=p(y_j|C)/p(y_j) is defined from the pretrained LLM and a prefix C fit to the reference data via Eq. (1); the Neyman-Pearson justification is external; the only overlapping self-citation (D2LLM, Liao et al. 2024) is used for cross-encoder terminology and is not load-bearing. However, the empirical validation is partially circular: the threshold τ, which directly controls subset composition and size, is selected by maximizing performance on the evaluation benchmarks (Figure 4, Appendix C.1), and the same benchmarks are then used to report the final comparisons. The headline improvements are thus partly tuned rather than predicted. The prefix-length choice is tuned on reference-data likelihood rather than test benchmarks, and the classifier-size ablation shows only small differences, so the central score remains independent. Overall this is partial evaluation circularity, not a derivation that reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claim is empirical and does not introduce new physical or ontological entities. The method relies on a learned prefix (a free parameter) and on two domain assumptions about representational sufficiency and likelihood comparability. The only mathematical axiom is Neyman-Pearson optimality, which is standard but applied here without verifying that the LLM's likelihoods are true data densities.

free parameters (4)
  • Domain prefix C = learned vector (prefix length 30, model hidden dimension)
    Trainable prefix optimized to maximize likelihood of reference data (Eq. 1). This is the core learned quantity of the method.
  • Selection threshold tau = 1.0
    Chosen by design and then validated by ablation (Appendix C.1) on the same benchmarks used for final comparisons.
  • Prefix length L = 30
    Selected from Figure 3 as the point with maximal log-likelihood on reference data, then fixed for all experiments.
  • Classifier model size = Qwen2.5-Coder-0.5B
    Ablation (Table 8) shows 0.5B is sufficient; the final experiments use this small model for efficiency.
axioms (3)
  • standard math Neyman-Pearson lemma: likelihood ratio is the optimal test statistic for binary hypothesis testing with fixed error rates.
    Used in Proposition 3.1 to justify the score s(y_j) as an optimal domain discriminant.
  • domain assumption A learned soft prefix trained on the reference data faithfully represents the target domain distribution.
    Section 3.1.1 states 'the prefix can effectively retain essential information about the domain defined by that dataset.' If this fails, the likelihood ratio will not select domain-relevant data.
  • domain assumption The LLM's next-token likelihoods provide comparable and sufficiently calibrated probabilities across diverse candidate documents.
    The method uses raw likelihoods p(y|C) and p(y) without length normalization or calibration, so the score may be biased by sequence length or model quirks.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection." pith.science (2026). https://pith.science/paper/WSHQC3KD

@misc{pith2026250906524,
  author       = {Pith},
  title        = {Pith review of: LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSHQC3KD}},
  note         = {Machine review of arXiv:2509.06524}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Adapting large language models (LLMs) to specific domains often faces a critical bottleneck: the scarcity of high-quality, human-curated data. While large volumes of unchecked data are readily available, indiscriminately using them for fine-tuning risks introducing noise and degrading performance. Strategic data selection is thus crucial, requiring a method that is both accurate and efficient. Existing approaches, categorized as similarity-based and direct optimization methods, struggle to simultaneously achieve these goals. In this paper, we introduce LAMDAS (LLM As an iMplicit classifier for domain-specific DAta Selection), a novel approach that leverages the pre-trained LLM itself as an implicit classifier, thereby bypassing explicit feature engineering and computationally intensive optimization process. LAMDAS reframes data selection as a one-class classification problem, identifying candidate data that "belongs" to the target domain defined by a small reference dataset. Extensive experimental results demonstrate that LAMDAS not only exceeds the performance of full-data training using a fraction of the data but also outperforms nine state-of-the-art (SOTA) baselines under various scenarios. Furthermore, LAMDAS achieves the most compelling balance between performance gains and computational efficiency compared to all evaluated baselines.

Figures

Figures reproduced from arXiv: 2509.06524 by Bingchang Liu, Hang Yu, Jianguo Li, Jian Wu, Peng Di, Wenjie Yang, Yue Zhang.

Figure 1
Figure 1. Figure 1: Performance gains versus data selection efficiency (selection time) for all methods in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution comparison of candidate data (yellow), LAMDAS-selected data (blue), DEITA [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of the prefix lengths on the log-likelihood of the reference data. The results indicate [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation Studies on the impact of varying thresholds on the selected SFT data resulting [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, abs/2108.07732, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    Color-filter: Conditional loss reduction filtering for targeted language model pre-training

    David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. Color-filter: Conditional loss reduction filtering for targeted language model pre-training. Advances in Neural Information Processing Systems, 37: 0 97618--97649, 2024

    work page 2024

  4. [4]

    Statistical inference

    George Casella and Roger Berger. Statistical inference. CRC press, 2024

    2024

  5. [5]

    Maybe only 0.5\ arXiv preprint arXiv:2305.09246, 2023

    Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5\ arXiv preprint arXiv:2305.09246, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    Pith/arXiv arXiv 2025

  9. [9]

    Sketchy moment matching: Toward fast and provable data selection for finetuning

    Yijun Dong, Viet Hoang Phan, Xiang Pan, and Qi Lei. Sketchy moment matching: Toward fast and provable data selection for finetuning. Advances in Neural Information Processing Systems, 37: 0 43367--43402, 2024

    work page 2024

  10. [10]

    Dsdm: Model-aware dataset selection with datamodels

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. In International Conference on Machine Learning, pp.\ 12491--12526. PMLR, 2024 a

    work page 2024

  11. [11]

    Dsdm: Model-aware dataset selection with datamodels

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. In International Conference on Machine Learning, pp.\ 12491--12526. PMLR, 2024 b

    work page 2024

  12. [12]

    Evans, Gordon B

    Nathaniel J. Evans, Gordon B. Mills, Guanming Wu, Xubo Song, and Shannon K. McWeeney. Data valuation with gradient similarity. ArXiv, 2024

    work page 2024

  13. [13]

    GIO : Gradient information optimization for training dataset selection

    Dante Everaert and Christopher Potts. GIO : Gradient information optimization for training dataset selection. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  14. [14]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Data selection via optimal control for language models

    Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  16. [16]

    SHED : Shapley-based automated dataset refinement for instruction fine-tuning

    Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. SHED : Shapley-based automated dataset refinement for instruction fine-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  17. [17]

    Evaluating sample utility for data selection by mimicking model weights

    Tzu-Heng Huang, Manjot Bilkhu, Frederic Sala, and Javier Movellan. Evaluating sample utility for data selection by mimicking model weights. ArXiv, abs/2501.06708, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  19. [19]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020

    Pith/arXiv arXiv 2001

  20. [20]

    Rule-based data selection for large language models

    Xiaomin Li, Mingye Gao, Zhiwei Zhang, Chang Yue, and Hong Hu. Rule-based data selection for large language models. ArXiv, abs/2410.04715, 2024

    arXiv 2024

  21. [21]

    One shot learning as instruction data prospector for large language models

    Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. One shot learning as instruction data prospector for large language models. In Annual Meeting of the Association for Computational Linguistics, 2023

    work page 2023

  22. [22]

    D 2 LLM : Decomposed and distilled large language models for semantic search

    Zihan Liao, Hang Yu, Jianguo Li, Jun Wang, and Wei Zhang. D 2 LLM : Decomposed and distilled large language models for semantic search. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14798--14814, Bangkok, Thailand, August 2024. ...

    doi:10.18653/v1/2024.acl-long.791 2024

  23. [23]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024 a

    work page 2024

  25. [25]

    Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models

    Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, and Haizhou Li. Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models. arXiv preprint arXiv:2406.14115, 2024 b

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    TSDS : Data selection for task-specific model finetuning

    Zifan Liu, Amin Karbasi, and Theodoros Rekatsinas. TSDS : Data selection for task-specific model finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 c

    work page 2024

  27. [27]

    When less is more: Investigating data pruning for pretraining llms at scale

    Max Marion, Ahmet \"U st \"u n, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. CoRR, 2023

    work page 2023

  28. [28]

    Sgpt: Gpt sentence embeddings for semantic search

    Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  30. [30]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning, pp.\ 27074--27113. PMLR, 2023

    work page 2023

  31. [31]

    O1 replication journey: A strategic progress report - part 1

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: A strategic progress report - part 1. ArXiv, abs/2410.18982, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Learning to retrieve prompts for in-context learning

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2655--2671, Seattle,...

    doi:10.18653/v1/2022.naacl-main.191 2022

  33. [33]

    McAuley, and Derek Zhiyuan Cheng

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed Huai hsin Chi, James Caverlee, Julian J. McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. ArXiv, abs/2402.09668, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Improving dense retrieval models with llm augmented data for dataset search

    Levy Silva and Luciano Barbosa. Improving dense retrieval models with llm augmented data for dataset search. Know.-Based Syst., 294 0 (C), June 2024. ISSN 0950-7051. doi:10.1016/j.knosys.2024.111740

    work page arXiv 2024

  35. [35]

    Improving pretraining data using perplexity correlations

    Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  36. [36]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur'elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    How do your code LLM s perform? empowering code instruction tuning with really good data

    Yejie Wang, Keqing He, Dayuan Fu, Zhuoma GongQue, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang, Xunliang Cai, and Weiran Xu. How do your code LLM s perform? empowering code instruction tuning with really good data. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conf...

    work page 2024

  38. [38]

    QuRating : Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating : Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024

    work page 2024

  39. [39]

    LESS : Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS : Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML), 2024

    work page 2024

  40. [40]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  41. [41]

    Wizard LM : Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizard LM : Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  42. [42]

    Diffusion models: A comprehensive survey of methods and applications

    Ling Yang, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56: 0 1 -- 39, 2022

    work page 2022

  43. [43]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. ArXiv, abs/2502.03387, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. In NeurIPS, 2024

    work page 2024

  45. [45]

    Siren's song in the ai ocean: a survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Beyond similarity: A gradient-based graph method for instruction tuning data selection

    Yang Zhao, Li Du, Xiao Ding, Yangou Ouyang, Hepeng Wang, Kai Xiong, Jinglong Gao, Zhouhao Sun, Dongliang Xu, Qing Yang, et al. Beyond similarity: A gradient-based graph method for instruction tuning data selection. CoRR, 2025

    work page 2025

  47. [47]

    Opencodeinterpreter: Integrating code generation with execution and refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    LIMA : Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA : Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  49. [49]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  50. [50]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  51. [51]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.