Investigating the Impact of Data Selection Strategies on Language Model Performance

Jiayao Gu; Liting Chen; Yihong Li

REVIEW 3 major objections 5 minor 16 references

Investigating the Impact of Data Selection Strategies on Language Model Performance

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that neural embedding-based data selection improves over random selection for pretraining, while n-gram-based DSIR remains stronger on most GLUE tasks.

desk verdict The central claim is contradicted by the paper's own table, and the 47k-vs-1.7M confound makes the comparison unidentifiable; the work has a clear idea but is not ready for review. read the letter →

arxiv 2501.03826 v1 pith:AE5DZZFH submitted 2025-01-07 cs.CL cs.LG

Jiayao Gu , Liting Chen , Yihong Li This is my paper · ORCID

classification cs.CLcs.LG

keywords dataselectionimportanceresamplinglanguagemodelpretrainingdistributionalignmentn-gramfeaturessentenceembeddingsGLUEbenchmarkhybrid

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the choice of pretraining data can be steered toward a target distribution and whether neural sentence embeddings add anything beyond n-gram statistics. It proposes HIR, Hybrid Importance Resampling, which forms a hybrid target distribution from an n-gram model and a Gaussian mixture model on sentence embeddings, with a parameter $\alpha$ weighting the two. With $\alpha=0$, HIR reduces to pure embedding-based selection; the paper claims this setting improves over random selection, and the reported table shows it ahead on four of six GLUE tasks after continued pretraining. The same table shows DSIR, the n-gram baseline, beating random on all six tasks and posting the best score on five of six. The intended conclusion is that token-level n-gram statistics remain the strongest single signal for pretraining data curation, while embedding-based selection offers a complementary, context-level signal.

What carries the argument

The central object is the Hybrid Importance Resampling (HIR) weighting scheme, a hybrid importance weight $$\$omega_i^{{\mathrm{hybrid}}$} = \left(\frac{\hat p_{\mathrm{ng}}(x_i)}{\hat q_{\mathrm{ng}}(x_i)}\right)^{\$\alpha$} \left(\frac{\hat p_{\mathrm{nn}}(x_i)}{\hat q_{\mathrm{nn}}(x_i)}\right)^{1-\$\alpha$}$$ that combines hashed n-gram statistics with Gaussian-mixture density estimates over sentence embeddings. Setting $\alpha=0$ makes the weight purely neural, the configuration tested in the report. This weight is what carries the argument: it is the mechanism by which raw documents are ranked and the top $k$ are selected, and it is what lets the paper attribute performance differences to the choice of feature type.

What would settle it

Run the same continued-pretraining and GLUE evaluation with HIR ($\alpha=0$) and DSIR each selecting exactly the same number of documents from the same raw pool, with no first-47,000 substitution and matched random seeds; if HIR no longer beats random selection on most tasks when sizes are equal, the consistency claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that distribution-matched data selection improves language model pretraining, and that n-gram and neural embedding features capture complementary aspects of the target distribution. Formally, HIR estimates $\hat p_{\text{ngram}}$ with a bag-of-hashed-n-grams model and $\hat p_{\text{nn}}$ with a diagonal Gaussian mixture model on 384-dimensional sentence embeddings, then defines the hybrid target $\hat p_{\text{hybrid}}(x)=\hat p_{\text{ngram}}(x)^{\alpha}\,\hat p_{\text{nn}}(x)^{1-\alpha}$ and resamples raw documents with probability proportional to $\hat p_{\text{hybrid}}(x)/\hat q_{\text{hybrid}}(x)$. Setting $\alpha=0$ isolates the neural signal. The paper argues that this neural-only variant aligns the raw data with the target at a broader contextual level and therefore improves over random selection, while DSIR's token-level n-gram weights are more directly aligned with masked-token pretraining and win five of the six GLUE tasks.

Load-bearing premise

The comparison assumes that substituting the first 47,000 documents of DSIR's 1.7 million-document selection with HIR's 47,000 documents keeps the two training sets comparable, so that any performance difference is due to the selection method rather than to dataset size or to which documents were swapped in.

Editorial extensions

If this is right

Distribution-matched selection improves downstream GLUE performance relative to random selection: DSIR beats random on all six tasks, and HIR beats random on four.
DSIR's n-gram importance weights win five of six tasks, so token-level statistics remain the strongest single feature type for pretraining data curation.
HIR with $\alpha=0$ wins on STS-B and beats random on four tasks without using n-gram features, so sentence-embedding alignment is a viable complementary signal.
Only $\alpha=0$ was tested; tuning $\alpha$ is the paper's stated next step for combining n-gram and neural signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper tests only the pure neural regime ($\alpha=0$); an intermediate $\alpha$ that lets n-gram and neural weights trade off is the natural next experiment and is explicitly left as future work.
The paper's discussion attributes HIR's underperformance partly to imprecise density estimates from a diagonal GMM in a high-dimensional embedding space; replacing the GMM with a more expressive density estimator is a direct way to test whether the neural feature type or its estimation is the bottleneck.
Because the reported comparison substitutes HIR's 47,000 documents into the first 47,000 positions of DSIR's 1.7 million-document set, an equal-size, same-pool comparison would separate the method's effect from the effect of which documents were swapped in.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Readable short study. The core result as stated — HIR consistently beats random — is not supported by Table 2, where HIR is numerically below random on MRPC and QNLI. More fundamentally, the experimental comparison is not identifiable: HIR selects 47k documents from a 1.4M-doc pool, while DSIR and random use 1.7M documents from a 58M-doc pool. Replacing the first 47k DSIR documents with HIR's doesn't equalize size or source; any difference could come from data volume or pool composition.

What is actually new: the specific test of a GMM fitted to SentenceTransformer embeddings as an importance resampler, compared on six GLUE tasks. That is a reasonable thing to try, and the paper is transparent that it only runs α=0, so the hybrid method is not actually evaluated. The authors also acknowledge the small scale and note the computational limits. That honesty earns some credit.

The main soft spots beyond the confound: the performance gaps are small and often within standard deviation, and the reported numbers don't support the 'consistent gains' language in the abstract. The method as described in Section 3.2 is never tested; α=0 is just GMM resampling. So the contribution is narrower than claimed.

The citation pattern is fine; DSIR is the right baseline, and the related work is adequate for a short report. This looks like a careful course project rather than a publication-ready paper. I would not cite it as evidence for neural feature selection. A serious referee would likely reject on the confound alone, so I would desk reject. If the authors re-run with matched subset sizes and pools, there might be a modest workshop paper.

Referee Report

3 major / 5 minor

Summary. The paper proposes HIR, a hybrid importance-resampling method that combines hashed n-gram statistics with SentenceTransformer embedding features for pretraining data selection. The authors compare HIR (with the hybrid weight fixed to α=0, i.e., the neural/GMM component only) against random selection and DSIR on the uncopyrighted Pile, followed by continued pretraining and fine-tuning on six GLUE tasks. The paper claims that data selection improves performance, that HIR with α=0 consistently improves over random selection, and that DSIR remains stronger on most token-level tasks.

Significance. If the central claim were supported, the paper would provide evidence that neural embedding features offer a complementary signal for data curation, which is a plausible and potentially useful direction. The paper has some strengths: it evaluates on an external benchmark (GLUE), it provides a GitHub repository with code, and the modeling setup (fitting GMM and n-gram distributions to raw and target data, then evaluating on downstream tasks) does not contain circular reasoning. However, the experimental evidence does not support the headline claims, and the main comparison is confounded by a large mismatch in pretraining dataset size and candidate-pool size.

major comments (3)

[Abstract and Section 6, Table 2] The abstract and Section 6 claim that HIR with α=0 'consistently improves over random selection.' Table 2 directly contradicts this: HIR is worse than random on MRPC (79.90±1.49 vs 82.68±0.61) and on QNLI (85.23±0.14 vs 85.32±0.33). Even on tasks where HIR's mean is higher, such as RTE (60.05±1.46 vs 56.92±1.50) and STSB (85.71±0.18 vs 85.06±0.12), the standard deviations overlap substantially, so no consistent or statistically robust improvement is demonstrated. The central claim of the paper is therefore not supported by the paper's own results.
[Section 6, experimental setup] The HIR condition is not comparable to the DSIR and random baselines. HIR selected 47,000 instances from the first 1.4 million documents of the raw dataset, while DSIR and random selected 1.7 million instances from the first 58 million documents. The statement that 'we replaced the first 47,000 instances selected by DSIR with the 47,000 instances selected by HIR' does not equalize pretraining dataset size; the HIR-pretrained model sees 47,000 documents and the baselines see 1.7 million. In addition, the candidate pools differ (1.4M vs 58M documents). Any observed difference between HIR and random, or between HIR and DSIR, could therefore be caused by the amount of training data or the composition of the raw pool, not by the selection method. This confound undermines the paper's main comparison and its claim that 'data selection improves performance.'
[Section 3.2 and Section 6] The proposed method is a hybrid of n-gram and neural features with a weighting parameter α, but the experiments only evaluate α=0, which reduces HIR to the GMM/neural component alone. The paper acknowledges this in Section 6 and the Limitations, but the abstract and conclusion nevertheless present the method as a hybrid and claim that 'the proposed hybrid importance reweighting (HIR) method ... consistently improves over random selection.' The hybrid model has not actually been tested, and the contribution that distinguishes HIR from DSIR (the weighted combination of n-gram and neural features) is never evaluated. To support the paper's claims, the authors would need to run experiments with α in (0,1) and demonstrate the claimed improvements under matched pretraining budgets.

minor comments (5)

[Section 3.2] The text says 'we proposed to incorporate the neural network features'; the tense should be 'we propose' for consistency with the rest of the paper.
[Section 3.1] The definition of γ is incomplete as printed: 'γ = 1Ps j=1 Pm i=1 zji Ps j=1 zj' is missing the denominator normalization. Please rewrite the estimator clearly and add equation numbers.
[Section 6, Table 2] The caption should state explicitly that the subscripts are standard deviations across the three random seeds, since the table otherwise reports numbers like '18.7310.22' that are easy to misread.
[Section 4] The pretraining section refers to hyperparameters from Xie et al. (2023) but does not specify the model architecture, number of parameters, optimizer, learning rate, or pretraining steps. Adding these details is necessary for reproducibility.
[Section 3.2] The text describes 'resampling' with probabilities proportional to importance weights, but the implementation selects the top-k samples by weight. These are different selection procedures; the paper should clarify which was used and justify the choice, since top-k selection does not correspond to importance resampling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found: HIR importance weights are fit to raw and target distributions and evaluated on external GLUE tasks; the central result is not forced by construction or by self-citation.

full rationale

I walked the paper's derivation chain. The target distribution p and raw distribution q are estimated independently from target and raw corpora: n-gram distributions are fit as bag-of-hashed-n-gram models, and neural distributions are fit as diagonal GMMs over SentenceTransformer embeddings. Importance weights are then formed as ratios of these independently fit densities, and the top-k samples are selected for continued pretraining. The resulting models are evaluated on GLUE, which is external to both the target set (Gutenberg/Wikipedia) and the raw set (uncopyrighted Pile). No fitted parameter is later renamed as a prediction; alpha is fixed to 0 rather than tuned to the evaluation tasks; and no uniqueness theorem or load-bearing self-citation is invoked. The authors cite Can Chen in acknowledgement and cite Chen et al. in related work, but Can Chen is not an author and those citations are not used to justify the central result. The reader-raised issues about the HIR-vs-random comparison are real but are experimental comparability and evidence-consistency concerns, not circularity: HIR was evaluated on a much smaller selection from a different pool, the substitution of the first 47,000 DSIR instances does not control for pool or size, and Table 2 actually shows HIR below random on MRPC and QNLI, contradicting the abstract's claim. The paper itself acknowledges the small-subset, one-pass, and no-alpha-tuning limitations. These concerns affect identifiability and evidentiary strength, but they do not make the derivation equivalent to its inputs. Therefore, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The empirical comparison rests on target-set choice, GMM density assumptions, and a dataset-size confound; these are load-bearing but not independently verified.

free parameters (3)

α (hybrid weight) = 0 (fixed)
Chosen by hand; only α=0 tested, so the hybrid weighting is never explored.
Number of GMM components = qnn=1000, pnn=50
Chosen without tuning; affects density estimates and importance weights.
Selection subset sizes = HIR=47k, DSIR/random=1.7M
Hand-set due to compute; confounds the comparison between methods.

assumptions (3)

domain assumption Wikipedia and Gutenberg (PG-19) form a target distribution whose matching improves GLUE downstream task performance.
Section 6 selects these as target sets and assumes transfer to GLUE; no evidence that this target predicts GLUE performance.
domain assumption A diagonal GMM with 1000 components on 384-dimensional embeddings accurately estimates raw and target densities for importance weights.
Section 3.2 uses it; Section 7 admits diagonal GMM in high dimension may not yield accurate probability estimates.
ad hoc to paper A 47,000-document HIR subset is comparable to a 1.7 million-document DSIR/random subset for measuring method quality.
Section 6 states this replacement; it is an experimental-design assumption specific to this paper and is not justified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Investigating the Impact of Data Selection Strategies on Language Model Performance." pith.science (2026). https://pith.science/paper/AE5DZZFH

@misc{pith2026250103826,
  author       = {Pith},
  title        = {Pith review of: Investigating the Impact of Data Selection Strategies on Language Model Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AE5DZZFH}},
  note         = {Machine review of arXiv:2501.03826}
}

read the original abstract

Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.

Figures

Figures reproduced from arXiv: 2501.03826 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 3 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. http://arxiv.org/abs/2402.16827 A survey on data selection for language models

arXiv 2024
[4]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020
[5]

Sbastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

arXiv 2023
[6]

Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. 2022. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719

arXiv 2022
[7]

Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. 2021. Generalized dataweighting via class-level gradient manipulation. Advances in Neural Information Processing Systems

work page 2021
[8]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

arXiv 2022

Show all 16 references

[9]

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. http://arxiv.org/abs/2004.10964 Don't stop pretraining: Adapt language models to domains and tasks

2020 arXiv
[10]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

2024
[11]

Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . CoRR, abs/1908.10084

2019 arXiv
[12]

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In International conference on machine learning. PMLR

2018
[13]

Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

2018 arXiv
[14]

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. 2023. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems

2023
[15]

Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. http://arxiv.org/abs/2409.14781 Pretraining data detection for large language models: A divergence-based calibration method

2024 arXiv
[16]

Hao Zhou, Chengming Hu, Ye Yuan, Yufei Cui, Yili Jin, Can Chen, Haolun Wu, Dun Yuan, Li Jiang, Di Wu, et al. 2024. Large language model (llm) for telecommunications: A comprehensive survey on principles, key techniques, and opportunities. arXiv preprint arXiv:2405.10825

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. http://arxiv.org/abs/2402.16827 A survey on data selection for language models

arXiv 2024

[4] [4]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020

[5] [5]

Sbastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

arXiv 2023

[6] [6]

Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. 2022. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719

arXiv 2022

[7] [7]

Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. 2021. Generalized dataweighting via class-level gradient manipulation. Advances in Neural Information Processing Systems

work page 2021

[8] [8]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

arXiv 2022