{"id":"5bc6b6ff-bc9b-43f1-9f36-96fa4f9e14f2","arxiv_id":"2501.03826","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Neural-embedding-based data selection underperforms DSIR on most GLUE tasks, and the paper's hybrid method is untested because only α=0 was run.","lead":"This preprint compares random, n-gram-based, and neural-embedding-based data selection for language model pretraining, and reports that n-gram selection wins most GLUE tasks. Its own results contradict the claim that the neural method consistently beats random, and the proposed hybrid method was never actually tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The HIR-vs-random claim is not identifiable because HIR was evaluated on 47k documents from a 1.4M-doc pool, while random/DSIR used 1.7M from a 58M-doc pool; the Section 6 substitution does not control for dataset size or source, and Table 2 already contradicts 'consistently improves'.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the comparison mixes dataset size and source with the selection method. Section 6's substitution of 47k HIR documents into a 1.7M DSIR set does not equalize either the amount of pretraining data or the raw pool from which documents were drawn. The paper itself acknowledges small-scale experimentation, but the design flaw is more severe than a scaling limitation: it prevents any causal attribution of the reported performance differences to HIR or DSIR. Even reading Table 2 at face value, the abstract's claim that HIR 'consistently improves over random selection' is false on MRPC and QNLI, so the central conclusion is internally inconsistent. Additionally, because alpha is never varied, the proposed hybrid distribution is not tested at all; only the alpha=0 neural-only variant appears. I am not attributing intent or dishonesty; the issue is that the experiment as reported cannot support the claim. A controlled re-run with equal dataset size and identical raw pool would settle the matter, and until then the appropriate verdict is REJECT.","tokens_in":7398,"tokens_out":3051,"duration_ms":27755,"concrete_test":"Re-run the Section 6 comparison with HIR at k=1.7M, sampling from the same first 58M-document raw dataset and target sets used for DSIR and random, then compare the six GLUE metrics. If HIR does not beat random on all six (or its standing changes), the consistency claim fails. A cheaper complementary check: run random, DSIR, and HIR all at k=47,000 from the same 1.4M-document pool to verify whether any method effect remains when dataset size and source are controlled.","verdict_should_be":"REJECT","load_bearing_attack":"Section 6 reports that DSIR and random select 1.7M instances from the first 58M documents of the uncopyrighted Pile, while HIR selects 47,000 instances from the first 1.4M documents. The paper then says 'we replaced the first 47,000 instances selected by DSIR with the 47,000 instances selected by HIR' to ensure consistency in dataset size. That substitution does not make the groups comparable: the HIR-pretrained model sees 47k documents, the DSIR and random models see 1.7M documents, and HIR's candidate pool (1.4M) differs from the others' (58M). Any observed advantage of HIR over random, or disadvantage relative to DSIR, could be caused by dataset size and raw-pool composition rather than by the selection method. The abstract's 'consistently improves over random selection' claim is also internally contradicted by Table 2: HIR is worse than random on MRPC (79.90 vs 82.68) and QNLI (85.23 vs 85.32), so the evidence does not support the central claim. In addition, alpha is fixed to 0, so the proposed hybrid method is never actually evaluated; only the GMM/neural component is tested. These issues make the headline result not only weak but not identifiable from the reported experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HIR, a hybrid importance-resampling method that combines hashed n-gram statistics with SentenceTransformer embedding features for pretraining data selection. The authors compare HIR (with the hybrid weight fixed to α=0, i.e., the neural/GMM component only) against random selection and DSIR on the uncopyrighted Pile, followed by continued pretraining and fine-tuning on six GLUE tasks. The paper claims that data selection improves performance, that HIR with α=0 consistently improves over random selection, and that DSIR remains stronger on most token-level tasks.","tokens_in":7747,"tokens_out":3225,"duration_ms":31354,"significance":"If the central claim were supported, the paper would provide evidence that neural embedding features offer a complementary signal for data curation, which is a plausible and potentially useful direction. The paper has some strengths: it evaluates on an external benchmark (GLUE), it provides a GitHub repository with code, and the modeling setup (fitting GMM and n-gram distributions to raw and target data, then evaluating on downstream tasks) does not contain circular reasoning. However, the experimental evidence does not support the headline claims, and the main comparison is confounded by a large mismatch in pretraining dataset size and candidate-pool size.","major_comments":[{"comment":"The abstract and Section 6 claim that HIR with α=0 'consistently improves over random selection.' Table 2 directly contradicts this: HIR is worse than random on MRPC (79.90±1.49 vs 82.68±0.61) and on QNLI (85.23±0.14 vs 85.32±0.33). Even on tasks where HIR's mean is higher, such as RTE (60.05±1.46 vs 56.92±1.50) and STSB (85.71±0.18 vs 85.06±0.12), the standard deviations overlap substantially, so no consistent or statistically robust improvement is demonstrated. The central claim of the paper is therefore not supported by the paper's own results.","section":"Abstract and Section 6, Table 2"},{"comment":"The HIR condition is not comparable to the DSIR and random baselines. HIR selected 47,000 instances from the first 1.4 million documents of the raw dataset, while DSIR and random selected 1.7 million instances from the first 58 million documents. The statement that 'we replaced the first 47,000 instances selected by DSIR with the 47,000 instances selected by HIR' does not equalize pretraining dataset size; the HIR-pretrained model sees 47,000 documents and the baselines see 1.7 million. In addition, the candidate pools differ (1.4M vs 58M documents). Any observed difference between HIR and random, or between HIR and DSIR, could therefore be caused by the amount of training data or the composition of the raw pool, not by the selection method. This confound undermines the paper's main comparison and its claim that 'data selection improves performance.'","section":"Section 6, experimental setup"},{"comment":"The proposed method is a hybrid of n-gram and neural features with a weighting parameter α, but the experiments only evaluate α=0, which reduces HIR to the GMM/neural component alone. The paper acknowledges this in Section 6 and the Limitations, but the abstract and conclusion nevertheless present the method as a hybrid and claim that 'the proposed hybrid importance reweighting (HIR) method ... consistently improves over random selection.' The hybrid model has not actually been tested, and the contribution that distinguishes HIR from DSIR (the weighted combination of n-gram and neural features) is never evaluated. To support the paper's claims, the authors would need to run experiments with α in (0,1) and demonstrate the claimed improvements under matched pretraining budgets.","section":"Section 3.2 and Section 6"}],"minor_comments":[{"comment":"The text says 'we proposed to incorporate the neural network features'; the tense should be 'we propose' for consistency with the rest of the paper.","section":"Section 3.2"},{"comment":"The definition of γ is incomplete as printed: 'γ = 1Ps j=1 Pm i=1 zji Ps j=1 zj' is missing the denominator normalization. Please rewrite the estimator clearly and add equation numbers.","section":"Section 3.1"},{"comment":"The caption should state explicitly that the subscripts are standard deviations across the three random seeds, since the table otherwise reports numbers like '18.7310.22' that are easy to misread.","section":"Section 6, Table 2"},{"comment":"The pretraining section refers to hyperparameters from Xie et al. (2023) but does not specify the model architecture, number of parameters, optimizer, learning rate, or pretraining steps. Adding these details is necessary for reproducibility.","section":"Section 4"},{"comment":"The text describes 'resampling' with probabilities proportional to importance weights, but the implementation selects the top-k samples by weight. These are different selection procedures; the paper should clarify which was used and justify the choice, since top-k selection does not correspond to importance resampling.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The central claims are not supported by the reported experiments, and the main comparison is confounded by dataset-size and pool-size mismatches. The hybrid method itself is never evaluated. These issues require redoing the experimental design rather than a localized revision, so I recommend rejection. If the authors resubmit, they should compare methods under equal pretraining data budgets and evaluate the full hybrid weight α."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Readable short study. The core result as stated — HIR consistently beats random — is not supported by Table 2, where HIR is numerically below random on MRPC and QNLI. More fundamentally, the experimental comparison is not identifiable: HIR selects 47k documents from a 1.4M-doc pool, while DSIR and random use 1.7M documents from a 58M-doc pool. Replacing the first 47k DSIR documents with HIR's doesn't equalize size or source; any difference could come from data volume or pool composition.\n\nWhat is actually new: the specific test of a GMM fitted to SentenceTransformer embeddings as an importance resampler, compared on six GLUE tasks. That is a reasonable thing to try, and the paper is transparent that it only runs α=0, so the hybrid method is not actually evaluated. The authors also acknowledge the small scale and note the computational limits. That honesty earns some credit.\n\nThe main soft spots beyond the confound: the performance gaps are small and often within standard deviation, and the reported numbers don't support the 'consistent gains' language in the abstract. The method as described in Section 3.2 is never tested; α=0 is just GMM resampling. So the contribution is narrower than claimed.\n\nThe citation pattern is fine; DSIR is the right baseline, and the related work is adequate for a short report. This looks like a careful course project rather than a publication-ready paper. I would not cite it as evidence for neural feature selection. A serious referee would likely reject on the confound alone, so I would desk reject. If the authors re-run with matched subset sizes and pools, there might be a modest workshop paper.","headline":"The central claim is contradicted by the paper's own table, and the 47k-vs-1.7M confound makes the comparison unidentifiable; the work has a clear idea but is not ready for review.","tokens_in":8213,"tokens_out":2354,"would_cite":false,"duration_ms":21981,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that neural embedding-based data selection improves over random selection for pretraining, while n-gram-based DSIR remains stronger on most GLUE tasks.","keywords":["data selection","importance resampling","language model pretraining","distribution alignment","n-gram features","sentence embeddings","GLUE benchmark","hybrid importance resampling"],"falsifier":"Run the same continued-pretraining and GLUE evaluation with HIR ($\\alpha=0$) and DSIR each selecting exactly the same number of documents from the same raw pool, with no first-47,000 substitution and matched random seeds; if HIR no longer beats random selection on most tasks when sizes are equal, the consistency claim is refuted.","tokens_in":7201,"feed_emoji":"🎯","tokens_out":14258,"duration_ms":119638,"temperature":0.7,"pith_summary":"The paper asks whether the choice of pretraining data can be steered toward a target distribution and whether neural sentence embeddings add anything beyond n-gram statistics. It proposes HIR, Hybrid Importance Resampling, which forms a hybrid target distribution from an n-gram model and a Gaussian mixture model on sentence embeddings, with a parameter $\\alpha$ weighting the two. With $\\alpha=0$, HIR reduces to pure embedding-based selection; the paper claims this setting improves over random selection, and the reported table shows it ahead on four of six GLUE tasks after continued pretraining. The same table shows DSIR, the n-gram baseline, beating random on all six tasks and posting the best score on five of six. The intended conclusion is that token-level n-gram statistics remain the strongest single signal for pretraining data curation, while embedding-based selection offers a complementary, context-level signal.","feed_headline":"Neural-embedding selection improves pretraining; n-gram DSIR leads","feed_subtitle":"Sentence-embedding selection beats random on four tasks; n-gram selection wins five of six.","key_machinery":"The central object is the Hybrid Importance Resampling (HIR) weighting scheme, a hybrid importance weight $$\\$omega_i^{{\\mathrm{hybrid}}$} = \\left(\\frac{\\hat p_{\\mathrm{ng}}(x_i)}{\\hat q_{\\mathrm{ng}}(x_i)}\\right)^{\\$\\alpha$} \\left(\\frac{\\hat p_{\\mathrm{nn}}(x_i)}{\\hat q_{\\mathrm{nn}}(x_i)}\\right)^{1-\\$\\alpha$}$$ that combines hashed n-gram statistics with Gaussian-mixture density estimates over sentence embeddings. Setting $\\alpha=0$ makes the weight purely neural, the configuration tested in the report. This weight is what carries the argument: it is the mechanism by which raw documents are ranked and the top $k$ are selected, and it is what lets the paper attribute performance differences to the choice of feature type.","core_discovery":"The central claim is that distribution-matched data selection improves language model pretraining, and that n-gram and neural embedding features capture complementary aspects of the target distribution. Formally, HIR estimates $\\hat p_{\\text{ngram}}$ with a bag-of-hashed-n-grams model and $\\hat p_{\\text{nn}}$ with a diagonal Gaussian mixture model on 384-dimensional sentence embeddings, then defines the hybrid target $\\hat p_{\\text{hybrid}}(x)=\\hat p_{\\text{ngram}}(x)^{\\alpha}\\,\\hat p_{\\text{nn}}(x)^{1-\\alpha}$ and resamples raw documents with probability proportional to $\\hat p_{\\text{hybrid}}(x)/\\hat q_{\\text{hybrid}}(x)$. Setting $\\alpha=0$ isolates the neural signal. The paper argues that this neural-only variant aligns the raw data with the target at a broader contextual level and therefore improves over random selection, while DSIR's token-level n-gram weights are more directly aligned with masked-token pretraining and win five of the six GLUE tasks.","pith_inferences":["The paper tests only the pure neural regime ($\\alpha=0$); an intermediate $\\alpha$ that lets n-gram and neural weights trade off is the natural next experiment and is explicitly left as future work.","The paper's discussion attributes HIR's underperformance partly to imprecise density estimates from a diagonal GMM in a high-dimensional embedding space; replacing the GMM with a more expressive density estimator is a direct way to test whether the neural feature type or its estimation is the bottleneck.","Because the reported comparison substitutes HIR's 47,000 documents into the first 47,000 positions of DSIR's 1.7 million-document set, an equal-size, same-pool comparison would separate the method's effect from the effect of which documents were swapped in."],"forward_implications":["Distribution-matched selection improves downstream GLUE performance relative to random selection: DSIR beats random on all six tasks, and HIR beats random on four.","DSIR's n-gram importance weights win five of six tasks, so token-level statistics remain the strongest single feature type for pretraining data curation.","HIR with $\\alpha=0$ wins on STS-B and beats random on four tasks without using n-gram features, so sentence-embedding alignment is a viable complementary signal.","Only $\\alpha=0$ was tested; tuning $\\alpha$ is the paper's stated next step for combining n-gram and neural signals."],"supporting_citations":[{"why":"Supplies the DSIR baseline, its hashed n-gram importance-resampling procedure, and the hyperparameter settings reused for pretraining and evaluation.","marker":"(Xie et al., 2023)"},{"why":"Supplies the sentence-transformer embeddings that HIR fits with Gaussian mixture models.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Defines the GLUE benchmark and the six tasks used to measure downstream performance.","marker":"(Wang, 2018)"},{"why":"Frames data selection as the problem of matching a large raw corpus to a smaller target distribution.","marker":"(Albalak et al., 2024)"},{"why":"Provides the heuristic classifier-based filtering approach that the distribution-matching methods are contrasted with.","marker":"(Brown et al., 2020)"}],"fun_headline_variants":["Neural embeddings beat random data selection for pretraining","Hybrid data selection: n-grams win tasks, embeddings win baseline","Data selection: n-gram DSIR tops GLUE, neural beats random","N-gram selection wins GLUE, neural selection beats random"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that substituting the first 47,000 documents of DSIR's 1.7 million-document selection with HIR's 47,000 documents keeps the two training sets comparable, so that any performance difference is due to the selection method rather than to dataset size or to which documents were swapped in.","fun_headline_variants_meta":{"raw":{"variants":["Neural embeddings beat random data selection for pretraining","Hybrid data selection: n-grams win tasks, embeddings win baseline","Data selection: n-gram DSIR tops GLUE, neural beats random","N-gram selection wins GLUE, neural selection beats random"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2810,"prompt_tokens":869,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1867}},"tokens_in":485,"tokens_out":1941,"duration_ms":12197,"temperature":1.0,"reasoning_tokens":1867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:45:37.001556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same continued-pretraining and GLUE evaluation with HIR ($\\alpha=0$) and DSIR each selecting exactly the same number of documents from the same raw pool, with no first-47,000 substitution and matched random seeds; if HIR no longer beats random selection on most tasks when sizes are equal, the consistency claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DSIR baseline, its hashed n-gram importance-resampling procedure, and the hyperparameter settings reused for pretraining and evaluation."}],"review_version":1}