{"id":"28af4494-cd9d-404f-99e4-fcbf1b5ab1ea","arxiv_id":"2507.03253","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A deletion-only program refiner, trained on expert end-to-end edits converted via minimum edit distance, improves LLM pretraining data and downstream accuracy.","lead":"RefineX cleans messy internet text for AI training by teaching a small model to delete only the low-quality parts, using edit programs distilled from a larger expert model. In pretraining tests at 350M and 750M scale, it beats raw, filtered, and prior program-based refinement baselines by 2.6-7.2% on average.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed-token-budget comparison is confounded by document count: RefineX removes ~37% of tokens (Fig. 4), so its 20B-token corpus contains more documents than baselines; no length-matched or random-deletion control exists, so gains may come from document diversity rather than learned deletion quality.","rationale":"The reader's weakest_assumption concerns whether deletion-only operations preserve the quality gain of full expert rewrites. That is related but not the decisive issue. The decisive issue is that the experimental protocol cannot distinguish learned deletion quality from document-quantity effects, because all comparisons fix total tokens, not documents, and RefineX shortens documents far more than baselines. I agree with the CONDITIONAL verdict, but for a sharper reason: without a control that deletes the same amount of content heuristically or randomly, the central claim that expert-guided learned deletions are the operative cause is unsupported. The paper deserves credit for its honest limitations section, detailed appendices, consistent gains at 350M and 750M, and the deletion-only property that introduces no new words. However, the DataMan-based quality analysis in Section 4.3 is not a sufficient substitute: DataMan shares two authors with this paper (Junyang Lin and Dayiheng Liu), and quality-score improvements on sampled instances do not establish downstream pretraining gains. The requested control is expensive but decisive, and it directly tests whether the mechanism claimed in the abstract actually drives the reported results.","tokens_in":37384,"tokens_out":6821,"duration_ms":93596,"concrete_test":"Run a matched-deletion control on the same source documents RefineX used: for each document, delete exactly the same number of characters as RefineX did, using (a) uniformly random span deletion and (b) a simple line-level heuristic (e.g., remove lines containing URLs or ad keywords). Train the same 750M LLaMA-2 model on the first 20B tokens of each control corpus and evaluate on the same LightEval tasks. If either control lands within ~0.3 average points of RefineX's 44.7 on Prox-D-filtered data, the gains are not attributable to learned deletion selection; if both are 1+ points lower, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2 compares all methods under a fixed 20B-token budget, but not under a fixed document budget. RefineX reduces average document length far more than Prox-C or the filters (Fig. 4: raw 1509→948 tokens; Comb 1857→1144; Prox-D 2473→1528). Consequently, to assemble 20B tokens, RefineX draws from substantially more source documents than the baseline corpora. The paper even advertises this as \"access to a more diverse set of documents\" (Section 4.2), but it never separates the effect of learned deletion quality from the effect of simply fitting more documents into the same token budget. The central claim that expert-guided deletion-only programs drive the 2.6–7.2% gains therefore rests on an untested comparison: any method that aggressively shortens documents, such as truncating each document to the RefineX output length or deleting the same number of characters by a simple heuristic, would also enlarge the document count and could reproduce most of the gain. This is most acute in the Prox-D row of Table 2, where RefineX's token reduction is largest (2473→1528, ~38%) and its margin over Prox-C is largest (+1.2 average points). The absence of a length-matched or deletion-matched control is a load-bearing gap, not a stylistic one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RefineX, a pipeline that refines pretraining corpora by training a 0.6B model to emit deletion-only programmatic edits. Distillation data is created by prompting Qwen2.5-72B to rewrite documents, computing minimal edit-distance operations between original and rewritten text, discarding insertions and replacements, and converting the surviving deletions into a small set of Python functions (remove_lines, remove_str, keep_all). RefineX is applied at scale to RedPajama subsets and compared against raw, rule-filtered, Prox-D, and Prox-C corpora under a fixed 20B-token budget, with 350M and 750M LLMs evaluated on 10 LightEval tasks. The paper reports consistent average improvements, with 750M gains of 2.6%–7.2% over baselines, and additional instance-level quality analysis using DataMan.","tokens_in":37709,"tokens_out":7610,"duration_ms":88168,"significance":"The core idea is timely and the execution is substantial: the paper provides full checkpoint-level results at two model scales, transparent algorithm descriptions, extensive appendix tables, and an external LightEval evaluation that is independent of the refinement pipeline. If the causal attribution is established, RefineX would be a practical contribution showing that distillation from end-to-end edits into deletion-only programs can improve pretraining data at low inference cost. However, the fixed-token-budget comparison does not isolate learned deletion quality from the increased document count caused by token removal, the instance-level quality metric is not independent of the training signal, and all pretraining runs are single-seed with no variance estimates. These issues are fixable with additional controls, so the contribution is promising but not yet fully established.","major_comments":[{"comment":"The fixed-token-budget comparison is confounded by document count. RefineX's deletion-only programs shorten documents dramatically (Fig. 4: raw 1509→948 tokens, Comb 1857→1144, Prox-D 2473→1528), so assembling a 20B-token corpus with RefineX requires drawing from substantially more source documents than the baseline corpora. The paper even describes this as \"access to a more diverse set of documents\" (Section 4.2). Without a length-matched control (e.g., truncating each document to the RefineX output length, or deleting the same number of characters with a random or simple rule-based heuristic) or a matched-document-count comparison, the average gains in Table 2 cannot be attributed to learned deletion quality rather than to the larger document count under the same token budget. This gap is load-bearing for the central claim of 2.6%–7.2% gains.","section":"§4.2, Table 2, Fig. 4, Appendix B"},{"comment":"The text-quality evaluation is not independent of the training signal. Appendix A.1 uses DataMan scores to sample the seed data for distillation, and Section 4.3 uses DataMan to measure the quality improvement of refined text. Moreover, DataMan's author list (Peng et al., 2025) includes two co-authors of this paper (Junyang Lin and Dayiheng Liu). The claim that RefineX \"effectively improves text quality\" (Table 3) is therefore assessed with the same model that shaped the distillation data, and the potential author-related bias is not disclosed. An external quality metric or a held-out quality test set not used in pipeline construction would be needed.","section":"§4.3, Table 3, Appendix A.1"},{"comment":"All pretraining comparisons are single-run and no variance estimates are reported. Many differences between methods are small (for example, 42.9 vs. 42.4 on Raw, and 43.2 vs. 42.6 on Gopher in Table 2), so it is not possible to assess whether the reported gains are statistically meaningful. At minimum, the authors should report standard deviations across evaluation subsets or multiple seeds, or provide a bootstrap interval for the average difference.","section":"§4.2, Tables 2 and 8"},{"comment":"The key design assumption, that deletion-only operations extracted from minimal edit distance preserve enough of the expert rewrite's benefit, is not directly validated. Insertions and replacements are discarded by construction, and the resulting deletion-only text is never checked against the expert output. The hand-set filtering thresholds in Appendix A.1 (insertion/replacement spans under 20 characters, deletions under 10 characters) are not a validation of this assumption. A control experiment that applies random deletions matched in number and length, or that compares against the full end-to-end rewrite on the same document budget, would clarify whether the downstream gains come from the deletion supervision or simply from token removal.","section":"§3.2, Eq. (2), Appendix A.1, Table 14"}],"minor_comments":[{"comment":"The method name is inconsistently typeset as \"REFINE X\" and \"REFINEX\"; one consistent form should be used throughout, including tables and figure captions.","section":"Global"},{"comment":"The legend label \"Prox-D + C\" is ambiguous and should be \"Prox-D + Prox-C\" for consistency with Table 2.","section":"Figure 3"},{"comment":"The text says \"The final reported average is computed over all nine benchmarks\" but Table 2 reports ten tasks; please clarify whether the average is over the nine early-signal benchmarks plus SciQ or over the ten listed tasks.","section":"Appendix C"},{"comment":"The notation (c'_i)|t|_i=1 is awkward; it would be clearer to define c'_i for i = 1 to |t| in a separate display.","section":"Eq. (1)"},{"comment":"The Empty column is high for low-score documents (68.17% for Score=1 with the 0.6B model); the authors should discuss the implications of converting many low-quality documents into empty strings, since this effectively combines in-document refinement with document-level filtering.","section":"Table 6 and Table 9"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk is the fixed-token-budget confound between token deletion and document count, which directly affects the headline 2.6%–7.2% claim. The DataMan author overlap should also be disclosed, and the single-seed nature of the pretraining comparisons should be addressed. If the authors add length-matched or random-deletion controls, provide variance estimates, and use an independent quality metric, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RefineX is a real, incremental step beyond ProX, and the experimental sweep is admirably broad. But the headline claim is currently underdetermined: because RefineX deletes substantially more tokens per document than the baselines, the 20B-token corpora contain different numbers of documents, and no control separates the effect of learned deletion quality from the effect of document diversity. That is the thing I'd want fixed before trusting the numbers.\n\nThe two-stage distillation idea is genuinely new: get an expert to write an end-to-end refined version, then run minimum edit distance to extract deletion-only operations, filter those, and train a 0.6B refiner to emit simple programs. That is a clean way to get supervision without burdening the expert with program generation, and the resulting refiner is cheap to run. The paper also deserves credit for evaluating on five data settings across two model sizes with an independent downstream benchmark (LightEval), and for being upfront about compute limits and expert-model dependence.\n\nThe main soft spot is a confound in the fixed-token-budget comparison. Figure 4 shows RefineX pulls average document length down from ~1500 to ~950 tokens on raw data; to fill 20B tokens it therefore draws on many more documents than the baselines, and the paper itself advertises 'access to a more diverse set of documents.' That means the 2.6-7.2% gains may come from seeing more documents at the same token budget, not from deletion quality. A control that truncates or randomly deletes documents to RefineX's length distribution is needed to make the central claim. Lacking that, the claim should be softened.\n\nOther issues are smaller. Runs are single-seed, with no variance estimates; the average margins are a few points and 'consistently' overstates what a 0.6-1.2 point spread shows. The text-quality metric DataMan is authored by the same group, and it is used both to sample the distillation data and to evaluate refined text, which is a circularity risk; the LightEval results are independent and are what actually carry the argument. The #Win counts also show small inconsistencies (e.g., the Prox-D + Prox-C row is 0/10 in Table 2 despite solid scores), which suggests some ties are being resolved in ways that aren't fully explained.\n\nWho is this for? Researchers and engineers working on LLM pretraining data curation, especially those building on program-based refinement. It deserves a serious referee: the method is clear, the experiments are large, and the missing control is exactly what peer review should catch. My recommendation is to send it out, with a request for the control and ideally multiple seeds.","headline":"RefineX is a genuine incremental extension of ProX with a clean two-stage distillation, but a missing length-matched control leaves its headline gains confounded with document count.","tokens_in":38243,"tokens_out":4457,"would_cite":true,"duration_ms":50783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small deletion-only refiner trained on expert rewrites improves LLM pretraining corpora more than filtering alone.","keywords":["pretraining data refinement","programmatic editing","deletion-only refinement","minimum edit distance","data distillation","LLM pretraining","data quality","RedPajama"],"falsifier":"A reader could settle the central claim by pretraining two equal-budget models on corpora produced by RefineX's distilled delete-only programs versus corpora produced by faithfully applying the full minimal-edit operation set (insertions and replacements included); if the full-edit variant wins, the deletion-only restriction is not the source of the reported gains.","tokens_in":37219,"feed_emoji":"✂️","tokens_out":6924,"duration_ms":73038,"temperature":0.7,"pith_summary":"RefineX claims that the costly step of having a powerful expert LLM rewrite noisy web text can be compressed into a cheap, small refiner that only deletes content. The paper argues that insertions and replacements mostly reflect the expert model's stylistic preferences, so discarding them avoids over-editing and model bias while keeping most of the quality gain. On a fixed 20-billion-token budget, models pretrained from scratch on RefineX-refined corpora beat models trained on raw, rule-filtered, and previously refined data across ten downstream tasks, with the largest reported gains of 2.6 to 7.2 percent at 750M scale. If true, this turns pretraining-data refinement into a surgical, nearly free operation that can be applied to every document in a corpus rather than a document-level filter.","feed_headline":"Deletion-only edits lift LLM pretraining by up to 7.2 percent","feed_subtitle":"A tiny distilled model learns expert delete programs, improving data quality with zero new words.","key_machinery":"The load-bearing object is the two-stage distillation pipeline. In stage one, expert end-to-end refinement produces a target text; in stage two, minimum edit distance (Levenshtein distance) decomposes the difference between original and target into insertions, replacements, and deletions, and only deletion spans survive. Those spans are mapped to a minimal function set, namely remove_lines(start_line, end_line), remove_str(line, del_str), and keep_all(), which the paper designs to stay token-cheap so the small refiner can predict them reliably. The mechanism turns an expensive free-form rewrite into a sparse, executable program and, by construction, guarantees that the output is always a subsequence of the input, which is what enforces authenticity and prevents hallucinated new content.","core_discovery":"The central claim is that expert-guided data refinement can be distilled into deletion-only edit programs without losing most of its value. The authors have an expert model rewrite web documents end to end, align the original and rewritten text with minimum edit distance, discard all insertions and replacements, convert the remaining deletion spans into three compact program functions, and train a 0.6-billion-parameter model to emit those programs. The trained refiner runs at corpus scale, leaving untouched text it judges clean and removing only junk spans. The paper reports that this pipeline consistently outperforms prior program-based refinement and all tested filtering baselines when used to pretrain 350M- and 750M-parameter models for 20B tokens, and that 10B tokens of its refined data match or exceed 20B tokens of one filtered baseline, so the contribution is the distillation recipe rather than the program executor.","pith_inferences":["Beyond the paper: the pipeline's ceiling is tied to the expert model, because if a stronger expert mostly rephrases rather than deletes, the deletion-only extraction may throw away the useful part of the supervision signal.","Beyond the paper: a testable extension would compare RefineX against a variant that also keeps short insertions and replacements with execution-safety checks, to see whether the authenticity guarantee is worth the lost edit types.","Beyond the paper: because the refined corpus is a strict token subset of the raw corpus, per-document token counts shrink predictably, which could make deduplication and token-budget planning for large pretraining runs easier.","Beyond the paper: if these gains replicate at billion-parameter scale, the method points toward programmatic data cleaning, in which data quality is controlled by executable, inspectable edits rather than by model judgment."],"forward_implications":["On a 750M model, RefineX-refined corpora raise average LightEval scores by 2.6 to 7.2 percent over the previous program-based refiner and over filtering baselines at the same 20B-token budget.","A model trained on 10B tokens of RefineX-refined data matches or exceeds a model trained on 20B tokens of a rule-filtered corpus, implying better data efficiency under fixed compute.","RefineX improves text quality within documents across every quality-score group it is applied to, and it introduces zero words absent from the original text, so raw-text diversity is preserved.","RefineX stacks on top of both rule-based and LLM-based document filters, making it a complement to existing filtering rather than a replacement.","Because the refiner only deletes, minor spelling and grammar imperfections remain in the data, and the paper reports downstream gains anyway, suggesting that removing junk matters more than polishing surface errors at this scale."],"supporting_citations":[{"why":"Introduces ProX, the prior program-based refinement pipeline that RefineX builds on and compares against as the strongest fine-grained baseline.","marker":"Zhou et al. 2024"},{"why":"Provides RedPajama-V2, the web corpus from which all raw, filtered, and refined pretraining sets are drawn.","marker":"Together, 2023"},{"why":"LightEval is the evaluation harness used to score the ten downstream tasks.","marker":"Fourrier et al., 2023"},{"why":"DataMan quality scores drive seed-data sampling for distillation and the per-instance text-quality evaluation.","marker":"Peng et al., 2025"},{"why":"Supplies the expert LLM used to generate end-to-end refined text for distillation supervision and the base model for the refiner.","marker":"Yang et al., 2025"},{"why":"Defines the FineWeb filtering baseline and the early-signal benchmark set that shapes the evaluation task selection.","marker":"Penedo et al., 2024a"},{"why":"Provides the normalized Levenshtein distance algorithm used to extract minimal edit operations between raw and expert-refined text.","marker":"Yujian & Bo, 2007"},{"why":"Specifies the LLaMA-2 architecture used for the 350M and 750M pretrained evaluation models.","marker":"Touvron et al., 2023b"}],"fun_headline_variants":["Delete-only edits boost LLM pretraining up to 7.2%","Surgical deletion refines LLM pretraining data","Mini refiner deletes junk to boost LLM pretraining","Deletion-only refinement: 7.2% better LLMs","Tiny refiner learns to delete for better LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the value of an expert rewrite is mostly carried by the text it would delete, so a deletion-only program extracted from the rewrite can stand in for the rewrite itself.","fun_headline_variants_meta":{"raw":{"variants":["Delete-only edits boost LLM pretraining up to 7.2%","Surgical deletion refines LLM pretraining data","Mini refiner deletes junk to boost LLM pretraining","Deletion-only refinement: 7.2% better LLMs","Tiny refiner learns to delete for better LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001208,"raw_usage":{"total_tokens":5016,"prompt_tokens":1024,"completion_tokens":3992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3905}},"tokens_in":640,"tokens_out":3992,"duration_ms":31527,"temperature":1.0,"reasoning_tokens":3905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:14:36.370015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the central claim by pretraining two equal-budget models on corpora produced by RefineX's distilled delete-only programs versus corpora produced by faithfully applying the full minimal-edit operation set (insertions and replacements included); if the full-edit variant wins, the deletion-only restriction is not the source of the reported gains.","supporting_citations":[{"cited_title":"Redpajama: an open dataset for training large language models, October 2023","cited_arxiv_id":null,"evidence_quote":"Provides RedPajama-V2, the web corpus from which all raw, filtered, and refined pretraining sets are drawn."},{"cited_title":"Lighteval: A lightweight framework for llm evaluation, 2023","cited_arxiv_id":null,"evidence_quote":"LightEval is the evaluation harness used to score the ten downstream tasks."},{"cited_title":"A normalized levenshtein distance metric","cited_arxiv_id":null,"evidence_quote":"Provides the normalized Levenshtein distance algorithm used to extract minimal edit operations between raw and expert-refined text."}],"review_version":1}