Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small deletion-only refiner trained on expert rewrites improves LLM pretraining corpora more than filtering alone.

desk verdict RefineX is a genuine incremental extension of ProX with a clean two-stage distillation, but a missing length-matched control leaves its headline gains confounded with document count. read the letter →

arxiv 2507.03253 v2 pith:L4U62CEJ submitted 2025-07-04 cs.CL cs.AI

Baolong Bi , Shenghua Liu , Xingzhang Ren , Dayiheng Liu , Junyang Lin , Yiwei Wang , Lingrui Mei , Junfeng Fang
show 2 more authors
Jiafeng Guo Xueqi Cheng
This is my paper
classification cs.CLcs.AI
keywords pretrainingdatarefinementprogrammaticeditingdeletion-onlyminimumeditdistancedistillationLLMqualityRedPajama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RefineX claims that the costly step of having a powerful expert LLM rewrite noisy web text can be compressed into a cheap, small refiner that only deletes content. The paper argues that insertions and replacements mostly reflect the expert model's stylistic preferences, so discarding them avoids over-editing and model bias while keeping most of the quality gain. On a fixed 20-billion-token budget, models pretrained from scratch on RefineX-refined corpora beat models trained on raw, rule-filtered, and previously refined data across ten downstream tasks, with the largest reported gains of 2.6 to 7.2 percent at 750M scale. If true, this turns pretraining-data refinement into a surgical, nearly free operation that can be applied to every document in a corpus rather than a document-level filter.

What carries the argument

The load-bearing object is the two-stage distillation pipeline. In stage one, expert end-to-end refinement produces a target text; in stage two, minimum edit distance (Levenshtein distance) decomposes the difference between original and target into insertions, replacements, and deletions, and only deletion spans survive. Those spans are mapped to a minimal function set, namely remove_lines(start_line, end_line), remove_str(line, del_str), and keep_all(), which the paper designs to stay token-cheap so the small refiner can predict them reliably. The mechanism turns an expensive free-form rewrite into a sparse, executable program and, by construction, guarantees that the output is always a subsequence of the input, which is what enforces authenticity and prevents hallucinated new content.

What would settle it

A reader could settle the central claim by pretraining two equal-budget models on corpora produced by RefineX's distilled delete-only programs versus corpora produced by faithfully applying the full minimal-edit operation set (insertions and replacements included); if the full-edit variant wins, the deletion-only restriction is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that expert-guided data refinement can be distilled into deletion-only edit programs without losing most of its value. The authors have an expert model rewrite web documents end to end, align the original and rewritten text with minimum edit distance, discard all insertions and replacements, convert the remaining deletion spans into three compact program functions, and train a 0.6-billion-parameter model to emit those programs. The trained refiner runs at corpus scale, leaving untouched text it judges clean and removing only junk spans. The paper reports that this pipeline consistently outperforms prior program-based refinement and all tested filtering baselines when used to pretrain 350M- and 750M-parameter models for 20B tokens, and that 10B tokens of its refined data match or exceed 20B tokens of one filtered baseline, so the contribution is the distillation recipe rather than the program executor.

Load-bearing premise

The load-bearing premise is that the value of an expert rewrite is mostly carried by the text it would delete, so a deletion-only program extracted from the rewrite can stand in for the rewrite itself.

Editorial extensions

If this is right

  • On a 750M model, RefineX-refined corpora raise average LightEval scores by 2.6 to 7.2 percent over the previous program-based refiner and over filtering baselines at the same 20B-token budget.
  • A model trained on 10B tokens of RefineX-refined data matches or exceeds a model trained on 20B tokens of a rule-filtered corpus, implying better data efficiency under fixed compute.
  • RefineX improves text quality within documents across every quality-score group it is applied to, and it introduces zero words absent from the original text, so raw-text diversity is preserved.
  • RefineX stacks on top of both rule-based and LLM-based document filters, making it a complement to existing filtering rather than a replacement.
  • Because the refiner only deletes, minor spelling and grammar imperfections remain in the data, and the paper reports downstream gains anyway, suggesting that removing junk matters more than polishing surface errors at this scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the pipeline's ceiling is tied to the expert model, because if a stronger expert mostly rephrases rather than deletes, the deletion-only extraction may throw away the useful part of the supervision signal.
  • Beyond the paper: a testable extension would compare RefineX against a variant that also keeps short insertions and replacements with execution-safety checks, to see whether the authenticity guarantee is worth the lost edit types.
  • Beyond the paper: because the refined corpus is a strict token subset of the raw corpus, per-document token counts shrink predictably, which could make deduplication and token-budget planning for large pretraining runs easier.
  • Beyond the paper: if these gains replicate at billion-parameter scale, the method points toward programmatic data cleaning, in which data quality is controlled by executable, inspectable edits rather than by model judgment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RefineX, a pipeline that refines pretraining corpora by training a 0.6B model to emit deletion-only programmatic edits. Distillation data is created by prompting Qwen2.5-72B to rewrite documents, computing minimal edit-distance operations between original and rewritten text, discarding insertions and replacements, and converting the surviving deletions into a small set of Python functions (remove_lines, remove_str, keep_all). RefineX is applied at scale to RedPajama subsets and compared against raw, rule-filtered, Prox-D, and Prox-C corpora under a fixed 20B-token budget, with 350M and 750M LLMs evaluated on 10 LightEval tasks. The paper reports consistent average improvements, with 750M gains of 2.6%–7.2% over baselines, and additional instance-level quality analysis using DataMan.

Significance. The core idea is timely and the execution is substantial: the paper provides full checkpoint-level results at two model scales, transparent algorithm descriptions, extensive appendix tables, and an external LightEval evaluation that is independent of the refinement pipeline. If the causal attribution is established, RefineX would be a practical contribution showing that distillation from end-to-end edits into deletion-only programs can improve pretraining data at low inference cost. However, the fixed-token-budget comparison does not isolate learned deletion quality from the increased document count caused by token removal, the instance-level quality metric is not independent of the training signal, and all pretraining runs are single-seed with no variance estimates. These issues are fixable with additional controls, so the contribution is promising but not yet fully established.

major comments (4)
  1. [§4.2, Table 2, Fig. 4, Appendix B] The fixed-token-budget comparison is confounded by document count. RefineX's deletion-only programs shorten documents dramatically (Fig. 4: raw 1509→948 tokens, Comb 1857→1144, Prox-D 2473→1528), so assembling a 20B-token corpus with RefineX requires drawing from substantially more source documents than the baseline corpora. The paper even describes this as "access to a more diverse set of documents" (Section 4.2). Without a length-matched control (e.g., truncating each document to the RefineX output length, or deleting the same number of characters with a random or simple rule-based heuristic) or a matched-document-count comparison, the average gains in Table 2 cannot be attributed to learned deletion quality rather than to the larger document count under the same token budget. This gap is load-bearing for the central claim of 2.6%–7.2% gains.
  2. [§4.3, Table 3, Appendix A.1] The text-quality evaluation is not independent of the training signal. Appendix A.1 uses DataMan scores to sample the seed data for distillation, and Section 4.3 uses DataMan to measure the quality improvement of refined text. Moreover, DataMan's author list (Peng et al., 2025) includes two co-authors of this paper (Junyang Lin and Dayiheng Liu). The claim that RefineX "effectively improves text quality" (Table 3) is therefore assessed with the same model that shaped the distillation data, and the potential author-related bias is not disclosed. An external quality metric or a held-out quality test set not used in pipeline construction would be needed.
  3. [§4.2, Tables 2 and 8] All pretraining comparisons are single-run and no variance estimates are reported. Many differences between methods are small (for example, 42.9 vs. 42.4 on Raw, and 43.2 vs. 42.6 on Gopher in Table 2), so it is not possible to assess whether the reported gains are statistically meaningful. At minimum, the authors should report standard deviations across evaluation subsets or multiple seeds, or provide a bootstrap interval for the average difference.
  4. [§3.2, Eq. (2), Appendix A.1, Table 14] The key design assumption, that deletion-only operations extracted from minimal edit distance preserve enough of the expert rewrite's benefit, is not directly validated. Insertions and replacements are discarded by construction, and the resulting deletion-only text is never checked against the expert output. The hand-set filtering thresholds in Appendix A.1 (insertion/replacement spans under 20 characters, deletions under 10 characters) are not a validation of this assumption. A control experiment that applies random deletions matched in number and length, or that compares against the full end-to-end rewrite on the same document budget, would clarify whether the downstream gains come from the deletion supervision or simply from token removal.
minor comments (5)
  1. [Global] The method name is inconsistently typeset as "REFINE X" and "REFINEX"; one consistent form should be used throughout, including tables and figure captions.
  2. [Figure 3] The legend label "Prox-D + C" is ambiguous and should be "Prox-D + Prox-C" for consistency with Table 2.
  3. [Appendix C] The text says "The final reported average is computed over all nine benchmarks" but Table 2 reports ten tasks; please clarify whether the average is over the nine early-signal benchmarks plus SciQ or over the ten listed tasks.
  4. [Eq. (1)] The notation (c'_i)|t|_i=1 is awkward; it would be clearer to define c'_i for i = 1 to |t| in a separate display.
  5. [Table 6 and Table 9] The Empty column is high for low-score documents (68.17% for Score=1 with the 0.6B model); the authors should discuss the implications of converting many low-quality documents into empty strings, since this effectively combines in-document refinement with document-level filtering.

Circularity Check

3 steps flagged · score 4.0 of 10

The central LightEval pretraining claim is externally grounded, but the text-quality evidence is partially self-referential: DataMan (co-authored) is used both to sample the distillation seed and to measure quality, the 'zero new words' result is definitional, and the comparison against E2Edel is favorably structured by the filtering of the supervision data.

  1. self definitional [Section 3.1, Eq. (1); Section 4.3, Table 4]
    "E(Odel, t) = (c′i)|t| i=1, where c′i = ‘’ if ci in (dj)|Odel| j=1 else ci ... REFINE X applies a moderate level of edits that enhance text quality, while ensuring reliability by relying solely on deletion operations. As shown in Table 4, it introduces no additional content (i.e., zero new words), thereby avoiding risks of hallucination or over-modification."

    Eq. (1) defines the refined text as the original character sequence with selected spans removed. Therefore 'zero new words' is entailed by the definition of the executor, not an empirically discovered property. The paper presents Table 4 as reliability evidence, but the result is true by construction for any deletion-only operator and does not distinguish RefineX from any other deletion-only method.

  2. self citation load bearing [Appendix A.1; Section 4.3; Appendix C]
    "To construct the seed dataset, we begin by scoring the raw collected corpus using DataMan (Peng et al., 2025), a state-of-the-art quality scoring model. To ensure that the seed data reflect the true distribution of the full corpus, we sample approximately 5 million documents in accordance with the score distribution. ... we begin by pre-classifying raw text data collected from RedPajama-V2 using DataMan (Peng et al., 2025)"

    DataMan is cited from Peng et al. (2025), whose author list includes RefineX co-authors Dayiheng Liu and Junyang Lin. The same DataMan scorer is used both to construct the distillation seed set (Appendix A.1) and to evaluate the quality improvement claims (Section 4.3, Tables 3 and 9). The 'quality improvement' evidence is therefore not fully independent of the authors' own prior tool, although the central downstream LightEval results do not depend on DataMan and remain externally grounded.

1 more flagged steps
  1. fitted input called prediction [Appendix A.1; Section 4.3, Table 3]
    "We discard any document containing insertion or replacement spans of 20 characters or more, and further filter out examples with fewer than 10 characters affected by deletions. ... A particularly encouraging result is that REFINEX significantly outperforms its own distillation source E2Edel, used as supervision during training. This suggests that the refine model not only learned the deletion-based programs effectively but also generalized beyond the examples it was distilled from."

    The supervision used to train RefineX is not unfiltered E2Edel but a hand-filtered subset of deletion programs, with short deletions and long insertions/replacements removed (Appendix A.1). Comparing the resulting model against full E2Edel is therefore favorable by construction: the training data was deliberately curated to exclude the operations the authors deem low-confidence. The claim that RefineX 'generalized beyond' its distillation source is not supported by this comparison, since the observed superiority over E2Edel is at least partly an artifact of the filtering choices rather than evidence of emergent behavior.

full rationale

The paper's main empirical claim, that RefineX yields 2.6%-7.2% average gains over baselines on LightEval tasks for a 750M model, is anchored in external, independently run downstream benchmarks (LightEval tasks: ARC-C, ARC-E, CSQA, HellaSwag, MMLU, OBQA, PIQA, SIQA, WinoG, SciQ). These results are not fitted to RefineX's own outputs and do not reduce to the method's definitions, so the central pretraining claim is not circular. However, the supporting text-quality narrative contains genuine self-referential elements. First, DataMan, a quality scorer co-authored by two RefineX authors, is used both to sample the distillation seed distribution and to measure the quality improvements in Section 4.3; this makes the quality-improvement evidence partially dependent on the authors' own prior tool, even though DataMan is an external published model and is not re-fit in this paper. Second, the 'zero new words' reliability result is a direct logical consequence of the deletion-only executor in Eq. (1), so presenting it as an evaluation finding is definitional rather than empirical. Third, the claim that RefineX outperforms its own distillation source E2Edel is confounded by the fact that the distillation data is a filtered subset of E2Edel with short deletions and long insertions/replacements removed; the comparison is therefore stacked in RefineX's favor. These issues are confined to the text-quality analysis and do not invalidate the independent LightEval comparison, so the overall circularity score is moderate rather than severe.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumptions that deletion-only edits capture the useful part of expert rewrites, that DataMan scores measure pretraining quality, and that small-scale LightEval results transfer to larger pretraining runs. The method itself adds no new physical or mathematical entities; its free parameters are hand-set filtering thresholds and chunk sizes.

free parameters (3)
  • distillation filtering thresholds = insertion/replacement >= 20 chars discarded; deletions < 10 chars discarded
    Hand-chosen cutoffs in Appendix A.1 that decide which expert E2E edits become supervision; they directly shape the refiner's behavior and are not derived from data.
  • chunk size for refinement inference = approximately 12,000 characters (described as 12k tokens in Section 3.2)
    Chosen window for splitting documents; larger than ProX's 1.5K tokens, affects context and program offset complexity (Section 3.3, Appendix A.3).
  • expert decoding parameters = top-p=0.8, top-k=20
    Decoding settings for Qwen2.5-72B end-to-end refinement (Appendix A.1); influence the distribution of edits distilled.
assumptions (5)
  • domain assumption Deletion-only refinement improves pretraining data while preserving diversity and naturalness (Section 3.1).
    The method forbids insertions and replacements by design; the paper argues spelling errors can remain because pretraining neutralizes them, but this is not empirically isolated from alternative edit policies.
  • domain assumption The minimal edit distance between raw and expert-refined text identifies the 'right' deletions (Section 3.2, Eq. 2).
    Levenshtein alignment minimizes edit count, not semantic quality; a different alignment could yield deletions that are just as valid or better.
  • domain assumption DataMan scores are a valid measure of pretraining utility (Appendix A.1, Section 4.3).
    DataMan is used both to sample distillation data and to evaluate quality gains, and its authors overlap with RefineX's author list, so the metric is not independently validated here.
  • domain assumption Downstream LightEval tasks (zero-shot accuracy on 10 benchmarks) reflect pretraining data quality (Section 4.1, Appendix C).
    Standard evaluation practice in this literature, but only 10 tasks at 350M-750M scale; conclusions may not transfer to larger models.
  • domain assumption The expert model Qwen2.5-72B-Instruct produces higher-quality text under the provided prompts (Section 3.2, Appendix A.2).
    No human validation of the E2E outputs is reported; quality is assumed from the model's reputation and DataMan scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs." pith.science (2026). https://pith.science/paper/L4U62CEJ

@misc{pith2026250703253,
  author       = {Pith},
  title        = {Pith review of: RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4U62CEJ}},
  note         = {Machine review of arXiv:2507.03253}
}
abstract

The foundational capabilities of large language models (LLMs) are deeply influenced by the quality of their pre-training corpora. However, enhancing data quality at scale remains a significant challenge, primarily due to the trade-off between refinement effectiveness and processing efficiency. While rule-based filtering remains the dominant paradigm, it typically operates at the document level and lacks the granularity needed to refine specific content within documents. Inspired by emerging work such as ProX, we propose $\textbf{RefineX}$, a novel framework for large-scale, surgical refinement of pre-training data through programmatic editing tasks. RefineX enables efficient and fine-grained data refinement while reliably preserving the diversity and naturalness of raw text. The core strength of RefineX lies in distilling high-quality, expert-guided end-to-end refinement results into minimal edit-based deletion programs. This high-precision distillation pipeline is used to train an efficient and reliable refine model that can systematically improve every instance in the corpus at scale. We evaluate RefineX across from-scratch pre-training at multiple model scales and find that it consistently outperforms models trained on raw, filtered, or alternatively refined data across diverse downstream tasks. On the 750M model, RefineX yields 2.6%-7.2% average gains on lighteval tasks, and achieves comparable performance using significantly fewer training tokens. Further analysis shows that RefineX reliably enhances text quality with both high efficiency and precision, outperforming prior approaches such as end-to-end generation and Prox-C. These results position RefineX as a scalable, effective, and reliable solution for optimizing pre-training data in modern LLM pipelines.

Figures

Figures reproduced from arXiv: 2507.03253 by the authors.

Figure 1
Figure 1. Overview of the program-based refinement pipeline and comparison of training data [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An overview of the REFINEX framework. (1) During training, REFINEX prompts an expert model to generate high-quality refined text under instructions, then extracts valid deletions via minimal edit distance. These are converted into program functions to supervise a reliable refine model. (2) At inference time, the trained model generates fine-grained refinement programs for each document, which are executed by a Pytho… view at source ↗
Figure 3
Figure 3. Downstream average performance (%) of model checkpoints with different num￾bers of training tokens during pretraining. We evaluate the effectiveness of data refinement by examining the performance of language models trained from scratch. Specifically, we begin by ap￾plying several document-level filtering methods, in￾cluding Gopher, C4, FW, COMB, and Prox-D, to the raw RedPajama-V2 corpus. The filtered datasets, alo… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Token count distributions before and after refinement on raw data, rule-based filtered data [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The top plot reflects the refinement efficiency, while the bottom plot shows the preference for leaving text untouched. From these results, we observe that while E2E deliv￾ers the largest quality gains, it is also prohibitively slow, particularly when inference depends…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.

Reference graph

Works this paper leans on

68 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Lightning AI. Litgpt. https://github.com/Lightning-AI/litgpt, 2023

    work page 2023

  3. [3]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

    work page 2024

  4. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Max Nye, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  5. [5]

    A critical analysis of the largest source for generative ai training data: Common crawl

    Stefan Baack. A critical analysis of the largest source for generative ai training data: Common crawl. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 2199--2208, 2024

    work page 2024

  6. [6]

    Parameters vs

    Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. Parameters vs. context: Fine-grained control of knowledge reliance in language models. arXiv preprint arXiv:2503.15888, 2025

    arXiv 2025

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  8. [8]

    On the resemblance and containment of documents

    Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp.\ 21--29. IEEE, 1997

    work page 1997

Show all 68 references
  1. [9]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  4. [12]

    Sailor: Open language models for south-east asia

    Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. Sailor: Open language models for south-east asia. CoRR, abs/2404.03608, 2024. doi:10.48550/ARXIV.2404.03608. URL https://doi.org/10.48550/arXiv.2404.03608

    2024 doi arXiv

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 0 18343...

    2022

  7. [15]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/lighteval

    2023

  8. [16]

    Incoder: A generative model for code infilling and synthesis

    Daniel Fried, Sewon Chen, Eric Wallace, et al. Incoder: A generative model for code infilling and synthesis. In arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  9. [17]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  10. [18]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    2023

  11. [19]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  12. [20]

    Rho-1: Not all tokens are what you need

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965, 2024

    2024 arXiv

  13. [21]

    Efficient inference for large reasoning models: A survey

    Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: A survey. arXiv preprint arXiv:2503.23077, 2025

    2025 arXiv

  14. [22]

    Openelm: An efficient language model family with open-source training and inference framework

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv preprint ar...

    2024 arXiv

  15. [23]

    Slang: New concept comprehension of large language models

    Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, and Xueqi Cheng. Slang: New concept comprehension of large language models. arXiv preprint arXiv:2401.12585, 2024

    2024 arXiv

  16. [24]

    Introducing meta llama 3: The most capable openly available llm to date, 2024

    Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai.meta.com/blog/meta-llama-3

    2024

  17. [25]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empiri...

    2018 doi

  18. [26]

    Towards fully exploiting llm internal states to enhance knowledge boundary perception

    Shiyu Ni, Keping Bi, Jiafeng Guo, Lulu Yu, Baolong Bi, and Xueqi Cheng. Towards fully exploiting llm internal states to enhance knowledge boundary perception. arXiv preprint arXiv:2502.11677, 2025

    2025 arXiv

  19. [27]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp.\ 1--22, 2023

    2023

  20. [28]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024 a

    2024 arXiv

  21. [29]

    Datatrove: large scale data processing, 2024 b

    Guilherme Penedo, Hynek Kydlíček, Alessandro Cappelli, Mario Sasko, and Thomas Wolf. Datatrove: large scale data processing, 2024 b . URL https://github.com/huggingface/datatrove

    2024

  22. [30]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Inf...

    2024

  23. [31]

    Dataman: Data manager for pre-training large language models

    Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. Dataman: Data manager for pre-training large language models. arXiv preprint arXiv:2502.19363, 2025

    2025 arXiv

  24. [32]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  25. [33]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  26. [34]

    Web data mining with organized contents using naive bayes algorithm

    Bathini Ravinder, Senthil Kumar Seeni, VS Prabhu, P Asha, SP Maniraj, and C Srinivasan. Web data mining with organized contents using naive bayes algorithm. In 2024 2nd International Conference on Computer, Communication and Control (IC4), pp.\ 1--6. IEEE, 2024

    2024

  27. [35]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  28. [36]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  29. [37]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Ananya Dwivedi-Yu, Yeganeh Kordi Hou, et al. Toolformer: Language models can teach themselves to use tools. In International Conference on Learning Representations (ICLR), 2024

    2024

  30. [38]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. ...

    2022 arXiv

  31. [39]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , June 2023

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , June 2023. URL https://huggingface.co/datasets/cerebras/SlimPajama-627B

    2023

  32. [40]

    Dolma: an open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

    2024

  33. [41]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02595, 2024

    2024 arXiv

  34. [42]

    C ommonsense QA : A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter o...

    2019 doi

  35. [43]

    Redpajama: an open dataset for training large language models, October 2023

    Together. Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  36. [44]

    M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das

    S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models, 2024

    2024

  37. [45]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  39. [47]

    Code llama: Open foundation models for code

    Albert Wang, Baptiste Roziere, Gautier Izacard, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  41. [49]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  42. [50]

    QuRating : Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating : Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024

    2024

  43. [51]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023

    2023

  44. [52]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  45. [53]

    React: Synergizing reasoning and acting in language models

    Shinn Yao, Jiong Zhao, Dian Yu, and et al. React: Synergizing reasoning and acting in language models. In NeurIPS, 2023

    2023

  46. [54]

    Craw4llm: Efficient web crawling for llm pretraining

    Shi Yu, Zhiyuan Liu, and Chenyan Xiong. Craw4llm: Efficient web crawling for llm pretraining. arXiv preprint arXiv:2502.13347, 2025

    2025 arXiv

  47. [55]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint arXiv:2406.06046, 2024

    2024 arXiv

  48. [56]

    A normalized levenshtein distance metric

    Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 29 0 (6): 0 1091--1095, 2007

    2007

  49. [57]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  50. [58]

    Map-neo: Highly capable and transparent bilingual large language model series

    Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.19327, 2024 a

    2024 arXiv

  51. [59]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024 b

    2024 arXiv

  52. [60]

    Editavalanche: A multi-granularity benchmark for edit-based language model evaluation

    Yuxian Zhang, Canwen Xu, Zhiyuan Liu, and Maosong Sun. Editavalanche: A multi-granularity benchmark for edit-based language model evaluation. arXiv preprint arXiv:2310.11603, 2023

    2023 arXiv

  53. [61]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023

  54. [62]

    Programming every example: Lifting pre-training data quality like experts at scale

    Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale. arXiv preprint arXiv:2409.17115, 2024

    2024 arXiv

  55. [63]

    Toolqa: A dataset for llm question answering with external tools

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36: 0 50117--50143, 2023

    2023

  56. [64]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  57. [65]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  58. [66]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  59. [67]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  60. [68]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.