Pith. sign in

REVIEW 3 major objections 4 minor 47 references

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that epiplexity—a measure of the structural information in data—can serve as an online training signal that improves out-of-distribution generalization.

desk verdict A credible first operationalization of epiplexity for data selection and generation, with a benchmark critique that may outlast the methods. read the letter →

arxiv 2608.11746 v1 pith:JDHGTVGK submitted 2026-08-12 cs.LG cs.CL

Ellen Su , Andres Potapczynski , Shikai Qiu , Edward Hughes , Andrew Gordon Wilson This is my paper · ORCID
classification cs.LGcs.CL
keywords epiplexitydataselectionsyntheticgenerationout-of-distributiongeneralizationcurriculumlearningscalinglawsREINFORCElanguagemodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that epiplexity, a measure of the structural information a compute-bounded learner can extract from data, is a practical signal for improving out-of-distribution generalization. To show this, it introduces EpiSelect, which fits cross-domain scaling laws to per-domain loss curves and reweights training batches toward domains with the largest predicted marginal epiplexity gain, and EpiGen, which uses REINFORCE to train a generator that produces synthetic text maximizing the learner's epiplexity. In experiments, EpiSelect outperforms a natural-distribution baseline and a prior adaptive selection method on zero-shot language tasks at two model sizes, and EpiGen improves GLUE fine-tuning performance over a pretrained GPT-2 baseline. The paper also finds that a simple baseline of training only on the largest Pile domain beats prior selection methods, motivating a switch to the Common Pile dataset for evaluating data selection.

What carries the argument

The load-bearing machinery is the prequential estimator of epiplexity, which approximates the structural information in data as the area between the training loss curve and the final loss, and the two online estimators built on it. For selection, the paper fits a cross-domain scaling law $\hat{L}_m(n_1,\dots,n_K)=\epsilon_m+\beta_m(\sum_k \gamma_{m,k} n_k)^{-\alpha_m}$ to the observed per-domain losses and takes the derivative with respect to tokens from each domain, $\partial\hat{S}/\partial n_k$, as a proxy for marginal epiplexity gain. For generation, it computes the reward as the drop in learner loss on an evaluation buffer before and after a batch, and updates the generator with the REINFORCE gradient $\nabla_{\theta_g} J = \mathbb{E}[(r_t-b)\sum_{x\in X_t}\nabla_{\theta_g}\log P_{\theta_g}(x)]$. These estimators turn epiplexity from an a posteriori quantity into a signal that can be maximized during training.

What would settle it

Take a held-out set of domains, fit the cross-domain scaling law on early checkpoints only, then compare the predicted per-domain epiplexity gains from Equation 4 with gains measured by actually training on a small addition of tokens from each domain; if the predictions are uncorrelated with measured gains, the selection signal is not tracking epiplexity.

Watch

Extended reading notes

Core claim

The paper's central claim is that data with higher epiplexity transfers better to unanticipated downstream tasks, so maximizing epiplexity during training should improve out-of-distribution generalization. It reports that epiplexity predicts zero-shot accuracy across five Pile domains (Pearson r=0.88), while a generic checkpoint summary like weight norm does not. It then operationalizes this claim: EpiSelect maximizes a predicted epiplexity gain computed from a fitted cross-domain scaling law, and EpiGen maximizes the learner's measured loss reduction over a buffer of generated data. In both settings, the epiplexity-maximizing procedure outperforms the baselines on zero-shot and fine-tuning tasks, which the authors take as support for the hypothesis.

Load-bearing premise

The whole method rests on assuming that the fitted cross-domain scaling law predicts the next marginal gain in epiplexity accurately enough to guide selection; if that parametric curve is wrong, EpiSelect is not actually selecting for epiplexity, and the observed gains could come from some other property of the reweighting.

Editorial extensions

If this is right

  • If maximizing epiplexity is the right objective, curriculum learning becomes a matter of estimating which batch yields the next largest structural gain, rather than relying on heuristic notions of difficulty.
  • Epiplexity-guided selection improves zero-shot accuracy at both 124M and 1.3B scale, suggesting the signal scales with model size rather than being a small-model artifact.
  • Because training only on PileCC outperforms prior selectors on The Pile, earlier comparisons of data selection methods on that benchmark may reflect domain-size imbalances rather than selection quality.
  • Synthetic data that maximizes epiplexity improves fine-tuning performance without any new real data, and mixing it with real data gives the largest gains, indicating a complementarity between synthetic and natural text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit testable extension is whether the same epiplexity-maximizing selection transfers to multimodal or non-text data, since the loss-curve estimator does not depend on the modality.
  • Because the selection signal comes entirely from fitted scaling-law parameters, a reader should not assume the proxy equals true epiplexity; the paper itself flags the absence of theoretical guarantees.
  • The five-domain correlation is suggestive but small; a natural stronger test would measure epiplexity across many more domains and check monotonicity rather than linear correlation.
  • The PileCC finding hints that diversity within a large web-crawl domain may be a hidden confound; if so, epiplexity may partly be a proxy for intra-domain diversity, and explicitly controlling for diversity could separate the two.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes two methods that use epiplexity as an online training signal: EpiSelect, which fits a cross-domain scaling law (Eq. 3) to per-domain loss curves and reweights domain sampling by the predicted marginal epiplexity gain (Eq. 4), and EpiGen, which trains a generator with REINFORCE using the learner's loss decrease on a buffer as reward. Experiments on LLaMA-style models trained on Common Pile report improved zero-shot LM Eval Harness accuracy over Natural and ADO baselines, and GPT-2-based synthetic data training reports improved GLUE fine-tuning scores. The paper also argues that The Pile is saturated as a data-selection benchmark because training only on PileCC outperforms SOTA selection methods.

Significance. If the causal chain from epiplexity to OOD generalization holds, the paper supplies a principled, computationally tractable objective for curriculum learning and synthetic data generation, with derivations in Section 3.2 and Appendix D that are correct, detailed scaling-law fitting, and external-benchmark evaluations that avoid circularity. Code links are provided. However, the empirical support at present is thin: the central correlation uses five domains, scaling-law fits are only validated in-sample, and no uncertainty is reported for the main accuracy tables. The strengths are real, but the attribution of the reported gains specifically to epiplexity maximization is not yet established.

major comments (3)
  1. [§3.1, Fig. 2 (left), Appendix B] The evidence for the central claim that epiplexity predicts OOD accuracy rests on n=5 domains with Pearson r=0.88 and p=0.05, which is marginal: with five points a single influential domain (PileCC) can drive the correlation, and the Spearman ρ=0.90 is reported without a p-value. Please add more domains, a bootstrap or permutation analysis, or otherwise quantify the stability of the correlation; without this, the 'strong predictor' claim in §3.1 is not adequately supported.
  2. [§3.2, Eq. (4), Appendix C.1] EpiSelect's selection signal is entirely a function of the fitted cross-domain scaling law (Eq. 3), but the law is validated only in-sample (median R²=0.88 on observed losses). Since the method uses the law to extrapolate dS/dn_k beyond observed token counts, in-sample accuracy does not establish that the induced ranking of domains is correct; a misspecified parametric form could yield the same in-sample R² while producing wrong extrapolations, for example by missing saturation or delayed cross-domain effects. Please add an out-of-sample validation: fit on the first part of the trajectory, predict later losses and the ranking of marginal epiplexity gains, and compare against measured gains. Without this, the Table 1 improvements cannot be attributed specifically to epiplexity maximization rather than to a generic upweighting of domains with currently steep loss curves. The Discussion's caveat about proxies is appropriate, but the attribution claim in the Abstract and Section 5 goes beyond what is currently demonstrated.
  3. [Tables 1 and 2] The headline comparisons are single-run results with no variance or significance. The 124M EpiSelect gain over ADO is 1.5 points on the average, the 1.3B gain is 0.6 points, and the EpiGen gain over FrozenGen on GLUE is 1.1 points; without multiple seeds or error bars these margins may be within run-to-run noise. Please report seeds, standard deviations, or a paired significance test, or state explicitly that the comparisons are single runs; as written, the 'state-of-the-art' and 'outperforms' claims are not statistically grounded.
minor comments (4)
  1. [Algorithm 1, Eq. (5)] The temperature τ is defined in the algorithm input as τ∈(0,1), but the text fixes τ=1; this range should be τ>0. Also, the clipped momentum mixture ωπ_k+(1−ω)̅π_k is not renormalized after clipping, so it may not remain a probability distribution; please specify the normalization step.
  2. [§3.4, Appendix A] The main text calls Common Pile '8TB' but Appendix A describes a token-balanced subset totaling 29.7B tokens; please make clear in the main text that the experiments use the token-balanced construction.
  3. [Eq. (6) and Algorithm 2] The reward is defined as the difference of sums over the buffer before and after K learner steps; the text says 'difference between the learner loss on the samples in the buffer before and after training on the current batch', which is clear, but the connection to the prequential estimator of Eq. (1) would benefit from an explicit statement that the current loss replaces the final-model loss.
  4. [Appendix C, scaling-law initialization] The text says 'we sample γm,k∼Dir(p) via a Dirichlet distribution'; please clarify that it is the vector γm,: that is sampled, not each entry independently, and state how the 'grid derived by the Cartesian product' interacts with this sampling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: epiplexity is operationalized as a loss-curve proxy, but downstream claims are validated on held-out external benchmarks.

full rationale

The paper's selection and generation objectives are explicitly framed as tractable proxies for epiplexity (Section 5: 'we relied on a tractable proxy for epiplexity rather than the quantity itself'), and the proxy is defined through training-loss dynamics (Eqs. 2, 6). Because epiplexity is defined as the area under the loss curve, maximizing the proxy is definitionally equivalent to maximizing predicted loss reduction; however, the paper does not treat this equivalence as evidence of OOD generalization. The central predictive claims are tested on held-out tasks (LM Evaluation Harness in Table 1, GLUE in Table 2) that are never used in fitting the scaling laws or training the generator, so the empirical loop is broken. The cross-domain scaling law (Eq. 3) is validated only in-sample (Appendix C.1, median R2=0.88), and extrapolation could be misspecified; this is a correctness risk, not a circularity. The citation to Finzi et al. [20] supplies the epiplexity definition and estimator from overlapping authors, but the paper's own new experiments independently support the hypothesis, so the self-citation is not load-bearing. The paper itself flags the proxy limitation and the potential for 'self-referential rewards' (Appendix G.1), which further confirms that the authors are not claiming the proxy is the true quantity. No step in the derivation reduces to its inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a proxy for epiplexity (fit to loss curves) rather than on the measure itself, on an assumed scaling-law form, and on a greedy optimization heuristic. No new physical entities are introduced. The fitted scaling-law parameters are the main free parameters; the number of parameters scales as K^2+3K for K domains, making overfitting a real risk.

free parameters (4)
  • Cross-domain scaling law parameters (alpha_m, beta_m, epsilon_m, gamma_{m,k}) per domain = Fitted online; final values not reported
    Equation 3 is fit to the training loss curves of each domain, and the selection signal in Equation 4 is computed from these fitted parameters. The paper reports median fit R2=0.88 but no out-of-sample evaluation.
  • Softmax temperature tau in EpiSelect = 1
    Equation 5 uses tau=1 to map marginal epiplexity gains to sampling probabilities; this fixed value controls the greediness of selection and is not ablated.
  • EMA baseline decay beta in EpiGen = 0.99
    Algorithm 2 uses this decay to track the reward baseline for REINFORCE; it is set without a reported sensitivity analysis.
  • Learner steps per generator step K = 10
    EpiGen performs K=10 learner updates per generator REINFORCE update; this schedule affects the reward estimate and is not ablated.
assumptions (4)
  • domain assumption The prequential estimator (Equation 1) approximates the true epiplexity as the area between the training loss curve and the final loss
    The paper's online surrogates (Equation 2 and Equation 4) replace the final loss with the current loss, and the Discussion states these are tractable proxies without theoretical guarantees. This is load-bearing because all rewards and selection signals are based on this proxy.
  • ad hoc to paper The cross-domain scaling law (Equation 3) correctly models per-domain loss as a function of token counts from all domains
    This parametric form is introduced by the authors for this paper. If it is misspecified, the derivative dS/dn_k does not track true epiplexity gain. The paper validates it via in-sample R2 only.
  • domain assumption Greedy softmax sampling over marginal epiplexity gain maximizes total epiplexity
    EpiSelect samples domains according to pi_k proportional to exp(dS/dn_k / tau) with no proof that this greedy policy optimizes total epiplexity over the training budget. The paper evaluates it empirically.
  • domain assumption The definition of epiplexity as the size of the MDL-optimal model under a compute budget is a meaningful measure of structural information
    Adopted from Finzi et al. [20]; the paper relies on this framing for its central hypothesis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization." pith.science (2026). https://pith.science/paper/JDHGTVGK

@misc{pith2026260811746,
  author       = {Pith},
  title        = {Pith review of: Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JDHGTVGK}},
  note         = {Machine review of arXiv:2608.11746}
}
read the original abstract

Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that could be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, provides a mechanism to reason about this relationship. In this paper, we show how to operationalize epiplexity as an online training signal for data selection and synthetic data generation. For selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, we define a generator's reward as the change in learner epiplexity over a buffer of previously generated data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning based tasks, supporting the hypothesis that data rich in structural information yield representations that transfer across domains.

Figures

Figures reproduced from arXiv: 2608.11746 by the authors.

Figure 1
Figure 1. Epiplexity for data selection and generation. (a) Epiplexity, S(t), can be estimated as the area between the learner’s training loss curve and its final loss as a function of training tokens. Data which is trivial to learn or unlearnable will yield low estimates of epiplexity and result in lower performance in OOD downstream tasks. (b) For data selection, we use scaling laws to forecast the per-domain loss curves an… view at source ↗
Figure 2
Figure 2. In all plots, we contrast epiplexity against the performance on a subset of the LM Eval Harness [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prior data selection methods generally sample more data from domains with higher epiplexity. We show the percentage of training data DoReMi [3] and ADO [4] sample from each domain (ordered lowest to highest epiplexity) in the Pile. Tracking Learning Progress Our implemen￾tation of epiplexity-guided synthetic data gen￾eration requires tracking the learning progress of a model throughout training. Rewarding a generato… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The Pile does not require data selection. Data selectors in the literature are trained to select domains on Pile, but solely training on the largest subdomain outperforms SOTA. Common Pile spans 30 domains, and the largest domain (Stack V2) accounts for roughly 14% of …
Figure 5
Figure 5. Figure 5: Few domains interact. Diagonal γm,k values dominate. In [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Synthetic data learning signal emerges from pretrained model weights. Average GLUE score for pretrained vs. ran￾dom initialization, with and without epiplexity￾guided training. Critically, none of the learner models evaluated in Ta￾ble 2 have seen any more tokens of re…
Figure 7
Figure 7. Figure 7: Mixing synthetic and real data yields the greatest improvement. Average GLUE score for learners trained on varying proportions of epiplexity￾guided synthetic data and OWT data. Here, we initialize the generator model as the trained, epiplexity-maximizing generator from…
Figure 8
Figure 8. Figure 8: Epiplexity predicts OOD downstream performance. Each point is a 124M model trained for 15B tokens on a single Pile domain. Epiplexity is correlated with mean zero-shot accuracy (Pearson r= 0.88, Spearman ρ= 0.90), whereas the weight norm is uncorrelated (r= 0.01, ρ= 0.…
Figure 9
Figure 9. Figure 9: Panel (a): Example online cross-domain scaling-law fit for the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Training dynamics of the epiplexity-guided generator. (Left) Learner loss on generated data decreases over training for the epiplexity-guided generator, while remaining flat for the frozen generator baseline. (Right) REINFORCE reward increases over training, indicatin…
Figure 11
Figure 11. Figure 11: Mixing OWT data into the evaluation buffer grounds the generator reward. (Left) The learner’s perplexity decreases monotonically as the fraction of OWT data in the buffer increases from 0% to 100%. (Center) Average zero-shot accuracy across 15 LM Eval Harness tasks si…
Figure 12
Figure 12. Figure 12: Truncated samples of synthetic data generated at step 1,000 (left) and step 60,000 (right). Qualitatively, the generator produces web-text-like content throughout training (e.g. news articles, Q&As, blog posts), consistent with its OpenWebText pretraining distribution…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages

  1. [20]

    Zico Kolter, and Andrew Gordon Wilson

    Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, J. Zico Kolter, and Andrew Gordon Wilson. From entropy to epiplexity: Rethinking information for computationally bounded intelligence. ArXiv, abs/2601.03220, 2026

    arXiv 2026

  2. [1]

    Scaling data-constrained language models.Advances in Neural Information Processing Systems, 36:50358–50376, 2023

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models.Advances in Neural Information Processing Systems, 36:50358–50376, 2023

    2023

  3. [2]

    Position: Will we run out of data? limits of llm scaling based on human-generated data

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobb- hahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  4. [3]

    Le, Tengyu Ma, and Adams Wei Yu

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V . Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.ArXiv, abs/2305.10429, 2023

    arXiv 2023

  5. [4]

    Zico Kolter

    Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J. Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws.ArXiv, abs/2410.11820, 2024

    arXiv 2024

  6. [5]

    Regmix: Data mixture as regression for language model pre-training.ArXiv, abs/2407.01492, 2024

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training.ArXiv, abs/2407.01492, 2024

    arXiv 2024

  7. [6]

    Doge: Domain reweighting with generalization estimation.ArXiv, abs/2310.15393, 2023

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation.ArXiv, abs/2310.15393, 2023

    arXiv 2023

  8. [7]

    Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré

    Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models.ArXiv, abs/2307.14430, 2023

    arXiv 2023

Show all 47 references
  1. [8]

    Data selection via optimal control for language models.ArXiv, abs/2410.07064, 2024

    Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models.ArXiv, abs/2410.07064, 2024

    2024 arXiv

  2. [9]

    Towards optimal learning of language models.ArXiv, abs/2402.17759, 2024

    Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, and Furu Wei. Towards optimal learning of language models.ArXiv, abs/2402.17759, 2024. 11

    2024 arXiv

  3. [10]

    Rho-1: Not all tokens are what you need.ArXiv, abs/2404.07965, 2024

    Zheng-Wen Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need.ArXiv, abs/2404.07965, 2024

    2024 arXiv

  4. [11]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY , USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi...

    2009

  5. [12]

    Strategic data ordering: Enhancing large language model performance through curriculum learning.ArXiv, abs/2405.07490, 2024

    Jisu Kim and Juhwan Lee. Strategic data ordering: Enhancing large language model performance through curriculum learning.ArXiv, abs/2405.07490, 2024

    2024 arXiv

  6. [13]

    Temporal difference learning and td-gammon.Commun

    Gerald Tesauro. Temporal difference learning and td-gammon.Commun. ACM, 38(3):58–68, March 1995. ISSN 0001-0782. doi: 10.1145/203330.203343. URL https://doi.org/10. 1145/203330.203343

    1995

  7. [14]

    Sifre, Dharshan Kumaran, Thore Graepel, Timothy P

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, L. Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, sh...

    2018

  8. [15]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. InInternational Conference on Machine Learning, 2024

    2024

  9. [16]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason E. Weston. Self-rewarding language models.ArXiv, abs/2401.10020, 2024

    2024 arXiv

  10. [17]

    Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning.ArXiv, abs/2506.24119, 2025

    Bo Liu, Leon Guertler, Simon Yu, Zi-Yan Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning.ArXiv, abs/2506...

    2025

  11. [18]

    Self- playing adversarial language game enhances llm reasoning.ArXiv, abs/2404.10642, 2024

    Pengyu Cheng, Tianhao Hu, Hang Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. Self- playing adversarial language game enhances llm reasoning.ArXiv, abs/2404.10642, 2024

    2024 arXiv

  12. [19]

    Absolute zero: Reinforced self-play reasoning with zero data.ArXiv, abs/2505.03335, 2025

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data.ArXiv, abs/2505.03335, 2025

    2025 arXiv

  13. [21]

    Sample efficient reinforce- ment learning with reinforce

    Junzi Zhang, Jongho Kim, Brendan O’Donoghue, and Stephen Boyd. Sample efficient reinforce- ment learning with reinforce. InProceedings of the AAAI conference on artificial intelligence, volume 35, 12, pages 10887–10895, 2021

    2021

  14. [22]

    Sutton, David A

    Richard S. Sutton, David A. McAllester, Satinder Singh, and Y . Mansour. Policy gradient methods for reinforcement learning with function approximation. InNeural Information Processing Systems, 1999

    1999

  15. [23]

    The pile: An 800gb dataset of diverse text for language modeling.ArXiv, abs/2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling.ArXiv, abs/2101.00027, 2020

    2020 arXiv

  16. [24]

    The common pile v0

    Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, A Feder Cooper, Aviya Skowron, et al. The common pile v0. 1: An 8tb dataset of public domain and openly licensed text.arXiv preprint arXiv:2506.05209, 2025

    2025 arXiv

  17. [25]

    Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal

    Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt, 2022. URL http...

    2022 arXiv

  18. [26]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Thomas Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models.ArXiv, abs/2001.08361, 2020. 12

    2001 arXiv

  19. [27]

    Rae, Oriol Vinyals, and L

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  20. [28]

    Maddison, Arthur Guez, L

    David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Made...

    2016

  21. [29]

    Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes.ArXiv, abs/0812.4360, 2009

    2009 arXiv

  22. [30]

    A possibility for implementing curiosity and boredom in model-building neural controllers

    Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. InProceedings of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats, page 222–227, Cambridge, MA, USA, 1991. MIT Press. ...

    1991

  23. [31]

    Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V . Hafner. Intrinsic motivation systems for autonomous mental development.IEEE Transactions on Evolutionary Computation, 11(2): 265–286, 2007. doi: 10.1109/TEVC.2006.890271

    2007

  24. [32]

    Active learning literature survey, 2009

    Burr Settles. Active learning literature survey, 2009

    2009

  25. [33]

    Bayesian active learning for classification and preference learning, 2011

    Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning, 2011. URLhttps://arxiv.org/abs/1112.5745

    2011 arXiv

  26. [34]

    Intrinsic motivation and automatic curricula via asymmetric self-play.ArXiv, abs/1703.05407, 2017

    Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play.ArXiv, abs/1703.05407, 2017

    2017 arXiv

  27. [35]

    Teacher–student curriculum learning.IEEE Transactions on Neural Networks and Learning Systems, 31:3732–3740, 2017

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum learning.IEEE Transactions on Neural Networks and Learning Systems, 31:3732–3740, 2017

    2017

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

  29. [37]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  30. [38]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeural Information Processing Systems, 2017

    2017

  31. [39]

    Rusu, Joel Veness, Marc G

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Ku- maran, Daan...

    2015

  32. [40]

    Self-improving reactive agents based on reinforcement learning, planning and teaching.Machine Learning, 8:293–321, 1992

    Longxin Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching.Machine Learning, 8:293–321, 1992

    1992

  33. [41]

    Lan- guage Models are Unsupervised Multitask Learners.OpenAI, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage Models are Unsupervised Multitask Learners.OpenAI, 2019

    2019

  34. [42]

    Openwebtext corpus.http: //Skylion007.github.io/OpenWebTextCorpus, 2019

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.http: //Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  35. [43]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018

    2018

  36. [44]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025.URL https://arxiv

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025.URL https://arxiv. org/abs/2504.13837, 204:1–16

    2025 arXiv

  37. [45]

    Steeves, Joel Hestness, and Nolan Dey

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R. Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama , June ...

    2023

  38. [46]

    how much

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems (NeurIPS), D...

    2024

  39. [2023]

    URLhttps://arxiv.org/abs/2307.09288

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.