Computation and Language · arXiv:2609.10702 v1 · submitted 9 Sep 2026 · 41 references
Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement
The paper says
“Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities.”
more of the abstract
Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 11 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Thorough three-stage BabyLM study with small gains; preservation confound is real but paper is transparent about it. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 41 references say.
The paper says, at §4.3–§4.4, Eq. (9) and Table 11
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The central improvement of the full method over ordinary continuation conflates a change in training design with an increase in cumulative word count. The full method adds 517,332 preservation-word presentations (Eq. 9, §4.3) that are not present in the ordinary-continuation baseline (89,168,037 vs. 89,685,369 words).
more of objection 1
The decomposition shows that (M,S) without preservation already outperforms ordinary continuation at equal exposure, and the additional increment from preservation is only 0.0439–0.0529 Overall. This incremental gain could be explained by the 0.58% increase in total presentations (≈0.58% of 89M) rather than by the preservation constraint itself. The authors acknowledge the extra presentations but do not provide a control that adds the same number of ordinary-word presentations to ordinary continuation, nor a control that applies the preservation loss on ordinary inputs without increasing total word count (e.g., by reducing other training steps). Without such a control, the incremental benefit of the preservation principle is not uniquely identified. This weakens the claim that ordinary-input preservation per se is beneficial beyond increased data exposure.
The paper says, at §4.4, Table 11: GlobalPIQA contribution
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
Relative to the parent model, GlobalPIQA accounts for approximately 74–79% of the overall gain. GlobalPIQA is a 203-item multiple-choice test with chance at 37.5%; the improvement is from 38.565 to 40.05. While this is a real increase, the dominance of this one metric means the overall improvement is heavily driven by a single task with relatively few items.
more of objection 2
The paper does not provide item-level confidence intervals or a discussion of whether this improvement is robust to evaluation-sample uncertainty. The claim that 'principle-guided model improvement' works should be qualified by noting that most of the gain comes from GlobalPIQA, and the other eight components contribute modest net changes.
The paper says, at §3.2, Table 9 and Figure 4
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The functional-intervention experiments on unseen symbols are informative and well-controlled. However, they are conducted on a small constructed task with controlled symbols, not on natural-language processing in the final model. The paper does not establish that the same mechanism (answer-selection signal) operates in the full-scale language model or that the observed improvements in the final model are driven by changes to this specific mechanism.
more of objection 3
The link between the mechanism studies and the final model improvement remains correlational and plausibly suggestive, not causal. This should be stated explicitly.
5 minor objections
The paper says, at §1.4, Table 3
Pith says: minor objection 1 of 5
Table 3 reports leaderboard scores with four decimal places for the internal models and two decimal places for external entries. The inconsistent precision is misleading; both should be reported to the same number of decimals.
The paper says, at §4.4, Figure 7 caption
Pith says: minor objection 2 of 5
The figure caption states 'Both seeds use the same evaluation sets, including the 203-item GlobalPIQA set.' This is important context and should also appear in the main text when discussing GlobalPIQA gains.
show 3 more
The paper says, at §4.5, Table 13
Pith says: minor objection 3 of 5
Table 13 reports gradient-norm ratio (11.835) and directional similarity (0.7288) but does not provide a standard error or bootstrap interval. Given that these quantities are computed from a single checkpoint, their stability across seeds or checkpoints is unknown.
The paper says, at §A.4, BLiMP scoring inconsistency
Pith says: minor objection 4 of 5
The paper acknowledges that 7 BLiMP items have identical text in both options, causing a reproducible discrepancy of 0.01–0.02 points between scoring procedures. This is well-documented, but the existence of even a small inconsistency undermines the precision with which BLiMP differences can be interpreted, especially given that many Overall differences in Table 11 are smaller than 0.10. A note on how this affects the comparison would be helpful.
The paper says, at General
Pith says: minor objection 5 of 5
The paper is very long (58 pages) and includes extensive appendices and research catalogs. While the completeness is commendable, the core scientific claims would benefit from a more concise presentation. Some sections (e.g., Appendix F) read more like a lab notebook than a journal article.
What would settle it
Hold parent, data, masking, and supervision fixed. Run two additional ordinary-continuation baselines that increase cumulative word count by exactly 517,332 words (the extra preservation word presentations) using ordinary 15% whole-word masking on the same Qwen rewrites. First control: extend ordinary continuation with extra ordinary training on the preservation rows (no KL term). Second control: extend ordinary continuation by repeating the last batch of ordinary data to match word count. Compare the full method's Overall to these extended ordinary-continuation controls at identical total exposure. If the full method's Overall remains higher (by >0.03 on both seeds), the preservation principle is supported; if the gap shrinks to within two-seed variation (~0.01), the advantage is consistent with extra data alone.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 11 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper reports a three-stage autonomous research program on BabyLM Strict-Small. The main empirical result is that a combination of dense masking, sparse supervision, and ordinary-input preservation improves the Overall score by about 0.23 points over the parent model, and about 0.15 over ordinary continuation. The gains are consistent across two seeds. That is the headline.
What is actually new: the integrated design of these techniques within the BabyLM framework, and the systematic discovery-to-application loop. The mechanistic findings about relation-specific context use and the separation of familiar performance from unseen reuse are real empirical results for this setting. The paper is admirably thorough: many controls, ablation studies, gradient diagnostics, source interventions, and detailed documentation of the research process. They release code, data, and models.
Soft spots: the effect size is tiny. 0.23 points on a 0-100 scale, and most of the gain comes from GlobalPIQA, which is near chance. The extra 517k preservation presentations are not matched in the ordinary continuation baseline, so the additional 0.04-0.05 Overall gain from preservation is likely partly explained by extra data. The paper acknowledges this but does not provide a control isolating the preservation mechanism. The entire evaluation uses one parent model, so generalization is untested. The mechanistic findings are plausible but not causally linked to the improvement—the paper explicitly says so, which is honest but limits the strength of the central claim.
Who this is for: researchers working on data-constrained language modeling, especially within the BabyLM community. The paper will be useful as a detailed case study of autonomous research and a specific training recipe.
Deserves peer review. The experimental work is careful, the reporting is transparent, and the research process is interesting even if the gains are small. The community benefit from having this documented.
The strongest objection
The central claim of principle-guided improvement rests partly on the comparison between the full method (dense masking + sparse supervision + ordinary-input preservation) and ordinary continuation. The full method includes 517,332 additional word presentations for preservation (Eq. 9, §4.3), whereas ordinary continuation and (M,S) have equal exposure (89,168,037 words). The full method's Overall gain over ordinary continuation (0.1538 and 0.1158) therefore conflates a training design change with an increase in cumulative word count. The reported decomposition shows that (M,S) without preservation already improves over ordinary continuation at equal exposure, but the added preservation contributes only a further 0.0439–0.0529 Overall. This additional increment could be explained by the 0.58% increase in total training data (517k / 89.2M) rather than by the preservation constraint itself. The paper does not provide a control that adds the same number of ordinary-word presentations to ordinary continuation, nor a control that applies preservation on ordinary inputs without the extra presentations (by reducing other training steps). Without such a control, the incremental benefit of the preservation principle is not uniquely identified.
The paper's strongest claim is that the principle-guided method improves Overall. The most vulnerable point is the confounding between the preservation mechanism and extra training presentations. The reader already flagged this as a complication. The test directly addresses whether the preservation principle adds value beyond more data. The comparison is simple and within the same computational budget. The concern does not require rejecting the paper; it suggests that the claim about preservation should be conditional on this control. The equal-exposure acquisition method (M,S) already shows a clean improvement, so the overall verdict remains CONDITIONAL as the reader set. The preservation claim needs the proposed test to be fully supported.
The referee's report
If the central claim holds, the paper makes three contributions. First, it provides a concrete demonstration that principle-guided design—separating visible inputs, supervised targets, and preservation constraints—can improve a language model under fixed data and exposure limits. Second, the Stage II mechanism findings (relation-type shapes context use; familiar performance does not ensure unseen reuse) are independently informative for understanding how limited-pretraining data can be used more effectively. Third, the open release of models, code, and research records supports reproducibility and further study. The paper also illustrates a Research RSI cycle, which may be of interest to the autonomous-science community. The significance is real but bounded by the reliance on a single parent model and the confound discussed below.
Pith's reading of the paper (a machine's summary, not the authors')
This paper argues that when data is scarce, how a model's experience is organized matters more than just how much text it sees. The authors conduct a three-stage autonomous research program on BabyLM Strict-Small, building a frontier model, then investigating how different forms of repetition and restatement shape context use and whether familiar performance guarantees reuse on unseen inputs. They find that exact repetition and aligned restatement produce different patterns of context use, and that recovering familiar accuracy does not mean new inputs can still use the learned computation. From this, they derive a design principle: organize experience around the contextual dependencies needed for prediction, treat visible information, supervised targets, and preservation as separate choices, and evaluate acquisition, generalization, and retention together. Applying this principle yields a second-generation model that consistently outperforms ordinary continued training from the same parent across two seeds, achieving the highest Overall score on the public leaderboard snapshot.
What carries the argument. The key machinery is the design principle of separating visible information, supervision targets, and preservation constraints during continued training. Concretely, the method applies dense masking to remove local clues in restatement text while supervising only a sparse subset of tokens, and uses KL divergence on ordinary-masked inputs to preserve the parent model's predictions. This allows the model to learn to rely on source context without losing existing language abilities.
Core claim. The paper claims that language models learn to use contextual information only when the training experience explicitly requires it. In controlled experiments, exact repetition and aligned restatement produce different effects on source use depending on the target relation and prediction window. Even when a model recovers familiar performance on a relational task, new symbols may not be able to access the same computation. These findings motivate a data-efficient learning principle: separate what information is visible during prediction, which tokens receive supervision, and which existing functions are preserved. The authors implement this by retaining source text, applying dense masking to hide local clues in restatements while supervising only a sparse set of targets, and preserving ordinary-input behavior through KL divergence against the frozen parent model. In two continuation seeds from the same initial model, this method improves the nine-metric Overall score compared to ordinary continued training, with the largest gains coming from GlobalPIQA and Entity Tracking.
The premise it rests on. That the mechanistic findings from controlled experiments (relation type shapes context use, familiar performance does not ensure unseen reuse) generalize to the full-scale language model training and are the primary drivers of the observed improvement, rather than other factors such as hyperparameter choices or the reuse of existing text without new rewriting.
If the paper is right (Pith's inferences, not the authors')
Language model training under tight data budgets can be improved by organizing experience around the contextual dependencies required for target predictions, rather than simply adding more text.
The separation of input masking, supervised targets, and preservation inputs provides a testable design space for continual learning in language models, applicable beyond BabyLM.
The method shows consistent gains across two continuation seeds, suggesting robustness to training randomness and potential transferability to other settings.
The observed gains are concentrated in GlobalPIQA (commonsense reasoning) and Entity Tracking (state tracking), while other metrics show smaller changes or trade-offs, guiding future optimization.
The principle of jointly testing acquisition, reuse (generalization), and retention provides a framework for evaluating whether a training method truly improves capability beyond fitting the training data.
The method's reliance on a frozen teacher model for preservation may limit applicability when the teacher is unavailable or when the architecture changes; adaptive preservation strategies could address this.
The large gain in GlobalPIQA suggests that better context use may enable models to learn commonsense reasoning from limited data without explicit reasoning objectives.
The finding that familiar performance does not guarantee unseen reuse implies that standard evaluation on held-out test sets may overestimate a model's ability to generalize to genuinely novel inputs, highlighting the need for targeted generalization tests.
The principle of organizing experience around contextual dependencies could be extended to other modalities (e.g., vision, multimodal) where limited labeled data forces models to rely on relational reasoning.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention.”
How the main claim is stated: asserted 0.81
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.47 · human participants 0.03 · controlled comparison 0.83 · randomized assignment 0.03 · states quantitative result 0.98 · compares with prior results 0.87 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. | background 1.00 | 0.07 | 0.04 | asserted 1.00 |
| Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. | contribution 0.56 | 0.86 | 0.03 | asserted 1.00 |
| Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. | method 0.53 | 0.69 | 0.09 | asserted 1.00 |
| Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. | method 0.96 | 0.87 | 0.04 | asserted 1.00 |
| Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. | result 0.98 | 0.92 | 0.06 | asserted 1.00 |
| In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. | result 0.92 | 0.82 | 0.06 | asserted 1.00 |
| These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. | interpretation 0.91 | 0.79 | 0.82 | asserted 0.81 |
| Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. | method 0.96 | 0.91 | 0.06 | asserted 0.99 |
| Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. | result 0.99 | 0.92 | 0.07 | asserted 1.00 |
| Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. | result 1.00 | 0.91 | 0.10 | asserted 1.00 |
| Further studies addressed compression, relational anchors, shared representations, and measurement. | future work 0.72 | 0.69 | 0.03 | asserted 0.94 |
| Models are available on Hugging Face; code and research records accompany the GitHub repository. | contribution 0.89 | 0.43 | 0.02 | asserted 1.00 |
| Together, these stages illustrate Research RSI: recursive self-improvement of the research process. | interpretation 0.93 | 0.73 | 0.21 | asserted 0.99 |
| Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them. | interpretation 0.86 | 0.24 | 0.15 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 0.76 · evidence from observation or experiment 0.88
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 10,053 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (41)
- [1] Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling Data-Constrained Language Models. Journal of Machine Learning Research , 26(53):1–66, 2025. URLhttp://jmlr.org/papers/v26/2... on Pith
- [2] Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjape, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLMChallenge: Sample-EfficientPretrainingonDevelopmentallyPlausibleCorpora. In Proceed-... on Pithdoi:10.18653/v1/2023.conll-babylm.1
- [3] BabyLM Organizing Team. BabyLM Challenge 2026. Official challenge website, 2026. URLhttps: //babylm.github.io/. Accessed 8 September 2026 on Pith
- [4] Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Y. Hu, Jing Liu, Jaap Jumelet, Tal Linzen, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox, and Adina Williams. Findings of the Third BabyLM Challenge: Accelerating L... on Pithdoi:10.18653/v1/2025
- [5] Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet, Tal Linzen, Aaron Mueller, Suchir Salhan, Raj Sanjay Shah, Alex Warstadt, and Ethan Gotlieb Wilcox. BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop. arXiv preprint arXiv:2602.200... on PitharXiv
- [6] Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English.Transactions of the Association for Computational Linguistics , 8:377–392, 2020. doi: 10.1162/tacl_a_00321. U... on Pithdoi:10.1162/tacl_a_00321
- [7] Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi U. Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian C. Paulun, Maria Ryskina, Ekin Akyürek, Ethan G. Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedoren... on Pith
- [8] Kanishka Misra, Julia Rayz, and Allyson Ettinger. COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computation... on Pithdoi:10.18653/v1/
- [9] NajoungKimandSebastianSchuster. EntityTrackinginLanguageModels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3835–3855. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long... on Pithdoi:10.18653/v1/2023.acl-long.213
- [10] Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah, et al. Global PIQA: Evaluating com- monsense reasoning across 100+ languages and cultures, 2025. URLhttps://arxiv.org/abs/25 10.24081v2. Version 2, 29 May 2026; first submitted 28 October 2025 on Pith
- [11] BabyLM Organizing Team. BabyLM Evaluation 2026. Official evaluation repository; Strict and Strict-Small implementation, 2026. URL https://github.com/babylm-org/babylm-eval/tr ee/6f825c291e2c4c78ad33b1935fd64d45f52642dc/strict . Reference scoring revision 6f825c2; accessed 8 Sep... on Pith
- [12] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language understand- ing. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks... on Pithdoi:10.18653/v1/w18-5446
- [13] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. InAdvances in Neural Information Processing Systems , volume 32. Curran ... on Pith
- [14] Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data.Behavior Research Methods, 56:5190–5213, 2024. doi: 10.3758/s13428-023-02... on Pithdoi:10.3758/s13428-023-02261
- [15] BabyLM Community. BabyLM Leaderboard 2026. Official Hugging Face Space and public source code, 2026. URL https://huggingface.co/spaces/BabyLM-community/BabyLM-Leaderboa rd-2026. Aggregation and sorting source: revision c84d8c1. Public Strict-Small table retrieved 8 September 20... on Pith
- [16] Tyler A. Chang and Benjamin K. Bergen. Word acquisition in neural language models.Transactions of the Association for Computational Linguistics , 10:1–16, 2022. doi: 10.1162/tacl_a_00444. URL https://aclanthology.org/2022.tacl-1.1/ on Pithdoi:10.1162/tacl_a_00444
- [17] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=XPZIaotutsD on Pith
- [18] Microsoft. DeBERTa: Official implementation and DeBERTa-v2 release notes. Official implementa- tion repository, 2021. URLhttps://github.com/microsoft/DeBERTa#whats-new-in-v2. Version 2 release notes dated 3 February 2021; accessed 9 September 2026 on Pith
- [19] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: fast convergence at large depth. InProceedings of the Thirty- Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 ofProceedings of Machi... on Pith
- [20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learn- ing for NLP. In Proceedings of the 36th International Conference on Machine Learning , vol- ume 97... on Pith
show 21 more
- [21] Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context Pretrain- ing: Language Modeling Beyond Document Boundaries. InInternational Conference on Lea... on Pith
- [22] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James L. McClelland, and Felix Hill. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. In Advances in Neural Information Processing Systems , volume 35, ... on Pithdoi:10.52202/068431-1371
- [23] Akari Haga, Akiyo Fukatsu, Miyu Oba, Arianna Bisazza, and Yohei Oseki. BabyLM Challenge: Exploring the effect of variation sets on language model training efficiency. InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning , pages 252–261. A... on Pith
- [24] Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. Parallel structures in pre-training data yield in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8582–8592. Asso- ciation ... on Pithdoi:10.18653/v1/2024.acl-long.465
- [25] Nicolas Zucchet, Francesco D’Angelo, Andrew Kyle Lampinen, and Stephanie Chan. The emergence of sparse attention: impact of data distribution and benefits of repetition. InAdvances in Neural Information Processing Systems, volume 38, pages 39439–39476. Curran Associates, Inc.,... on Pithdoi:10.52202/085713-1179
- [26] Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal Abstractions of Neural Networks. In Advances in Neural Information Processing Systems , volume 34, pages 9574–9586. Curran Associates, Inc., 2021. URLhttps://proceedings.neurips.cc/paper_files/paper/202 1/f... on Pith
- [27] Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. InInternational Conference on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=Hf17y6u9BC on Pith
- [28] Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The Transient Nature of Emergent In-Context Learning in Transformers. In Advances in Neural In- formation Processing Systems , volume 36, pages 27801–27819. Curran Associates, Inc., 2023. do... on Pithdoi:10.52202/075280-1208
- [29] Suraj Anand, Michael A. Lepori, Jack Merullo, and Ellie Pavlick. Dual process learning: Controlling the use of in-context vs. in-weights strategies with weight forgetting. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://proceedings.iclr... on Pith
- [30] Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked languagemodeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2985–3000. Association for Computational Linguisti... on Pithdoi:10.18653/v1/2023.eacl-main.217
- [31] Zhizhong Li and Derek Hoiem. Learning without forgetting. InComputer Vision – ECCV 2016 , volume 9908 ofLecture Notes in Computer Science , pages 614–629. Springer, 2016. doi: 10.1007/97 8-3-319-46493-0_37. URL https://link.springer.com/chapter/10.1007/978-3-319-46493-0 _37 on Pithdoi:10.1007/97
- [32] Jiahai Feng and Jacob Steinhardt. How do Language Models Bind Entities in Context? InInter- national Conference on Learning Representations , 2024. URL https://openreview.net/forum?i d=zb3b6oKO77 on Pith
- [33] Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Representational Analysis of Binding in Language Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17468–17493. Association for Computational Linguistics, 2024. doi: 10.18653... on Pithdoi:10.18653/v1/2024.emnlp
- [34] Shuxing Yang, Fujia Chen, Rui Zhao, Junyao Wu, Yize Wang, Haiyao Luo, Ning Han, Qiaolu Chen, Yuze Hu, Wenhao Li, Mingzhu Li, Hongsheng Chen, and Yihao Yang. End-to-end autonomous scientific discovery on a real optical platform.arXiv preprint arXiv:2604.27092, 2026. URLhttps: /... on PithPith's reviewarXiv
- [35] Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research.Nature, 651:914–919, 2026. doi: 10.1038/s41586-026-10265-5. URL https://www.nature.com/articles/s41586-026-10265-5 on Pithdoi:10.1038/s41586-026-10265-5
- [36] Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation. In First Conference on Language Modeling , on Pith
- [37] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel machine: Open- ended evolution of self-improving agents. InThe Fourteenth International Conference on Learning Qiushi Engine 57 Representations, 2026. URL https://openreview.net/forum?id=pUpzQZTvGY on Pith
- [38] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023. URLhttps://proceedings.neurips.cc/paper_files/paper/2023/ has... on Pith
- [39] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, et al. Accelerating scientific discovery with Co-Scientist. Nature, 655:487–496, 2026. doi: 10.1038/s41586-026-10644-y. URL https://www.na ture.com/articles/s41586-026-10644-y on Pithdoi:10.1038/s41586-026-10644-y
- [40] Mingguang Chen, Licheng Wang, and Bo Qu. Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops.arXiv preprint arXiv:2607.07663, 2026. URLhttps: //arxiv.org/abs/2607.07663v2. Qiushi Engine 58 on PithPith's reviewarXiv
- [2024] URL https://openreview.net/forum?id=46Zgqo4QIU on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment