Pith. sign in

REVIEW 2 major objections 3 minor 64 references

FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation

T0 review · 2 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FormulaSPIN claims that spreadsheet formula generation can self-improve beyond static supervised data by playing a two-player game against its own previous version, using formula execution as an automatic judge, reaching 74.9% exact match a

desk verdict FormulaSPIN has a genuinely useful empirical method with a solid evaluation, but the central convergence claim is asserted, not proven, and is likely false as stated. read the letter →

arxiv 2607.19354 v1 pith:55LMMOU6 submitted 2026-05-21 cs.AI

Cy Xie This is my paper
classification cs.AI
keywords spreadsheetformulasnaturallanguagetoformulaself-playfine-tuningexecutionfeedbackadaptivecurriculumexecution-basedvotinggenerationsemanticequivalence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that spreadsheet formula generation can escape the ceiling of static supervised data by having a model play against its own previous version, using formula execution as an automatic judge. It proposes FormulaSPIN, a self-play fine-tuning loop in which the current model is trained to prefer ground-truth formulas over formulas the previous model generated, but with a crucial adjustment: execution results sort candidate formulas into Trivial, Coarse, and Fine categories, and only semantic errors get full negative weight. This avoids the contradictory gradients that arise when two different formulas compute the same result, such as SUM(A1:A5) versus A1+A2+A3+A4+A5, where one may be the reference in one example and the penalized output in another. The paper reports that after three or four iterations, without any additional annotations, FormulaSPIN reaches 74.9% exact match and 87.1% execution accuracy on NL2Formula-70K, outperforming plain supervised fine-tuning, continued training, and preference-optimized models. A sympathetic reader would care because it suggests a general route for tasks with cheap deterministic verification to self-improve from scarce data without human preference labels.

What carries the argument

The load-bearing object is the execution-filtered self-play loss, a logistic loss over the relative log-probability of the reference formula versus the opponent's sample, weighted by w(f') = 1 for Coarse samples, βmed for Fine samples, and 0 for Trivial samples. The adaptive curriculum weight βmed = βmax * n_fine / (n_fine + n_coarse), with βmax = 0.25, lets the training signal shift from punishing wrong results to gently steering non-canonical-but-correct formulas toward idiomatic forms. The companion inference mechanism, ExecVote, groups sampled candidates by execution output and returns the most probable member of the largest group, turning semantic equivalence into a voting signal.

What would settle it

Train FormulaSPIN with a deliberately high fixed βmed, such as 0.25 from iteration zero, on a dataset where ground-truth formulas are verbose and the canonical form is execution-equivalent; if the model's probability of the canonical form falls while execution accuracy stays flat across iterations, the reweighted objective is not converging to the data distribution. A more direct check: compute the stationary condition of the loss and show whether the implied opponent distribution equals the previous model's distribution; if it is a reweighted measure, the claimed fixed-point theorem does not

Watch

Extended reading notes

Core claim

FormulaSPIN turns formula generation into a two-player game. At each iteration the previous model acts as opponent by sampling candidate formulas, and the main player is trained with a reweighted logistic loss over relative log-probabilities so that it assigns higher probability to the ground truth than to the opponent's sample. The central adjustment is formula-aware filtering: executing both the reference and the candidate on the table places each sample into Trivial (exact match or execution failure, weight 0), Coarse (wrong result, weight 1), or Fine (correct result but different syntax, weight βmed capped at 0.25). The weight βmed grows as the Fine/Coarse ratio rises, creating an automa

Load-bearing premise

The claim rests on the assumption that reweighting the self-play objective, by zeroing Trivial samples and down-weighting Fine ones, leaves the fixed point unchanged so the model still converges to the true data distribution; if reweighting shifts that fixed point, the iterative loop's improvement is not grounded in a convergence guarantee.

Editorial extensions

If this is right

  • Iterative self-play over a fixed supervised dataset yields consistent gains: +1.9% exact match at the first iteration, accumulating to +6.7% by the fourth iteration over the supervised baseline, while continuing supervised fine-tuning on the same data degrades execution accuracy by 1.9 points.
  • The largest gains are on complex formulas (+11.0% execution accuracy from the supervised base to the final iteration), suggesting that compositional patterns are discovered rather than memorized from static labels.
  • Training-time gains persist out of distribution: FormulaSPIN reaches 69.1% exact match and 78.5% execution accuracy on the Sheetpedia set, and adding K=10 candidate voting raises execution accuracy to 84.9%.
  • The adaptive curriculum is necessary: removing it costs 1.9% exact match, and fixed weight schedules underperform the adaptive schedule.
  • Execution-based voting without a trained verifier gives a reliable accuracy boost at modest inference cost, about 1.7x for K=10 candidates.
  • If the reweighted objective truly preserves the fixed point, the training loop should plateau after 3-4 iterations, matching the paper's reported convergence pattern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A reader should treat the convergence claim as an open point: the reweighted objective's opponent is a reweighted version of the previous model's distribution, not that distribution itself, so the fixed-point identity requires proof rather than direct inheritance from the vanilla self-play objective.
  • Inference: The same execution-filtering recipe should transfer to other domains with cheap deterministic oracles, but the transfer is not automatic: in domains where distinct correct outputs rarely produce identical observable results, the Coarse/Fine distinction collapses and the method reduces to ordinary self-play.
  • Inference: The paper's own manual audit that roughly 18% of ground-truth formulas are non-canonical suggests that exact-match is partly a style-matching score; an evaluation that accepts any execution-equivalent formula would likely show higher true semantic accuracy.
  • Inference: A direct ablation worth running is to set the Fine-sample weight to zero for all iterations. If performance still climbs across iterations, the stylistic-nudging term is not the source of the gains; if it stops climbing, the Fine samples carry real learning signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. FormulaSPIN proposes a self-play fine-tuning method for natural-language-to-spreadsheet-formula generation. Starting from an SFT model, each iteration samples candidate formulas from the previous model, filters/excludes them by execution outcome, and categorizes them as Trivial, Coarse, or Fine. These samples are used in a weighted SPIN-style logistic objective (Eqs. 1–3) with an adaptive weight for Fine samples, and at inference ExecVote samples multiple formulas and votes by execution result. Experiments on NL2Formula-70K and Sheetpedia report state-of-the-art EM/EA, an ablation showing advantage over vanilla SPIN, and consistent gains across base models. The paper claims that the execution-aware reweighting preserves SPIN's convergence guarantee, but this is asserted without proof.

Significance. If the empirical results hold, FormulaSPIN is a practically useful contribution: it uses execution feedback to avoid contradictory gradients in self-play, introduces an adaptive curriculum, and provides a training-free voting mechanism. The paper uses public benchmarks, open base models, and releases code, and the multi-base-model and ablation evidence strengthen the empirical case. However, the theoretical anchor is currently overclaimed: the convergence guarantee in §3.2 is unsupported and, as stated, is likely false. The empirical contribution may survive if the guarantee is retracted and the objective is presented as a heuristic, but the present framing is not acceptable. There is also an internal inconsistency in the reported iteration-0 numbers that must be reconciled.

major comments (2)
  1. [§3.2, Eqs. (1)–(3)] The statement that "the SPIN convergence guarantee remains intact" is asserted without proof and is not implied by Chen et al.'s theorem. In Eq. (1) the opponent draw f' ~ pθt is multiplied by w(f') from Eq. (2), and the Fine/Coarse classification (hence w) depends on f and T through E(f',T)=E(f,T); βmed in Eq. (3) is a further global data-dependent constant. The effective negative distribution is pθt(f')w(f,f'), which is neither pθt nor normalized. Degenerate case: if all opponent samples are Trivial or Fine with βmed=0, then L(θ;θt)=0 for every θ, so the global optimum set is the whole simplex, contradicting the claimed pθ=p_data iff. More generally, no fixed-point analysis of the weighted logistic loss is provided. Because this claim is the stated reason that iterative self-play "drives the model toward the target distribution," it is load-bearing. Please prove the claim or remove it
  2. [Table 1 vs. Table 8] Table 1 reports FormulaSPIN(t0)=70.1 EM and describes this as +1.9% over the SFT base at iteration 0, while Table 8 reports FormulaSPIN iter0=68.5 EM. Later iterations agree (72.5/74.1/74.9), so this is not a simple labeling difference but an inconsistent iteration-0 value. The iteration-0 number is the first self-play gain and sets the starting point for the rest of the curve and for the vanilla-SPIN comparison. Please correct the discrepancy and re-check the claimed +1.9% and the analysis in §4.4 that depends on it.
minor comments (3)
  1. [§4.3, Table 2] The 7.0-point EA gain from K=10 is computed on only 128 test samples, while the main test set is far larger and Table 1 reports EA=87.1 for the same model but Table 2 greedy gives 86.7. Report the full-test result or clearly state the subset and provide error bars.
  2. [§4.4, Algorithm 1] The text credits "confidence-based filtering and rejection sampling" for the high quality of self-generated training formulas, but Algorithm 1 only skips exact matches and execution errors. Specify where such filtering occurs; otherwise the claimed virtuous cycle is not reproducible from the paper.
  3. [Eq. (2)] The weight is written w(f'), but Fine/Coarse classification depends on the pair (f,f') through E(f',T)=E(f,T). Make this dependence explicit, e.g., w(f,f'), to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central results are held-out empirical gains; §3.2's convergence claim is an unproven external citation transfer, not a circular reduction.

full rationale

The paper's derivation chain is not circular. FormulaSPIN's core claims—+6.7% EM over SFT and 74.9% EM/87.1% EA on NL2Formula-70K—are measured against held-out test sets, not reconstructed from training quantities. The execution filtering in Eq. (2) uses ground-truth execution results as labels to categorize generated samples (Trivial/Coarse/Fine); that is ordinary supervised label usage, not a fitted parameter being reported as a prediction. The adaptive weight βmed in Eq. (3) is computed from counts of Fine/Coarse samples on the training set and is ablated on the test set; no reported metric is equal to the training objective by construction. ExecVote is an inference-time selection rule over the model's own samples and execution-equivalence classes; it is not a fitted verifier. There are no self-citations carrying the argument; the only load-bearing theoretical citation is to SPIN (Chen et al., 2024b), by different authors, as external work. The one flagged weakness is in §3.2: the sentence 'the SPIN convergence guarantee remains intact: the global optimum of L is achieved if and only if pθ = p_data under this redefined target' asserts a result without proof for the reweighted objective. Since w(f') in Eq. (2) makes the effective negative distribution pθt(f')w(f') instead of pθt(f'), the external SPIN theorem may not apply. This is a soundness/correctness concern, not a circular reduction: no equation in the paper is shown to reduce to its own inputs, and the guarantee is not derived from the paper's own assumptions. Per the review rules, this concern is weighed as a correctness risk and does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The core method relies on several choices and background assumptions; the most significant free parameter is βmax, and the most fragile assertion is that SPIN's convergence theorem survives reweighting.

free parameters (5)
  • βmax (maximum Fine-sample weight) = 0.25
    Hand-chosen cap in Eq. (3) controlling how strongly execution-equivalent (Fine) negatives are pushed down; not derived from theory.
  • λ (SPIN logit scaling coefficient) = 0.1 for iterations 0–2; 5.0 for iteration 3
    Manually set per-iteration schedule; not justified by analysis.
  • Number of self-play iterations = 4
    Stopped at iteration 3/4 due to observed plateau (Figure 6); early stopping on validation.
  • ExecVote sample count K = 10
    Chosen from Table 2 as the best trade-off; results reported on 128 test samples only.
  • Sampling temperature for ExecVote = >1 (exact value unspecified)
    Temperature used to sample K candidates is not reported, making the inference setup incomplete.
assumptions (4)
  • ad hoc to paper SPIN convergence theorem remains valid for the reweighted objective
    Section 3.2 asserts the guarantee is intact after execution filtering and βmed down-weighting; no proof is given, and the effective opponent distribution is reweighted.
  • domain assumption Custom spreadsheet execution simulator matches real Excel semantics
    Section 4.1 mentions a 'Linux-based customized spreadsheet formula execution simulator'; no comparison against Microsoft Excel or Google Sheets is provided.
  • domain assumption Execution equivalence on the given table implies semantic equivalence of formulas
    Used to label candidates as Fine vs Coarse; can fail if the table does not exercise edge cases (e.g., zero-length ranges, text vs number coercion).
  • domain assumption Ground-truth formulas are a valid sample from the target distribution
    Standard supervised assumption, but Appendix B notes 18.3% of ground truths are non-canonical, which the method handles by down-weighting Fine samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation." pith.science (2026). https://pith.science/paper/55LMMOU6

@misc{pith2026260719354,
  author       = {Pith},
  title        = {Pith review of: FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55LMMOU6}},
  note         = {Machine review of arXiv:2607.19354}
}
read the original abstract

Spreadsheet applications are used by hundreds of millions worldwide, yet writing formulas remains a significant barrier. Existing approaches rely on static supervised data, which quickly saturates on limited annotations. In this paper, we introduce FORMULASPIN, a self-play framework that breaks the ceiling of supervised fine-tuning by enabling iterative self-improvement without any additional data. Vanilla SPIN fails on this task: it uniformly penalizes every non-matching output, so execution-equivalent alternatives are punished as negatives in one example while serving as ground truth in another, producing contradictory gradients. Our framework resolves this by exploiting formula generation's unique advantage: binary executability provides implicit supervision that separates semantic errors from valid stylistic variants. We frame training as a two-player game in which the main player learns to prefer ground-truth formulas over those from its previous version, while execution feedback sorts outputs into distinct granularities-enabling an adaptive curriculum that shifts from semantic correctness to stylistic refinement. To further increase accuracy, we incorporate ExecVote, a semantic-level voting mechanism that naturally handles multiple valid formulations. Experiments on multiple benchmarks demonstrate that FORMULASPIN achieves state-of-the-art performance, with 74.9% exact match and 87.1% execution accuracy on NL2FORMULA, matching models trained with additional preference annotations while outperforming both traditional SFT and frontier proprietary models. These findings underscore self-play's potential to tackle scarce data tasks and open the door to extending it beyond executable domains.

Figures

Figures reproduced from arXiv: 2607.19354 by the authors.

Figure 1
Figure 1. Performance comparison on NL2Formula￾70K (Zhao et al., 2024) and Sheetpedia (Tian et al., 2025). FormulaSPIN achieves state-of-the-art results on both benchmarks without additional annotations, outper￾forming the SFT baseline by +6.7% EM (Exact Match) / +6.0% EA (Execution Accuracy) on NL2Formula￾70K and +5.7% EM / +6.8% EA on Sheetpedia, while matching SFT-DPO trained with GPT-4.1-labeled pref￾erence pairs as well … view at source ↗
Figure 2
Figure 2. The FormulaSPIN framework. The model iteratively self-improves via a two-player game, utilizing Execution Filtering to construct an adaptive curriculum that progresses from semantic correctness (Coarse samples) to stylistic refinement (Fine samples). At inference, an execution-based voting strategy selects the final formula based on execution agreement among sampled candidates. reference formula f than to f ′ . As s… view at source ↗
Figure 3
Figure 3. Contradictory gradients under uniform weight [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Execution filtering and the Formula-Aware Self-Play objective. The opponent pθt generates f ′ , which is sorted by executing against T : Trivial sam￾ples (exact matches or execution failures) are discarded; Coarse samples (wrong result) receive full weight to teach sem…
Figure 5
Figure 5. Figure 5: Execution Accuracy by formula complexity. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance progression across iterations on [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An example where SFT produces a legacy array formula requiring special entry mode, while For￾mulaSPIN generates the modern idiomatic function. Error Analysis. We conduct stratified ran￾dom sampling (300 examples each from Sim￾ple, Medium, and Complex buckets) on For￾mu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 12 linked inside Pith

  1. [1]

    2025 , url=

    Claude 3.7 Sonnet and Claude Code , author=. 2025 , url=

    2025

  2. [2]

    2025 , url=

    Gemini 2.5: Our most intelligent AI model , author=. 2025 , url=

    2025

  3. [3]

    arXiv preprint arXiv:2501.12948 , year=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  4. [4]

    , booktitle=

    Price, Patti J. , booktitle=. Evaluation of Spoken Language Systems: The. 1990 , publisher=

    1990

  5. [5]

    Proceedings of the National Conference on Artificial Intelligence (AAAI) , pages=

    Learning to Parse Database Queries Using Inductive Logic Programming , author=. Proceedings of the National Conference on Artificial Intelligence (AAAI) , pages=. 1996 , url=

    1996

  6. [6]

    Proceedings of the European Conference on Machine Learning (ECML) , pages=

    Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing , author=. Proceedings of the European Conference on Machine Learning (ECML) , pages=. 2001 , url=

    2001

  7. [7]

    Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) , pages=

    Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars , author=. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) , pages=. 2005 , url=

    2005

  8. [8]

    Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Learning Synchronous Grammars for Semantic Parsing with Lambda Calculus , author=. Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=. 2007 , url=

    2007

Show all 64 references
  1. [9]

    Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Semantic Parsing via Paraphrasing , author=. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=. 2014 , url=

    2014

  2. [10]

    Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Language to Logical Form with Neural Attention , author=. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=. 2016 , url=

    2016

  3. [11]

    Zhong, Victor and Xiong, Caiming and Socher, Richard , journal=. Seq2. 2017 , url=

    2017

  4. [12]

    2018 , url=

    Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir , booktitle=. 2018 , url=

    2018

  5. [13]

    , booktitle=

    Yu, Tao and Zhang, Rui and Er, Heyang and Li, Suyi and Xue, Eric and Pang, Bo and Lin, Xi Victoria and Tan, Yi Chern and Shi, Tianze and Li, Zihan et al. , booktitle=. 2019 , url=

    2019

  6. [14]

    Proceedings of the 38th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) , pages=

    Automating String Processing in Spreadsheets Using Input-Output Examples , author=. Proceedings of the 38th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) , pages=. 2011 , url=

    2011

  7. [15]

    Communications of the ACM , volume=

    Spreadsheet Data Manipulation Using Examples , author=. Communications of the ACM , volume=. 2012 , url=

    2012

  8. [16]

    ACM SIGPLAN Notices , volume=

    Synthesis from Examples , author=. ACM SIGPLAN Notices , volume=. 2012 , url=

    2012

  9. [17]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Neuro-Symbolic Program Synthesis , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  10. [18]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  11. [19]

    2021 , url=

    Chen, Xinyun and Maniatis, Petros and Singh, Rishabh and Sutton, Charles and Dai, Hanjun and Lin, Max and Zhou, Denny , booktitle=. 2021 , url=

    2021

  12. [20]

    2021 , url=

    Wang, Zhiruo and Dong, Haoyu and Jia, Ran and Li, Jia and Fu, Zhiyi and Han, Shi and Zhang, Dongmei , booktitle=. 2021 , url=

    2021

  13. [21]

    2022 , url=

    Cheng, Zhoujun and Dong, Haoyu and Wang, Zhiruo and Jia, Ran and Guo, Jiaqi and Gao, Yan and Han, Shi and Lou, Jian-Guang and Zhang, Dongmei , booktitle=. 2022 , url=

    2022

  14. [22]

    2022 , url=

    Liu, Qian and Chen, Bei and Guo, Jiaqi and Ziyadi, Morteza and Lin, Zeqi and Chen, Weizhu and Lou, Jian-Guang , booktitle=. 2022 , url=

    2022

  15. [23]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    TableFormer: Robust Transformer Modeling for Table-Text Encoding , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , pages=. 2022 , url=

    2022

  16. [24]

    2024 , url=

    Zhao, Wei and Hou, Zhitao and Wu, Siyuan and Gao, Yan and Dong, Haoyu and Han, Shi and Zhang, Dongmei , booktitle=. 2024 , url=

    2024

  17. [25]

    Advances in Neural Information Processing Systems , year=

    Sheetpedia: A 300K-Spreadsheet Corpus for Spreadsheet Intelligence and LLM Fine-Tuning , author=. Advances in Neural Information Processing Systems , year=

  18. [26]

    Neural Computation , volume=

    Long Short-Term Memory , author=. Neural Computation , volume=. 1997 , url=

    1997

  19. [27]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=. 2020 , url=

    2020

  20. [28]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  21. [29]

    arXiv preprint arXiv:2112.09332 , year=

    WebGPT: Browser-assisted Question-answering with Human Feedback , author=. arXiv preprint arXiv:2112.09332 , year=

    arXiv

  22. [30]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  23. [31]

    Competition-level Code Generation with

    Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R. Competition-level Code Generation with. Science , volume=. 2022 , publisher=

    2022

  24. [32]

    International Conference on Learning Representations (ICLR) , year=

    Synchromesh: Reliable Code Generation from Pre-trained Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  25. [33]

    arXiv preprint arXiv:2302.13971 , year=

    Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  26. [34]

    , journal=

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela et al. , journal=. The. 2024 , url=

    2024

  27. [35]

    arXiv preprint arXiv:2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    arXiv

  28. [36]

    Temporal Difference Learning and

    Tesauro, Gerald , journal=. Temporal Difference Learning and. 1995 , url=

    1995

  29. [37]

    , journal=

    Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian et al. , journal=. Mastering the Game of. 2017 , publisher=

    2017

  30. [38]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal Policy Optimization Algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:1710.03748 , year=

    Emergent Complexity via Multi-Agent Competition , author=. arXiv preprint arXiv:1710.03748 , year=

    arXiv

  32. [40]

    Le, Hung and Wang, Yue and Gotmare, Akhilesh Deepak and Savarese, Silvio and Hoi, Steven C. H. , booktitle=. 2022 , url=

    2022

  33. [41]

    2022 , url=

    Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah , booktitle=. 2022 , url=

    2022

  34. [42]

    arXiv preprint arXiv:2312.06585 , year=

    Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models , author=. arXiv preprint arXiv:2312.06585 , year=

    arXiv

  35. [43]

    arXiv preprint arXiv:2401.01335 , year=

    Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models , author=. arXiv preprint arXiv:2401.01335 , year=

    arXiv

  36. [44]

    arXiv preprint arXiv:2408.01072 , year=

    A Survey on Self-play Methods in Reinforcement Learning , author=. arXiv preprint arXiv:2408.01072 , year=

  37. [45]

    Advances in Neural Information Processing Systems , volume=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , volume=. 2022 , url=

    2022

  38. [46]

    , journal=

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron et al. , journal=. Constitutional. 2022 , url=

    2022

  39. [47]

    International Conference on Learning Representations (ICLR) , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  40. [48]

    Advances in Neural Information Processing Systems , year=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. Advances in Neural Information Processing Systems , year=

  41. [49]

    2023 , url=

    Lee, Harrison and Phatale, Samrat and Mansoor, Hassan and Lu, Kellie and Mesnard, Thomas and Bishop, Colton and Carbune, Victor and Rastogi, Abhinav , journal=. 2023 , url=

    2023

  42. [50]

    Transactions on Machine Learning Research , year=

    Execution-based Code Generation using Deep Reinforcement Learning , author=. Transactions on Machine Learning Research , year=

  43. [51]

    2023 , url=

    Liu, Jiate and Xia, Yiqin and Lu, Yichen and Ao, Xiang and Zhou, Kun and Zhang, Xia and Zhang, Yang and Wang, Junlan and Wen, Ji-Rong , journal=. 2023 , url=

    2023

  44. [52]

    International Conference on Learning Representations (ICLR) , year=

    Teaching Large Language Models to Self-Debug , author=. International Conference on Learning Representations (ICLR) , year=

  45. [53]

    , journal=

    Guo, Shangmin and Zhang, Biao and Liu, Tianlin and Liu, Tianqi and Khalman, Misha and Llinares, Felipe and Rame, Alexandre and Mesnard, Thomas and Zhao, Yao et al. , journal=. Direct Language Model Alignment from Online. 2024 , url=

    2024

  46. [54]

    Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for

    Xiong, Wei and Dong, Hanze and Ye, Chenlu and Wang, Ziqi and Zhong, Han and Ji, Heng and Jiang, Nan and Zhang, Tong , journal=. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for. 2024 , url=

    2024

  47. [55]

    arXiv preprint arXiv:2401.10020 , year=

    Self-Rewarding Language Models , author=. arXiv preprint arXiv:2401.10020 , year=

    arXiv

  48. [56]

    Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , journal=. Scaling. 2024 , url=

    2024

  49. [57]

    Advances in Neural Information Processing Systems (NeurIPS) , pages=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , pages=. 2024 , url=

    2024

  50. [58]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Curriculum Learning for Small Code Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  51. [59]

    2021 , url=

    Scholak, Torsten and Schucher, Nathan and Bahdanau, Dzmitry , booktitle=. 2021 , url=

    2021

  52. [60]

    2025 , url=

    Cao, Lang and Xu, Jingxian and Liu, Hanbing and Wang, Jinyu and Zhou, Mengyu and Dong, Haoyu and Han, Shi and Zhang, Dongmei , journal=. 2025 , url=

    2025

  53. [61]

    Advances in Neural Information Processing Systems , volume=

    Learning to summarize with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  54. [62]

    2025 , url=

    Zhang, Yuhao and Duan, Shaoming and Su, Jinhang and Liu, Chuanyi and Han, Peiyi , journal=. 2025 , url=

    2025

  55. [63]

    arXiv preprint arXiv:2210.11610 , year=

    Large language models can self-improve , author=. arXiv preprint arXiv:2210.11610 , year=

    arXiv

  56. [64]

    GitHub repository , year=

    Stanford Alpaca: An instruction-following LLaMA model , author=. GitHub repository , year=

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.