Pith. sign in

REVIEW 2 major objections 6 minor 15 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

RL composes primitive skills into new reasoning strategies

2026-07-09 03:43 UTC pith:TQPC6CUX

load-bearing objection The stress-test concern about generation budget is a real gap, but the mechanistic findings survive it. the 2 major comments →

arxiv 2607.07646 v1 pith:TQPC6CUX submitted 2026-07-08 cs.AI cs.CL

RL Post-Training Builds Compositional Reasoning Strategies

Azwar Abdulsalam , Nishil Patel , Andrew Saxe This is my paper
classification cs.AI cs.CL
keywords reinforcement learningcompositional reasoningpost-trainingrewrite grammarprocedural chunkingGRPOrejection fine-tuningskill composition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether reinforcement learning (RL) post-training merely amplifies behaviors already latent in a base model, or whether it can compose primitive skills into genuinely new higher-level reasoning strategies. The authors study this in a fully observable rewrite-grammar environment where every generated step can be audited against a known rule set. They find that RL does more than reweight: it first strengthens primitive reduction steps, then discovers valid composed procedures—sequential compositions that collapse chains of primitive steps and parallel compositions that combine independent steps simultaneously. These compositions are not isolated lucky samples; they are reused and consolidated into a stable repertoire. Crucially, RL achieves this selectivity through within-prompt contrast between successful and failed rollouts, whereas rejection fine-tuning (RFT) clones whatever appears in successful trajectories, including invalid shortcuts. The paper also shows that strategy emergence depends not on how often the base model sees primitive operations, but on whether pretraining organizes them into sustained reduction procedures that RL can later compress.

Core claim

The central discovery is a phased compositional mechanism: RL post-training first strengthens primitive contractions, then discovers and consolidates valid composed procedures (macro and parallel contractions) that were never present in the pretraining data. This mechanism is driven by selectivity, not exploration volume—GRPO's within-group success-failure contrast suppresses spurious rewrites that RFT clones—and it is gated by whether pretraining provides chained reduction procedures, not merely exposure to primitive facts. The result is that RL solves held-out problems the base model cannot solve even at 16x the sampling budget.

What carries the argument

The rewrite-grammar environment with globally unique right-hand sides enables exact auditing of every generated rewrite into four categories: primitive, macro (sequential composition), parallel (parallel composition), and spurious. The GRPO finite-group analysis shows that the sign of the feature-level update is determined by the contrast between feature frequency in successful versus failed completions from the same prompt, which RFT cannot replicate because it discards failures.

Load-bearing premise

The central claim that RL builds compositional reasoning strategies rests on the rewrite-grammar environment being representative of reasoning more broadly. The grammar has globally unique right-hand sides, making primitive contractions unambiguous—a structural property that does not hold in natural-language reasoning, where intermediate steps cannot be mechanically classified as valid or invalid. Whether the same phased chunking mechanism operates in LLM post-training on数学或码

What would settle it

If one constructed a grammar where right-hand sides are not globally unique (so primitive contractions are ambiguous), and the same phased compositional emergence failed to appear, it would suggest that the mechanism depends critically on the environment's unambiguous structure rather than on a general property of RL post-training.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If RL's compositional chunking mechanism generalizes beyond rewrite grammars, then RL post-training on math or code reasoning may be building reusable multi-step procedures rather than merely amplifying existing solution probabilities, which would change how we interpret capability gains from RLVR.
  • The finding that RFT clones invalid shortcuts alongside useful ones suggests that imitation-style post-training may embed spurious reasoning patterns that are invisible without process-level auditing, with potential downstream reliability consequences.
  • The pretraining substrate result implies that curriculum design for pretraining should optimize for sustained procedural chaining rather than raw exposure to primitive operations, if the goal is to enable later RL to discover compositional strategies.
  • The phased emergence pattern—primitive strengthening before compositional discovery—suggests that early RL training curves may understate eventual capability gains, and that evaluation protocols with short training horizons may miss compositional strategy formation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The selectivity mechanism identified here—GRPO's within-prompt contrast suppressing spurious actions—could potentially be tested in natural-language reasoning settings by classifying intermediate steps as valid or invalid using a verifier model, even when a ground-truth grammar is unavailable.
  • If the pretraining substrate result generalizes, it predicts that two base models with identical primitive skill coverage but different procedural chaining structure should diverge significantly under RL post-training, which is testable with controlled pretraining ablations on larger models.
  • The consolidation phase where discovered macro rules become reused suggests that RL may be implicitly learning a form of option-like hierarchical policy structure inside a flat sequence model, which could be probed by examining attention patterns or internal representations associated with consolidated macro rules.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper studies whether RL post-training can compose primitive skills into higher-level strategies, using a controlled rewrite-grammar environment where every generated rewrite can be audited against a known grammar. A Transformer is pretrained from scratch on primitive rewrite chains and then post-trained via GRPO (or rejection fine-tuning) on a goal-directed contraction task with binary final-answer reward. The paper reports four main findings: (1) RL discovers valid non-primitive strategies—macro contractions (sequential compositions) and parallel contractions—that the base model was never exposed to during pretraining; (2) these strategies emerge in a phased manner (primitive strengthening first, then compositional chunking) and are consolidated into a reusable repertoire; (3) RL outperforms RFT not through greater exploration volume but through selectivity—RFT produces many spurious shortcuts while RL suppresses them; and (4) pretraining structure (specifically, chained reduction procedures rather than raw contraction frequency) gates the emergence of compositional strategies. A finite-group analysis of GRPO explains the selectivity mechanism via within-prompt success–failure contrast.

Significance. The paper makes a genuine methodological contribution by providing a fully observable testbed in which post-training strategies can be exactly classified rather than inferred from aggregate metrics. The four-way taxonomy (primitive, macro, parallel, spurious) and the trace-level auditability are well-motivated and cleanly executed. The GRPO feature-count derivation (Section 5) is a compact and insightful formalization of why same-prompt contrast suppresses spurious actions, and the matched-fraction pretraining control (Section 7) is a thoughtful experimental design that separates marginal exposure from procedural structure. The phased-emergence and consolidation findings are the strongest mechanistic results. The scope is appropriately limited to a synthetic grammar, and the authors acknowledge this in Section 8.

major comments (2)
  1. Section 6, Table 1: The 'beyond base model' claim conflates sampling budget (pass@k) with generation budget (max tokens per attempt). Table 1 varies k (64, 256, 1024) but every attempt is capped at 256 tokens. Difficulty levels are explicitly defined by whether the primitive contraction solution fits within 256 tokens (Section 3.2): Difficulty 2 needs one extra primitive step, Difficulty 3 needs two, etc. Thus Buckets 4–5 are solvable with primitives the base model already possesses—they just require more tokens per attempt. The base model scores 0% on Buckets 4–5 at pass@1024, but this may simply reflect the 256-token cap rather than a capability gap. If the base model were given a 1024- or 2048-token generation budget, it might solve a substantial fraction of Buckets 4–5 using only primitive contractions. This would reframe the central 'beyond base model' claim (contribution 3, Section
  2. 6, and the abstract) from 'RL expands the capability frontier' to 'RL learns token-efficient shortcuts within a fixed generation budget.' The mechanistic findings about compositional strategy emergence (Sections 4–5, 7) would still stand, but the frontier-expansion framing is currently unsupported without varying the generation budget. The authors should either (a) evaluate the base model at pass@k with a larger generation budget (e.g., 512, 1024 tokens) to test whether primitive-only solutions become accessible, or (b) reframe the claim to explicitly state that the expansion is relative to a fixed generation budget. As written, the claim is stronger than what the experiment supports.
minor comments (6)
  1. Section 3.2: The mapping from difficulty levels to token counts is stated qualitatively ('one additional primitive contraction step beyond the budget') but never quantified. How many tokens does one primitive contraction step consume on average, and how many extra steps do Buckets 4–6 require? This would help readers assess the generation-budget concern above.
  2. Figure 2 caption: 'appliexd' should be 'applied'.
  3. Section 5: The GRPO feature-count derivation uses the notation k+ and k− for counts of successful/failed completions containing feature F, but k is also used for the pass@k sampling budget throughout the paper. Consider disambiguating.
  4. Appendix B: The GRPO and RFT hyperparameters are listed, but the total number of post-training iterations (20k, inferred from figures) and batch size are not explicitly stated in the main text or appendix.
  5. Section 8: The discussion acknowledges scope limitations ('we do not claim that large language models must exhibit the same phases or taxonomy'), but the globally unique right-hand sides (Section 3) make primitive contractions unambiguous—a property that does not hold in natural-language reasoning. This limitation could be stated more prominently, as it bears on the generalizability of the central claim.
  6. Table 1: The 'Overall' column reports 17.50% at pass@1024, but the bucket-level numbers (62.50%, 18.75%, 6.25%, 0%, 0%) average to 17.50%, confirming equal bucket sizes. Consider noting that each bucket has 16 problems, as stated in the caption, to make the averaging transparent.

Simulated Author's Rebuttal

2 responses · 0 unresolved

The referee raises a single major comment regarding whether the 'beyond base model' claim in Section 6 conflates sampling budget (pass@k) with generation budget (max tokens per attempt). The referee correctly notes that all attempts are capped at 256 tokens, and that difficulty levels are defined relative to this cap. We agree this is a valid concern and will address it by running the suggested experiments and revising the framing accordingly.

read point-by-point responses

  1. Referee: Section 6, Table 1: The 'beyond base model' claim conflates sampling budget (pass@k) with generation budget (max tokens per attempt). Table 1 varies k (64, 256, 1024) but every attempt is capped at 256 tokens. Difficulty levels are explicitly defined by whether the primitive contraction solution fits within 256 tokens (Section 3.2): Difficulty 2 needs one extra primitive step, Difficulty 3 needs two, etc. Thus Buckets 4–5 are solvable with primitives the base model already possesses—they just require more tokens per attempt. The base model scores 0% on Buckets 4–5 at pass@1024, but this may simply reflect the 256-token cap rather than a capability gap. If the base model were given a 1024- or 2048-token generation budget, it might solve a substantial fraction of Buckets 4–5 using only primitive contractions. This would reframe the central 'beyond base model' claim from 'RL expands the cap

    Authors: The referee is correct that Table 1 varies the sampling budget k while holding the generation budget fixed at 256 tokens, and that the difficulty levels are defined relative to this 256-token cap. We agree that this leaves open the possibility that the base model could solve Buckets 4–5 with primitive-only solutions if given a larger per-attempt token budget. This is a genuine gap in the experimental support for the frontier-expansion framing as currently stated. We will address this in two ways. First, we will run the experiment the referee suggests: evaluating the base model at pass@k with generation budgets of 512 and 1024 tokens on Buckets 4–5. If the base model can solve a substantial fraction of these buckets with primitive-only solutions under a larger token budget, we will reframe the claim as the referee proposes in option (b): explicitly stating that the expansion is relative to a fixed generation budget, and that RL's contribution is discovering token-efficient compositional shortcuts rather than expanding the capability frontier in an absolute sense. Second, regardless of the outcome of that experiment, we will revise the abstract and Section 6 to make the fixed-generation-budget qualifier explicit in the claim, since the current phrasing ('RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets') does not currently make clear that the generation budget is held constant. We note that the mechanistic findings in Sections 4–5 and 7 (phased compositional emergence, consolidation, selectivity over RFT, pretraining structure gating) are independent of this framing and would stand either way. The referee's point specifically concerns the frontier-expansion claim in Contribution 3, not the strategy-discc revision: no

  2. Referee: ...capability frontier' to 'RL learns token-efficient shortcuts within a fixed generation budget.' The mechanistic findings about compositional strategy emergence (Sections 4–5, 7) would still stand, but the frontier-expansion framing is currently unsupported without varying the generation budget. The authors should either (a) evaluate the base model at pass@k with a larger generation budget (e.g., 512, 1024 tokens) to test whether primitive-only solutions become accessible, or (b) reframe the claim to explicitly state that the expansion is relative to a fixed generation budget. As written, the claim is stronger than what the experiment supports.

    Authors: We will adopt both options. We will run the larger-generation-budget experiment (option a) and report the results, and we will also revise the framing (option b) to make the fixed-generation-budget qualifier explicit throughout the abstract, Section 6, and the contribution list. If the larger-budget experiment shows that the base model can solve Buckets 4–5 with primitive-only solutions, we will reframe the claim as the referee suggests. If it shows that the base model still cannot solve them (e.g., because the model's primitive contraction competence is itself unreliable at the needed chain lengths), the claim would be better supported but we would still state the generation-budget condition explicitly for clarity. In either case, the revised manuscript will not claim frontier expansion without the qualification the referee identifies. revision: no

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims are supported by independent experiments and externally defined grammar rules, not by fitted parameters or self-citation chains.

full rationale

The paper's derivation chain is largely self-contained. The composed strategies (macro and parallel contractions) are defined by the known grammar (Section 3), not by a fitted parameter or by the target result. The 'beyond base model' claim (Section 6, Table 1) is evaluated against an external benchmark: the pretrained model's own pass@k at increasing sampling budgets. The pretraining ablation (Section 7, Figure 6) uses a matched-fraction control that is independently motivated by the need to separate contraction frequency from contraction chaining. The GRPO selectivity argument (Section 5) is derived from first principles (the score-function identity and group-relative advantage normalization) and does not reduce to a self-citation. The only minor concern is that the taxonomy of 'valid composed strategies' is defined by the same grammar used to generate the pretraining data, but this is a design choice for an auditable environment, not a circular derivation: the paper does not claim that RL discovers the grammar, only that it discovers valid shortcuts within it, and the base-model comparison establishes that these shortcuts are not merely amplified from pretraining. No step in the chain reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities, particles, forces, or postulated objects. The 'macro rules' and 'parallel contractions' are classifications of existing grammar operations, not new entities. The free parameters are standard RL/GRPO hyperparameters and environment design choices, none of which are fitted to the target result. The axioms are either standard mathematical derivations (GRPO contrast, RFT alignment) or domain assumptions about the grammar environment. No ad-hoc-to-paper axioms are needed.

free parameters (9)
  • rho (contraction weight) = 4 (default), 1/2/4 (ablation)
    Controls local upweighting of contraction moves during pretraining. Not fitted to the target result but chosen as a design parameter; ablated in Section 7.
  • G (group size) = 4
    GRPO group size. Standard hyperparameter, not tuned to the result.
  • epsilon (clip ratio) = 0.2
    PPO/GRPO clip parameter. Standard value.
  • beta (KL coefficient) = 1e-3
    KL penalty to reference policy. Standard value.
  • temperature = 0.8
    Sampling temperature during RL. Standard value.
  • top-k = 5
    Top-k sampling during RL. Restrictive but not tuned to the result.
  • N (alphabet size) = 20
    Grammar alphabet size. Design choice.
  • max generation length = 256 tokens
    Generation budget that defines difficulty levels. Design choice.
  • training mixture = 60% D1, 20% D2, 20% D3
    RL training distribution over difficulty levels. Design choice.
axioms (5)
  • standard math GRPO with binary rewards provides within-prompt contrast that suppresses features enriched in failed completions
    Derived in Section 5 from the GRPO objective. The derivation is a feature-level approximation, as the authors note.
  • standard math On-policy RFT with binary rewards is first-order aligned with the policy gradient at theta=theta_old
    Derived in Section 5 using the score-function identity. Correct under stated assumptions.
  • domain assumption The rewrite grammar with globally unique right-hand sides makes primitive contractions unambiguous
    Stated in Section 3. This is a design choice that simplifies auditing but does not hold in natural-language reasoning.
  • domain assumption Difficulty defined by optimal primitive contraction length relative to the 256-token budget captures meaningful problem hardness
    Stated in Section 3.2. This is reasonable but ties difficulty to the specific budget choice.
  • domain assumption The phased emergence of macro then parallel contractions reflects a genuine mechanism rather than training dynamics artifacts
    Observed in Figure 2 over 3 seeds. The claim is empirical, not derived from first principles.

pith-pipeline@v1.1.0-glm · 16839 in / 3320 out tokens · 554255 ms · 2026-07-09T03:43:44.324039+00:00 · methodology

0 comments
read the original abstract

Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.

Figures

Figures reproduced from arXiv: 2607.07646 by Andrew Saxe, Azwar Abdulsalam, Nishil Patel.

Figure 1
Figure 1. Figure 1: Controlled environment and training pipeline. (a) Each source character maps to one or more globally unique right-hand-side strings. (b) Pretraining data consists of multi-step sequences in which the output of one step becomes the input to the next. (c) Each training triple encodes a single primitive expansion or contraction. (d) RL prompts are constructed by repeated forward expansion from a target symbol… view at source ↗
Figure 2
Figure 2. Figure 2: RL undergoes a delayed transition from primitive contractions to compressed valid shortcuts. Macro contractions accelerate and overtake primitive contractions around iteration 12,500; parallel contractions emerge later. Bottom examples show the structural analogy to algebraic simplification: primitive contractions correspond to local simplifications, macro contractions to reusable multi-step maneuvers, and… view at source ↗
Figure 3
Figure 3. Figure 3: RL discovers and consolidates reusable macro rules. (a) Number of newly discovered macro rules at each checkpoint. (b) Reuse count of previously discovered macro rules. (c) Reuse fraction among all macro actions. RL continues discovering new macro rules while increasingly reusing earlier ones, indicating consolidation into a stable repertoire. The group standard deviation is σ 2 = 1 G [PITH_FULL_IMAGE:fig… view at source ↗
Figure 4
Figure 4. Figure 4: RL discovers valid non-primitive strategies while sup￾pressing spurious shortcuts. (a,b) Counts of valid macro and paral￾lel contractions. (c) Count of spurious contractions. (d) Fraction of non-primitive contractions that are spurious. RFT generates many non-primitive actions, but they are mostly invalid; RL produces fewer spurious rewrites and increasingly concentrates behavior into valid macro and paral… view at source ↗
Figure 5
Figure 5. Figure 5: RL exhibits a delayed crossover over RFT, with the largest advantage on harder held-out problems. Pass@16 on con￾traction problems across five difficulty levels and overall. Mean over three seeds. RFT improves rapidly early in training but plateaus, while RL improves more gradually and continues climb￾ing through the end of training [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Strategy emergence depends on local contraction structure, not overall contraction frequency. (a) Solid bars (left axis): overall contraction rate. Hatched bars (right axis): contraction-after-contraction probability P(Ct = 1 | Ct−1 = 1). The matched-fraction control reaches a comparable overall rate to ρ = 2 but has substantially less chaining. (b,c) Macro and parallel contraction counts during RL post-tr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 11 internal anchors

  1. [1]

    Reasoning with Exploration: An Entropy Perspective

    Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W. X., Zhang, Z., and Wei, F. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

    Chu, T., Zhai, Y ., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q. V ., Levine, S., and Ma, Y . Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Let's Verify Step by Step

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

    Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y ., Kautz, J., and Dong, Y . Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 2025a. 8 RL Post-Training Builds Compositional Reasoning Strategies Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Und...

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    X., Zhang, Z., Wen, Z., Zhang, Z., and Zhou, J

    Tang, X., Zhan, Y ., Li, Z., Zhao, W. X., Zhang, Z., Wen, Z., Zhang, Z., and Zhou, J. Rethinking sample polarity in reinforcement learning with verifiable rewards.arXiv preprint arXiv:2512.21625,

    work page arXiv

  8. [8]

    Solving math word problems with process- and outcome-based feedback

    Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process- and outcome- based feedback.arXiv preprint arXiv:2211.14275,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Emergent hierarchical reasoning in llms through rein- forcement learning.arXiv preprint arXiv:2509.03646,

    Wang, H., Xu, Q., Liu, C., Wu, J., Lin, F., and Chen, W. Emergent hierarchical reasoning in llms through rein- forcement learning.arXiv preprint arXiv:2509.03646,

    work page arXiv

  10. [10]

    The invisible leash: Why rlvr may or may not escape its origin.arXiv preprint arXiv:2507.14843,

    Wu, F., Xuan, W., Lu, X., Liu, M., Dong, Y ., Harchaoui, Z., and Choi, Y . The invisible leash: Why rlvr may or may not escape its origin.arXiv preprint arXiv:2507.14843,

    work page arXiv

  11. [11]

    A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

    Xiong, W., Yao, J., Xu, Y ., Pang, B., Wang, L., Sahoo, D., Li, J., Jiang, N., Zhang, T., Xiong, C., and Dong, H. A minimalist approach to llm reasoning: From rejection sampling to reinforce.arXiv preprint arXiv:2504.11343,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    From f(x) and g(x) to f(g(x)): Llms learn new skills in rl by composing old ones.arXiv preprint arXiv:2509.25123,

    Yuan, L., Chen, W., Zhang, Y ., Cui, G., Wang, H., You, Z., Ding, N., Liu, Z., Sun, M., and Peng, H. From f(x) and g(x) to f(g(x)): Llms learn new skills in rl by composing old ones.arXiv preprint arXiv:2509.25123,

    work page arXiv

  13. [13]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

    Yue, Y ., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Zelikman, E., Wu, Y ., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning.arXiv preprint arXiv:2203.14465,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining

    Zhao, R., Meterez, A., Kakade, S., Pehlevan, C., Jelassi, S., and Malach, E. Echo chamber: Rl post-training am- plifies behaviors learned in pretraining.arXiv preprint arXiv:2504.07912,

    work page internal anchor Pith review Pith/arXiv arXiv