{"id":"1af1c578-8457-4fbf-9d78-eec64c777ffa","arxiv_id":"2607.07646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":9,"one_line_summary":"RL post-training composes primitive rewrite skills into reusable macro and parallel contraction strategies that solve problems inaccessible to the base model under large sampling budgets.","lead":"This paper shows that RL post-training can compose primitive skills into new higher-level reasoning strategies in a controlled rewrite-grammar environment, going beyond mere amplification of existing behaviors. The fully observable setup lets readers see exactly which strategies emerge and why, making it relevant to anyone studying how AI systems learn to reason.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 'beyond base model' claim conflates sampling budget with generation budget: the base model is evaluated at pass@1024 but always with 256-token generation, and the paper never tests whether a larger generation budget would let the base model solve Buckets 4–5 with primitive contractions alone.","rationale":"The reader identified generalizability beyond the rewrite-grammar setting as the weakest assumption. That is a legitimate scope concern, but it is external — about whether the mechanism transfers to natural-language reasoning. The concern I identify is internal: within the paper's own setting, the 'beyond base model' claim conflates two distinct budgets. The sampling budget (pass@k) is varied in Table 1, but the generation budget (256 tokens) is held fixed, and the difficulty levels are defined relative to that fixed budget. This means the base model's 0% on Buckets 4–5 could be an artifact of the token cap rather than a genuine capability gap. That said, this concern does not invalidate the paper's core mechanistic contribution. The phased emergence of macro/parallel contractions (Figure 2), their consolidation into a reusable repertoire (Figure 3), the selectivity advantage over RFT (Figure 4), and the pretraining ablation (Figure 6) are all independent of whether the base model could solve the problems with more tokens. The mechanism — RL discovering and consolidating valid compositions through contrastive selection — stands on its own. The concern specifically weakens the 'beyond base model' framing in Section 6 and the abstract, not the mechanistic claims about composition. The CONDITIONAL verdict remains appropriate: the paper makes a solid mechanistic contribution in a controlled setting, with the acknowledged scope limitation and the additional qualification that the 'frontier expansion' claim is specifically about finite-token-budget frontiers, not absolute capability.","tokens_in":13891,"tokens_out":8514,"duration_ms":546681,"concrete_test":"Evaluate the pretrained base model on Buckets 4–5 at pass@16 (or pass@64) with progressively larger generation budgets: 512, 1024, and 2048 tokens per attempt. If success rate rises above ~20% on Buckets 4–5 at 1024 tokens, the 'beyond base model' claim weakens to a token-efficiency claim and the abstract should be revised accordingly. If success remains near 0% even at 2048 tokens, the capability-gap framing is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's claim that 'RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets' (Section 6, Table 1) rests on the base model scoring 0% on Buckets 4–5 at pass@1024. But Table 1 varies only the sampling budget (number of attempts), not the generation budget (max tokens per attempt). Every attempt is still capped at 256 tokens. The difficulty levels are explicitly defined by whether the primitive contraction solution fits within 256 tokens (Section 3.2): Difficulty 2 needs one extra primitive step beyond the budget, Difficulty 3 needs two, etc. So Buckets 4–5 are solvable with primitives the base model already possesses — they just require more tokens per attempt. The base model fails not because it lacks the primitives but because each 256-token attempt is too short for the primitive solution, and it never produces valid shortcuts (macro/parallel contractions) because it was never exposed to them in pretraining. If the base model were given a 1024- or 2048-token generation budget, it might solve a substantial fraction of Buckets 4–5 using only primitive contractions. This would reframe the central claim from 'RL expands the capability frontier' to 'RL learns token-efficient shortcuts' — a weaker statement. The paper's title claim ('Builds Compositional Reasoning Strategies') would still be supported by the mechanistic findings (phased emergence, consolidation, selectivity), but the 'beyond base model' framing in the abstract and Section 6 would need qualification. The paper does not address this confound anywhere.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper studies whether RL post-training can compose primitive skills into higher-level strategies, using a controlled rewrite-grammar environment where every generated rewrite can be audited against a known grammar. A Transformer is pretrained from scratch on primitive rewrite chains and then post-trained via GRPO (or rejection fine-tuning) on a goal-directed contraction task with binary final-answer reward. The paper reports four main findings: (1) RL discovers valid non-primitive strategies—macro contractions (sequential compositions) and parallel contractions—that the base model was never exposed to during pretraining; (2) these strategies emerge in a phased manner (primitive strengthening first, then compositional chunking) and are consolidated into a reusable repertoire; (3) RL outperforms RFT not through greater exploration volume but through selectivity—RFT produces many spurious shortcuts while RL suppresses them; and (4) pretraining structure (specifically, chained reduction procedures rather than raw contraction frequency) gates the emergence of compositional strategies. A finite-group analysis of GRPO explains the selectivity mechanism via within-prompt success–failure contrast.","tokens_in":14039,"tokens_out":2982,"duration_ms":156732,"significance":"The paper makes a genuine methodological contribution by providing a fully observable testbed in which post-training strategies can be exactly classified rather than inferred from aggregate metrics. The four-way taxonomy (primitive, macro, parallel, spurious) and the trace-level auditability are well-motivated and cleanly executed. The GRPO feature-count derivation (Section 5) is a compact and insightful formalization of why same-prompt contrast suppresses spurious actions, and the matched-fraction pretraining control (Section 7) is a thoughtful experimental design that separates marginal exposure from procedural structure. The phased-emergence and consolidation findings are the strongest mechanistic results. The scope is appropriately limited to a synthetic grammar, and the authors acknowledge this in Section 8.","major_comments":[{"comment":"Section 6, Table 1: The 'beyond base model' claim conflates sampling budget (pass@k) with generation budget (max tokens per attempt). Table 1 varies k (64, 256, 1024) but every attempt is capped at 256 tokens. Difficulty levels are explicitly defined by whether the primitive contraction solution fits within 256 tokens (Section 3.2): Difficulty 2 needs one extra primitive step, Difficulty 3 needs two, etc. Thus Buckets 4–5 are solvable with primitives the base model already possesses—they just require more tokens per attempt. The base model scores 0% on Buckets 4–5 at pass@1024, but this may simply reflect the 256-token cap rather than a capability gap. If the base model were given a 1024- or 2048-token generation budget, it might solve a substantial fraction of Buckets 4–5 using only primitive contractions. This would reframe the central 'beyond base model' claim (contribution 3, Section","section":null},{"comment":"6, and the abstract) from 'RL expands the capability frontier' to 'RL learns token-efficient shortcuts within a fixed generation budget.' The mechanistic findings about compositional strategy emergence (Sections 4–5, 7) would still stand, but the frontier-expansion framing is currently unsupported without varying the generation budget. The authors should either (a) evaluate the base model at pass@k with a larger generation budget (e.g., 512, 1024 tokens) to test whether primitive-only solutions become accessible, or (b) reframe the claim to explicitly state that the expansion is relative to a fixed generation budget. As written, the claim is stronger than what the experiment supports.","section":null}],"minor_comments":[{"comment":"Section 3.2: The mapping from difficulty levels to token counts is stated qualitatively ('one additional primitive contraction step beyond the budget') but never quantified. How many tokens does one primitive contraction step consume on average, and how many extra steps do Buckets 4–6 require? This would help readers assess the generation-budget concern above.","section":null},{"comment":"Figure 2 caption: 'appliexd' should be 'applied'.","section":null},{"comment":"Section 5: The GRPO feature-count derivation uses the notation k+ and k− for counts of successful/failed completions containing feature F, but k is also used for the pass@k sampling budget throughout the paper. Consider disambiguating.","section":null},{"comment":"Appendix B: The GRPO and RFT hyperparameters are listed, but the total number of post-training iterations (20k, inferred from figures) and batch size are not explicitly stated in the main text or appendix.","section":null},{"comment":"Section 8: The discussion acknowledges scope limitations ('we do not claim that large language models must exhibit the same phases or taxonomy'), but the globally unique right-hand sides (Section 3) make primitive contractions unambiguous—a property that does not hold in natural-language reasoning. This limitation could be stated more prominently, as it bears on the generalizability of the central claim.","section":null},{"comment":"Table 1: The 'Overall' column reports 17.50% at pass@1024, but the bucket-level numbers (62.50%, 18.75%, 6.25%, 0%, 0%) average to 17.50%, confirming equal bucket sizes. Consider noting that each bucket has 16 problems, as stated in the caption, to make the averaging transparent.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The stress-test concern about the generation budget is the most important issue to flag. The paper's title claim ('Builds Compositional Reasoning Strategies') is supported by the mechanistic analysis regardless of this concern, but the 'beyond base model' framing in the abstract and Section 6 is currently stronger than the evidence warrants. A single additional experiment (base model at larger generation budget) or a reframing would resolve this. Given that this is a workshop paper with otherwise sound and well-executed experiments, minor revision is appropriate. The GRPO feature-count derivation and the matched-fraction pretraining control are genuine strengths that should be highlighted to the authors."},"author_rebuttal":{"model":"glm-5.2","summary":"The referee raises a single major comment regarding whether the 'beyond base model' claim in Section 6 conflates sampling budget (pass@k) with generation budget (max tokens per attempt). The referee correctly notes that all attempts are capped at 256 tokens, and that difficulty levels are defined relative to this cap. We agree this is a valid concern and will address it by running the suggested experiments and revising the framing accordingly.","responses":[{"response":"The referee is correct that Table 1 varies the sampling budget k while holding the generation budget fixed at 256 tokens, and that the difficulty levels are defined relative to this 256-token cap. We agree that this leaves open the possibility that the base model could solve Buckets 4–5 with primitive-only solutions if given a larger per-attempt token budget. This is a genuine gap in the experimental support for the frontier-expansion framing as currently stated. We will address this in two ways. First, we will run the experiment the referee suggests: evaluating the base model at pass@k with generation budgets of 512 and 1024 tokens on Buckets 4–5. If the base model can solve a substantial fraction of these buckets with primitive-only solutions under a larger token budget, we will reframe the claim as the referee proposes in option (b): explicitly stating that the expansion is relative to a fixed generation budget, and that RL's contribution is discovering token-efficient compositional shortcuts rather than expanding the capability frontier in an absolute sense. Second, regardless of the outcome of that experiment, we will revise the abstract and Section 6 to make the fixed-generation-budget qualifier explicit in the claim, since the current phrasing ('RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets') does not currently make clear that the generation budget is held constant. We note that the mechanistic findings in Sections 4–5 and 7 (phased compositional emergence, consolidation, selectivity over RFT, pretraining structure gating) are independent of this framing and would stand either way. The referee's point specifically concerns the frontier-expansion claim in Contribution 3, not the strategy-discc","revision_made":"no","referee_comment":"Section 6, Table 1: The 'beyond base model' claim conflates sampling budget (pass@k) with generation budget (max tokens per attempt). Table 1 varies k (64, 256, 1024) but every attempt is capped at 256 tokens. Difficulty levels are explicitly defined by whether the primitive contraction solution fits within 256 tokens (Section 3.2): Difficulty 2 needs one extra primitive step, Difficulty 3 needs two, etc. Thus Buckets 4–5 are solvable with primitives the base model already possesses—they just require more tokens per attempt. The base model scores 0% on Buckets 4–5 at pass@1024, but this may simply reflect the 256-token cap rather than a capability gap. If the base model were given a 1024- or 2048-token generation budget, it might solve a substantial fraction of Buckets 4–5 using only primitive contractions. This would reframe the central 'beyond base model' claim from 'RL expands the cap"},{"response":"We will adopt both options. We will run the larger-generation-budget experiment (option a) and report the results, and we will also revise the framing (option b) to make the fixed-generation-budget qualifier explicit throughout the abstract, Section 6, and the contribution list. If the larger-budget experiment shows that the base model can solve Buckets 4–5 with primitive-only solutions, we will reframe the claim as the referee suggests. If it shows that the base model still cannot solve them (e.g., because the model's primitive contraction competence is itself unreliable at the needed chain lengths), the claim would be better supported but we would still state the generation-budget condition explicitly for clarity. In either case, the revised manuscript will not claim frontier expansion without the qualification the referee identifies.","revision_made":"no","referee_comment":"...capability frontier' to 'RL learns token-efficient shortcuts within a fixed generation budget.' The mechanistic findings about compositional strategy emergence (Sections 4–5, 7) would still stand, but the frontier-expansion framing is currently unsupported without varying the generation budget. The authors should either (a) evaluate the base model at pass@k with a larger generation budget (e.g., 512, 1024 tokens) to test whether primitive-only solutions become accessible, or (b) reframe the claim to explicitly state that the expansion is relative to a fixed generation budget. As written, the claim is stronger than what the experiment supports."}],"tokens_in":13736,"tokens_out":970,"duration_ms":94334,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper introduces a genuinely auditable rewrite-grammar testbed where every emitted rewrite can be classified as primitive, macro, parallel, or spurious against a known grammar. That's a real methodological contribution. The four-way taxonomy and the phased emergence story — primitives first, then macro contractions, then parallel — are cleanly shown across three seeds. The GRPO feature-count derivation in Section 5 is correct and isolates the within-prompt contrast mechanism that distinguishes GRPO from RFT. The pretraining ablation with the matched-fraction control is well designed: it separates marginal contraction exposure from chained reduction structure, and the result is clear. The RFT comparator is strong — same prompts, same group size, same data. This is solid work for a workshop venue, and the mechanistic claims about selectivity and consolidation hold up to scrutiny. The reader's assessment is fair and I largely agree with it. Now, the stress-test concern. It lands. The paper defines difficulty by whether the primitive solution fits within a 256-token generation budget, then evaluates the base model at pass@1024 but always with that same 256-token cap. So Buckets 4–5 are unsolvable by the base model not because it lacks the primitives but because each attempt is too short for the primitive-only path. If the base model were given 1024 or 2048 tokens per attempt, it might solve a meaningful fraction of those problems with primitives alone. This reframes the 'beyond base model' claim: RL discovers token-efficient shortcuts, which is still important and still compositional, but it is not the same as expanding the capability frontier in the strongest sense. The title claim — 'builds compositional reasoning strategies' — survives this, because the macro and parallel contractions are genuinely new valid procedures, not just longer primitive chains. But the abstract framing about 'held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets' is misleading as written, since the sampling budget is larger but the generation budget is not. This is a fixable problem: either run the base model with a larger generation budget and show it still fails, or qualify the claim to 'token-efficient compositional strategies.' Two minor points: only GRPO is tested, so the title's 'RL post-training' is broader than the evidence supports. And no code or data is shipped, which limits independent verification of the trajectory classifications — though the grammar and training details in the appendices are thorough enough to reproduce. This paper is for researchers working on RL post-training mechanisms and the amplification-vs-composition debate. It deserves a serious referee who can push the authors to address the generation-budget confound, which is the one issue that materially affects the central claim. I'd recommend a revise-and-resubmit cycle at a venue that allows it.","headline":"The stress-test concern about generation budget is a real gap, but the mechanistic findings survive it.","tokens_in":14733,"tokens_out":644,"would_cite":true,"duration_ms":163749,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"RL composes primitive skills into new reasoning strategies","keywords":["reinforcement learning","compositional reasoning","post-training","rewrite grammar","procedural chunking","GRPO","rejection fine-tuning","skill composition"],"falsifier":"If one constructed a grammar where right-hand sides are not globally unique (so primitive contractions are ambiguous), and the same phased compositional emergence failed to appear, it would suggest that the mechanism depends critically on the environment's unambiguous structure rather than on a general property of RL post-training.","tokens_in":13910,"feed_emoji":"🧩","tokens_out":1043,"duration_ms":151533,"temperature":0.7,"pith_summary":"This paper asks whether reinforcement learning (RL) post-training merely amplifies behaviors already latent in a base model, or whether it can compose primitive skills into genuinely new higher-level reasoning strategies. The authors study this in a fully observable rewrite-grammar environment where every generated step can be audited against a known rule set. They find that RL does more than reweight: it first strengthens primitive reduction steps, then discovers valid composed procedures—sequential compositions that collapse chains of primitive steps and parallel compositions that combine independent steps simultaneously. These compositions are not isolated lucky samples; they are reused and consolidated into a stable repertoire. Crucially, RL achieves this selectivity through within-prompt contrast between successful and failed rollouts, whereas rejection fine-tuning (RFT) clones whatever appears in successful trajectories, including invalid shortcuts. The paper also shows that strategy emergence depends not on how often the base model sees primitive operations, but on whether pretraining organizes them into sustained reduction procedures that RL can later compress.","feed_headline":"RL doesn't just amplify skills—it composes new reasoning strategies","feed_subtitle":"In a fully auditable rewrite-grammar environment, RL discovers and consolidates valid multi-step shortcuts that the base model never produce","key_machinery":"The rewrite-grammar environment with globally unique right-hand sides enables exact auditing of every generated rewrite into four categories: primitive, macro (sequential composition), parallel (parallel composition), and spurious. The GRPO finite-group analysis shows that the sign of the feature-level update is determined by the contrast between feature frequency in successful versus failed completions from the same prompt, which RFT cannot replicate because it discards failures.","core_discovery":"The central discovery is a phased compositional mechanism: RL post-training first strengthens primitive contractions, then discovers and consolidates valid composed procedures (macro and parallel contractions) that were never present in the pretraining data. This mechanism is driven by selectivity, not exploration volume—GRPO's within-group success-failure contrast suppresses spurious rewrites that RFT clones—and it is gated by whether pretraining provides chained reduction procedures, not merely exposure to primitive facts. The result is that RL solves held-out problems the base model cannot solve even at 16x the sampling budget.","pith_inferences":["The selectivity mechanism identified here—GRPO's within-prompt contrast suppressing spurious actions—could potentially be tested in natural-language reasoning settings by classifying intermediate steps as valid or invalid using a verifier model, even when a ground-truth grammar is unavailable.","If the pretraining substrate result generalizes, it predicts that two base models with identical primitive skill coverage but different procedural chaining structure should diverge significantly under RL post-training, which is testable with controlled pretraining ablations on larger models.","The consolidation phase where discovered macro rules become reused suggests that RL may be implicitly learning a form of option-like hierarchical policy structure inside a flat sequence model, which could be probed by examining attention patterns or internal representations associated with consolidated macro rules."],"forward_implications":["If RL's compositional chunking mechanism generalizes beyond rewrite grammars, then RL post-training on math or code reasoning may be building reusable multi-step procedures rather than merely amplifying existing solution probabilities, which would change how we interpret capability gains from RLVR.","The finding that RFT clones invalid shortcuts alongside useful ones suggests that imitation-style post-training may embed spurious reasoning patterns that are invisible without process-level auditing, with potential downstream reliability consequences.","The pretraining substrate result implies that curriculum design for pretraining should optimize for sustained procedural chaining rather than raw exposure to primitive operations, if the goal is to enable later RL to discover compositional strategies.","The phased emergence pattern—primitive strengthening before compositional discovery—suggests that early RL training curves may understate eventual capability gains, and that evaluation protocols with short training horizons may miss compositional strategy formation."],"fun_headline_variants":["RL post-training discovers composed reasoning strategies absent from pretraining data","Phased mechanism: RL strengthens primitives, then consolidates composed procedures","Selectivity drives RL composition; rejection fine-tuning plateaus on invalid shortcuts","RL builds reliable strategies from weak procedural ingredients in base model","Pretraining reduction chains gate emergence of RL compositional strategies"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The central claim that RL builds compositional reasoning strategies rests on the rewrite-grammar environment being representative of reasoning more broadly. The grammar has globally unique right-hand sides, making primitive contractions unambiguous—a structural property that does not hold in natural-language reasoning, where intermediate steps cannot be mechanically classified as valid or invalid. Whether the same phased chunking mechanism operates in LLM post-training on数学或码","fun_headline_variants_meta":{"raw":{"variants":["RL post-training discovers composed reasoning strategies absent from pretraining data","Phased mechanism: RL strengthens primitives, then consolidates composed procedures","Selectivity drives RL composition; rejection fine-tuning plateaus on invalid shortcuts","RL builds reliable strategies from weak procedural ingredients in base model","Pretraining reduction chains gate emergence of RL compositional strategies"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":688,"prompt_tokens":601,"completion_tokens":87,"prompt_tokens_details":null},"tokens_in":601,"tokens_out":87,"duration_ms":18091,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T03:43:44.324039+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one constructed a grammar where right-hand sides are not globally unique (so primitive contractions are ambiguous), and the same phased compositional emergence failed to appear, it would suggest that the mechanism depends critically on the environment's unambiguous structure rather than on a general property of RL post-training.","supporting_citations":[],"review_version":1}