Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Pith's one-line read Reinforcement learning synthesizes new compositional reasoning only when the base model has already mastered the atomic skills of recalling facts from memory and processing facts from context.

desk verdict Solid controlled evidence that RL generalizes to compositional reasoning only after atomic SFT; the mechanistic 'synthesis' claim rests on a pass@k argument that doesn't hold up. read the letter →

arxiv 2512.01970 v3 pith:RU2E63QX submitted 2025-12-01 cs.AI cs.CL

classification cs.AIcs.CL
keywords reinforcementlearningskillsynthesisatomicskillscomplementaryreasoningsupervisedfine-tuningzero-shotgeneralizationparametriccontextual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether reinforcement learning (RL) genuinely creates new reasoning skills or merely makes existing ones more likely to appear. It answers using a controlled synthetic testbed: a database of fictional biographies that separates two atomic skills, recalling facts stored in model weights and processing facts given in context. The central claim is that RL can fuse these atomics into a new composite strategy only when the base model has already been supervised on both atomics separately. Models supervised directly on the composite task memorize the training paths (about 90 percent on seen cases) and collapse on novel ones (about 18 percent), while RL on top of atomic training reaches far better zero-shot generalization. If the claim holds, it gives a concrete training recipe: teach atomic skills first with supervised fine-tuning, then use RL to combine them.

What carries the argument

The load-bearing object is the decomposition of complementary reasoning into a conjunction of two atomic capabilities, formalized as C_COMP ⇔ C_MEM ∧ C_CTX, and implemented as a synthetic knowledge graph of fictional biographies where the boundary between parametric and contextual knowledge is enforced by construction. The pass@k comparison—evaluating whether the SFT base model's sampling catches up to the RL model as k grows from 1 to 512—is the operative test used to distinguish synthesis from amplification. RL is applied with a binary outcome reward on the final answer, not on the reasoning chain.

What would settle it

Take the atomic-SFT base and sample k up to 2^16 or more on zero-shot complementary questions before RL; if pass@k eventually catches up to the RL model's zero-shot accuracy, the persistent gap at 512 is a sampling-depth artifact rather than evidence of a new circuit. Alternatively, run RL with the same composite data but an answer-independent reward; if zero-shot gains persist, generalization comes from exposure rather than reinforcement.

Watch

Extended reading notes

Core claim

The paper defines Complementary Reasoning as a task that requires both Parametric Reasoning (facts encoded in model weights) and Contextual Reasoning (facts supplied in the input context). Through a synthetic biography dataset built on a controlled knowledge graph, it compares supervised fine-tuning (SFT) on atomics, SFT on the composite task, and RL starting from different bases. It finds that SFT alone on the composite task yields roughly 90 percent in-distribution accuracy but only about 18 percent on zero-shot relational paths, evidence that SFT memorizes path shortcuts rather than learning to integrate skills. RL applied to a base that was SFT-trained on atomics produces large gains acr

Load-bearing premise

The pass@k non-convergence test assumes that 512 samples are enough to reveal any latent strategy in the SFT model; if the composed strategy exists in the base distribution with probability lower than roughly 1/512, the diagnosis of synthesis over amplification would be wrong.

Editorial extensions

If this is right

  • A model SFT-trained on atomic skills and then RL-trained on a small slice of composite data can match or beat a model SFT-trained on the full composite dataset, using less than 10% of the composite data.
  • SFT on the composite task creates a memorization trap: high in-distribution scores with near-zero out-of-distribution transfer, so composite-task accuracy alone is a poor predictor of RL readiness.
  • Removing either atomic skill from the base model collapses RL-driven generalization, so both parametric and contextual foundations are necessary prerequisites, not optional aids.
  • For atomic-trained bases, RL creates reasoning that is not reachable by sampling from the base distribution, whereas for composite-trained bases RL only boosts the probability of already-present paths.
  • The atomic-first recipe maintains its advantage across model sizes from 0.5B to 3B parameters, with the largest zero-shot gap at the largest scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to evaluate the same atomic-first/RL-later ordering on temporally controlled news QA, where facts dated after training are guaranteed to be contextual; if the ordering effect disappears there, the synthetic result may depend on the clean relation-path structure.
  • The pass@k criterion is behavioral rather than mechanistic; a stronger test of synthesis would be circuit-level evidence, such as checking whether RL training creates attention heads that bridge context and memory that were absent before RL.
  • The paper's zero-shot setting includes relations that never appear in QA training but do appear in the parametric biographies; whether this counts as true synthesis depends on whether RL discovers how to query latent knowledge or merely surfaces a query pattern already latent in the SFT distribution.
  • The recipe implies that expensive composite reasoning traces may be unnecessary at scale: cheap atomic data for SFT plus a modest amount of RL on composite prompts could suffice, but this implication is the authors' scalable-path suggestion and remains untested on real-world knowledge-intensive benchmarks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies whether reinforcement learning (RL) synthesizes new compositional reasoning skills or merely amplifies skills already present in a supervised fine-tuned (SFT) base model. Using a controlled synthetic biography dataset, the authors decompose Complementary Reasoning (COMP) into two atomic skills—Parametric Reasoning (MEM) and Contextual Reasoning (CTX)—and evaluate generalization at three levels: I.I.D., Composition, and Zero-shot. The main empirical findings are that (i) SFT directly on COMP reaches high in-distribution accuracy but collapses on zero-shot relational combinations; (ii) RL on COMP data yields large generalization gains only when the base model was first SFT-ed on both atomic skills; and (iii) pass@k non-convergence is interpreted as evidence that RL has created a new reasoning circuit rather than reweighting existing ones. The paper also reports model-scaling, PCA, uncertainty, sample-efficiency, and error analyses.

Significance. If the central claims hold, the paper makes a useful practical contribution: it identifies a concrete data recipe—SFT on atomic skills followed by RL on a small amount of composite data—that outperforms direct SFT on the composite task, especially for zero-shot generalization. The controlled synthetic setup is a genuine strength: it avoids pretraining contamination, enforces strict parametric/contextual separation, and evaluates novel relational paths. The authors also include several complementary analyses (model scaling in Appendix C, training dynamics in Appendix E.1, PCA in Appendix E.3, uncertainty in Appendix E.4) that go beyond a single accuracy table. However, the strongest mechanistic conclusion—that RL synthesizes genuinely new logic circuits—is not established by the finite-k pass@k argument, and the paper reports no variance information. The practical recipe is plausible and well-supported in broad terms, but the mechanistic interpretation is overstated relative to the evidence.

major comments (5)
  1. [Title/Abstract vs Section 4.3 and Table 1b] The abstract states that SFT on the composite task 'collapses' to 18% on novel paths, Section 4.3 reports 26.25% zero-shot for SFT_COMP, and Table 1b reports 18.41%. These are different numbers for the same headline result. The abstract's central quantitative claim should match the table and the main text; currently a reader cannot tell which number is the reported result.
  2. [6.2, Figure 7] The pass@k non-convergence criterion is not sufficient to distinguish synthesis from low-probability amplification. For a latent correct trajectory with per-sample probability p, pass@k ≈ 1-(1-p)^k; at k up to 2^9, a trajectory with p ≈ 1/512 produces a persistent gap indistinguishable from the reported one. The two explanations are separable only by the scaling of the gap with k across many orders of magnitude, or by mechanistic evidence. The authors themselves acknowledge in the Limitations that mechanistic evidence is absent. The strong language about 'logic circuits' and 'fundamentally absent from the SFT distribution' should be softened or supported with additional k-scaling experiments.
  3. [6.2, Figure 7] The comparison between SFT_MEM+CTX and SFT_COMP in the pass@k analysis is confounded. The SFT_COMP baseline is trained on 90% of COMP data, while SFT_MEM+CTX is trained on 100% of MEM and CTX data; the two base distributions differ in composition, data quantity, and coverage, not only in possession of atomic skills. A cleaner control would equate total data volume and relation/path coverage across the two bases, or vary both dimensions systematically.
  4. [5.2 vs Appendix D, Table 7] Section 5.2 concludes that RL is 'uniquely necessary' for zero-shot generalization from atomic skills. However, Appendix D, Table 7 reports that SFT_MEM+CTX+COMP (SFT on a mix of all atomic and composite data) reaches zero-shot 43.25, close to the proposed SFT_MEM+CTX→RL_COMP zero-shot 50.87. The appendix acknowledges this but the main-text claim overstates the necessity of RL. The claim should be revised to 'RL gives the best zero-shot performance under the tested data budgets' or the comparison should be made directly with matched data.
  5. [All experiments (Tables 2, 6, 7, 8; Figures 2–7)] No error bars, seeds, or statistical tests are reported anywhere in the paper. Many core comparisons—'consistently outperforms at every data scale,' 'gains nearly double in all settings,' 'SFT curves converge while RL curves do not'—are single-run numbers. Given the noise typical of small RL training runs, at least a few seeds (or a bootstrap over test samples) are needed to establish that the reported gaps are reliable.
minor comments (5)
  1. [Table 8] The 30% and 90% data sections both contain a row labeled 'SFT_COMP→RL_COMP' with entropy values 2.90 and 4.10, respectively. Based on the context and the surrounding rows, these appear to be intended as 'SFT_MEM+CTX→RL_COMP'; please correct.
  2. [Figures 2 and 5] Color labels are inconsistent: Figure 2 describes 'red bars' vs 'green bars' in the text but 'yellow bars' vs 'blue bars' in the caption; Figure 5 uses red/orange and green/blue in the text and caption. Please harmonize color naming with the actual figures.
  3. [Section 4.1] The inference that COMP is 'inherently more difficult' because the authors generated ~180k COMP samples vs ~88k MEM and ~3k CTX samples conflates dataset construction choices with task difficulty. A statement about how the sample sizes were chosen would strengthen the interpretation.
  4. [Abstract / GitHub URL] The URL 'https://github.com/sitaocheng/from atomic to composite' contains spaces; the intended URL is presumably 'https://github.com/sitaocheng/from-atomic-to-composite' or similar. Also, at submission time the repository is not yet available, which limits reproducibility of the numerical claims.
  5. [Section 4.2] The paper alternates between 'Qwen-2.5-1.5B' and 'Qwen-1.5B-Base'; please use a single model naming convention throughout, including in Appendix C and E.1.

Circularity Check

2 steps flagged · score 3.0 of 10

Partial construct-level circularity: COMP is defined as the conjunction of MEM and CTX, and the pass@k criterion defines synthesis as non-convergence; the empirical recipe itself is not derived from these definitions.

  1. self definitional [Section 3, Problem Definition: Complementary Reasoning (equation for C_COMP)]
    "Formally, we define the capability requirement as a logical conjunction: CCOMP ⇐ ⇒ CMEM ∧ CCTX . This implies that a failure in either atomic skill (CMEM or CCTX) necessitates a failure in the complementary task."

    The paper's central claim that 'atomic skills are a strict prerequisite' is partly a restatement of this definition: if COMP is defined as requiring both MEM and CTX, then any model lacking an atomic skill cannot perform COMP by construction. The subsequent experiments test training orders and data recipes, which gives the claim empirical content, but the 'prerequisite' language is already entailed by the task definition rather than independently derived.

  2. self definitional [Section 6.2, The Role of RL: Synthesizer vs. Amplifier, Figure 7]
    "1) Synthesis Signal: If a significant performance gap persists even at large k (i.e., the curves remain parallel or divergent), it implies that RL has synthesized novel reasoning paths that the SFT model fundamentally cannot discover via sampling. 2) Amplification Signal: If the performance of the SFT model eventually catches up to the RL model as k increases (i.e., the curves merge), it implies that the reasoning paths are latent in the SFT distribution, and RL merely amplifies their generation probability."

    The paper defines 'synthesis' and 'amplification' operationally in terms of whether pass@k curves merge by k=2^9, then reports the observed non-convergence as evidence that RL 'has synthesized a logic circuit'. Under this operationalization, the conclusion is the criterion: any persistent gap is labeled synthesis. Finite-k non-convergence cannot distinguish a genuinely new circuit from a latent correct trajectory with per-sample probability below 1/512, so the mechanistic claim is not independently tested. The authors' Limitations concede that mechanistic evidence is absent, saying future work should investigate 'mechanistic interpretability of how RL circuits recruit atomic attention heads'.

full rationale

The paper is primarily an empirical controlled study, and its headline recipe — SFT on MEM+CTX followed by RL on COMP outperforms SFT on COMP followed by RL — is supported by accuracy comparisons, data-efficiency curves, scaling experiments, ablation of atomic-skill necessity, and error-pattern analysis. These comparisons are not fitted to the conclusion, and the recipe's empirical superiority is not derived from the definition of COMP. The main circularity concern is at the construct level. First, COMP is defined as the logical conjunction of MEM and CTX, so the 'atomic prerequisite' is partly true by definition; however, the paper does not stop at the definition and tests alternative training orders, giving the practical recipe independent content. Second, the synthesizer-versus-amplifier conclusion is operationalized through the pass@k criterion: non-convergence at k≤2^9 is labeled 'synthesis', making the mechanistic label co-extensive with the observation. This is a validity and inference issue more than a fitted-input circularity, but it does mean the strong mechanistic claim is not independently established. Self-citations appear only as background and are not load-bearing. Overall, the central empirical findings remain self-contained, so the circularity score is low-to-moderate rather than severe.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numerical free parameters are fitted in a derivation sense; the paper is an empirical study. The main load-bearing assumptions are the definitional decomposition of COMP into atomic skills, the contamination-free nature of synthetic biographies, the pass@k interpretive criterion, and the representativeness of the Qwen family.

assumptions (4)
  • domain assumption Complementary reasoning is exactly the logical conjunction of parametric and contextual reasoning (C_COMP ⇔ C_MEM ∧ C_CTX).
    Section 3 defines COMP this way; if composite reasoning requires additional skills (e.g., planning, tool use), the experimental conclusions may not transfer to other composite tasks.
  • domain assumption Synthetic biographies with novel entities provide a faithful model of the parametric/contextual boundary without pretraining contamination.
    Section 4.1 explicitly adopts this design to control knowledge; its validity for real-world RAG or continual learning is untested.
  • ad hoc to paper Pass@k non-convergence is a reliable discriminator between skill synthesis and probability amplification.
    Introduced in Section 6.2 and used to conclude that RL creates new logic circuits; not independently validated and may depend on sampling budget k.
  • domain assumption The Qwen-2.5 family is representative of LLM post-training behavior.
    All experiments use Qwen-0.5B/1.5B/3B; the authors acknowledge this limitation and only claim the finding is likely architecture-agnostic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies." pith.science (2026). https://pith.science/paper/RU2E63QX

@misc{pith2026251201970,
  author       = {Pith},
  title        = {Pith review of: Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RU2E63QX}},
  note         = {Machine review of arXiv:2512.01970}
}
read the original abstract

Does Reinforcement Learning (RL) merely amplify existing skills, or synthesize novel skills? We investigate this question through the lens of Complementary Reasoning: the critical practical capability of integrating internal knowledge with external context, a prerequisite for reliable Continual Learning and Retrieval-Augmented Generation. To avoid pre-training contamination, we construct a controlled semanticsynthetic dataset of biographies and decompose this capability into two atomic skills: Parametric Reasoning (retrieving facts encoded in model weights) and Contextual Reasoning (processing novel in-context information). We present two findings. First, models supervised directly on the composite task reach high accuracy on seen facts and reasoning paths (90%) but collapse on novel facts and reasoning paths (18%), indicating that Supervised Fine-Tuning (SFT) relies on rote memorization rather than genuine skill integration. Second, RL bridges this generalization gap, acting as a skill synthesizer rather than a mere amplifier--but only under a strict prerequisite: it synthesizes new composite strategies only when the base model has first mastered the independent atomic skills via SFT. These results suggest that decoupled atomic training followed by RL offers a scalable path to complex novel reasoning.

Figures

Figures reproduced from arXiv: 2512.01970 by the authors.

Figure 2
Figure 2. Comparison of reinforcement learning on different base models across complementary [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Necessity of atomic skills for RL generalization. We conduct RL with the same amount [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Performance of training with dif￾ferent strategies over 12.8k COMP samples. Necessity of Reinforcement Learning for Zero￾shot generalization (Training Strategy). Given a model with sufficient atomic skills (SFTMEM+CTX), is RL strictly necessary to synthesize them? We com￾pare three training strategies using the same 12.8k COMP samples: SFT, LoRA (rank=256), and RL [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Performance with the same amount of SFT and RL data. We compare SFT [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Few-shot adaptation of SFTMEM+CTX with different training strategies. Sufficient atomic skills enable few-shot adaptation. We further investigate how much COMP data is actually needed to “trigger” gen￾eralization once sufficient atomic skills are es￾tablished. Fixing S…
Figure 7
Figure 7. Figure 7: Pass@k comparison for SFTMEM+CTX and SFTCOMP. It shows that RL synthesizes new compositional skills only based on models with sufficient atomic skills. present in the base distribution (Amplification) (Setlur et al., 2025)? To distinguish between these roles of RL, we …
Figure 8
Figure 8. Figure 8: Model scaling analysis of Qwen model family across 0.5B, 1.5B, and 3B parameters. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Training dynamics of SFT with parametric and contextual reasoning data over training [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The impact of training steps (i.e., training loss) of the SFT model on RL generalization. E.1 THE TRAINING DYNAMICS OF SFTMEM+CTX Settings To check when and how the ability of complementary reasoning ability (COMP) emerges through the training of parametric and contex…
Figure 11
Figure 11. Figure 11: PCA Analysis of SFTMEM+CTX →RLCOMP and SFTCOMP →RLCOMP. The scatter points represent the layer-wise shifts. The large markers represent the global centroid shift (z ∗ ) for each reasoning type. guish between the underlying mechanisms of parametric and contextual reaso…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 unverdicted novelty 6.5 of 10

    SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.

Reference graph

Works this paper leans on

4 extracted references · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.