{"id":"70b38bd8-7be7-4a18-b5be-ae3888911509","arxiv_id":"2512.01970","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a controlled synthetic QA benchmark, RL generalized to unseen relation combinations only when the model had already been SFT-trained on both parametric and contextual atomic skills.","lead":"This paper tests whether reinforcement learning creates new reasoning skills or merely boosts existing ones, using a synthetic dataset that separates memory-based and context-based reasoning. It reports that RL generalizes to novel combinations only if the model first mastered the two atomic skills through supervised fine-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pass@k non-convergence at k=512 cannot distinguish synthesis from low-probability amplification; only k-scaling can settle it.","rationale":"The central claim of the paper is not merely that atomic SFT followed by RL improves complementary reasoning—that is supported by multiple direct comparisons in Figures 2, 3, and 5. The stronger and more novel claim is that RL acts as a skill synthesizer, creating genuinely new reasoning circuits, rather than as a probability amplifier. The only evidence offered for this stronger claim is the non-convergence of pass@k curves in Figure 7. This is the load-bearing step: if the gap merely reflects low but nonzero latent probability under the SFT model, the practical recipe remains intact but the paper's mechanistic contribution collapses. The reader's weakest_assumption identifies exactly this issue, and the paper's own Limitations section concedes that mechanistic interpretability is left to future work. The concern is not an internal inconsistency, but an overinterpretation of a finite-k sampling experiment. With k≤512, any latent trajectory with per-sample probability below roughly 2e-3 is essentially invisible to the pass@k metric. Given that the correct CoT paths in this synthetic task are multi-hop sequences, such low probabilities are entirely plausible, and the paper provides no estimate of the base rate. The confounded baseline—SFT COMP trained on 90% of COMP data versus SFT MEM+CTX trained on the full atomic set—further prevents attributing the observed gap specifically to skill synthesis. A pass@k scaling experiment with much larger k, or a direct estimate of the underlying per-sample probability, would settle whether the curves eventually merge. If they merge, the amplification account is confirmed; if the gap persists with stable slope, the synthesis claim gains genuine support. Therefore the reader's CONDITIONAL verdict is unchanged: the empirical recipe is credible, but the mechanistic conclusion requires additional evidence.","tokens_in":20444,"tokens_out":2914,"duration_ms":35167,"concrete_test":"Run a controlled replication of Figure 7 with k extended to 2^15 (or equivalently estimate the per-sample probability of a correct completion under SFT MEM+CTX from ~10k samples per test item and compute the k at which pass@k would approach the RL curve). If SFT MEM+CTX curves eventually merge with the RL curves at larger k, the synthesis claim fails; if the gap persists with a stable parallel slope beyond the extrapolated coverage horizon, the synthesis interpretation is supported. To remove the confound, also retrain SFT COMP with a matched data budget (e.g., same total number of samples as MEM+CTX) before comparing pass@k curves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanistic claim—that RL synthesizes new logic circuits rather than amplifying latent ones—rests almost entirely on the pass@k argument in Section 6.2 and Figure 7. The logic is: because SFT MEM+CTX sampling curves do not catch up to RL within k≤2^9, RL must have created a reasoning path absent from the base distribution. This inference is not sound. pass@k at finite k measures the upper tail of the sampling distribution. A latent correct trajectory with per-sample probability p much smaller than 1/512 will also produce a persistent gap at k=512, even if RL merely re-weights that existing trajectory. The two explanations are distinguishable only by the scaling of the gap with k, not by non-convergence at a single k. The comparison is also confounded: the SFT COMP baseline was trained on 90% of COMP data, while SFT MEM+CTX was trained on 100% of MEM+CTX data, so the base distributions differ in composition and coverage, not only in atomic-skill possession. The authors themselves acknowledge in the Limitations section that mechanistic evidence is absent and that future work should investigate 'how RL circuits recruit atomic attention heads.' Thus, Section 6.2 overstates the mechanistic conclusion: the reported evidence supports the practical recipe but not the strong synthesizer interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether reinforcement learning (RL) synthesizes new compositional reasoning skills or merely amplifies skills already present in a supervised fine-tuned (SFT) base model. Using a controlled synthetic biography dataset, the authors decompose Complementary Reasoning (COMP) into two atomic skills—Parametric Reasoning (MEM) and Contextual Reasoning (CTX)—and evaluate generalization at three levels: I.I.D., Composition, and Zero-shot. The main empirical findings are that (i) SFT directly on COMP reaches high in-distribution accuracy but collapses on zero-shot relational combinations; (ii) RL on COMP data yields large generalization gains only when the base model was first SFT-ed on both atomic skills; and (iii) pass@k non-convergence is interpreted as evidence that RL has created a new reasoning circuit rather than reweighting existing ones. The paper also reports model-scaling, PCA, uncertainty, sample-efficiency, and error analyses.","tokens_in":20743,"tokens_out":4270,"duration_ms":50069,"significance":"If the central claims hold, the paper makes a useful practical contribution: it identifies a concrete data recipe—SFT on atomic skills followed by RL on a small amount of composite data—that outperforms direct SFT on the composite task, especially for zero-shot generalization. The controlled synthetic setup is a genuine strength: it avoids pretraining contamination, enforces strict parametric/contextual separation, and evaluates novel relational paths. The authors also include several complementary analyses (model scaling in Appendix C, training dynamics in Appendix E.1, PCA in Appendix E.3, uncertainty in Appendix E.4) that go beyond a single accuracy table. However, the strongest mechanistic conclusion—that RL synthesizes genuinely new logic circuits—is not established by the finite-k pass@k argument, and the paper reports no variance information. The practical recipe is plausible and well-supported in broad terms, but the mechanistic interpretation is overstated relative to the evidence.","major_comments":[{"comment":"The abstract states that SFT on the composite task 'collapses' to 18% on novel paths, Section 4.3 reports 26.25% zero-shot for SFT_COMP, and Table 1b reports 18.41%. These are different numbers for the same headline result. The abstract's central quantitative claim should match the table and the main text; currently a reader cannot tell which number is the reported result.","section":"Title/Abstract vs Section 4.3 and Table 1b"},{"comment":"The pass@k non-convergence criterion is not sufficient to distinguish synthesis from low-probability amplification. For a latent correct trajectory with per-sample probability p, pass@k ≈ 1-(1-p)^k; at k up to 2^9, a trajectory with p ≈ 1/512 produces a persistent gap indistinguishable from the reported one. The two explanations are separable only by the scaling of the gap with k across many orders of magnitude, or by mechanistic evidence. The authors themselves acknowledge in the Limitations that mechanistic evidence is absent. The strong language about 'logic circuits' and 'fundamentally absent from the SFT distribution' should be softened or supported with additional k-scaling experiments.","section":"6.2, Figure 7"},{"comment":"The comparison between SFT_MEM+CTX and SFT_COMP in the pass@k analysis is confounded. The SFT_COMP baseline is trained on 90% of COMP data, while SFT_MEM+CTX is trained on 100% of MEM and CTX data; the two base distributions differ in composition, data quantity, and coverage, not only in possession of atomic skills. A cleaner control would equate total data volume and relation/path coverage across the two bases, or vary both dimensions systematically.","section":"6.2, Figure 7"},{"comment":"Section 5.2 concludes that RL is 'uniquely necessary' for zero-shot generalization from atomic skills. However, Appendix D, Table 7 reports that SFT_MEM+CTX+COMP (SFT on a mix of all atomic and composite data) reaches zero-shot 43.25, close to the proposed SFT_MEM+CTX→RL_COMP zero-shot 50.87. The appendix acknowledges this but the main-text claim overstates the necessity of RL. The claim should be revised to 'RL gives the best zero-shot performance under the tested data budgets' or the comparison should be made directly with matched data.","section":"5.2 vs Appendix D, Table 7"},{"comment":"No error bars, seeds, or statistical tests are reported anywhere in the paper. Many core comparisons—'consistently outperforms at every data scale,' 'gains nearly double in all settings,' 'SFT curves converge while RL curves do not'—are single-run numbers. Given the noise typical of small RL training runs, at least a few seeds (or a bootstrap over test samples) are needed to establish that the reported gaps are reliable.","section":"All experiments (Tables 2, 6, 7, 8; Figures 2–7)"}],"minor_comments":[{"comment":"The 30% and 90% data sections both contain a row labeled 'SFT_COMP→RL_COMP' with entropy values 2.90 and 4.10, respectively. Based on the context and the surrounding rows, these appear to be intended as 'SFT_MEM+CTX→RL_COMP'; please correct.","section":"Table 8"},{"comment":"Color labels are inconsistent: Figure 2 describes 'red bars' vs 'green bars' in the text but 'yellow bars' vs 'blue bars' in the caption; Figure 5 uses red/orange and green/blue in the text and caption. Please harmonize color naming with the actual figures.","section":"Figures 2 and 5"},{"comment":"The inference that COMP is 'inherently more difficult' because the authors generated ~180k COMP samples vs ~88k MEM and ~3k CTX samples conflates dataset construction choices with task difficulty. A statement about how the sample sizes were chosen would strengthen the interpretation.","section":"Section 4.1"},{"comment":"The URL 'https://github.com/sitaocheng/from atomic to composite' contains spaces; the intended URL is presumably 'https://github.com/sitaocheng/from-atomic-to-composite' or similar. Also, at submission time the repository is not yet available, which limits reproducibility of the numerical claims.","section":"Abstract / GitHub URL"},{"comment":"The paper alternates between 'Qwen-2.5-1.5B' and 'Qwen-1.5B-Base'; please use a single model naming convention throughout, including in Appendix C and E.1.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a practical and plausible takeaway. The main reason for major revision is the gap between the strong mechanistic interpretation in Section 6.2 and the actual evidence, plus the internal inconsistency in the headline zero-shot number and the absence of variance information. The confounded pass@k comparison and the Appendix D result on SFT with mixed data further weaken the strong 'synthesis vs amplification' and 'RL is necessary' framings. I believe these issues can be addressed within the scope of the paper, but the mechanistic claim needs to be either substantially softened or backed by scaling experiments and a proper control."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the practical finding is probably real: on a clean synthetic testbed, RL only generalizes to novel composite reasoning when the base model has been SFT'd on both atomic skills, and SFT directly on the composite task rote-memorizes. Second, don't take the pass@k non-convergence as proof that RL creates new reasoning circuits; that argument conflates low-probability latent trajectories with genuinely absent ones.\n\nThe paper does a lot right. The synthetic biography setup strictly separates parametric and contextual knowledge, the three-level generalization split (I.I.D., Composition, Zero-shot) is sensible, and the scale is substantial. The main comparison is well controlled: same RL data, varying SFT composition. The finding that SFT on composite data reaches 90% I.I.D. but collapses on zero-shot, while atomic SFT plus RL does the reverse, is a clean and useful empirical result. It also replicates across Qwen 0.5B, 1.5B, and 3B. That deserves credit.\n\nNow the soft spots, in proportion. The most concrete problem is internal inconsistency: the abstract reports 18% zero-shot for SFT-COMP, but Section 4.3 says 26.25% citing the same table. Table 1b itself shows 18.41. That needs fixing before the paper is citable. There are also no error bars or repeated seeds reported, and code/data are not yet released, so the numbers cannot be verified independently.\n\nThe bigger conceptual issue is Section 6.2. Pass@k at k up to 512 cannot distinguish synthesis from amplification. If the correct trajectory has per-sample probability well below 1/512, the SFT sampling curve will not catch up at k=512 even if RL merely re-weights that existing trajectory. The authors' own Limitations section concedes that mechanistic evidence is absent. The comparison is also confounded: SFT-COMP on 90% of COMP data vs SFT-MEM+CTX on 100% of atomic data are not matched in composition or coverage. So the practical recipe stands, but the “synthesizer vs amplifier” language overreaches the evidence.\n\nOne more thought: the reader worried that COMP being defined as the conjunction of MEM and CTX bakes in the atomic prerequisite. I think that is not a flaw; it is the intended design. The interesting empirical claim is that the ordering matters—atomic SFT first, then RL—and the data support that.\n\nBottom line: this deserves a serious referee. The main empirical contribution is solid and useful for anyone thinking about post-training data allocation. The mechanistic framing needs to be toned down, the numbers fixed, and the code released. I would send it to review, and I would cite the empirical finding once the inconsistencies are resolved.","headline":"Solid controlled evidence that RL generalizes to compositional reasoning only after atomic SFT; the mechanistic 'synthesis' claim rests on a pass@k argument that doesn't hold up.","tokens_in":21237,"tokens_out":1815,"would_cite":true,"duration_ms":20863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning synthesizes new compositional reasoning only when the base model has already mastered the atomic skills of recalling facts from memory and processing facts from context.","keywords":["reinforcement learning","skill synthesis","atomic skills","complementary reasoning","supervised fine-tuning","zero-shot generalization","parametric reasoning","contextual reasoning"],"falsifier":"Take the atomic-SFT base and sample k up to 2^16 or more on zero-shot complementary questions before RL; if pass@k eventually catches up to the RL model's zero-shot accuracy, the persistent gap at 512 is a sampling-depth artifact rather than evidence of a new circuit. Alternatively, run RL with the same composite data but an answer-independent reward; if zero-shot gains persist, generalization comes from exposure rather than reinforcement.","tokens_in":20339,"feed_emoji":"🧩","tokens_out":3999,"duration_ms":41714,"temperature":0.7,"pith_summary":"The paper asks whether reinforcement learning (RL) genuinely creates new reasoning skills or merely makes existing ones more likely to appear. It answers using a controlled synthetic testbed: a database of fictional biographies that separates two atomic skills, recalling facts stored in model weights and processing facts given in context. The central claim is that RL can fuse these atomics into a new composite strategy only when the base model has already been supervised on both atomics separately. Models supervised directly on the composite task memorize the training paths (about 90 percent on seen cases) and collapse on novel ones (about 18 percent), while RL on top of atomic training reaches far better zero-shot generalization. If the claim holds, it gives a concrete training recipe: teach atomic skills first with supervised fine-tuning, then use RL to combine them.","feed_headline":"RL builds new reasoning only on atomic-first models","feed_subtitle":"Supervised training on the composite task memorizes; RL on an atomic base generalizes to unseen relation paths.","key_machinery":"The load-bearing object is the decomposition of complementary reasoning into a conjunction of two atomic capabilities, formalized as C_COMP ⇔ C_MEM ∧ C_CTX, and implemented as a synthetic knowledge graph of fictional biographies where the boundary between parametric and contextual knowledge is enforced by construction. The pass@k comparison—evaluating whether the SFT base model's sampling catches up to the RL model as k grows from 1 to 512—is the operative test used to distinguish synthesis from amplification. RL is applied with a binary outcome reward on the final answer, not on the reasoning chain.","core_discovery":"The paper defines Complementary Reasoning as a task that requires both Parametric Reasoning (facts encoded in model weights) and Contextual Reasoning (facts supplied in the input context). Through a synthetic biography dataset built on a controlled knowledge graph, it compares supervised fine-tuning (SFT) on atomics, SFT on the composite task, and RL starting from different bases. It finds that SFT alone on the composite task yields roughly 90 percent in-distribution accuracy but only about 18 percent on zero-shot relational paths, evidence that SFT memorizes path shortcuts rather than learning to integrate skills. RL applied to a base that was SFT-trained on atomics produces large gains acr","pith_inferences":["A testable extension is to evaluate the same atomic-first/RL-later ordering on temporally controlled news QA, where facts dated after training are guaranteed to be contextual; if the ordering effect disappears there, the synthetic result may depend on the clean relation-path structure.","The pass@k criterion is behavioral rather than mechanistic; a stronger test of synthesis would be circuit-level evidence, such as checking whether RL training creates attention heads that bridge context and memory that were absent before RL.","The paper's zero-shot setting includes relations that never appear in QA training but do appear in the parametric biographies; whether this counts as true synthesis depends on whether RL discovers how to query latent knowledge or merely surfaces a query pattern already latent in the SFT distribution.","The recipe implies that expensive composite reasoning traces may be unnecessary at scale: cheap atomic data for SFT plus a modest amount of RL on composite prompts could suffice, but this implication is the authors' scalable-path suggestion and remains untested on real-world knowledge-intensive benchmarks."],"forward_implications":["A model SFT-trained on atomic skills and then RL-trained on a small slice of composite data can match or beat a model SFT-trained on the full composite dataset, using less than 10% of the composite data.","SFT on the composite task creates a memorization trap: high in-distribution scores with near-zero out-of-distribution transfer, so composite-task accuracy alone is a poor predictor of RL readiness.","Removing either atomic skill from the base model collapses RL-driven generalization, so both parametric and contextual foundations are necessary prerequisites, not optional aids.","For atomic-trained bases, RL creates reasoning that is not reachable by sampling from the base distribution, whereas for composite-trained bases RL only boosts the probability of already-present paths.","The atomic-first recipe maintains its advantage across model sizes from 0.5B to 3B parameters, with the largest zero-shot gap at the largest scale."],"fun_headline_variants":["Atomic-first RL synthesizes reasoning; composite SFT just memorizes","RL needs atomic skills to build new reasoning paths","Supervised on composite memorizes; RL on atomics generalizes","Skill synthesis via RL requires atomic foundation","RL only amplifies unless atoms are trained first"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pass@k non-convergence test assumes that 512 samples are enough to reveal any latent strategy in the SFT model; if the composed strategy exists in the base distribution with probability lower than roughly 1/512, the diagnosis of synthesis over amplification would be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Atomic-first RL synthesizes reasoning; composite SFT just memorizes","RL needs atomic skills to build new reasoning paths","Supervised on composite memorizes; RL on atomics generalizes","Skill synthesis via RL requires atomic foundation","RL only amplifies unless atoms are trained first"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1089,"prompt_tokens":732,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":476,"tokens_out":357,"duration_ms":4400,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:03:04.375600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the atomic-SFT base and sample k up to 2^16 or more on zero-shot complementary questions before RL; if pass@k eventually catches up to the RL model's zero-shot accuracy, the persistent gap at 512 is a sampling-depth artifact rather than evidence of a new circuit. Alternatively, run RL with the same composite data but an answer-independent reward; if zero-shot gains persist, generalization comes from exposure rather than reinforcement.","supporting_citations":[],"review_version":1}