{"id":"4e64ef99-4319-4414-8eb5-1e88d55cd31e","arxiv_id":"2511.16886","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Recursive reasoning in TRMs is reinterpreted as policy improvement, and a stepwise denoising supervision scheme (DIS) cuts forward passes 18× while improving small-model ARC scores.","lead":"This paper argues that each recursive reasoning step in Tiny Recursive Models can be viewed as a policy-improvement step, and introduces a training scheme that supervises every step toward progressively cleaner targets. The scheme reportedly cuts forward passes 18× and raises ARC-AGI-1 pass@2 from 12% to 24% on a 0.8M-parameter model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 13's identification of post-reasoning policy with p(a|s,o=1) is asserted, not derived; the policy-improvement claim reduces to relabeling if it fails.","rationale":"The reader's weakest assumption identifies Eq. 13 as the crux; I agree. This is the single most load-bearing step: every subsequent formal result (Bayes inversion, product policy, Proposition 4.1) depends on interpreting the post-reasoning policy as p(a|s,o=1). Without a derivation or empirical support for this identification, the paper's central claim is not established. The concern is internal inconsistency, not disagreement with consensus: the justification offered (conditioning on x) does not entail conditioning on optimality, and the formalization defines 'improvement' in terms of the model's own distributions, making the conclusion tautological once Eq. 13 is granted. I considered other potential issues—lack of code/error bars, compute comparison—but these are secondary and do not undermine the theoretical claim as directly. The empirical DIS results are suggestive and may merit replication, but they do not validate the theory; they only show that a corruption-based curriculum can improve TRM performance. The proposed test would directly assess whether the log-ratio signal correlates with task success, which is the empirical content of Eq. 13. If the test passes, the theory gains support; if it fails, the central claim is refuted. Therefore the reader's REJECT verdict stands, though confidence remains moderate because the test has not been run.","tokens_in":13030,"tokens_out":3714,"duration_ms":33961,"concrete_test":"On a trained TRM (without DIS), evaluate the Advantage Margin condition (Eq. 18) on held-out examples: at each recursion step t, compute the log-ratio A_t(a) = log π+t(a|s) - log π̂t(a|s) for the ground-truth token and for actions sampled from the reference policy. Then measure the rank correlation between A_t and an external correctness signal (e.g., token-level accuracy or final puzzle success). If the correlation is not significantly positive across steps, the identification of the log-ratio with an improvement signal (Eq. 13-17) is empirically falsified, and the policy-improvement interpretation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a TRM step is a policy improvement operator rests on Eq. 13: π+t(a|s) ≈ p(a|s,o=1). This is the bridge that converts the post-reasoning distribution into an 'optimality-conditioned' policy, enabling the Bayes inversion in Eq. 14 and the geometric policy family in Eq. 16. The paper's justification (§3.2) is that zt+1_L is conditioned on the input embedding x, unlike 'standard one-step feed-forward inference.' This conflates conditioning on the input with conditioning on task success. Any learned function of x is conditioned on x; that says nothing about whether the output is optimal. The reference policy also depends on x (through zt_H), so the contrast is false. If Eq. 13 does not hold, the log-ratio in Eq. 17 is not an advantage-like signal, and the product policy in Eq. 16 is just an arbitrary geometric interpolation of two model distributions—not an RL improvement operator. The formalization then becomes circular: the 'improvement factor' is defined as the log-ratio of the model's own pre/post policies, so the conclusion that the step improves the policy holds by construction, not by any external notion of improvement. Proposition 4.1 only shows that training to concentrate on a target makes that target's log-ratio margin positive; it never links the log-ratio to expected return or task reward. Thus the theoretical core is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that each latent-reasoning step in Tiny Recursive Models (TRMs) can be formalized as a policy-improvement operator: the post-reasoning distribution π+t is interpreted as the policy conditioned on optimality p(a|s,o=1); by Bayes inversion, the log-ratio between post- and pre-reasoning policies becomes an advantage-like signal; and a geometric family of policies π_{t,w} is derived. Based on this, the authors propose Deep Improvement Supervision (DIS), which supervises post-reasoning outputs on intermediate targets generated by a discrete corruption schedule. Experiments on N-Queens and ARC-AGI show DIS improving over TRM baselines at equal capacity while using fewer latent steps.","tokens_in":13419,"tokens_out":8090,"duration_ms":70866,"significance":"If valid, the formalization would unify latent recursive reasoning with KL-regularized RL and provide a principled training objective. The empirical DIS method is interesting: a 0.8M-parameter model achieves 24% pass@2 on ARC-AGI-1, and the paper includes pseudocode and a proof appendix. However, the central identification (Eq. 13) is assumed, not derived, and the improvement claim is largely circular because the advantage signal is defined as a log-ratio of the model's own pre/post policies. The theory therefore does not yet explain the empirical gains, and the gains themselves are not isolated from confounds.","major_comments":[{"comment":"The key identification π+t(a|s)≈p(a|s,o=1) is asserted without derivation or empirical support. The justification that z_{t+1}^L is conditioned on the input embedding x does not distinguish it from any learned function of x; the reference policy π̂t also depends on x through z_H^t. No definition of the random variable o or its relationship to task reward is provided. This assumption is load-bearing: the Bayes inversion in Eq. (14), the improvement score in Eq. (17), and the policy family in Eq. (16) all collapse if (13) fails. As written, the paper conflates conditioning on the input with conditioning on task success.","section":"§3.2, Eq. (13)"},{"comment":"The claimed policy-improvement property is circular. A_t(s,a) is defined as log π+t(a|s) − log π̂t(a|s), i.e., the log-ratio of the model's own output distributions. The theorem quoted from Frans et al. (2025) guarantees improvement for product policies of the form π ∝ π̂ f(A_π̂) when A_π̂ is a true advantage function Q_π̂ − V_π̂. Here no Q or V is defined or learned, and A_t is not shown to be a monotone function of any such advantage. Consequently, the statement that recursion implements a policy-improvement operator is a relabeling: it holds by construction for any two distributions of the same model, not by any connection to expected return or task reward.","section":"§3.4, Eqs. (14)-(17)"},{"comment":"The proof does not establish the stated result. Step 2 asserts that minimizing CE(ℓ_c, y†_s) drives π+t(y†_s)→1, which is an idealized assumption about optimization success and model capacity, not a consequence of the loss. More importantly, the conclusion is a distributional concentration statement about the model's own log-ratio A_t; it never relates A_t to the scoring distribution P or to any external notion of improvement. The assumption log P(y†_s)/P(y†_s−1)>0 is introduced only in the final sentence and is not used in the derivation. Thus the proposition is a consistency property of the geometric policy family, not a theorem about reward or performance improvement.","section":"Appendix B, Proposition 4.1"},{"comment":"The 18× forward-pass reduction compares DIS's 18 steps (Nsup=6,T=1,n=2) to TRM's worst-case 336 steps (Nsup=16,T=3,n=6). TRM has a learned halting mechanism, so its actual test-time compute is likely lower; no measurements of actual passes with halting are reported. Furthermore, DIS differs from the TRM baselines in several components simultaneously: the corruption-schedule targets, the integer time-step conditioning, the fixed Nsup, and the absence of a halting head. Without ablations (e.g., TRM with the same recursion schedule but standard target supervision; DIS with Nsup=16), the performance gains cannot be attributed to the policy-improvement mechanism. The paper also reports no error bars or multiple seeds, so the 40.4 vs 41.3 and 27.1 vs 40.0 differences are not assessed for statistical significance.","section":"§5, Table 1 and compute claims"}],"minor_comments":[{"comment":"The title on the first page ('Your Latent Reasoning is Secretly Policy Improvement Operator') differs from the arXiv title ('Latent Reasoning in TRMs is Secretly a Policy Improvement Operator'); please harmonize.","section":"Title"},{"comment":"The definition of o=1 as 'a equals the ground-truth token' is not immediately compatible with the ARC grid-output setting; clarify how o is defined for non-token outputs.","section":"§3.1, footnote 1"},{"comment":"The sentence 'The task specific hyperparameters as the hidden layers size and reasoning steps are presented below' is garbled; also Table 1 does not list all hyperparameters for the runs.","section":"§5.3"},{"comment":"The claim 'we can avoid dead compute steps' is never directly measured; consider reporting per-step contribution to accuracy to support this.","section":"Abstract / §5.4"},{"comment":"The statement that the authors 'replicated the TRM experiments and achieved slightly lower results' is appreciated, but it is unclear which hyperparameters differed from the original TRM; please specify.","section":"§5.4"},{"comment":"The LLM-generated trajectory experiment is reported as achieving 10% accuracy with no experimental details; either provide a full setup or remove the claim.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"The stress-test concern in the reader's report is valid: Eq. (13) is the unproven bridge, and without it the policy-improvement claim reduces to a relabeling of pre/post distributions. The DIS empirical findings may be publishable as an empirical curriculum-learning result, but the current theoretical framing does not add explanatory power and the experiments lack the ablations needed to attribute gains. I would suggest the authors substantially reframe the contribution or provide direct evidence that the post-reasoning policy is indeed optimality-conditioned (e.g., by measuring calibration with respect to task success)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing worth knowing about this paper is that the empirical recipe—train a tiny recursive model with stepwise denoising targets derived by corrupting the ground truth—might be genuinely useful. The numbers are striking: 24% pass@2 on ARC-AGI-1 with a 0.8M parameter model and 18× fewer forward passes than TRM. If that replicates, it's a real result.\n\nBut the theoretical framing does not hold up. The paper tries to argue that a single TRM step is a policy improvement operator because the post-reasoning policy can be interpreted as p(a|s,o=1) (Eq. 13). The justification is that the latent update is conditioned on the input x, which is a non-sequitur: every learned function of x is conditioned on x; that says nothing about optimality. The stress-test note is right—this is asserted, not derived.\n\nOnce you grant Eq. 13, the rest of the machinery follows by Bayes inversion and log-ratio reweighting, but the 'improvement factor' is just the log-ratio of the model's own pre- and post-reasoning distributions. So the conclusion that recursion improves policy is built in by definition. Proposition 4.1 makes this plain: it shows that if you train the post-reasoning policy to assign high probability to the target, then the log-ratio advantage for that target is positive. That's a tautology about the training objective, not a statement about expected return.\n\nThe empirical section is also under-supported: no code, no seeds, no error bars. The compute comparison is not apples-to-apples—they compare their T=1,n=2 against TRM's T=3,n=6 and note they removed the halting head. The 18× reduction in forward passes is partly a consequence of those architectural simplifications, not purely the supervision scheme.\n\nI don't want to overstate the problems. The DIS idea is simple and could be useful in small-data regimes. The authors are honest about the LLM-trajectory failure and about the fixed-step limitation. But the paper as written overclaims: the theory section should be either substantially strengthened or reframed as intuition.\n\nOn balance, the paper should go to peer review, because the empirical claim, if true, deserves scrutiny and replication. But it shouldn't be accepted as-is. I'd ask for the code, seeds, and a proper ablation that controls for the architecture changes. The policy-improvement formalization should be demoted to a heuristic unless the authors can actually justify the optimality-conditioning step.\n\nWeigh in: I'd bring this to a reading group to discuss the training method, but I wouldn't cite it yet.\n\nBest.","headline":"Clever training trick, but the policy-improvement theory is circular; the empirical claims need code and error bars before being trusted.","tokens_in":13911,"tokens_out":3527,"would_cite":false,"duration_ms":29661,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single latent-reasoning step in a recursive model is formally a policy improvement update, and exploiting this view yields a training method that matches accuracy with 18x fewer forward passes.","keywords":["latent reasoning","recursive models","policy improvement","advantage margin","deep improvement supervision","credit assignment","ARC-AGI","discrete corruption"],"falsifier":"Take a trained recursive reasoning model and look at a step where the model already outputs the correct answer; decode the pre- and post-reasoning policies and check whether the log-ratio advantage of the correct token is actually above the average across tokens. If it is not, that step fails the paper's own improvement condition and the policy-improvement interpretation does not describe the model's behaviour.","tokens_in":12908,"feed_emoji":"🧠","tokens_out":6958,"duration_ms":56623,"temperature":0.7,"pith_summary":"Small looped reasoning models improve their answers by iterating latent states, but it was unclear when an extra iteration actually helps. This paper argues that each latent reasoning step is secretly a policy improvement update: the model's current guess plays the role of a reference policy, and the output after one update plays the role of a policy conditioned on success. Decoding before and after the update yields two distributions whose log-ratio is an advantage-like signal, and the correct token should have above-average log-ratio for the step to be useful. On that basis the paper proposes Deep Improvement Supervision, which supplies per-step targets that get progressively closer to the true answer, and shows that this removes wasted compute and cuts the number of forward passes 18-fold while matching or exceeding prior recursive models on ARC-AGI tasks.","feed_headline":"Each latent reasoning step is a policy improvement update","feed_subtitle":"A stepwise supervision scheme built on this view cuts forward passes 18x on ARC-AGI while matching accuracy.","key_machinery":"The load-bearing object is the log-ratio advantage A_t(s,a) = log π⁺_t(a|s) − log π̂_t(a|s), the difference between the post-reasoning and pre-reasoning output distributions of a single recursion step. This ratio simultaneously defines an implicit optimality likelihood (via Bayes' rule) and the one-step improvement family π_{t,w} ∝ π̂_t^{1−w} π⁺_t^w. It does the work of converting an uninterpreted recursive update into a testable policy-improvement operator, and gives the training criterion: the correct action must have above-average log-ratio advantage under the interpolated policy.","core_discovery":"The paper's central claim is that a single recursive reasoning step implements a policy improvement update: it takes a reference policy (the model's current guess) and transforms it into an improved policy using an internally computed signal. Decoding the latent state before and after one update gives two output distributions, and the paper interprets the post-update distribution as approximating the policy conditioned on the event that the answer is optimal. Applying Bayes' rule turns this into a multiplicative reweighting of the reference policy, with the log-ratio of the two distributions serving as an advantage-like improvement score. The resulting update family is a geometric interpolat","pith_inferences":["If the geometric interpretation is taken seriously, the interpolation weight w could be tuned at inference time as a trust knob over the model's own reasoning, a move the paper does not explore.","The theory predicts that the benefit of extra recursion steps should vanish exactly when the log-ratio advantage stops separating the correct token; tracking this margin over training would give a principled stopping rule.","The paper's underperformance of LLM-generated improvement trajectories (10% vs 24% for the corruption schedule) suggests that the form of the intermediate targets—monotone and local—matters more than their semantic quality; a testable extension is to generate targets by solving subproblems incrementally."],"forward_implications":["Every recursion step can be audited: if the correct token's log-ratio advantage is not above average, that step is dead compute and can be skipped or corrected.","Supervising each step with an improving target sequence converts long-horizon credit assignment into per-step supervised sub-goals, which is what allows a 0.8M-parameter model to reach 24% on ARC-AGI-1.","Because the post-reasoning policy defines a whole family of interpolated policies, the same architecture implicitly parameterizes a continuum of trust levels in its own reasoning, indexed by w.","A fixed number of supervision steps can replace a learned halting mechanism, eliminating the extra forward pass typically needed to train the halting signal.","The improvement-margin condition (Eq. 18) is a directly testable diagnostic for when latent recursion helps rather than degrades a prediction."],"fun_headline_variants":["Latent reasoning is a policy update, cutting passes 18x","Each latent step is a policy improvement, enabling 18x speedup","Why latent reasoning works: it's a policy improvement step","Recursive steps are policy updates, eliminating dead compute","Policy improvement view of latent reasoning trims 18x passes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the assumption that the post-reasoning output distribution is close to the policy conditioned on the answer being optimal; the paper asserts this rather than deriving it, and its stated justification conflates conditioning on the input with conditioning on task success.","fun_headline_variants_meta":{"raw":{"variants":["Latent reasoning is a policy update, cutting passes 18x","Each latent step is a policy improvement, enabling 18x speedup","Why latent reasoning works: it's a policy improvement step","Recursive steps are policy updates, eliminating dead compute","Policy improvement view of latent reasoning trims 18x passes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3300,"prompt_tokens":709,"completion_tokens":2591,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":2519}},"tokens_in":453,"tokens_out":2591,"duration_ms":14862,"temperature":1.0,"reasoning_tokens":2519,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:01:48.571883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained recursive reasoning model and look at a step where the model already outputs the correct answer; decode the pre- and post-reasoning policies and check whether the log-ratio advantage of the correct token is actually above the average across tokens. If it is not, that step fails the paper's own improvement condition and the policy-improvement interpretation does not describe the model's behaviour.","supporting_citations":[],"review_version":1}