{"id":"1d092156-f06a-4db2-86ca-41524d79e0d4","arxiv_id":"2603.00963","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LCO replaces PPO-style policy gradients with regression toward the advantage-derived optimal logits/policy, restoring logits-level convexity and yielding more stable LLM RL training.","lead":"Reinforcement learning for large language models is unstable; this paper argues the reason is that PPO's loss is not convex in the model's pre-softmax logits, while supervised fine-tuning's loss is. It proposes Logits Convex Optimization, which regresses the policy to a target derived from the RL objective, and reports more stable training and better scores on math, reading, and instruction-following benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theoretical guarantees assume a fixed target z*; the LCO algorithm as run uses a continuously updated target, so Prop. 4.4/4.8–4.9 do not apply to the reported method.","rationale":"The reader identified Assumption 4.2 (NTK-style linearity) as the weakest assumption. That is a valid concern: if the first-order Taylor expansion fails, Proposition 4.4 provides no bridge from logits convexity to parameter-space behavior. However, a more direct and potentially more damaging issue is that the theory analyzes a fixed target z*, while the algorithm as described and evaluated uses a moving target. Section 4.1 derives z* from a behavioral policy πold, and the convergence proofs in Appendix H treat zold as the initial logits, giving a stationary objective. Yet Section 6.1 claims LCO 'facilitates a dynamic optimization process that continuously aligns with evolving targets,' meaning the target is recomputed as the policy changes. With a non-stationary target, the convexity of each instantaneous loss does not yield the stated linear convergence or a fixed point toward which gradient descent is guaranteed to head. Thus, even under Assumption 4.2, the central stability theorem does not apply to the actual algorithm. The empirical results may still be valid and useful, but the theoretical explanation of stability is not supported by the presented proofs. The reader's verdict of CONDITIONAL remains appropriate: the paper should either clarify and analyze the moving-target version or restrict the theory to a fixed-target setting and adjust the claims. I do not see grounds to reject the empirical contribution outright, so the verdict is unchanged.","tokens_in":23529,"tokens_out":9926,"duration_ms":96895,"concrete_test":"Run LCO-KLD on MATH500 with Qwen-2.5-3B in two variants: (A) fixed target z* computed once from the initial SFT logits zold plus DPO advantage A/β and never updated; (B) moving target recomputed each gradient step from the current policy's logits plus the same A/β. Compare final Pass@1, gradient-norm traces, and the per-step target displacement ∥z*_{k+1}−z*_k∥. If (A) matches (B), the fixed-target theory covers the experiment; if (B) is substantially better or different, the reported results depend on the dynamic target and the fixed-target propositions do not explain them.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is not only Assumption 4.2 but a mismatch between the theory and the deployed algorithm. All theoretical guarantees—Proposition 4.4's gradient directionality and Propositions 4.8–4.9's linear convergence—are derived for a fixed optimal target z* = zold + A/β, where Appendix H.1 explicitly calls zold 'the initial logits.' In the experiments, however, LCO is described as a 'dynamic optimization process that continuously aligns with evolving targets' (Sec. 4.1, Sec. 6.1), so zold and hence z* change as the policy updates. Under a moving target the loss is non-stationary; logits convexity relative to a shifting minimizer does not imply the proved convergence bounds or the 'not misled by spurious stationary points' guarantee, since there is no fixed θ* toward which the gradient is directionally aligned. The empirical gains could therefore stem from advantage-weighted regression dynamics rather than from the logits-convexity stabilization the paper claims. This concern is internal: it holds even if one grants Assumption 4.2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies instability in RL fine-tuning of LLMs and proposes that convexity of the loss with respect to the final logits is a key stabilizing property. It observes that the SFT negative log-likelihood is logits-convex, whereas the clipped PPO surrogate is not, and uses this contrast to motivate a family of objectives called Logits Convex Optimization (LCO). Three LCO variants are defined: regression to the optimal logits z* = z_old + A/β via MSE or log-cosh, and forward KL alignment to the corresponding optimal policy π*. The theoretical sections claim that, under an NTK-style linearity assumption on logits as a function of parameters, logits convexity yields favorable parameter-space gradient directionality (Prop. 4.4), bounded and decreasing gradient norms (Prop. 4.7), and linear convergence for LCO-MSE/LCH (Props. 4.8–4.9). Experiments on math reasoning, QA-Feedback, and AlpacaEval 2.0 across several Qwen, Llama, and Mistral backbones report that LCO variants outperform PPO, GRPO, DAPO, and GSPO, with smoother gradient dynamics and better sample efficiency.","tokens_in":23752,"tokens_out":6593,"duration_ms":78498,"significance":"If the theoretical claims were fully supported, the paper would provide a useful and simple design principle for RL fine-tuning: choose surrogate losses that are convex in logit space. The empirical evaluation is a genuine strength: results are reported across multiple model families, tasks, and random seeds, and the LCO variants consistently beat strong baselines on several benchmarks. The paper also gives explicit, checkable derivations of logits convexity for SFT, PPO, and the proposed losses, and the proposed objectives are easy to implement. However, the load-bearing theoretical bridge from logits convexity to parameter-space stability rests on an unverified NTK-scale linearity assumption, and there is a mismatch between the fixed-target theory and the dynamic-target algorithm actually run in the experiments. The empirical findings may stand on their own, but the central theoretical guarantee as stated does not apply to the reported method.","major_comments":[{"comment":"There is a mismatch between the theoretical setting and the deployed algorithm. Proposition 4.4 and Propositions 4.8–4.9 are derived for a fixed optimal target z* = z_old + A/β; Appendix H.1 explicitly identifies z_old as 'the initial logits.' In the experiments, however, §6.1 states that LCO 'facilitates a dynamic optimization process that continuously aligns with evolving targets.' With a moving target there is no fixed θ* toward which the gradient is directionally aligned, and the linear-convergence proofs of Props. 4.8–4.9 do not apply. The non-stationarity is not a minor presentational gap: it concerns the central claim that the reported method enjoys the proven stability guarantees. Please either extend the theory to a moving target, or restrict the guarantees to a regime in which targets are frozen and clearly state that the deployed algorithm is a heuristic extension.","section":"§4.3 / §6.1 / Appendix H.1"},{"comment":"Assumption 4.2 is the only bridge from logits convexity to parameter-space gradient behavior, but it is asserted rather than verified. The first-order Taylor expansion z_θ* ≈ z_θ + ∇_θ z_θ^T (θ*−θ) with negligible higher-order terms is an NTK-regime condition. LLMs fine-tuned in practice are not in that regime; individual parameters can change enough that logit residuals are material, and no experiment in the paper checks the size of the residual. If Assumption 4.2 fails, Proposition 4.4 provides no parameter-space guarantee and Props. 4.8–4.9 do not follow. At minimum, the paper should state this as an idealized condition and provide an empirical diagnostic (e.g., measure the Taylor residual on actual checkpoints).","section":"§4.3, Eq. (14)"},{"comment":"The inequality in Eq. (15) is not valid as stated for the set Θ* defined in the text. The authors define Θ* = {θ* : L(z_θ*) − L* < ε}, i.e., θ* is only near-optimal. For a logits-convex L, the first-order characterization gives ⟨∇_z L, z−z*⟩ ≥ L(z) − L(z*) ≥ −ε, not necessarily ≥ 0. The claimed ≥0 requires z* to be an actual minimizer of L. This weakens the 'not misled by spurious stationary points' conclusion: the directional guarantee has an ε slack. The proof should either use exact minimizers or state the bound with the ε term.","section":"§4.3, Prop. 4.4 and definition of Θ*"},{"comment":"The justification for the target z* = z_old + A/β as the unique minimizer of the convergence bound relies on the claim that advantage normalization makes the mean of A over the vocabulary zero. This is not true for the sparse sampled-action estimator in Eq. (11), where exactly one entry of A is nonzero and all others are zero, so the mean is A(s_t,a_t)/|V|, not zero. The sparse setting is explicitly used in the experiments reported in Table 5. The paper should either restrict the zero-mean claim to the dense advantage estimators, or account for the nonzero mean in the convergence-bound minimization.","section":"§4.4, Eq. (11) and Table 5"},{"comment":"Proposition 4.7 is used to support the claim that LCO 'effectively precludes the sudden, high-magnitude gradient spikes' seen in PPO. The bounds in Eqs. (16)–(18) are upper bounds involving σ_max, the maximum singular value of ∇_θ z_θ. If σ_max itself grows during training, the bound can grow even as the loss decreases. The statement that gradient norms 'scale with remaining error and diminish progressively' therefore requires an additional assumption that σ_max remains bounded. The experiments may show this empirically, but the theoretical claim as written is stronger than what Prop. 4.7 proves.","section":"§4.3, Prop. 4.7"}],"minor_comments":[{"comment":"Typo: 'proposeLogits Convex Optimization' should read 'propose Logits Convex Optimization'.","section":"§1"},{"comment":"The statement that PPO is 'not logits convex at any time step' is too strong. The Hessian computation in Eqs. (39)–(42) is restricted to the active, unclipped region; in clipped regions the gradient is zero and the Hessian is PSD (zero). The correct statement is that the PPO surrogate is not convex in general on the active region; there exist states where the Hessian is indefinite.","section":"§4.3, Lemma 4.5"},{"comment":"The text oscillates between calling z* 'a specific solution' and 'the unique optimal logits.' Because of softmax translation invariance, z* in Eq. (7) is one representative of an equivalence class. The later discussion in §4.4 acknowledges this, but the earlier wording should be made consistent.","section":"Appendix D / §4.1"},{"comment":"The DPO-based advantage estimator A(s_t,a_t) := log φ_DPO(a_t|s_t)/φ_ref(a_t|s_t) is presented as a direct consequence of the Bellman equation. The derivation is compressed; please spell out the assumptions on the reward model and token-level MDP so the reader can verify the identification.","section":"§4.2, Eq. (13)"},{"comment":"The phrase 'in modern reinforcement learning practice, it is standard to employ advantage normalization at each time step' conflates normalizing advantages over a batch of sampled actions with normalizing a dense advantage vector over the full vocabulary. These are different operations and only the latter yields mean zero over V.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The conditional verdict from the first reader is justified. The strongest concern is not just Assumption 4.2, but the fixed-target versus dynamic-target mismatch: the paper's own §6.1 says the method uses continuously evolving targets, while the proofs in §4.3 and Appendix H assume a fixed z*. This is a central gap, not a presentational one. The empirical results are substantial and could support a weaker, more honest claim: LCO is a stable, well-performing policy-optimization heuristic whose design is motivated by logits convexity under an idealized linearization. I do not see evidence of misconduct, but the introduction and abstract currently overstate the theoretical guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a stability story that is stretched past its theoretical reach. The core observation—SFT's loss is convex in logits while PPO's clipped surrogate is not—is correct and worth stating clearly. The LCO losses are simple regressions or distillations toward the closed-form optimal policy of the KL-regularized RL objective, and the paper shows they train stably and beat PPO, GRPO, DAPO, and GSPO across three tasks and four backbones. That empirical result is the real contribution, and it is credible: multiple seeds, reported error bars, consistent wins, and a sample-efficiency plot that makes sense. Credit where due: the experiments are broad and honestly reported.\n\nThe theory is where I part ways. Proposition 4.4 depends on Assumption 4.2, an NTK-scale linearity of logits in parameters that LLMs plainly violate. Even granting that assumption, the proved inner-product inequality is weak—it says nothing about perpendicular gradient components, which can still send updates sideways. Lemma 4.5 also overstates the PPO case: in clipped regions the objective is constant, so the Hessian is zero and the loss is locally convex; only the active region is non-convex. Those are fixable.\n\nThe bigger problem is internal. All the guarantees—Prop 4.4 and Props 4.8–4.9—use a fixed target z* = z_old + A/β, with z_old explicitly called 'the initial logits' in Appendix H.1. But the experiments run a dynamic target that 'continuously aligns with evolving targets' (Sec. 6.1), and the paper credits that dynamism for beating π*. So the theory never applies to the reported method. The gains may simply be advantage-weighted regression with a moving target—a known-working recipe—and the logits-convexity story is not what the proofs establish. No code or data are released, and the LCO-LCH convergence bound relies on a tanh≈x linearization that holds only near the optimum.\n\nWho gets value: anyone working on LLM alignment who wants a practical, stable alternative to PPO-family methods. The recipe is plausible and worth adopting. Who should be careful: anyone citing the stability theory as established. I would send this to peer review, because the empirical study deserves scrutiny and the fixed-versus-dynamic target gap is a sharp, answerable question for the authors. The next version needs to either prove something for the moving target or weaken the claims substantially.","headline":"Useful empirical recipe wrapped in a theory that doesn't cover the algorithm actually run; the fixed-target proofs are the main weakness.","tokens_in":24320,"tokens_out":2431,"would_cite":false,"duration_ms":27096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that RL instability in large language models is governed by whether the training loss is convex in the model's logits, and that replacing PPO's clipped surrogate with logits-convex targets stabilizes and strengthens policy","keywords":["logits convexity","policy optimization","large language model RL","PPO instability","KL-regularized RL","forward KL","sample efficiency","gradient directionality"],"falsifier":"Record both inner products in Equation (15) during an actual LCO training run near convergence; if at any step the parameter-space inner product ⟨∇θL, θ−θ*⟩ is clearly negative while the logit-space inner product ⟨∇zL, z−z*⟩ is non-negative, then the first-order linearization of logits has failed and Proposition 4.4 provides no guarantee at that point. Alternatively, exhibit one training state where the curvature of the PPO loss with respect to the logits in the non-clipped region is positive semidefinite, which would contradict Lemma 4.5's blanket non-convexity claim.","tokens_in":23330,"feed_emoji":"🎯","tokens_out":8815,"duration_ms":85831,"temperature":0.7,"pith_summary":"The paper tries to explain why reinforcement learning is so much less stable than supervised fine-tuning when training large language models, and to fix it. Its claim is that the deciding factor is 'logits convexity': whether the loss, viewed as a function of the pre-softmax logits, is convex. SFT's negative log-likelihood has this property; PPO's clipped surrogate does not, which the paper links to gradient explosions and training collapse. It then introduces Logits Convex Optimization (LCO), a family of losses that regress or distill toward the closed-form optimal policy and are convex in logits. If the argument is right, logits convexity is a design principle for stable LLM RL, and LCO should consistently beat PPO-family methods, which the reported experiments support.","feed_headline":"Convex logits keep LLM RL training stable; PPO lacks them","feed_subtitle":"Replacing clipped surrogates with convex-logit targets beats PPO-family baselines on math, QA, and instruction following.","key_machinery":"The load-bearing object is 'logits convexity' — positive semi-definiteness of the loss's Hessian with respect to the pre-softmax logits. Proposition 4.4 is the mechanism: via a local first-order Taylor expansion of logits in the parameters, logits convexity transfers to a non-negative inner product between the parameter-space gradient and the displacement toward near-optimal parameters. The LCO objectives are constructed to make this property hold at every time step, and Proposition 4.7 converts it into gradient norms that shrink as the loss shrinks.","core_discovery":"The central claim is that gradient-based policy optimization in LLMs is stabilized when the loss is convex with respect to the logits. Under a first-order linearization of the logits as a function of parameters (Assumption 4.2), logits convexity yields an inner-product inequality — the parameter-space gradient and the logit-space gradient align with the direction toward near-optimal parameters — so gradient descent is not drawn into spurious stationary points. The paper proves SFT's NLL loss is logits-convex, PPO's clipped surrogate is not, and the LCO losses (MSE, log-cosh, forward KL) are all logits-convex. The optimal target is derived in closed form: π* is proportional to π_old exp(A/β),","pith_inferences":["Editorial extension: the paper's mechanism can be tested directly by watching the smallest Hessian eigenvalue of the policy loss with respect to logits during training; stable runs should keep it non-negative, and PPO-style runs should show negative directions exactly when gradient spikes occur.","Editorial extension: the closed-form optimal target is not tied to PPO's clipping heuristic, so any RL or distillation algorithm that matches this target with a convex loss — including GRPO/DAPO variants or online distillation — should inherit similar stability.","Editorial extension: the analysis suggests future RL losses could be designed by choosing a surrogate whose logit Hessian is PSD, making clipping and trust regions less necessary for stability and replacing them with target construction.","Editorial extension: since the directionality proof relies on a local linearization of logits in parameters, its predictions are most testable near convergence; comparing gradient-direction violations in early versus late training would reveal where the linearity assumption needs to be replaced."],"forward_implications":["If logits convexity is the key to stable LLM RL, then SFT's stability is not incidental: it follows from the PSD Hessian of the cross-entropy loss, and any surrogate that lacks this property is prone to gradient spikes and collapse.","Gradient norms for LCO losses are bounded by monotone functions of the loss value, giving a self-stabilizing optimization trajectory that avoids the exploding gradients seen in PPO.","Because LCO matches the same closed-form optimum as the KL-regularized objective, it is consistent with PPO's goal while being more stable, and the reported results show it outperforms PPO, GRPO, DAPO, and GSPO across math reasoning, QA-Feedback, AlpacaEval, and OOD MMLU.","LCO also improves sample efficiency: LCO-KLD reaches PPO-level MATH500 performance in about half the samples, and LCO-LCH in about a third.","The specific target z* = z_old + A/β is justified as the shift that minimizes the initial logit-space discrepancy, giving the tightest convergence bound for the MSE and log-cosh variants when advantages are mean-centered."],"fun_headline_variants":["Logits convexity stabilizes RL training; PPO lacks it","Replace PPO's clipped loss with convex-logit targets","Why PPO is unstable: missing logits convexity","LCO: convex logit targets for stable LLM RL","Stable RL via logits convexity, unlike PPO"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The argument depends on Assumption 4.2: near the optimum, the network's logits must be an approximately linear (first-order) function of the parameters, so that convexity in logit space carries over to parameter-space gradient directionality — a regime real LLMs are typically far from.","fun_headline_variants_meta":{"raw":{"variants":["Logits convexity stabilizes RL training; PPO lacks it","Replace PPO's clipped loss with convex-logit targets","Why PPO is unstable: missing logits convexity","LCO: convex logit targets for stable LLM RL","Stable RL via logits convexity, unlike PPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1125,"prompt_tokens":731,"completion_tokens":394,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":475,"tokens_out":394,"duration_ms":3963,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:45:47.296121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record both inner products in Equation (15) during an actual LCO training run near convergence; if at any step the parameter-space inner product ⟨∇θL, θ−θ*⟩ is clearly negative while the logit-space inner product ⟨∇zL, z−z*⟩ is non-negative, then the first-order linearization of logits has failed and Proposition 4.4 provides no guarantee at that point. Alternatively, exhibit one training state where the curvature of the PPO loss with respect to the logits in the non-clipped region is positive semidefinite, which would contradict Lemma 4.5's blanket non-convexity claim.","supporting_citations":[],"review_version":1}