{"id":"2bf00e98-0a94-4d09-86f1-506a9f44e338","arxiv_id":"2509.02522","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PACS trains an LLM to predict its own correctness signal with cross-entropy loss, and the gradient of that loss contains a policy-update term, yielding large reported gains on AIME math benchmarks.","lead":"This paper proposes a new way to train reasoning language models: treat the right/wrong answer check as a label and train the model to predict that label with a standard classification loss, while a hidden term still behaves like a policy update. The authors report large score improvements over PPO and GRPO on math exams, making the approach worth testing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gradient analysis in Eqs. 3/6 and the implementation story in Sec. 3.4.1 describe different updates: one includes a REINFORCE score-function term, the other only the direct ψ-derivative. The paper never states which is used, and the two disagree on how correct low-ψ samples are updated.","rationale":"The reader's weakest assumption concerns drift of the log-probability ratio and the reset schedule. I identified a different, more fundamental issue: the paper's theoretical gradient (Eqs. 3 and 6) includes a score-function term that the intuitive loss analysis in Sec. 3.4.1 ignores, and the two updates behave differently for correct samples with low ψ. Since the paper does not state which gradient is implemented, the central claim that PACS 'inherently recovers the classical policy gradient' is not pinned down by the text. The concern is not an ad hominem or a disagreement with empirical results; it is an internal consistency issue between the formal derivation and the described algorithm. The empirical comparison may still be valid, and the method may work well in practice, but the theoretical contribution and the stability explanation need to specify which objective is actually optimized. Because this ambiguity affects the central claim without disproving the empirical findings, I maintain the reader's CONDITIONAL verdict rather than escalating to REJECT.","tokens_in":17038,"tokens_out":15839,"duration_ms":156745,"concrete_test":"Inspect the released PACS training code and compute the gradient of the implemented loss for one correct rollout with ψ<0. If the implemented update increases πθ(o|q), the REINFORCE term l∇θ log πθ is absent and Eq. 6 is not the implemented gradient; if it decreases πθ(o|q), the full expectation gradient is used. Then rerun the Qwen2.5-7B AIME 2025 experiment with the two variants forced explicitly (full Eq. 6 gradient vs. direct ψ-derivative only) and compare pass@256 and training entropy; the central claim should be evaluated only on the variant the code actually implements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that optimizing the cross-entropy loss in Eq. 1 'inherently recovers the classical policy gradient update' via the actor/critic decomposition of Eq. 6. The derivation of Eq. 6 is correct for the exact expectation E_{q,o~πθ}[l(q,o;πθ)], but it relies on the score-function term ∇θ log πθ(o|q) in Eq. 3. However, Sec. 3.4.1 analyzes the loss as if the sampled outputs were fixed data: it says that for a correct sample the loss reduces to log σ(ψ), so minimizing it increases ψ and hence πθ(oi|q). That is the direct derivative −(R−σ)∇θψ only, and it omits the l∇θ log πθ term. These two updates are not equivalent. For a correct sample with ψ<0 (σ<0.5), the direct derivative increases πθ, but the full Eq. 6 update, with ∇θψ ∝ ∇θ log πθ, contains the additional term l∇θ log πθ = log σ ∇θ log πθ, whose coefficient is negative; the net coefficient log σ + (1−σ) is negative, so the full update decreases the probability of that correct sample. Thus the theoretical gradient and the intuitive 'supervised' description can move in opposite directions. The paper never specifies whether the implementation computes the full expectation gradient or only backpropagates through ψ for the already-sampled rollouts. If the latter, PACS is not recovering the policy gradient from Eq. 6; it is a reward-weighted classification loss, and the actor-critic story is unsupported. If the former, the algorithm has a tendency to suppress correct samples with below-median log-probability ratios, which would be a serious stability problem. This ambiguity is load-bearing because it determines whether the central theoretical claim describes the actual method.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PACS, a method for RLVR that defines a policy-parameterized score function ψ(q,o;πθ) as an RLOO advantage over log-probability-ratio reward proxies and optimizes a binary cross-entropy loss against the verifiable binary reward. The authors derive a gradient decomposition (Eq. 6) into a policy-gradient term and a reward-estimation term and claim that this constitutes an implicit actor-critic coupling that inherently recovers the classical policy gradient update. Experiments on Qwen2.5-3B and Qwen2.5-7B across MATH-500, AMC23, AIME 2024, and AIME 2025 compare PACS with PPO and GRPO, reporting consistent improvements, particularly on AIME at high pass@k. An ablation of a weighting mechanism and a β sensitivity analysis are included, and code is released.","tokens_in":17431,"tokens_out":12150,"duration_ms":113265,"significance":"If the claims hold, PACS would be a simple and effective RLVR trainer with meaningful gains over PPO and GRPO on mathematical reasoning benchmarks, and the open-source release is a practical strength. However, the conceptual significance is limited: the gradient decomposition in Eq. (6) is a chain-rule identity for any on-policy differentiable objective, and the 'critic' is not an independent value estimate but a reparameterization of the same policy. The empirical results are interesting, but the theoretical interpretation, the unspecified class-imbalance weighting, and the unspecified reference-policy reset schedule currently prevent the method from being fully evaluated or reproduced.","major_comments":[{"comment":"Equations (3) and (6) are correct for the full expectation gradient, but they include the score-function term l(q,o;πθ)∇θ log πθ(o|q). The implementation discussion in Section 3.4.1 instead analyzes only the direct derivative through ψ: it says that minimizing the loss for a correct sample maximizes ψ and hence increases πθ(oi|q), which corresponds to the −(R−σ)∇θψ term alone. The manuscript never states whether the implementation backpropagates through ψ for fixed rollouts or computes the full expectation gradient. If the former, the implemented loss is a reward-weighted classification loss, and the central claim that PACS 'inherently recovers the classical policy gradient update' is unsupported; if the latter, the intuitive description in Section 3.4.1 is incomplete because the l∇θ log πθ term can oppose the direct term, e.g., for a correct sample with small σ the score-function term has a negative weight. This distinction is load-bearing for the paper's main claim and must be resolved explicitly.","section":"Section 3.3, Eq. (6); Section 3.4.1"},{"comment":"The class-imbalance weighting mechanism is load-bearing: the ablation in Table 2 shows large drops without it (e.g., AIME 2024 pass@32 falls from 43.90 to 32.58 for Qwen2.5-7B), yet Section 3.4.2 gives no formula for the weights, no statement of how they enter Equation (9), and no hyperparameter values. Citing King and Zeng (2001) is not sufficient to specify the method. Please provide the exact weighted loss and the weight-setting rule.","section":"Section 3.4.2"},{"comment":"The score function depends on the log-probability ratio β log(πθ/πref), and the paper states that this ratio may grow over time and that πref is periodically hard-reset to a recent snapshot with optimizer-state reinitialization. No reset frequency, snapshot-selection rule, or optimizer-state reset procedure is given, and β is chosen empirically (β=1) via the sweep in Figure 3 rather than derived. Because ψ is the entire learning signal, the stability and behavior of PACS hinge on these unspecified choices, so the method is not yet fully defined.","section":"Section 3.4, Eqs. (7)–(8)"},{"comment":"Calling Equation (1) a supervised learning task is misleading: the expectation is over o∼πθ(·|q), so the objective is an on-policy RL objective whose gradient decomposition in Equation (6) is a chain-rule identity for any differentiable on-policy loss. The 'critic' role is not an independent value estimate but a reparameterization of the same policy's log-probabilities, so the 'implicit actor-critic coupling' claim amounts to a naming choice rather than a new algorithmic mechanism. The abstract and Section 3.3 should be revised to avoid overclaiming.","section":"Section 3.2, Eq. (1); Abstract"}],"minor_comments":[{"comment":"At pass@256, the base model outperforms PACS on AMC23 (Base 99.18 vs PACS 98.51) and AIME 2024 (Base 62.55 vs PACS 59.78); this is acknowledged in Section C.2 but should be addressed in the main results to qualify the claim of superior performance at large sampling budgets.","section":"Table 2"},{"comment":"The DeepScaleR reference (Luo et al., 2025) is listed as a Notion blog without a URL or access date; please provide a complete citation.","section":"References"},{"comment":"The 3D heatmaps are difficult to read for the exact β and k values; a tabular version or clearer axis labels would improve interpretability.","section":"Figure 3"},{"comment":"There are typographical artifacts such as 'im Plicit' and 'aSupervised' in the abstract and Section 3.2; please proofread the manuscript.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central theoretical narrative as written is not supported by the implementation details provided. My recommendation of major revision is based on the belief that the authors can resolve the issue by specifying the exact implemented gradient and either revising the claims or explicitly framing PACS as a reward-weighted classification objective with empirical gains. If the implementation indeed uses only the direct derivative through ψ, the title and abstract should be changed to avoid claiming that the method inherently recovers the classical policy gradient update."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PACS is a plausible RLVR variant with large reported gains on AIME, but the paper doesn't currently pin down what algorithm is actually being run, which makes the theoretical story hard to evaluate. The core idea is a new combination: use an RLOO advantage built from log-probability ratios as the score, then train that score with binary cross-entropy where the reward is the label. The building blocks are known, but the specific mix is not something I've seen, and the empirical improvements over PPO and GRPO at pass@256 on AIME 2024/2025 are substantial if real. The algebra in Eq. 6 is correct: the gradient of the expected loss decomposes into a REINFORCE term and a pathwise derivative. The writing is generally clear, and the ablations on advantage estimators and the weighting mechanism are useful.\n\nThe soft spots are more than cosmetic. Most important: the paper never says whether the implementation uses the full expectation gradient from Eq. 6 or just backpropagates through ψ for fixed samples. Those updates are different. For a correct sample with ψ below the group average, the full gradient actually suppresses that sample's probability, while the fixed-sample version increases it. The intuitive 'supervised' description in Sec. 3.4.1 matches the fixed-sample version, which would mean the actor-critic interpretation is unjustified. If they do use the full gradient, there's a stability problem that needs explaining. The code may answer this, but the paper shouldn't leave it implicit.\n\nSecond, two load-bearing hyperparameters are absent: the class-imbalance weighting from King & Zeng is cited but never given a formula, and the reference-policy reset schedule has no period. The ablation shows the weighting is worth many points on AIME, so this isn't a detail. Third, the abstract and intro mislabel the AIME 2024 result (59.78 pass@256) as AIME 2025; the actual AIME 2025 number is 58.22. That's a factual error. Fourth, no error bars or seeds, which matters less given the gap sizes but still.\n\nBottom line: the empirical recipe might be useful, and the paper deserves a serious referee, but it needs major revision before publishability: state the exact update, specify the weighting and reset schedule, and correct the benchmark mix-up. I'd bring it to peer review, and I'd also bring the ambiguity to a reading group, because the theory-vs-implementation gap is instructive.","headline":"Clear writing and correct algebra, but the actual update is underspecified, the key hyperparameters are missing, and the abstract mislabels a benchmark; the empirical results are interesting enough to take through review.","tokens_in":18000,"tokens_out":7511,"would_cite":false,"duration_ms":68482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A supervised cross-entropy loss recovers the policy gradient for RLVR","keywords":["reinforcement learning with verifiable rewards","implicit actor-critic","supervised learning reformulation","cross-entropy loss","log-probability ratio","RLOO advantage","LLM mathematical reasoning","policy gradient"],"falsifier":"Train PACS without the periodic reset of the reference policy, or monitor the sigmoid-transformed score against true reward on a held-out set: if gradients diverge or the score stops tracking actual correctness, the claimed actor-critic coupling is not the source of stability.","tokens_in":16834,"feed_emoji":"🧮","tokens_out":11070,"duration_ms":81983,"temperature":0.7,"pith_summary":"PACS recasts reinforcement learning with verifiable rewards (RLVR) as a supervised classification task: the outcome reward is treated as a label, and the policy itself is trained, with a cross-entropy loss, to predict whether each of its own sampled responses is correct. The paper's central claim is that this supervised objective is not an approximation but an exact reformulation of policy optimization: the gradient decomposes into a standard policy-gradient term plus a reward-prediction correction, so one network acts as both actor and critic in a single update. If the claim holds, RLVR post-training can avoid separate value models and high-variance Monte Carlo advantages, and can borrow the stability of supervised learning. On AIME-level math benchmarks with a 7B model, PACS reports pass@256 gains of roughly 11 to 15 points over the PPO and GRPO baselines while maintaining higher policy entropy during training.","feed_headline":"Supervised loss recovers policy gradients, beats PPO and GRPO on AIME","feed_subtitle":"If true, RLVR training gets supervised stability with policy-gradient credit assignment.","key_machinery":"The load-bearing object is the RLOO-style advantage score $\\psi$ defined in Eq. 7, with a policy log-probability-ratio proxy $\\hat r = \\beta\\log(\\pi_\\theta/\\pi_{\\rm ref})$ from Eq. 8. The load-bearing identity is the gradient decomposition of the cross-entropy loss into an actor term $l\\nabla_\\theta\\log\\pi_\\theta$ and a critic term $(R-\\sigma(\\psi))\\nabla_\\theta\\psi$. This identity is what turns a supervised classifier into an implicit actor-critic method. Stability relies on two supporting mechanisms: periodically hard-resetting $\\pi_{\\rm ref}$ to a recent snapshot of $\\pi_\\theta$ so the log-ratio proxy does not drift, and class-imbalance weights that rebalance correct versus incorrect samples.","core_discovery":"The discovery is that a binary cross-entropy loss, whose labels are the verifiable outcome rewards and whose logits come from a policy-parameterized advantage-like score, carries the same gradient as classic policy-gradient RLVR while adding an implicit critic term. With score function $\\psi(q,o_i;\\pi_\\theta) = \\hat r(q,o_i;\\pi_\\theta) - \\frac{1}{G-1}\\sum_{j\\ne i}\\hat r(q,o_j;\\pi_\\theta)$ and reward proxy $\\hat r = \\beta\\log(\\pi_\\theta/\\pi_{\\rm ref})$, the loss gradient is\n$$-\\mathbb E\\left[\\, l(q,o;\\pi_\\$\\theta$)\\nabla_\\$\\theta$\\log\\pi_\\$\\theta$(o|q) + \\big(R(q,o)-\\$\\sigma$(\\psi)\\big)\\nabla_\\$\\theta$\\psi(q,o;\\pi_\\$\\theta$)\\right],$$\nwhere $l$ is the per-sample cross-entropy. The first bracket is the actor: a policy-gradient update weighted by how well the current score predicts the reward. The second bracket is the critic: a prediction-error residual that adjusts the score toward the true reward. Because both use the same parameters, the paper argues the actor and critic are coupled implicitly, eliminating the temporal mismatch of separate value networks. The paper further reports that the resulting method consistently outperforms PPO and GRPO on MATH 500, AMC23, AIME 2024, and AIME 2025 with Qwen2.5-3B and Qwen2.5-7B.","pith_inferences":["If the supervised formulation is the right lens, the supervised-learning toolbox (class-imbalance weighting, curriculum ordering, label smoothing, and early stopping on validation accuracy) should transfer directly to RLVR without altering the policy-gradient structure.","The paper treats the reset schedule of $\\pi_{\\rm ref}$ as a practical choice; a testable extension is to trigger the reset adaptively from the drift of $\\log(\\pi_\\theta/\\pi_{\\rm ref})$, removing the schedule hyperparameter while preserving the claimed stability.","The ablation results suggest the advantage estimator matters most on hard tasks: on AIME, the RLOO variant beats Dr. GRPO by several points, while on MATH 500 they nearly tie; a sparser-reward domain such as program synthesis should show a larger RLOO-versus-Dr.-GRPO gap if that pattern is real."],"forward_implications":["RLVR training can be implemented as a supervised classification loop over the policy's own rollouts, with no separate value network, no generalized advantage estimation, and no clipping ratio.","The same gradient structure means PACS retains the credit-assignment behavior of a policy gradient while replacing Monte Carlo advantage noise with a prediction-error signal.","Training dynamics reported in the paper show higher sustained gradient norms and non-collapsing entropy, so the method keeps exploring longer than PPO or GRPO.","On AIME 2024 and AIME 2025, the reported pass@256 gains of roughly 11 to 15 points over PPO and GRPO on a 7B model would make PACS a stronger drop-in trainer for math reasoning at moderate scale."],"supporting_citations":[{"why":"Defines the PPO algorithm that serves as the main value-model-based RLVR baseline.","marker":"Schulman et al., 2017"},{"why":"Defines GRPO, the main value-model-free RLVR baseline, and the group-relative advantage style that PACS modifies.","marker":"Shao et al., 2024"},{"why":"Introduces the RLOO estimator used to compute the leave-one-out advantage score.","marker":"Kool et al., 2019"},{"why":"Shows REINFORCE-style leave-one-out optimization works for LLM feedback, the basis for PACS's advantage computation.","marker":"Ahmadian et al., 2024"},{"why":"Supplies the log-probability-ratio form of the reward proxy $\\hat r = \\beta\\log(\\pi_\\theta/\\pi_{\\rm ref})$.","marker":"Rafailov et al., 2023"},{"why":"Supplies the periodic hard-reset of the reference policy that keeps the reward proxy stable during training.","marker":"Liu et al., 2025a"},{"why":"Provides the rare-events class-imbalance weighting used to rebalance correct and incorrect samples.","marker":"King & Zeng, 2001"},{"why":"Provides the DeepScaleR dataset used as the training corpus in all experiments.","marker":"Luo et al., 2025"},{"why":"Provides the unbiased pass@k estimator used to report every benchmark number.","marker":"Chen et al., 2021"}],"fun_headline_variants":["Implicit actor-critic via supervised loss, beats PPO/GRPO on math","PACS: RLVR as a supervised learning problem with implicit critic","Cross-entropy loss hides a policy gradient, outperforms PPO and GRPO","No separate critic: cross-entropy loss yields policy gradients, beats GRPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's stability and performance rest on the assumption that the log-probability-ratio score keeps meaningfully measuring response quality throughout training, which the paper handles by periodically resetting the reference policy rather than proving it will hold.","fun_headline_variants_meta":{"raw":{"variants":["Implicit actor-critic via supervised loss, beats PPO/GRPO on math","PACS: RLVR as a supervised learning problem with implicit critic","Cross-entropy loss hides a policy gradient, outperforms PPO and GRPO","No separate critic: cross-entropy loss yields policy gradients, beats GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000905,"raw_usage":{"total_tokens":3955,"prompt_tokens":1072,"completion_tokens":2883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2799}},"tokens_in":688,"tokens_out":2883,"duration_ms":20609,"temperature":1.0,"reasoning_tokens":2799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:37:57.582219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PACS without the periodic reset of the reference policy, or monitor the sigmoid-transformed score against true reward on a held-out set: if gradients diverge or the score stops tracking actual correctness, the claimed actor-critic coupling is not the source of stability.","supporting_citations":[],"review_version":2}