{"id":"9eafb3fa-58b3-4f9f-9a91-92936e513acd","arxiv_id":"2508.18884","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"HAEPO weights each trajectory by its softmax-normalized cumulative log-likelihood, adds entropy and KL penalties, and matches or slightly surpasses PPO, GRPO, and DPO on small RL and summarization tasks.","lead":"HAEPO is a new training loss for reinforcement learning and LLM alignment that scores whole trajectories through a softmax over their log-probabilities, with entropy and KL regularization to stay stable. On small benchmarks it is at par with PPO/GRPO/DPO and slightly better in a 40-person human study, but it is 3x slower than PPO on CartPole.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Implemented gradient Eq. (6) omits the sampling-distribution term that the appendix (Eq. 36) proves necessary; HAEPO as implemented does not optimize E[-Σ R_k w_k].","rationale":"The reader's weakest assumption identifies exactly the most load-bearing defect. The central claim—'unbiased, low-variance' and 'aligns closely with true rewards'—depends on the gradient estimator being the gradient of E[-Σ R_k w_k]. Eq. 6 is not; the appendix's own Eq. 36 shows the additional sampling term. The paper's finite-difference sanity check only validates the analytic gradient of the sample loss, not the unbiasedness of the Monte Carlo estimator, so it does not resolve the discrepancy. This is not an external disagreement but an internal inconsistency between the derivation and the implementation. That said, the empirical results could still be valid for the heuristic update, and the empirical evidence (bandit, Random Walk, CartPole, TL;DR) is mixed but not fatal, so the appropriate verdict remains conditional pending the check; no change to the reader's CONDITIONAL verdict is needed.","tokens_in":16586,"tokens_out":7871,"duration_ms":81458,"concrete_test":"Implement both estimators on a small episodic MDP (e.g., the appendix's 5-step chain). Compute the Monte Carlo average of Eq. (6) over many batches and compare with a central finite-difference estimate of ∇_θ E_{τ~πθ}[-Σ R_k w_k] (or with the sample average of the corrected estimator in Eq. 36). If the difference exceeds ~10% of the true gradient norm, the omitted term is material. A second, complementary check: rerun the CartPole and bandit experiments with the corrected Eq. (36) estimator in place of Eq. (6); if learning curves or final returns change substantially, the published results are specific to the biased update rather than to the claimed HAEPO objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core theoretical claim is that the HAEPO update is unbiased. The main text defines L_orig(θ) = -Σ_k R_k w_k(θ) (Eq. 3) and derives ∇θ L_orig = -Σ_k R_k w_k [∇L_k - Σ_j w_j ∇L_j] (Eq. 6). But Eq. 6 is only the gradient of the sample loss, holding sampled trajectories fixed. The appendix correctly derives the gradient of the expected loss E_{τ~πθ}[-Σ R_k w_k]: it contains a second term, -(Σ R_k w_k)∇θ log p(τ;θ) (Eqs. 35-36). That second term is the standard REINFORCE/policy-gradient term and is not generally zero. Therefore the implemented gradient is a biased estimator of the gradient of the stated objective; the 'unbiased, low-variance' guarantee in §3.5 and the appendix does not apply to the algorithm actually run. Since every subsequent claim about 'aligns closely with true rewards' and 'provably lowers variance' rests on this unbiasedness, the theoretical foundation of HAEPO is unsupported unless the missing term is shown to be negligible. This is an internal inconsistency, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HAEPO, a trajectory-level policy optimization method. Each rollout is compressed to the sum of its action log-probabilities, a Plackett-Luce softmax over a batch converts these into normalized weights, and the loss is the negative of the return-weighted sum of these weights, augmented by an entropy bonus and a KL penalty to a frozen reference policy. The method is evaluated on multi-armed bandits, a 500-step random walk, CartPole, and TL;DR summarization with LLMs, comparing against PPO, DPO, and GRPO. The main text derives the gradient of the sample loss, claims unbiasedness and variance reduction, and presents empirical evidence of faster or comparable convergence and lower GPU memory use.","tokens_in":16883,"tokens_out":4583,"duration_ms":47127,"significance":"If the theoretical claims were correct, HAEPO would be a simple, low-memory trajectory-level alternative to GRPO/PPO, with a clear listwise interpretation. The paper includes several strengths: the method is simple and interpretable, the appendix contains an unusually explicit derivation and a finite-difference check, and the experiments span both small RL environments and an LLM alignment task with human evaluation. However, the central theoretical claim — that the implemented update is an unbiased, low-variance estimator of the expected return objective — is contradicted by the paper's own appendix. The implemented gradient (Eq. 6) omits the sampling-distribution correction that the unbiasedness proof (Eq. 36) shows is necessary. Consequently the contribution, as presented, is a heuristic trajectory-level loss on a fixed batch, not an unbiased policy-gradient method. The empirical results may still be useful, but the paper's stated theoretical foundation and 'provably lowers variance' claims need substantial revision or removal.","major_comments":[{"comment":"The gradient implemented in Eq. (6) is the gradient of the sample loss -Σ_k R_k w_k with respect to θ while holding the sampled trajectories fixed. The appendix correctly derives the gradient of the expectation E_{τ~p(τ;θ)}[-Σ_k R_k w_k] and obtains Eq. (35), whose Monte Carlo estimator in Eq. (36) includes the additional term - (Σ_k R_k w_k) ∇_θ log p(τ;θ). This term is generally nonzero. Therefore the update actually run in the experiments is not an unbiased estimator of the gradient of the objective defined in Eq. (34). The unbiasedness proof and the low-variance claims in §3.5 do not apply to the algorithm as implemented. Please either include the missing sampling-distribution term and rerun the experiments, or explicitly reframe HAEPO as a deterministic sample-loss minimization heuristic and provide a different justification for why it improves expected return.","section":"§3.4, Eq. (6) and Appendix, Eq. (35)–(36)"},{"comment":"The claim that z-score normalization 'provably lowers variance' is not supported by any proof in the main text or appendix. Moreover, the statement that the transformation 'leaves the expectation unchanged so the gradient remains unbiased' is incorrect for the full z-score: replacing R_k by (R_k - μ)/σ multiplies the entire gradient by 1/σ, which changes the scale of the expected gradient and is not equivalent to subtracting a constant baseline. Subtracting a constant b from all returns in a batch does leave the gradient unchanged because Σ_k w_k ∇_θ log w_k = 0, but dividing by the sample standard deviation does not. Please provide a precise statement with a proof, or remove the 'provably' and 'unbiased' claims for this normalization.","section":"§3.5, Z-Score Normalization"},{"comment":"The gradient decomposition in Eq. (10) writes the entropy contribution as β E_{k~w}[∇_θ log w_k]. Differentiating the entropy term β Σ_k w_k log w_k from Eq. (7), however, gives β Σ_k w_k (1 + log w_k) ∇_θ log w_k, which is the form correctly used in Eq. (30). Eq. (11)'s collapsed form -E[(R̃_k - β - λD_k)∇_θ log w_k] also omits the (1 + log w_k) factor. The main-text derivation and the appendix are therefore inconsistent. Please correct the main-text equations so that the reported gradient matches the defined loss.","section":"§3.8, Eqs. (10)–(11) vs. Appendix Eq. (30)"}],"minor_comments":[{"comment":"The setup says each method is run with three random seeds, but Table 1 and the text report results over five seeds. Please align these numbers. Also, 'a 50-50 split of 1,000 supervised examples and 1,000 RL updates' is ambiguous; clarify the dataset size and update count.","section":"§4.4, TL;DR setup and Table 1"},{"comment":"The KL penalty is applied to the Plackett-Luce weight distributions w and w_ref, not to the policy distributions. This is a batch-level, trajectory-weight trust region, not a policy trust region. Please state this explicitly to avoid overclaiming the connection to PPO-style KL penalties.","section":"§3.7, Eq. (8)–(9)"},{"comment":"For sum normalization, if the batch sum of returns is negative, the normalized returns ˜R_k can be negative and do not sum to +1 in the usual sense. The claim that the rescaled rewards satisfy Σ_k ˜R_k = 1 assumes a positive sum. Please clarify the handling of negative or zero-sum batches.","section":"§3.5, Sum-Normalization"},{"comment":"The finite-difference check verifies gradients of the loss with respect to the inputs newlp, oldlp, and returns, not with respect to policy parameters θ. This does not validate the unbiasedness of the estimator in Eq. (36). Please clarify what the check actually demonstrates.","section":"Appendix, Code Sanity Check"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the mismatch between the implemented gradient and the claimed unbiased estimator. This is not a matter of presentation; it affects the central theoretical contribution. The authors could fix it either by adding the missing term or by reframing the method as a heuristic loss and providing an alternative justification. Given the paper's empirical breadth and clear writing, a major revision seems appropriate rather than rejection. I would also suggest the editor ask for a proof or removal of the 'provably lowers variance' claim for z-score normalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core theoretical claim—that the HAEPO update is unbiased—does not hold for the algorithm actually run. The main text derives Eq. (6) as the gradient of the sample loss, holding trajectories fixed, but the appendix correctly derives the gradient of the expected loss and includes an extra term, −(Σ R_k w_k) ∇_θ log p(τ;θ). That term is the standard REINFORCE correction and is not generally zero. The implemented update drops it, so HAEPO is not optimizing E[−Σ R_k w_k], and the \"unbiased, low-variance\" guarantee does not apply. The paper never shows the omitted term is negligible, so this is a load-bearing flaw, not a cosmetic one.\n\nSecond, the paper does have real content. The loss—softmax over summed trajectory log-likelihoods, with entropy and KL penalties—is a legitimate combination that, to my knowledge, isn't verbatim in the literature. The derivation is clear, the finite-difference check of the analytic gradient is honest, and the limitations section is candid. The empirical story on CartPole is honest about the 3× slowdown. The TL;DR results are plausible but small.\n\nThe softer spots: the z-score \"provably lowers variance\" claim is unproven; the bandit experiments lack standard bandit baselines (UCB, Thompson, Exp3); and the human evaluation is 40 participants recruited via social media, with no significance tests. These are addressable.\n\nWho is this for? A reader interested in trajectory-level RLHF variants might find the formulation useful, but only after the bias is fixed. If the authors add the missing term or show it vanishes, the method could be a genuine low-memory alternative. As it stands, the theory and experiments describe different algorithms, so I'd be skeptical of the \"better or at par\" conclusion.\n\nI'd send it to peer review, but with a strong request to address the gradient issue and rerun experiments with the corrected update. It's a serious paper with a fixable flaw.","headline":"The implemented HAEPO update omits the sampling-distribution term the paper's own appendix proves necessary, so the unbiasedness claim doesn't hold for the algorithm actually run; the paper is still a clear, honest, and partially useful contribution with fixable flaws.","tokens_in":17396,"tokens_out":2305,"would_cite":false,"duration_ms":21899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A history-aggregated objective that collapses each rollout to a cumulative log-likelihood and normalizes it with a Plackett-Luce softmax is claimed to make policy optimization faster, more exploratory, and more memory-efficient than PPO, GR","keywords":["reinforcement learning","policy optimization","Plackett-Luce weighting","trajectory-level credit assignment","exploration","LLM alignment","entropy regularization","KL trust region"],"falsifier":"On the random-walk or CartPole setup, compare training with the implemented HAEPO gradient (Eq. 6/9) against training with the full unbiased estimator from the appendix (Eq. 36), which appends −(Σ_k R_k w_k)∇_θ log p(τ;θ) to the update. If including this term changes the learning curves materially—or if its gradient norm is not small relative to the implemented gradient—the claim that HAEPO is an unbiased, low-variance estimator of expected return is falsified.","tokens_in":16496,"feed_emoji":"🎯","tokens_out":10971,"duration_ms":93491,"temperature":0.7,"pith_summary":"HAEPO is a policy-gradient method that treats each full rollout as one ranked item: it sums the log-probabilities of all actions in an episode, runs a Plackett-Luce softmax over the batch to get a normalized weight per trajectory, and multiplies those weights by normalized returns to form the update. The paper argues that this trajectory-level listwise weighting preserves distinctions among sequences that per-token aggregation dilutes, and that adding an entropy bonus plus a soft KL penalty to a frozen reference policy keeps the update from collapsing into premature exploitation. Empirically, the authors report faster convergence and lower variance on bandit and random-walk tasks, competitive sample efficiency with smoother curves on CartPole, and higher human preference ratings than GRPO or DPO on TL;DR summarization while using 26.4 percent less peak GPU memory than GRPO. If these results hold, HAEPO offers a simpler, critic-free and memory-frugal alternative to PPO-style and group-relative methods for long-horizon RL and LLM alignment.","feed_headline":"Trajectory-level Plackett-Luce loss matches or beats PPO, GRPO, DPO","feed_subtitle":"HAEPO weights whole rollouts by cumulative log-likelihood and uses 26% less GPU memory than GRPO.","key_machinery":"The load-bearing object is the Plackett-Luce weight w_k = exp(L_k)/Σ_j exp(L_j), where L_k = Σ_t log π_θ(a_t|s_t) is the cumulative log-likelihood of rollout k. Because ∇_θ log w_k = ∇_θ L_k − Σ_j w_j ∇_θ L_j, the reward-weighted gradient automatically subtracts a batch-averaged score, giving a centered estimator without a learned baseline. The entropy term β Σ w log w and the KL penalty λ Σ w (log w − log w_ref) are applied to the weight distribution, not to individual actions, so they govern how the whole batch of trajectories spreads or shifts.","core_discovery":"The central claim is that an objective of the form L(θ) = −Σ_k w_k R̃_k + β Σ_k w_k log w_k + λ Σ_k w_k (log w_k − log w_k^ref), with w_k the Plackett-Luce weight over cumulative log-likelihoods, is a stable, low-variance estimator of expected return that yields better exploration than per-token or pairwise losses. The paper derives the gradient as an expectation over PL weights of a centered score function, and reports that on four benchmarks — stochastic bandits, a sparse random walk, CartPole, and TL;DR human-feedback fine-tuning — HAEPO converges quickly, explores broadly, aligns with human preference better or on par with PPO, GRPO, and DPO, and cuts GPU memory relative to GRPO.","pith_inferences":["Editorial inference: The implemented gradient (Eq. 6) drops the sampling-distribution term that the appendix's unbiased estimator (Eq. 36) includes; a head-to-head comparison of the two updates on a long-horizon task would reveal whether the omitted term is negligible or whether HAEPO is optimizing a different objective.","Editorial inference: HAEPO's return-weighted PL normalization is a natural fit for listwise preference datasets with graded or full ranking feedback; the paper only demonstrates it on scalar-return environments, not on explicit preference rankings.","Editorial inference: The reported VRAM reduction suggests HAEPO could enable longer contexts or larger batches on a fixed GPU than GRPO, but that extrapolates beyond the tested 1B/1.5B models and 1,000-update schedule.","Editorial inference: The paper's single-agent limitation points to an open question of whether PL weighting over joint trajectories can be stabilized in multi-agent settings, since the reference-policy KL would then need to account for opponents' changing policies."],"forward_implications":["HAEPO can be implemented as a simple loss on per-rollout log-probability sums, with no critic network and no per-token clipping; that is what keeps its per-update cost and memory footprint small.","On sparse long-horizon tasks such as the 500-step random walk, trajectory-level PL weighting should keep rare successful walks visible in the gradient, improving credit assignment relative to per-step policy gradients.","For LLM alignment, HAEPO's listwise reward weighting is a drop-in alternative to GRPO's group-relative baseline, with the reported 26.4% peak-VRAM savings on a 1.5B model.","The two regularization coefficients β_ent and β_kl control the exploration-stability trade-off; the paper notes these require tuning per task and reward scale.","Because HAEPO stores one log-probability sum per trajectory and runs a softmax over the batch, its overhead grows linearly with batch size M; the paper lists this as a computational limitation for very large batches."],"supporting_citations":[{"why":"Supplies the Plackett-Luce first-choice probability model that defines HAEPO's trajectory weights w_k.","marker":"[Plackett, 1975]"},{"why":"Supplies the score-function/log-derivative gradient that the HAEPO update is built on.","marker":"[Williams, 1992]"},{"why":"The PPO baseline and the trust-region/clipping idea that HAEPO replaces with a soft KL penalty.","marker":"[Schulman et al., 2017]"},{"why":"The GRPO baseline; HAEPO is compared against its token-level group-relative update and reports lower peak VRAM.","marker":"[Shao et al., 2024]"},{"why":"The DPO baseline; HAEPO is positioned as a trajectory-level alternative to DPO's pairwise preference loss.","marker":"[Rafailov et al., 2023a]"},{"why":"Provides the likelihood-ratio identity the appendix uses to prove unbiasedness of the gradient estimator.","marker":"[Glynn, 1990]"},{"why":"Supplies the RL framework, the random-walk example, and the policy-gradient theorem cited for the estimator.","marker":"[Sutton and Barto, 2018]"}],"fun_headline_variants":["HAEPO: History-weighted loss matches or beats PPO, GRPO, DPO","Aggregate trajectory history: match PPO, GRPO, DPO, use less GPU","Plackett-Luce on log-likelihoods: match or beat PPO, GRPO, DPO","Cumulative log-likelihood weighting: explore better, use less GPU","HAEPO: matches or beats PPO, GRPO, DPO with 26% less GPU"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that ignoring the way the policy's own sampling distribution changes during an update does not bias learning; if that ignored effect matters, HAEPO is not optimizing the objective it claims to optimize.","fun_headline_variants_meta":{"raw":{"variants":["HAEPO: History-weighted loss matches or beats PPO, GRPO, DPO","Aggregate trajectory history: match PPO, GRPO, DPO, use less GPU","Plackett-Luce on log-likelihoods: match or beat PPO, GRPO, DPO","Cumulative log-likelihood weighting: explore better, use less GPU","HAEPO: matches or beats PPO, GRPO, DPO with 26% less GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002722,"raw_usage":{"total_tokens":10243,"prompt_tokens":793,"completion_tokens":9450,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":9329}},"tokens_in":537,"tokens_out":9450,"duration_ms":65118,"temperature":1.0,"reasoning_tokens":9329,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:07:19.089858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the random-walk or CartPole setup, compare training with the implemented HAEPO gradient (Eq. 6/9) against training with the full unbiased estimator from the appendix (Eq. 36), which appends −(Σ_k R_k w_k)∇_θ log p(τ;θ) to the update. If including this term changes the learning curves materially—or if its gradient norm is not small relative to the implemented gradient—the claim that HAEPO is an unbiased, low-variance estimator of expected return is falsified.","supporting_citations":[],"review_version":1}