{"id":"60554564-bdc4-412d-8631-a054d5a70748","arxiv_id":"2507.01915","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"GAPO combines multiple-gradient descent with gradient rescaling to balance helpfulness and harmlessness in RLHF, and P-GAPO adds user preference weights.","lead":"The paper presents GAPO, a method for fine-tuning large language models on several conflicting goals at once, such as being helpful and harmless. It rescales the gradient of each objective before combining them, and claims a better balance than simply weighting the rewards.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'converges to Pareto optimal' claim is unproven: Theorems 3.2-3.3 are only local, Lemma A.1's Taylor/Gram-Schmidt step is algebraically wrong, and Section 3.3's last-layer-gradient MGDA falls outside the theorem's full-gradient assumptions.","rationale":"The reader's REJECT verdict is appropriate. I partially agree with the reader's weakest assumption: the last-layer-gradient approximation is a real gap between theory and implementation, but the more fundamental problem is that even the idealized full-gradient algorithm has no convergence theorem. Theorems 3.2 and 3.3 are local statements, and the proof of Lemma A.1 is algebraically incorrect, so the abstract's claim that 'GAPO converges towards a Pareto optimal solution' is not established by the paper's mathematics. The empirical evaluation, which trains and tests on the same reward and cost models and uses a small GPT-4o prompt set without significance testing, further weakens the empirical contribution, but the theoretical gap alone is sufficient to reject the central claim as stated. No new concern changes the verdict; the rejection stands.","tokens_in":22965,"tokens_out":7391,"duration_ms":87990,"concrete_test":"Re-derive Lemma A.1 independently: write the first-order Taylor expansion of J_i(theta + eta*Delta) and verify that the eta term is eta * <Delta, grad J_i(theta)>, not eta * sum_k <Delta, grad J_k(theta)>. Then re-run the proof of Theorem 3.3 with the corrected lemma to see whether the ratio ||grad J_i||^p / ||grad J_j||^p still follows. To test the implementation gap, on a small two-objective LM or quadratic task, compare alpha weights computed from full gradients versus last-layer-only gradients and check whether the resulting update is a common ascent direction for both objectives; if any objective decreases while the other increases, the last-layer version does not inherit GAPO's stated guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises convergence to a Pareto-optimal solution, but no theorem in the paper establishes convergence. Theorem 3.2 shows only that, at a fixed non-Pareto-stationary parameter, the normalized MGDA direction is a common descent direction; because the objective gradients move with theta and no step-size schedule, compactness, or Lyapunov argument is supplied, repeated local ascent does not imply convergence to a Pareto stationary point, let alone Pareto optimality. Theorem 3.3 is a ratio statement, not a convergence statement. The given proofs are also unsound: Lemma A.1 (Appendix A) claims the first-order change in J_i is eta * sum_{k=1}^m <Delta(theta), grad J_k(theta)> after a Gram-Schmidt expansion, but Taylor expansion gives eta * <Delta(theta), grad J_i(theta)>; the sum over k appears by dropping the dot product with grad J_i. Theorems 3.1 and 3.3 rely on this lemma, so their conclusions are not established. Separately, Section 3.3 ('Practical Implementation') computes MGDA weights from last-layer gradients only, whereas Theorems 3.2 and 3.3 assume full-parameter gradients; no transfer argument shows that the last-layer approximation preserves common descent or balance. Thus the paper's central theoretical claim is unsupported both for the idealized method and for the method actually run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GAPO, a multi-objective RLHF method that replaces linear scalarization of rewards with a multiple-gradient-descent (MGDA) update, and P-GAPO, a preference-weighted variant. The authors claim that their theoretical analysis shows convergence to a Pareto-optimal solution, and they report experiments on Mistral-7B in which GAPO improves helpfulness and harmlessness trade-offs relative to several baselines. The paper is positioned as a gradient-based alternative to MORLHF and Safe RLHF. The central theoretical claims, however, are not established: the proof of Lemma A.1 is algebraically incorrect, no theorem actually proves convergence, and the implemented algorithm uses last-layer gradients that fall outside the scope of the theorems.","tokens_in":23230,"tokens_out":6449,"duration_ms":78088,"significance":"If the convergence and Pareto-optimality claims were rigorously established, GAPO would provide a principled replacement for linear scalarization in multi-objective RLHF, with P-GAPO offering a tractable way to trace a Pareto front. The experimental setup is well described, including reward models, datasets, hyperparameters, and evaluation prompts, which is a strength for reproducibility. The empirical comparisons against safe RLHF, fast RL, MGDA, MORLHF, and rewarded soups are informative. However, the paper's advertised contribution is the theoretical guarantee of convergence to Pareto optimality, and that guarantee is not delivered. The proof error and the gap between the theory and the executed algorithm are load-bearing, so the empirical results, while suggestive, cannot compensate for the unsupported central claim.","major_comments":[{"comment":"The proof of Lemma A.1 is mathematically incorrect. A first-order Taylor expansion gives J_i(θ') - J_i(θ) = η⟨Δ(θ), ∇J_i(θ)⟩ + o(η). The proof instead introduces a Gram-Schmidt expansion of Δ(θ) as a sum over all i of ⟨Δ,∇L_i⟩/||∇L_i||²∇L_i plus residual terms, and then substitutes this into the Taylor term. This expansion is not a valid orthogonal decomposition, and even if it were, substituting it would not produce the displayed identity ηΣ_{k=1}^m ⟨Δ(θ),∇L_k(θ)⟩; the factor ∇L_i is not applied to the sum as written. The claimed identity is false in general, for example with two non-orthogonal gradient vectors. Because Theorems 3.1 and 3.3 both invoke Lemma A.1, their proofs do not establish the stated results.","section":"Appendix A.1, Lemma A.1"},{"comment":"The abstract states that GAPO 'converges towards a Pareto optimal solution', but no theorem in the paper establishes convergence. Theorem 3.2 shows only that, at a fixed non-Pareto-stationary parameter θ, the normalized MGDA direction is a common ascent direction. Theorem 3.3 is a one-step ratio statement about the relative improvement of two objectives when the learning rate tends to zero. There is no step-size schedule, no boundedness or compactness assumption on the parameter iterates, no treatment of stochastic gradient noise, and no Lyapunov or accumulation-point argument. Repeated application of a local improvement direction need not converge to a Pareto-stationary point, let alone to a Pareto-optimal point. Thus the headline convergence claim is unsupported by the theoretical analysis.","section":"Abstract and Section 3.2"},{"comment":"The MGDA weights in the implemented algorithm are computed using only the gradients of the last layer of the policy network ('we opt to use the gradients of the parameters in the last layer of the LM policies as a substitute'). Theorems 3.2 and 3.3, however, assume full-parameter gradients ∇θJ_i(θ) in the objective functions and in the MGDA minimization problem. The paper provides no argument that the last-layer approximation preserves the common-descent property ⟨Δ_N(θ), ∇N J_i(θ)⟩ > 0 or the balance property of Theorem 3.3. Consequently, the theoretical results do not apply to the algorithm actually run in the experiments, and the empirical results cannot be interpreted through the provided theorems.","section":"Section 3.3, Practical Implementation"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'when they are conflict' in the abstract, 'harmelss' in Figure 1, 'evalated' in the Figure 3 caption, and inconsistent use of L_i versus J_i in the proofs of Appendix A. The notation should be made consistent throughout.","section":"General"},{"comment":"Equation (14) contains an apparent typo: the left-hand side is written as \\widehat r^h_s but the right-hand side uses r^s_t; the superscript should likely be s rather than h.","section":"Appendix B.2"},{"comment":"The empirical comparisons do not report variance or multiple independent seeds. Given that the headline experimental claim is a Pareto-front improvement, reporting standard deviations or at least a small number of seeds would strengthen the conclusions.","section":"Section 4.2"},{"comment":"Theorem 3.1 is stated as a convergence-free property of a single MGDA step, but the proof in Appendix A.1 uses the incorrect Lemma A.1. The theorem may be salvageable through the KKT conditions of Problem (5), but the current proof is not valid.","section":"Section 3.1"}],"recommendation":"reject","confidential_remarks":"The central contribution of the paper is the claimed convergence of GAPO to a Pareto-optimal solution. That claim is not supported: Lemma A.1's proof is algebraically incorrect, no convergence theorem appears anywhere in the paper, and the implemented last-layer-gradient variant is not covered by the stated theorems. These are not merely presentation issues; they undermine the main theoretical selling point. The empirical results are suggestive, but they do not repair the theory. I recommend rejection. A resubmission that supplies a rigorous convergence analysis (or honestly reframes the contribution as a local/empirical method) and that either justifies the last-layer approximation or matches the implementation to the theory could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies gradient normalization plus MGDA to multi-objective RLHF on Mistral-7B. The empirical part is honest and the P-GAPO preference front is a genuinely useful addition. But the theory that anchors the abstract is not just incomplete; Lemma A.1's proof is algebraically wrong, and the last-layer-gradient implementation sits outside the theorems.\n\nWhat is new: they run this on helpfulness/harmlessness with two datasets and GPT-4o eval, and show GAPO p=1 beats Safe RLHF, MGDA, and Fast RL on average. The P-GAPO front dominates linear scalarization and model merging on their points. That is a legitimate engineering result, and the paper is transparent about practical details.\n\nThe theory is the soft spot. The abstract says GAPO converges to a Pareto-optimal solution, but no convergence theorem exists. Theorem 3.2 only shows a local common-descent direction; Theorem 3.3 is a ratio statement. Lemma A.1's Taylor expansion includes a Gram-Schmidt expansion of Delta(theta) and then drops the dot product with grad L_i, producing a spurious sum over k. That is simply incorrect, and Theorems 3.1 and 3.3 depend on it. Separately, Section 3.3 says they use last-layer gradients for the MGDA weights, while Theorems 3.2 and 3.3 assume full-parameter gradients; no argument bridges that gap. So the central claim is unsupported both for the idealized method and for the method actually run.\n\nEmpirically, the reward and cost models used for training are the same ones used for evaluation, which inflates the numbers. The GPT-4o eval uses 129 helpfulness and 83 harmlessness prompts, no significance testing, and the choice of p=1 over p=2 looks post hoc. No code or data are released.\n\nWhat holds up: the empirical trend is plausible and consistent across two test sets, the baselines are not misreported, and the limitations section acknowledges the narrow model and evaluation gap. This is not a dishonest paper; it is an overclaimed one.\n\nThis paper is for people working on multi-objective alignment who want to see whether gradient-based balancing can beat scalarization. They will get value from the experiments, not the proofs. It deserves referee time because the question is important and the experiments are reproducible in principle, but the theory needs a full rewrite or removal.\n\nRecommendation: send it to review, but expect the reviewers to require fixing the theory or dropping the convergence claims, and to ask for code.","headline":"Useful empirical comparison drowned by an unsupported convergence claim and a provably wrong lemma.","tokens_in":23808,"tokens_out":2135,"would_cite":false,"duration_ms":23326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GAPO rescales and reweights each objective's gradient so an LLM improves conflicting alignment goals at once, provably reaching Pareto-stationary solutions and lifting helpfulness and harmlessness together on Mistral-7B.","keywords":["multi-objective RLHF","gradient rescaling","multiple-gradient descent","Pareto optimality","preference-based alignment","helpfulness","harmlessness","LLM alignment"],"falsifier":"At a checkpoint from a GAPO run, compute the MGDA weights twice—once from the last-layer gradients used in the paper and once from gradients over all trainable parameters—and apply each update to a fixed prompt batch. If the last-layer direction violates $\\langle \\Delta_N,\\nabla^\\mathrm{N}_\\theta J_i\\rangle>0$ for either objective while the full-gradient direction satisfies it, or if a full-gradient run reaches points that dominate the reported GAPO frontier, then the implemented algorithm is not the one the theorems protect.","tokens_in":22719,"feed_emoji":"⚖️","tokens_out":15690,"duration_ms":173083,"temperature":0.7,"pith_summary":"The paper argues that conflicting alignment objectives for large language models, such as helpfulness and harmlessness, can be balanced by choosing an update direction rather than by mixing rewards. Its method, GAPO, normalizes each objective's gradient and then solves a small weighting problem so that the weighted sum of gradients is as short as possible, which either certifies a Pareto-stationary point or gives a direction that improves every objective. A preference-aware variant, P-GAPO, replaces the weighting problem with a user-supplied preference vector over normalized gradients, letting one training setup sweep a Pareto front. The paper reports that on a 7-billion-parameter policy GAPO raises both helpfulness and harmlessness relative to scalarization and constrained-RL baselines, with P-GAPO's front dominating the alternatives in the balanced region. If correct, this gives alignment practitioners a direct, controllable gradient-level trade-off mechanism instead of hand-tuned reward weights.","feed_headline":"Gradient rescaling beats reward mixing for conflicting LLM goals","feed_subtitle":"GAPO balances helpfulness and harmlessness in one update rule and adds a user-preference dial for the trade-off.","key_machinery":"The mechanism that carries the argument is multiple-gradient descent (MGDA) with normalized gradients. MGDA solves for coefficients $\\alpha_i$ that minimize the norm of $\\sum_i \\alpha_i\\nabla_\\theta J_i(\\theta)$; a zero minimum marks a Pareto-stationary point, and a nonzero minimum is a direction that improves every objective. GAPO's modification is to run this weighting problem on the rescaled gradients $\\nabla_\\theta J_i(\\theta)/\\|\\nabla_\\theta J_i(\\theta)\\|_2^p$ and then update with $\\sum_i\\alpha^\\mathrm{N}_i\\nabla^\\mathrm{N}_\\theta J_i(\\theta)$. The parameter $p$ is the dial: $p=1$ makes gradients unit-length before weighting, while $p=2$ makes the rescaled gradient inversely proportional to its original length, and Theorem 3.3 pins down how the per-objective progress ratio depends on this choice. P-GAPO bypasses the weighting problem entirely and uses a user-supplied preference vector $\\lambda$ to combine the unit-normalized gradients, which is what lets one training setup sweep out a Pareto front.","core_discovery":"At its core, the paper claims that multi-objective alignment can be solved by gradient-adaptive descent rather than by reward scalarization. Given $m$ objectives $J_i(\\theta)$, GAPO first computes normalized gradients $\\nabla^\\mathrm{N}_\\theta J_i(\\theta)=\\nabla_\\theta J_i(\\theta)/\\|\\nabla_\\theta J_i(\\theta)\\|_2^p$, then solves the multiple-gradient descent problem $\\min_{\\alpha} \\|\\sum_i \\alpha_i \\nabla^\\mathrm{N}_\\theta J_i(\\theta)\\|_2^2$ subject to $\\sum_i\\alpha_i=1$, $\\alpha_i\\ge 0$. The paper proves (Theorem 3.2) that if $\\theta$ is not Pareto-stationary the resulting update direction is nonzero and has positive inner product with every normalized gradient, so all objectives improve; and (Theorem 3.3) that in the infinitesimal-step limit each objective's gain is proportional to $\\|\\nabla_\\theta J_i(\\theta)\\|_2^p$, which shifts effort toward objectives that still have large gradients. P-GAPO replaces the solving step with a user preference vector $\\lambda$, using $\\sum_i \\lambda_i \\nabla^\\mathrm{N}_\\theta J_i(\\theta)$ as the update and so tracing a front over preferences. Empirically, on a 7-billion-parameter policy with helpfulness and harmlessness rewards, GAPO with $p=1$ reports the highest average of the two objective scores among the compared methods, and the P-GAPO front improves on the linear-scalarization front in the balanced-preference region and completely dominates the weight-interpolation baseline. The paper describes the theoretical target as convergence toward a Pareto optimal solution, with Pareto stationarity as the formal necessary condition that the algorithm actually reaches.","pith_inferences":["The paper computes MGDA weights from last-layer gradients only but states its theorems for full gradients; testing whether the two weight vectors agree across training would show whether the executed algorithm inherits the proven Pareto and balance guarantees.","Because P-GAPO applies the user vector after unit normalization, the resulting front's spacing reflects gradient geometry rather than reward scales; checking whether users' stated $\\lambda$ values match the achieved helpfulness/harmlessness ratios would test the preference-controllability claim.","The balance theorem is stated for arbitrary $m$ objectives, yet the experiments cover only two; a three-objective run would reveal whether gradient rescaling stays stable as the weighting problem grows."],"forward_implications":["Multi-objective RLHF can be run without hand-fixed reward scalarization: the update direction itself settles the trade-off, and the paper states the procedure can wrap any policy-gradient RLHF or DPO-based aligner.","Since per-objective progress is proportional to the $p$-th power of the gradient norm, the algorithm automatically channels more progress into the objective that is still far from converged, which should counter the overcautious-refusal failure mode of constrained-RL safety tuning.","P-GAPO lets one training setup produce many Pareto points by varying the user preference vector, instead of retraining a model for each reward weight.","On the two datasets tested, the reported frontier places GAPO above the scalarization and weight-interpolation baselines in both helpfulness and harmlessness, so a stable result would shift the practical safety-utility trade-off outward."],"supporting_citations":[{"why":"It supplies the MGDA framework: the minimal-norm weighted gradient sum gives a common descent direction or certifies Pareto stationarity.","marker":"Désidéri (2012)"},{"why":"It brings MGDA into deep multi-task learning and is the template for applying multiple-gradient descent to neural network objectives.","marker":"Sener and Koltun (2018)"},{"why":"It motivates the normalized-gradient form that GAPO uses to rescale each objective before weighting.","marker":"Zhou et al. (2023)"},{"why":"It provides the preference-based MGDA idea that P-GAPO adapts to user preference vectors.","marker":"Zhang et al. (2024b)"},{"why":"It supplies the constrained-RL baseline, the reward and cost models, and the dataset split used in the experiments.","marker":"Dai et al. (2023)"},{"why":"It provides the PPO base algorithm whose surrogate losses yield the per-objective gradients that GAPO reweights.","marker":"Schulman et al. (2017)"},{"why":"It gives the theoretical argument that linear scalarization cannot reach the full Pareto front, motivating the gradient-based alternative.","marker":"Hu et al. (2023)"},{"why":"It provides the weight-interpolation baseline whose Pareto front is compared against P-GAPO.","marker":"Rame et al. (2023)"},{"why":"It provides the dynamic-reward-composition baseline, Fast RL, that GAPO is compared against.","marker":"Li et al. (2024a)"}],"fun_headline_variants":["GAPO rescales gradients per objective, not mixed rewards","Pareto-optimal LLM alignment via adaptive gradient descent","Multi-objective RLHF: balance trade-offs with gradient scaling","User-preference dial sets Pareto front for LLM alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gradients used in GAPO's weighting step are the true gradients of the objectives over all model parameters; the implementation instead uses only the final layer's gradients, and no proof connects that shortcut to the theorem.","fun_headline_variants_meta":{"raw":{"variants":["GAPO rescales gradients per objective, not mixed rewards","Pareto-optimal LLM alignment via adaptive gradient descent","Multi-objective RLHF: balance trade-offs with gradient scaling","User-preference dial sets Pareto front for LLM alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000846,"raw_usage":{"total_tokens":3768,"prompt_tokens":1116,"completion_tokens":2652,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":2582}},"tokens_in":732,"tokens_out":2652,"duration_ms":22687,"temperature":1.0,"reasoning_tokens":2582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:41:21.254007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At a checkpoint from a GAPO run, compute the MGDA weights twice—once from the last-layer gradients used in the paper and once from gradients over all trainable parameters—and apply each update to a fixed prompt batch. If the last-layer direction violates $\\langle \\Delta_N,\\nabla^\\mathrm{N}_\\theta J_i\\rangle>0$ for either objective while the full-gradient direction satisfies it, or if a full-gradient run reaches points that dominate the reported GAPO frontier, then the implemented algorithm is not the one the theorems protect.","supporting_citations":[],"review_version":1}