{"id":"761e3013-b4d6-4c3b-9eed-e8431369342f","arxiv_id":"2412.05469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HaM jointly trains multiple LLM policies by maximizing the hypervolume of their objective vectors, yielding a diverse Pareto-covering set of behaviors without pre-specified human preferences.","lead":"This paper trains several language model policies at the same time, each balancing competing goals such as helpfulness and harmlessness differently. The method, called HaM, maximizes the hypervolume of the policies' outcome vectors and is claimed to beat single-preference baselines on several benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The training objective's reward normalization and its link to evaluation-time rewards are underspecified; until shown otherwise, the reported Pareto-front superiority may be an artifact of Eq. (4) clipping and reward scaling.","rationale":"Proposition 1 and Theorem 1 are mathematically correct for any point coordinates, but the coordinates are defined by Eq. (3)-(4). The hypervolume maximization is only as meaningful as the mapping from θ to evaluation-space rewards. The reader's weakest assumption identifies exactly this point. My stress-test sharpens it: the [0,1] normalization is not just a convention; because Eq. (4) contains a tunable z and a clip at zero, different but equally unreported normalizations of the same reward models produce different HaM objective landscapes. The paper's empirical comparisons use one unstated choice, so the headline 'empirically superior across objectives' is not reproducible from the text. I do not see an internal inconsistency in the algorithm itself; the idea is coherent and the approximations are analyzed. The appropriate remedy is a normalization ablation plus a check that training-time hypervolume tracks evaluation-time hypervolume. That is a conditional-accept situation, not a rejection.","tokens_in":17041,"tokens_out":4994,"duration_ms":55978,"concrete_test":"Re-run the Section 4.1 harmless-helpful experiment with the same raw reward-model scores used for evaluation, under two contrasting normalizations for training: (A) min-max normalize r_j(x,y) across D to [0,1], and (B) use raw scores directly (or a constant shift such that negative scores are allowed). Record the HaM training objective value and the evaluation hypervolume/domination relation. If the Pareto front versus RiC/SCA changes or the training objective saturates/clips under (B), the reported claim depends on an unreported normalization and Eq. (3)-(4) are under-specified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"HaM's central claim is that maximizing hypervolume of the vectors v_k=(¯L_1(θ_k),...,¯L_J(θ_k)) produces diverse policies whose evaluation-time Pareto front is superior. This requires two conditions that the paper never establishes. First, Eq. (3) assumes r_j(x,y)∈[0,1], but the HuggingFace reward models used in Section 4 output unbounded scores; the paper does not state how they are normalized, and Eq. (4)'s max((L_j+z)/z,0) can clip all coordinates to a constant or zero if z is wrong, killing the hypervolume signal. Second, the training-space coordinates are reward-weighted log-likelihoods evaluated on the fixed dataset D, not expected reward-model scores of generated responses under θ_k. Hypervolume in this proxy space need not correlate with hypervolume in the evaluation space; no experiment or proposition connects them. The Section 4 domination claims are therefore only as strong as this unvalidated proxy plus an unreported reward preprocessing choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HaM, an a-posteriori multi-objective alignment method for LLMs. It defines J per-objective reward-weighted log-likelihood objectives, normalizes and clips them, and jointly optimizes K policy heads by maximizing the hypervolume of the resulting K reward vectors. The authors prove that this objective equals the hypervolume indicator (Proposition 1) and provide a high-probability bound for the mini-batch approximation (Theorem 1). To make the method practical, they share a transformer backbone among policies and use separate policy heads, reducing space complexity. Experiments on Opt-350m and Opt-6.7B compare HaM with RiC and scalarization (SCA) on tasks involving harmlessness, helpfulness, humor, faithfulness, and hallucination, reporting Pareto-front plots.","tokens_in":17182,"tokens_out":5530,"duration_ms":56725,"significance":"If the empirical claims hold, HaM is a valuable contribution: it introduces a-posteriori multi-objective optimization to MOAHF, gives a clean hypervolume objective with an inclusion-exclusion form, and provides an elegant shared-backbone parameterization. The proofs in Appendix A appear correct, and the mini-batch concentration bound is a useful formal addition. However, the paper's central empirical claim of superiority over RiC and SCA is currently supported only by single-run Pareto plots, and the relationship between the training-time objective and the evaluation-time reward models is not established. These are fixable but load-bearing gaps.","major_comments":[{"comment":"Equation (3) assumes r_j(x,y) is in [0,1], but the HuggingFace reward models used in Sections 4.1-4.4 return unbounded scores. The paper does not state how these scores are normalized, nor how the clipping parameter z in Eq. (4) is chosen. Depending on z, the max{·,0} clipping can either collapse all coordinates to zero or leave them effectively unclipped, so the hypervolume signal in Eq. (6) depends on an unreported preprocessing choice. Because the evaluation axes in Figures 3-8 plot raw reward-model scores, the reader cannot determine whether the reported Pareto-front superiority is robust to this choice. Please report the normalization used, the value of z for each task, and a sensitivity analysis.","section":"Section 3.1, Eq. (3)-(4); Section 4 training protocol"},{"comment":"The HaM objective in Eq. (6) is the hypervolume of vectors v_k = (\\bar L_j(\\theta_k))_j, where \\bar L_j is a reward-weighted log-likelihood computed on the fixed dataset D. In contrast, the Pareto fronts in Section 4 are computed from reward-model scores of generated responses. No theoretical statement or experiment connects these two spaces. It is therefore possible that a large hypervolume in the training space does not translate into a good Pareto front in the evaluation space, which would undermine the empirical comparison with RiC and SCA. Please add an analysis or experiment demonstrating that policies selected by Eq. (6) also dominate in the evaluation reward space, for example by reporting the correlation between training-space coordinates and evaluation rewards, or by conducting an oracle-selection study.","section":"Section 3.3 vs. Section 4 evaluation protocol"},{"comment":"All Pareto-front comparisons are based on single runs. There are no error bars, multiple seeds, or statistical tests, and the fronts consist of only K=5 points (or 10 scalarization vectors for SCA/RiC). Statements such as 'HaM dominates RiC' or 'by the number of domination wins, HaM is the best method' are visual judgments subject to optimization noise and evaluation randomness. Please report hypervolume indicator values with confidence intervals over several seeds, or otherwise quantify the variability of the fronts and the significance of the domination claims.","section":"Section 4; Figures 3-8"}],"minor_comments":[{"comment":"The complexity reduction is stated as O(J^2 J K) in the text, which appears to be a typo; it should be O(J 2^J K) or should match the random-scalarization analysis. Also, s_w(\\theta) divides by w_j, so the text should clarify how zero entries of w on the simplex boundary are handled.","section":"Section 3.6, Eq. (10)"},{"comment":"The matrix \\theta_k is described as being in R^{L\\times d} with L 'the number of tokens'; in a language-model head the relevant dimension is the vocabulary size. Please use a standard notation such as V for the vocabulary to avoid confusion.","section":"Section 3.5"},{"comment":"The theorem assumes that B prompt-response pairs are drawn uniformly at random from D, but does not state with or without replacement. The concentration argument relies on independence; if the implementation samples without replacement, the independence assumption should be justified or sampling with replacement should be stated explicitly.","section":"Section 3.4, Theorem 1"},{"comment":"The entry 'Threshold for MORS 0.7-quantile for each reward dimension' uses an undefined acronym MORS; please define it in the text or remove it.","section":"Table 5"},{"comment":"The vertical axes in the three-dimensional plots are labeled 'Z value' rather than an objective name; this makes the plots hard to read. Please label axes consistently with R1, R2, R3 as in the earlier figures.","section":"Section 4.4, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The theoretical part (Proposition 1 and Theorem 1) appears sound and the proposed parameterization is elegant. The main risk is that the empirical claims are not yet supported: reward normalization is unspecified, the training-to-evaluation proxy is unvalidated, and the Pareto-front comparisons lack uncertainty quantification. These are fixable with additional experiments and details, so major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper does something genuinely new: it applies a-posteriori multi-objective optimization to LLM alignment, and does it with a shared transformer backbone plus per-policy heads, so K policies cost roughly one policy in memory. That is a real practical idea. The theory is standard but correct — Proposition 1 is just inclusion–exclusion for hypervolume, and Theorem 1 is a clean concentration bound. The writing is clear and the positioning against RiC and SCA is honest. I believe the authors believe the method works, and the qualitative examples in Appendix B suggest it does produce visibly different behaviors across heads.\n\nThe soft spots are real, though. The training objective in Eq. (3) assumes r_j(x,y) ∈ [0,1], but the HuggingFace reward models used in evaluation output unbounded scores. The paper never says how rewards are normalized, and the normalization constant z in Eq. (4) is left unreported. If z is wrong, the clipping in Eq. (4) can kill the hypervolume signal entirely. That is a simple thing to fix, but it makes the experiments hard to trust as reported. More substantially, the training coordinates are reward-weighted log-likelihoods on a fixed dataset, while the evaluation coordinates are reward-model scores on generated responses. The paper never shows that maximizing hypervolume in the proxy space transfers to the evaluation space. A single ablation connecting L_j to actual reward scores would have settled this; its absence is the biggest gap.\n\nAlso, the Pareto-front plots are single runs with no error bars. Some of them show clear domination, but others show intersecting fronts, and the paper falls back on counting domination wins. That is not a rigorous claim of superiority. SCA with only five scalarizations is a weak baseline, though RiC is a reasonable one. These are fixable problems, but as it stands the headline claim is not yet supported.\n\nThe paper deserves peer review — it addresses an important problem, the method is coherent, and the theoretical parts check out. A serious referee should ask for multiple seeds, released code, exact reward normalization and z values, and a proxy-to-evaluation validation. If those come in, this could be a useful contribution. For now, I would not cite it as evidence of superior Pareto fronts, but I might mention it as related work if I were writing on multi-objective alignment.\n\nMy recommendation: send to review, but the authors should be told that the empirical claims need major revision before acceptance.","headline":"HaM is a sensible and clearly presented a-posteriori MOO method for LLM alignment with a real practical contribution in the shared-backbone multi-head design, but the empirical superiority claims ride on single runs and an unvalidated training proxy, so the paper needs major strengthening before those claims land.","tokens_in":17784,"tokens_out":1810,"would_cite":false,"duration_ms":21472,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes HaM, which trains multiple LLM policies jointly by maximizing hypervolume to cover the Pareto front without preset human preferences.","keywords":["multi-objective alignment","hypervolume maximization","Pareto front","large language models","a-posteriori multi-objective optimization","reward-weighted log-likelihood","human feedback"],"falsifier":"Train HaM with one objective's training rewards multiplied by a large constant while keeping the evaluation reward models unchanged: if the hypervolume is merely rescaled and the Pareto front in evaluation space stays the same, the proxy is robust; if the front shifts or collapses, the claim that HaM covers the true Pareto front fails. A second decisive check is to compare HaM trained on the paper's proxy rewards against HaM trained directly on the evaluation reward models, and look for a large drop in evaluated Pareto-front quality under the proxy.","tokens_in":16808,"feed_emoji":"⚖️","tokens_out":11438,"duration_ms":103600,"temperature":0.7,"pith_summary":"This paper tries to establish that multi-objective alignment of large language models can be done a-posteriori: rather than committing to human preference weights before training, one can train several diverse policies at once and present the whole trade-off frontier afterward. Its proposed algorithm, HaM, maximizes the hypervolume of the union of rectangles formed by normalized reward-weighted log-likelihoods of the policies, which is shown to be equivalent to the standard hypervolume indicator. If the claim holds, a system designer would get a set of policies covering objectives such as harmlessness, helpfulness, humor, faithfulness, and hallucination from a single training run, with memory cost close to that of one model because policies share a transformer backbone. The paper evaluates HaM's Pareto fronts against the rewards-in-context and scalarization baselines on two and three objectives, reporting that HaM wins most domination comparisons and that no baseline dominates HaM in the reported plots. It also contributes a concentration bound for mini-batch estimation of the hypervolume objective and a random scalarization option that removes the exponential dependence on the number of policies.","feed_headline":"One training run covers the whole LLM trade-off frontier","feed_subtitle":"Instead of picking human preferences first, HaM trains several policies whose trade-offs cover the frontier.","key_machinery":"The load-bearing object is the inclusion-exclusion hypervolume estimator in Eq. (6), a closed-form formula for the hypervolume indicator: the volume of the union of J-dimensional rectangles with lower-left corner at the origin and upper-right corner at each policy's normalized reward-weighted log-likelihood vector. This formula converts multi-objective diversity into a single scalar that is monotone and submodular in the policy set, so greedy gradient-based optimization with Adam is a natural fit. The other pieces of machinery make that scalar optimizable in practice: a multi-headed parameterization (shared transformer backbone plus one output head per policy) keeps space use near that of a single model; mini-batching replaces the dataset sum with B random pairs and gives a high-probability error bound of $O(JK\\sqrt{\\log(JK/\\delta)/(2B)})$; and an optional random hypervolume scalarization replaces the sum over all $2^K$ subsets with random scalarizations, trading exactness for an $O(\\sqrt{2J/N})$ estimation error.","core_discovery":"The central claim is that maximizing the hypervolume of a set of policy reward vectors is a valid a-posteriori objective for multi-objective LLM alignment. For K policies with parameters $\\Theta=(\\theta_k)_{k=1}^K$ and J objectives, define $\\bar{L}_j(\\theta_k)$ as the reward-weighted log-likelihood for objective j under policy k, clipped and linearly rescaled to $[0,1]$. HaM maximizes $L_{\\mathrm{ham}}(\\Theta)=\\sum_{S\\in 2^{[K]}\\setminus\\emptyset}(-1)^{|S|-1}\\prod_{j=1}^J \\min_{k\\in S}\\bar{L}_j(\\theta_k)$, the inclusion-exclusion volume of the union of K anchored hyperrectangles; Proposition 1 identifies this with the hypervolume indicator of the points $(\\bar{L}_1(\\theta_k),\\dots,\\bar{L}_J(\\theta_k))$. Maximizing this volume pushes each policy toward high objective values while penalizing overlap among the policies, which is what makes the learned policies both Pareto-optimal and diverse. The paper presents this as the first a-posteriori multi-objective alignment method, implements the policies as separate heads on a shared transformer, and claims empirically superior Pareto fronts over the a-priori baselines.","pith_inferences":["A testable extension the paper does not run is to normalize each objective's training rewards before optimization, since the hypervolume geometry in Eq. (6) is sensitive to reward scale.","The shared-backbone design suggests a cheaper variant: freeze the backbone after initial fine-tuning and optimize only the policy heads, then check whether the same Pareto front is covered.","The same hypervolume objective could be applied at decoding time to rerank a set of candidate responses, extending the method beyond the training-time policies the paper evaluates.","If the front improvements are genuine, they motivate combining HaM with interactive preference refinement: show the user the learned front and let them choose or adjust the operating point after seeing actual responses."],"forward_implications":["After training, a user can choose among K policies without having specified preferences in advance, because the policies are spread across the Pareto front rather than concentrated around one scalarization.","Storing K policies costs roughly the memory of one model, since the policies share a transformer backbone and differ only in lightweight heads.","Mini-batching makes each HaM iteration cost $O(JKB + JK2^K)$ instead of $O(JKn + JK2^K)$, with a proved high-probability error of order $O(JK/\\sqrt{B})$.","The optional random hypervolume scalarization removes the exponential $2^K$ term from the per-iteration cost, at the price of an $O(\\sqrt{2J/N})$ estimation error.","In the reported experiments, HaM's Pareto front is never dominated by a baseline front, and it dominates the RiC or SCA baseline in most settings across the harmless-helpful, harmless-humor, faithful-hallucination, and three-objective tasks."],"supporting_citations":[{"why":"Defines the hypervolume indicator and hypervolume-maximization selection criterion; Proposition 1 equates the HaM objective to this indicator.","marker":"Emmerich et al. (2005)"},{"why":"Supplies the inclusion-exclusion estimator and differentiable expected hypervolume machinery that Eq. (6) generalizes to K policies and J objectives.","marker":"Daulton et al. (2020)"},{"why":"Establishes the a-posteriori multi-objective optimization viewpoint, the rationale for covering the Pareto front, which the paper claims to apply to LLM alignment for the first time.","marker":"Miettinen (1998)"},{"why":"Provides the rewards-in-context baseline and the observation that preference-to-reward mapping is nonlinear, which motivates a-posteriori coverage; also supplies the online data-augmentation protocol.","marker":"Yang et al. (2024)"},{"why":"Provides the random hypervolume scalarization identity and its concentration rate, used to cut the exponential dependence on K.","marker":"Zhang and Golovin (2020)"},{"why":"Proves monotonicity and submodularity of the hypervolume indicator, used to justify greedy gradient optimization of the HaM objective.","marker":"Ulrich and Thiele (2012)"},{"why":"Supplies the sub-Gaussian concentration inequality used in Theorem 1's bound on mini-batch hypervolume estimation error.","marker":"Boucheron et al. (2013)"},{"why":"Supplies the human preference dataset used in the harmless-helpful and harmless-humor experiments.","marker":"Bai et al. (2022)"},{"why":"Supplies the summarization dataset used in the faithful-hallucination experiments.","marker":"Stiennon et al. (2020)"},{"why":"Supplies the low-rank adaptation method that makes the shared-backbone multi-head policy representation space-efficient.","marker":"Hu et al. (2022)"}],"fun_headline_variants":["HaM: One pass to span the LLM trade-off frontier","Diverse LLM policies via hypervolume max","Cover all preferences: HaM's a-posteriori MOO","Train once, get the whole Pareto set of LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training objective in Eq. (3), a reward-weighted log-likelihood $\\bar{L}_j(\\theta)=\\mathbb{E}[r_j(x,y)\\log p(y\\mid x;\\theta)]$ normalized to $[0,1]$, is an adequate stand-in for the reward-model scores later used to evaluate the Pareto front; if the two diverge, the claimed Pareto-front improvements would not reflect genuine multi-objective alignment.","fun_headline_variants_meta":{"raw":{"variants":["HaM: One pass to span the LLM trade-off frontier","Diverse LLM policies via hypervolume max","Cover all preferences: HaM's a-posteriori MOO","Train once, get the whole Pareto set of LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1200,"prompt_tokens":943,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":559,"tokens_out":257,"duration_ms":3598,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:41:44.765247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HaM with one objective's training rewards multiplied by a large constant while keeping the evaluation reward models unchanged: if the hypervolume is merely rescaled and the Pareto front in evaluation space stays the same, the proxy is robust; if the front shifts or collapses, the claim that HaM covers the true Pareto front fails. A second decisive check is to compare HaM trained on the paper's proxy rewards against HaM trained directly on the evaluation reward models, and look for a large drop in evaluated Pareto-front quality under the proxy.","supporting_citations":[],"review_version":1}