{"id":"91a5309a-d0ef-477e-8cd9-2a96cf1d30d7","arxiv_id":"2505.23708","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single weight-conditioned reinforcement learning policy can span the Pareto front of motion tracking objectives, enabling post-training reward tuning and hierarchical weight selection for character control.","lead":"AMOR trains one character-control policy that takes reward weights as an input, so users can rebalance motion tracking objectives after training without retraining. The paper shows this lets a user tune a bipedal robot for dynamic dances, and lets a high-level policy adjust weights on the fly for novel tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linear scalarization in §4.1 (Eq. 2) only realizes the convex coverage set; the paper asserts convexity for the seven tracking objectives without verifying it, so the 'entire Pareto front / all preferences' claim is unsupported if any unsupported Pareto point exists.","rationale":"I read the paper as a practical claim: a single weight-conditioned policy enables post-training trade-off selection, and the hierarchical extension leverages that capability. The strongest version of the claim is that the policy spans the full Pareto front of the seven tracking rewards. The load-bearing premise is the convexity/CCS assumption in Section 4.1: without it, linear scalarization cannot reach unsupported Pareto-optimal behaviors, and 'covering the entire space of preferences' fails. I considered the more operational concerns—missing error bars in Table 2, unreleased code, and the absence of an explicit held-out motion split—but those affect the strength of the demonstrations rather than the mechanism underlying the main novelty. The convexity issue is more central because it sets an upper bound on what the method can express, regardless of engineering quality. The paper earns credit for the weight-conditioning design, the Figure 6 smoothness-adaptation evidence, and the real-robot transfer, all of which support the narrower tuning claim. The concrete test above would settle whether the stronger Pareto-coverage claim holds; until it is run, the appropriate verdict remains conditional, matching the reader's CONDITIONAL verdict. I therefore recommend UNCHANGED.","tokens_in":18597,"tokens_out":10539,"duration_ms":115753,"concrete_test":"Pick a representative motion and build a high-quality reference Pareto set using PPO with Chebyshev scalarizations (or a multi-policy MORL method) over a dense preference grid, with identical seeds and episode counts. For each Pareto-optimal reference point J, solve the LP: is there w∈Δ^7 such that w·J ≥ w·J' for all other reference points J'? If some Pareto-optimal J has no such w, it is unsupported, the front is non-convex, and AMOR's linear weight conditioning cannot reach that trade-off—the 'entire Pareto front' claim fails. Then, for each supported reference point, run AMOR with the supporting w and check that its measured returns match; a mismatch would separate non-convexity from approximation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that one weight-conditioned policy 'spans the Pareto front' and 'covers the entire space of preferences'—requires that every Pareto-optimal trade-off be reachable by some linear weight vector w in Δ^7. As Section 4.1 (Eq. 2 and footnote 1) acknowledges, linear scalarization realizes only the convex coverage set (CCS). On a non-convex Pareto front, there are Pareto-optimal return vectors J that do not maximize w·J for any w; AMOR's only trade-off dial is w, so such behaviors are unreachable. The paper's support for convexity is a citation to continuous-control MORL work, not a check of these seven tracking objectives. Figure 4 does not fill this gap: if each sampled J(w) were exactly optimal for its w, pairwise non-dominance of the 8192 sampled points follows for positive weights even on a non-convex front, so the plot cannot validate full coverage. The objectives include nonsmooth tracking-error norms, contact switches, and hard balance constraints, precisely where concave frontier segments are plausible. If an unsupported Pareto point exists, the 'entire space of preferences' claim is false, although the narrower 'on-the-fly tuning for user-chosen w' functionality could still hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AMOR, a policy conditioned on both a motion context and a reward-weight vector, trained with a multi-objective extension of PPO (MOPPO) in which each episode samples weights uniformly from the 7-dimensional simplex. The authors claim that a single learned policy can track a dataset of kinematic reference motions while spanning the Pareto front of seven tracking objectives, enabling zero-shot, on-the-fly weight tuning after training. They demonstrate this on a simulated humanoid, use manually time-varying weights to transfer a dancing and a double-pirouette motion to a physical bipedal robot, and introduce a high-level policy (HLP) that selects low-level weights using a discriminator-based implicit reward. The paper closes with comparisons of HLP versus fixed weights in terms of joint MAE and discriminator logits.","tokens_in":18848,"tokens_out":5435,"duration_ms":53906,"significance":"If the central claims hold, AMOR is a practically valuable contribution: it replaces the expensive retrain-per-weight-set loop with post-training weight selection, which is especially relevant for sim-to-real transfer, and it offers an interpretable hierarchical mechanism for re-weighting objectives during execution. The paper has notable strengths: the MOPPO derivation in Section 4.2 is clean and sound, the robot experiments are carried out on real hardware, the weight-selection visualization in Fig. 8 is informative, and the use of a separately trained discriminator for evaluation is a reasonable implicit assessment. However, the strongest claim—that the policy spans the full Pareto front and covers the entire space of preferences—rests on an unsupported convexity assumption, and several quantitative claims lack the statistical detail needed to support them.","major_comments":[{"comment":"The paper defines the Pareto front through linear scalarization, asserting that for continuous robotic control the front is \"typically convex,\" and this assertion is the only support for the claims in Sections 3 and 7.2 that the policy covers \"the entire space of preferences\" and tracks \"arbitrary reference motions under arbitrary weightings.\" Linear scalarization is guaranteed to realize only the convex coverage set; if any Pareto-optimal return vector lies on a concave region of the front, no w in the simplex reaches it. The seven objectives include nonsmooth tracking-error norms, contact switches, and balance constraints, where concave segments are plausible, and the citation to continuous-control MORL does not establish convexity for this specific reward vector. Figure 4 cannot resolve this: if the policy were exactly optimal for each sampled w, pairwise non-dominance of the samples would follow even on a non-convex front, so the plot does not test full coverage. Please either verify or empirically probe convexity of the empirical front (for example, by checking whether any sampled return vector is dominated or is not supported by any positive linear weight, or by comparing against a population-based MORL method), or restrict the claims to the convex coverage set and to all linear preferences, which would still leave the main workflow valuable.","section":"Section 4.1, Eq. (2) and footnote 1"},{"comment":"The headline result for the hierarchical policy—\"significant improvements\" in both MAE and discriminator logits—is presented without any variance estimate, confidence interval, or statistical test. The differences are 10.02 versus 9.55 degrees MAE and -18.02 versus -15.48 logits; with 80,000 episodes these differences could be statistically significant, but the reader cannot tell whether they are stable across training seeds, initialization, or motion subsets. Please report means over multiple independent training runs with standard errors, or at minimum bootstrap intervals, and state the number of seeds used.","section":"Section 7.3, Table 2"},{"comment":"The statement that \"all of the 8192 points are Pareto non-dominated w.r.t. all objectives\" needs a precise operational definition. The panels show only pairwise slices of a 7-dimensional return space, so the reader cannot verify a 7D dominance claim from the figure; moreover, for an approximate policy trained by MOPPO, exact non-dominance of every sampled point is surprising and suggests either that the policy is nearly optimal for every sampled weight (which should be demonstrated) or that the dominance check was performed in a way that is not described. Please specify the algorithm and tolerance used for the 7D dominance test, report how many sampled points were excluded or re-sampled, and show the distribution of scalarized returns.","section":"Section 7.2, Fig. 4"}],"minor_comments":[{"comment":"The symbol J(pi) is used for both the vector return and the scalarized return in the same displayed equation; please distinguish the two notations, for instance by writing J_w(pi) or by putting J(pi)\\cdot w on the left-hand side.","section":"Section 4.1, Eq. (3)"},{"comment":"The per-reward prior scales are said to be \"informed by [Serifi et al. 2024],\" but the text does not state whether these are exactly the VMP scales or adjusted for AMOR; because the scales change the effective preference space implied by w, please state explicitly how they were set and whether the \"arbitrary weights\" claim refers to scaled or unscaled rewards.","section":"Section 5, Table 1"},{"comment":"The comparison with the fixed-weight VMP controller for the double pirouette is presented without quantitative metrics; please report at least tracking error, completion rate, or fall rate for VMP versus AMOR on the real robot, or explicitly label the comparison as qualitative.","section":"Section 7.2, robot experiments"},{"comment":"The high-level policy training omits several implementation details needed for reproduction, including the discriminator window size V, the gradient-penalty coefficient c_gp, the HLP reward scale, and the observation and action normalization for the weight outputs; the two HLP variants in Fig. 9 are also not defined in the text.","section":"Sections 6 and 7.3"},{"comment":"The inset training-comparison figure is not numbered and has no caption; please integrate it as a numbered figure and define the curves, in particular what \"MOPPO [fixed]\" means and how the reported reward is computed.","section":"Section 7.2, training comparison"}],"recommendation":"major_revision","confidential_remarks":"The paper is well written and the engineering contribution is substantial, but the authors should decide whether they want to defend the full Pareto-front coverage claim or retreat to the convex coverage set. The latter is probably sufficient for the paper's practical contribution and is much easier to support rigorously. I also recommend that the authors release at least the training configurations, since the paper currently gives no code or data release statement. The missing error bars in Table 2 and the unexplained 7D non-dominance claim in Fig. 4 are issues that can be fixed with additional analysis rather than new algorithmic ideas."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: AMOR is a serious, workmanlike paper that takes an established idea—preference-conditioned multi-objective RL—and makes it work for 7-objective physics-based character control, including on a real robot. The new pieces are the context-conditioned weight policy, the high-level policy that picks weights online using a discriminator reward, and the demonstration that on-the-fly weight tuning lets you transfer a double pirouette to hardware in about a day instead of a five-day retrain. That last result is the real contribution, and it is convincing. The training curves, the Pareto plots, and the robot jitter reduction all hang together, and the authors are honest about the gap between PPO and MOPPO. The citation pattern is fine: prior MORL work and successor features are acknowledged, and the VMP self-citations support the reward scaling choices.\n\nThe soft spots are real but not fatal. The stress-test concern about linear scalarization is legitimate: the paper defines the Pareto front via Eq. 2 and footnotes that this is the convex coverage set, but then the abstract and related work claim the policy 'spans the Pareto front' and 'covers the entire space of preferences.' Those claims require convexity of the seven tracking objectives, which is asserted by citation, not verified. Figure 4 does not help: 8192 sampled points being pairwise non-dominated is consistent with a policy that only reaches a small slice of a non-convex front. I would not sink the paper on this—the user-facing feature is weight-conditioned behavior, which works for any user-chosen weight in the simplex—but the authors should either verify convexity for their specific rewards or soften the language.\n\nThe second issue is statistical reporting. Table 2 shows a MAE improvement from 10.02 to 9.55 and logits from -18.02 to -15.48, described as 'significant improvements,' but there are no error bars, no number of seeds, and no significance test. For a deep RL paper this is a standard ask, not a fatal flaw, and it is easy to fix. Third, no code or data are released, which limits reproducibility for a methods paper whose training costs five days on one GPU.\n\nOverall, the central feasibility claim—a single weight-conditioned policy enables post-training tuning across a useful range of trade-offs—holds up. The Pareto coverage language needs tightening, not rejection. I would bring this to a reading group and cite it if I worked in character control or MORL for robotics. Send it to peer review; a serious referee can push the authors on the convexity claim and the missing statistics, but the paper has a real result underneath.","headline":"Solid, practical application of preference-conditioned MORL to motion tracking, with a real robot payoff, but the 'entire Pareto front' claim outruns the evidence.","tokens_in":791,"tokens_out":903,"would_cite":true,"duration_ms":31874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AMOR trains a single character-control policy that accepts arbitrary reward weights after training, so trade-offs between tracking accuracy, smoothness, and other objectives can be re-tuned without retraining.","keywords":["multi-objective reinforcement learning","physics-based character control","motion tracking","reward weight conditioning","Pareto front","sim-to-real transfer","hierarchical reinforcement learning"],"falsifier":"Pick one motion, densely sample weight vectors from the simplex, and compute the seven-dimensional return of the trained policy for each; if any point that is Pareto-optimal for the true tracking problem is not achievable by $\\pi(\\cdot, w)$ for any linear $w$ — for instance, if the set of achieved returns is non-convex or contains a point dominated by a stochastic mixture of two other achieved returns — then the 'arbitrary weightings' claim is falsified for that motion.","tokens_in":18395,"feed_emoji":"🤖","tokens_out":7205,"duration_ms":62280,"temperature":0.7,"pith_summary":"AMOR is a training scheme for physics-based character controllers that moves reward tuning out of the training loop. Instead of fixing a weighted sum of seven tracking objectives before learning, it trains one policy that also takes the weight vector as an input, so after training a user or an algorithm can change the trade-off between accuracy, smoothness, and other terms and immediately see a different behavior. The paper claims that a single such policy approximates the Pareto front of those objectives across many motions, and demonstrates the workflow on a simulated humanoid and on a real bipedal robot, where raising the smoothness weight reduces sim-to-real jitter. It also trains a high-level policy that chooses time-varying weights during execution, using a learned discriminator as the reward, which improves tracking quality and makes the implicit reward legible as a sequence of interpretable weight choices. If the claim holds, the standard retrain-to-retune cycle in character control and robot locomotion would be replaced by real-time weight adjustment.","feed_headline":"One policy covers every reward trade-off after training","feed_subtitle":"AMOR's weight-conditioned policy lets users or algorithms adjust motion-quality trade-offs on the fly, on simulated and real robots.","key_machinery":"The load-bearing object is the weight-conditioned policy $\\pi(a_t \\mid s_t, c_t, w)$ trained by MOPPO, a multi-objective extension of PPO. MOPPO replaces the scalar value function with a vector-valued value function $V^\\pi(s, c, w) = \\mathbb{E}^\\pi\\left[\\sum_{t\\ge 0} \\gamma^t r_t \\mid s_0=s, c_0=c\\right]$ and forms the policy-gradient update from the scalarized vector advantage $A^\\pi(s_t, c_t, a_t)\\cdot w$; weights are drawn per episode from the simplex via a Dirichlet distribution with $\\alpha=1$, forcing one network to cover the whole space of linear scalarizations. The motion context $c_t$ combines the current motion frame with a latent code from a variational autoencoder over a window of past and future frames, so the same weight-conditioned policy generalizes across motion clips. The hierarchical extension adds a high-level policy whose action space is the simplex itself: it outputs weights through a softmax layer and is trained with standard PPO on a discriminator reward, letting the low-level AMOR policy remain frozen.","core_discovery":"The paper's central claim is that a single weight-conditioned policy $\\pi(a_t \\mid s_t, c_t, w)$ can, once converged, track arbitrary reference motions under arbitrary linear weightings of the seven reward terms, so that no retraining is needed to change the behavior. To support this, the authors train the policy with MOPPO, a multi-objective extension of PPO in which each episode samples a weight vector $w$ uniformly from the $(m-1)$-dimensional simplex and the policy updates use a vector-valued advantage function $A^\\pi(s_t, c_t, a_t)$ scalarized by $w$. Evaluating 8192 sampled weight vectors on three motions (idle, walking, dancing) shows returns that are mutually Pareto non-dominated, and manual weight changes visibly shift behavior, such as trading lower-body tracking for smoothness. On a 20-DoF bipedal robot, on-the-fly tuning reduced measured joint jitter and enabled a double pirouette that the fixed-weight baseline could not perform. A second claim is that a high-level policy $\\pi(w_t \\mid s_t, c_t)$ trained against a discriminator-based implicit reward can select weights dynamically within a skill; in simulation this hierarchical controller achieves lower mean joint-position error and higher discriminator logits than fixed uniform weights.","pith_inferences":["The same weight-conditioning recipe should transfer to other contact-rich continuous-control tasks, such as dexterous manipulation or whole-body loco-manipulation, where retraining for each reward trade-off is equally expensive; the paper only demonstrates character and robot locomotion.","Training across the full simplex may act as an implicit regularizer that makes the policy more robust to dynamics mismatches than a fixed-weight policy, since it has learned to behave under many different objectives; the paper does not isolate or test this effect.","One could close the gap between MOPPO and fixed-weight PPO by using the high-level policy's weight distribution as a curriculum for the low-level policy, focusing training capacity on weight vectors that are actually useful rather than uniformly sampling the simplex.","The high-level policy's time-varying weights could be mined automatically to discover motion phases or failure modes, such as jitter-prone segments, without manual inspection."],"forward_implications":["Reward weights become a post-training interface: users can tune behavior in real time, and tuning a new behavior no longer requires a multi-day retraining run.","Because the policy is also conditioned on the motion context, one AMOR policy covers many motion clips under many trade-offs, not a single clip with a single weight.","A high-level policy can change weights within a single skill, so trade-offs that vary over time, such as high velocity early in a pirouette and high smoothness at the end, are reachable without manual scheduling.","The weight sequence chosen by the high-level policy doubles as an interpretable account of what the discriminator-based reward rewards, giving insight into implicit rewards.","For robotics, the framework turns sim-to-real transfer into a weight-selection problem: raising smoothness on the real system reduced jitter and made a demanding dance motion feasible."],"supporting_citations":[{"why":"Defines the MORL problem and the Pareto-front objective that AMOR builds on.","marker":"Hayes et al. 2022"},{"why":"Supplies the convex coverage set and linear-dominance formulation used to justify weight conditioning.","marker":"Roijers et al. 2013"},{"why":"Provides the preference-conditioned single-policy MORL approach that AMOR extends to character control.","marker":"Alegre et al. 2023"},{"why":"Is the PPO algorithm whose clipped objective MOPPO generalizes to vector rewards.","marker":"Schulman et al. 2017"},{"why":"Supplies the motion-context VAE, the seven reward terms and their prior scalings, and the VMP tracking baseline.","marker":"Serifi et al. 2024"},{"why":"Introduces the discriminator-based implicit reward used to train AMOR's high-level policy.","marker":"Peng et al. 2021"},{"why":"Shows conditioning the discriminator on the latent motion code, which the HLP training adopts to avoid mode collapse.","marker":"Tessler et al. 2023"}],"fun_headline_variants":["Train once, tune rewards anytime","Choose robot behavior trade-offs after training","Single policy spans the Pareto front of rewards","Adjust motion quality on the fly without retraining","AMOR's weight-conditioned policy adapts instantly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheme's promise that every relevant behavior can be dialed in with some weight vector depends on the Pareto front of the seven tracking objectives being convex enough that linear scalarization reaches every Pareto-optimal trade-off; the paper cites prior multi-objective RL work for this property but does not verify convexity for these particular objectives.","fun_headline_variants_meta":{"raw":{"variants":["Train once, tune rewards anytime","Choose robot behavior trade-offs after training","Single policy spans the Pareto front of rewards","Adjust motion quality on the fly without retraining","AMOR's weight-conditioned policy adapts instantly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1688,"prompt_tokens":1008,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":624,"tokens_out":680,"duration_ms":6151,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:39:00.582654+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick one motion, densely sample weight vectors from the simplex, and compute the seven-dimensional return of the trained policy for each; if any point that is Pareto-optimal for the true tracking problem is not achievable by $\\pi(\\cdot, w)$ for any linear $w$ — for instance, if the set of achieved returns is non-convex or contains a point dominated by a stochastic mixture of two other achieved returns — then the 'arbitrary weightings' claim is falsified for that motion.","supporting_citations":[],"review_version":1}