AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning

Agon Serifi; David M\"uller; Espen Knoop; Lucas N. Alegre; Moritz B\"acher; Ruben Grandia

REVIEW 3 major objections 5 minor 2 references

AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AMOR trains a single character-control policy that accepts arbitrary reward weights after training, so trade-offs between tracking accuracy, smoothness, and other objectives can be re-tuned without retraining.

desk verdict Solid, practical application of preference-conditioned MORL to motion tracking, with a real robot payoff, but the 'entire Pareto front' claim outruns the evidence. read the letter →

arxiv 2505.23708 v1 pith:GNV6R5JU submitted 2025-05-29 cs.RO cs.GR

Lucas N. Alegre , Agon Serifi , Ruben Grandia , David Müller , Espen Knoop , Moritz Bächer This is my paper

classification cs.ROcs.GR

keywords multi-objectivereinforcementlearningphysics-basedcharactercontrolmotiontrackingrewardweightconditioningParetofrontsim-to-realtransferhierarchical

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AMOR is a training scheme for physics-based character controllers that moves reward tuning out of the training loop. Instead of fixing a weighted sum of seven tracking objectives before learning, it trains one policy that also takes the weight vector as an input, so after training a user or an algorithm can change the trade-off between accuracy, smoothness, and other terms and immediately see a different behavior. The paper claims that a single such policy approximates the Pareto front of those objectives across many motions, and demonstrates the workflow on a simulated humanoid and on a real bipedal robot, where raising the smoothness weight reduces sim-to-real jitter. It also trains a high-level policy that chooses time-varying weights during execution, using a learned discriminator as the reward, which improves tracking quality and makes the implicit reward legible as a sequence of interpretable weight choices. If the claim holds, the standard retrain-to-retune cycle in character control and robot locomotion would be replaced by real-time weight adjustment.

What carries the argument

The load-bearing object is the weight-conditioned policy $\pi(a_t \mid s_t, c_t, w)$ trained by MOPPO, a multi-objective extension of PPO. MOPPO replaces the scalar value function with a vector-valued value function $V^\pi(s, c, w) = \mathbb{E}^\pi\left[\sum_{t\ge 0} \gamma^t r_t \mid s_0=s, c_0=c\right]$ and forms the policy-gradient update from the scalarized vector advantage $A^\pi(s_t, c_t, a_t)\cdot w$; weights are drawn per episode from the simplex via a Dirichlet distribution with $\alpha=1$, forcing one network to cover the whole space of linear scalarizations. The motion context $c_t$ combines the current motion frame with a latent code from a variational autoencoder over a window of past and future frames, so the same weight-conditioned policy generalizes across motion clips. The hierarchical extension adds a high-level policy whose action space is the simplex itself: it outputs weights through a softmax layer and is trained with standard PPO on a discriminator reward, letting the low-level AMOR policy remain frozen.

What would settle it

Pick one motion, densely sample weight vectors from the simplex, and compute the seven-dimensional return of the trained policy for each; if any point that is Pareto-optimal for the true tracking problem is not achievable by $\pi(\cdot, w)$ for any linear $w$ — for instance, if the set of achieved returns is non-convex or contains a point dominated by a stochastic mixture of two other achieved returns — then the 'arbitrary weightings' claim is falsified for that motion.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single weight-conditioned policy $\pi(a_t \mid s_t, c_t, w)$ can, once converged, track arbitrary reference motions under arbitrary linear weightings of the seven reward terms, so that no retraining is needed to change the behavior. To support this, the authors train the policy with MOPPO, a multi-objective extension of PPO in which each episode samples a weight vector $w$ uniformly from the $(m-1)$-dimensional simplex and the policy updates use a vector-valued advantage function $A^\pi(s_t, c_t, a_t)$ scalarized by $w$. Evaluating 8192 sampled weight vectors on three motions (idle, walking, dancing) shows returns that are mutually Pareto non-dominated, and manual weight changes visibly shift behavior, such as trading lower-body tracking for smoothness. On a 20-DoF bipedal robot, on-the-fly tuning reduced measured joint jitter and enabled a double pirouette that the fixed-weight baseline could not perform. A second claim is that a high-level policy $\pi(w_t \mid s_t, c_t)$ trained against a discriminator-based implicit reward can select weights dynamically within a skill; in simulation this hierarchical controller achieves lower mean joint-position error and higher discriminator logits than fixed uniform weights.

Load-bearing premise

The scheme's promise that every relevant behavior can be dialed in with some weight vector depends on the Pareto front of the seven tracking objectives being convex enough that linear scalarization reaches every Pareto-optimal trade-off; the paper cites prior multi-objective RL work for this property but does not verify convexity for these particular objectives.

Editorial extensions

If this is right

Reward weights become a post-training interface: users can tune behavior in real time, and tuning a new behavior no longer requires a multi-day retraining run.
Because the policy is also conditioned on the motion context, one AMOR policy covers many motion clips under many trade-offs, not a single clip with a single weight.
A high-level policy can change weights within a single skill, so trade-offs that vary over time, such as high velocity early in a pirouette and high smoothness at the end, are reachable without manual scheduling.
The weight sequence chosen by the high-level policy doubles as an interpretable account of what the discriminator-based reward rewards, giving insight into implicit rewards.
For robotics, the framework turns sim-to-real transfer into a weight-selection problem: raising smoothness on the real system reduced jitter and made a demanding dance motion feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same weight-conditioning recipe should transfer to other contact-rich continuous-control tasks, such as dexterous manipulation or whole-body loco-manipulation, where retraining for each reward trade-off is equally expensive; the paper only demonstrates character and robot locomotion.
Training across the full simplex may act as an implicit regularizer that makes the policy more robust to dynamics mismatches than a fixed-weight policy, since it has learned to behave under many different objectives; the paper does not isolate or test this effect.
One could close the gap between MOPPO and fixed-weight PPO by using the high-level policy's weight distribution as a curriculum for the low-level policy, focusing training capacity on weight vectors that are actually useful rather than uniformly sampling the simplex.
The high-level policy's time-varying weights could be mined automatically to discover motion phases or failure modes, such as jitter-prone segments, without manual inspection.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: AMOR is a serious, workmanlike paper that takes an established idea—preference-conditioned multi-objective RL—and makes it work for 7-objective physics-based character control, including on a real robot. The new pieces are the context-conditioned weight policy, the high-level policy that picks weights online using a discriminator reward, and the demonstration that on-the-fly weight tuning lets you transfer a double pirouette to hardware in about a day instead of a five-day retrain. That last result is the real contribution, and it is convincing. The training curves, the Pareto plots, and the robot jitter reduction all hang together, and the authors are honest about the gap between PPO and MOPPO. The citation pattern is fine: prior MORL work and successor features are acknowledged, and the VMP self-citations support the reward scaling choices.

The soft spots are real but not fatal. The stress-test concern about linear scalarization is legitimate: the paper defines the Pareto front via Eq. 2 and footnotes that this is the convex coverage set, but then the abstract and related work claim the policy 'spans the Pareto front' and 'covers the entire space of preferences.' Those claims require convexity of the seven tracking objectives, which is asserted by citation, not verified. Figure 4 does not help: 8192 sampled points being pairwise non-dominated is consistent with a policy that only reaches a small slice of a non-convex front. I would not sink the paper on this—the user-facing feature is weight-conditioned behavior, which works for any user-chosen weight in the simplex—but the authors should either verify convexity for their specific rewards or soften the language.

The second issue is statistical reporting. Table 2 shows a MAE improvement from 10.02 to 9.55 and logits from -18.02 to -15.48, described as 'significant improvements,' but there are no error bars, no number of seeds, and no significance test. For a deep RL paper this is a standard ask, not a fatal flaw, and it is easy to fix. Third, no code or data are released, which limits reproducibility for a methods paper whose training costs five days on one GPU.

Overall, the central feasibility claim—a single weight-conditioned policy enables post-training tuning across a useful range of trade-offs—holds up. The Pareto coverage language needs tightening, not rejection. I would bring this to a reading group and cite it if I worked in character control or MORL for robotics. Send it to peer review; a serious referee can push the authors on the convexity claim and the missing statistics, but the paper has a real result underneath.

Referee Report

3 major / 5 minor

Summary. The paper proposes AMOR, a policy conditioned on both a motion context and a reward-weight vector, trained with a multi-objective extension of PPO (MOPPO) in which each episode samples weights uniformly from the 7-dimensional simplex. The authors claim that a single learned policy can track a dataset of kinematic reference motions while spanning the Pareto front of seven tracking objectives, enabling zero-shot, on-the-fly weight tuning after training. They demonstrate this on a simulated humanoid, use manually time-varying weights to transfer a dancing and a double-pirouette motion to a physical bipedal robot, and introduce a high-level policy (HLP) that selects low-level weights using a discriminator-based implicit reward. The paper closes with comparisons of HLP versus fixed weights in terms of joint MAE and discriminator logits.

Significance. If the central claims hold, AMOR is a practically valuable contribution: it replaces the expensive retrain-per-weight-set loop with post-training weight selection, which is especially relevant for sim-to-real transfer, and it offers an interpretable hierarchical mechanism for re-weighting objectives during execution. The paper has notable strengths: the MOPPO derivation in Section 4.2 is clean and sound, the robot experiments are carried out on real hardware, the weight-selection visualization in Fig. 8 is informative, and the use of a separately trained discriminator for evaluation is a reasonable implicit assessment. However, the strongest claim—that the policy spans the full Pareto front and covers the entire space of preferences—rests on an unsupported convexity assumption, and several quantitative claims lack the statistical detail needed to support them.

major comments (3)

[Section 4.1, Eq. (2) and footnote 1] The paper defines the Pareto front through linear scalarization, asserting that for continuous robotic control the front is "typically convex," and this assertion is the only support for the claims in Sections 3 and 7.2 that the policy covers "the entire space of preferences" and tracks "arbitrary reference motions under arbitrary weightings." Linear scalarization is guaranteed to realize only the convex coverage set; if any Pareto-optimal return vector lies on a concave region of the front, no w in the simplex reaches it. The seven objectives include nonsmooth tracking-error norms, contact switches, and balance constraints, where concave segments are plausible, and the citation to continuous-control MORL does not establish convexity for this specific reward vector. Figure 4 cannot resolve this: if the policy were exactly optimal for each sampled w, pairwise non-dominance of the samples would follow even on a non-convex front, so the plot does not test full coverage. Please either verify or empirically probe convexity of the empirical front (for example, by checking whether any sampled return vector is dominated or is not supported by any positive linear weight, or by comparing against a population-based MORL method), or restrict the claims to the convex coverage set and to all linear preferences, which would still leave the main workflow valuable.
[Section 7.3, Table 2] The headline result for the hierarchical policy—"significant improvements" in both MAE and discriminator logits—is presented without any variance estimate, confidence interval, or statistical test. The differences are 10.02 versus 9.55 degrees MAE and -18.02 versus -15.48 logits; with 80,000 episodes these differences could be statistically significant, but the reader cannot tell whether they are stable across training seeds, initialization, or motion subsets. Please report means over multiple independent training runs with standard errors, or at minimum bootstrap intervals, and state the number of seeds used.
[Section 7.2, Fig. 4] The statement that "all of the 8192 points are Pareto non-dominated w.r.t. all objectives" needs a precise operational definition. The panels show only pairwise slices of a 7-dimensional return space, so the reader cannot verify a 7D dominance claim from the figure; moreover, for an approximate policy trained by MOPPO, exact non-dominance of every sampled point is surprising and suggests either that the policy is nearly optimal for every sampled weight (which should be demonstrated) or that the dominance check was performed in a way that is not described. Please specify the algorithm and tolerance used for the 7D dominance test, report how many sampled points were excluded or re-sampled, and show the distribution of scalarized returns.

minor comments (5)

[Section 4.1, Eq. (3)] The symbol J(pi) is used for both the vector return and the scalarized return in the same displayed equation; please distinguish the two notations, for instance by writing J_w(pi) or by putting J(pi)\cdot w on the left-hand side.
[Section 5, Table 1] The per-reward prior scales are said to be "informed by [Serifi et al. 2024]," but the text does not state whether these are exactly the VMP scales or adjusted for AMOR; because the scales change the effective preference space implied by w, please state explicitly how they were set and whether the "arbitrary weights" claim refers to scaled or unscaled rewards.
[Section 7.2, robot experiments] The comparison with the fixed-weight VMP controller for the double pirouette is presented without quantitative metrics; please report at least tracking error, completion rate, or fall rate for VMP versus AMOR on the real robot, or explicitly label the comparison as qualitative.
[Sections 6 and 7.3] The high-level policy training omits several implementation details needed for reproduction, including the discriminator window size V, the gradient-penalty coefficient c_gp, the HLP reward scale, and the observation and action normalization for the weight outputs; the two HLP variants in Fig. 9 are also not defined in the text.
[Section 7.2, training comparison] The inset training-comparison figure is not numbered and has no caption; please integrate it as a numbered figure and define the curves, in particular what "MOPPO [fixed]" means and how the reported reward is computed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: weights are inputs, the Pareto-front claim is the training objective, and the high-level policy is evaluated with a separately trained discriminator. The convexity caveat is a correctness/scope issue, not a circular step.

full rationale

The paper does not fit a parameter and then rename it as a prediction. The weight vector w is sampled from the simplex and is an input to both the policy and the MOPPO update (Section 4.2, Eq. 5); the per-objective scales in Table 1 are hand-set hyperparameters inherited from VMP [Serifi et al. 2024], not fitted to the reported returns. The central "spanning the Pareto front" statement is the algorithm's optimization objective: Section 4.1 defines the reachable set via linear scalarization (Eq. 2), and MOPPO maximizes w·J, so the conditional behavior is an empirical outcome of training rather than a hidden re-derivation of an input. The HLP comparison is not self-confirming because Section 7.3 explicitly states that the evaluation discriminator is trained separately and is not the discriminator used to train the HLP. Self-citations to VMP and to Alegre et al. / Felten et al. are components or context; they do not carry the load-bearing argument for the multi-objective result. The main caveat, that linear scalarization only realizes the convex coverage set (footnote 1) and that convexity of the seven tracking objectives is asserted rather than verified, is a real scope/correctness limitation of the "entire Pareto front" claim, not a circular reduction. Figure 4's all-sampled-points-non-dominated observation is also largely forced by near-optimality for each sampled w, so it is weak evidence of full coverage; this weakens the coverage claim but does not constitute an equation-level circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three domain assumptions: convexity of the Pareto front under linear scalarization, zero-shot generalization of a single network across the weight simplex and motion contexts, and sufficiency of the VMP VAE representation. The only fitted knobs are the per-reward prior scaling constants in Table 1, which are adopted from a self-cited paper and not tuned in this work.

free parameters (2)

Per-reward prior scales, humanoid = upper/lower/feet 1.0; rbs/root/vel 1.0; smooth terms 1e-5, 1e-5, 1e-5, 1e-6
Table 1 lists hand-assigned scalings for the seven objectives; they determine the effective weight space and are taken from VMP (Serifi et al. 2024) without fitting or sensitivity analysis.
Per-reward prior scales, robot = up/lo/feet 7.0; rbs/root 1.0; vel 2.0; smooth terms 1e-4, 1.5, 0.45, 2.5e-6
Table 1; the authors state smoothness needs much higher weight on hardware, and these constants influence the sim-to-real results and the reported jitter reduction.

assumptions (3)

domain assumption The Pareto front of the seven tracking objectives is convex, so every relevant trade-off is reachable by a linear weight vector in the simplex.
Section 4.1, Eq. (2) and footnote 1: the paper assumes the convex coverage set characterization, citing Felten et al. 2023 and Xu et al. 2020 for continuous control. This is not verified for these motion-tracking objectives.
domain assumption A single policy trained with per-episode random weight vectors from a uniform Dirichlet distribution generalizes zero-shot to all weights in the simplex and all motion contexts.
Section 4.2 and 5: the training samples w ~ Dirichlet(1) per episode; the paper relies on function approximation to interpolate across the entire weight space rather than demonstrating coverage.
domain assumption The VAE motion-context encoder from VMP provides a sufficient context representation for tracking diverse motions.
Section 5, Motion Context: the paper directly reuses VMP's VAE and refers to that paper for details, including loss, normalization, and training.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning." pith.science (2026). https://pith.science/paper/GNV6R5JU

@misc{pith2026250523708,
  author       = {Pith},
  title        = {Pith review of: AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GNV6R5JU}},
  note         = {Machine review of arXiv:2505.23708}
}

read the original abstract

Reinforcement learning (RL) has significantly advanced the control of physics-based and robotic characters that track kinematic reference motion. However, methods typically rely on a weighted sum of conflicting reward functions, requiring extensive tuning to achieve a desired behavior. Due to the computational cost of RL, this iterative process is a tedious, time-intensive task. Furthermore, for robotics applications, the weights need to be chosen such that the policy performs well in the real world, despite inevitable sim-to-real gaps. To address these challenges, we propose a multi-objective reinforcement learning framework that trains a single policy conditioned on a set of weights, spanning the Pareto front of reward trade-offs. Within this framework, weights can be selected and tuned after training, significantly speeding up iteration time. We demonstrate how this improved workflow can be used to perform highly dynamic motions with a robot character. Moreover, we explore how weight-conditioned policies can be leveraged in hierarchical settings, using a high-level policy to dynamically select weights according to the current task. We show that the multi-objective policy encodes a diverse spectrum of behaviors, facilitating efficient adaptation to novel tasks.

Figures

Figures reproduced from arXiv: 2505.23708 by the authors.

**Figure 1.** Our method uses multi-objective reinforcement learning to enable on-the-fly tuning of reward weights post-training, which can be used to transfer [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** shows the structure of AMOR. At its core, it is an RL-based controller that tracks a kinematic reference motion, similarly to related work [Serifi et al. 2024]. The policy is trained to output actions a𝑡 that maximize the reward of a simulated character, given the current state of the character s𝑡 , and a context vector c𝑡 encoding task-relevant information. Specifically, in our context, c𝑡 includes a time-varying k… view at source ↗

**Figure 3.** High-Level Policy Overview. We learn a high-level policy (HLP) that generates reward weights for a pretrained AMOR based on the current motion context. In this stage, a discriminator is trained to distinguish between reference and simulated motions, with its output serving as an implicit reward for the HLP. Intuitively, this reward function rewards the agent for performing transitions that appear indistinguishable … view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: displays the Pareto fronts obtained by tracking three different motions (Idle, Walking, Dancing) on the humanoid by following 𝜋. Because each point J(𝜋) on the Pareto front is a 7-dimensional vector, we depict pairwise comparisons between the unscaled cumulative rewar…

**Figure 7.** Figure 7: Weight Tuning. The left column shows the measured DoF [ ◦ ], and the right column shows the measured DoF velocities [ ◦ /s] for the three pitch joints of the robot’s right leg during the dance motion in the supplementary video. The purple curve shows the measurements o…

**Figure 5.** Figure 5: Weight Influence. (Top) Kinematic reference. (Middle) Visual performance when prioritizing tracking reward weights. (Bottom) Visual performance when prioritizing smoothness reward terms. 0.4 0.6 0.8 1.0 Cumulative Smoothness Reward 0.00 0.01 0.02 Weight: 0.1 0.5 0.9 …

**Figure 6.** Figure 6: Prioritizing Objectives. Distributions of the unweighted cumulative smoothness reward for three distinct weight values, each evaluated on 32768 episodes. -30 0 30 -350 0 350 -30 0 30 -350 0 350 0 1 2 -30 0 30 0 1 2 -350 0 350 Hip Pitch Knee Pitch Ankle Pitch Time [s] …

**Figure 8.** Figure 8: Adaptive Weights. Visualization of the behavior of AMOR’s high-level policy 𝜋 (w𝑡 |s𝑡 , ct) through different types of motion. The resulting reward weights w𝑡 ∈ Δ𝑚 are plotted on a logarithmic scale; this better captures their multiplicative effect on the reward. 1 2 3…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

[2017]

arXiv:1707.06347 http://arxiv.org/abs/1707.06347 Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojciech Matusik

Proximal Policy Optimization Algorithms.CoRRabs/1707.06347 (2017). arXiv:1707.06347 http://arxiv.org/abs/1707.06347 Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojciech Matusik

arXiv 2017
[2018]

Graph.37, 4, Article 131 (July 2018), 14 pages

Interactive exploration of design trade-offs.ACM Trans. Graph.37, 4, Article 131 (July 2018), 14 pages. doi:10.1145/3197517.3201385 Agon Serifi, Ruben Grandia, Espen Knoop, Markus Gross, and Moritz Bächer. 2024. VMP: Versatile Motion Priors for Robustly Tracking Motion on Physical Characters. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Co...

arXiv 2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2017]

arXiv:1707.06347 http://arxiv.org/abs/1707.06347 Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojciech Matusik

Proximal Policy Optimization Algorithms.CoRRabs/1707.06347 (2017). arXiv:1707.06347 http://arxiv.org/abs/1707.06347 Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojciech Matusik

arXiv 2017

[2] [2018]

Graph.37, 4, Article 131 (July 2018), 14 pages

Interactive exploration of design trade-offs.ACM Trans. Graph.37, 4, Article 131 (July 2018), 14 pages. doi:10.1145/3197517.3201385 Agon Serifi, Ruben Grandia, Espen Knoop, Markus Gross, and Moritz Bächer. 2024. VMP: Versatile Motion Priors for Robustly Tracking Motion on Physical Characters. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Co...

arXiv 2018