Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NEAR trains a noise-conditioned energy-based reward from expert state transitions and anneals the noise level during RL to imitate humanoid motions at AMP-comparable performance.

desk verdict NEAR's non-adversarial energy-reward idea is promising, but Eq. (2)'s sign error makes the method as written anti-imitative; the paper needs code verification. read the letter →

arxiv 2501.14856 v2 pith:WUV3W3OT submitted 2025-01-24 cs.RO cs.AI

Anish Abhijit Diwan , Julen Urain , Jens Kober , Jan Peters This is my paper · ORCID
classification cs.ROcs.AI
keywords imitationlearningmotionadversarialenergyenergy-basedframeworkfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many robots learn by imitating an expert. The hardest version, imitation from observation, gives only the expert's poses, for example joint positions over time, not the commands that produced them. The robot must also learn the physics of its own body. Standard methods use an adversarial discriminator that tries to tell expert poses from the robot's poses and is used as a reward. These methods work, but they can be unstable because the discriminator and the policy chase each other.

NEAR takes a different path. It first trains a neural network to estimate an energy function for the expert's motions. The energy is high for poses close to the expert's data and low for unusual poses. To make the energy smooth everywhere, the data is corrupted with different amounts of noise, and the network is trained to denoise it. This is the same score matching idea used in modern generative models.

Once the energy network is trained, it is used as a fixed reward: the robot gets a score for how expert-like its next state is. A policy is then trained with reinforcement learning, specifically PPO. The paper adds an annealing schedule: training starts with a heavily smoothed energy function, high noise, and gradually switches to sharper energies as the robot improves, which keeps the reward informative even when the robot is still learning. The paper tests NEAR on humanoid walking, running, punching, and martial arts in the IsaacGym simulator and compares with AMP. NEAR achieves similar pose error and smoother trajectories in several tasks, while avoiding the discriminator's instability. Its main weakness is in single-demo tasks and at moments when the noise level changes, where the reward can become uninformative.

Extended reading notes

Core claim

From Section 1: 'We propose to use the learnt energy functions as reward functions and present a new imitation learning algorithm called NEAR that has better, more stable learning dynamics, and learns smooth and unambiguous reward signals. NEAR produces motions comparable to state-of-the-art adversarial imitation learning methods like AMP.' If the paper is correct, energy-based rewards learned once via denoising score matching, with annealing over noise levels, yield imitation performance on humanoid locomotion and martial arts comparable to AMP, without needing a concurrently trained discriminator.

Load-bearing premise

The learned energy network is trained only on perturbed expert state transitions, so its values outside the support of those perturbed distributions are unconstrained. The paper assumes that a sufficiently large initial noise sigma1 plus the annealing rule keeps the support of the policy's state-transition distribution inside the support of the current perturbed distribution q_sigma_k for all k. This containment is asserted in Section 4.2 and Appendix A.1 ('the manifold of policy-generated samples always lies in P'), but no proof or formal bound is given. Figure 5 and the Limitations section show that when this assumption fails, at noise-level switches, the reward becomes ill-defined and returns drop. If a policy explores states where the energy network has not been trained, the reward is arbitrary and can mislead RL.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes NEAR, an imitation-from-observation algorithm that first trains a noise-conditioned energy network on expert state transitions via denoising score matching and then uses the learned energy as a reward for reinforcement learning, with an annealing schedule that switches among noise levels according to the policy's progress. Experiments on humanoid locomotion, martial arts, and goal-conditioned tasks compare NEAR against AMP and report comparable or better performance on pose error, spectral arc length, task return, and wall-clock time, together with ablations on annealing and task-reward composition.

Significance. If the technical issues below are resolved, the core idea is attractive: a stationary, pretrained reward that avoids adversarial min-max training and can be combined with any RL algorithm. The manuscript has concrete strengths: code and videos are promised, each algorithm is run with five independent seeds, ablations isolate annealing and reward composition, and the limitations section is unusually candid about the failure mode at noise-level switches. However, the central construction currently contains a sign inconsistency in the score-matching objective, and the key support-containment assumption is asserted rather than proved and is contradicted by the paper's own Figure 5. These issues must be addressed before the empirical claims can be accepted.

major comments (4)
  1. [Section 4.1, Eq. (2); Section 4.2; Appendix A.2.1] There is a sign inconsistency in the central reward construction. For the Gaussian perturbation qσ(x′|x)=N(x′|x,σ²I), the conditional score is ∇_{x′} log qσ(x′|x) = −(x′−x)/σ², but Eq. (2) trains ∇_{x′}eθ toward +(x′−x)/σ². Combined with the statement in Section 4.2 that the energy sign is flipped so that higher eθ means closer to pD, the reward r=eθ would be maximized by moving away from expert transitions if Algorithm 1 were implemented literally. Appendix A.2.1 repeats the incorrect identity s(x′,σ)=∇x′eθ=(x′−x)/σ². If the released code uses the standard NCSN target (x−x′)/σ², then the equations do not describe the implemented method; if it uses Eq. (2), the reported imitation results would not follow from the stated objective. This must be corrected in both directions before the empirical claims can be traced to the algorithm.
  2. [Section 4.2, Appendix A.1, Section 6 (Limitations), Figure 5] The assumption that supp(πθG) is contained in the support of the current perturbed distribution qσk is asserted but never proved. Appendix A.1 ends with 'the annealing strategy ... and a sufficiently large σ ensure that the manifold of policy-generated samples always lies in P', but no formal bound or mechanism is given. The paper's own Limitations section and Figure 5 document that at noise-level switches the energy becomes ill-defined on part of the policy manifold and the return drops. Since the central claim is that the reward is always well-defined and informative, this gap is load-bearing. A concrete test would be to measure the fraction of policy rollouts whose state transitions fall outside the estimated support of qσk before and after each switch, or to add a support-regularization term that provably keeps the policy inside P.
  3. [Appendix B.2.3] All reported metrics (DTW pose error and SAL) are computed only on the k=20 most rewarding trajectories selected from the rollouts, and policy-evaluation metrics are recorded as means over the k=20 most rewarding environments. This selection bias can inflate the scores of both algorithms and, more importantly, can change the relative ordering when one method has heavier-tailed trajectory quality. The paper should report metrics over all rollouts (or all non-terminated episodes) and justify the top-k selection independently of the algorithms being compared.
  4. [Appendix B.2.2] For the spin-kick task, reference states are drawn from a beta distribution (β=3.0, α=1.0) instead of uniformly, and this task-specific initial-state shaping is described in the NEAR training details. If this modification was applied only to NEAR and not to the AMP baseline, it confounds the single-clip comparison in Table 1 because the initial-state distribution strongly affects the difficulty of imitation from observation. Please clarify whether AMP was run with the same reference-state distribution, or add an ablation showing NEAR with the uniform initialisation.
minor comments (5)
  1. [Section 5.2, Table 2] The sentence 'NEAR also outperforms AMP in stylised goal-conditioned tasks' is not supported by the Target Reaching & Punching row, where NEAR has a lower mean task return (3.6±2.64) than AMP (3.85±0.76), with large variance; please qualify the claim.
  2. [Algorithm 1, Section 5.1] The annealing progress formula in Algorithm 1 ('progress = eθ(A,σk)/mean energy on switching to σk − 1') is not defined precisely; the text in Section 4.2 describes tracking average return but does not specify the window size or how the initial return is estimated. The reward transformation in Section 5.1 also introduces r′ without stating how it interacts with the annealing progress signal.
  3. [Appendix A.1] The notation in the proof is confusing: qσ is described as supported in a closed manifold P, but a convolution with a Gaussian kernel is supported on all of R^d whenever pD has bounded support. Please clarify whether P is meant to be the effective support (e.g., high-probability region) rather than the mathematical support.
  4. [Section 4.1, Eq. (2)] The sum in the definition of LDSM has the index bound 'i=i' instead of 'i=1', and the notation x′ versus x is used inconsistently between Eq. (2) and the surrounding text.
  5. [Figure 5] Figure 5 lacks error bars and a precise definition of the plotted quantity ('mini-batch energy return'); it would be helpful to show the raw energy reward before the tanh transform and to indicate which noise-level switch corresponds to each drop.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the energy reward is trained offline on expert transitions and evaluated against external pose/SAL metrics; the Eq. (2) sign inconsistency is a correctness issue, not a circular reduction.

full rationale

The derivation is not circular. The energy network e_theta is trained by denoising score matching on expert state transitions only (Eq. 2) and then held fixed while the policy maximizes it as a reward (Eq. 3 and Algorithm 1); no energy parameter or annealing threshold is fit to the reported pose-error or spectral-arc-length metrics. The annealing rule uses the current energy return to move among pre-specified noise levels, which is a curriculum criterion rather than a posterior fit, and the Limitations section explicitly reports that the support-containment assumption can fail, so the paper does not redefine failure away. The self-referential character of using the learned reward to judge progress is not a circular derivation of the empirical claims. The serious issue is a sign inconsistency: Eq. (2) targets (x' - x)/sigma^2 while Sec. 4.2 flips the energy so higher energy means closer to pD, giving gradient (x - x')/sigma^2; as written this would make the reward anti-imitative. That is a correctness and reproducibility problem, not circular equivalence, so it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard assumptions about score matching and universal approximation, plus two domain assumptions that are not formally established: the containment of the policy support in the perturbed distribution support, and the out-of-distribution generalization of the learned energy. The free parameters are mostly standard RL hyperparameters, but the noise scale and the unreported annealing threshold are load-bearing for the method's success.

free parameters (5)
  • Noise scale bounds (sigma1=20, sigmaL=0.01) and number of levels L=50 = sigma1=20, sigmaL=0.01, L=50
    Dataset-dependent hyperparameters controlling the dilation of the expert distribution; the paper states they must be 'sufficiently large' or 'small' but provides no selection criteria beyond intuition (Section 4.2, Table 4).
  • Annealing threshold alpha = not reported
    Appears in Algorithm 1 as the switching condition for noise levels, but no value is listed in Table 4, so the annealing schedule cannot be exactly reproduced.
  • Reward centering window k=3 and tanh scale 10 = k=3, scale=10
    Additional reward transformation hyperparameters in Section 5.1 that affect learning dynamics; fixed heuristically.
  • Reference state sampling beta distribution for spin-kick = alpha=1.0, beta=3.0
    Task-specific adjustment introduced after observing learning difficulty (Appendix B.2.2); this is a post-hoc tuning choice.
  • Task reward weights = wtask = wenergy = 0.5
    When composing rewards, equal weighting is chosen; no sensitivity analysis is provided.
assumptions (5)
  • standard math Convolution of a probability distribution with a Gaussian kernel yields a smooth density and smooth score.
    Used in Lemma A.1 to argue q_sigma is smooth; standard result but applied to a distribution pD supported on a closed manifold.
  • standard math Neural networks of sufficient size can approximate the score function to arbitrary precision on compact domains.
    Universal approximation theorem invoked in Appendix A.1 to conclude the learned energy is smooth.
  • domain assumption The support of the policy's state-transition distribution remains inside the support of the perturbed distribution q_sigma_k throughout training.
    Since the energy network is trained only on perturbed expert samples, its output outside supp(q_sigma_k) is unconstrained; Section 4.2 and Appendix A.1 assume annealing ensures containment. This is the load-bearing assumption.
  • domain assumption Denoising score matching on finite samples yields an accurate energy function in data-sparse and off-manifold regions.
    The paper relies on score matching generalization to provide informative rewards in areas of X not densely covered by expert data; no bound is given.
  • domain assumption Matching the expert's state-transition distribution is sufficient for imitating the expert's underlying actions.
    This is the standard IfO premise (Section 2, Eq. 1) that the policy can be learned from state transitions only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation." pith.science (2026). https://pith.science/paper/WUV3W3OT

@misc{pith2026250114856,
  author       = {Pith},
  title        = {Pith review of: Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUV3W3OT}},
  note         = {Machine review of arXiv:2501.14856}
}
read the original abstract

This paper introduces a new imitation learning framework based on energy-based generative models capable of learning complex, physics-dependent, robot motion policies through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert's motion data distribution and learns smooth, and well-defined representations of the data distribution's energy function using denoising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined in the manifold of policy-generated samples. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algorithms like Adversarial Motion Priors (AMP). Our framework sidesteps the optimisation challenges of adversarial imitation learning techniques and produces results comparable to AMP in several quantitative metrics across multiple imitation settings.

Figures

Figures reproduced from arXiv: 2501.14856 by the authors.

Figure 1
Figure 1. A comparison of reward functions (probability density approximations) learnt in a 2D [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Degradation of an adversarially learnt policy (AMP) in a stylised walking imitation task. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Annealing (during RL) ensures that the agent always receives a focused and well-defined [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Snapshots of the policies trained with NEAR. Mummy-style walking and spin-kick are [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Annealing at higher noise levels causes a drop in the energy reward’s return. Interestingly, ablation studies show that the addition of a task reward reduced the overall unpredictability of NEAR at the later stages of training. A reason for this could be the reduction …
Figure 6
Figure 6. Figure 6: An illustration of a dense and sparse pD. In the case of the sparse distribution, annealing would have a lower impact as the newly initialised policy would start out receiving much higher rewards. B EXPERIMENT DETAILS B.1 TASKS The task reward and goal features for eac…
Figure 7
Figure 7. Figure 7: Discriminator variance experiments. The policy was first trained for [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Discriminator non-smoothness experiments. Plots show the mean and std. discriminator [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Perfect discriminator experiments on the walking task (multi-clip dataset with 74 motions [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 11 canonical work pages

  1. [1]

    Towards principled methods for training generative adversarial networks

    Martin Arjovsky and L´eon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862,

    arXiv

  2. [7]

    Annealed importance sampling

    11 Published as a conference paper at ICLR 2025 Radford M Neal. Annealed importance sampling. Statistics and computing, 11:125–139,

    work page 2025

  3. [8]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438,

    arXiv

  4. [19]

    The average dynamic time warping pose error is then computed as the average DTW score of allτm across all expert trajectories ˆτj with ∥ˆxi − xi∥2 as the cost function

    Then, the k = 20 most rewarding trajectories are selected to form a set of policy trajectories Dπ = {τm}k m=1 where τm = {xi}Lm i=1 and each trajectory has an arbitrary length Lm. The average dynamic time warping pose error is then computed as the average DTW score of allτm across all expert trajectories ˆτj with ∥ˆxi − xi∥2 as the cost function. To ensur...

    work page 2018

  5. [20]

    The smoothness of the character’s trajectory is an interesting metric to determine the policy’s ability to perform periodic motions in a controlled manner

    is a measure of the smoothness of a motion. The smoothness of the character’s trajectory is an interesting metric to determine the policy’s ability to perform periodic motions in a controlled manner. The underlying idea behind SAL is that smoother motions typically change slowly over time and are comprised of fewer and low-valued frequency domain componen...

    work page 2018

  6. [21]

    The agent aims to reach the goal position at the bottom right (the episode ends when the agent’s position is within some threshold of the goal)

    In this experiment, the agent is initialised randomly in a small window at the top portion of the L- shaped maze. The agent aims to reach the goal position at the bottom right (the episode ends when the agent’s position is within some threshold of the goal). Expert demonstrations were collected, so the expert’s trajectory did not reach the target directly...

    work page 2017

  7. [22]

    to train the policy. The discriminator is slightly modified by removing the sigmoid activation at the output layer and instead computing the loss on sigmoid(D()) 10 (same setup as the main experiments in this paper). Training is continued normally until some cut-off point. The cut-off point is varied across runs to obtain varying levels of intersection be...

    work page 2017

  8. [1024]

    The decoder has (1024, 512,

    neurons and maps the input to a 2048-dimensional latent space. The decoder has (1024, 512,

    work page

Show all 22 references
  1. [1983]

    On convergence and stability of gans

    Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv:1705.07215,

    arXiv

  2. [1988]

    When using the environment-supplied task reward, we set wtask = wenergy = 0.5

    and advantages are computed using generalised advantage estimation (Schulman et al., 2015). When using the environment-supplied task reward, we set wtask = wenergy = 0.5. The NCSN neural network is a fully-connected network with an auto-encoder style architecture. Here, the en...

    2015

  3. [1989]

    Diffusion imita- tion from observation

    Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, Dai-Jie Wu, and Shao-Hua Sun. Diffusion imita- tion from observation. arXiv preprint arXiv:2410.05429,

    arXiv

  4. [1999]

    Calm: Conditional adversarial latent models for directable virtual characters

    Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–9,

    2023

  5. [2008]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137,

    arXiv

  6. [2010]

    Further, we standardise samples before passing them to the network

    to improve consistency across different independent training runs. Further, we standardise samples before passing them to the network. The NCSN noise scale was defined as a geometric sequence with σ1 = 20, σL = 0 .01, and L = 50 following the advice from Song & Ermon (2020). F...

    2020

  7. [2011]

    Lemma A.1

    is smooth and well-defined in the manifold of perturbed samples. Lemma A.1. Let pD be a distribution with support contained in a closed manifold M ⊆Rd. We assume that pD is continuous in this manifold. Let qσ be a distribution supported in a closed manifold P ⊆ Rd obtained by ...

    1989

  8. [2015]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  9. [2017]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470,

    arXiv

  10. [2018]

    Recent advances in imitation learning from obser- vation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. arXiv preprint arXiv:1905.13566,

    1905 arXiv

  11. [2020]

    How to train your energy-based models

    Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288,

    arXiv

  12. [2021]

    George Cybenko

    doi: 10.1109/ MM.2021.3061394. George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of control, signals and systems, 2(4):303–314,

    2021

  13. [2023]

    Generative adversarial imitation from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158,

    arXiv

  14. [2024]

    12 Published as a conference paper at ICLR 2025 A P ROOFS & EXTENDED EXPLANATIONS A.1 P ROOF OF ENERGY FUNCTION SMOOTHNESS In this section, we prove that an energy function learnt via denoising score matching (Vincent,

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.