REVIEW 5 major objections 4 minor 12 references
Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement
T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a single state-value function, learned from action-free expert observations and the agent's own imperfect data, can carry imitation learning to the level of a reward-trained oracle, without reward engineering or…
desk verdict A solid empirical paper on imitation from observations: VfO's state-value trick works reasonably well and SIBench is the genuine new contribution, but the theory is heuristic and the headline claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the state-value function $v(s)$ trained on the mixed expert/background dataset with binary rewards, whose fixed point is $v_{\bar{\pi}}(s_t) = \mathbb{E}_{(s_{t+1}, s_{t+2}, \dots) \sim \bar{\pi}} \sum_{i \ge 0} \gamma^i p(z_{t+i} = E \mid s_{t+i})$ — the cumulative discounted likelihood that future states are expert-visited. Because a state-value function can be evaluated without knowing the action, it is the one object that lets action-free expert data speak to action-labeled background data; a Q-function cannot, since background transitions carry no reward signal. Policy improvement then uses exponentiated advantage-weighted regression over background transitions, $L_\pi = -\mathbb{E}_{(s, a) \in D_B}[\exp((\gamma v(s') + r - v(s))/\lambda) \log \pi(a|s)]$, which up-weights actions whose resulting states score higher in expert-visit likelihood. A learned discriminator can replace the binary reward (VfO-disc), but the binary version (VfO-bin) is the paper's primary demonstration that no reward model is needed at all.
What would settle it
Collect a background dataset whose reachable states have zero overlap with the expert's state distribution (for example, data from a policy confined to a region of the state space the demonstrations never enter) and train VfO-bin on it: the Eq. 3 and Eq. 4 account predicts the advantage signal vanishes and policy return does not improve, while a claim that VfO imitates from the value function alone would predict continued improvement.
Extended reading notes
Core claim
On its own terms, the paper's central finding is that a state-value function can carry the entire imitation-from-observations burden: learn $v(s)$ by temporal-difference regression on a mixture of action-free expert trajectories (reward 1) and action-labeled background trajectories (reward 0), then improve the policy by advantage-weighted regression over background transitions. The value function's fixed point is the cumulative discounted probability that future states were visited by the expert within the mixed data (Eq. 4), so reweighting background actions by their advantage pushes the policy toward expert states without ever seeing an expert action or a reward. The paper reports that on the self-improvement benchmark and in the full iterative loop, VfO-bin performs similarly to AWR trained with ground-truth rewards across nearly all experiments, and improves on the data even when the background comes from policies far below expert level. It further reports that the dual and value-based baselines that dominate bimodal benchmarks (SMODICE, DILO) largely fail on the graded-quality data distribution, which the paper reads as evidence that prior evaluation regimes measured filtering rather than self-improvement.
Load-bearing premise
The method's entire learning signal is advantage-weighted regression over background transitions, so the background data must contain actions that actually lead toward expert-visited states; equally load-bearing, the paper offers no formal guarantee that maximizing the cumulative expert-visit likelihood in Eq. 4 yields expert-like behavior, treating that identification as an intuition.
Editorial extensions
If this is right
- An agent can bootstrap to near-oracle skill from its own low-quality rollouts: in the 20-iteration self-improvement loop, VfO-bin ends with performance similar to the AWR oracle and ahead of BC, BCO, SMODICE, and DILO.
- The offline SIBench proxy predicts online self-improvement: in every case where SIBench showed improvement, the full iterative loop improved too, and Hopper's plateau matched SIBench's zero-crossing between background returns of 1,500 and 2,000.
- The advantage of VfO is specific to graded-quality data: on bimodal expert/random mixtures the method underperforms behavior cloning, so algorithm rankings flip depending on which data regime is used for evaluation.
- VfO works from image observations on the Robomimic Lift task, where both VfO-bin and the AWR oracle improve, indicating the value-transfer mechanism survives high-dimensional inputs.
- Dual and value-based IfO baselines (SMODICE, DILO) largely fail on SIBench's graded-quality data, which the paper attributes to weak Bellman-residual signal when good and bad trajectories overlap heavily.
Reading between the lines
- A corollary the paper leaves implicit: VfO-bin's improvement should track the reachability overlap between the background policy's action coverage and the expert's state distribution, so measuring that overlap could predict in advance which tasks VfO will improve and which it will fail on.
- Because VfO-bin needs no discriminator, no reward model, and no expert actions, it is a natural drop-in for pipelines whose only task signal is raw human video plus the robot's own exploration; the paper motivates this direction but explicitly defers large-scale and cross-embodiment experiments to future work.
- The Eq. 4 reading of the value as a discounted expert-visit probability suggests VfO-bin is implicitly estimating a state-occupancy ratio between expert and mixed data, and a formal treatment of that link could yield convergence guarantees the paper does not attempt.
- The same value-transfer trick could plausibly reweight unlabeled corpora in other domains wherever a few 'good' examples stand in for the expert demonstrations, but the paper presents no evidence for that extension beyond the control setting it studies.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies imitation learning from observations (IfO), where an agent has access to action-free expert demonstrations plus a background dataset that contains actions but no rewards, and is expected to learn a policy that imitates the expert. The proposed method, VfO, trains a state-value function on a mixture of expert and background transitions using either binary rewards (1 on expert states, 0 on background states) or discriminator-based rewards, and then updates the policy by exponentiated advantage-weighted regression on the background transitions. The paper also introduces SIBench, an offline benchmark constructed from D4RL and Robomimic data generated by behavior-cloned policies of varying quality, and reports experiments on state-based and image-based tasks, on bimodal data mixtures, and in an iterative self-improvement loop. The main claims are that VfO is competitive with AWR trained on ground-truth rewards on SIBench, that SIBench correlates with iterative self-improvement, and that VfO can bootstrap self-improvement from low-quality self-collected data.
Significance. If the central claims hold, the paper makes a useful contribution: a simple, non-adversarial IfO method that uses action-free expert data plus unlabeled background data without reward engineering, and a benchmark that better reflects self-improvement data distributions than the standard bimodal mixtures used in prior work. The experimental coverage is broad (four D4RL domains, three Robomimic tasks, state and image observations, offline and iterative settings, and ablations over the temperature and mixture ratio), and the paper is transparent about baseline re-implementations and limitations. The main weaknesses are that the policy-improvement mechanism behind Eq. (3) is not formally grounded, and the headline conclusions overstate the breadth and strength of the evidence. With those issues addressed, the empirical findings would be a solid step toward scalable IfO.
major comments (5)
- [Section 3.2, Eq. (3)] The policy update in Eq. (3) is described as exponentiated advantage-weighted regression and therefore inherits AWR's policy-improvement rationale, but the value function v in Eq. (2) is trained on the mixture of expert and background data and approximates the value of the virtual policy π̄ in Eq. (1), not the value of the background policy π_B that generated the actions used in the regression. AWR's derivation assumes the advantage is computed with respect to the behavior policy whose actions are being weighted, so the weights in Eq. (3) are not the advantages of π_B and no policy-improvement guarantee applies. The strong SIBench results are thus not explained by the stated mechanism. Please either provide a formal justification (for example, conditions under which the fixed point of Eq. (3) improves the expert-return objective, even in a simplified tabular or linear setting) or explicitly characterize Eq. (3) as a heuristic and add experiments that probe the failure mode where background actions cannot reach expert-visited states.
- [Section 6 vs. Sections 4.4 and 4.5] The conclusion states that 'across nearly all experiments in our analysis, VfO is competitive with RL from ground-truth rewards when using just a few action-free trajectories to define desired behavior.' This is inconsistent with the bimodal results in Figure 4, where both VfO variants underperform BC and are clearly worse than DILO and SMODICE, and with the image-based SIBench results in Figure 5, where only Lift shows meaningful improvement. Moreover, the expert dataset for D4RL consists of all 1000 demonstrations (Section 4.1), so 'a few action-free trajectories' is inaccurate. The claim should be narrowed to the state-based SIBench setting, with the Hopper and Robomimic caveats stated, and the demonstration count should be reported precisely.
- [Section 4.6] The self-improvement experiments are seeded with 'bad but non-zero performance to avoid regions with low signal-to-noise ratios' (Section 4.6), and the Robomimic self-improvement results are described as mixed ('VfO-bin is good on two out of three tasks'). The conclusion (Section 6) nevertheless claims 'bootstrapping imitation learning from observations to mastery via self-collection starting from low quality data' as a demonstrated result. Starting from a known non-zero-quality seed does not demonstrate bootstrapping from low-quality data in the sense implied, and the mixed Robomimic results should be reflected in the conclusion. Please qualify the claim and clarify what 'low quality' means in this experiment.
- [Sections 4.1 and Appendix B] SIBench is introduced as a new benchmark contribution, but no datasets or code are released, and the two main baseline comparisons (SMODICE and DILO) are re-implemented with documented deviations (Appendix B: no entropy regularization and no gradient penalty for SMODICE; no orthogonal-gradient update and different action discretization for DILO). Since some of these deviations are known to affect baseline behavior, and since the benchmark data cannot be downloaded, the empirical comparisons and the proposed benchmark are difficult to reproduce or build on. Please release the benchmark construction code and datasets (or, at minimum, a precise generator specification with all policy training details), and state whether the baseline deviations affect the reported rankings.
- [Section 3.2, Eqs. (1)-(2)] The definition of the virtual policy π̄ is internally ambiguous. Eq. (1) defines a state-conditioned mixture of π_E and π_B with state-dependent mixing weight p(z|s,α), while the text says this is equivalent to deciding at the beginning of an episode whether to follow the expert or background policy. These two descriptions are not the same process, and the TD data in Eq. (2), sampled as (1−α) from D_E and α from D_B, corresponds to an episode-level mixture of the two data-generating policies rather than the state-conditioned policy in Eq. (1). Consequently, the interpretation in Eq. (4) of v as the cumulative discounted likelihood of expert visitation 'within the mixed dataset' is not backed by a well-defined policy-evaluation target. Please clarify the sampling process and state precisely which policy, if any, the learned value function evaluates.
minor comments (4)
- [Section 4.6] There is a typo: 'Overal' should be 'Overall'.
- [Figures 2-5 and 8-9] The paper reports mean and standard deviation across 5 seeds (Section 4.1), but the figures do not show error bars or confidence regions. Adding them would make the differences between algorithms easier to assess.
- [Algorithm 1] The mixture parameter α appears only in the value loss, not in the policy loss; the pseudocode would be clearer if the text stated explicitly that this is intentional, since a reader may otherwise expect α to also weight the policy update.
- [Appendix D, Table 1] The term 'discount ratio' for γ is nonstandard; 'discount factor' is the usual terminology.
Circularity Check
No significant circularity: VfO's derivation (Eqs. 1-3) is self-contained, its evaluation uses held-out returns with documented failure cases, and self-citations are non-load-bearing.
full rationale
The derivation chain is self-contained. The virtual mixture policy (Eq. 1) defines the policy whose value is evaluated; the value loss (Eq. 2) is standard TD on the mixed data under a fixed reward (1 for expert states, 0 for background, or a discriminator output); the policy loss (Eq. 3) is AWR-style advantage-weighted regression on background actions. No parameter is fitted and then re-reported as a prediction: the reward is a fixed definition, the value function is fitted to that reward, and the policy is scored by held-out environment returns, which are not the training objective and are not forced by construction. Eq. 4 is a mathematical identity following from the binary reward definition and is presented as an interpretation ('the learned value can be interpreted as...'), not as an empirical prediction; the subsequent remark that maximizing it 'prioritises visiting expert states' is a mechanism claim that is empirically validated and partially falsified (VfO-bin underperforms on bimodal data in Fig. 4 and on Hopper in Fig. 2), which would be impossible if results were tautological. The comparison with the AWR oracle is meaningful because the oracle consumes ground-truth rewards VfO never sees, and the observed similarity fails in several settings. Self-citations (Springenberg et al. 2024 for the multi-scale encoder and the claim that offline actor-critic RL scales; Wulfmeier et al. 2024 for the IQlearn equivalence) are present but non-load-bearing: the encoder is an implementation detail reproduced by the paper's own ablations, and the IQlearn remark concerns a related method, not VfO's derivation. Acknowledged limitations (same-embodiment only; 'bad but non-zero performance' seeding in Sec. 4.6; no large-scale experiments yet; VfO's weakness on bimodal data) are honest scope statements that narrow, rather than conceal, the central claim. The main residual concern is a correctness risk, not circularity: advantages in Eq. 3 are computed against the virtual mixture policy rather than the behavior policy, so the AWR policy-improvement guarantee is not formally inherited; this is an unproven assumption, not a definitional equivalence. Score 1 reflects several self-citations that never bear the load of the central claim.
Assumptions & free parameters
free parameters (2)
- temperature lambda =
1.0 (D4RL), 0.1 (Robomimic)
- mixture ratio alpha =
0.5
assumptions (5)
- ad hoc to paper The mixed data can be treated as coming from a virtual policy pi-bar that mixes expert and background policies (Eq. 1).
- domain assumption The background dataset provides sufficient state-action coverage near expert states for advantage-weighted regression to extract expert-like actions.
- ad hoc to paper A state value trained with reward 1 on expert states and 0 elsewhere is a useful objective for imitation (Eq. 4 interpretation).
- standard math Standard TD learning and AWR convergence assumptions for offline RL.
- domain assumption Same embodiment and dynamics for expert and background data.
Cite this review
Pith. "Pith review of Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement." pith.science (2026). https://pith.science/paper/X55KYRKL
@misc{pith2026250706701,
author = {Pith},
title = {Pith review of: Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement},
year = {2026},
howpublished = {\url{https://pith.science/paper/X55KYRKL}},
note = {Machine review of arXiv:2507.06701}
}
read the original abstract
Imitation Learning from Observation (IfO) offers a powerful way to learn behaviors at large-scale: Unlike behavior cloning or offline reinforcement learning, IfO can leverage action-free demonstrations and thus circumvents the need for costly action-labeled demonstrations or reward functions. However, current IfO research focuses on idealized scenarios with mostly bimodal-quality data distributions, restricting the meaningfulness of the results. In contrast, this paper investigates more nuanced distributions and introduces a method to learn from such data, moving closer to a paradigm in which imitation learning can be performed iteratively via self-improvement. Our method adapts RL-based imitation learning to action-free demonstrations, using a value function to transfer information between expert and non-expert data. Through comprehensive evaluation, we delineate the relation between different data distributions and the applicability of algorithms and highlight the limitations of established methods. Our findings provide valuable insights for developing more robust and practical IfO techniques on a path to scalable behaviour learning.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[2]
URL https://openreview.net/forum? id=TtYSU29zgR. Davchev, T., Sushkov, O. O., Regli, J.-B., Schaal, S., Ay- tar, Y ., Wulfmeier, M., and Scholz, J. Wish you were here: Hindsight goal selection for long-horizon dexterous manipulation. ArXiv, abs/2112.00597, 2021. Di Palo, N., Byravan, A., Hasenclever, L., Wulfmeier, M., Heess, N., and Riedmiller, M. Toward...
arXiv 2021
-
[6]
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S
URL https://openreview.net/forum? id=S1xKd24twB. Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y ., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022. Riedmiller, M., Springenberg, J. T., Hafner, R., and Heess, N. Collect & infer-a fresh look at data-...
arXiv 2022
-
[8]
URL https://openreview.net/forum? id=4iEoOIQ7nL. Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning to reinforcement learn, 2016. Wang, Q., Xiong, J., Han, L., sun, p., Liu, H., and Zhang, T. Exponentially weighted imitation learning for batched historical data. In Bengio,...
work page 2016
-
[11]
URL https://proceedings.mlr.press/ v229/zitkovich23a.html. Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, Z., Aytar, Y ., Denil, M., de Freitas, N., and Reed, S. Offline learning from demonstrations and unlabeled experience, 2020. URL https://arxiv.org/abs/ 2011.13885. 13 Value from Observations A. Implementation Details The policies are sim...
arXiv 2024
-
[12]
and model-free RL approaches (Lampe et al., 2023; Bousmalis et al., 2024; Springenberg et al., 2024). While many of these works rely on externally defined reward functions, related signals, or vision-language models as reward sources (Ma et al., 2024), ours directly uses demonstration data to define optimal behaviour (Abbeel & Ng, 2004). Self-improvement ...
work page 2023
-
[2014]
cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper. pdf. Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. Advances in neural information processing systems, 29, 2016. Hong, Z.-W., Kumar, A., Karnik, S., Bhandwaldar, A., Sri- vastava, A., Pajarinen, J., Laroche, R., Gupta, A., and Agrawa...
arXiv 2014
-
[2018]
cc/paper_files/paper/2018/file/ 4aec1b3435c52abbdf8334ea0e7141e0-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 4aec1b3435c52abbdf8334ea0e7141e0-Paper. pdf. Watson, J., Huang, S., and Heess, N. Coherent soft imitation learning. Advances in Neural Information Processing Systems, 36, 2024. Wulfmeier, M., Ondruska, P., and Posner, I. Maximum en- tropy deep inverse reinforcement learning. arXiv preprint a...
arXiv 2018
-
[2020]
URL https://openreview.net/forum? id=Hyg-JC4FDr. Kwon, T., Palo, N. D., and Johns, E. Language models as zero-shot trajectory generators. IEEE Robotics and Automation Letters, 9(7):6728–6735, 2024. doi: 10.1109/ LRA.2024.3410155. Lampe, T., Abdolmaleki, A., Bechtle, S., Huang, S., Sprin- genberg, J. T., Bloesch, M., Groth, O., Hafner, R., Her- tweck, T., ...
arXiv 2024
Show all 12 references
-
[2021]
Chebotar, Y ., Hausman, K., Lu, Y ., Xiao, T., Kalashnikov, D., Varley, J., Irpan, A., Eysenbach, B., Julian, R., Finn, C., et al
URL https://openreview.net/forum? id=7PkfLkyLMRM. Chebotar, Y ., Hausman, K., Lu, Y ., Xiao, T., Kalashnikov, D., Varley, J., Irpan, A., Eysenbach, B., Julian, R., Finn, C., et al. Actionable models: Unsupervised offline re- inforcement learning of robotic skills. arXiv prepri...
2021 arXiv
-
[2022]
Kostrikov, I., Agrawal, K
URL https://openreview.net/forum? id=BrPdX1bDZkQ. Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, J. Discriminator-actor-critic: Addressing sam- ple inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Rep...
1912 arXiv
-
[2023]
org/CorpusID:265609417
URL https://api.semanticscholar. org/CorpusID:265609417. Wulfmeier, M., Bloesch, M., Vieillard, N., Ahuja, A., Born- schein, J., Huang, S., Sokolov, A., Barnes, M., Desjardins, G., Bewley, A., Bechtle, S. M. E., Springenberg, J. T., Momchev, N., Bachem, O., Geist, M., and Ried...
2024 arXiv
-
[2024]
org/CorpusID:269605913
URL https://api.semanticscholar. org/CorpusID:269605913. Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109. ...
2012
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.