Success in Humanoid Reinforcement Learning under Partial Observation

Wuhao Wang; Zhiyong Chen

REVIEW 4 major objections 5 minor 21 references

Success in Humanoid Reinforcement Learning under Partial Observation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A parallel history encoder lets model-free reinforcement learning succeed at humanoid locomotion under partial observability, matching full-state performance with as little as one-third of the state.

desk verdict The empirical result looks real and novel, but the paper is not self-contained: the central history encoder is never specified, so the main causal claim cannot be checked. read the letter →

arxiv 2507.18883 v1 pith:4FB3AY7T submitted 2025-07-25 cs.AI cs.RO

Wuhao Wang , Zhiyong Chen This is my paper · ORCID

classification cs.AIcs.RO

keywords humanoidlocomotionpartialobservabilityhistoryencoderparalleltemporalencodingmodel-freereinforcementlearningstatereconstructionmassadaptationHumanoid-v4benchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that high-dimensional humanoid locomotion can be learned when the controller sees only part of the robot's state, a setting where prior reinforcement-learning methods have failed. In the Humanoid-v4 benchmark, a model-free policy using a fixed-length parallel history encoder reaches the same level of performance as a baseline with full state access, while seeing only one-third to two-thirds of the simulator's 348-dimensional state. The same policy also keeps working when body-part masses change, which the authors take as evidence that the encoder reconstructs enough context from recent observations. If true, this would make humanoid control far less dependent on complete, low-noise sensing.

What carries the argument

The carrying mechanism is the parallel history encoder: a network that processes a fixed-length sequence of past observations all at once, assigning equal weight to each time step rather than compressing history sequentially the way recurrent networks do. Integrated into a standard model-free actor-critic algorithm, this encoder gives the policy temporal context from which it can infer state information missing from the current observation. The paper's hypothesis is that this reconstruction of essential context is what makes stable locomotion possible.

What would settle it

On the same humanoid task, keep the same available dimensions but corrupt every retained observation with zero-mean noise and a one-step delay, and randomly drop readings from one joint. If the encoder-based policy's final return falls to the level of the recurrent baselines, the success depends on clean missing dimensions rather than on true reconstruction under partial observability.

Watch

Extended reading notes

Core claim

The central claim is that partial observability does not prevent successful humanoid control if the policy is given a fixed window of recent observations and a parallel encoder that treats every time step in that window equally. Removing one or two semantic blocks of the state—velocity, mass/inertia, or force—while always keeping joint positions, the encoder-based method trains stably and matches or beats a full-state baseline in several configurations; recurrent sequential baselines do not learn in the same settings. The paper also reports that training with body masses varied between 50% and 100% of default values still produces a policy that handles ±50% mass changes on every body part. It concludes that the encoder reconstructs essential latent context from recent observations, and that velocity is the most critical missing attribute while force is the least.

Load-bearing premise

The load-bearing premise is that removing whole blocks of simulator state, while always keeping joint positions, faithfully represents what partial observability means; if real sensing adds noise, delay, occlusion, or hidden quantities that positions cannot recover, the reported performance may not transfer.

Editorial extensions

If this is right

If the claim holds, humanoid controllers could be built with far fewer sensors, since the method works with 34–73% of the original state dimensions.
Sensor design should prioritize joint positions and velocity, since removing velocity hurts most and removing force hurts least.
Architecture matters more than memory length: sequential recurrent processing fails on this task, while equal-weight parallel history succeeds.
Body-mass and inertia information are not strictly needed for locomotion, so policies trained without them may transfer to robots with altered or unknown mass distributions.
Configurations that remove velocity and mass converge slowly but partially recover with more training, so the encoder's reconstruction improves with experience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the block-removal protocol leaves the retained channels clean; adding sensor noise, dropout, or delay to those channels would test whether the encoder's reconstruction survives realistic sensing imperfections.
Beyond the paper: because the window is fixed-length and parallel, the encoder has no mechanism for very long-range dependencies; tasks where a hidden quantity drifts over hundreds of steps may still require recurrent or state-space memory.
Beyond the paper: the full-state baseline is a single model-free algorithm; comparisons with other full-state methods and with a state-inference oracle would clarify how much of the gap the encoder actually closes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

For you in one breath: this is the first time I've seen stable partial-observability training on Humanoid-v4 with full-state-level returns, and the margin over RMF and ODERMF is large enough to take seriously. But the paper as written cannot support its own headline, because the thing it credits for the result—the "novel history encoder"—is never actually described. The authors say, in the paper itself, that architecture and mechanism details will be in a separate report. That is an honest admission, but it is also a load-bearing hole: with no sequence length, layer structure, positional encoding, or integration into the policy and value networks, there is no way to verify the encoder exists as claimed, no way to reproduce the curves, and no way to assign credit to the encoder versus simple history stacking, observation normalization, or hyperparameter tuning.

What is genuinely good: the one-attribute removal tables are internally consistent, the comparison against RMF and ODERMF is credible in direction and size, and the mass-variation evaluation is a useful robustness check. The citation pattern looks fine—RMF and ODERMF are real, relevant baselines, and the self-citation issue is not present. The benchmark result, if the method actually works, is a meaningful step for POMDP continuous control.

The soft spots beyond the missing encoder: the two-attribute experiments omit the baselines altogether, so we cannot see whether the recurrent methods would also partially improve with more training. The variance in the two-attribute tables is high, especially Remove VM at 3M (2978 ± 1310), which the text acknowledges with "meaningful performance" but should be weighed against the abstract's "comparable" wording. And the partial-observability model itself is just dimension deletion with joint positions always kept; no sensor noise, delay, or latent variables. That is a legitimate simplification, but it means the real-world transfer claim is weaker than the paper suggests.

My bottom line: the paper deserves a serious referee, not a desk reject in the strong sense, because the reported numbers are specific and potentially important. But a referee should be told to demand the full method, code, and baseline results before accepting anything. As submitted, I would not recommend acceptance; I would recommend a major revision that puts the architecture in the paper.

This is a paper for people working on memory and partial observability in continuous control, and for editors who want to push the community toward actually specifying model-free POMDP algorithms. If the authors provide the omitted material, this could become a useful benchmark paper.

Referee Report

4 major / 5 minor

Summary. The paper claims the first stable training of a humanoid locomotion policy under partial observability in Gymnasium Humanoid-v4, using a 'novel history encoder' that processes fixed-length observation sequences in parallel and is integrated into a model-free TD3-style algorithm. Experiments remove one or two semantic blocks from the 348-dimensional MuJoCo state vector (velocity, mass/inertia, force), with joint positions always retained, and compare against two recurrent baselines (RMF and ODERMF) and a full-state TD3 baseline. The authors report that the proposed method matches or exceeds full-state TD3 in several configurations, extends to 3 million gradient steps, and adapts to body-mass perturbations. The numerical tables are internally consistent and show large margins over the recurrent baselines in the one-attribute-removal settings, but the manuscript does not specify the encoder architecture, omits the recurrent baselines from the two-attribute experiments, reports very high variance in two-attribute results, and provides no ablation or analysis supporting the causal role of the encoder.

Significance. If the central claim is correct, this would be a useful empirical result: a model-free, memory-based method that succeeds at high-dimensional humanoid POMDP control with reduced observation dimensionality, plus a clean taxonomy of which state attributes matter. The one-attribute-removal comparisons are concrete and the observation-attribute decomposition is a good idea. However, the significance is currently limited by three factors. First, the method itself is not specified, so the result cannot be reproduced or even checked; the paper explicitly defers the architecture to a separate report. Second, the two-attribute results, which are those closest to the paper's 'one-third of states' claim, lack baseline comparisons and have large seed variance, so the 'stable training' claim is overstated. Third, the encoder's causal role is asserted rather than evidenced; no ablation, code, or analysis distinguishes the encoder from hyperparameter choices or simple history stacking. These are not cosmetic issues; they bear directly on the paper's main contribution.

major comments (4)

[Results, second paragraph] The central mechanism of the paper, the 'novel history encoder,' is never specified. The manuscript says only that 'most hyperparameters follow the original TD3 configuration' and that 'further details will be provided in a separate report' about the architecture. There is no description of the sequence length, network depth, positional encoding, normalization, or how the encoder is integrated with the actor and critic. Without this information, the comparison against RMF and ODERMF cannot be reproduced, and credit cannot be assigned to the encoder as opposed to observation preprocessing, the shared codebase, or hyperparameters. Because the abstract and title attribute the success to this encoder, this missing specification is a load-bearing gap that must be fixed in the manuscript itself.
[Table 3 and 'Removal of Two Observation Attributes'] The two-attribute results, which are the evidence for the paper's 'one-third of states' claim, are presented without the RMF and ODERMF baselines. The text says those baselines fail when one attribute is removed and therefore are not included, but this does not demonstrate their behavior in the two-attribute settings; it is possible that some two-attribute configurations are easier or harder for recurrent methods in a way that changes the comparison. As a result, the claim that the proposed method is uniformly superior under two-attribute removal is not supported. The full-state TD3 baseline is also not repeated in Table 3, making the comparison to 'full observability' less direct; the authors reference the same TD3 number, but a reader cannot see the statistic alongside the partial-observation rows.
[Table 3, Remove VM and Remove VF rows] The reported standard deviations are extremely large relative to the means: for example, Ours (1M) Remove VM has 974.6 ± 535.3 and Ours (1M) Remove VF has 1382.1 ± 730.9; the 3M Remove VM value is 2978.3 ± 1310.6. These values are consistent with runs that frequently fail and only occasionally succeed, and they undermine the paper's stated claim that 'our method remains robust across all partial observability configurations' and the abstract's phrase 'stable training.' The paper should report per-seed curves or at least confidence intervals, and the claim of stability should be qualified for configurations with high variance.
[Discussion and Abstract] The paper hypothesizes that the encoder 'reconstructs essential contextual information from recent observations,' but no evidence is offered for this mechanism. There is no latent-state probing, no ablation comparing the parallel encoder against a simple concatenation of recent observations, no control experiment varying the history length, and no analysis showing that the encoder recovers the removed velocity or mass information. The observed behavior could plausibly arise from the retained position features alone plus memory, or from the extra computation of processing a sequence. The causal claim in the abstract ('the key to this success is a novel history encoder') therefore goes beyond what the experiments demonstrate.

minor comments (5)

[Results, first paragraph] The environment name is written as 'Mujoco Humanoid-v4'; the standard name is 'MuJoCo Humanoid-v4' and the capitalization should be corrected.
[Table 4] The table reports only the average return for each mass-variation scenario, although the text says each scenario was evaluated ten times. Reporting standard deviations or individual trial values would allow the reader to assess the reliability of the mass-adaptation claim.
[Results, 'Removal of One Observation Attribute'] The paper states that all models use 'matched network capacity and consistent hyperparameters,' but it does not specify the hidden sizes, LSTM layers, or optimizer details for RMF and ODERMF. Such details are needed for a fair comparison and for replication.
[Abstract and Introduction] The claim that 'no prior work has demonstrated stable training' is a strong literature claim. The manuscript does not include a systematic search or comparison of prior partial-observability results on this exact benchmark; a more carefully qualified statement or a reference to an existing negative result would be appropriate.
[Figure captions] Figures 1 and 2 would be easier to interpret if the captions stated the number of seeds, the error-band convention, and the smoothing procedure used to plot the curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: partial-observation ablations are fixed before training and compared against external TD3; the under-specified encoder is an omitted-evidence problem, not a circular one.

full rationale

The claimed derivation chain is: define six partial-observation environments by deleting fixed components of the 348-dimensional state vector, train the proposed method plus RMF and ODERMF baselines, and compare episodic returns against TD3 with full observations. None of these steps fits a parameter to the target result, renames a known pattern as a prediction, or imports a load-bearing conclusion from the authors' own prior work. The observation configurations in Table 1 are fixed by design rather than selected based on the output, and the comparison is made against an external TD3 baseline. The mass-variation experiments are parameter perturbations applied during or after training, so the evaluation is not forced by construction. One passage does assert missing support: the Results section states, 'Further details will be provided in a separate report that discusses the architecture and mechanisms of our method.' I flag this explicitly. It is a genuine verifiability and reproducibility gap: the 'novel history encoder' is not specified, so the paper's causal claim that the encoder is 'the key to this success' cannot be independently checked from the manuscript. However, an omitted mechanism description is not a circularity — it does not show that the conclusion is equivalent to the premises by definition, nor does it rely on a self-citation chain or a fitted parameter renamed as a prediction. Therefore the overall circularity score is 0, with the noted caveat that the central mechanism is under-described rather than circularly derived.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper relies on several unstated modeling choices. It does not introduce new physical entities, and it reports no fitted constants other than undisclosed hyperparameters. The main burden falls on the adequacy of state removal as partial observability, the sufficiency of a finite history window, and the fairness of the omitted baseline comparisons.

free parameters (2)

History length = not reported
The core encoder processes a fixed-length sequence of past observations; the length is a free design choice controlling how much context is available, and it is never specified.
Encoder architecture hyperparameters = not reported
Depth, width, and any temporal or attention mechanism are not described, so the design choices that produce the reported results are undisclosed free parameters.

assumptions (5)

domain assumption Removing dimensions from the Humanoid-v4 state vector is an adequate simulation of partial observability.
Invoked in the Results section when constructing the six observation configurations; no sensor noise or latency is modeled.
domain assumption A fixed-length window of past observations is sufficient to recover the latent state.
The abstract hypothesizes that the encoder reconstructs essential contextual information from recent observations, but no evidence about window length or sufficiency is given.
domain assumption Episodic return in Humanoid-v4 is a valid measure of locomotion success.
All success claims are based on training curves and final reward in the simulator; there is no real-robot validation.
domain assumption The RMF and ODERMF baselines were implemented fairly with matched capacity.
The paper claims consistent hyperparameters and matched capacity but provides no implementation details and excludes these baselines from the two-attribute removal experiments.
domain assumption No prior work achieved stable partial-observation humanoid training in Humanoid-v4.
The first-ever claim is asserted without a systematic literature survey beyond the cited POMDP baselines.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Success in Humanoid Reinforcement Learning under Partial Observation." pith.science (2026). https://pith.science/paper/4FB3AY7T

@misc{pith2026250718883,
  author       = {Pith},
  title        = {Pith review of: Success in Humanoid Reinforcement Learning under Partial Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4FB3AY7T}},
  note         = {Machine review of arXiv:2507.18883}
}

read the original abstract

Reinforcement learning has been widely applied to robotic control, but effective policy learning under partial observability remains a major challenge, especially in high-dimensional tasks like humanoid locomotion. To date, no prior work has demonstrated stable training of humanoid policies with incomplete state information in the benchmark Gymnasium Humanoid-v4 environment. The objective in this environment is to walk forward as fast as possible without falling, with rewards provided for staying upright and moving forward, and penalties incurred for excessive actions and external contact forces. This research presents the first successful instance of learning under partial observability in this environment. The learned policy achieves performance comparable to state-of-the-art results with full state access, despite using only one-third to two-thirds of the original states. Moreover, the policy exhibits adaptability to robot properties, such as variations in body part masses. The key to this success is a novel history encoder that processes a fixed-length sequence of past observations in parallel. Integrated into a standard model-free algorithm, the encoder enables performance on par with fully observed baselines. We hypothesize that it reconstructs essential contextual information from recent observations, thereby enabling robust decision-making.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages

[1]

Cipriano, P

M. Cipriano, P. Ferrari, N. Scianca, L. Lanari, G. Oriolo, Humanoid motion generation in a world of stairs. Robotics and Autonomous Systems 168, 104495 (2023)

work page 2023
[2]

Estevez, J

D. Estevez, J. G. Victores, R. Fernandez-Fernandez, C. Balaguer, Enabling garment-agnostic laundry tasks for a Robot Household Companion. Robotics and Autonomous Systems 123, 103330 (2020)

work page 2020
[3]

I. J. da Silva, D. H. Perico, T. P. D. Homem, R. A. da Costa Bianchi, Deep Reinforcement Learning for a Humanoid Robot Soccer Player.Journal of Intelligent & Robotic Systems102 (3), 69 (2021)

work page 2021
[4]

R. P. Singh, M. Benallegue, M. Morisawa, R. Cisneros, F. Kanehiro, Learning Bipedal Walking On Planned Footsteps For Humanoid Robots, in2022 IEEE-RAS 21st International Conference on Humanoid Robots, vol. 1 (2022), pp. 686–693

work page 2022
[5]

Bellman, Dynamic Programming (Princeton University Press, Princeton, NJ) (1957), intro- duces the formalism of Markov decision processes (MDPs) and the principle of optimality

R. Bellman, Dynamic Programming (Princeton University Press, Princeton, NJ) (1957), intro- duces the formalism of Markov decision processes (MDPs) and the principle of optimality

work page 1957
[6]

K. J. ˚Astr¨om, Optimal control of Markov processes with incomplete state information.Journal of Mathematical Analysis and Applications 10 (1), 174–205 (1965)

work page 1965
[7]

Q. Liu, A. Chung, C. Szepesvari, C. Jin, When Is Partially Observable Reinforcement Learning Not Scary, in Proceedings of Thirty Fifth Conference on Learning Theory, P.-L. Loh, M. Ra- ginsky, Eds. (PMLR), vol. 178 of Proceedings of Machine Learning Research (2022), pp. 5175–5220

work page 2022
[8]

A. Wang, A. C. Li, T. Q. Klassen, R. T. Icarte, S. A. Mcilraith, Learning Belief Representations for Partially Observable Deep RL, in Proceedings of the 40th International Conference on Machine Learning , A. Krause, et al. , Eds. (PMLR), vol. 202 of Proceedings of Machine Learning Research (2023), pp. 35970–35988

work page 2023

Show all 21 references

[9]

X. Chen, Y. M. Mu, P. Luo, S. Li, J. Chen, Flow-based Recurrent Belief State Learning for POMDPs, in Proceedings of the 39th International Conference on Machine Learning , 9 K. Chaudhuri, et al., Eds. (PMLR), vol. 162 of Proceedings of Machine Learning Research (2022), pp. 3444–3468

2022
[10]

W. Wei, L. Zhang, L. Li, H. Song, J. Liang, Set-membership Belief State-based Reinforcement Learning for POMDPs, in Proceedings of the 40th International Conference on Machine Learning, A. Krause, et al. , Eds. (PMLR), vol. 202 of Proceedings of Machine Learning Research (2023...

2023
[11]

Hochreiter, J

S. Hochreiter, J. Schmidhuber, Long short-term memory.Neural Computation9 (8), 1735–1780 (1997)

1997
[12]

L. Meng, R. Gorbet, D. Kuli ´c, Memory-based Deep Reinforcement Learning for POMDPs, in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems , vol. 1 of IROS (2021), pp. 5619–5626

2021
[13]

T. Ni, B. Eysenbach, R. Salakhutdinov, Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs, inProceedings of the 39th International Conference on Machine Learning (PMLR), vol. 162 of Proceedings of Machine Learning Research (2022), pp. 16691–16723

2022
[14]

X. Zhao, D. Zhang, L. Han, T. Zhang, B. XU, ODE-based Recurrent Model-free Reinforce- ment Learning for POMDPs, in Thirty-seventh Conference on Neural Information Processing Systems, vol. 37 of NeurIPS (2023), pp. 22827–22849

2023
[15]

Arcieri, et al., POMDP Inference and Robust Solution via Deep Reinforcement Learning: An Application to Railway Optimal Maintenance

G. Arcieri, et al., POMDP Inference and Robust Solution via Deep Reinforcement Learning: An Application to Railway Optimal Maintenance. Machine Learning 113 (10), 7967–7995 (2024)

2024
[16]

Lemmel, R

J. Lemmel, R. Grosu, Real-Time Recurrent Reinforcement Learning, in Proceedings of the 39th AAAI Conference on Artificial Intelligence, vol. 39 of AAAI (2025), pp. 18189–18197

2025
[17]

A. Gu, T. Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752 (2023)

2023 arXiv
[18]

T. Dao, A. Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, inInternational Conference on Machine Learning (ICML), vol. 235 of ICML (2024), pp. 10041–10071. 10

2024
[19]

Ota, Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces (2024), arXiv:2403.19925

T. Ota, Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces (2024), arXiv:2403.19925

2024 arXiv
[20]

Rimon, T

Z. Rimon, T. Jurgenson, O. Krupnik, G. Adler, A. Tamar, MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (2024), arXiv:2403.09859

2024 arXiv
[21]

Towers, et al., Gymnasium (2023), doi:10.5281/zenodo.8127026,https://zenodo.org/ record/8127025

M. Towers, et al., Gymnasium (2023), doi:10.5281/zenodo.8127026,https://zenodo.org/ record/8127025. 11

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Cipriano, P

M. Cipriano, P. Ferrari, N. Scianca, L. Lanari, G. Oriolo, Humanoid motion generation in a world of stairs. Robotics and Autonomous Systems 168, 104495 (2023)

work page 2023

[2] [2]

Estevez, J

D. Estevez, J. G. Victores, R. Fernandez-Fernandez, C. Balaguer, Enabling garment-agnostic laundry tasks for a Robot Household Companion. Robotics and Autonomous Systems 123, 103330 (2020)

work page 2020

[3] [3]

I. J. da Silva, D. H. Perico, T. P. D. Homem, R. A. da Costa Bianchi, Deep Reinforcement Learning for a Humanoid Robot Soccer Player.Journal of Intelligent & Robotic Systems102 (3), 69 (2021)

work page 2021

[4] [4]

R. P. Singh, M. Benallegue, M. Morisawa, R. Cisneros, F. Kanehiro, Learning Bipedal Walking On Planned Footsteps For Humanoid Robots, in2022 IEEE-RAS 21st International Conference on Humanoid Robots, vol. 1 (2022), pp. 686–693

work page 2022

[5] [5]

Bellman, Dynamic Programming (Princeton University Press, Princeton, NJ) (1957), intro- duces the formalism of Markov decision processes (MDPs) and the principle of optimality

R. Bellman, Dynamic Programming (Princeton University Press, Princeton, NJ) (1957), intro- duces the formalism of Markov decision processes (MDPs) and the principle of optimality

work page 1957

[6] [6]

K. J. ˚Astr¨om, Optimal control of Markov processes with incomplete state information.Journal of Mathematical Analysis and Applications 10 (1), 174–205 (1965)

work page 1965

[7] [7]

Q. Liu, A. Chung, C. Szepesvari, C. Jin, When Is Partially Observable Reinforcement Learning Not Scary, in Proceedings of Thirty Fifth Conference on Learning Theory, P.-L. Loh, M. Ra- ginsky, Eds. (PMLR), vol. 178 of Proceedings of Machine Learning Research (2022), pp. 5175–5220

work page 2022

[8] [8]

A. Wang, A. C. Li, T. Q. Klassen, R. T. Icarte, S. A. Mcilraith, Learning Belief Representations for Partially Observable Deep RL, in Proceedings of the 40th International Conference on Machine Learning , A. Krause, et al. , Eds. (PMLR), vol. 202 of Proceedings of Machine Learning Research (2023), pp. 35970–35988

work page 2023