Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation

Mehmet Turan Yard{\i}mc{\i}

REVIEW 4 major objections 5 minor 17 references

Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that separating the value functions for locomotion and manipulation makes a humanoid robot reach targets 3.5x faster than a single shared critic.

desk verdict Honest and clear, but the 3.5x claim is confounded—the paper is a useful pilot and benchmark proposal, not a demonstrated causal result. read the letter →

arxiv 2606.11891 v2 pith:EIX5U53Q submitted 2026-06-10 cs.RO cs.LG

Mehmet Turan Yard{i}mc{i} This is my paper · ORCID

classification cs.ROcs.LG

keywords humanoidloco-manipulationdualcriticarchitectureunifiedmulti-objectivereinforcementlearningcurriculumimitationfine-tuningrewardengineeringreachingefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Working with a humanoid robot that must walk and reach with one policy, the paper tries to establish that the architecture of the value function—one unified critic that estimates the combined value of all rewards versus two separate critics with disjoint reward signals—is a primary determinant of reaching efficiency, not a minor implementational detail. In a standardized comparison, the dual-critic policy reached targets in 6.5 simulation steps versus 22.6 for the unified critic, achieved 14.3 versus 7.0 validated reaches per 1,000 steps, and reached a 65.2% validated rate versus 53.8%. The paper further finds that five anti-gaming reward mechanisms do not improve on the plain dual critic, arguing that the architectural change matters more than additional reward engineering. The authors candidly note that the two runs also differ in arm action dimensionality, curriculum schedule, and one locomotion reward weight, and are single-seed, so the result is presented as an efficiency gap between two policies rather than an isolated causal effect. If the finding survives a clean single-variable ablation, it would make critic architecture a first-class design choice in multi-objective humanoid RL and in RL fine-tuning of imitation-learned policies.

What carries the argument

The load-bearing mechanism is the dual actor-critic architecture: two independent actor-critic branches, one for locomotion and one for the arm, each with its own value function trained on a disjoint reward signal (velocity tracking and balance for locomotion; reaching distance and displacement for the arm). The unified alternative feeds a single 109-dimensional observation into one critic that estimates the combined value of both reward streams. The paper's mechanistic claim is that the unified critic's value landscape spreads locomotion-oriented gradients onto arm actions, damping their magnitude and producing slower, more conservative reaching; separating the critics isolates each objecti

What would settle it

Run a single-variable ablation in which the dual-critic and unified-critic policies use the same arm action space, the same curriculum schedule, and the same reward weights, with multiple seeds; if the unified-critic policy achieves the same or better time-to-reach and throughput as the dual-critic policy, the central claim is falsified. Alternatively, if a dual-critic policy with a 12-DoF arm action space still shows 3.5x faster reaching than its unified counterpart, the architectural explanation is supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that replacing a single value function—which receives the concatenated observation of both locomotion and manipulation objectives—with two independent value functions that consume disjoint reward signals changes the efficiency of a trained humanoid loco-manipulation policy. In a standardized evaluation, the dual-critic policy reaches a target in 6.5 simulation steps on average versus 22.6 steps for the unified-critic policy, a 3.5x improvement; it scores 14.3 versus 7.0 validated reaches per 1,000 steps; and its validated reach rate is 65.2% versus 53.8%. The paper also reports that adding five anti-gaming reward mechanisms to the dual-criti

Load-bearing premise

The causal claim rests on the assumption that the only meaningful difference between the two training runs is the critic architecture; the paper itself notes that they also differ in arm action dimensionality (12 vs 5 degrees of freedom), curriculum schedule, one locomotion reward weight, and single-seed training.

Editorial extensions

If this is right

If the central claim holds, critic architecture deserves to be listed alongside reward design and policy class in the standard design checklist for multi-objective humanoid RL.
Standard training metrics (reward value, curriculum level, cumulative reach counts) are insufficient to compare multi-objective policies; time-to-reach and validated throughput should be reported routinely.
In RL fine-tuning of imitation-learned manipulation policies, a unified critic may overwrite or suppress the pre-trained arm behavior; separating critics by objective is a candidate safeguard against catastrophic forgetting.
Anti-gaming reward mechanisms may be redundant once the architectural interference is removed, saving engineering effort.
The 3.5x speed gap, if real, translates directly to more validated reaches per unit of compute, which matters for sim-to-real and real-time deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the dual-critic result holds under a clean single-variable ablation, the same separation-by-objective principle could transfer to other multi-task RL settings—e.g., vision-guided reaching, bimanual manipulation, or concurrent navigation and interaction—where one shared value function may similarly filter gradients.
A testable extension the paper leaves implicit: measure the evolution of arm action magnitude during training under both architectures; if the unified critic's suppression starts only when locomotion reward dominates, the causal story about gradient competition can be confirmed directly.
The efficiency gap suggests a potentially cheap intervention for existing unified-critic pipelines: re-train with separated critics before investing in reward shaping or new demonstrations; the comparison says this could be worth trying even without changing the policy class.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Honest, clearly written, and the benchmark metrics are sensible—but the paper's central causal claim is overreach. The comparison between the unified and dual critics varies at least four things at once: arm action dimensionality (12 vs 5 DoF), residual versus direct action interface, curriculum schedule, and one locomotion reward weight. Each run is a single seed. The paper itself admits this in the 'Confounding factors' paragraph, yet the abstract and conclusion still call critic architecture 'a primary design choice.' That is not supported by the data.

What is genuinely useful: the standardized evaluation methodology—validated reach rate, time-to-reach, throughput—is better than training reward curves, and the observation that these metrics reveal differences invisible to curriculum level and reward is worth taking seriously. The author also ships code and checkpoints and spells out the exact ablation that would settle the causal question. That is more transparency than most.

The soft spots are structural, not minor. The claim that the 5-DoF arm is a strict subset of the 12-DoF action space does not remove the confound: a residual policy on a small action space and a direct policy on a larger one are different policy classes with different exploration properties. S6u also stalled at curriculum Level 10 while S6s reached Level 12, so the unified run had less effective training on harder levels. The arm action magnitude difference (1.22 vs 2.54) is cited as mechanistic evidence for suppression, but it is just as consistent with the action parametrization difference. And the S7 comparison cannot support claims about reward engineering versus architecture because it varies training length and adds an extra set of mechanisms.

For all that, the paper is not a waste. As a descriptive report of two trained policies, with a proposed benchmark and a clear specification of the needed ablation, it has value. But as a demonstration that critic architecture matters causally, it fails. I would want the authors to either reframe the paper as a pilot or motivating study, or run the single-variable ablation they describe. Who gets value: someone working on humanoid loco-manipulation or IL+RL fine-tuning who wants a clear example of why confounded comparisons happen in these training pipelines. I would bring it to a reading group as a case study in evaluation methodology and causal overclaiming. If it came to my desk, I would send it to peer review, but with the strong expectation that the conclusion must be reconciled with the acknowledged confounds.

Referee Report

4 major / 5 minor

Summary. The paper compares unified-critic and dual-critic architectures for a humanoid loco-manipulation policy on the Unitree G1 in NVIDIA Isaac Lab. It reports that the dual-critic run reaches targets 3.5× faster, achieves 2× higher throughput, and attains a higher validated reach rate than the unified-critic run, and that adding five anti-gaming reward mechanisms provides no further benefit. The paper concludes that critic architecture is a primary design choice in multi-objective humanoid RL, with implications for RL fine-tuning of imitation-learned policies. The authors do, however, include a caveat that the two main runs differ in curriculum schedule, arm action dimensionality, and one locomotion reward weight, and that each is a single seed.

Significance. If the causal claim were supported, the result would be significant: it would elevate critic architecture to a first-class design variable in multi-objective humanoid RL and would offer a concrete mechanism (gradient isolation) for protecting imitation-learned skills during RL fine-tuning. The paper also ships code, checkpoints, and a project page, and it explicitly states the single-variable ablation needed to establish causality. However, as presented, the headline claim is not supported by the data because the comparison is confounded and unreplicated. The mechanistic explanation—that a unified critic suppresses arm action magnitudes—is plausible but not isolated from the other differences between the compared runs.

major comments (4)

[§IV.C 'Confounding factors'; Table II] The central causal claim is not supported by the experiments. S6u and S6s differ not only in critic architecture but also in arm action dimensionality (12 vs. 5 DoF), action interface (direct 12-DoF arm actions vs. 5-DoF residual arm actions), curriculum progress (Level 10 vs. Level 12), and, according to the abstract's caveat, one locomotion reward weight; each is a single seed. The '5-DoF arm is a strict subset' argument does not remove these confounds: a 5-DoF residual policy and a 12-DoF direct policy are different policy classes with different optimization landscapes, and a smaller action space can independently improve learning efficiency. The paper's own more cautious statement—'an efficiency gap between two trained policies rather than an isolated effect of the critic'—directly contradicts the abstract and conclusion's assertion that 'critic architecture is a primary design choic
[§IV.B, Table II (S7 comparison)] The conclusion that anti-gaming reward mechanisms 'provide no additional benefit' is also confounded. S7 differs from S6s by more than the five reward mechanisms: it uses a frozen locomotion branch, a freshly initialized arm policy, 55-dim arm observations versus 52-dim, and reached only Level 7/12. Any of these differences could explain the lower validated reach rate (60.9% vs. 65.2%). The claim that 'once the architectural bottleneck is resolved, additional reward engineering is unnecessary' is therefore not established by the evidence.
[§IV.A, Table II] All reported configurations are single seeds, and the evaluation appears to use one environment, so no variance estimate is available. The headline differences—65.2% vs. 53.8% validated reach rate, 14.3 vs. 7.0 reaches per 1,000 steps—could be within run-to-run noise. Multi-seed training or, at minimum, confidence intervals over evaluation episodes are required before quantitative claims of 3.5× and 2× improvements can be accepted.
[§III.A vs. §IV.A] There is an internal inconsistency in the action-space description. §III.A states that the robot has 5 arm joints with wrist and hand joints fixed, but the unified critic's arm actor is described as outputting 12 arm+finger actions. This ambiguity matters for the 'strict subset' argument and for reproducibility. Please clarify what the 12-DoF action actually controls.

minor comments (5)

[Abstract] The abstract reproduced at the top of the submission includes the caveat 'We report this as an efficiency gap between two trained policies rather than an isolated effect of the critic,' but the full-text abstract omits this caveat and instead says 'controlled comparison' and 'primary design choice.' These versions should be reconciled.
[Fig. 1] The caption says 'The unified variant (not shown),' but the figure is described as a key architectural comparison. A small schematic of the unified input concatenation would help readers.
[§III.A] Typo: 'receivedisjoint' should be 'receive disjoint.'
[Table II] The table would benefit from confidence intervals or at least the number of evaluation episodes. Also clarify the exact meaning of 'Validated / 1K steps.'
[§IV.A] Please clarify whether the reported 17,000 steps/second is per-environment steps or total environment transitions across 4096 parallel environments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the comparison is empirical and the evaluation metric is defined independently of the architecture.

full rationale

This paper is an empirical comparison of two critic architectures; it does not derive a prediction from a fitted parameter or from a definitional identity. The headline quantities (validated reach rate, time-to-reach, throughput) are defined by an external, standardized evaluation procedure and are not used to fit the critic or any other model component. The paper's own discussion explicitly disclaims that the comparison is an isolated causal effect, listing arm action dimensionality, curriculum schedule, one locomotion reward weight, and single-seed training as confounds. That is a threat to validity, not circularity: the results are not forced by construction, and no parameter is renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The stated limitations actually undercut the causal strength of the claims, which is the opposite of circular reasoning. Consistent with the reader's take, the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on an implicit causal identification that is violated by the experiment's design: multiple variables change between the two compared policies. The hand-chosen evaluation thresholds are reasonable but uncalibrated, and the lack of seed variance means all effect sizes have no uncertainty estimate. No new entities are postulated.

free parameters (6)

Evaluation position threshold (epsilon_pos) = 0.06 m
Hand-chosen threshold defining a 'validated reach'; changes in this value would alter all reach-rate and time-to-reach numbers.
Displacement threshold (d_disp) = 0.10 m
Hand-chosen threshold in Eq. 1 requiring end-effector displacement; directly affects validated reach counts.
Timeout (t_max) = 150 steps
Hand-chosen evaluation horizon; changes affect timeout rate and time-to-reach statistics.
Locomotion reward weight = not reported
The paper states S6u and S6s differ in one locomotion reward weight; this is both a confound and an unreported free parameter.
Curriculum advancement threshold = not specified
The threshold for validated reach rate required to advance curriculum levels is not given, and it controls how far each policy trained.
Anti-gaming reward weights and constraints = not reported
The five mechanisms (minimum distance, velocity-toward-target reward, stillness penalty, etc.) involve hand-chosen weights and thresholds that are not listed.

assumptions (3)

domain assumption Single-seed PPO runs are treated as representative of each architecture.
No seeds or repeats are reported; the 3.5x gap may be within seed noise. The paper lists single-seed training as a limitation but still draws causal conclusions.
ad hoc to paper The 5-DoF arm is a strict subset of the 12-DoF action space, making critic architecture the most likely explanatory variable.
Used in 'Confounding factors' to argue the critic is the likely cause; this is an assumption, not a proof, because the two policy classes also differ in their residual/absolute action formulation.
domain assumption The standardized evaluation thresholds constitute a valid measure of reaching efficiency.
The paper defines its own validated-reach metric and uses it for all comparisons; the metric's external validity is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation." pith.science (2026). https://pith.science/paper/EIX5U53Q

@misc{pith2026260611891,
  author       = {Pith},
  title        = {Pith review of: Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIX5U53Q}},
  note         = {Machine review of arXiv:2606.11891}
}

read the original abstract

Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within a single policy. A natural design choice is whether to use a single (unified) critic that estimates the combined value of all objectives, or separate (dual) critics with disjoint reward signals. We compare the two on the Unitree G1 humanoid (23 active DoF, of which 17 are policy-controlled) in NVIDIA Isaac Lab, training loco-manipulation policies through sequential curricula that progress from stationary reaching to walking with variable-orientation targets. Under a matched compute budget, the dual-critic run reaches targets 3.5x faster (6.5 vs. 22.6 simulation steps), achieves 2x higher throughput (14.3 vs. 7.0 validated reaches per 1,000 steps), and attains a higher validated reach rate (65.2% vs. 53.8%) than the unified-critic run in a standardized evaluation. Adding five anti-gaming reward mechanisms on top of the dual critic yields no further improvement (60.9% vs. 65.2%). We report this as an efficiency gap between two trained policies rather than an isolated effect of the critic: the two runs also differ in curriculum schedule, arm action dimensionality and one locomotion reward weight, and each is a single seed. The results are nonetheless suggestive for the emerging paradigm of RL fine-tuning of imitation-learned policies, where a unified critic may suppress pre-trained arm behavior through competing locomotion gradients. We argue that critic architecture deserves explicit treatment as a design variable in multi-objective humanoid RL, and specify the single-variable ablation required to establish its causal contribution. Code, trained checkpoints and a project page are available at https://mturan33.github.io/critic-architecture-matters/

Figures

Figures reproduced from arXiv: 2606.11891 by the authors.

**Figure 2.** Dual-critic policy (S6s) during play evaluation. The Unitree G1 reaches [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 5 linked inside Pith

[1]

Learning fine-grained bimanual manipulation with low-cost hardware,

T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” inProc. RSS, 2023

2023
[2]

Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,

Z. Fu, T. Z. Zhao, and C. Finn, “Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,”arXiv preprint arXiv:2401.02117, 2024

arXiv 2024
[3]

Real-world humanoid locomotion with reinforcement learning,

I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Real-world humanoid locomotion with reinforcement learning,”Sci- ence Robotics, vol. 9, no. 89, 2024

2024
[4]

HOVER: Versatile neural whole-body controller for humanoid robots,

T. He et al., “HOVER: Versatile neural whole-body controller for humanoid robots,”arXiv preprint arXiv:2410.21229, 2024

arXiv 2024
[5]

ULC: A unified and fine-grained controller for humanoid loco-manipulation,

W. Sun, L. Feng, B. Cao, Y . Liu, Y . Jin, and Z. Xie, “ULC: A unified and fine-grained controller for humanoid loco-manipulation,”arXiv preprint arXiv:2507.06905, 2025

arXiv 2025
[6]

Deep whole-body control: Learning a unified policy for manipulation and locomotion,

Z. Fu, X. Cheng, and D. Pathak, “Deep whole-body control: Learning a unified policy for manipulation and locomotion,” inProc. CoRL, 2022

2022
[7]

Expressive whole-body control for humanoid robots,

X. Cheng, Y . Ji, J. Chen, R. Yang, G. Yang, and X. Wang, “Expressive whole-body control for humanoid robots,” inProc. RSS, 2024

2024
[8]

Curriculum learning,

Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProc. ICML, 2009

2009

Show all 17 references

[9]

Solving Rubik’s cube with a robot hand,

OpenAI et al., “Solving Rubik’s cube with a robot hand,”arXiv preprint arXiv:1910.07113, 2019

1910 arXiv
[10]

Curriculum learning for reinforcement learning domains: A framework and survey,

S. Narvekar et al., “Curriculum learning for reinforcement learning domains: A framework and survey,”JMLR, vol. 21, no. 181, pp. 1–50, 2020

2020
[11]

Concrete problems in AI safety,

D. Amodei et al., “Concrete problems in AI safety,”arXiv preprint arXiv:1606.06565, 2016

2016 arXiv
[12]

Defining and characterizing reward hacking,

J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” inProc. NeurIPS, 2022

2022
[13]

The effects of reward misspec- ification: Mapping and mitigating misaligned models,

A. Pan, K. Bhatia, and J. Steinhardt, “The effects of reward misspec- ification: Mapping and mitigating misaligned models,” inProc. ICLR, 2022

2022
[14]

Orbit: A unified simulation framework for interactive robot learning environments,

M. Mittal et al., “Orbit: A unified simulation framework for interactive robot learning environments,”IEEE Robot. Autom. Lett., vol. 8, no. 6, pp. 3740–3747, 2023

2023
[15]

Prox- imal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[16]

Teach a robot to FISH: Versatile imitation from one minute of demonstrations,

S. Haldar, J. Mathur, D. Bernstein, and L. Pinto, “Teach a robot to FISH: Versatile imitation from one minute of demonstrations,” inProc. RSS, 2023

2023
[17]

Serl: A software suite for sample-efficient robotic reinforcement learning,

J. Luo et al., “Serl: A software suite for sample-efficient robotic reinforcement learning,” inProc. ICRA, 2024

2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Learning fine-grained bimanual manipulation with low-cost hardware,

T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” inProc. RSS, 2023

2023

[2] [2]

Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,

Z. Fu, T. Z. Zhao, and C. Finn, “Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,”arXiv preprint arXiv:2401.02117, 2024

arXiv 2024

[3] [3]

Real-world humanoid locomotion with reinforcement learning,

I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Real-world humanoid locomotion with reinforcement learning,”Sci- ence Robotics, vol. 9, no. 89, 2024

2024

[4] [4]

HOVER: Versatile neural whole-body controller for humanoid robots,

T. He et al., “HOVER: Versatile neural whole-body controller for humanoid robots,”arXiv preprint arXiv:2410.21229, 2024

arXiv 2024

[5] [5]

ULC: A unified and fine-grained controller for humanoid loco-manipulation,

W. Sun, L. Feng, B. Cao, Y . Liu, Y . Jin, and Z. Xie, “ULC: A unified and fine-grained controller for humanoid loco-manipulation,”arXiv preprint arXiv:2507.06905, 2025

arXiv 2025

[6] [6]

Deep whole-body control: Learning a unified policy for manipulation and locomotion,

Z. Fu, X. Cheng, and D. Pathak, “Deep whole-body control: Learning a unified policy for manipulation and locomotion,” inProc. CoRL, 2022

2022

[7] [7]

Expressive whole-body control for humanoid robots,

X. Cheng, Y . Ji, J. Chen, R. Yang, G. Yang, and X. Wang, “Expressive whole-body control for humanoid robots,” inProc. RSS, 2024

2024

[8] [8]

Curriculum learning,

Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProc. ICML, 2009

2009