{"id":"0d25ded3-cc66-4d3f-afb8-1af357f0f77a","arxiv_id":"2606.11891","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"In a confounded single-seed comparison, a dual-critic design reached targets 3.5x faster than a unified critic, but the causal role of critic architecture was not isolated.","lead":"This paper compares two ways of structuring the learning critic in a simulated humanoid robot that walks and reaches, and finds one variant reaches targets much faster. The comparison is not clean—the two setups differ in several ways—so the headline conclusion is not yet proven.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim that critic architecture drives the 3.5x/2x gains is undercut by confounded comparison: S6u vs S6s differ in arm action dimensionality (12 vs 5 DoF), curriculum schedule, one reward weight, and residual-vs-direct action interface; single seed.","rationale":"The reader's weakest assumption is the same as the load-bearing concern I identified: the causal attribution of the efficiency gains to critic architecture is confounded by action dimensionality, curriculum schedule, reward weight, and single-seed training. The paper's own limitations section and 'Confounding factors' paragraph acknowledge these issues, but the abstract and conclusion still present the finding as 'critic architecture is a primary design choice.' This overreach is the core problem. I agree with the reader's REJECT verdict: the descriptive results may be a useful benchmark, but the paper does not support the causal claim. My proposed concrete test is a direct single-variable ablation that would settle whether the critic architecture alone explains the 3.5x speed and 2x throughput differences. Since my analysis corroborates the reader's verdict, no change to the verdict is needed; the verdict should remain REJECT. The paper would need rerunning with the proposed control to be considered for acceptance as a causal study.","tokens_in":5952,"tokens_out":3953,"duration_ms":42346,"concrete_test":"Run a 2x2 ablation grid in the same Isaac Lab codebase: (1) unified critic with 5-DoF residual arm actions, (2) unified critic with 12-DoF direct arm actions, (3) dual critic with 5-DoF residual actions (replicate S6s), (4) dual critic with 12-DoF direct actions. Keep the curriculum schedule, reward weights, and all hyperparameters identical across all four conditions, and train each with at least three random seeds. Then evaluate under the standardized benchmark (3,000 steps standing and walking, same thresholds). If condition (1) matches the fast S6s results while condition (2) remains slow, the critic is not the cause; if all unified variants are slow and all dual variants are fast regardless of action dimensionality, the critic claim is supported. At minimum, train a unified critic with exactly the 5-DoF residual action space, same curriculum and reward weights as S6s, and compare to","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline conclusion—that dual critics outperform a unified critic because of critic architecture—rests on treating S6u vs S6s as a controlled comparison. The authors themselves list in 'Confounding factors' that the runs differ in arm action dimensionality (12 vs. 5 DoF), curriculum schedule, and one locomotion reward weight. Their argument that '5-DoF arm is a strict subset of the 12-DoF action space' does not remove the confound: S6s uses 5-DoF arm residual actions while S6u uses 12-DoF arm+finger actions, so the action interface (residual vs. direct) and action-space size differ. A smaller action space can independently improve sampling efficiency by pruning irrelevant DoF, and the residual formulation changes the optimization landscape. Furthermore, S6u only reached curriculum Level 10, while S6s reached Level 12, so S6u may have had fewer effective training iterations on harder levels; the efficiency gap might reflect curriculum progress rather than critic architecture. Single-seed training provides no variance estimate, so the 65.2% vs. 53.8% validated reach rate could be within seed noise. The paper itself acknowledges it is reporting 'an efficiency gap between two trained policies rather than an isolated effect of the critic,' yet the abstract and conclusion still assert that 'critic architecture is a primary design choice.' That overstates the causal claim. The suggested mechanistic explanation (unified critic suppresses arm actions) is plausible, but it is not identified from the confounded comparison. The S7 vs. S6s comparison also varies training length (Level 7 vs. 12) and adds anti-gaming rewards, so it cannot support conclusions about reward engineering versus architecture. Thus the load-bearing assumption—that critic architecture is the sole or primary difference—is not satisfied.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares unified-critic and dual-critic architectures for a humanoid loco-manipulation policy on the Unitree G1 in NVIDIA Isaac Lab. It reports that the dual-critic run reaches targets 3.5× faster, achieves 2× higher throughput, and attains a higher validated reach rate than the unified-critic run, and that adding five anti-gaming reward mechanisms provides no further benefit. The paper concludes that critic architecture is a primary design choice in multi-objective humanoid RL, with implications for RL fine-tuning of imitation-learned policies. The authors do, however, include a caveat that the two main runs differ in curriculum schedule, arm action dimensionality, and one locomotion reward weight, and that each is a single seed.","tokens_in":6391,"tokens_out":6849,"duration_ms":73357,"significance":"If the causal claim were supported, the result would be significant: it would elevate critic architecture to a first-class design variable in multi-objective humanoid RL and would offer a concrete mechanism (gradient isolation) for protecting imitation-learned skills during RL fine-tuning. The paper also ships code, checkpoints, and a project page, and it explicitly states the single-variable ablation needed to establish causality. However, as presented, the headline claim is not supported by the data because the comparison is confounded and unreplicated. The mechanistic explanation—that a unified critic suppresses arm action magnitudes—is plausible but not isolated from the other differences between the compared runs.","major_comments":[{"comment":"The central causal claim is not supported by the experiments. S6u and S6s differ not only in critic architecture but also in arm action dimensionality (12 vs. 5 DoF), action interface (direct 12-DoF arm actions vs. 5-DoF residual arm actions), curriculum progress (Level 10 vs. Level 12), and, according to the abstract's caveat, one locomotion reward weight; each is a single seed. The '5-DoF arm is a strict subset' argument does not remove these confounds: a 5-DoF residual policy and a 12-DoF direct policy are different policy classes with different optimization landscapes, and a smaller action space can independently improve learning efficiency. The paper's own more cautious statement—'an efficiency gap between two trained policies rather than an isolated effect of the critic'—directly contradicts the abstract and conclusion's assertion that 'critic architecture is a primary design choic","section":"§IV.C 'Confounding factors'; Table II"},{"comment":"The conclusion that anti-gaming reward mechanisms 'provide no additional benefit' is also confounded. S7 differs from S6s by more than the five reward mechanisms: it uses a frozen locomotion branch, a freshly initialized arm policy, 55-dim arm observations versus 52-dim, and reached only Level 7/12. Any of these differences could explain the lower validated reach rate (60.9% vs. 65.2%). The claim that 'once the architectural bottleneck is resolved, additional reward engineering is unnecessary' is therefore not established by the evidence.","section":"§IV.B, Table II (S7 comparison)"},{"comment":"All reported configurations are single seeds, and the evaluation appears to use one environment, so no variance estimate is available. The headline differences—65.2% vs. 53.8% validated reach rate, 14.3 vs. 7.0 reaches per 1,000 steps—could be within run-to-run noise. Multi-seed training or, at minimum, confidence intervals over evaluation episodes are required before quantitative claims of 3.5× and 2× improvements can be accepted.","section":"§IV.A, Table II"},{"comment":"There is an internal inconsistency in the action-space description. §III.A states that the robot has 5 arm joints with wrist and hand joints fixed, but the unified critic's arm actor is described as outputting 12 arm+finger actions. This ambiguity matters for the 'strict subset' argument and for reproducibility. Please clarify what the 12-DoF action actually controls.","section":"§III.A vs. §IV.A"}],"minor_comments":[{"comment":"The abstract reproduced at the top of the submission includes the caveat 'We report this as an efficiency gap between two trained policies rather than an isolated effect of the critic,' but the full-text abstract omits this caveat and instead says 'controlled comparison' and 'primary design choice.' These versions should be reconciled.","section":"Abstract"},{"comment":"The caption says 'The unified variant (not shown),' but the figure is described as a key architectural comparison. A small schematic of the unified input concatenation would help readers.","section":"Fig. 1"},{"comment":"Typo: 'receivedisjoint' should be 'receive disjoint.'","section":"§III.A"},{"comment":"The table would benefit from confidence intervals or at least the number of evaluation episodes. Also clarify the exact meaning of 'Validated / 1K steps.'","section":"Table II"},{"comment":"Please clarify whether the reported 17,000 steps/second is per-environment steps or total environment transitions across 4096 parallel environments.","section":"§IV.A"}],"recommendation":"reject","confidential_remarks":"The paper is transparent about its limitations and provides code and checkpoints, which is commendable. However, the central claim is not supported by the current experiments, and the abstract/conclusion overstate the evidence. The authors outline the necessary single-variable ablation; a resubmission with that controlled comparison and multi-seed results could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest, clearly written, and the benchmark metrics are sensible—but the paper's central causal claim is overreach. The comparison between the unified and dual critics varies at least four things at once: arm action dimensionality (12 vs 5 DoF), residual versus direct action interface, curriculum schedule, and one locomotion reward weight. Each run is a single seed. The paper itself admits this in the 'Confounding factors' paragraph, yet the abstract and conclusion still call critic architecture 'a primary design choice.' That is not supported by the data.\n\nWhat is genuinely useful: the standardized evaluation methodology—validated reach rate, time-to-reach, throughput—is better than training reward curves, and the observation that these metrics reveal differences invisible to curriculum level and reward is worth taking seriously. The author also ships code and checkpoints and spells out the exact ablation that would settle the causal question. That is more transparency than most.\n\nThe soft spots are structural, not minor. The claim that the 5-DoF arm is a strict subset of the 12-DoF action space does not remove the confound: a residual policy on a small action space and a direct policy on a larger one are different policy classes with different exploration properties. S6u also stalled at curriculum Level 10 while S6s reached Level 12, so the unified run had less effective training on harder levels. The arm action magnitude difference (1.22 vs 2.54) is cited as mechanistic evidence for suppression, but it is just as consistent with the action parametrization difference. And the S7 comparison cannot support claims about reward engineering versus architecture because it varies training length and adds an extra set of mechanisms.\n\nFor all that, the paper is not a waste. As a descriptive report of two trained policies, with a proposed benchmark and a clear specification of the needed ablation, it has value. But as a demonstration that critic architecture matters causally, it fails. I would want the authors to either reframe the paper as a pilot or motivating study, or run the single-variable ablation they describe. Who gets value: someone working on humanoid loco-manipulation or IL+RL fine-tuning who wants a clear example of why confounded comparisons happen in these training pipelines. I would bring it to a reading group as a case study in evaluation methodology and causal overclaiming. If it came to my desk, I would send it to peer review, but with the strong expectation that the conclusion must be reconciled with the acknowledged confounds.","headline":"Honest and clear, but the 3.5x claim is confounded—the paper is a useful pilot and benchmark proposal, not a demonstrated causal result.","tokens_in":6872,"tokens_out":4223,"would_cite":false,"duration_ms":42845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that separating the value functions for locomotion and manipulation makes a humanoid robot reach targets 3.5x faster than a single shared critic.","keywords":["humanoid loco-manipulation","dual critic architecture","unified critic","multi-objective reinforcement learning","curriculum learning","imitation learning fine-tuning","reward engineering","reaching efficiency"],"falsifier":"Run a single-variable ablation in which the dual-critic and unified-critic policies use the same arm action space, the same curriculum schedule, and the same reward weights, with multiple seeds; if the unified-critic policy achieves the same or better time-to-reach and throughput as the dual-critic policy, the central claim is falsified. Alternatively, if a dual-critic policy with a 12-DoF arm action space still shows 3.5x faster reaching than its unified counterpart, the architectural explanation is supported.","tokens_in":5866,"feed_emoji":"🤖","tokens_out":5598,"duration_ms":54133,"temperature":0.7,"pith_summary":"Working with a humanoid robot that must walk and reach with one policy, the paper tries to establish that the architecture of the value function—one unified critic that estimates the combined value of all rewards versus two separate critics with disjoint reward signals—is a primary determinant of reaching efficiency, not a minor implementational detail. In a standardized comparison, the dual-critic policy reached targets in 6.5 simulation steps versus 22.6 for the unified critic, achieved 14.3 versus 7.0 validated reaches per 1,000 steps, and reached a 65.2% validated rate versus 53.8%. The paper further finds that five anti-gaming reward mechanisms do not improve on the plain dual critic, arguing that the architectural change matters more than additional reward engineering. The authors candidly note that the two runs also differ in arm action dimensionality, curriculum schedule, and one locomotion reward weight, and are single-seed, so the result is presented as an efficiency gap between two policies rather than an isolated causal effect. If the finding survives a clean single-variable ablation, it would make critic architecture a first-class design choice in multi-objective humanoid RL and in RL fine-tuning of imitation-learned policies.","feed_headline":"Dual critics make humanoids reach targets 3.5x faster","feed_subtitle":"Separate value functions for walking and reaching beat one shared critic—before any reward redesign is needed.","key_machinery":"The load-bearing mechanism is the dual actor-critic architecture: two independent actor-critic branches, one for locomotion and one for the arm, each with its own value function trained on a disjoint reward signal (velocity tracking and balance for locomotion; reaching distance and displacement for the arm). The unified alternative feeds a single 109-dimensional observation into one critic that estimates the combined value of both reward streams. The paper's mechanistic claim is that the unified critic's value landscape spreads locomotion-oriented gradients onto arm actions, damping their magnitude and producing slower, more conservative reaching; separating the critics isolates each objecti","core_discovery":"On the paper's own terms, the central discovery is that replacing a single value function—which receives the concatenated observation of both locomotion and manipulation objectives—with two independent value functions that consume disjoint reward signals changes the efficiency of a trained humanoid loco-manipulation policy. In a standardized evaluation, the dual-critic policy reaches a target in 6.5 simulation steps on average versus 22.6 steps for the unified-critic policy, a 3.5x improvement; it scores 14.3 versus 7.0 validated reaches per 1,000 steps; and its validated reach rate is 65.2% versus 53.8%. The paper also reports that adding five anti-gaming reward mechanisms to the dual-criti","pith_inferences":["If the dual-critic result holds under a clean single-variable ablation, the same separation-by-objective principle could transfer to other multi-task RL settings—e.g., vision-guided reaching, bimanual manipulation, or concurrent navigation and interaction—where one shared value function may similarly filter gradients.","A testable extension the paper leaves implicit: measure the evolution of arm action magnitude during training under both architectures; if the unified critic's suppression starts only when locomotion reward dominates, the causal story about gradient competition can be confirmed directly.","The efficiency gap suggests a potentially cheap intervention for existing unified-critic pipelines: re-train with separated critics before investing in reward shaping or new demonstrations; the comparison says this could be worth trying even without changing the policy class."],"forward_implications":["If the central claim holds, critic architecture deserves to be listed alongside reward design and policy class in the standard design checklist for multi-objective humanoid RL.","Standard training metrics (reward value, curriculum level, cumulative reach counts) are insufficient to compare multi-objective policies; time-to-reach and validated throughput should be reported routinely.","In RL fine-tuning of imitation-learned manipulation policies, a unified critic may overwrite or suppress the pre-trained arm behavior; separating critics by objective is a candidate safeguard against catastrophic forgetting.","Anti-gaming reward mechanisms may be redundant once the architectural interference is removed, saving engineering effort.","The 3.5x speed gap, if real, translates directly to more validated reaches per unit of compute, which matters for sim-to-real and real-time deployment."],"fun_headline_variants":["Dual critics cut humanoid reach steps 3.5x vs unified critic","Separate value functions boost loco-manipulation efficiency","Critic architecture: split rewards improve humanoid reach rate","Two critics vs one: humanoid reaches 65% vs 54% success","Better critic splitting speeds humanoid target reaching"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The causal claim rests on the assumption that the only meaningful difference between the two training runs is the critic architecture; the paper itself notes that they also differ in arm action dimensionality (12 vs 5 degrees of freedom), curriculum schedule, one locomotion reward weight, and single-seed training.","fun_headline_variants_meta":{"raw":{"variants":["Dual critics cut humanoid reach steps 3.5x vs unified critic","Separate value functions boost loco-manipulation efficiency","Critic architecture: split rewards improve humanoid reach rate","Two critics vs one: humanoid reaches 65% vs 54% success","Better critic splitting speeds humanoid target reaching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1654,"prompt_tokens":882,"completion_tokens":772,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":698}},"tokens_in":626,"tokens_out":772,"duration_ms":8930,"temperature":1.0,"reasoning_tokens":698,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:44:33.779963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a single-variable ablation in which the dual-critic and unified-critic policies use the same arm action space, the same curriculum schedule, and the same reward weights, with multiple seeds; if the unified-critic policy achieves the same or better time-to-reach and throughput as the dual-critic policy, the central claim is falsified. Alternatively, if a dual-critic policy with a 12-DoF arm action space still shows 3.5x faster reaching than its unified counterpart, the architectural explanation is supported.","supporting_citations":[],"review_version":2}