{"id":"2ea57e6c-5202-4ad1-917a-70598158716e","arxiv_id":"2608.09762","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement learning framework that shares a decomposed critic between arm and gripper policies improves real-world pick-and-place success rates by 20 to 70 percentage points over a strong baseline.","lead":"This paper combines two existing reinforcement learning techniques, centralized training with decentralized execution and a decomposed multi-head critic, to train robotic arms to pick and place objects in the real world. It reports higher success rates than a state-of-the-art baseline across several tasks, while using a custom compliant gripper and larger randomization ranges.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pot reset result is confounded by the new compliant gripper: Section IV-B credits the hardware with improved grasping, but the paper never states the HIL-SERL baseline used the same gripper.","rationale":"I read the paper in good faith and take its central claim to be the empirical superiority of HARC over HIL-SERL under large domain randomization. The most load-bearing threat to that claim is not the HRA decomposition condition, though that is a real secondary concern; it is the uncontrolled hardware variable. Section IV-B explicitly credits the compliant palm with improved handle grasping reliability, which is a mechanical improvement independent of the learning algorithm. If the baseline HIL-SERL was run with a different gripper, the pot reset success gap is not evidence about the algorithm. Since the manuscript never states that the baseline used the identical compliant gripper, a skeptical reader cannot verify the headline claim. The proposed concrete test would settle this: run HIL-SERL with the same compliant gripper and randomization. If the baseline still fails, the confound is not the sole explanation; if the baseline improves, the reported comparison needs qualification. The reader's verdict of CONDITIONAL is appropriate, so I do not change it, but I want the hardware-matching control to be an explicit condition. My agreement with the reader is partial because they noted the hardware confound in their rationale but chose the HRA condition as the weakest assumption; I consider the hardware control more directly load-bearing for the central empirical claim.","tokens_in":11173,"tokens_out":8969,"duration_ms":81845,"concrete_test":"Ask the authors to report and, if needed, run the hardware-matched control: HIL-SERL on pot reset with the exact compliant gripper and the same 40x40 cm / 90 degree randomization used for HARC, and/or HARC with the original 1-DoF gripper. If HIL-SERL with the compliant gripper still achieves 0% and HARC with the original gripper still achieves 55%, the concern does not land; otherwise Table II and the abstract's \"baseline fails\" claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—HARC substantially improves sample efficiency and success over HIL-SERL—rests on Table II. The pot reset row (0% vs 55%) is the weakest link. In Section IV-B, the paper states: \"Our compliant palm improves handle grasping reliability beyond the original 1-DoF revolute joint gripper.\" This is a hardware improvement, not an algorithmic one. The comparison to HIL-SERL is valid only if the baseline was evaluated with the identical compliant gripper and identical randomization. The manuscript does not say that. If HIL-SERL used the older/original gripper, the 0 to 55 success gap and the abstract's claim of \"successfully accomplishing a task where the baseline consistently fails\" may be entirely due to mechanical advantage. This confound is load-bearing because pot reset is one of four headline tasks and the only \"baseline fails\" claim; if it is a hardware artifact, the abstract's central claim is overstated even if the other three tasks stand. The same ambiguity affects the custom manipulator tasks: Section III-D introduces a novel task-adaptive gripper as a contribution, but the paper never states whether the HIL-SERL baseline runs used that same gripper. The fix is not to delete the result but to control hardware across methods and report the hardware used for each baseline run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HARC, a real-world online RL framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). It separates a manipulation policy into a continuous Cartesian arm actor and a discrete gripper actor that share a centralized multi-head critic, with one value head for the sparse task reward and one for a potential-based grasp reward. The authors claim that this decomposition, together with a reformulated discrete SAC objective, substantially improves sample efficiency and success rate over the HIL-SERL baseline on tennis-ball pick-and-place, banana pick-and-place, pot reset, and simulated block relocation under larger domain randomization. The paper also contributes a compliant two-finger gripper design used in the real-world experiments.","tokens_in":11435,"tokens_out":7756,"duration_ms":74969,"significance":"If the RL objectives are formally correct and the hardware is matched across methods, the reported results would be a useful step toward real-world online RL under larger workspace randomization, an area where prior systems are often constrained to small perturbations. The paper includes a complete system description, real-robot deployment on two platforms, and a simulation benchmark, which are strengths. However, the central algorithmic equation is internally inconsistent as written, and the hardware used for the baseline is not explicitly controlled, so the headline empirical claims cannot currently be taken at face value. The HRA benefit is also asserted through a conditioning argument rather than demonstrated by ablations or measurements.","major_comments":[{"comment":"The TD targets in Eq. (8) are internally inconsistent. V_k(s') is defined as an expectation over the next discrete action, V_k(s') = E_{a'_d ~ pi_psi}[min Q-bar_k(s', a')], but y_task then subtracts alpha_d log pi_psi(a'_d | s') outside that expectation. After the expectation, a'_d is a bound variable and cannot reappear outside; the resulting expression is not the standard discrete SAC target. The correct discrete SAC target would place the entropy term inside the expectation, e.g., y = r + gamma E_{a'_d ~ pi_psi}[min Q-bar(s', a'_d) - alpha_d log pi_psi(a'_d | s')], or, if a single sampled action is used, y = r + gamma (min Q-bar(s', a'_d) - alpha_d log pi_psi(a'_d | s')). Moreover, because the critic is defined over the hybrid action a = (a_c, a_d), V_k(s') must also specify how the next continuous action a'_c enters; as written no expectation over a'_c is shown. Table I additionally reports \"Entropy backup False,\" which makes the presence of the entropy term in y_task confusing. This issue is load-bearing because the paper's central claim relies on the reformulated critic and actor objectives for the categorical gripper policy.","section":"III-C, Eq. (8)"},{"comment":"The hardware used for the HIL-SERL baseline is not reported, which confounds the main comparison. Section IV-B states, \"Our compliant palm improves handle grasping reliability beyond the original 1-DoF revolute joint gripper,\" and Section III-D introduces a new compliant gripper as a contribution. The paper never states whether the HIL-SERL baseline for the pot reset task, or for the custom-manipulator tasks, used the same compliant gripper and the same randomization. If the baseline ran with the original gripper, the Table II result of 0% vs 55% on pot reset, and the abstract's claim of \"successfully accomplishing a task where the baseline consistently fails,\" could be due primarily to the mechanical advantage of the new gripper rather than to HARC. The authors should state explicitly which gripper was mounted for each baseline run and, if the hardware differed, rerun the baseline with the identical gripper.","section":"IV-B and Table II"},{"comment":"The benefit of the HRA decomposition is asserted through a conditioning argument but is not verified. The paper correctly notes in Section III-C that HRA is most beneficial when \"each reward component depends on only a small, largely disjoint subset of the state variables.\" It then asserts that the task reward is driven mainly by the front view and end-effector pose while the grasp reward is associated with the wrist camera and grasp state, but it provides no measurement, gradient analysis, or ablation supporting this disjointness. If the reward components are not largely disjoint in their dependence on the observations, the variance-reduction argument for the decomposed critic collapses. Since the method's advantage over the monolithic critic is central to the paper, the authors should either provide an ablation with a single shared critic under identical hardware, or report a quantitative proxy for the degree of reward-head disjointness.","section":"III-C"},{"comment":"The performance comparison is presented as point estimates without confidence intervals or number of independent runs. Table II reports a single success rate per method and task, and the learning curves in Fig. 4 appear to be single runs. With 20 evaluation episodes, the binomial standard error at 60% success is about 11 percentage points, so the reported gaps are meaningful only if the evaluation protocol is stable and the runs are representative. The paper also does not isolate the contributions of CTDE, HRA, the discrete SAC actor, or the grasp reward weight lambda_g; an ablation study would be needed to attribute the improvement to the proposed components. These points do not invalidate the empirical claims, but they should be reported transparently for a real-world RL paper.","section":"IV-C and Table II"}],"minor_comments":[{"comment":"The sentence beginning \"Additional hyperparameters on All discrete gripper actors...\" is duplicated and grammatically incomplete; please rewrite it as a single clear statement about the observations used by the discrete actors.","section":"IV-A"},{"comment":"The target entropy is listed as -3 in the shared RLPD parameters and as 0.1 in the HARC parameters. The paper should clarify which target entropy applies to which actor and why the values differ, especially because the discrete actor has three actions.","section":"Table I"},{"comment":"The grasp reward is called a potential-based reward shaping (PBRS) term, but r_grasp(s,a) = gamma Phi(s') - Phi(s) + P is not pure PBRS unless P is part of the reward rather than a shaping term. If P is a gripper-switching penalty, write it as a negative term and define its sign explicitly; otherwise the reader cannot tell whether P encourages or discourages switching.","section":"III-C, Eq. (6)"},{"comment":"The conversion from \"demo buffer transitions\" to \"equivalent episodes\" is not defined clearly enough in the main text. Please specify the average transitions per episode and whether the comparison controls for episode length and intervention duration.","section":"IV-C, Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The Eq. (8) target inconsistency and the unstated hardware matching for the baseline are both fixable within the scope of the manuscript, but they are load-bearing for the central claim. If the authors correct the objective, report the gripper used for each baseline run, and add the requested ablations or measurements, the paper could become a solid contribution to real-world online RL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a systems paper. The real contribution is a specific integrated pipeline—CTDE with two SAC actors (continuous arm, discrete gripper), a shared multi-head critic decomposed into task and grasp heads, and a new compliant gripper—plus real-robot results under 5–25x larger randomization than HIL-SERL. If the results hold, they address a real bottleneck in online robot learning. But one headline result, pot reset, is likely confounded by the new gripper, and the paper's central TD target is written incorrectly. Neither issue makes the system uninteresting, but both need fixing before the claims are solid.\n\nWhat the paper does well: it is unusually concrete about the training setup—RLPD hyperparameters, intervention counts, randomization ranges, episode lengths. The integration is nontrivial: discrete SAC for gripper, shared critic for hybrid actions, and reward decomposition are combined in one asynchronous HIL pipeline. The authors also state the HRA condition (reward components depend on largely disjoint state subsets) and acknowledge that the decomposition is exact only for additive rewards. That is the right level of honesty.\n\nSoft spots, in order:\n\n1. Hardware confound. Section IV-B says the compliant palm 'improves handle grasping reliability beyond the original 1-DoF revolute joint gripper,' but the paper never states that the HIL-SERL baseline ran with the identical gripper and randomization. The pot reset row (0% vs 55%) is the only task where the baseline 'consistently fails,' and if the baseline used the old gripper the comparison is mostly mechanical advantage, not learning. This needs a sentence and ideally a hardware-controlled run.\n\n2. Eq. (8) is internally inconsistent. V_k(s') is defined as an expectation over the next discrete action, yet the task target subtracts α_d logπψ(a'_d|s') outside that expectation. The continuous-action entropy is also missing from the target. It may be a typo from transcribing discrete SAC, but as written it is not a valid SAC target.\n\n3. Missing evidence. No ablations (HRA off, shared critic off, discrete SAC vs DQN), no seeds or confidence intervals on Table II, and no statement about how many evaluation episodes or which randomization samples were used. The 5–25x claim is presented as fact but the baseline randomization numbers are only given as 2–8cm and 1–10°, without the same detail for HARC.\n\nWho is this for: people building real-world online RL systems. It is a useful system demonstration and a good starting point for a serious referee, but it is not yet a clean scientific comparison. Recommendation: send to peer review, but require the authors to correct Eq. (8), release code and videos, report error bars, run at least one ablation of the HRA head, and explicitly state that the baseline used the same gripper and randomization. If the pot reset confound cannot be resolved, soften the abstract.","headline":"Useful system paper that needs a hardware control and a corrected critic target before its headline gains carry weight.","tokens_in":12002,"tokens_out":3178,"would_cite":false,"duration_ms":28207,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a reward-decomposed critic, with separate heads for the sparse task reward and a dense grasp-potential reward, lets real-world robot policies train under 5-25x larger domain randomization than a state-of-the-art…","keywords":["real-world reinforcement learning","robot manipulation","centralized training with decentralized execution","hybrid reward architecture","hybrid action space","human-in-the-loop RL","domain randomization","multi-head critic"],"falsifier":"Run the pot-reset task with a monolithic critic—same encoders, same actor decomposition, same rewards summed into one Q-function—and compare success rate and critic gradient norms against the multi-head version; if collapse and large gradient norms do not reappear, the paper's attribution of stability to reward decomposition is falsified.","tokens_in":10962,"feed_emoji":"🤖","tokens_out":9376,"duration_ms":77447,"temperature":0.7,"pith_summary":"This paper argues that the main obstacle to real-world online reinforcement learning for manipulation is not exploration alone but critic instability: a single Q-function must regress a long-horizon sparse return from noisy RGB observations, and this becomes untenable as workspaces, object poses, and viewpoints vary widely. The proposed HARC framework combines centralized training with decentralized execution and a Hybrid Reward Architecture, splitting the critic into a task-value head and a grasp-value head, with a dense potential-based grasp reward derived from gripper torque or width. Under domain randomization roughly 5-25 times larger than the prior baseline, the framework raises success rates from 60% to 80% on tennis-ball pick-and-place, from 60% to 90% on banana pick-and-place, from 0% to 55% on pot reset, and from 25% to 95% on simulated block relocation, while using fewer expert demonstrations and interventions. If these results hold, real-world RL becomes practical for wider-object-placement manipulation tasks that current online RL systems handle only under tightly constrained setups.","feed_headline":"Split critic lifts real-world robot RL success from 25% to 95%","feed_subtitle":"Separate task and grasp critic heads deliver 55-95% success where the baseline stalls near 0-60%.","key_machinery":"The central object is the reward-decomposed multi-head critic, which instantiates the identity $Q_\\theta(s,a)=\\sum_{k\\in\\{\\text{task},\\text{grasp}\\}} Q^k_\\theta(s,a)$. A potential-based grasp signal $\\Phi(s)=\\operatorname{clip}((x-x_{\\min})/(x_{\\max}-x_{\\min}),0,1)$, with $x$ being gripper torque or width, produces $r_{\\mathrm{grasp}}=\\gamma\\Phi(s')-\\Phi(s)+P$. The heads share a feature trunk but have separate TD targets; actor updates use $Q_{\\text{task}}+\\lambda_g Q_{\\text{grasp}}$, and the discrete gripper actor uses a categorical objective over the softmax logits. This decomposition turns one noisy long-horizon value regression into two simpler regressions, which the paper identifies as the reason critic training remains stable under real RGB observations.","core_discovery":"The paper's central claim is that a centralized, reward-decomposed critic lets a continuous arm policy and a discrete gripper policy be trained directly on hardware with far larger domain randomization than previous human-in-the-loop RL, while using less expert data. Concretely, the authors report success-rate improvements from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, from 0% to 55% on pot reset, and from 25% to 95% on simulated block relocation, with total expert data falling from 80, 102, 132, 189 equivalent episodes to 69, 76, 111, 115. The explanation offered is variance reduction: decomposing the critic into task and grasp heads gives each head a simpler temporal-difference target and reduces the destabilizing effect of noisy RGB observations.","pith_inferences":["A testable extension is to vary the amount of pixel noise in simulation and measure how much the success-rate gap between the multi-head and monolithic critic widens; the paper's variance-reduction story predicts a monotone relationship.","The same reward-decomposition recipe should transfer to other auxiliary signals—contact, proximity, or task-progress classifiers—provided each depends on a small, largely disjoint sensor subset, a regime the paper does not explore.","The stated disjointness condition implies that the method will lose its advantage if the grasp potential is computed from the same front-view features that drive the task reward; ablating that coupling would locate the boundaries of the framework."],"forward_implications":["A training budget of about 160 wall-clock minutes on one GPU can reach 55-95% success on manipulation tasks with wide workspace and pose randomization, rather than only a few centimeters of variation.","Expert supervision becomes cheaper: the total equivalent-episode count of demonstrations and interventions falls from 80, 102, 132, 189 to 69, 76, 111, 115 across the four tasks.","The gripper policy can be trained from wrist-camera images alone, while the arm policy uses the side view, so heterogeneous per-actor observations are compatible with the shared critic.","Because execution remains decentralized, the learned policies require only local observations at deployment, preserving the actor-critic pipeline's practical interface."],"supporting_citations":[{"why":"HIL-SERL is the human-in-the-loop baseline HARC is compared against and whose success rates and expert-data counts HARC improves on.","marker":"[13]"},{"why":"Hybrid Reward Architecture is the source of the reward-decomposition condition and the multi-head critic idea.","marker":"[16]"},{"why":"RLPD supplies the sample-efficient off-policy training loop with offline data and high update-to-data ratio that the pipeline is built on.","marker":"[17]"},{"why":"MADDPG's centralized-critic CTDE formulation is adopted for the joint critic and separate actors.","marker":"[21]"},{"why":"Soft Actor-Critic is the underlying continuous actor-critic learner extended with a discrete gripper actor.","marker":"[26]"},{"why":"The discrete SAC objective provides the categorical-action loss used for the gripper policy.","marker":"[29]"},{"why":"The ResNet-based image encoder extracts visual features and supplies the pre-trained classifier used for sparse task rewards.","marker":"[27]"},{"why":"The modular revolute-joint gripper provides the actuation module and compliant finger mechanics for the novel gripper used in the tasks.","marker":"[30]"}],"fun_headline_variants":["Split critic lifts real-world robot RL from 25% to 95% success","Task and grasp critic heads lift robot RL success to 95%","Real-world robot RL hits 95% success with split critic","Decomposed critic triples success in real-world robot RL","Critic decomposition boosts real-world robot learning to 95%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the sparse task reward and the potential-based grasp reward depend on small, largely disjoint subsets of the observations, so splitting the critic reduces target variance; if the grasp potential shares the same informative features as the task reward, the framework's advantage over a single critic disappears.","fun_headline_variants_meta":{"raw":{"variants":["Split critic lifts real-world robot RL from 25% to 95% success","Task and grasp critic heads lift robot RL success to 95%","Real-world robot RL hits 95% success with split critic","Decomposed critic triples success in real-world robot RL","Critic decomposition boosts real-world robot learning to 95%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2842,"prompt_tokens":1033,"completion_tokens":1809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1718}},"tokens_in":649,"tokens_out":1809,"duration_ms":13408,"temperature":1.0,"reasoning_tokens":1718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:15:18.262690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pot-reset task with a monolithic critic—same encoders, same actor decomposition, same rewards summed into one Q-function—and compare success rate and critic gradient norms against the multi-head version; if collapse and large gradient norms do not reappear, the paper's attribution of stability to reward decomposition is falsified.","supporting_citations":[{"cited_title":"Hybrid reward architecture for reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Hybrid Reward Architecture is the source of the reward-decomposition condition and the multi-head critic idea."},{"cited_title":"A high-force gripper with embedded multimodal sensing for powerful and perception driven grasping,","cited_arxiv_id":null,"evidence_quote":"The modular revolute-joint gripper provides the actuation module and compliant finger mechanics for the novel gripper used in the tasks."}],"review_version":1}