{"id":"e17bd43c-842b-4476-bce8-f0d9f67d8232","arxiv_id":"2411.14913","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HyDo combines diffusion-model policies with maximum entropy RL in a hybrid discrete/continuous action space, improving success rates on non-prehensile manipulation tasks.","lead":"The paper presents HyDo, a reinforcement learning algorithm that uses diffusion models to explore diverse continuous motion actions while also selecting contact points, for non-prehensile robot manipulation. On simulated and real-robot 6D pose alignment tasks, it reports higher success rates than the HACMan baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 53%→72% '6D pose alignment' claim aggregates planar and 6D results; the 6D-only improvement is 54%→64% and is not statistically significant.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper cannot be fully accepted without corrections. However, I identify a different primary weak point than the reader's weakest_assumption. The reader focuses on the theoretical validity of using path entropy as a surrogate for action entropy in the maximum-entropy objective. That is a legitimate concern about the 'principled objective' framing. The more load-bearing issue for the central claim, as stated in the abstract, is that the headline 53%→72% number is not a 6D-task result; it is an aggregate over planar and 6D trials. The 6D-only comparison shows a 10-point improvement with overlapping confidence and a Fisher exact p-value around 0.40, so the phrase 'significantly improved ... 6D pose alignment' is not supported by the data as reported. This is a concrete, verifiable error in the presentation of the main empirical evidence. Correcting it would require rewording the abstract and adding per-task significance tests, which is consistent with a CONDITIONAL verdict. The theoretical issue should also be fixed, but it is secondary because the empirical diversity and success-rate improvements could still hold even if the derivation is informal. Hence I keep the reader's CONDITIONAL verdict and mark agreement as partial.","tokens_in":12751,"tokens_out":6082,"duration_ms":57687,"concrete_test":"Recompute Table II with planar and 6D results reported separately, and run a two-sided Fisher exact test on the 6D-only counts (HACMan 27/50 vs HyDo+CM 32/50). If p > 0.05, the abstract and Section V-B.2 should be corrected to state that the 53%→72% improvement is the aggregate over planar and 6D goals, and the 6D-only improvement should be described as non-significant or supported by additional trials.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim, stated in the abstract and introduction, is that HyDo improves success rates 'from 53% to 72% on a real-world 6D pose alignment task.' Table II does not support this reading. The table shows two blocks: planar goals (left) and 6D goals (right). The 53% figure is HACMan's combined success rate (26/50 planar + 27/50 6D = 53/100), and 72% is HyDo+CM's combined rate (40/50 planar + 32/50 6D = 72/100). On the 6D task alone, the comparison is HACMan 27/50 (54%) versus HyDo+CM 32/50 (64%). A two-sided Fisher exact test on these 6D-only counts gives p ≈ 0.40, so the word 'significantly' is not supported for the 6D task. The text in Section V-B.2 also reports only the aggregate averages, making it easy for readers to infer the 6D-specific improvement is 53→72. Because the abstract's headline example is the strongest evidence for the central claim of improved real-world success rates, this misreport is the most load-bearing weakness: the advertised 6D result is not established by the reported experiments, even though the aggregate and simulation results remain suggestive. The theoretical concern about the path-entropy surrogate flagged by the reader is also worth addressing, but the empirical misreport is more immediately decisive for the paper's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HyDo, a hybrid off-policy maximum-entropy reinforcement learning algorithm for non-prehensile manipulation. The continuous motion-parameter policy is modeled as a diffusion or consistency model, while discrete contact-point selection is handled by a Q-value-based softmax policy. The authors derive a surrogate maximum-entropy objective via structured variational inference and evaluate HyDo in simulation and zero-shot sim2real tasks, reporting improved success rates and behavior diversity compared with HACMan baselines.","tokens_in":13148,"tokens_out":4498,"duration_ms":44349,"significance":"If the central claims hold, the work is a useful step for non-prehensile manipulation: it combines diffusion/consistency policies with maximum-entropy exploration in a hybrid action space, and the simulation study uses IQM with confidence intervals, which is a strength. The qualitative diversity analysis and ablations over denoising steps also provide useful evidence. However, the headline real-world result is misreported, and the theoretical identification of the path-entropy term with action entropy is not established; both issues affect the paper's main claims.","major_comments":[{"comment":"The claim in the abstract and introduction that HyDo improves success from 53% to 72% 'on a real-world 6D pose alignment task' is not supported by Table II. The 53% figure is HACMan's combined planar+6D total (26/50 + 27/50 = 53/100) and the 72% figure is HyDo+CM's combined total (40/50 + 32/50 = 72/100). For the 6D task alone, HACMan achieves 27/50 (54%) and HyDo+CM achieves 32/50 (64%); a two-sided Fisher exact test on these counts gives p ≈ 0.40, so the word 'significantly' is not justified for the 6D-specific comparison. Please correct the abstract and introduction, report planar and 6D results separately with exact counts and statistical tests, and do not describe the combined average as a 6D-task result.","section":"Abstract, Section I, Section V-B.2, Table II"},{"comment":"The regularizer Σ_{k=0}^K log πθ(a_{k-1} | a_k, k, s) is a sum of reverse-chain log-densities, not the entropy H(πθ(a0|s)) of the action distribution πθ(a0|s). The structured variational inference derivation lower-bounds log p(O1:T) by an expectation of reward minus a path log-density term; it does not show that this path term equals or bounds -log πθ(a0|s) or H(πθ(a0|s)). Thus the claim that the method optimizes a principled maximum-entropy objective for the diffusion policy is not established. Please either provide a proof that the path entropy is a valid surrogate for the action entropy, or recast the objective as a heuristic regularizer and temper the theoretical contribution accordingly.","section":"Section IV-B, Eq. (4)"},{"comment":"The real-robot evaluation uses only 10 trials per object per condition and 50 trials per condition overall. Even for the combined planar+6D comparison, no confidence intervals or significance tests are reported, and for the 6D-only comparison the observed gap (27/50 vs 32/50) is within sampling error. The manuscript should report exact binomial confidence intervals and appropriate tests for each task configuration, and should avoid the phrase 'significantly improved' unless a test supports it.","section":"Section V-B.2, Table II"}],"minor_comments":[{"comment":"The summation in Eq. (4) runs from k=0 to K, but the term for k=0 refers to log πθ(a_{-1} | a_0, 0, s), which is undefined; the summation should start at k=1, consistent with Eq. (6) and the variational derivation.","section":"Section IV-B, Eq. (4)"},{"comment":"The sentence reporting average success rates of 53%, 64%, 68%, and 72% should state explicitly that these numbers are averaged over planar and 6D goals combined; the table shows separate planar and 6D columns, so the current text invites the misinterpretation that the 6D task alone improved from 53% to 72%.","section":"Section V-B.2"},{"comment":"The behavior-entropy formula uses log_{|B|} π(β), which is a normalized entropy, but the notation is not defined; please clarify that this is normalized by log|B| so that the reported values are comparable across tasks with different |B|.","section":"Section V-C"},{"comment":"There are several typos and stylistic issues, including 'varity' in Section I and 'intracble' in Section IV-B; the paper would benefit from a careful proofreading pass.","section":"Global"},{"comment":"The manuscript does not provide hyperparameter values for α1, α2, β, and the number of denoising steps K for the main experiments, nor does it include code or a reproducibility statement; adding these would strengthen the empirical contribution.","section":"Section V and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The aggregate planar+6D improvement (53% to 72%) may be statistically meaningful, and the simulation results are suggestive, but the abstract's specific 6D reading is not supported. The theoretical gap between path entropy and action entropy is also real and load-bearing. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also note that the authors cite several of their own related works; this is not inappropriate, but the manuscript should ensure that the novel contribution is clearly distinguished from those works."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. The paper proposes HyDo, a hybrid off-policy RL method that uses a diffusion policy for continuous motion parameters and explicit entropy regularization on both discrete and continuous action spaces. The combination is new, the simulation evaluation is well done, and the diversity analysis is thoughtful. But the headline result in the abstract and introduction is misreported. The 53% to 72% figure is the combined planar-plus-6D success rate across all objects. The 6D-only comparison is HACMan 27/50 (54%) vs HyDo+CM 32/50 (64%), which a Fisher exact test will not call significant (p ~ 0.4). That is a real problem because the paper sells this as evidence of improved real-world success, and the 6D task is the harder, more relevant one.\n\nWhat is genuinely good: the algorithm design is sensible. Modeling the motion parameter policy as a diffusion model and adding an entropy term to the actor and critic targets is a natural extension of SAC to diffusion policies. The theoretical section gives a variational lower bound for the maximum-reward likelihood objective; it is a standard ELBO argument, and it is not circular. The simulation results are consistent across unseen categories and instances, with IQM and 95% CIs, and the ablations separating the entropy term from the diffusion component are informative. The behavior entropy evaluation, following Jia et al., is a nice touch.\n\nWhere the soft spots are: the abstract overclaims, as above. Also, the real robot experiments use only 50 trials per condition, so even the aggregate 53→72 has wide uncertainty; that is a standard limitation in this literature but should be presented humbly. The theoretical claim that the path-entropy sum is a valid surrogate for the entropy of π(a0|s) is not established. It is fine as a variational lower bound, but calling it 'maximum entropy' for the action distribution is a modeling choice that needs a word of caution. Minor: no code release mentioned; that should be encouraged.\n\nNet: this is a useful within-subfield contribution, not a game-changer. It deserves peer review, but only after the authors fix the reporting and tone down the abstract.\n\nRecommendation: engage with it as a referee if you get the chance; it's a solid paper with a fixable flaw.","headline":"Solid diffusion-policy RL method with a real reporting problem: the 53→72 headline aggregates planar and 6D runs; the 6D-only gain is not statistically significant.","tokens_in":13609,"tokens_out":2139,"would_cite":true,"duration_ms":21273,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid diffusion-policy actor-critic with a maximum-entropy objective improves exploration and raises real-robot success in non-prehensile manipulation to 72%.","keywords":["reinforcement learning","diffusion policies","non-prehensile manipulation","hybrid action space","maximum entropy RL","sim2real transfer","consistency models","soft actor-critic"],"falsifier":"Run HyDo with the entropy term replaced by a direct sample-based estimate of the entropy of the executed action distribution $\\pi(a^0|s)$ (e.g., histogram or k-NN entropy over many sampled actions at fixed states); if success rates do not drop or the ranking of policies does not match the path-entropy surrogate, the mechanism attributed to the entropy term is falsified.","tokens_in":12592,"feed_emoji":"🤖","tokens_out":9301,"duration_ms":76817,"temperature":0.7,"pith_summary":"The paper sets out to show that a robot learning non-prehensile manipulation—pushing, flipping, aligning 6D poses—explores more effectively when the continuous part of its hybrid action policy is a diffusion model and the training objective includes a maximum-entropy regularizer. The proposed algorithm, HyDo, selects discrete contact points by Q-value maximization and generates continuous motion parameters by sampling a diffusion policy, with an entropy term over the full denoising chain derived as a lower bound via structured variational inference. In simulation, HyDo beats the HACMan baseline on unseen object categories and instances in 6D pose alignment. In zero-shot transfer to a real Franka Panda robot, the average success rate over planar and 6D goals rises from 53% for HACMan to 68% for HyDo and 72% for HyDo+CM, while behavior-diversity metrics also increase. If this holds, the payoff is a general recipe for diverse exploration in hybrid-action robotic manipulation that transfers to unseen objects and real hardware.","feed_headline":"Diffusion policies lift robot manipulation success to 72%","feed_subtitle":"A maximum-entropy diffusion policy in hybrid RL learns more diverse pushing and flipping behaviors, enabling zero-shot sim-to-real transfer.","key_machinery":"The object that carries the argument is the HyDo actor-critic objective, in which the continuous motion-parameter policy $\\pi_m$ is a diffusion model (a policy that generates actions by denoising random noise over $K$ steps) and the discrete contact-point policy $\\pi_{\\text{loc}}$ is a softmax over per-point Q-values. The load-bearing identity is the lower-bound maximum-entropy objective $J_\\pi(\\theta) = \\sum_t \\mathbb{E}_{s_t,a^{0:K}_t\\sim\\pi_\\theta}[ r(s_t,a^0_t) - \\alpha \\sum_{k=0}^K \\log \\pi_\\theta(a^{k-1}_t | a^k_t, k, s_t) ]$, obtained from a variational lower bound on $\\log p(O_{1:T})$ with a factorized variational distribution $q$ over the denoising chain. This one expression supplies the actor loss (via reparameterized gradients) and the entropy term in the soft Bellman target, and the same entropy logic extends to the location policy through $\\log \\pi_{\\text{loc}}$. The consistency-model variant replaces the DDPM sampler with a consistency-model inference step, reusing the same objective and update rules.","core_discovery":"The paper's central claim is that the combination of a diffusion-model motion policy and a maximum-entropy objective—both applied inside a hybrid actor-critic that already chooses discrete contact points—produces exploration that is measurably more diverse and more successful for 6D non-prehensile manipulation. The new objective is $J_\\pi(\\theta) = \\sum_t \\mathbb{E}_{s_t,a^{0:K}_t\\sim\\pi_\\theta}[ r(s_t,a^0_t) - \\alpha \\sum_{k=0}^K \\log \\pi_\\theta(a^{k-1}_t | a^k_t, k, s_t) ]$, which the authors derive as a lower bound on the maximum-reward likelihood using structured variational inference with a variational distribution $q$ that factorizes over the denoising chain. This objective feeds both the reparameterized actor gradient and the soft Bellman target, while the location policy keeps its softmax-over-Q form. The authors show that HyDo and HyDo+CM (a consistency-model variant) outperform HACMan and its diffusion/consistency ablations on unseen categories and instances in simulation, achieve higher behavior entropy on fixed tasks, and raise the real-robot success average from 53% to 72% in zero-shot transfer.","pith_inferences":["The paper's headline 53% to 72% averages planar and 6D real-robot trials; reading Table II directly, the 6D-only gain is from 54% (HACMan) to 62% (HyDo) and 64% (HyDo+CM).","If the variational lower bound is loose, the diversity gains may reflect diffusion-policy expressiveness or implicit regularization rather than true maximum-entropy exploration; a direct test is to replace the path-entropy term with a sample-based estimate of the true action entropy.","The same actor-critic pattern—Q-maximized discrete selection plus entropy-regularized diffusion sampling for continuous parameters—should transfer to other hybrid-action robot tasks such as grasping, tool use, or any contact-rich manipulation where motion parameters are continuous.","The consistency-model result suggests that diffusion-policy exploration in online RL can be made cheap enough for real-time control, so pairing entropy-regularized diffusion with faster samplers is a promising direction for closed-loop manipulation."],"forward_implications":["On simulated 6D pose alignment, HyDo achieves higher interquartile-mean success than HACMan on unseen categories (0.843 vs 0.760) and unseen instances (0.884 vs 0.818).","Adding entropy regularization to diffusion or consistency-model policies improves success by roughly 10 to 15 percentage points over the unregularized diffusion/consistency baselines HACMan+Diff and HACMan+CM.","In zero-shot real-robot transfer, the diffusion-based entropy-regularized variants outperform HACMan on both planar and 6D goals, with HyDo+CM reaching 72% average success.","The consistency-model variant HyDo+CM matches or exceeds the diffusion variant's success while needing fewer denoising steps; at $K=5$ it reaches 0.787 success with 7.51 ms inference time versus 0.684 for HyDo.","All HyDo variants show higher behavior entropy than HACMan and HyDo (w/o Diff), indicating more diverse action sequences under identical start and goal poses."],"supporting_citations":[{"why":"Supplies the HACMan hybrid actor-critic map framework and the non-prehensile manipulation task setup that HyDo extends.","marker":"[7]"},{"why":"Provides the maximum-entropy off-policy actor-critic objective that the paper adapts to diffusion policies.","marker":"[15]"},{"why":"Defines the denoising diffusion probabilistic model used as the continuous motion-parameter policy.","marker":"[12]"},{"why":"Provides the consistency-model alternative that yields the HyDo+CM variant with faster inference.","marker":"[37]"},{"why":"Establishes diffusion policies as a policy class for actor-critic RL and supplies the reparameterized gradient for the actor loss.","marker":"[16]"},{"why":"Supplies the structured variational inference derivation used to justify the lower-bound entropy objective.","marker":"[40]"},{"why":"Introduces the hybrid action-space formulation and a max-entropy term for continuous motion parameters, which HyDo extends to both spaces.","marker":"[6]"},{"why":"Provides the behavior-entropy metric used to quantify policy diversity.","marker":"[10]"},{"why":"Shows consistency models can be used as policies in online RL, supporting the HyDo+CM variant.","marker":"[13]"}],"fun_headline_variants":["Diffusion policies boost non-prehensile robot skills to 72% real-world success","Hybrid diffusion policy explores better, hits 72% on real robot tasks","Non-prehensile manipulation: diffusion policy lifts success to 72%","Maximum-entropy diffusion RL: diverse robot actions, 72% success","Diffusion-based hybrid RL learns diverse pushing and flipping to 72% sim2real"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on assuming that the summed log-probabilities of the diffusion denoising steps measure the true diversity of the action the robot executes, an equality the paper derives as a lower bound but never proves.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion policies boost non-prehensile robot skills to 72% real-world success","Hybrid diffusion policy explores better, hits 72% on real robot tasks","Non-prehensile manipulation: diffusion policy lifts success to 72%","Maximum-entropy diffusion RL: diverse robot actions, 72% success","Diffusion-based hybrid RL learns diverse pushing and flipping to 72% sim2real"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2957,"prompt_tokens":1016,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1838}},"tokens_in":632,"tokens_out":1941,"duration_ms":14432,"temperature":1.0,"reasoning_tokens":1838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:43:25.877777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HyDo with the entropy term replaced by a direct sample-based estimate of the entropy of the executed action distribution $\\pi(a^0|s)$ (e.g., histogram or k-NN entropy over many sampled actions at fixed states); if success rates do not drop or the ranking of policies does not match the path-entropy surrogate, the mechanism attributed to the entropy term is falsified.","supporting_citations":[{"cited_title":"HACMan: Learning hybrid actor-critic maps for 6d non-prehensile manipulation,","cited_arxiv_id":null,"evidence_quote":"Supplies the HACMan hybrid actor-critic map framework and the non-prehensile manipulation task setup that HyDo extends."},{"cited_title":"Denoising diffusion probabilistic models,","cited_arxiv_id":null,"evidence_quote":"Defines the denoising diffusion probabilistic model used as the continuous motion-parameter policy."},{"cited_title":"Diffusion policies as an expres- sive policy class for offline reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Establishes diffusion policies as a policy class for actor-critic RL and supplies the reparameterized gradient for the actor loss."},{"cited_title":"A hybrid ap- proach for learning to shift and grasp with elaborate motion primitives,","cited_arxiv_id":null,"evidence_quote":"Introduces the hybrid action-space formulation and a max-entropy term for continuous motion parameters, which HyDo extends to both spaces."},{"cited_title":"Towards diverse behaviors: A benchmark for imitation learning with human demonstrations,","cited_arxiv_id":null,"evidence_quote":"Provides the behavior-entropy metric used to quantify policy diversity."},{"cited_title":"Consistency models as a rich and efficient policy class for reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Shows consistency models can be used as policies in online RL, supporting the HyDo+CM variant."}],"review_version":1}