{"id":"2e1866b3-f497-4b31-b5d9-c11ee430d546","arxiv_id":"2412.00084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An ablation study shows that observation sequences, action sequences, receding horizon control, U-Net backbones, and FiLM conditioning each help Diffusion Policy in task-dependent ways, with absolute-control and hard tasks benefiting most.","lead":"This paper breaks the Diffusion Policy robot-learning method into five design components and tests how much each one matters on eight manipulation tasks. The results give practitioners a rule of thumb for which parts to keep or drop depending on the task type.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hammer is a repeated counterexample: the paper's own tables contradict its takeaways on U-Net, FiLM, and receding horizon for hard/long-horizon tasks.","rationale":"The reader's weakest assumption concerns missing error bars, seeds, and statistical tests, which is a valid concern. However, the more decisive problem is that the reported point estimates themselves contradict the paper's headline conclusions. For Hammer, classified as hard/difficult and long-horizon, the effects are exactly zero or slightly negative for three of the five components (U-Net, FiLM, receding horizon), while the paper claims these components are crucial in exactly those categories. This internal inconsistency is independent of sampling uncertainty—no confidence interval can turn a reported 17% vs 17% tie into a 'significant improvement.' The aggregated figures obscure the discrepancy by averaging across tasks, and the post hoc task taxonomies allow the author to cherry-pick categories that fit the narrative. A multi-seed replication of the Hammer ablation would settle whether the reported numbers are reliable, but even without that, the paper must be revised to reclassify Hammer or qualify the takeaways. Since these issues are addressable through a major revision, the conditional verdict remains appropriate.","tokens_in":8175,"tokens_out":8505,"duration_ms":77989,"concrete_test":"Run a 10-seed replication of the Adroit Hammer ablation for Tables 3-5 using the same 25 human demonstrations and the paper's (to-be-released) hyperparameters. Report mean success rate plus 95% confidence interval for U-Net vs MLP, FiLM vs direct inputs, and with vs without receding horizon control. If the confidence intervals overlap zero or the point estimates remain within 1-2% of each other as reported, then takeaways 4.4-4.6 cannot stand as stated; the authors must either reclassify Hammer or explicitly qualify the hard/long-horizon claims. Additionally, recompute the hard/long-horizon aggregates excluding Hammer; if the conclusions flip or the effect sizes shrink materially, the taxonomy itself is doing the work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central takeaways are not supported by its own tables. Takeaway 4.5 says U-Net is crucial for hard tasks, but Table 4 labels Hammer as 'Hard' and reports U-Net 17% vs MLP 17%, an exact tie. Takeaway 4.6 says FiLM conditioning significantly improves hard tasks, but Table 5 labels Hammer as 'Difficult' and reports FiLM 17% vs Direct Inputs 18%, a slight reversal. Takeaway 4.4 says receding horizon control is essential for long-horizon tasks, but Table 3 lists Hammer's horizon as 270 steps and reports 17% with and without receding horizon control, again a tie. These are not small differences to be resolved by error bars; they are zero or reverse effects in exactly the categories the takeaways claim are strongly affected. The aggregated bar charts in Figures 10-12 average over tasks, allowing large gains in Relocate and Door to mask Hammer's null contributions. Because the task taxonomies (easy/hard, long/short, delta/absolute) are introduced post hoc, the 'task-dependent' conclusions appear to be selective summaries rather than robust empirical patterns. The absence of seeds and confidence intervals compounds the problem: even for tasks with large point differences, the reliability is unknown.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies five components of Diffusion Policy—observation sequence input, action sequence execution, receding horizon control, denoising network architecture, and FiLM conditioning—and reports ablation experiments across eight ManiSkill and Adroit tasks. For each component, it presents a table of success rates, grouped bar charts, and a takeaway stating when the component matters (e.g., absolute vs. delta control, long vs. short horizon, hard vs. easy tasks). The paper concludes with practical recommendations for when to keep or remove each component.","tokens_in":8453,"tokens_out":3479,"duration_ms":33771,"significance":"If the reported patterns held, this would be a useful practical reference for researchers modifying Diffusion Policy, since the five components are clearly identified and the ablation covers two commonly used benchmarks with multiple task types. The paper is also honest in showing per-task tables rather than only aggregate numbers. However, the central claims currently exceed the evidence: the tables contain no error bars, seed counts, or statistical tests, and several individual task results directly contradict the stated takeaways. The significance of the paper therefore depends on whether the authors can provide statistical support and reconcile the counterexamples.","major_comments":[{"comment":"Takeaway 4.5 states that U-Net denoising architecture is essential for hard tasks, but Table 4 labels Hammer as a hard task and reports U-Net 17% vs. MLP 17%, an exact tie. This is not a small gap to be resolved by error bars; it is a null effect in exactly the category the takeaway claims is strongly affected. The aggregate plot in Fig. 11 averages over six hard tasks and therefore hides this counterexample. The takeaway and the recommendation need to be revised to acknowledge that Hammer does not support the claim, or the authors need to provide a principled, pre-specified reason why Hammer should be excluded.","section":"Sec. 4.5 / Table 4"},{"comment":"Takeaway 4.6 says FiLM conditioning significantly improves performance on hard tasks, but Table 5 lists Hammer as a difficult task and reports FiLM 17% vs. Direct Inputs 18%, a slight reversal. Additionally, Pen, classified as easy, shows FiLM 71% vs. Direct Inputs 75%. The claim of a significant improvement on hard tasks is therefore contradicted by one of the six difficult tasks in the same table. The paper should either weaken this takeaway to an exception-aware statement or provide additional evidence, such as confidence intervals, showing that the aggregate difference is meaningful despite individual reversals.","section":"Sec. 4.6 / Table 5"},{"comment":"Takeaway 4.4 states that receding horizon control is essential for long-horizon tasks, but Table 3 lists Hammer with a task horizon of 270/400 steps and reports 17% with receding horizon control and 17% without, an exact tie. Since Hammer is one of the longest-horizon tasks in the table, this directly undermines the claimed relationship between horizon length and the benefit of receding horizon control. The paper needs to either explain this exception, revise the takeaway to \"beneficial for most long-horizon tasks,\" or provide evidence beyond a single success-rate value that the effect is real.","section":"Sec. 4.4 / Table 3"},{"comment":"No error bars, number of seeds, or statistical tests are reported anywhere in the experimental section. Many comparisons rely on small differences that could easily be random variation, such as Hammer 17% vs. 11% in Table 1, Hammer 17% vs. 27% in Table 2, Pen 71% vs. 73% in Table 3, and Hammer 17% vs. 18% in Table 5. Without multiple seeds and confidence intervals, the reader cannot distinguish a real component effect from evaluation noise. This is load-bearing because all five takeaways are empirical claims about differences in success rates. The authors should rerun with multiple seeds, report mean and variance, and ideally perform paired comparisons across tasks.","section":"Sec. 4.1 / Tables 1-5"},{"comment":"The task taxonomies (absolute vs. delta control, long vs. short horizon, easy vs. hard) appear to be assigned post hoc in a way that tracks the observed outcomes. For example, Hammer is classified as hard in Table 4, difficult in Table 5, long-horizon in Table 3, and as a real-time-control exception in Sec. 4.3; the paper does not give a pre-specified criterion for these labels. Since the takeaways depend on these categories, post hoc classification risks circularity: the categories are defined after seeing which tasks show large effects. The authors should define the classification criteria before presenting results, or demonstrate that the conclusions are robust to reasonable alternative classifications.","section":"Sec. 4.2-4.6"}],"minor_comments":[{"comment":"The text says \"The empirical results in Tab. 2 and Fig. 4 indicate...\" but Fig. 4 is a schematic of action sequence execution, not the performance comparison; the performance comparison appears in Fig. 9. Please correct the cross-reference.","section":"Sec. 4.3"},{"comment":"There is an incomplete placeholder citation \"[?]\" in the sentence about state-of-the-art image generation works; this should be replaced with a concrete reference.","section":"Sec. 3.4"},{"comment":"Identical conditions appear to have different success rates across tables: PushChair with the full Diffusion Policy is reported as 61% in Table 1 and Table 2 but 60% in Tables 3, 4, and 5. This suggests either different runs or inconsistent reporting; please clarify whether all tables refer to the same baseline runs and, if not, report per-seed variation.","section":"Table 3"},{"comment":"The experimental setup does not report the exact values of observation horizon To, action horizon Ta, prediction horizon Tp, learning rate, batch size, number of denoising steps, or architecture sizes for U-Net and MLP. Without these values, the ablations are not reproducible.","section":"Sec. 4.1"},{"comment":"Reference [27] is given as \"A Vaswani\" and should be expanded to the full Attention Is All You Need citation; several other references are also incomplete (e.g., [24]).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical ablation with no derivations and no code or data release mentioned. The central issue is that the five takeaways are stated categorically, but the paper's own tables contain repeated counterexamples (especially Hammer) and there are no error bars or seed counts. I do not think this warrants rejection because the authors could fix the manuscript by adding multi-seed statistics, revising the takeaways to be exception-aware, and tightening the task taxonomy. However, the current version is not acceptable as-is for a journal; the experimental rigor and the consistency between claims and tables need substantial work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a cleanly written ablation of the five components of Diffusion Policy on ManiSkill and Adroit, organized around a sensible practitioner message: which component matters depends on the task. But the paper overstates its own tables, and the stress-test is right: Hammer shows up as a tie or reversal for exactly the takeaways that claim big effects. That is not a nitpick.\n\nWhat is actually new: per-task success-rate numbers on eight tasks for observation horizon, action horizon, receding horizon, denoising architecture, and FiLM conditioning, shown in detailed tables rather than one aggregate score. The task-dependent framing (absolute vs delta, easy vs hard, long vs short) is a reasonable way to structure results. The writing is clear and the categories are sensible in principle.\n\nWhere it goes soft: no seeds, error bars, or statistical tests, so all point differences are unverified. More importantly, several headline takeaways contradict the paper's own tables. Table 4 labels Adroit Hammer as hard, but U-Net and MLP are both 17%; Table 5 labels it difficult, but FiLM is 17% vs direct inputs 18%; Table 3 lists its horizon as 270 (long), but receding horizon gives 17% with and without. The text says U-Net is 'crucial' for hard tasks, FiLM 'significantly improves' hard tasks, and receding horizon is 'essential' for long-horizon tasks. Hammer is a counterexample in all three, and the aggregated bar charts hide it by averaging. That is not a small effect size issue; it is a zero or reverse effect in the exact category the takeaway targets. The paper needs either more data to show Hammer is noise or a more nuanced conclusion.\n\nMinor but real: no code or hyperparameters, which limits reproducibility for a practitioner-oriented paper. The easy/hard categories are post hoc, making the takeaways look fitted rather than predictive.\n\nBottom line: the per-task tables are a useful starting point and the question matters, so the paper deserves referee time. But this version should not be used as guidance as-is. A reader following the recommendations will be misled on Hammer. The author should add seeds and intervals, adjust the claims to fit the data, and explicitly discuss the exceptions.\n\nFor peer review: send it out, but with a clear request to address the contradictions before publication.","headline":"Hammer undoes the punchline: the paper's own tables contradict its strongest takeaways, so treat it as a useful but overclaimed ablation.","tokens_in":8952,"tokens_out":2809,"would_cite":false,"duration_ms":26830,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion Policy's components are task-dependent, not universally necessary.","keywords":["Diffusion Policy","imitation learning","ablation study","robotic manipulation","action sequence execution","receding horizon control","FiLM conditioning","U-Net denoising"],"falsifier":"Run each of the five ablations on the same eight tasks with at least ten random seeds and report confidence intervals; the central claim would be falsified if the with-versus-without gaps (for example, Relocate 64% with FiLM versus 2% with direct inputs, or Hammer 17% with action sequences versus 27% without) shrink to overlap zero across seeds.","tokens_in":7940,"feed_emoji":"🤖","tokens_out":5019,"duration_ms":42072,"temperature":0.7,"pith_summary":"This paper decomposes the Diffusion Policy imitation-learning method into five design choices—observation sequence input, action sequence execution, receding horizon control, denoising network architecture, and FiLM conditioning—and asks how much each contributes to task success. Through ablation experiments on eight ManiSkill and Adroit tasks, it argues that no component helps everywhere: observation history is important for absolute-control tasks but nearly irrelevant for delta-control tasks, action sequence execution boosts most tasks by 10–20% but hurts the reactive Adroit Hammer task, receding horizon control matters for long-horizon tasks, and U-Net and FiLM conditioning are what make hard tasks solvable. The intended payoff is a component-level map that lets practitioners keep, simplify, or drop parts of Diffusion Policy according to task type rather than treating the whole architecture as indivisible.","feed_headline":"No single Diffusion Policy component helps every task","feed_subtitle":"Five ablations on ManiSkill and Adroit show when to keep observation history, action sequences, and U-Net.","key_machinery":"The central object is the conditional denoising diffusion process used as a visuomotor policy, which generates a sequence of actions from an observation sequence. The paper's evidence-generating mechanism is controlled ablation: for each of the five named components, it runs the full Diffusion Policy against a version with that one component removed or replaced, holding benchmarks, demonstrations, and observation modality fixed across the eight tasks. These paired success-rate comparisons carry the argument, and the paper groups results by task properties—control mode, task horizon, and task difficulty—to turn the raw tables into the stated design rules.","core_discovery":"On the paper's own terms, the central discovery is that each of the five components of Diffusion Policy has a task-dependent contribution, and the paper states a takeaway for each. Observation sequence input is crucial for tasks requiring absolute control (for example, Adroit Relocate drops from 64% to 47% success without it) but has little effect on delta-control tasks such as StackCube and PegInsertionSide. Action sequence execution improves most tasks by 10–20%, yet Adroit Hammer does better with single-action execution (27% versus 17%), because responsiveness to real-time feedback matters more than action consistency. Receding horizon control gives roughly a 15% improvement on long-horizon tasks while slightly hurting the short-horizon Pen task. U-Net denoising is essential on hard tasks—PegInsertionSide rises from 21% with MLP to 80% with U-Net, and Relocate from 7% to 64%—while MLP suffices for easy tasks. FiLM conditioning similarly improves hard tasks dramatically, such as Relocate from 2% with direct inputs to 64% with FiLM, while making little difference on easy tasks.","pith_inferences":["A testable extension the paper leaves implicit is a decision rule: given a new task's control mode, horizon, and difficulty, one could predict which components matter without running the full ablation.","The Hammer result suggests a responsiveness-consistency tradeoff in action-sequence policies that could be measured independently, for example by injecting perturbations and comparing recovery speed.","If these single-run gaps are confirmed with repeated seeds, the findings imply that many downstream variants that quietly remove components may be unknowingly trading performance for simplicity.","The delta-versus-absolute distinction points to a Markovianity explanation: in delta control, the current observation already carries the relative state, so history adds little information."],"forward_implications":["For delta-control tasks, practitioners can drop observation history without much loss, simplifying the policy input.","For reactive tasks like Adroit Hammer, action horizon should be shortened toward single-action roll-outs to preserve responsiveness.","For long-horizon tasks, receding horizon control should be kept, while short-horizon tasks can execute full predicted sequences.","Hard tasks need the expressive U-Net denoiser and FiLM conditioning; easy tasks can use a cheaper MLP with direct observation inputs.","Component choices should be informed by control mode, horizon, and difficulty rather than a one-size-fits-all Diffusion Policy default."],"supporting_citations":[{"why":"introduces Diffusion Policy, the architecture whose five components are decomposed and ablated","marker":"[4]"},{"why":"provides the ManiSkill benchmark and its 1000 demonstrations used for the four ManiSkill tasks","marker":"[14]"},{"why":"the ManiSkill2 benchmark source for the manipulation tasks and demonstrations","marker":"[8]"},{"why":"supplies the Adroit benchmark and the 25 human teleoperation demonstrations used for the four Adroit tasks","marker":"[19]"},{"why":"defines FiLM conditioning, the observation-conditioning mechanism ablated in Section 4.6","marker":"[16]"},{"why":"defines the U-Net architecture that is compared against MLP in the denoising-network ablation","marker":"[23]"},{"why":"introduces the denoising diffusion probabilistic model underlying the diffusion policy","marker":"[10]"}],"fun_headline_variants":["Diffusion Policy: each component matters only in some tasks","Task-dependent gains from Diffusion Policy's five parts","No one-size-fits-all for Diffusion Policy components","Which Diffusion Policy part helps? Depends on the task","Ablating Diffusion Policy: each piece shines differently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported success-rate differences between the full policy and each ablated variant are real effects; the paper gives no error bars, seed counts, or statistical tests, so if gaps like Hammer's 17% versus 11% are random variation, the task-dependent takeaways lose their support.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion Policy: each component matters only in some tasks","Task-dependent gains from Diffusion Policy's five parts","No one-size-fits-all for Diffusion Policy components","Which Diffusion Policy part helps? Depends on the task","Ablating Diffusion Policy: each piece shines differently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1280,"prompt_tokens":900,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":516,"tokens_out":380,"duration_ms":3914,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:30.672521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the five ablations on the same eight tasks with at least ten random seeds and report confidence intervals; the central claim would be falsified if the with-versus-without gaps (for example, Relocate 64% with FiLM versus 2% with direct inputs, or Hammer 17% with action sequences versus 27% without) shrink to overlap zero across seeds.","supporting_citations":[{"cited_title":"Film: Visual reasoning with a general conditioning layer","cited_arxiv_id":null,"evidence_quote":"defines FiLM conditioning, the observation-conditioning mechanism ablated in Section 4.6"}],"review_version":1}