{"id":"ad80debd-88b0-46ef-ba2a-57a9aadf13c8","arxiv_id":"2504.14573","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Attention weights in a cross-modal transformer trained on furniture assembly cluster into distinct patterns for different manipulation primitives, suggesting the possibility of unsupervised skill segmentation, but the paper does not implement or evaluate such segmentation.","lead":"This robotics paper proposes a cross-modality attention mechanism for merging camera, tactile, and proprioceptive data in a diffusion policy, and studies whether its attention patterns mark distinct manipulation skills. It reports that attention weights differ across manually defined stages of a furniture assembly task, but it does not actually run the proposed skill segmentation or modality selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'unsupervised segmentation' claim is not actually tested: no segmentation algorithm is run, and the only evidence is a t-SNE visualization of embeddings from a single trajectory, colored by manual labels the paper itself calls ambiguous.","rationale":"The reader's verdict is REJECT, and the strongest concern is that the central claim is not demonstrated. The reader's weakest_assumption focuses on the reliability of the manual labels; my concern is adjacent but broader: even if the labels were perfectly reliable, no unsupervised segmentation algorithm is applied or evaluated, so the headline claim about CMA-enabled segmentation cannot be assessed. Section 4.4's t-SNE plot is a visualization of a single trajectory, not a segmentation result, and Section 5 states the segmentation and hierarchical policy are future work. Because the paper itself admits the key contribution is not implemented, and because no success rates or multi-seed statistics are reported, the REJECT verdict remains appropriate. I mark agreement as partial because the reader's stated weakest assumption (label consistency) is one way the evidence could fail, while the more decisive problem is the absence of any quantitative test of unsupervised segmentation. A concrete control experiment, comparing a real segmentation algorithm on CMA embeddings against random-initialization and raw-feature baselines, would settle the question.","tokens_in":6213,"tokens_out":4078,"duration_ms":37898,"concrete_test":"Run an actual unsupervised segmentation on the trained CMA outputs, e.g., K-means (with K chosen by silhouette score) or change-point detection on the sequence of CMA embedding vectors across all 50 expert trajectories, and evaluate boundary/segment agreement against the manual labels using Adjusted Rand Index or segment F1. Include two controls: (i) embeddings from a randomly initialized CMA, and (ii) raw proprioceptive state features. If the trained CMA's segmentation does not significantly beat both controls, the claim that CMA 'enables unsupervised segmentation' is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's advertised contribution (Section 1; Section 3) is that cross-modality attention (CMA) 'enables unsupervised segmentation of expert trajectories into primitive actions.' To establish that, one would need an algorithm that produces segment boundaries from CMA outputs and a quantitative evaluation of those boundaries. Instead, Section 4.2 shows attention weights differ across the authors' manually defined primitives, Section 4.3 compares validation losses of whole-trajectory vs. per-primitive policies, and Section 4.4 shows a t-SNE plot of embeddings from an entire trajectory. None of these constitutes unsupervised segmentation: t-SNE is a visualization, not a clustering or change-point method; the clusters are interpreted with the same manual labels whose reliability Section 4.4 undermines ('due to the ambiguity in human labels the clustering is not perfect'; the Screw action splits into two clusters). Because t-SNE can produce visually separated clusters in high-dimensional noise, and because the CMA embeddings condition on two consecutive timesteps of state and image features with clear temporal progress, the observed separation could reflect temporal phase or labeling artifacts rather than recovered skill structure. The paper's own Section 5 lists unsupervised segmentation and the hierarchical policy as future work, confirming that the central mechanism was not implemented or evaluated. The sample-efficiency comparison also rests on validation loss from one seed with no task success rates, so even the secondary claims are not robustly supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Cross-Modality Attention (CMA) mechanism integrated into a 1D-conditional U-Net diffusion policy for multi-modal imitation learning in furniture assembly. The stated contributions are (1) a CMA architecture for modality selection, (2) an unsupervised segmentation method for expert trajectories into primitive actions, and (3) improved sample efficiency from training per-primitive policies. Experiments use 50 expert trajectories from FurnitureSim one-leg assembly, manually segmented into six primitives (Reach Base, Grip and Move Base, Reach Leg, Grip and Move Leg, Insert, Screw). Results show attention weight patterns that differ across primitives (Fig. 2), lower validation losses for individual per-primitive policies than for a whole-trajectory policy (Fig. 4), and a t-SNE visualization suggesting clustered embeddings (Fig. 5). The paper's own Section 5 states that unsupervised segmentation and hierarchical policy training are future work.","tokens_in":6510,"tokens_out":4899,"duration_ms":45454,"significance":"If the central claim were substantiated, the ability of a cross-modal attention mechanism to discover primitive action structure without human labels would be a valuable advance for long-horizon manipulation. The paper also presents a concrete architecture and a sensible attention-analysis procedure. However, the evidence as presented is almost entirely qualitative: no unsupervised segmentation algorithm is run, no quantitative segmentation evaluation is reported, and the sample-efficiency comparison rests on validation loss from a single seed with an unequal training setup. The paper's abstract and contributions overstate what the experiments demonstrate. The strengths are the use of a standard benchmark, clear architectural description, and a useful illustration of attention-weight variability across manually defined phases.","major_comments":[{"comment":"The advertised contribution of 'unsupervised segmentation of expert trajectories into primitive actions' is not tested. No segmentation algorithm is implemented or run; Figure 5 is a t-SNE visualization, not a segmentation result, and t-SNE can produce visually separated clusters in high-dimensional noise. Moreover, the clusters are interpreted using the same manual labels that Section 4.4 itself calls ambiguous. Section 5 explicitly lists unsupervised segmentation and hierarchical policy training as future work, confirming that the central mechanism was not evaluated. To support the claim, the authors would need to present an actual segmentation method (e.g., clustering with model selection, change-point detection, or attention-boundary detection) and quantitative boundary/segment evaluation on held-out trajectories, or demonstrate that a hierarchical policy trained from the automatically discovered segments solves the task.","section":"Section 3 and Section 4.4"},{"comment":"The sample-efficiency comparison is confounded. The baseline is one policy trained on the full set of 50 trajectories for 1.2M steps, while each of the six primitive policies is trained on a disjoint subset of the data (approximately 8 trajectories) for 200k steps, with total steps matched to the baseline. The per-primitive policies see far less data and fewer distinct trajectories, and the comparison conflates the effect of segmenting the input distribution with the effect of changing dataset size per policy, number of policies, and total optimization budget per dataset item. In addition, only validation loss is reported, not rollout success rate, and no standard deviations or multiple seeds are given, so the claim of improved 'accuracy' (Section 4.3) is unsupported.","section":"Section 4.3, Figure 4"},{"comment":"The evidence that CMA 'learns characteristics of different primitive actions' is circular. The CMA model is trained on the same demonstration trajectories that were manually segmented in Section 4.1, and these same labels are then used to interpret the attention weights (Fig. 2) and the t-SNE clusters (Fig. 5). Since the model was trained to predict actions from observations that progress through the trajectory, the observed attention differences could reflect temporal phase, labeling artifacts, or spurious correlations rather than emergent skill structure. A non-circular test would require evaluating the attention or embeddings on held-out trajectories, or ablating one modality and showing that attention patterns change in a task-meaningful way.","section":"Sections 4.2 and 4.4"},{"comment":"The abstract states that the method 'leverages this segmentation to train a hierarchical policy capable of solving long-horizon, contact-rich manipulation tasks,' but Section 5 states that training a hierarchical policy is future work and no hierarchical policy is trained or evaluated anywhere in the paper. This is a direct discrepancy between the claims and the reported experiments, and it affects the first sentence of the abstract as well as the third listed contribution.","section":"Abstract and Section 5"}],"minor_comments":[{"comment":"The tensor shape is written as '(T· N× B× D' with an unclosed parenthesis; the embedding dimension D is later stated to be 128 in Section 3.3 but should be defined in the architecture description.","section":"Section 3"},{"comment":"The text 'Thrid stage' contains a typo; it should read 'Third stage.'","section":"Figure 1"},{"comment":"No hyperparameters for the t-SNE visualization (e.g., perplexity, learning rate, number of iterations) are given, which limits reproducibility.","section":"Section 4.4"},{"comment":"Figure 4 appears to show a single run without error bars or seed information; the paper should state how many seeds were used and include variance estimates.","section":"Section 4.3"},{"comment":"The manual segmentation procedure is described only as 'similar to what's defined in Lin et al. [27];' more detail on the annotation protocol and inter-labeler consistency would help assess the reliability of the ground-truth labels used throughout the evaluation.","section":"Section 4.1"},{"comment":"The use of two consecutive timesteps in CMA is mentioned but not formalized; the paper should specify how temporal and modality dimensions are arranged in the attention computation, and why two timesteps is sufficient.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The paper's central advertised contribution is absent from the experiments, and the abstract overstates what is demonstrated. The underlying hypothesis is interesting, but the current manuscript does not contain the required evidence. I would encourage the authors to perform and evaluate an actual unsupervised segmentation procedure and a properly controlled sample-efficiency comparison before resubmitting, rather than framing these as already demonstrated results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is an empirical observation: in a cross-modal diffusion policy trained on FurnitureSim, averaged last-layer attention weights look different across six manually defined manipulation primitives, and training per-primitive policies gives lower validation loss than a single full-trajectory policy. That is a reasonable starting point, and the paper is honest enough about its architecture and data. It also cites prior cross-modal attention work fairly, rather than pretending the mechanism is new.\n\nThe trouble is that the paper's advertised contribution—unsupervised primitive segmentation—never actually gets tested. Section 4.4 is just a t-SNE of one trajectory, which is a visualization, not a clustering or change-point method. The clusters are interpreted using the same manual labels that Section 4.4 itself calls ambiguous, and t-SNE can produce separated clusters in high-dimensional noise, especially when the inputs are two consecutive timesteps with clear temporal progress. The sample-efficiency claim is also thin: six policies on non-overlapping subsets versus one policy on all data, with validation loss from a single seed and no success rates. That is suggestive at best.\n\nThe internal contradiction is the thing to flag. The abstract and Section 1 claim \"unsupervised segmentation\" as a contribution, but Section 5 explicitly lists it as future work. Either the authors did not run the experiment they are claiming, or they overstate what the t-SNE shows. The manual segmentation is also unverified, and they admit the labels are imperfect; that undermines the only ground truth used to judge the attention patterns.\n\nNone of this makes the paper worthless. The observation that attention weights encode skill structure is worth knowing, and the hypothesis that you could segment with a real unsupervised method is plausible and testable. But the current form is a preliminary study, not a complete result. I would want to see an actual segmentation algorithm, quantitative boundary evaluation, and task success rates before taking the central claim seriously.\n\nWho gets value from this? Someone working on hierarchical imitation learning or multi-modal policy architectures might find the attention-pattern observation a useful suggestion. It is a workshop-level exploratory paper in its current state.\n\nMy recommendation: do not desk-reject, but send it to review with the expectation of major revision. A serious referee can push the authors to either implement and evaluate a real unsupervised segmentation method or reframe the paper as a focused empirical analysis of attention patterns. As written, it should not be accepted.","headline":"A plausible but untested hypothesis: the paper shows attention weights differ across manually labeled skills, but it never actually runs an unsupervised segmentation, so the advertised contribution is not yet demonstrated.","tokens_in":7002,"tokens_out":1840,"would_cite":false,"duration_ms":20515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that cross-modality attention in a diffusion policy learns to select the most informative sensor at each timestep and, in doing so, learns attention patterns that segment expert trajectories into primitive actions…","keywords":["cross-modality attention","modality selection","unsupervised skill segmentation","diffusion policy","hierarchical imitation learning","tactile sensing","furniture assembly"],"falsifier":"Record force-torque contact events while the robot performs the one-leg assembly task and use them to define objective action boundaries; if unsupervised clustering of CMA embeddings does not place cluster boundaries at these contact events across most trajectories, the claim that CMA segments primitives would be refuted.","tokens_in":6040,"feed_emoji":"🤖","tokens_out":5971,"duration_ms":53493,"temperature":0.7,"pith_summary":"This paper tries to show that a cross-modality attention (CMA) mechanism—transformer attention over image, proprioceptive, and tactile embeddings—does two jobs at once in a long-horizon furniture assembly policy. First, it selects which modalities matter at each timestep when generating actions. Second, after imitation training on expert demonstrations, its attention weights and embeddings carry enough skill structure to split trajectories into primitive actions without human segmentation labels. The payoff, if correct, is a route to hierarchical policies for contact-rich tasks that avoids labor-intensive and ambiguous manual labeling.","feed_headline":"Attention weights reveal robot skills hidden in expert demos","feed_subtitle":"A cross-modal diffusion policy learns to pick useful sensors and to segment assembly moves without human labels.","key_machinery":"The load-bearing object is the Cross-Modality Attention (CMA) module: a transformer-style self-attention block, here 8 heads and 2 layers, applied to embeddings of camera, grip-camera, proprioceptive, and tactile inputs stacked across two consecutive timesteps. Its softmax attention weights $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})$ are averaged over heads in the last layer, and these averaged weights are the evidence that primitive actions have distinct attention signatures. The module also produces the conditional embedding that drives the 1D-conditional U-Net diffusion policy, so the same learned attention serves both action generation and primitive discovery.","core_discovery":"Working on the one-leg FurnitureSim assembly task with 50 expert demonstrations, the paper reports three experimental findings. After imitation training, the last-layer attention weights of the CMA module, averaged over heads, show distinct patterns for six manually defined primitives; this is taken as evidence that the mechanism has learned primitive characteristics. Training a separate policy for each primitive for 200,000 steps each, matching the 1,200,000-step total, gives lower validation loss than training one policy on whole trajectories, with the screwing primitive as an exception. T-SNE projections of CMA embeddings show clusters that mostly match the manual labels, but with systematic discrepancies—some Grip-and-Move frames land in Reach Base, and Screw splits into screwing and gripper repositioning—which the authors attribute to ambiguity in human labels and cite as motivation for unsupervised segmentation.","pith_inferences":["A direct test the authors did not run: use the unsupervised clusters themselves as the primitive definitions, train the hierarchical policy on them, and compare task success against policies trained on manual labels; this would separate the segmentation claim from the label choice.","If CMA attention weights are stable across seeds and tasks, the mechanism could serve as a generic sensor router for larger multi-modal policies, letting each timestep route only the modalities the current phase of manipulation needs.","Because the paper argues human labels are ambiguous, a stronger evaluation would compare clusters against objective phase annotations such as contact onset, gripper release, and insertion completion rather than the manual six-way split."],"forward_implications":["A hierarchical policy can be trained by clustering CMA embeddings into skills and selecting a skill-specific policy, eliminating the need for human-labeled trajectory segmentation.","Training separate policies on individual primitives should require fewer total samples than whole-trajectory training, because each primitive has a more consistent input distribution.","The attention weights can be used to drop uninformative modalities for each primitive, cutting input dimensionality during execution.","The approach is intended to transfer to other long-horizon, contact-rich assembly tasks where tactile or audio feedback is informative.","Manual primitive labels are not necessarily the true segment boundaries; learned clustering can reveal structure that human labels miss, such as the split of the screwing action into screwing and gripper repositioning."],"supporting_citations":[{"why":"Supplies the FurnitureSim environment and the one-leg assembly task used for all experiments.","marker":"[8]"},{"why":"Provides the 1D-conditional U-Net diffusion policy backbone that CMA conditions.","marker":"[11]"},{"why":"Provides the 50 expert trajectories that are segmented and used for imitation learning.","marker":"[15]"},{"why":"Supplies the prior result that cross-modal attention outperforms concatenation, motivating the CMA design.","marker":"[19]"},{"why":"Defines the six primitive-action labels used to evaluate attention characteristics and segmentation.","marker":"[27]"},{"why":"Supplies the frozen R3M visual encoder whose embeddings are one of the CMA input modalities.","marker":"[30]"}],"fun_headline_variants":["Cross-modal attention picks sensors and splits demos into skills","Robot learns which senses matter and segments skills from demos","Attention mechanism selects modalities and discovers robot skills","Sensing focus and skill discovery via cross-modality attention","Cross-modal attention: choose sensors, segment skills in demos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the authors' manual division of each trajectory into six primitives (Reach Base, Grip and Move Base, Reach Leg, Grip and Move Leg, Insert, Screw) is a reliable and consistent labeling; if those labels are inconsistent, the observed alignment between attention weights and primitives could reflect labeling artifacts rather than real skill structure.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modal attention picks sensors and splits demos into skills","Robot learns which senses matter and segments skills from demos","Attention mechanism selects modalities and discovers robot skills","Sensing focus and skill discovery via cross-modality attention","Cross-modal attention: choose sensors, segment skills in demos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1563,"prompt_tokens":779,"completion_tokens":784,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":395,"tokens_out":784,"duration_ms":6582,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:45:15.027865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record force-torque contact events while the robot performs the one-leg assembly task and use them to define objective action boundaries; if unsupervised clustering of CMA embeddings does not place cluster boundaries at these contact events across most trajectories, the claim that CMA segments primitives would be refuted.","supporting_citations":[{"cited_title":"Diffusion policy: Vi- suomotor policy learning via action diffusion,","cited_arxiv_id":null,"evidence_quote":"Provides the 1D-conditional U-Net diffusion policy backbone that CMA conditions."},{"cited_title":"Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation,","cited_arxiv_id":null,"evidence_quote":"Supplies the FurnitureSim environment and the one-leg assembly task used for all experiments."},{"cited_title":"Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assembly","cited_arxiv_id":"2404.17684","evidence_quote":"Defines the six primitive-action labels used to evaluate attention characteristics and segmentation."}],"review_version":1}