{"id":"1a08945f-c193-4e3b-8a8f-be1ecc3a4e22","arxiv_id":"2501.04693","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FuSe fine-tunes generalist robot policies on touch and audio data using language as a bridge, enabling multimodal prompts and compositional cross-modal tasks.","lead":"This paper introduces FuSe, a training recipe that lets pre-trained robot policies use touch and audio sensors by connecting those senses through language. The method improves grasping and button-pressing success in real robot tests, especially when vision is blocked.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline 'zero-shot descriptions of grasped objects' is never quantitatively evaluated; the only support for the generative head is an action-success ablation, so a central claimed capability rests on assertion.","rationale":"I read the paper as proposing a recipe, validated by real-robot experiments, for grounding new sensor modalities in language while finetuning generalist policies. The method is clearly described, code/data/models appear released, and the ablations (Fig 7) give some support for the two auxiliary losses improving bag-grasping. Unseen-object evaluation is a genuine strength. My concern is not with the internal consistency of the training objective but with the mapping between claims and evidence: the abstract presents zero-shot object description as a headline capability, yet no experiment measures it. The reader's weakest assumption about language annotations not capturing sensor features is related but not the same; my check targets the missing evaluation directly. If the description experiment failed, the central claim would need to be narrowed to action success and prompting, not 'unlocking descriptions'. The existing lack of error bars and significance tests also weakens the numeric 'over 20%' claim, but that is already captured by the reader's conditional verdict; the description gap is the more specific load-bearing omission. For these reasons I recommend keeping the CONDITIONAL verdict rather than moving to ACCEPT.","tokens_in":12848,"tokens_out":5604,"duration_ms":59605,"concrete_test":"Run a dedicated description-generation experiment: grasp N seen and N unseen objects, record the text emitted by the generative head, and score against human-annotated ground-truth property labels (e.g., soft/hard, smooth/rough, crinkled, squishy, sound type) using exact-match, per-property precision/recall, and a text-similarity metric, with 95% confidence intervals over at least 10 rollouts per object. Include a control model trained with L_gen removed. If the model does not significantly beat the control or achieve meaningful accuracy on unseen objects, the abstract's description claim should be removed or explicitly marked as illustrative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract, Fig 1, and Sec I claim FuSe unlocks zero-shot descriptions of grasped objects, e.g., 'The object I have grasped feels soft and squishy'. This capability is attributed to the language generation loss L_gen in Sec III. However, Sec IV reports no evaluation of generated descriptions: no held-out objects, no human ratings, no accuracy or semantic-similarity metric, and not even qualitative examples from the deployed policy. The only empirical evidence involving L_gen is the Fig 7 ablation, which measures shopping-bag grasping success, not description quality. Because all training annotations for L_gen are templated plus ChatGPT rephrasings, the generative head could be memorizing per-object or per-template associations; unseen-object description performance is required to support 'zero-shot'. The manuscript's limitation section mentions training resources and observation history but does not acknowledge this missing evaluation. This gap does not invalidate the action-success results, but it removes evidential support for one of the three headline capabilities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FuSe, a fine-tuning recipe that extends pre-trained generalist visuomotor policies (Octo and a 3B PaliGemma-based VLA) to heterogeneous sensor modalities, specifically touch and audio, by using natural language as a common cross-modal grounding. The training objective combines the original action loss with a multimodal CLIP-style contrastive loss and a language generation loss. The authors collect and release a 26,866-trajectory real-world dataset with vision, touch, audio, proprioception, action, and language annotations across tabletop grasping, shopping-bag grasping, and button-pressing tasks. Experiments report that FuSe outperforms vision-only fine-tuning, from-scratch training, and a ResNet baseline, especially in the partially observable shopping-bag task, and that it enables multimodal prompting, compositional cross-modal prompting, and zero-shot descriptions of grasped objects.","tokens_in":13161,"tokens_out":4943,"duration_ms":52430,"significance":"If the claims hold, FuSe is a practical and timely recipe for adapting large generalist robot policies to non-visual sensors without large paired multimodal datasets. The paper's strengths include a large open-sourced real-world dataset, evaluation on unseen test objects, two architecturally different backbone policies, and an ablation showing that both auxiliary losses contribute to action success. The work is potentially useful for the broader robot-learning community, but the current version overclaims one of its three headline capabilities and lacks uncertainty quantification on the main quantitative results.","major_comments":[{"comment":"The headline capability 'zero-shot descriptions of grasped objects' is never evaluated anywhere in the experiments. Section III introduces the generative loss L_gen, but Section IV reports no metric for generated descriptions (e.g., object retrieval accuracy, semantic similarity, human evaluation), no held-out-object evaluation, and not even qualitative example outputs from the deployed policy. Figure 7 only measures grasp success, and Figure 6 uses the generative head only as an intermediate step in an action-conditioning pipeline. Since all L_gen training annotations are templated and ChatGPT-rephrased, the model could be retrieving memorized template associations; evidence that generated descriptions generalize to unseen objects is required before claiming 'zero-shot descriptions'. Either add a direct evaluation of the generated text or remove/qualify this capability claim.","section":"Abstract, Fig. 1, Section IV"},{"comment":"The main quantitative results are presented as bar-chart point estimates without error bars, confidence intervals, or significance tests. The text states that each scenario is run for 5 rollouts, but the plotted aggregated success rates do not show variance. This makes the central 'over 20% improvement' claim statistically unquantifiable as stated, and some comparisons appear close (for example, in Table I the touch-ambiguous grasp success is 0.35 in both the tactile-only and vision-plus-tactile conditions). Please report per-condition trial counts and standard errors or confidence intervals, and perform significance tests where appropriate, or explicitly present the results as preliminary.","section":"Section IV-A, Figures 5-8"},{"comment":"The phrase 'increase success rates by over 20% compared to all considered baselines' is undefined. It does not specify whether this is an absolute percentage-point increase, a relative improvement, an average over tasks, or the minimum improvement across baselines. The per-task figures show different gaps, so this headline number must be accompanied by the exact aggregation rule and the underlying per-task numbers.","section":"Abstract"},{"comment":"The use of 'zero-shot' needs a precise definition. The evaluations include unseen test objects, but the language prompts appear to be drawn from the same templated-and-rephrased instruction distribution used during training, so calling the prompting 'zero-shot' is ambiguous: the policy may not be zero-shot with respect to the instruction modality combination. Please define which aspects are unseen at training time (objects, prompts, modality combinations, or task compositions) and state explicitly what was excluded from the training data for each reported capability.","section":"Section IV-B, IV-C"}],"minor_comments":[{"comment":"There are typos in the prose: 'hetereogeneous' in Section I and 'heteregoneous' in Section III should be corrected.","section":"Sections I and III"},{"comment":"The phrase 'the pre-training phrase' should read 'the pre-training phase'; the observation that OXE pretraining appears to harm language understanding is stated but not quantified, so consider adding a brief supporting result or softening the claim.","section":"Section IV-G"},{"comment":"The column layout of Table I is difficult to parse because it combines condition labels with success metrics across two task rows; consider using separate, clearly labeled columns for each condition and metric, and include the number of trials per condition.","section":"Table I"},{"comment":"The ablation in Figure 7 appears to report a single point estimate per condition; please state whether these results are averaged over multiple seeds or rollouts and report the associated variance.","section":"Section IV-F, Figure 7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk in the current manuscript is overclaiming the description-generation capability, which is presented as a headline contribution but lacks any direct evaluation. The action-success results are more solid, though they would benefit from error bars and a precise definition of the headline improvement. Both issues are fixable within a revision: either add a proper evaluation of generated descriptions on held-out objects or remove the claim, and report uncertainty on the main bar charts. The paper otherwise fits the venue's scope and the related-work coverage appears appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: FuSe is a well-executed finetuning recipe for adding touch and audio to generalist robot policies, and the action-success results in real-world tasks look believable. The reader's conditional verdict is fair, and the stress-test note lands: one of the three headline capabilities, zero-shot descriptions of grasped objects, is claimed in the abstract and Figure 1 but never actually evaluated.\n\nWhat's new and what works: FuSe combines a CLIP-style contrastive loss and a sensory-grounded language generation loss to tie new sensor modalities (touch, audio) to natural language, then finetunes a pretrained generalist policy. The same recipe works on two very different backbones, Octo and a 3B PaliGemma VLA. They also contribute a 27K-trajectory dataset with vision, touch, audio, language, and actions, and they open-source the data, code, and models. The experiments cover three real-robot tasks with multiple baselines, ablations, and held-out objects. The shopping-bag task is the right stress test: vision is occluded and touch/audio genuinely help. The ablations show both auxiliary losses matter and that training from scratch fails, which supports the central claim that pretraining plus cross-modal grounding is the key.\n\nSoft spots, in proportion: First, the main bar charts (Figures 5, 6, 8) lack error bars and significance tests. With only 5 rollouts per scenario, some reported gaps could be within noise; this is a moderate issue, easily fixed. Second, the \"over 20%\" improvement is vague—absolute percentage points or relative? It should be pinned down. This is minor. Third, and most important, the zero-shot description capability is asserted, not demonstrated. Section IV has no evaluation of generated descriptions: no held-out objects, no human ratings, no semantic similarity metric, not even a qualitative example from the deployed policy. The only evidence tying to the generative head is the shopping-bag ablation (which measures grasping success, not description quality) and the multi-step compositional task (which uses the generative head but never checks the generated text). Given that training annotations are templated plus ChatGPT rephrasing, the generative head could be memorizing associations; unseen-object description performance is needed to support \"zero-shot.\" The limitation section mentions compute but not this gap. This does not invalidate the action-success results, but it removes evidential support for one of the three advertised capabilities.\n\nWho this is for: researchers working on adding non-visual sensors to generalist robot policies. They will get a usable recipe, a useful dataset, and a clear set of baselines. The citation pattern looks fine, with relevant prior work on touch-language-vision alignment and audio-visual imitation acknowledged. The paper deserves a serious referee—the core contribution is solid and the gap is fixable. I would send it to review, requesting either a direct evaluation of generated descriptions or a recalibrated abstract that does not claim the capability as demonstrated.","headline":"FuSe's action-success results are solid and believable, but the 'zero-shot descriptions' headline capability is never directly evaluated; the paper deserves review, but the claims need fixing.","tokens_in":13559,"tokens_out":3342,"would_cite":true,"duration_ms":33472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning generalist robot policies with touch and audio via language grounding raises task success by over 20 percent.","keywords":["robot manipulation","multimodal finetuning","language grounding","tactile sensing","audio sensing","generalist policies","vision-language-action models","zero-shot prompting"],"falsifier":"Give FuSe the same training data but replace every touch- and audio-related phrase in the instructions with random or visual-only language, then test on prompts such as \"pick the object that feels corded\"; if the policy still succeeds, the claimed language grounding is not what drives the result. A cleaner check is to record whether test success on the shopping-bag task drops when the tactile images are corrupted, which would show the policy actually depends on the new sensors.","tokens_in":125,"feed_emoji":"🤖","tokens_out":6107,"duration_ms":118330,"temperature":0.7,"pith_summary":"This paper claims that a robot policy pre-trained to act from vision and language can be extended to use touch and audio with only a modest amount of new data, if natural language is used as the shared bridge between all sensors. The authors propose FuSe, a finetuning recipe that adds two auxiliary losses to the standard action-matching loss: a contrastive loss that aligns each sensory view with matched language instructions, and a generative loss that predicts the instruction from the sensory observation. On a real robot, FuSe policies outperform vision-only finetuning and from-scratch training by over 20 percent, and they gain zero-shot abilities such as following prompts like \"pick the round object that feels squishy\" or \"press the button that plays piano.\" The same recipe transfers to a 3-billion-parameter vision-language-action model, not just the transformer-based generalist policy used for the main experiments.","feed_headline":"Language grounding lets robot policies use touch and audio","feed_subtitle":"A two-loss finetuning recipe lifts real-world task success by over 20 percent over vision-only baselines.","key_machinery":"The central mechanism is the pair of auxiliary losses that ground every new sensor in language. A multimodal contrastive loss takes observation embeddings built from any subset of vision, touch, and audio, and pulls them toward matched language instructions while pushing them away from unmatched ones; a language-generation head then predicts the instruction text from those same embeddings, using one shared transformer with modality tokens to distinguish inputs. Together these losses force the new sensors to attach to meanings the pre-trained policy already understands, which is what allows zero-shot cross-modal prompting without joint multimodal-action training data. A pre-trained tactile encoder and a spectrogram-based audio encoder provide the raw sensor features that feed this grounding process.","core_discovery":"The central claim is that natural language is a sufficient common embedding space to connect vision, touch, and audio for action selection, so that paired multimodal-action data is not required to teach a policy to reason across senses. FuSe finetunes a pre-trained generalist policy with a behavior-cloning loss plus a multimodal contrastive loss and a language-generation loss, both computed over every combination of the available modalities. This makes the touch and audio encoders learn to project their signals into the semantic vocabulary the policy already knows from vision and language pre-training. The finetuned policy can then follow multimodal prompts, compose concepts across modalities such as matching a button's sound to an object's color, and describe a grasped object in words, even for objects that were absent from the finetuning data.","pith_inferences":["An untested consequence is that the same language-grounding recipe should extend to other rare modalities, such as proprioceptive force cues or thermal sensing, whenever those signals can be described in words; the paper only demonstrates touch and audio.","If the auxiliary losses are the active ingredient, then FuSe should also improve when the language annotations are generated automatically from sensor readings, which would reduce the human annotation burden in a way the paper does not evaluate.","The gap on unseen objects suggests FuSe learns property-level semantics rather than memorizing training scenes; a stress test that scales the number of objects and distractors would show whether the 20 percent advantage persists as scene complexity grows.","The generative head could be reused as an automatic annotator: a FuSe policy that can describe what it grasps could label new multimodal demonstrations, bootstrapping larger training sets for future multimodal policies."],"forward_implications":["Adding touch and audio to a vision-only generalist policy raises average task success by over 20 percent, with the largest gain in the occluded shopping-bag scenario where vision is least reliable.","FuSe policies can follow zero-shot multimodal prompts that disambiguate objects by touch or sound when a single modality is ambiguous.","The same policy can perform compositional cross-modal reasoning, such as pressing the training button that plays the same sound as a novel button, and can describe a grasped object in words without explicit training for that description task.","Both auxiliary losses are necessary; ablations show that omitting either the contrastive loss or the generative loss degrades performance, particularly on unseen test objects.","The recipe transfers across architectures, working for both a transformer-based generalist policy and a 3-billion-parameter vision-language-action model."],"supporting_citations":[{"why":"Supplies the pre-trained transformer-based generalist policy that FuSe finetunes in the main experiments.","marker":"[4]"},{"why":"Provides the large multi-robot dataset used to pre-train the backbone, which FuSe preserves through finetuning.","marker":"[9]"},{"why":"Supplies the pre-trained tactile encoder used to featurize touch images before they enter the policy.","marker":"[14]"},{"why":"Supplies the contrastive alignment objective that FuSe adapts to match sensory views with language instructions.","marker":"[49]"},{"why":"Supplies the vision-language backbone used to build the 3-billion-parameter vision-language-action variant of FuSe.","marker":"[28]"},{"why":"Used to generate rephrased language templates that widen the range of free-form instructions at test time.","marker":"[50]"}],"fun_headline_variants":["Language grounds touch and audio for robot policies","Robot policies get touch and audio via language grounding","Zero-shot cross-modal robot control from language grounding","Touch and audio join robot vision using language","Multimodal robot manipulation grounded in language"],"cache_read_input_tokens":15744,"weakest_assumption_plain":"The method assumes that language descriptions of what objects feel and sound like carry enough information to teach the policy which sensor signals matter, so that the contrastive and generative losses can substitute for paired cross-modal action data; if the annotations miss the task-relevant sensory features, the new modalities will be ignored.","fun_headline_variants_meta":{"raw":{"variants":["Language grounds touch and audio for robot policies","Robot policies get touch and audio via language grounding","Zero-shot cross-modal robot control from language grounding","Touch and audio join robot vision using language","Multimodal robot manipulation grounded in language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1326,"prompt_tokens":949,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":565,"tokens_out":377,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:25:13.810482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give FuSe the same training data but replace every touch- and audio-related phrase in the instructions with random or visual-only language, then test on prompts such as \"pick the object that feels corded\"; if the policy still succeeds, the claimed language grounding is not what drives the result. A cleaner check is to record whether test success on the shopping-bag task drops when the tactile images are corrupted, which would show the policy actually depends on the new sensors.","supporting_citations":[{"cited_title":"Octo: An open-source generalist robot policy,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained transformer-based generalist policy that FuSe finetunes in the main experiments."},{"cited_title":"Open X-Embodiment: Robotic learning datasets and RT-X models,","cited_arxiv_id":null,"evidence_quote":"Provides the large multi-robot dataset used to pre-train the backbone, which FuSe preserves through finetuning."},{"cited_title":"A touch, vision, and language dataset for multimodal alignment,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained tactile encoder used to featurize touch images before they enter the policy."},{"cited_title":"Chatgpt: Optimizing language models for dialogue,","cited_arxiv_id":null,"evidence_quote":"Used to generate rephrased language templates that widen the range of free-form instructions at test time."}],"review_version":1}