Pith. sign in

REVIEW 1 major objections 6 minor 44 references

S2A2 shows that adding acoustic spatial maps and spectrograms to vision lets imitation-learning robots pick the sounding object and place it in the matching box.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:44 UTC pith:OPCAGQLK

load-bearing objection Genuinely new integration of array audition with imitation learning; the key ablation is confounded and real-robot evidence is thin, but the core S2A2-vs-vision result survives. the 1 major comments →

arxiv 2607.26047 v1 pith:OPCAGQLK submitted 2026-07-28 cs.RO

S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information

Kaneyoshi Hiratsuka , Benjamin Yen , Ryosuke Kojima This is my paper
classification cs.RO
keywords imitation learningrobot manipulationacoustic spatial mapsound source localizationspectrogrammultimodal policyMUSICSpotforming
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces acoustic-aware manipulation tasks—picking and placing tasks where the target or destination is determined by sound, not by what is visible. It proposes S2A2, a framework that feeds a robot policy three kinds of observation: an RGB image, a spatial map of where sound is coming from, and a spectrogram of what the sound is. The central finding is that when a task needs both sound position and sound identity, S2A2 succeeds far more often than vision-only policies—up to about 90% versus about 10% on the most demanding simulated task. The paper also shows that giving a policy an acoustic representation irrelevant to the task can hurt performance, and that the benefit depends on which policy architecture is used. Real-robot trials confirm the pattern on two of the tasks.

Core claim

The paper's central claim is that manipulation policies for tasks where sound determines the target should integrate two complementary acoustic representations alongside vision: a MUSIC-based acoustic spatial map, which projects the likelihood of a sound source's direction onto a 2D workspace plane, and a Spotforming-derived spectrogram, which isolates and amplifies the target source's timbre. Neither representation alone is enough when a robot must both find which object is sounding and decide which box to place it in. In simulation, S2A2 reaches 78.7–89.7% success on the Localization & Identification task depending on the policy, while vision-only baselines sit at 9.3–11.0%. Real-robot exp

What carries the argument

Two acoustic processing pipelines feed a multimodal policy. The acoustic spatial map pipeline uses MUSIC on each microphone array to compute direction-of-arrival scores, then projects them with distance attenuation onto a 224×224 plane corresponding to the workspace. The spectrogram pipeline picks the highest peak in that map, beamforms toward it, applies Spotforming (NMF-based source separation) to suppress noise and sidelobes, and produces a single-channel log-power spectrogram. Both acoustic representations are encoded by small ResNets and concatenated with RGB-vision features and proprioception before being passed to the action-generation policy, which the paper implements with four diff

Load-bearing premise

The pipeline assumes at most one task-relevant dominant sound source at a time: the spectrogram branch takes only the highest peak in the acoustic spatial map, and MUSIC is run with a fixed assumed source count, so competing simultaneous sources would degrade the inputs and undermine the claimed policy gains—the paper itself defers multi-source environments to future work.

What would settle it

Run the Localization & Identification task with two sound-emitting objects present, one that should be grasped and one distractor, while keeping the visual observation unchanged; if success drops to the vision-only baseline level, the single-source assumption is the load-bearing element of the method.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In tasks requiring both sound position and timbre, providing only one acoustic representation leaves success near vision-only levels; both are necessary for the reported gains.
  • Acoustic input that is irrelevant to the task can reduce success, so the framework implies that representations should be matched to the acoustic capabilities a task actually requires.
  • The magnitude of the acoustic benefit depends strongly on the policy: the largest gains appear for the transformer action-chunking and diffusion-style policies, while a discretizing policy and a large pretrained flow policy show smaller or more variable improvement.
  • Real-robot trials on the Localization & Identification and Exploratory tasks reproduce the simulation result that S2A2 beats a vision-only baseline, indicating the acoustic representations transfer beyond synthetic audio.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to replace single-peak selection with multi-source separation, which could extend S2A2 to environments with several simultaneous sound sources—the paper's own stated limitation.
  • Because the assumed number of MUSIC sources is fixed (3 in simulation, 1 on the real robot), a future version could estimate source count adaptively rather than requiring it as a hyperparameter.
  • The observed performance drop when unnecessary acoustic channels are present suggests that learned gating or task-conditioned modality selection could improve sample efficiency beyond simply concatenating features.
  • The sensitivity analysis showing an optimal number of microphone arrays points toward a scaling question: how sparse can the microphone array be before the localization signal is too weak for the policy to use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper introduces 'acoustic-aware manipulation tasks' and an imitation-learning framework, S2A2, that fuses RGB images with two acoustic representations: a MUSIC-derived acoustic spatial map (which object is sounding) and a Spotforming-based spectrogram (what sound it is). Four policy backbones (ACT, Diffusion Policy, VQ-BeT, π0) are evaluated in simulation on four tasks designed to require source localization, sound identification, both, or active exploration; S2A2 is reported to give the largest gains on the Localization & Identification task (e.g., 78.7±2.1% with ACT and 89.7±2.1% with Diffusion Policy vs. 11.0±3.5% and 9.3±6.1% vision-only baselines). A real-robot experiment with ACT on the L&I and exploratory tasks is reported as confirming the simulation trend. The central claim is that acoustic spatial and spectral streams make independent, complementary contributions to policy learning.

Significance. If the results hold, the paper makes a useful contribution by connecting robot audition (MUSIC, Spotforming, microphone arrays) to imitation learning and by proposing a benchmark of four tasks that isolate spatial, spectral, and exploratory acoustic demands. The simulation study is systematic: three seeds, four policy families, task-matched ablations, and sensitivity analyses in Appendix C. The real-robot setup is described in unusual detail (Appendix E), which is commendable. However, the main mechanistic claim about independent contributions of spatial and spectral streams rests on an ablation that is confounded as currently defined, and the real-robot confirmation rests on a single-seed, numerically under-reported experiment. These issues are fixable and do not appear to invalidate the headline S2A2-vs-vision result, but they do affect the strength of the paper's internal causal story.

major comments (1)
  1. [§4.2, Appendix E, Fig. 5(b)] The real-robot validation is reported with a single seed: Appendix E states 'evaluated the success rate over 100 trials with a single seed,' and no numeric success rates or confidence intervals are given in the main text or appendix. Figure 5(b) shows bars but the actual values and error bars are not reported in the text. With one seed and no trial-level statistics, the claim in Section 4.2 that the real-robot experiments 'confirm' the simulation findings is weaker than the simulation evidence. Please report the exact per-condition success counts, provide multiple seeds or at least a confidence interval, and clarify whether the two tasks were evaluated on the same policy checkpoint.
minor comments (6)
  1. [Abstract] Grammar: 'We implemented S2A2 models that integrates policies' should be 'that integrate policies.'
  2. [Fig. 5(b)] The bar chart is difficult to read without numeric labels; add success-rate values on the bars or in a table.
  3. [§2.3] The sentence 'This work can be positioned as the first to connect these robot audition techniques to manipulation imitation learning' is stronger than the evidence; suggest softening to 'one of the first' or citing concurrent work if any.
  4. [Acknowledgments] The placeholder text ('If a paper is accepted...') is not appropriate for a submitted manuscript and should be replaced by actual acknowledgments or removed.
  5. [§A.3] The MUSIC assumed source count differs between simulation (3) and real robot (1) without an analysis of sensitivity to this parameter. Since multi-source robustness is explicitly deferred to future work, a brief note explaining why the simulation uses 3 when the tasks contain one source would help.
  6. [§B.1] The t-SNE visualizations are discussed qualitatively but no quantitative separation metric (e.g., silhouette score) is provided. Since the claim that latent structure explains success is central to Fig. 4, a simple numeric measure would strengthen the presentation.

Circularity Check

1 steps flagged

w/o Spat ablation is self-definitional because the spectrogram it retains is generated from the acoustic spatial map; central S2A2-vs-baseline result remains independent.

specific steps
  1. self definitional [Section 3.3.2 (Spectrogram Pipeline) / Section 4.1 (w/o Spat. condition) / Appendix D.2]
    ""From the acoustic spatial map S, the source presence likelihood is computed, and peak extraction selects source candidates. Since only one source is present at a time, the highest peak is taken as the candidate. Source separation by Spotforming is applied to the candidate to generate the spectrogram." ... "w/o Spat. removes the acoustic spatial map pipeline, using visual information and acoustic signal information from the spectrogram.""

    The w/o Spat. condition is defined as removing the acoustic spatial map while keeping the spectrogram, but the spectrogram pipeline itself is defined in terms of the spatial map: peak extraction selects the source from the acoustic spatial map S, and Appendix D.2 beamforms toward that selected azimuth. Therefore the spectrogram input is not independent spectrogram-only signal information; it carries acoustic spatial information by construction. Table 2 results for w/o Spat. (and the claimed independent contribution of the spectral stream) are therefore not cleanly separable from spatial information, and the L&I gap between w/o Spat. and S2A2 cannot be attributed solely to the presence of the policy-level spatial-map channel.

full rationale

The derivation itself is not circular: acoustic features come from external MUSIC/Spotforming algorithms, policies are trained with the behavioral-cloning loss Eq. (1), and success is measured against task outcomes (grasp correct object, place in correct box) that are independent of the acoustic feature values. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and self-citations (e.g., [34] co-authored by Kojima) are not load-bearing. Selecting hyperparameters on the same tasks in Appendix C is standard tuning, not circularity. The one concrete circular/self-definitional issue is the w/o Spat. ablation, whose spectrogram is generated from the spatial map by construction, so the independent-contribution claim for spectral information is confounded. Because the main S2A2-versus-baseline result does not depend on this ablation and the policy outputs are measured externally, the overall circularity is moderate rather than central.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced; the acoustic spatial map and Spotforming spectrogram are representations of existing signals. The central claim depends on standard array-processing assumptions and on hand-set/tuned constants in the audio pipeline. The number of arrays and update rate are tuned on the same benchmark, which is the most visible self-referential element in the ledger.

free parameters (6)
  • Number of microphone arrays N = 4
    Chosen as the best value on the localization task in the sensitivity analysis (Appendix C, Fig. 10a). This tunes the input dimensionality of the acoustic spatial map to the evaluation tasks.
  • Acoustic information update rate = 10 Hz
    Selected from a 1-30 FPS sweep (Fig. 10b); performance saturated at 10 Hz. Tuned on the same task suite used for final success-rate reporting.
  • MUSIC assumed source count = 3 (simulation) / 1 (real robot)
    A fixed modeling input to the noise-subspace estimation (Appendix A.3). The simulation sets 3 even though one task-relevant source is present; the real robot uses 1.
  • Spotforming NMF constants (R, threshold delta, max mask ratio) = 50, 1.6e-3, 0.2
    Hand-set constants in the NMF source-separation stage (Section D.2) with no sensitivity analysis; they control how much of the spectrogram is retained.
  • Distance attenuation d_floor and gamma = 10.0 m and 1.0 (sim); 0.0 and 0.0 (real)
    Hand-chosen in Eq. (11) to stabilize the projection of DOA scores to the 2D map; values differ between simulation and real robot (Appendix A.3).
  • Exploratory-task sound trigger speed = 0.5 mm/s
    Arbitrary simulation threshold at which the 'active' object starts emitting sound when moved (Appendix H); defines the task.
axioms (5)
  • standard math MUSIC direction-of-arrival estimation: the noise subspace is orthogonal to steering vectors of the true source directions, given the assumed number of sources.
    Used in Eq. (10) of Appendix D.1; standard result in array signal processing.
  • domain assumption Pyroomacoustics image-source simulation with a 10m x 10m x 3m room and third-order reflections is a faithful enough proxy for real acoustic conditions to transfer policy improvements.
    The main evidence is simulation; fidelity of the acoustic simulation is not independently measured against the real recordings.
  • domain assumption A single task-relevant sound source dominates, so selecting the highest peak in the acoustic spatial map yields the correct source for Spotforming.
    Peak selection (Section 3.3.2) and MUSIC source count assume at most one source; multi-source environments are deferred in Section 6.
  • domain assumption Expert demonstrations generated with privileged information (ground-truth sound position and sound label) are appropriate training data for imitation learning.
    Appendix H describes the expert that uses privileged information; standard IL assumption, but demonstration quality controls the ceiling.
  • domain assumption The objects are visually indistinguishable in the provided images, so vision-only policies cannot solve the tasks by appearance.
    Task design in Section 3.2 and the real setup rely on this; any accidental visual cue would weaken the baseline comparison.

pith-pipeline@v1.3.0-alltime-deepseek · 18904 in / 17005 out tokens · 158196 ms · 2026-08-01T00:44:29.405502+00:00 · methodology

0 comments
read the original abstract

Acoustic information provides rich cues about object location, material properties, and changes caused by contact or motion. This paper introduces a new set of acoustic-aware manipulation tasks for imitation learning, in which robots must use auditory cues to determine manipulation targets. These tasks require sound source localization and identification for active exploration in robotic manipulation. Also, we propose a multimodal imitation learning framework, Spatial-Spectral Audio Action (S2A2), that integrates visual features with acoustic spatial and acoustic signal information for the acoustic-aware manipulation tasks. We implemented S2A2 models that integrates policies such as ACT, Diffusion Policy, VQ-BeT, and $\pi_0$, into our framework. Simulation experiments showed that the proposed method is the most effective for tasks requiring both position and timbre. Furthermore, real-robot experiments confirm the applicability of the proposed tasks and framework to real-world manipulation.

Figures

Figures reproduced from arXiv: 2607.26047 by Benjamin Yen, Kaneyoshi Hiratsuka, Ryosuke Kojima.

Figure 1
Figure 1. Figure 1: Overview of the S2A2 framework. 2.2 Manipulation with Acoustic Modality The acoustic modality has been used in many perception tasks, including material identification [20, 32], estimation of granular material flow rate and amount [16, 17], and liquid height estimation [18, 19]. For robotic manipulation, policy learning [6, 14] and action generation [12] using contact sounds have been proposed, and pretrai… view at source ↗
Figure 2
Figure 2. Figure 2: Simulation settings. A simulation environment integrating the physics simulator Genesis [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task success rate versus training steps. Training curves of the four policies (ACT, Diffu [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Latent space visualization. (a) Comparison for Diffusion Policy on the L&I task. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Real-robot examples. (a) Top: the robot during the exploratory task. Middle: the acoustic [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: t-SNE visualization of ACT latent representations for the four acoustic-aware manipula [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: t-SNE visualization of Diffusion Policy latent representations for the four acoustic-aware [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: t-SNE visualization of VQ-BeT latent representations for the four acoustic-aware manip [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: t-SNE visualization of π0 latent representations for the four acoustic-aware manipulation tasks. The four quadrants correspond to Localization, Identification, Exploratory, and L&I task. In each quadrant, the rows show the same embedded samples colored by different task-related vari￾ables: success/failure, sound coordinate, or sound type. The columns compare representations from the intermediate layer and … view at source ↗
Figure 10
Figure 10. Figure 10: Sensitivity analysis of the audio representation. (a) Effect of the number of microphone [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example real-robot sequence of the S2A2 model with ACT performing the L&I task. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example real-robot sequence of the S2A2 model with ACT performing the exploratory [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991. doi:10.1162/neco.1991.3.1.88

    doi:10.1162/neco.1991.3.1.88 1991

  2. [2]

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration.Robotics and Autonomous Systems, 57(5):469–483, 2009. ISSN 0921-8890. doi:https://doi.org/10.1016/j.robot.2008.10.024. URLhttps://www.sciencedirect.com/ science/article/pii/S0921889008001772

    doi:10.1016/j.robot.2008.10.024 2009

  3. [3]

    T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An al- gorithmic perspective on imitation learning. 7(1):1–179. ISSN 1935-8253, 1935-8261. doi:10.1561/2300000053. URLhttps://www.emerald.com/ftrob/article/7/1-2/1/ 1328576/An-Algorithmic-Perspective-on-Imitation-Learning

    doi:10.1561/2300000053 1935

  4. [4]

    M. U. Din, W. Akram, L. S. Saoud, J. Rosell, and I. Hussain. Vision language action models in robotic manipulation: A systematic review, 2025. URLhttps://arxiv.org/abs/2507. 10672

    2025

  5. [5]

    M. Du, O. Y . Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. InRobotics: Science and Systems XVIII. Robotics: Science and Systems Foundation. ISBN 978-0-9923747-8-5. doi:10.15607/RSS.2022.XVIII.009. URL http://www.roboticsproceedings.org/rss18/p009.pdf

    work page doi:10.15607/rss.2022.xviii.009 2022

  6. [6]

    H. Li, Y . Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. InCoRL, 2022

    2022

  7. [7]

    Y . Chen, M. V . d. Merwe, A. Sipos, and N. Fazeli. Visuo-tactile transformers for manipu- lation. In K. Liu, D. Kulic, and J. Ichnowski, editors,Proceedings of The 6th Conference on Robot Learning, volume 205 ofProceedings of Machine Learning Research, pages 2026–2040. PMLR, 14–18 Dec 2023. URLhttps://proceedings.mlr.press/v205/chen23d.html

    2026

  8. [8]

    Ablett, O

    T. Ablett, O. Limoyo, A. Sigal, A. Jilani, J. Kelly, K. Siddiqi, F. Hogan, and G. Dudek. Mul- timodal and force-matched imitation learning with a see-through visuotactile sensor.IEEE Transactions on Robotics, 41:946–959, 2025. doi:10.1109/TRO.2024.3521864

    arXiv 2025

  9. [9]

    F. Liu, C. Li, Y . Qin, J. Xu, P. Abbeel, and R. Chen. Vitamin: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface, 2025. URLhttps://arxiv.org/ abs/2504.06156

    Pith/arXiv arXiv 2025

  10. [10]

    Y . Li, Y . Chen, Z. Zhao, P. Li, T. Liu, S. Huang, and Y . Zhu. Simultaneous tactile-visual perception for learning multimodal robot manipulation, 2026. URLhttps://arxiv.org/ abs/2512.09851

    arXiv 2026

  11. [11]

    Suresh, H

    S. Suresh, H. Qi, T. Wu, T. Fan, L. Pineda, M. Lambeta, J. Malik, M. Kalakrishnan, R. Calandra, M. Kaess, J. Ortiz, and M. Mukadam. Neuralfeels with neural fields: Vi- suotactile perception for in-hand manipulation.Science Robotics, 9(96):eadl0628, 2024. doi:10.1126/scirobotics.adl0628. URLhttps://www.science.org/doi/abs/10.1126/ scirobotics.adl0628. 9

    work page doi:10.1126/scirobotics.adl0628 2024

  12. [12]

    Thankaraj and L

    A. Thankaraj and L. Pinto. That sounds right: Auditory self-supervision for dynamic robot manipulation. In J. Tan, M. Toussaint, and K. Darvish, editors,Proceedings of The 7th Confer- ence on Robot Learning, volume 229 ofProceedings of Machine Learning Research, pages 1036–1049. PMLR, 06–09 Nov 2023. URLhttps://proceedings.mlr.press/v229/ thankaraj23a.html

    2023

  13. [13]

    Mejia, V

    J. Mejia, V . Dean, T. Hellebrekers, and A. Gupta. Hearing touch: Audio-visual pretraining for contact-rich manipulation, 2024. URLhttps://arxiv.org/abs/2405.08576

    Pith/arXiv arXiv 2024

  14. [14]

    Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, B. Burchfiel, and S. Song. Maniwav: Learning robot manipulation from in-the-wild audio-visual data, 2024. URLhttps://arxiv.org/ abs/2406.19464

    Pith/arXiv arXiv 2024

  15. [15]

    R. Wang, H. Geng, T. Li, P. Wu, F. Wang, G. Anumanchipalli, T. Darrell, B. Li, P. Abbeel, J. Malik, and A. A. Efros. The sound of simulation: Learning multimodal sim-to-real robot policies with generative audio. In J. Lim, S. Song, and H.-W. Park, editors,Proceedings of The 9th Conference on Robot Learning, volume 305 ofProceedings of Machine Learning Res...

    2025

  16. [16]

    Clarke, T

    S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors,Proceedings of The 2nd Conference on Robot Learning, volume 87 ofProceedings of Machine Learning Research, pages 529–550. PMLR, 29–31 Oct 2018. URLhttps:// proceedin...

    2018

  17. [18]

    Liang, S

    H. Liang, S. Li, X. Ma, N. Hendrich, T. Gerkmann, F. Sun, and J. Zhang. Making sense of audio vibration for liquid height estimation in robotic pouring. In2019 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), page 5333–5339. IEEE Press, 2019. doi:10.1109/IROS40897.2019.8968303. URLhttps://doi-org.kyoto-u. idm.oclc.org/10.1109/...

    arXiv 2019

  18. [19]

    Z. Wang, G. Tian, and H. Pan. Robot gaining robust pouring skills through fusing vision and audio.ISA Transactions, 135:428–437, 2023. ISSN 0019-0578. doi:https://doi.org/10.1016/ j.isatra.2022.09.022. URLhttps://www.sciencedirect.com/science/article/pii/ S0019057822004815

    2023

  19. [20]

    M. Eppe, M. Kerzel, E. Strahl, and S. Wermter. Deep neural object analysis by interactive auditory exploration with a humanoid robot. In2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page 284–289. IEEE Press, 2018. doi:10.1109/IROS. 2018.8593838. URLhttps://doi-org.kyoto-u.idm.oclc.org/10.1109/IROS.2018. 8593838

    arXiv 2018

  20. [21]

    Macesanu, B

    L. Macesanu, B. Folefack, S. Singh, R. Ray, B. Abbatematteo, and R. Mart ´ın-Mart´ın. Caver: Curious audiovisual exploring robot, 2026. URLhttps://arxiv.org/abs/2511.07619

    arXiv 2026

  21. [22]

    MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

    G. Tatiya, J. Francis, H.-H. Wu, Y . Bisk, and J. Sinapov. MOSAIC: Learning Unified Multi- Sensory Object Property Representations for Robot Learning via Interactive Perception.arXiv e-prints, art. arXiv:2309.08508, Sept. 2023. doi:10.48550/arXiv.2309.08508

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.08508 2023

  22. [23]

    X. Wei, H. Zhang, X. Cao, S. Xie, W. Ge, Y . Li, and C. Wang. Audio-vla: Adding contact audio perception to vision-language-action model for robotic manipulation, 2025. URLhttps: //arxiv.org/abs/2511.09958. 10

    arXiv 2025

  23. [24]

    Nakadai and H

    K. Nakadai and H. G. Okuno. Robot audition and computational auditory scene analysis. 2 (9):2000050. ISSN 2640-4567, 2640-4567. doi:10.1002/aisy.202000050. URLhttps:// advanced.onlinelibrary.wiley.com/doi/10.1002/aisy.202000050

    doi:10.1002/aisy.202000050

  24. [25]

    Jalayer, M

    R. Jalayer, M. Jalayer, and A. Baniasadi. A review on sound source localization in robotics: Focusing on deep learning methods.Applied Sciences, 15(17):9354, Aug. 2025. ISSN 2076-

    2025

  25. [26]

    R. Schmidt. Multiple emitter location and signal parameter estimation.IEEE Transactions on Antennas and Propagation, 34(3):276–280, 1986. doi:10.1109/TAP.1986.1143830

    arXiv 1986

  26. [27]

    A. A. Salama. Direction of arrival estimation: A tutorial survey of classical and modern meth- ods, 2025. URLhttps://arxiv.org/abs/2508.11675

    Pith/arXiv arXiv 2025

  27. [28]

    Kagimoto, K

    Y . Kagimoto, K. Itoyama, K. Nishida, and K. Nakadai. Spotforming by NMF using multiple microphone arrays. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9253–9258. IEEE. ISBN 978-1-6654-7927-1. doi:10.1109/IROS47612. 2022.9981808. URLhttps://ieeexplore.ieee.org/document/9981808/

    arXiv 2022

  28. [29]

    Z. Li, A. Chapin, E. Xiang, R. Yang, B. Machado, N. Lei, E. Dellandrea, D. Huang, and L. Chen. Robotic manipulation via imitation learning: Taxonomy, evolution, benchmark, and challenges, 2025. URLhttps://arxiv.org/abs/2508.17449

    Pith/arXiv arXiv 2025

  29. [30]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V . Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. San- keti, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y . Lu, S. Levine, L. Lee, T.-W. E. Lee, I. Leal, Y . Kuang, D. Kalashnikov, R. Jul...

    2023

  30. [31]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246

    Pith/arXiv arXiv 2024

  31. [32]

    S. Jin, H. Liu, B. Wang, and F. Sun. Open-environment robotic acoustic perception for object recognition.Frontiers in Neurorobotics, V olume 13 - 2019, 2019. ISSN 1662-

    2019

  32. [33]

    J. Wang, Y . He, D. Su, K. Itoyama, K. Nakadai, J. Wu, S. Huang, Y . Li, and H. Kong. Slam- based joint calibration of multiple asynchronous microphone arrays and sound source localiza- tion.IEEE Transactions on Robotics, 40:4024–4044, 2024. doi:10.1109/TRO.2024.3410456

    arXiv 2024

  33. [34]

    Gabriel, R

    D. Gabriel, R. Kojima, K. Hoshiba, K. Itoyama, K. Nishida, and K. Nakadai. 2d sound source position estimation using microphone arrays and its application to a vr-based bird song analysis system.Advanced Robotics, 33(7-8):403–414, 2019

    2019

  34. [35]

    Yamada, K

    T. Yamada, K. Itoyama, K. Nishida, and K. Nakadai. Sound source tracking by drones with microphone arrays. In2020 IEEE/SICE International Symposium on System Integration (SII), pages 796–801. IEEE, 2020. 11

    2020

  35. [36]

    T. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRobotics: Science and Systems XIX. Robotics: Science and Systems Foundation. ISBN 978-0-9923747-9-2. doi:10.15607/RSS.2023.XIX.016. URLhttp:// www.roboticsproceedings.org/rss19/p016.pdf

    doi:10.15607/rss.2023.xix.016 2023

  36. [37]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. 44(10):1684–1704. ISSN 0278-3649, 1741-3176. doi:10.1177/02783649241273668. URLhttps://journals.sagepub.com/ doi/10.1177/02783649241273668

    doi:10.1177/02783649241273668

  37. [38]

    S. Lee, Y . Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 2699...

    2024

  38. [39]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π 0: A vision-language-action flow model for general robot control, 2026. URLhttps://arxiv. o...

    Pith/arXiv arXiv 2026

  39. [40]

    G. Authors. Genesis: A generative and universal physics engine for robotics and beyond, December 2024. URLhttps://github.com/Genesis-Embodied-AI/Genesis

    2024

  40. [41]

    Scheibler, E

    R. Scheibler, E. Bezzam, and I. Dokmani ´c. Pyroomacoustics: A python package for audio room simulations and array processing algorithms. In2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 351–355. doi:10.1109/ICASSP. 2018.8461310. URLhttp://arxiv.org/abs/1710.04196

    arXiv 2018

  41. [42]

    L. v. d. Maaten and G. Hinton. Visualizing data using t-SNE. 9(86):2579–2605. URLhttp: //jmlr.org/papers/v9/vandermaaten08a.html

  42. [43]

    D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using resid- ual quantization. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11513–11522, 2022. doi:10.1109/CVPR52688.2022.01123. 12 A Implementation Details This appendix describes the implementation details of the S2A2 model. Specifically...

    arXiv 2022

  43. [3417]

    URLhttp://dx.doi.org/10.3390/app15179354

    doi:10.3390/app15179354. URLhttp://dx.doi.org/10.3390/app15179354

    work page doi:10.3390/app15179354

  44. [5218]

    URLhttps://www.frontiersin.org/journals/ neurorobotics/articles/10.3389/fnbot.2019.00096

    doi:10.3389/fnbot.2019.00096. URLhttps://www.frontiersin.org/journals/ neurorobotics/articles/10.3389/fnbot.2019.00096

    arXiv 2019