Pith. sign in

REVIEW 5 major objections 6 minor 42 references

Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A speech LLM can learn direction from beamformed multi-channel audio, reporting 92% direction accuracy and 3.84% WER on simulated far-field speech.

desk verdict A clean, internally consistent empirical study of a reasonable SLLM-plus-beamforming idea, but the smart-glasses and cross-talk claims outrun the evidence; still deserves a referee. read the letter →

arxiv 2506.14973 v1 pith:EUADPMPT submitted 2025-06-17 eess.AS cs.AI

Jiamin Xie , Ju Lin , Yiteng Huang , Tyler Vuong , Zhaojiang Lin , Zhaojun Yang , Peng Su , Prashant Rawat
show 3 more authors
Sangeeta Srivastava Ming Sun Florian Metze
This is my paper
classification eess.AScs.AI
keywords speechlargelanguagemodeldirectionalrecognitionmulti-talkerASRsourcelocalizationbeamformingsmartglassesserializedoutputtrainingcontrastivedirectiondataaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a speech large language model, given multi-channel audio from a smart-glasses microphone array, can learn to think in terms of direction: it can output the arrival direction of a speaker and the transcript in one generative step, and it can transcribe only the speech coming from a direction named in the text prompt. The route is to serialise the direction as a text tag alongside the words (S-DOT) and to contrastively add a distractor speaker from an undesired direction during training (CDDA). If the claim holds, one model replaces separate source-localization, speech-separation, and ASR components with a single prompt-conditioned decoder. On simulated far-field audio the paper reports 92% direction accuracy and 3.84% word error rate over twelve azimuths for single-talker speech, and a 98.4% success rate for target-direction recognition in two-talker frontal conversations.

What carries the argument

Three mechanisms carry the argument. A fixed bank of NLCMV beamformers turns the seven microphone channels into twelve directional channels spaced at 30 degrees, so direction-of-arrival information appears as inter-channel amplitude differences rather than as learned microphone geometry. Serialized directional output training (S-DOT) rewrites the training transcript as alternating direction tags and word sequences, such as '30 degrees' followed by that speaker's words, so the language model learns to emit location and content in a single token stream. Contrastive direction data augmentation (CDDA) adds a distractor talker from a direction outside the target set, training the model to stay silent for non-target directions. The audio encoder and pretrained decoder stay frozen while a linear projection and an adapter are fine-tuned, except in the edge-case variant where one decoder layer is opened.

What would settle it

A decisive test would run the same model on real multi-channel recordings from a seven-microphone smart-glasses array in natural rooms, with speakers moving within an utterance, and compare direction accuracy and target-direction success rate to the simulated figures of 92% and 98.4%; a large drop would indicate the model exploits simulation-specific beamformer artifacts rather than true direction-of-arrival structure.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that spatial directivity can be taught to a speech LLM as a property of the text output rather than as a separate regression head. Directional-SpeechLLaMA takes the flattened outputs of twelve fixed NLCMV beamformers, projects them into a pre-trained audio encoder's space through a linear layer, and is fine-tuned in three stages: multi-channel ASR, serialized direction-tagged output training, and target-direction ASR fine-tuning, with contrastive direction data augmentation that mixes in a distractor speaker from a non-target direction. The reported outcome is a single autoregressive model that both localizes the source and transcribes it, reaching 92% accuracy over the twelve seen azimuths with 3.84% WER on the single-talker far-field task, and 98.4% success rate with 3.81% success WER on target-direction ASR in two-talker frontal conversations.

Load-bearing premise

The load-bearing assumption is that simulated data—audiobook speech convolved with real room impulse responses and processed by a fixed NLCMV beamformer—preserves the spatial cues of an actual smart-glasses array, and that a stationary speaker on a discrete 12-direction grid is representative of real use.

Editorial extensions

If this is right

  • A single autoregressive model can answer 'what was said and from where' in one output stream, so downstream systems could drop a separate direction-of-arrival classifier and a separate speech-separation front-end.
  • Prompting with a direction label—focus on 30 degrees, for example—suppresses bystander speech strongly enough to reach a 98.4% target-direction success rate and 3.81% success WER in simulated two-talker frontal conversations.
  • A model trained on only five frontal directions keeps 92% left/right accuracy when audio comes from directions it never saw, and distance-based label recovery lifts multi-talker unseen-direction success to 94.3%.
  • Unseen room impulse responses from seen directions cost only a small drop in success rate, from 98.4% to 95.0%, while success WER improves to 3.46%.
  • Handling the edge cases of two speakers in one direction or a requested direction that is absent requires unfreezing one decoder layer; with that change the model reports case-label accuracy of 94.3% for one speaker, 91.6% for two, and 86.3% for the empty case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct next experiment would replace the 12 discrete direction tags with a continuous azimuth regression, since S-DOT encodes direction as an output token and the grid itself is a training choice.
  • The CDDA distractor mechanism is a spatial form of negative training, so the same 'output nothing for non-targets' recipe could transfer to text-prompted extraction of arbitrary sound classes.
  • The paper does not ablate the fixed beamformers; feeding raw multi-channel audio to the linear projection would show whether the beamformer bank is necessary or just a convenient packaging of spatial cues.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Directional-SpeechLLaMA, a speech large language model that consumes multi-channel beamformed audio from a 7-microphone smart-glasses array (Project Aria geometry) to perform three tasks: single-talker far-field ASR with source localization, multi-talker target-direction ASR, and handling of edge cases such as same-direction talkers and absent target directions. The technical contributions are Serialized Directional Output Training (S-DOT), which inserts direction tokens into serialized output transcripts, and Contrastive Direction Data Augmentation (CDDA), which adds a distractor speaker from an undesired direction during training. The model is trained in three stages: multi-channel ASR, direction-aware ASR, and target-direction task fine-tuning. Experiments on simulated LibriSpeech mixtures with real room impulse responses and a fixed NLCMV beamforming front-end report 3.84% WER and 92% direction accuracy on 12 directions for single-talker ASR, and a 3.81% success WER with 98.4% success rate for target-direction ASR on five frontal directions in two-talker conversations. The paper also reports a degradation to 21.32% success WER at 25% overlap and shows that CDDA improves seen-direction performance at the cost of generalizing to unseen directions.

Significance. If the results hold, the paper demonstrates a credible path for injecting spatial directivity into speech LLMs, which is a relatively uninvestigated capability. The multi-stage fine-tuning recipe and the CDDA idea are clearly presented and the internal evaluation on simulated data is consistent and reproducible in principle. A notable strength is the use of a fixed, parameter-free beamforming front-end and the explicit separation of seen/unseen directions in parts of the evaluation. However, the significance is tempered by the absence of any real-device validation on actual smart-glasses recordings, the high WER under overlapping speech, and the relaxed success criteria for unseen directions. These gaps mean the central practical claim — that the model captures spatial cues on real smart glasses — remains unestablished.

major comments (5)
  1. [Sec. 4 (Experimental Setup) and Sec. 4.4] The paper frames the system as leveraging a smart-glasses microphone array, but all experiments are conducted on simulated LibriSpeech audio convolved with room impulse responses and processed by a fixed NLCMV beamformer; no actual Project Aria recordings are used. Real glasses involve head movement, ego-noise, sensor mismatch, off-grid source angles, and time-varying acoustic conditions, all of which alter inter-channel relationships and beam-pattern leakage. As a result, the load-bearing claim that the model effectively captures spatial cues on real smart glasses is unsupported. Please either add even a small real-device evaluation or substantially soften the application claims and describe the work as a simulation-based feasibility study.
  2. [Sec. 4.4, Table 4] The abstract claims 'bystander cross-talk suppression', but the model degrades to 21.32% success WER at 25% overlap, which is not strong suppression for a conversational setting. Only a single overlap ratio is reported, and the authors attribute the failure to the pre-trained model's lack of exposure to overlapping speech. This is a central limitation for the multi-talker scenario and should be addressed with an overlap-ratio sweep (e.g., 0%, 10%, 50%) and a discussion of whether the CDDA distractor training can be extended to overlapping distractors. As presented, the headline 'strong performance in multi-talker speech scenarios' overstates the results.
  3. [Sec. 3.2 and Table 2] The success criteria for unseen directions are relaxed: for audio from unseen directions, success requires only that the output transcription matches the target transcription, with direction labels recovered by proxy heuristics (Any Direction, Sign Match, Shortest Distance). This makes the unseen-direction metrics not comparable to the seen-direction metrics, which require an exact direction-label match. Moreover, the best model with CDDA (row (h)) does not report unseen directions at all, and the text states CDDA 'neglects unseen directions'. This sharply limits the generalizability claim. Please report seen and unseen directions with consistent success criteria and explicitly discuss the trade-off between CDDA gains and the loss of unseen-direction handling.
  4. [Sec. 4.2 and Table 2] There is no comparison with existing directional speech recognition systems, such as AGADIR [29] or the directional ASR work of Lin et al. [21], which share authors and are highly relevant. The only ASR baseline is a single-channel Whisper model, and the multi-talker ablation does not include MC-SpeechLLaMA without S-DOT. Consequently, it is unclear how much of the gain comes from the speech LLM architecture, the beamforming front-end, or the proposed S-DOT/CDDA methods. Adding comparisons with a non-LLM directional ASR system and with MC-SpeechLLaMA trained on the multi-talker task would strengthen the central claim that text-direction prompting helps.
  5. [Sec. 3.2 and Table 2] The success word error rate (sWER) is computed only on utterances where the direction prediction is correct, which can be misleading when the model frequently outputs an empty hypothesis on wrong-direction cases. The SR metric accounts for direction accuracy, but the interaction between sWER and SR is not transparent. Please clarify the exact denominator for sWER in each condition and also report an unconditional WER that treats wrong-direction and empty outputs as errors, so readers can assess the true end-to-end performance of target-direction ASR.
minor comments (6)
  1. [Sec. 2.1] The phrase 'without introducing additional parameters' is misleading because the linear projection layer and the adapter layer are trainable; the intended meaning appears to be that no extra spatial feature extractor is learned beyond the fixed beamformers. Please rephrase.
  2. [Algorithm 1] In line 8, the reference transcript R_j_multi contains only the two target directions, while a third distractor speaker is present in the audio. The algorithm should clarify that CDDA trains the model to ignore the distractor, and specify how the distractor's transcript is handled in the reference.
  3. [Table 2] The table header 'Seen Directs.Metrics (%)' and 'Unseen Directs.Metrics (%)' is confusing because the same metric names (sWER, SR) mean different things in the two blocks. Please use separate, clearly labeled columns for seen and unseen conditions, and rename SR in the unseen block to something like 'transcription success rate'.
  4. [Sec. 4.2] The sentence comparing rows (e)-(h) notes that CDDA performs best 'albeit at the cost of neglecting unseen directions', but this caveat is easy to miss. Since it is a significant limitation, it should be stated prominently in the abstract and conclusion.
  5. [Figure 1] The figure labels 'MicArray5' and 'LinearAdaptor' contain typos, and the stage labels 'NUM' and 'N1:N' are unclear. Please redraw the figure with consistent terminology.
  6. [Sec. 3.3] The 'case label header' used to distinguish one-speaker, two-speaker, and empty-output cases is not specified. Please describe the exact tokens (e.g., 'One', 'Two', 'Empty') and how they are inserted into the reference sequence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Directional-SpeechLLaMA's claimed results are empirical, held-out evaluations rather than reductions to fitted inputs or self-citation chains.

full rationale

The paper's derivation chain is empirical rather than circular. S-DOT and CDDA are training objectives and data augmentations that supervise the model to output direction labels alongside transcripts; the evaluation measures WER, direction accuracy, sWER, and success rate on held-out simulated LibriSpeech-plus-RIR test sets, including unseen RIRs and overlap conditions. The direction labels in the reference transcripts (Eq. 1 and 2) are ground-truth simulation parameters, not fitted quantities, and the reported accuracies are classification results on independent test data, not consequences of the training objective by construction. The NLCMV beamforming front-end is adopted from the authors' prior AGADIR work [29], and the model builds on SpeechLLaMA [8]; these are components used in the system, but the central claim that direction labels improve ASR and localization is tested against held-out data rather than derived from those citations. No uniqueness theorem, ansatz, or fitted parameter is imported as a forced conclusion. The absence of real-device validation is a legitimate external-validity limitation, but it is not a circularity; the simulation-to-real gap concerns representativeness of the testbed, not logical dependence of the results on their inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper relies on standard machine-learning axioms and domain assumptions about beamforming and simulation fidelity. No new physical entities are introduced. The free parameters are mostly training and evaluation choices; the direction grid and frontal subset are the most consequential because they define the scope of the claimed capability.

free parameters (6)
  • Direction set size = 12 discrete directions at 30 degree spacing
    The model is trained and evaluated only on this discrete azimuth grid; real-world directions are continuous and the model cannot output labels outside the grid.
  • Frontal direction subset = -60, -30, 0, 30, 60 degrees
    The multi-talker task trains only on these five directions; unseen-direction evaluation then requires relaxed success criteria because the model cannot output labels outside this training set.
  • Learning rate = 1e-4
    Standard choice for LLM fine-tuning; not central to the claim but affects results.
  • Warmup steps = 4000
    Standard training hyperparameter, part of the fine-tuning recipe.
  • Evaluation overlap ratio = 25%
    The only overlap condition tested; performance at other overlap ratios is unknown, and 25% already causes a large WER increase.
  • CDDA sampling counts = not specified
    Algorithm 1 describes the augmentation procedure but does not specify how many augmented examples are generated or how the distractor set is scaled, making the exact training set hard to replicate.
assumptions (4)
  • domain assumption Beamformed multi-channel audio preserves direction-of-arrival information that a linear projection and fine-tuned adapter can extract.
    Invoked in Sec 2.1.2 to justify the front-end and adapter design.
  • domain assumption Simulated RIRs from real environments accurately model spatial diversity on the Aria glasses.
    Invoked in Sec 4 to ground the simulated dataset; no real-device evaluation confirms this transfer.
  • ad hoc to paper Speakers remain stationary and speak from a single fixed direction during a conversation.
    Stated in Sec 2.2; real conversations involve movement, so this simplifying assumption limits the claimed applicability to real scenarios.
  • domain assumption A text prompt such as 'Repeat after me in X degrees' suffices to condition the LLM on the target direction.
    The prompt design is central to the target-direction ASR task (Sec 3.2) and is not ablated against alternative prompt formulations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition." pith.science (2026). https://pith.science/paper/EUADPMPT

@misc{pith2026250614973,
  author       = {Pith},
  title        = {Pith review of: Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EUADPMPT}},
  note         = {Machine review of arXiv:2506.14973}
}
read the original abstract

Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech recognition capabilities. However, the ability of Speech LLMs to comprehend and process multi-channel audio with spatial cues remains a relatively uninvestigated area of research. In this work, we present directional-SpeechLlama, a novel approach that leverages the microphone array of smart glasses to achieve directional speech recognition, source localization, and bystander cross-talk suppression. To enhance the model's ability to understand directivity, we propose two key techniques: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA). Experimental results show that our proposed directional-SpeechLlama effectively captures the relationship between textual cues and spatial audio, yielding strong performance in both speech recognition and source localization tasks.

Figures

Figures reproduced from arXiv: 2506.14973 by the authors.

Figure 1
Figure 1. Illustration of Directional-SpeechLLaMA training in a multi-stage framework for multi-talker directional ASR filtering from multi-channel waveforms for overlapping speech separation. Similarly, MIMO-speech [24] employed a multi￾source neural beam-former to enhance speech for each speaker in an acoustic mixture, thereby enabling multi-talker speech separation and recognition. A recent review by Zmolikova et al. [25] … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 29 canonical work pages

  1. [29]

    Enhancing end-to-end multi-channel speech separation via spatial feature learning,

    R. Gu, S.-X. Zhang, L. Chen, Y . Xu, M. Yu, D. Su, Y . Zou, and D. Yu, “Enhancing end-to-end multi-channel speech separation via spatial feature learning,” inICASSP. IEEE, 2020, pp. 7319– 7323

    work page 2020

  2. [21]

    Zotter and M

    F. Zotter and M. Frank,Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019

    work page 2019

  3. [1]

    Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition

    Introduction Recent research has demonstrated that a decoder-only large lan- guage model (LLM), pre-trained on a vast text corpus, can be ef- fectively adapted to comprehend multi-modal input, such as im- ages and audio, by prompting the LLM with modality-specific embeddings [1–6]. In particular, the integration of LLMs into speech processing has given ri...

    work page Pith review arXiv 2025

  4. [2]

    Multi-channel Speech Large Language Model Our training of multi-channel SLLM builds on recent approach that integrate speech capabilities into LLMs via audio encoders [6,7]

    Methods 2.1. Multi-channel Speech Large Language Model Our training of multi-channel SLLM builds on recent approach that integrate speech capabilities into LLMs via audio encoders [6,7]. The audio encoder is first trained on a large corpus of au- dio to acquire general acoustic representations independently of the LLM. To adapt the model to a downstream t...

    work page

  5. [3]

    The linear projection layer, inserted before the audio encoder, plays a crucial role in highlighting rel- evant information from different channels

    The audio encoder weights remain frozen to preserve its pre-trained knowledge, which operates in a high-dimensional space (1536 dimensions). The linear projection layer, inserted before the audio encoder, plays a crucial role in highlighting rel- evant information from different channels. It flattens the multi- channel features and projects them into the ...

    work page

  6. [4]

    what is said from where?

    Evaluation Tasks 3.1. Single-talker Far-field ASR and Source Localization The single-talker task integrates source localization and ASR, aiming to answer the question “what is said from where?”. The model is prompted to predict the direction of arrival in the hori- zontal plane and the corresponding transcript, aligning with the objectives of Stage 2 trai...

    work page

  7. [5]

    Room impulse responses (RIRs) from real environments are used to model spatial diver- sity, generating 12 distinct directions at 30° resolution

    Experimental Setup and Results We simulate 7-channel LibriSpeech (LS) datasets based on the array configuration of the Project Aria glasses [34], using the original LS corpus [35] as the source. Room impulse responses (RIRs) from real environments are used to model spatial diver- sity, generating 12 distinct directions at 30° resolution. To focus on typic...

    work page

  8. [6]

    We propose two key techniques to infuse directional knowledge: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA)

    Conclusions This paper presents Directional-SpeechLLaMA, a speech large language model designed to process multi-channel directional audio. We propose two key techniques to infuse directional knowledge: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA). Through a multi-stage training framework, we demonstrat...

    work page

Show all 42 references
  1. [7]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  3. [9]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”NeurIPS, vol. 35, pp. 23 716–23 736, 2022

    2022

  4. [10]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inICML. PMLR, 2023, pp. 19 730– 19 742

    2023

  5. [11]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,”arXiv preprint arXiv:2305.10790, 2023

    2023 arXiv

  6. [12]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  7. [13]

    Prompt- ing large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinliet al., “Prompt- ing large language models with speech recognition abilities,” in ICASSP. IEEE, 2024, pp. 13 351–13 355

    2024

  8. [14]

    On decoder-only architecture for speech- to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liuet al., “On decoder-only architecture for speech- to-text and large language model integration,” in2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  9. [15]

    Di- arizationlm: Speaker diarization post-processing with large lan- guage models,

    Q. Wang, Y . Huang, G. Zhao, E. Clark, W. Xia, and H. Liao, “Di- arizationlm: Speaker diarization post-processing with large lan- guage models,”arXiv preprint arXiv:2401.03506, 2024

    2024 arXiv

  10. [16]

    Enhancing speaker diarization with large language models: A contextual beam search approach,

    T. J. Park, K. Dhawan, N. Koluguri, and J. Balam, “Enhancing speaker diarization with large language models: A contextual beam search approach,” inICASSP. IEEE, 2024, pp. 10 861– 10 865

    2024

  11. [17]

    Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,”arXiv preprint arXiv:2308.16692, 2023

    2023 arXiv

  12. [18]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language mod- els are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  13. [19]

    A consolidated perspective on multimicrophone speech enhance- ment and source separation,

    S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “A consolidated perspective on multimicrophone speech enhance- ment and source separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 4, pp. 692–730, 2017

    2017

  14. [20]

    Brandstein and D

    M. Brandstein and D. Ward,Microphone arrays: signal process- ing techniques and applications. Springer Science & Business Media, 2013

    2013

  15. [22]

    Microphone arrays,

    G. W. Elko and J. Meyer, “Microphone arrays,”Springer hand- book of speech processing, pp. 1021–1041, 2008

    2008

  16. [23]

    Superdirectional microphone arrays,

    G. W. Elko, “Superdirectional microphone arrays,”Acoustic sig- nal processing for telecommunication, pp. 181–237, 2000

    2000

  17. [24]

    Deep beamforming networks for multi-channel speech recognition,

    X. Xiao, S. Watanabe, H. Erdogan, L. Lu, J. Hershey, M. L. Seltzer, G. Chen, Y . Zhang, M. Mandel, and D. Yu, “Deep beamforming networks for multi-channel speech recognition,” in ICASSP. IEEE, 2016, pp. 5745–5749

    2016

  18. [25]

    Probabilistic spa- tial dictionary based online adaptive beamforming for meeting recognition in noisy and reverberant environments,

    N. Ito, S. Araki, M. Delcroix, and T. Nakatani, “Probabilistic spa- tial dictionary based online adaptive beamforming for meeting recognition in noisy and reverberant environments,” inICASSP. IEEE, 2017, pp. 681–685

    2017

  19. [26]

    Recognizing overlapped speech in meetings: A multichan- nel separation approach using neural networks,

    T. Yoshioka, H. Erdogan, Z. Chen, X. Xiao, and F. Alleva, “Recognizing overlapped speech in meetings: A multichan- nel separation approach using neural networks,”arXiv preprint arXiv:1810.03655, 2018

    2018 arXiv

  20. [27]

    Directional speech recognition for speaker disambiguation and cross-talk suppression,

    J. Lin, N. Moritz, R. Xie, K. Kalgaonkar, C. Fuegen, and F. Seide, “Directional speech recognition for speaker disambiguation and cross-talk suppression,” inProc. INTERSPEECH 2023, 2023, pp. 3522–3526

    2023

  21. [28]

    Directional source sep- aration for robust speech recognition on smart glasses,

    T. Feng, J. Lin, Y . Huang, W. He, K. Kalgaonkar, N. Moritz, L. Wan, X. Lei, M. Sun, and F. Seide, “Directional source sep- aration for robust speech recognition on smart glasses,”arXiv preprint arXiv:2309.10993, 2023

    2023 arXiv

  22. [30]

    Mimo-speech: End-to-end multi-channel multi-speaker speech recognition,

    X. Chang, W. Zhang, Y . Qian, J. Le Roux, and S. Watanabe, “Mimo-speech: End-to-end multi-channel multi-speaker speech recognition,” in2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 237–244

    2019

  23. [31]

    Neural target speech extraction: An overview,

    K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. ˇCernock`y, and D. Yu, “Neural target speech extraction: An overview,”IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023

    2023

  24. [32]

    Bat: Learning to reason about spatial sounds with large language models,

    Z. Zheng, P. Peng, Z. Ma, X. Chen, E. Choi, and D. Harwath, “Bat: Learning to reason about spatial sounds with large language models,”arXiv preprint arXiv:2402.01591, 2024

    2024 arXiv

  25. [33]

    Can large language models understand spatial audio?

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, J. Zhang, L. Lu, Z. Ma, Y . Wanget al., “Can large language models understand spatial audio?”arXiv preprint arXiv:2406.07914, 2024

    2024 arXiv

  26. [34]

    Teleconference application and b-format microphone array for directional audio coding,

    J. Ahonen, V . Pulkki, and T. Lokki, “Teleconference application and b-format microphone array for directional audio coding,” in Audio Engineering Society Conference: 30th International Con- ference: Intelligent Audio Environments. Audio Engineering So- ciety, 2007

    2007

  27. [35]

    Agadir: Towards array-geometry agnostic direc- tional speech recognition,

    J. Lin, N. Moritz, Y . Huang, R. Xie, M. Sun, C. Fuegen, and F. Seide, “Agadir: Towards array-geometry agnostic direc- tional speech recognition,” inICASSP 2024-2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 951–11 955

    2024

  28. [36]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,”NeurIPS, 2017

    2017

  29. [37]

    Seri- alized output training for end-to-end overlapped speech recogni- tion,

    N. Kanda, Y . Gaur, X. Wang, Z. Meng, and T. Yoshioka, “Seri- alized output training for end-to-end overlapped speech recogni- tion,”arXiv preprint arXiv:2003.12687, 2020

    2003 arXiv

  30. [38]

    Supervised contrastive learning,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,”NeurIPS, vol. 33, pp. 18 661–18 673, 2020

    2020

  31. [39]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inICML. PMLR, 2023, pp. 28 492–28 518

    2023

  32. [40]

    Project aria: A new tool for egocentric multi-modal ai research,

    J. Engel, K. Somasundaram, M. Goesele, A. Sun, A. Gamino, A. Turner, A. Talattof, A. Yuan, B. Souti, B. Meredithet al., “Project aria: A new tool for egocentric multi-modal ai research,” arXiv preprint arXiv:2308.13561, 2023

    2023 arXiv

  33. [41]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” inICASSP. IEEE, 2015, pp. 5206–5210

    2015

  34. [42]

    Au- diochatllama: Towards general-purpose speech abilities for llms,

    Y . Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y . Shangguan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “Au- diochatllama: Towards general-purpose speech abilities for llms,” inProceedings of the 2024 Conference of the North American Chapter of the Association f...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.