pith. sign in

arxiv: 2607.02129 · v1 · pith:BTDPUBZ6new · submitted 2026-07-02 · 💻 cs.SD

Speaker head orientation estimation with a single microphone array using phase spectrogram features

Balint Turi , Archontis Politis , Parthasaarathy Sudarsanam , Tuomas Virtanen This is my paper

Pith reviewed 2026-07-03 05:51 UTC · model grok-4.3

classification 💻 cs.SD
keywords speaker head orientation estimationmicrophone arrayphase spectrogramdeep neural networkvoice directivityaudio signal processinghead orientationfine-tuning
0
0 comments X

The pith

Phase spectrograms from a single microphone array fed into a hybrid neural network estimate speaker head orientation after training on simulated data and fine-tuning on real recordings.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to determine the direction a person is facing from audio captured by one microphone array. It uses the phase part of the short-time Fourier transform as input to a deep network that mixes convolutional layers for local patterns, recurrent layers for sequences, and self-attention for global context. Training begins on a large simulated set built with realistic voice radiation patterns, then shifts to real recordings. This yields state-of-the-art accuracy in quiet and noisy rooms, with personalization lowering the average angular error to 11.3 degrees. The result matters for applications such as meeting analysis, smart spaces, and driver monitoring that need spatial awareness without extra hardware.

Core claim

Phase spectrogram features from a single microphone array, processed by a deep neural network that combines convolutional, recurrent, and self-attention layers, support accurate speaker head orientation estimation when the network is first trained on large-scale simulated data generated with voice directivity patterns and then fine-tuned on real recordings, outperforming baselines in both clean and noisy conditions.

What carries the argument

Phase spectrogram features from the short-time Fourier transform, supplied as input to a hybrid convolutional-recurrent-self-attention neural network.

If this is right

  • The network generalizes from simulation to real data under both clean and noisy conditions.
  • Personalization to individual users and environments reduces mean angular error to 11.3 degrees.
  • The method outperforms prior handcrafted-feature and raw-waveform baselines.
  • Head orientation can be recovered from a single array without needing multiple arrays or visual input.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same phase-based pipeline could be tested for continuous tracking of moving speakers rather than static orientation.
  • Combining the audio estimates with simple visual cues might further lower error in multimodal settings.
  • The simulation-plus-fine-tuning recipe may transfer to related tasks such as sound source localization with one array.

Load-bearing premise

The simulated dataset created with voice directivity patterns approximates real acoustic conditions closely enough that fine-tuning on real recordings produces good generalization.

What would settle it

Run the trained model on a fresh collection of real recordings from unseen rooms and speakers without any fine-tuning step and check whether the mean angular error stays near 11.3 degrees or rises sharply.

read the original abstract

Estimating a speaker's head orientation from audio can provide valuable information in smart environments, meetings, and driver monitoring. We propose a novel approach that leverages the phase component of the short-time Fourier transform from a single microphone array as input to a deep neural network combining convolutional, recurrent, and self-attention layers. Unlike prior methods that use physics-informed handcrafted features or raw waveform inputs, our approach enables robust learning from simulated and real data. Trained on a large-scale dataset generated with voice directivity patterns and fine-tuned on real recordings, our model achieves state-of-the-art accuracy, outperforming baselines under both clean and noisy conditions. Personalization experiments further demonstrate significant gains, reaching a mean angular error of 11.3 degrees when adapting to individual users and environments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes a CNN-RNN-attention DNN that takes phase spectrogram features from a single microphone array as input for estimating speaker head orientation. It describes pre-training on a large-scale simulated dataset generated with voice directivity patterns, followed by fine-tuning on real recordings, and claims state-of-the-art accuracy that outperforms baselines under clean and noisy conditions, with personalization experiments reaching a mean angular error of 11.3 degrees.

Significance. If the empirical claims hold after proper validation, the phase-spectrogram approach combined with large-scale simulation pre-training could advance practical audio-only orientation estimation for smart environments and meetings. The architecture choice and personalization results are potentially useful contributions, but the significance depends on demonstrating that the simulation-to-real transfer actually captures orientation cues rather than artifacts.

major comments (2)
  1. [Abstract] Abstract: The SOTA and outperformance claims are asserted without naming any baselines, reporting dataset sizes, providing error bars, statistical significance tests, or controls for simulation-to-real domain shift; these details are load-bearing for the central performance assertions.
  2. [§3 (Data Generation)] The simulation-to-real transfer (pre-training on voice-directivity-generated phase spectrograms then fine-tuning) is presented as enabling the 11.3° result, yet no quantitative evidence (distribution distances, feature statistics, or ablation on directivity model calibration) is supplied to show the simulated phase features are close enough to real data for the model to learn genuine orientation cues rather than simulation artifacts.
minor comments (2)
  1. [§2 (Proposed Method)] The description of the CNN-RNN-attention architecture would benefit from an explicit equation or diagram showing how the phase spectrogram tensor is shaped and fed into the first convolutional layer.
  2. [§4 (Experiments)] Table captions or result figures should include the exact number of real recordings used for fine-tuning and personalization to allow reproducibility assessment.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. Below we address each major point directly, indicating where revisions to the manuscript are warranted.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The SOTA and outperformance claims are asserted without naming any baselines, reporting dataset sizes, providing error bars, statistical significance tests, or controls for simulation-to-real domain shift; these details are load-bearing for the central performance assertions.

    Authors: The abstract is intentionally concise per journal guidelines and summarizes the core contribution. Full details appear in the manuscript: baselines are defined and compared in §4 and §5, dataset sizes and generation procedure in §3, error bars and per-condition results in §5 (including noisy conditions), and simulation-to-real transfer via pre-training plus fine-tuning is quantified through the reported accuracy gains. Statistical significance testing was not performed; we can add it in a revision if the editor requires. We will expand the abstract by one sentence to name the primary baselines and report the 11.3° personalized error if length permits. revision: partial

  2. Referee: [§3 (Data Generation)] The simulation-to-real transfer (pre-training on voice-directivity-generated phase spectrograms then fine-tuning) is presented as enabling the 11.3° result, yet no quantitative evidence (distribution distances, feature statistics, or ablation on directivity model calibration) is supplied to show the simulated phase features are close enough to real data for the model to learn genuine orientation cues rather than simulation artifacts.

    Authors: The manuscript shows the benefit of the pre-training stage through direct comparison of models trained from scratch versus pre-trained then fine-tuned, with the latter reaching the reported 11.3° error on real recordings. No explicit distribution-distance metrics or directivity-model ablation tables were included. We will add a supplementary table reporting cosine similarity between simulated and real phase-spectrogram statistics across frequency bands and an ablation removing the directivity model to address this concern. revision: yes

Circularity Check

0 steps flagged

No circularity; standard empirical ML pipeline with independent evaluation

full rationale

The paper presents a CNN-RNN-attention model trained on simulated phase spectrograms (generated using voice directivity patterns) and fine-tuned on real recordings, with performance measured as mean angular error on test data. This is a conventional supervised learning setup where results derive from data splits and optimization, not from any self-definitional equations, fitted parameters renamed as predictions, or load-bearing self-citations of uniqueness theorems. No steps reduce the claimed accuracy (e.g., 11.3°) to quantities defined by the model itself or prior author work. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the fidelity of acoustic simulation for voice directivity and the informativeness of phase features for the task; the DNN weights constitute the primary fitted elements.

free parameters (1)
  • DNN model parameters
    Weights of the convolutional, recurrent, and self-attention layers are fitted during training on simulated and real data.
axioms (1)
  • domain assumption The phase component of the STFT encodes directional information related to speaker head orientation
    The method relies on this to use phase spectrograms as input features without explicit physics modeling.

pith-pipeline@v0.9.1-grok · 5672 in / 1373 out tokens · 40780 ms · 2026-07-03T05:51:39.575313+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Speaker head orientation estimation with a single microphone array using phase spectrogram features

    INTRODUCTION Estimating a speaker’s head orientation has become increas- ingly important in modern human–machine interaction, as it provides cues about attention, intent, and conversational con- text. For example, in smart home environments, orientation information can disambiguate which device a user’s com- mand is directed to [1, 2, 3]. In meeting rooms...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    PROBLEM DEFINITION AND METHODOLOGY The objective of this work is to estimate the head orientation of a single speaker in the azimuthal plane within a reverber- ant room environment, using only a single microphone array with specifications similar to those found in smart home au- dio devices. In this setting, both the speaker and the array may occupy any p...

  3. [3]

    Per Angle Classifier

    EV ALUA TION We evaluate our proposed method on both simulated and real datasets. The simulated setup allows controlled testing across acoustic conditions and baselines. We then examine the effect of personalization through user- and room-specific fine-tuning, and finally validate generalization on real record- ings. Performance is measured using classifi...

  4. [4]

    CONCLUSION We presented a novel approach to speaker head orientation estimation that leverages phase features from the short-time Fourier transform in combination with a deep neural network. Through experiments on both simulated and real datasets, we demonstrated that this representation provides superior ro- bustness to noise and generalization across sp...

  5. [5]

    Soundr: Head position and orientation prediction using a mi- crophone array,

    J. Yang, G. Banerjee, V . Gupta, M. S. Lam, and J. A. Landay, “Soundr: Head position and orientation prediction using a mi- crophone array,” inACM CHI, 2020

    2020

  6. [6]

    Model-based head orientation estima- tion for smart devices,

    Q. Yang and Y . Zheng, “Model-based head orientation estima- tion for smart devices,” inACM IMWUT, 2021

    2021

  7. [7]

    Direction- of-voice (dov) estimation for intuitive speech interaction with smart devices ecosystems,

    K. Ahuja, A. Kong, M. Goel, and C. Harrison, “Direction- of-voice (dov) estimation for intuitive speech interaction with smart devices ecosystems,” inACM UIST, 2020

    2020

  8. [8]

    A study on visual focus of atten- tion recognition from head pose in a meeting room,

    S. O. Ba and J.-M. Odobez, “A study on visual focus of atten- tion recognition from head pose in a meeting room,” inMLMI, 2006

    2006

  9. [9]

    Head orientation and gaze direc- tion in meetings,

    R. Stiefelhagen and J. Zhu, “Head orientation and gaze direc- tion in meetings,” inACM CHI, 2002

    2002

  10. [10]

    An orientation sensor-based head tracking system for driver behaviour monitoring,

    Y . Zhao, L. G¨orne, I.-M. Yuen, D. Cao, M. Sullman, D. Auger, C. Lv, H. Wang, R. Matthias, L. Skrypchuk, and A. Mouza- kitis, “An orientation sensor-based head tracking system for driver behaviour monitoring,”Sensors, vol. 17, no. 11, 2017

    2017

  11. [11]

    Deep learning for head pose esti- mation: A survey,

    A. Asperti and D. Filippini, “Deep learning for head pose esti- mation: A survey,”SN Comp. Sci., vol. 4, no. 4, 2023

    2023

  12. [12]

    A baseline algorithm for estimating talker orientation using acoustical data from a large- aperture microphone array,

    J. M. Sachar and H. F. Silverman, “A baseline algorithm for estimating talker orientation using acoustical data from a large- aperture microphone array,” inICASSP, 2004

    2004

  13. [13]

    A robust method to extract talker azimuth orientation using a large-aperture microphone array,

    A. Levi and H. Silverman, “A robust method to extract talker azimuth orientation using a large-aperture microphone array,” IEEE TASLP, vol. 18, no. 2, 2010

    2010

  14. [14]

    Real-time sound source orientation estima- tion using a 96 channel microphone array,

    H. Nakajima, K. Kikuchi, T. Daigo, Y . Kaneda, K. Nakadai, and Y . Hasegawa, “Real-time sound source orientation estima- tion using a 96 channel microphone array,” inIEEE/RSJ IROS, 2009

    2009

  15. [15]

    Audio person tracking in a smart-room environment,

    A. Abad, C. Segura, D. Macho, J. Hernando, and C. Nadeu, “Audio person tracking in a smart-room environment,” inIn- terspeech, 2006

    2006

  16. [16]

    GCC-PHAT based head orienta- tion estimation,

    C. Segura and J. Hernando, “GCC-PHAT based head orienta- tion estimation,” inInterspeech, 2012

    2012

  17. [17]

    The generalized correlation method for estimation of time delay,

    C. Knapp and G. Carter, “The generalized correlation method for estimation of time delay,”IEEE TASLP, vol. 24, no. 4, pp. 320–327, 1976

    1976

  18. [18]

    Speaker orientation estimation based on hybridation of GCC-PHAT and HLBR,

    C. Segura, A. Abad, J. Hernando, and C. Nadeu, “Speaker orientation estimation based on hybridation of GCC-PHAT and HLBR,” inInterspeech, 2008

    2008

  19. [19]

    Multimodal head orientation towards attention track- ing in smartrooms,

    C. Segura, C. Canton-Ferrer, A. Abad, J. R. Casas, and J. Her- nando, “Multimodal head orientation towards attention track- ing in smartrooms,” inICASSP, 2007

    2007

  20. [20]

    Head orientation estimation from multiple microphone ar- rays,

    R. C. Felsheim, A. Brendel, P. A. Naylor, and W. Kellermann, “Head orientation estimation from multiple microphone ar- rays,” inEUSIPCO, 2021

    2021

  21. [21]

    Single-channel head orientation estimation based on discrimination of acoustic transfer function,

    R. Takashima, T. Takiguchi, and Y . Ariki, “Single-channel head orientation estimation based on discrimination of acoustic transfer function,” inInterspeech, 2011

    2011

  22. [22]

    Estimation of talker’s head orientation based on discrimination of the shape of cross-power spectrum phase coefficients,

    R. Takashima, T. Takiguchi, and Y . Ariki, “Estimation of talker’s head orientation based on discrimination of the shape of cross-power spectrum phase coefficients,” inInterspeech, 2012

    2012

  23. [23]

    Learning speaker-listener mutual head orientation by leveraging hrtf and voice directivity on headphones,

    H. Takawale and N. Roy, “Learning speaker-listener mutual head orientation by leveraging hrtf and voice directivity on headphones,” inICASSP, 2024

    2024

  24. [24]

    Pyannote.audio: Neural building blocks for speaker diariza- tion,

    H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “Pyannote.audio: Neural building blocks for speaker diariza- tion,” inICASSP, 2020

    2020

  25. [25]

    Phase-aware deep speech enhance- ment: It’s all about the frame length,

    T. Peer and T. Gerkmann, “Phase-aware deep speech enhance- ment: It’s all about the frame length,”JASA Express Letters, vol. 2, no. 10, 2022

    2022

  26. [26]

    Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,

    S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,”IEEE JSTSP, vol. 13, no. 1, pp. 34–48, 2019

    2019

  27. [27]

    Assessment of self-attention on learned features for sound event localization and detection,

    P. Sudarsanam, A. Politis, and K. Drossos, “Assessment of self-attention on learned features for sound event localization and detection,” inDCASE, 2021

    2021

  28. [28]

    DirPat - Database and Viewer of 2D/3D Directivity Patterns of Sound Sources and Receivers,

    M. Brandner, M. Frank, and D. Rudrich, “DirPat - Database and Viewer of 2D/3D Directivity Patterns of Sound Sources and Receivers,” in144th AES Convention, 2018, e-Brief 425

    2018

  29. [29]

    Generation and analysis of an acoustic radiation pattern database for forty-one musical instruments,

    N. R. Shabtai, G. Behler, M. V orl ¨ander, and S. Weinzierl, “Generation and analysis of an acoustic radiation pattern database for forty-one musical instruments,”JASA, vol. 141, 2017

    2017

  30. [30]

    Long-term horizon- tal vocal directivity of opera singers: Effects of singing projec- tion and acoustic environment,

    D. Cabrera, P. J. Davis, and A. Connolly, “Long-term horizon- tal vocal directivity of opera singers: Effects of singing projec- tion and acoustic environment,”Journal of V oice, vol. 25, no. 6, pp. e291–e303, Nov 2011

    2011

  31. [31]

    CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit,

    J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit,” 2019, Version 0.92

    2019

  32. [32]

    Pyroomacous- tics: A python package for audio room simulation and array processing algorithms,

    R. Scheibler, E. Bezzam, and I. Dokmanic, “Pyroomacous- tics: A python package for audio room simulation and array processing algorithms,” inICASSP, 2018

    2018

  33. [33]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” inICLR, 2015

    2015

  34. [34]

    Wham!: Extending speech separation to noisy environments,

    G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux, “Wham!: Extending speech separation to noisy environments,” inInterspeech, Sept. 2019

    2019

  35. [35]

    Rendering of source spread for arbitrary playback setups based on spatial covari- ance matching,

    L. McCormack, A. Politis, and V . Pulkki, “Rendering of source spread for arbitrary playback setups based on spatial covari- ance matching,” inWASPAA, 2021

    2021

  36. [36]

    Generat- ing coherence-constrained multisensor signals using balanced mixing and spectrally smooth filters,

    D. Mirabilii, S. J. Schlecht, and E.A.P. Habets, “Generat- ing coherence-constrained multisensor signals using balanced mixing and spectrally smooth filters,”JASA, vol. 149, no. 3, pp. 1425–1433, 2021

    2021

  37. [37]

    Speaker distance estimation in enclosures from single- channel audio,

    M. Neri, A. Politis, D.A. Krause, M. Carli, and T. Virta- nen, “Speaker distance estimation in enclosures from single- channel audio,”IEEE/ACM TASLP, vol. 32, pp. 2242–2254, 2024

    2024