The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models

Oli Danyi Liu; Peter Bell; Yi Wang

REVIEW 4 major objections 6 minor 28 references

The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AV-HuBERT encodes phonemes on audio time, not visual time, so its small lead over audio-only HuBERT is a downsampling artifact.

desk verdict A useful caution about AV-HuBERT's temporal encoding, but the paper never tests the 100–300 ms natural asynchrony it claims the model misses. read the letter →

arxiv 2506.20361 v1 pith:FS5JKWAI submitted 2025-06-25 eess.AS cs.SDeess.IV

Yi Wang , Oli Danyi Liu , Peter Bell This is my paper

classification eess.AScs.SDeess.IV

keywords audio-visualspeechself-supervisedlearningAV-HuBERTphoneticdecodabilitytemporaldynamicsmultimodalperceptionlinearclassifierstime-to-voice

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human listeners use the fact that lip movements precede voicing by 100-300 ms. This paper asks whether a self-supervised audio-visual model, AV-HuBERT, captures that asynchronicity in the time course of its phonetic representations. Using per-frame linear classifiers to plot phonetic decodability around phone onsets, the authors find that AV-HuBERT's embeddings become decodable only about 20 ms earlier than audio-only HuBERT's, and that this gap is explained by AV-HuBERT's four-frame audio downsampling rather than by visual anticipation. Delaying the audio input shifts the model's decodability peak by the same amount, showing its visual encoding is aligned to the audio track. The conclusion is that AV-HuBERT, and by extension transformer speech models, should be used with caution when modeling temporally sensitive aspects of multimodal speech perception.

What carries the argument

The central machinery is the phonetic decodability curve: a set of linear logistic-regression classifiers, one per time step in a 1200 ms context window centered on phone onset, trained to predict phone identity from that step's feature frame. The curve's peak and rising edge reveal when phoneme information becomes available. The argument's other load-bearing component is AV-HuBERT's audio downsampling: four 10 ms log-Mel frames are stacked and concatenated with one 40 ms visual frame, which shifts audio features earlier by an expected 20 ms and can fully account for the observed 20 ms gap.

What would settle it

Delay only the visual stream by 200 ms while keeping audio synchronized, and observe whether the decodability peak moves earlier; the paper predicts it stays on audio time, whereas true visual encoding would shift it toward the visual lead.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in Section 5, is that AV-HuBERT's encoding of speech temporal dynamics is dominated by its audio input, with its visual encoding time course overwhelmingly aligned to the audio encoding, so that the natural 100-300 ms visual lead in speech is overlooked. The evidence is that the phoneme decodability window of AV-HuBERT peaks at phone onset (0 ms) while HuBERT's peaks 20 ms later; the 20 ms advantage disappears when accounting for AV-HuBERT's stacking of four 10 ms audio frames per 40 ms feature frame. When audio is artificially delayed by 20 or 80 ms, the decodability peak shifts correspondingly, and at an extreme delay of 800 ms, when audio can no longer help, the curve shows no early visual peak. The authors attribute this to cross-modal transfer learning: AV-HuBERT's pretraining targets are MFCC-based audio pseudo-labels, so both modalities learn acoustic timing.

Load-bearing premise

The comparison assumes that AV-HuBERT and HuBERT differ only in visual input, but the two models were pretrained on different datasets (LRS3 vs LibriSpeech) with different targets, so the observed timing difference could partly come from data domain or target design rather than audio-visual integration.

Editorial extensions

If this is right

If correct, AV-HuBERT should not be used to simulate the temporal dynamics of human audio-visual speech perception, despite its McGurk-like behavior in prior work.
The 20 ms advantage reported in earlier comparisons should be reinterpreted as a feature-extraction artifact, not evidence of visual facilitation.
Timing comparisons between audio-only and audio-visual SSL models must control for frame rates and downsampling before attributing differences to the visual modality.
Because the effect appears across layers 3, 6, 9, and 12, the limitation is a property of the whole model, not a single layer.
Training targets such as audio-derived pseudo-labels can shape the temporal correlation of transformer features, so any model used for temporally sensitive brain comparisons needs target-design checks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to pretrain an audio-visual model with visual-derived or jointly derived targets and evaluate the same decodability curves; the framework predicts a genuine visual lead would appear only if the target itself carries visual timing.
The decodability-curve method could be applied to other audio-visual models, such as those using cross-attention fusion rather than early concatenation, to see whether fusion architecture determines temporal alignment.
A finer analysis on bilabial and labial stops with measured articulation-to-acoustic lags could reveal whether the model shows any consonant-specific visual lead that the aggregate curve hides.
If confirmed on a matched training set, the result would suggest that current audio-visual SSL pretraining pipelines may be learning to ignore asynchrony rather than exploit it, with implications for lip-reading and hearing-aid applications.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper has a real, timely question and one new empirical result — AV-HuBERT's phoneme decodability is only about 20 ms earlier than HuBERT's — but the experiments don't actually test the 100–300 ms natural audio-visual asynchrony that motivates the claim. So the headline conclusion is plausible but not directly supported.

What the paper does well: applying the phonetic decodability-window method to an audio-visual SSL model is a sensible new move. The normalization of curves is reasonable. And the delayed-audio manipulation in Section 4.3 is a genuinely clever diagnostic: shifting the audio delay shifts the decodability peak by roughly the same amount, which supports the claim that audio dominates the timing. The writing is clear and the authors explicitly acknowledge the pretraining data confound in Section 2.1.

Soft spots, in order of severity. First, the central 20 ms figure is read off a 40 ms frame-rate curve, so it is half a time step, with no error bars or significance test. Second, the AV-HuBERT vs HuBERT comparison is uncontrolled: different pretraining corpora, different targets, different data domains. The paper says it normalizes to mitigate this, but normalization does not remove a timing confound. Third, and this is the one that matters most: the asynchronicity simulation in Section 4.3 uses delays of 20, 80, and 800 ms. The natural time-to-voice range the paper cites is 100–300 ms. The 20 and 80 ms conditions are below or at the edge of that range, and 800 ms pushes the delayed audio entirely outside the 200 ms context window, so that condition tests what happens when audio is effectively absent, not whether visual encoding follows audio timing. A null result there does not establish audio dominance at natural asynchrony. The authors should have tested 100–300 ms delays with the audio still in the context window. Fourth, the conclusion broadens to "all transformer models" — that generalization goes well beyond the evidence.

Is the paper worth a serious referee? Yes. The question is timely, the method is inherited from credible prior work, and the negative result, if true, would be a useful caution for the audiovisual-SSL-as-brain-model community. But in its current form the central quantitative claim is not robust. A referee should send it back for the missing delay conditions, error bars, and ideally a better-controlled comparison (e.g., a HuBERT trained on LRS3 audio, or an AV-HuBERT variant). My own verdict is conditional at best.

Referee Report

4 major / 6 minor

Summary. This paper asks whether the audio-visual self-supervised model AV-HuBERT captures the natural 100–300 ms lead of visual over acoustic cues in speech. The authors extract layer-9 features from AV-HuBERT and audio-only HuBERT, train per-time-step logistic-regression phone classifiers over a 1200 ms window centered on forced-aligned phone onsets, and compare the resulting decodability curves. They report that AV-HuBERT's peak decodability precedes HuBERT's by about 20 ms, attribute this gap to AV-HuBERT's 4-frame audio stacking rather than to genuine visual anticipation, and support this interpretation with an audio-delay manipulation (20, 80, 800 ms) plus a layer-wise peak table. They conclude that AV-HuBERT's visual encoding is time-locked to its audio stream and that the model is therefore unsuitable for modeling temporally sensitive aspects of human multimodal speech perception.

Significance. The question is timely: AV-HuBERT is increasingly used as a model of human audio-visual integration, and the temporal dynamics of phonetic encoding are central to such comparisons. The paper's methodology is transparent and reproducible in spirit: it uses public checkpoints, a standard linear-probing procedure, and a clever delay manipulation that directly tests which input modality controls decodability timing. If the conclusions were established, they would constitute a useful caution for the cognitive-auditory modeling community and a concrete engineering insight about frame-stacking artifacts. However, the headline 20 ms difference is currently at or below the temporal resolution of the AV-HuBERT features, the model comparison is confounded by pretraining data and targets, and the delay experiment skips the ecologically relevant 100–300 ms range. The central claim is therefore plausible but not yet supported at the advertised precision.

major comments (4)

[Section 2.2, Section 4.1, Table 1] The central quantitative claim — that AV-HuBERT becomes decodable 'approximately 20 ms' earlier than HuBERT — is read off decodability curves whose AV-HuBERT time step is 40 ms: Section 2.2 states that the 1200 ms window contains 30 AV-HuBERT frames, so a 20 ms difference is half a frame. Table 1 reports peak times to the nearest millisecond (e.g., 0 ms for AV-HuBERT and 20 ms for HuBERT) without error bars, confidence intervals, or a statistical test across the three cross-validation folds. The 20 ms gap may simply be quantization of a 40 ms grid; please report the per-fold peak distributions, a bootstrap or permutation test on the peak difference, and state explicitly how peaks were interpolated or read from the 40 ms grid.
[Section 2.1, Section 4.1] The HuBERT vs AV-HuBERT comparison is confounded by pretraining differences that the paper itself acknowledges in Section 2.1: AV-HuBERT is pretrained on 433 h of LRS3 with cross-modal pseudo-label transfer and an additional visual encoder, while HuBERT is pretrained on 960 h of LibriSpeech with standard audio-only HuBERT targets. The normalization step in Section 4.1 rescales each curve by its maximum, but it cannot remove timing distortions caused by data domain, speaker diversity, or training-target design. Consequently the 20 ms peak difference cannot be uniquely attributed to audio-visual integration; at minimum the paper should state this limitation prominently and support the cross-model comparison with the within-model delay evidence rather than treating the two models as differing only in modality.
[Section 4.3] The asynchronous-input experiment tests audio delays of 20, 80, and 800 ms only, yet the phenomenon motivating the paper is the natural 100–300 ms time-to-voice range cited in the Introduction from Chandrasekaran et al. [8]. At 20 ms the delay is below the 40 ms AV-HuBERT frame resolution, and at 80 ms it is at the lower edge of the relevant range; the 800 ms condition places the delayed audio entirely outside the 200 ms limited-context window, effectively removing the audio stream rather than simulating natural asynchrony. A null result in the 800 ms condition therefore shows at most that AV-HuBERT cannot recover phone timing when audio is absent, not that its visual encoding is audio-locked at ecologically valid lags. Please add audio-delay conditions of 100, 200, and 300 ms with the audio stream still inside the context window and report the corresponding peak shifts.
[Section 4.2] The explanation that frame stacking advances audio features by an expected 20 ms assumes that a phone onset is uniformly distributed across the four stacked audio frames. This assumption is stated without empirical support; forced-alignment boundaries are derived on a different time grid, and a non-uniform onset distribution would change the expected shift. Please report the empirical distribution of phone-onset positions within the four-frame stacks (or a sensitivity analysis), and correct the related statement that HuBERT's 'shorter frame window length (20 ms)' is responsible for its smaller shift, since the relevant quantity is the 20 ms frame stride rather than a window length.

minor comments (6)

[Section 3.2] Section 3.2: '1 15 samples' appears to be a missing fraction, probably 1/15; please fix the typo.
[Section 2.1] Section 2.1: the shorthand 'HuBERTs' for the pair is confusing; use 'HuBERT and AV-HuBERT' throughout.
[Section 4.3, Figure 4] Section 4.3 and Figure 4: the subfigure captions do not state the condition represented by the black dotted lines until the body text; add the condition to the caption.
[Section 5] Section 5: the final sentence extrapolates from two models to 'all transformer models used in temporal sensitive modelling'; this overgeneralization is not supported by the experiments, so restrict the claim to the tested models or add evidence.
[Table 1] Table 1: state the peak-reading convention (argmax over raw frames? any interpolation?) so the 0/20 ms values can be interpreted at the stated 40 ms resolution.
[Section 2.2] Section 2.2: the formula 'it covers 2w+1 = T/sample rate frames' is not typeset cleanly and the definition of w is implicit; make the notation explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: decodability peaks are measured empirically and the audio-dominance conclusion is tested by explicit delay manipulation.

full rationale

The paper's derivation chain is empirical rather than circular. It measures phonetic decodability peaks by training linear classifiers on frozen pretrained embeddings (Section 2.2), reads peak times off the resulting curves (Sections 4.1 and 4.2, Table 1), and tests the audio-dominance interpretation by artificially delaying audio inputs and observing that peak positions track the audio delay (Section 4.3). The 20 ms advantage of AV-HuBERT is not a fitted parameter renamed as a prediction; it is read from normalized decodability curves, and the paper explicitly attributes it to a known architectural property (stacking four 10 ms audio frames into one 40 ms AV-HuBERT frame), an explanation that is then probed by delay manipulation rather than assumed. The extreme 800 ms delay condition is used to test whether visual encoding alone can produce anticipatory peaks; the null result is an observation, not a consequence of the definition of the decodability window. The cited method [12] is a self-cited analysis tool by a co-author, but the load-bearing conclusion does not rest on that citation: the peaks are measured directly from the models under comparison, and the delay manipulation is independent of [12]. The acknowledged pretraining-data confound (Section 2.1) is a validity threat, not a circular step, because the central audio-dominance claim is established within AV-HuBERT itself via the delay experiment. No equation is shown to equal its input by construction, no fitted parameter is relabeled as a prediction, and no self-citation is used to forbid alternative explanations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted constants. Its central claim depends on the validity of decodability-window analysis, the comparability of two differently pretrained models, and the accuracy of forced-alignment timing. The uniform-distribution assumption for downsampling is the only ad hoc modeling step.

assumptions (4)

domain assumption The peak of the linear-classifier decodability curve indexes when phoneme information becomes available in model representations.
The entire method, adapted from Gwilliams et al. and Liu et al., assumes that classifier accuracy at a given time step reflects the temporal availability of phonetic information, not other confounds such as contextual cues.
domain assumption Layer 9 features from HuBERT and AV-HuBERT are comparably informative for phoneme decoding.
The paper selects layer 9 because prior work found high phonetic correlation, but the two models may differ in layer-wise information organization due to different training data and objectives.
domain assumption Forced-alignment phone boundaries from the Montreal Forced Aligner are accurate enough for 20-40 ms timing comparisons.
The paper relies on forced alignment for phone onsets; alignment errors of tens of milliseconds could directly affect the measured decodability peak times.
ad hoc to paper The 20 ms expected advancement from stacking four audio frames assumes the phone onset is uniformly distributed across the four frames.
This uniform-distribution assumption, introduced in Section 4.2 to explain the observed 20 ms gap, is plausible but is a modeling assumption rather than a measured quantity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models." pith.science (2026). https://pith.science/paper/FS5JKWAI

@misc{pith2026250620361,
  author       = {Pith},
  title        = {Pith review of: The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FS5JKWAI}},
  note         = {Machine review of arXiv:2506.20361}
}

read the original abstract

Human speech perception is multimodal. In natural speech, lip movements can precede corresponding voicing by a non-negligible gap of 100-300 ms, especially for specific consonants, affecting the time course of neural phonetic encoding in human listeners. However, it remains unexplored whether self-supervised learning models, which have been used to simulate audio-visual integration in humans, can capture this asynchronicity between audio and visual cues. We compared AV-HuBERT, an audio-visual model, with audio-only HuBERT, by using linear classifiers to track their phonetic decodability over time. We found that phoneme information becomes available in AV-HuBERT embeddings only about 20 ms before HuBERT, likely due to AV-HuBERT's lower temporal resolution and feature concatenation process. It suggests AV-HuBERT does not adequately capture the temporal dynamics of multimodal speech perception, limiting its suitability for modeling the multimodal speech perception process.

Figures

Figures reproduced from arXiv: 2506.20361 by the authors.

**Figure 1.** Phonetic decodability curves on Grid data cross-validation experiments, representing (a) absolute prediction accuracy on all phonemes, (b) absolute prediction accuracy on plosive stop consonants, (c) relative accuracy on all phonemes, (d) relative accuracy on plosive stop consonants, with x axis labels representing time (ms) relative to phone onset (0). Plain orange line denotes HuBERT and triangular-marked red line… view at source ↗

**Figure 2.** Phonetic decodability curves from audio-visual features in the cross-validation experiments, full-context on Grid (full Grid, triangular-marked red lines); limited-context window of 200ms on Grid (limited Grid, circle-marked blue lines) or full-context on LRS3 (full LRS3, plain green lines), plotted on all phonemes (a), or on plosive stop consonants (b) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Relative prediction accuracy curves on Grid data based on limited-context window, (a) all phoneme and (b) plosive stops . Circle-marked blue lines for AV-HuBERT. Plain orange lines for HuBERT phone onset, for all phonemes (Fig. 1a) and for plosive stop consonants (Fig. 1b), respectively, plotted with data from Grid. HuBERT curves are mildly shifted towards right than centered at the phone onset time, compared to th… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Phonetic decodability curves on asynchronous audio-visual signals from Grid data: (a) absolute prediction accuracy on all phonemes (b) relative accuracy on all phonemes, (c) absolute prediction accuracy on all phonemes (d) absolute prediction accuracy on plosive stops.…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 22 canonical work pages

[8]

Using artificial neural networks to ask ‘why’ questions of minds and brains,

N. Kanwisher, M. Khosla, and K. Dobs, “Using artificial neural networks to ask ‘why’ questions of minds and brains,”Trends in Neurosciences, vol. 46, no. 3, pp. 240–254, 2023

work page 2023
[1]

Introduction Recent years have seen a rising trend in using artificial neural networks (ANN) to model sensory processes in humans [1]. For speech processing, it has been shown that the neural activity in the auditory cortex of human listeners correlates with represen- tations in a recurrent neural network model trained for speech recognition, with corresp...

work page
[2]

Related works and method Our analysis relies on using pre-trained SSL models as feature extractors, and training phoneme classifiers to identify the de- codable window of a phone. 2.1. HuBERT and Audio-visual HuBERT We used an audio-only and an audio-visual SSL model as the feature extractors for the corresponding modality. We chose the A V-HuBERT model [...

work page
[3]

Dataset We analyzed the phonetic decodability window on two datasets separately

Task description 3.1. Dataset We analyzed the phonetic decodability window on two datasets separately. The phone boundaries and labels for both datasets were obtained through the Montreal forced aligner on English arpabet dictionary [18, 19]. Grid: The GRID dataset [20] includes speech videos col- lected in a controlled environment, with 34 speaker sittin...

work page
[4]

Experiments 4.1. Full context window We initially experimented with HuBERTs features extracted given full utterance as inputs, which potentially allows them to encode information from other phones in the utterance. Layer 9 of HuBERT / A V-HuBERT is selected for feature extraction, be- cause it has relatively high correlation to phonetic information among ...

work page
[5]

We found that A V-HuBERT’s encoding of speech temporal dynam- ics is dominated by its audio input

Conclusion In this paper, we investigated how audio-visual SSL models capture the natural asynchronicity between audio and visual cues in speech, by using a sequence of linear classifier to de- code phone identity from each frame of A V-HuBERT/HuBERT model features in a context window centered at phone onset. We found that A V-HuBERT’s encoding of speech ...

work page
[6]

The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models

reported that the McGurk effect could emerge without spe- cific training on the latest audio-visual self-supervised learn- ing (SSL) speech representation model, Audio-visual HuBERT (A V-HuBERT) [7]. They thus suggested that audio-visual self- supervised learning (SSL) models could be useful for simulat- ing how humans perform audio-visual integration. A ...

work page Pith review arXiv 2025
[7]

We would like to thank Biao Zeng from University of South Wales and Hao Tang, Sharon Goldwater from ILCC, Univer- sity of Edinburgh for useful discussion

Acknowledgements This work was supported by the UKRI Centre for Doctoral Training (CDT) in Natural Language Processing, funded by the UKRI grant EP/S022481/1 and the University of Edinburgh. We would like to thank Biao Zeng from University of South Wales and Hao Tang, Sharon Goldwater from ILCC, Univer- sity of Edinburgh for useful discussion. We would li...

work page

Show all 28 references

[9]

Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recur- rent automatic speech recognition systems,

M. Keshishian, G. Mischler, S. Thomas, B. Kingsbury, S. Bickel, A. D. Mehta, and N. Mesgarani, “Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recur- rent automatic speech recognition systems,”bioRxiv, pp. 2025– 01, 2025

2025
[10]

Hearing lips and seeing voices,

H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976

1976
[11]

Multisensory integration: current issues from the perspective of the single neuron,

B. E. Stein and T. R. Stanford, “Multisensory integration: current issues from the perspective of the single neuron,”Nature reviews neuroscience, vol. 9, no. 4, pp. 255–266, 2008

2008
[12]

Multimodal deep learning

J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y . Nget al., “Multimodal deep learning.” inICML, vol. 11, 2011, pp. 689– 696

2011
[13]

On the role of noise in audiovisual integration: Evidence from artificial neural networks that exhibit the McGurk effect,

L. Grasse and M. S. Tata, “On the role of noise in audiovisual integration: Evidence from artificial neural networks that exhibit the McGurk effect,”arXiv preprint arXiv:2411.05715, 2024

2024
[14]

Learning audio-visual speech representation by masked multimodal cluster prediction,

B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,”arXiv preprint arXiv:2201.02184, 2022

2022 arXiv
[15]

The natural statistics of audiovisual speech,

C. Chandrasekaran, A. Trubanova, S. Stillittano, A. Caplier, and A. A. Ghazanfar, “The natural statistics of audiovisual speech,” PLoS computational biology, vol. 5, no. 7, p. e1000436, 2009

2009
[16]

Bimodal speech: early suppressive visual effects in human auditory cor- tex,

J. Besle, A. Fort, C. Delpuech, and M.-H. Giard, “Bimodal speech: early suppressive visual effects in human auditory cor- tex,”European journal of Neuroscience, vol. 20, no. 8, pp. 2225– 2234, 2004

2004
[17]

Visual speech speeds up the neural processing of auditory speech,

V . Van Wassenhove, K. W. Grant, and D. Poeppel, “Visual speech speeds up the neural processing of auditory speech,”Proceedings of the National Academy of Sciences, vol. 102, no. 4, pp. 1181– 1186, 2005

2005
[18]

Asynchronicity between visual and auditory information in audiovisual speech: Evidence from four types of consonant- words/b/,/t/,/k/and/g,

B. Zeng, K. Evans, M. Carne, L. Game, and E. Persson 1, “Asynchronicity between visual and auditory information in audiovisual speech: Evidence from four types of consonant- words/b/,/t/,/k/and/g,” inProc. AVSEC 2024, 2024, pp. 43–46

2024
[19]

A predictive learning model can simulate temporal dynamics and context ef- fects found in neural representations of continuous speech,

O. D. Liu, H. Tang, N. Feldman, and S. Goldwater, “A predictive learning model can simulate temporal dynamics and context ef- fects found in neural representations of continuous speech,”arXiv preprint arXiv:2405.08237, 2024

2024 arXiv
[20]

Neural dynamics of phoneme sequences reveal position-invariant code for content and order,

L. Gwilliams, J.-R. King, A. Marantz, and D. Poeppel, “Neural dynamics of phoneme sequences reveal position-invariant code for content and order,”Nature communications, vol. 13, no. 1, p. 6606, 2022

2022
[21]

HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

2021
[22]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

2016
[23]

Dynamic encoding of acoustic features in neural responses to continuous speech,

B. Khalighinejad, G. C. da Silva, and N. Mesgarani, “Dynamic encoding of acoustic features in neural responses to continuous speech,”Journal of Neuroscience, vol. 37, no. 8, pp. 2176–2185, 2017

2017
[24]

Representation learning with contrastive predictive coding,

A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[25]

Montreal forced aligner: Trainable text-speech align- ment using Kaldi

M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using Kaldi.” inInterspeech, vol. 2017, 2017, pp. 498–502

2017
[26]

Prosodylab-aligner: A tool for forced alignment of laboratory speech,

K. Gorman, J. Howell, and M. Wagner, “Prosodylab-aligner: A tool for forced alignment of laboratory speech,”Canadian Acous- tics, vol. 39, no. 3, pp. 192–193, 2011

2011
[27]

An audio- visual corpus for speech perception and automatic speech recog- nition,

M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio- visual corpus for speech perception and automatic speech recog- nition,”The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421–2424, 2006

2006
[28]

LRS3-TED: a large-scale dataset for visual speech recognition,

T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: a large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [8]

Using artificial neural networks to ask ‘why’ questions of minds and brains,

N. Kanwisher, M. Khosla, and K. Dobs, “Using artificial neural networks to ask ‘why’ questions of minds and brains,”Trends in Neurosciences, vol. 46, no. 3, pp. 240–254, 2023

work page 2023

[2] [1]

Introduction Recent years have seen a rising trend in using artificial neural networks (ANN) to model sensory processes in humans [1]. For speech processing, it has been shown that the neural activity in the auditory cortex of human listeners correlates with represen- tations in a recurrent neural network model trained for speech recognition, with corresp...

work page

[3] [2]

Related works and method Our analysis relies on using pre-trained SSL models as feature extractors, and training phoneme classifiers to identify the de- codable window of a phone. 2.1. HuBERT and Audio-visual HuBERT We used an audio-only and an audio-visual SSL model as the feature extractors for the corresponding modality. We chose the A V-HuBERT model [...

work page

[4] [3]

Dataset We analyzed the phonetic decodability window on two datasets separately

Task description 3.1. Dataset We analyzed the phonetic decodability window on two datasets separately. The phone boundaries and labels for both datasets were obtained through the Montreal forced aligner on English arpabet dictionary [18, 19]. Grid: The GRID dataset [20] includes speech videos col- lected in a controlled environment, with 34 speaker sittin...

work page

[5] [4]

Experiments 4.1. Full context window We initially experimented with HuBERTs features extracted given full utterance as inputs, which potentially allows them to encode information from other phones in the utterance. Layer 9 of HuBERT / A V-HuBERT is selected for feature extraction, be- cause it has relatively high correlation to phonetic information among ...

work page

[6] [5]

We found that A V-HuBERT’s encoding of speech temporal dynam- ics is dominated by its audio input

Conclusion In this paper, we investigated how audio-visual SSL models capture the natural asynchronicity between audio and visual cues in speech, by using a sequence of linear classifier to de- code phone identity from each frame of A V-HuBERT/HuBERT model features in a context window centered at phone onset. We found that A V-HuBERT’s encoding of speech ...

work page

[7] [6]

The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models

reported that the McGurk effect could emerge without spe- cific training on the latest audio-visual self-supervised learn- ing (SSL) speech representation model, Audio-visual HuBERT (A V-HuBERT) [7]. They thus suggested that audio-visual self- supervised learning (SSL) models could be useful for simulat- ing how humans perform audio-visual integration. A ...

work page Pith review arXiv 2025

[8] [7]

We would like to thank Biao Zeng from University of South Wales and Hao Tang, Sharon Goldwater from ILCC, Univer- sity of Edinburgh for useful discussion

Acknowledgements This work was supported by the UKRI Centre for Doctoral Training (CDT) in Natural Language Processing, funded by the UKRI grant EP/S022481/1 and the University of Edinburgh. We would like to thank Biao Zeng from University of South Wales and Hao Tang, Sharon Goldwater from ILCC, Univer- sity of Edinburgh for useful discussion. We would li...

work page