Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Fusion of Discrete Representations and Self-Augmented Representations for Multilingual Automatic Speech Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fusing two discrete speech-unit streams cuts character error rate by up to 24 percent while keeping transmission bitrate near 665 bits per second.

desk verdict The fusion result holds up; the bitrate confound raised by the stress-test does not. read the letter →

arxiv 2411.18107 v1 pith:6RPDJUB2 submitted 2024-11-27 cs.SD eess.AS

Shih-heng Wang , Jiatong Shi , Chien-yu Huang , Shinji Watanabe , Hung-yi Lee This is my paper · ORCID
classification cs.SDeess.AS
keywords automaticspeechrecognitiondiscreterepresentationself-supervisedlearningfusionmechanismcross-attentionmultilingualASRself-augmentedbitrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that automatic speech recognition can get the accuracy benefits of continuous self-supervised features while keeping the small storage, fast I/O, and short sequences of discrete units. It proposes to fuse two discrete representations through a cross-attention layer that learns to align the two streams after de-duplication and subword compression have made them non-linearly misaligned. On LibriSpeech-100h and ML-SUPERB, the fusion lowers character error rate by 19% and 24% relative to a single discrete stream, matching or slightly beating a continuous-feature baseline while using roughly 0.3% of its bitrate. The paper also introduces self-augmented discrete representations, derived by reshaping or taking deltas of a single SSL feature, that remove the need for a second SSL model and give more language-robust gains.

What carries the argument

The fusion mechanism is a modified transformer encoder layer: the primary discrete representation passes through self-attention, while a cross-attention layer treats it as the query and the secondary representation as key and value, after the secondary embedding has gone through a down-then-up adapter projection. A learnable scalar weight $\alpha$ combines the self-attention and cross-attention outputs before the MLP, acting as a gate between the streams. The self-augmented representations are the same MMS-1B features reshaped from $(T, D_{ssl})$ to $(2T, D_{ssl}/2)$ before discretization, or their frame-to-frame delta features discretized separately.

What would settle it

Train the identical model on the same data but with the secondary stream replaced by random tokens of the same length and vocabulary; if the CER drop relative to the one-stream baseline still approaches the reported 19–24%, the improvement would be shown to come from added capacity or regularization rather than from complementary information in the second stream.

Watch

Extended reading notes

Core claim

The central claim is that a learnable cross-attention fuse of two discrete speech-unit streams improves ASR accuracy over either stream alone, without sacrificing the compression benefits of discreteness. Concretely, with the multilingual MMS-1B features as the primary stream and the English WavLM-Large features as the secondary, the model achieves a 19% relative CER reduction on LibriSpeech and a 24% relative reduction on ML-SUPERB compared with the non-fusion MMS-1B baseline, at a bitrate of roughly 665 bits per second versus about two million for continuous features. The paper further claims that discrete streams produced by reshaping or differencing a single SSL representation work nearly as well and are more consistent across the 143 languages of ML-SUPERB.

Load-bearing premise

The gains rest on the assumption that cross-attention can learn a meaningful alignment between two independently de-duplicated and BPE-compressed unit sequences; if the attention alignment fails for a new pair of streams, the reported CER improvements would not generalize even though the architecture and training recipe are unchanged.

Editorial extensions

If this is right

  • Discrete-unit ASR can reach or slightly exceed continuous-feature accuracy at a transmission bitrate of 0.3% of continuous, making low-bandwidth and on-device recognition practical.
  • The fusion mechanism is architecturally scalable: adding more streams means adding more cross-attention layers and expanding the weighted sum, so the same idea extends beyond two discrete representations.
  • Because self-augmented Delta and Reshape streams need only one SSL forward pass, the second stream can be produced on a cheap CPU, taking about 14% of the time needed to run a second SSL model.
  • The language-wise analysis frames a design trade-off: fusing with WavLM-Large gives the largest accuracy gain, while fusing with Delta or Reshape yields fewer declining languages and lower cross-language variance.
  • Both orderings of the two streams (primary and secondary) work, with a slight preference for putting the stronger representation first, indicating the learned gate controls information flow robustly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The learned gate $\alpha$ could be read as a per-layer reliability measure; a future variant might adjust it dynamically per utterance or per language when one stream becomes uninformative, for example in code-switched or noisy audio.
  • The cross-attention fusion pattern should transfer to any pair of token sequences with nonlinear misalignment, such as text units from different subword tokenizers or multimodal discrete tokens, since the attention itself performs the alignment.
  • A testable extension is to make the gate $\alpha$ conditional on a language embedding, directly targeting the residual cross-language variance the paper reports, rather than training one global gate for all 143 languages.
  • The paper's bitrate accounting counts only the discrete token streams; a fuller system-level comparison would also include model parameters and the K-Means codebook, though the relative claim that discrete streams are orders of magnitude cheaper than continuous features stands.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a fusion mechanism for discrete speech representations in multilingual ASR. Two discrete unit streams (e.g., from MMS-1B and WavLM-Large) are integrated via cross-attention layers inserted into a Transformer encoder, with a learnable weighted sum between self-attention and cross-attention outputs. The authors also introduce 'self-augmented' discrete representations (Reshape and Delta) derived from a single SSL model to avoid a second forward pass. Experiments on LibriSpeech-100h and ML-SUPERB report consistent CER improvements over a non-fusion MMS-1B discrete baseline, up to 19% and 24% relative improvement, while keeping bitrates near the discrete level (around 665 bits/s vs 2,048,000 bits/s for continuous features). The paper also analyzes primary/secondary stream order, language-wise robustness, and inference cost.

Significance. If the reported gains are real, the work is a practical contribution to efficient ASR with discrete units: it offers a way to recover some of the accuracy lost when using discrete SSL features, while preserving the storage, I/O, and sequence-length benefits of discrete representations. The paper is clearly written and follows the Interspeech 2024 Discrete Speech Unit Challenge protocol, which makes the numbers directly comparable with that benchmark. Strengths include the direct experimental measurements, the inclusion of a parameter-cost estimate (+24% parameters), and the explicit bitrate formula. The main weakness is that the central comparison is not cleanly bitrate-matched in a way that controls for input sequence length, leaving the attribution of the gains to 'fusion' vs. simply higher bitrate unresolved.

major comments (3)
  1. [§4.4, Table 3] The 'Non-Fusion high bitrate' baseline (MMS-1B, bitrate 556.15) is obtained by removing de-duplication and BPE from the discrete representation, which nearly doubles the input sequence length (Table 2: Train goes from ~203 to ~393 frames). Transformer self-attention complexity and optimization behavior depend strongly on sequence length, so this control does not isolate the effect of higher information rate from the effect of longer input sequences. Consequently, the comparison between the fusion variant (MMS-1B + WavLM-Large, 665.13 bits/s) and this baseline cannot support the claim that the 19%/24% relative CER gains come from the fusion mechanism per se. A single-stream control with the same bitrate and the same de-duplicated/BPE-compressed sequence length is needed.
  2. [§5.1, Table 3] The self-augmented variants (Reshape at 1024.90 bits/s and Delta at 648.52 bits/s) are compared only against the non-fusion MMS-1B baseline at 280.86 bits/s. This confounds bitrate with the proposed fusion/augmentation approach. Without a bitrate-matched single-stream control (e.g., a larger codebook or a larger BPE vocabulary applied to MMS-1B alone), the reported 6% and 19% improvements for self-augmented representations cannot be attributed to the method rather than to the increased bitrate.
  3. [§5.1, Table 3 and §4.4] The paper's claim that 'fusion variants achieve performance slightly superior to the continuous representation baseline while using only 0.3% of its bitrate' is interesting but not directly related to the bitrate-matched issue. The continuous baseline (2.34/10.89) is reasonably close to the fusion result (1.89/10.87), and this is a meaningful efficiency gain. However, the comparison between fusion and continuous uses different model architecture and input features, so it is not a controlled ablation; this is acceptable as an efficiency demonstration but should not be over-interpreted as evidence about the fusion mechanism.
minor comments (5)
  1. [§5.1, Table 3] No error bars or multi-seed statistics are reported. Given that some differences are small (e.g., 1.89 vs 1.92 on LibriSpeech), reporting variability (or at least stating the single-seed nature) would improve reliability.
  2. [§3.1, Eq. (3)] The initialization of the learnable alpha parameter is not specified. It would be useful to know whether it is initialized to favor the primary stream or uniform.
  3. [§5.2, 'Inference cost analysis'] The inference-cost measurement compares only feature-extraction time (WavLM-Large forward pass vs. delta transformation), not end-to-end ASR inference time. The conclusion that fusion with Delta 'significantly reduces inference costs' should be scoped to feature extraction.
  4. [§2.2, Table 2] The text says sequence length 'can be halved,' but the table shows a 48% reduction (Train) and 46% (Dev), which is close to but not exactly half. Consider saying 'nearly halved.'
  5. [§4.3] The description of the adapter uses 'a down-projection linear layer, a non-linear activation, and an up-projection linear layer,' but does not state the activation function; please name it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the CER and bitrate claims are direct measurements on held-out test sets, not derived from fitted equations or self-referential definitions.

full rationale

The paper's central claims are empirical evaluations: the fusion mechanism's CER and bitrate are measured on LibriSpeech and ML-SUPERB test sets following a fixed challenge protocol. No derived constant or fitted equation is presented, and the improvement (19%/24% relative CER reduction) is a direct comparison of held-out test-set measurements against baselines. The self-citations (e.g., ML-SUPERB benchmark, the challenge protocol, and prior discrete-unit ASR work) supply external benchmarks and standard preprocessing recipes; they do not define the fusion mechanism's output or preordain its measured performance. The fusion architecture is specified independently (cross-attention with learnable weighted sum), and the experimental baselines are separately trained models. Concerns about the high-bitrate baseline not being sequence-length-matched, or about post-hoc selection of the primary representation, are experimental-design limitations that affect interpretation of the cause of the improvement, but they are not circular reasoning: the reported numbers are not forced by construction, by definition, or by a self-citation chain. Thus no circular step meeting the evidentiary bar is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The free parameters are standard experimental choices, but two of them (SSL layer index and fusion gate) directly control the reported gains. The axioms are domain assumptions inherited from prior SSL and ASR work, with the cross-attention alignment assumption being the most load-bearing.

free parameters (6)
  • K-means cluster count = 2000
    Set for all SSL models; controls granularity of discrete units and hence ASR accuracy and bitrate.
  • BPE vocabulary size = 3000
    Chosen following [21] but smaller; controls subword compression and token sequence length.
  • SSL layer index = WavLM layer 21, MMS-1B layer 48
    Chosen without reported ablation; representation quality and complementarity depend on the layer.
  • Adapter bottleneck size = 128
    Selected for the cross-attention input projection; affects capacity of secondary stream integration.
  • Fusion weight alpha = learned per model
    Scalar gate in Eq. (3) is optimized on training data; central to the fusion mechanism.
  • Delta window width = 9
    Librosa default used to compute delta features; impacts the dynamic representation.
assumptions (4)
  • domain assumption Discrete units from K-Means clustering over SSL features retain enough information for competitive ASR.
    Invoked throughout Sections 2.1 and 4.2; inherited from [13,21], not re-validated in this paper.
  • domain assumption Cross-attention can learn useful alignment between de-duplicated and BPE-compressed unit sequences despite non-linear length differences.
    Core assumption of the fusion mechanism in Section 3.1; if false, fusion would not improve over baselines.
  • domain assumption WavLM-Large and MMS-1B features are complementary for multilingual ASR.
    Stated in Section 4.2 with motivation from pretraining data; not directly measured.
  • domain assumption ML-SUPERB and LibriSpeech test sets and the CER metric adequately reflect ASR quality.
    Standard benchmark assumption in Section 4.5; no independent validation of metric choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fusion of Discrete Representations and Self-Augmented Representations for Multilingual Automatic Speech Recognition." pith.science (2026). https://pith.science/paper/6RPDJUB2

@misc{pith2026241118107,
  author       = {Pith},
  title        = {Pith review of: Fusion of Discrete Representations and Self-Augmented Representations for Multilingual Automatic Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RPDJUB2}},
  note         = {Machine review of arXiv:2411.18107}
}
read the original abstract

Self-supervised learning (SSL) models have shown exceptional capabilities across various speech-processing tasks. Continuous SSL representations are effective but suffer from high computational and storage demands. On the other hand, discrete SSL representations, although with degraded performance, reduce transmission and storage costs, and improve input sequence efficiency through de-duplication and subword-modeling. To boost the performance of discrete representations for ASR, we introduce a novel fusion mechanism that integrates two discrete representations. The fusion mechanism preserves all the benefits of discrete representation while enhancing the model's performance by integrating complementary information. Additionally, we explore "self-augmented'' discrete representations, which apply transformations to a single continuous SSL representation, eliminating the fusion mechanism's dependency on multiple SSL models and further decreasing its inference costs. Experimental results on benchmarks, including LibriSpeech and ML-SUPERB, indicate up to 19% and 24% relative character error rate improvement compared with the non-fusion baseline, validating the effectiveness of our proposed methods.

Figures

Figures reproduced from arXiv: 2411.18107 by the authors.

Figure 3
Figure 3. Differing from the standard encoder layer, which typi [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 39 canonical work pages

  1. [1]

    Fusion of Discrete Representations and Self-Augmented Representations for Multilingual Automatic Speech Recognition

    INTRODUCTION Self-supervised learning (SSL) models have demonstrated excep- tional success across a variety of speech-processing tasks [1–11]. Prior works mostly focused on leveraging continuous SSL represen- tations [11, 12], which, despite their effectiveness, are notorious for their high storage and computational costs. To address these issues, recent ...

    work page Pith review arXiv 2024

  2. [2]

    Discretization process Figure 1 provides a high-level overview of our fusion pipeline

    DISCRETE REPRESENTATION FOR ASR 2.1. Discretization process Figure 1 provides a high-level overview of our fusion pipeline. In this section, we would like to introduce how to derive the final dis- crete representations d for fusion. Figure 2 shows the process of discretization of speech inputs (the Self Augment part will be in- troduced in Section 3.2). F...

    work page

  3. [3]

    METHODOLOGIES In this section, we first introduce the general picture and details of our fusion mechanism (Sec. 3.1). Next, we introduce two repre- sentation augmentation methods to derive self-augmented discrete representations (Sec. 3.2). 3.1. Discrete representation fusion mechanism As discussed in Section 1, due to the non-linear misalignment be- twee...

    work page

  4. [4]

    Dataset We evaluated the proposed method on LibriSpeech-100h [29] and ML-SUPERB [12]

    EXPERIMENTAL SETTINGS 4.1. Dataset We evaluated the proposed method on LibriSpeech-100h [29] and ML-SUPERB [12]. The setting is aligned with the Discrete Speech Unit Challenge4. LibriSpeech-100h evaluates the English ASR ca- pability, providing 100 hours of clean English paired data. On the other hand, ML-SUPERB assesses the multilingual ASR capability, c...

    work page 2024

  5. [5]

    Quantitative result ASR results

    RESULT & ANALYSIS 5.1. Quantitative result ASR results. Table 3 showcases the CER of our baselines and fu- sion variants on the LibriSpeech and ML-SUPERB datasets. Rows without a secondary representation (-) indicate non-fusion baselines. The results demonstrate that all fusion variants consistently out- perform the non-fusion baselines, even with compara...

    work page

  6. [6]

    self-augmented

    CONCLUSION We propose a novel fusion mechanism that integrates two non- linearly misaligned discrete representations by utilizing an attention- based approach. Most importantly, our proposed methods preserve all of the advantages of discrete representation but with enhanced performance. Additionally, we developed “self-augmented” rep- resentations by effi...

    work page

  7. [7]

    HuBERT: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    Wei-Ning Hsu et al., “HuBERT: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

    work page 2021

  8. [8]

    A Survey of Multi- lingual Models for Automatic Speech Recognition,

    Hemant Yadav and Sunayana Sitaram, “A Survey of Multi- lingual Models for Automatic Speech Recognition,” in LREC, 2022, pp. 5071–5079

    work page 2022

Show all 40 references
  1. [9]

    Cross-lingual Automatic Speech Recog- nition Exploiting Articulatory Features,

    Qingran Zhan et al., “Cross-lingual Automatic Speech Recog- nition Exploiting Articulatory Features,” in APSIPA ASC) , 2019, pp. 1912–1916

    2019

  2. [10]

    wav2vec 2.0: A framework for self- supervised learning of speech representations,

    Alexei Baevski et al., “wav2vec 2.0: A framework for self- supervised learning of speech representations,” Advances in Neural Information Processing Systems , vol. 33, pp. 12449– 12460, 2020

    2020

  3. [11]

    XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,

    Arun Babu et al., “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,” in Proc. Inter- speech, 2022, pp. 2278–2282

    2022

  4. [12]

    W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

    Yu-An Chung et al., “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in ASRU, 2021, pp. 244–250

    2021

  5. [13]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    Sanyuan Chen et al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, pp. 1505–1518, 2021

    2021

  6. [14]

    Self-supervised speech repre- sentation learning: A review,

    Abdelrahman Mohamed et al., “Self-supervised speech repre- sentation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022

    2022

  7. [15]

    Multi-resolution huBERT: Multi-resolution speech self-supervised learning with masked unit prediction,

    Jiatong Shi et al., “Multi-resolution huBERT: Multi-resolution speech self-supervised learning with masked unit prediction,” in Proc. ICLR, 2024

    2024

  8. [16]

    Scaling speech technology to 1,000+ languages,

    Vineel Pratap, , et al., “Scaling speech technology to 1,000+ languages,” arXiv, 2023

    2023

  9. [17]

    SUPERB: Speech Processing Universal PERformance Benchmark,

    Shu wen Yang et al., “SUPERB: Speech Processing Universal PERformance Benchmark,” in Proc. Interspeech, 2021, pp. 1194–1198

    2021

  10. [18]

    ML-SUPERB: Multilingual Speech Uni- versal PERformance Benchmark,

    Jiatong Shi et al., “ML-SUPERB: Multilingual Speech Uni- versal PERformance Benchmark,” in Proc. Interspeech, 2023, pp. 884–888

    2023

  11. [19]

    Exploring speech recognition, transla- tion, and understanding with discrete speech units: A compar- ative study,

    Xuankai Chang et al., “Exploring speech recognition, transla- tion, and understanding with discrete speech units: A compar- ative study,” 2023

    2023

  12. [20]

    An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks,

    Kai-Wei Chang et al., “An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks,” in Proc. Interspeech, 2022, pp. 5005–5009

    2022

  13. [21]

    Towards universal speech discrete tokens: A case study for asr and tts,

    Yifan Yang et al., “Towards universal speech discrete tokens: A case study for asr and tts,” in ICASSP 2024 - 2024 IEEE In- ternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), 2024

    2024

  14. [22]

    Acoustic bpe for speech generation with dis- crete tokens,

    Feiyu Shen et al., “Acoustic bpe for speech generation with dis- crete tokens,” in ICASSP 2024 - 2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11746–11750

    2024

  15. [23]

    V oxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks,

    Soumi Maiti et al., “V oxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks,” in ICASSP 2024 - 2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 13326–13330

    2024

  16. [24]

    TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript- Conditioned Speech Separation and Recognition,

    Hakan Erdogan et al., “TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript- Conditioned Speech Separation and Recognition,” in Proc. In- terspeech, 2023, pp. 3462–3466

    2023

  17. [25]

    Akvsr: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,

    Jeong Hun Yeo et al., “Akvsr: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,” IEEE Transactions on Multimedia , pp. 1–13, 2024

    2024

  18. [26]

    Lip reading for low-resource lan- guages by learning and combining general speech knowl- edge and language-specific knowledge,

    Minsu Kim et al., “Lip reading for low-resource lan- guages by learning and combining general speech knowl- edge and language-specific knowledge,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15359–15371

    2023

  19. [27]

    Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learning,

    Xuankai Chang, , et al., “Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learning,” in Proc. Interspeech, 2023, pp. 1399–1403

    2023

  20. [28]

    EFFUSE: Efficient self-supervised fea- ture fusion for e2e asr in multilingual and low resource scenar- ios,

    Tejes Srivastava et al., “EFFUSE: Efficient self-supervised fea- ture fusion for e2e asr in multilingual and low resource scenar- ios,” in Proc. Interspeech, 2024

    2024

  21. [29]

    FeaRLESS: Feature Refinement Loss for Ensembling Self-Supervised Learning Features in Robust End-to-end Speech Recognition,

    Szu-Jui Chen et al., “FeaRLESS: Feature Refinement Loss for Ensembling Self-Supervised Learning Features in Robust End-to-end Speech Recognition,” in Proc. Interspeech, 2022, pp. 3058–3062

    2022

  22. [30]

    Combining spectral and self-supervised features for low resource speech recognition and translation,

    Dan Berrebbi et al., “Combining spectral and self-supervised features for low resource speech recognition and translation,” in Proc. Interspeech, 2022

    2022

  23. [31]

    Many-to-many spoken language translation via unified speech and text representation learning with unit- to-unit translation,

    Minsu Kim et al., “Many-to-many spoken language translation via unified speech and text representation learning with unit- to-unit translation,” 2023

    2023

  24. [32]

    Intelligible Lip-to-Speech Synthesis with Speech Units,

    Jeongsoo Choi et al., “Intelligible Lip-to-Speech Synthesis with Speech Units,” in Proc. INTERSPEECH 2023, 2023, pp. 4349–4353

    2023

  25. [33]

    Tmt: Tri-modal translation between speech, image, and text by processing different modalities as different languages,

    Minsu Kim et al., “Tmt: Tri-modal translation between speech, image, and text by processing different modalities as different languages,” 2024

    2024

  26. [34]

    Towards practical and efficient image- to-speech captioning with vision-language pre-training and multi-modal tokens,

    Minsu Kim et al., “Towards practical and efficient image- to-speech captioning with vision-language pre-training and multi-modal tokens,” in ICASSP 2024 - 2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 7970–7974

    2024

  27. [35]

    Librispeech: An asr corpus based on public domain audio books,

    Vassil Panayotov et al., “Librispeech: An asr corpus based on public domain audio books,” in 2015 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  28. [36]

    The interspeech 2024 challenge on speech processing using discrete units,

    Xuankai Chang et al., “The interspeech 2024 challenge on speech processing using discrete units,” in Proc. Interspeech, 2024

    2024

  29. [37]

    Attention is all you need,

    Ashish Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems . 2017, vol. 30, Cur- ran Associates, Inc

    2017

  30. [38]

    ESPnet: End-to-end speech processing toolkit,

    Shinji Watanabe et al., “ESPnet: End-to-end speech processing toolkit,” in Proc. Interspeech, 2018, pp. 2207–2211

    2018

  31. [39]

    librosa: Audio and music signal analy- sis in python,

    Brian McFee et al., “librosa: Audio and music signal analy- sis in python,” in Proceedings of the 14th python in science conference, 2015, vol. 8

    2015

  32. [40]

    Adam: A method for stochastic optimization,

    Diederik P. Kingma et al., “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014

    2014 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.