Pith. sign in

REVIEW 4 major objections 4 minor 39 references

CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Feeding BART discrete tokens that encode general-sound semantics, obtained by vector-quantizing BEATs audio representations, improves automated audio captioning over codec-style waveform tokens.

desk verdict Useful, honest incremental AAC result; the semantic-token story is plausible but not fully isolated from token length and codebook training. read the letter →

arxiv 2506.00800 v1 pith:BPJO2KFD submitted 2025-06-01 eess.AS cs.LGcs.SD

Daiki Takeuchi , Binh Thien Nguyen , Masahiro Yasuda , Yasunori Ohishi , Daisuke Niizumi , Noboru Harada This is my paper
classification eess.AScs.LGcs.SD
keywords automatedaudiocaptioningdiscretetokenresidualvectorquantizationrepresentationBEATsEnCLAPCapsClotho
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that automated audio captioning improves when the language model receives discrete audio tokens that carry semantic content about general sounds, rather than tokens optimized only for reconstructing waveforms. The authors build CLAP-ART, which vector-quantizes features from the pretrained audio representation model BEATs and feeds the resulting tokens to BART together with a CLAP audio embedding. On the AudioCaps and Clotho benchmarks, CLAP-ART outperforms the EnCLAP baseline on every reported metric. If the claim is right, caption quality can be raised by choosing what the discrete input tokens represent, without extra training data or a larger language model.

What carries the argument

The central mechanism is the audio representation tokenizer (ART), which converts continuous, semantically rich audio representations into discrete tokens. BEATs-RVQ, the main variant, takes 768-dimensional feature vectors from BEATs' audio encoder and passes them through 16 residual vector quantization layers with 1024-entry codebooks, projecting each vector to its nearest codebook entry and quantizing the residual at each layer. The resulting multi-layer discrete tokens are embedded and concatenated with a projected CLAP audio embedding to form the input sequence for BART.

What would settle it

Run CLAP-ART and EnCLAP with matched token sequence lengths, for example by resampling BEATs-RVQ tokens to EnCodec's temporal rate; if the SPIDEr advantage disappears, the improvement is due to input format rather than semantic content.

Watch

Extended reading notes

Core claim

The central claim is that semantic-rich discrete tokens derived from semantically rich audio representations are better inputs than reconstruction-oriented codec tokens for fine-tuning a language model in automated audio captioning. CLAP-ART instantiates this by replacing EnCodec tokens with tokens from an audio representation tokenizer (ART): BEATs-AT uses BEATs' acoustic tokenizer directly, and BEATs-RVQ applies residual vector quantization to BEATs' audio encoder outputs, with k-means codebooks built on the training set. The experiments show CLAP-ART outperforming EnCLAP across METEOR, CIDEr, SPICE, SPIDEr, and FENSE on both AudioCaps and Clotho, and the ablation without the CLAP embedding degrades far less than EnCLAP's, supporting the attribution of the gain to the semantic content of the ART tokens.

Load-bearing premise

The method assumes that the BEATs features that survive vector quantization actually carry the general-sound semantics that improve caption generation, rather than the gain coming from other differences such as token sequence length or codebook design.

Editorial extensions

If this is right

  • Replacing reconstruction-oriented EnCodec tokens with semantic quantized tokens from BEATs improves captioning on AudioCaps and Clotho across all five reported metrics.
  • ART discrete tokens alone, without the CLAP embedding, retain enough semantic information to reach captioning scores close to the full EnCLAP baseline, making discrete semantic tokens a strong standalone input.
  • Increasing the number of RVQ layers from 1 to 16 raises performance, indicating that multi-layer quantization packs more useful semantic information for caption generation.
  • Tokens derived from speech-focused models such as SpeechTokenizer and HuBERT-RVQ do not improve over EnCodec, so the relevant semantics are those of general sounds rather than speech phonemes or linguistic content.
  • With a BART-base model trained only on AudioCaps, CLAP-ART reaches SPIDEr scores near a 7-billion-parameter LLM-based method that uses extra data, which the paper reads as evidence that token representation choice matters as much as language-model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the comparison does not hold token sequence length fixed: BEATs features are flattened into a long vector sequence while EnCodec tokens follow their own frame rate, so a matched-length experiment would separate the semantic-content effect from the input-format effect.
  • Because the BEATs model was trained with AudioSet supervision, a natural extension is to test CLAP-ART on audio domains outside AudioSet categories; larger gains on AudioCaps, a subset of AudioSet, suggest the benefit may depend on overlap between the representation model's pretraining domain and the captioning domain.
  • The RVQ codebooks are computed on the training set before fine-tuning; learning or adapting the codebooks jointly with the captioning objective could be a testable improvement that may further tune the tokens to the captioning task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript proposes CLAP-ART, an automated audio captioning method that feeds BART with discrete tokens derived from semantically rich audio representations: BEATs-AT, which naively uses BEATs' acoustic tokenizer, and BEATs-RVQ, which applies residual vector quantization to BEATs' audio encoder features. These tokens are concatenated with a CLAP audio embedding and used to fine-tune BART, following the EnCLAP framework. The authors report that CLAP-ART outperforms EnCLAP on AudioCaps and Clotho across all evaluated metrics, with ablations removing the CLAP embedding and varying the number of RVQ layers. The paper's central hypothesis is that semantic-rich discrete tokens are more beneficial for AAC than reconstruction-oriented EnCodec tokens.

Significance. If the causal attribution were established, this would be a useful design principle for AAC: tokenizers should preserve semantic content rather than waveform fidelity. The paper has genuine strengths: it uses six random seeds with standard deviations, reproduces the EnCLAP baseline, includes a w/o-CLAP ablation that directly tests the informativeness of the discrete tokens, and compares against SpeechTokenizer- and HuBERT-based alternatives. These make the main comparative result—that BEATs-derived tokens help—fairly robust. However, the title and conclusions make a stronger causal claim about semantic richness than the current experiments isolate, because the central comparison varies several factors at once.

major comments (4)
  1. [Sec. 4.1 and Table 1] The central attribution of gains to 'semantic richness' is confounded with token sequence length and codebook training. In Section 4.1, BEATs-RVQ flattens the BEATs encoder output into F*T 768-dimensional vectors, whereas EnCLAP's EnCodec tokens have a time length L in Eq. (1); these lengths are not matched. Additionally, the EnCodec codebooks are learned end-to-end for reconstruction, while the BEATs-RVQ codebooks are obtained by k-means on the training features. The +3.5 SPIDEr gain in Table 1 could therefore be explained by differences in input sequence length, codebook capacity, or codebook learning objective rather than by semantic content per se. A control that varies only the source representation while keeping the tokenizer pipeline, sequence length, and codebook size fixed (for example, applying the same RVQ/k-means procedure to EnCodec encoder features) is needed to support the title's causal claim.
  2. [Sec. 5.2] The paper acknowledges that BEATs iter3+ was pre-trained on AudioSet and that AudioCaps is a subset of AudioSet, but it does not treat this as a threat to the central claim. The largest improvements are on AudioCaps (SPIDEr +3.5 for BEATs-RVQ in Table 1), while the Clotho gains are modest (+1.3), and the Clotho experiments are pre-trained on AudioCaps (Section 5.1), so they do not remove the overlap. Because the hypothesis concerns general sounds, the authors should either evaluate on a benchmark disjoint from the AR pre-training data or use an AR model whose pre-training does not include labels from the evaluation domain; at minimum, the conclusion should be tempered to acknowledge that the observed benefit may partly reflect AudioSet-label leakage.
  3. [Sec. 5.4 and Table 3] The claim that increasing the number of RVQ layers improves performance by packing more semantic information is based only on mean scores in Table 3, with no standard deviations or significance tests, although six seeds were used for the other tables. The differences between adjacent settings are small (e.g., SPIDEr 49.7 for N=8 vs. 49.8 for N=16 in Table 3), and the trend is not strictly monotonic (SPIDEr 48.8 for N=2 vs. 48.6 for N=4). Reporting the six-seed variability would make it possible to assess whether the multi-layer advantage is reliable, and the conclusion should be scaled to what the data support.
  4. [Secs. 3 and 5.2, Eqs. (3)-(4)] The paper's hypothesis has two parts: the tokens should be semantic-rich and discrete. The experiments replace EnCodec tokens with BEATs-derived tokens, but they never compare BEATs-RVQ discrete tokens against continuous BEATs features in the same BART fine-tuning framework. Such a comparison is needed to show that quantization itself contributes to the improvement; without it, the gains could be due to the BEATs features alone, and the claim that discrete tokens are beneficial for BART would not be isolated. The w/o-CLAP ablation in Table 2 shows that the ART tokens alone carry information, but it does not test the continuous-input alternative.
minor comments (4)
  1. [Eq. (4)] Equation (4) contains a typo: 'ICLAP-AR T' should be 'I_{CLAP-ART}' or similar.
  2. [Sec. 4.1] In Section 4.1, the notation 'F × T' for patch dimensions and then 'length F T' is easy to confuse with the time length T used elsewhere; consider renaming the patch dimensions to avoid the clash.
  3. [Sec. 5.5] In Table 4 and the surrounding text, the 'Reference' rows are the best of six runs and are clearly labeled, but the sentence suggesting the result is 'comparable to the use of LLM' should explicitly remind the reader that LOAE's score comes from the original paper and is not averaged over seeds.
  4. [Reproducibility] The manuscript does not provide a link to an implementation of CLAP-ART; releasing code and checkpoints would strengthen the reproducibility of the reported six-seed results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CLAP-ART's gains are empirical benchmark results, not derived from fitted constants or self-citations.

full rationale

The paper is an empirical AAC method paper. Its central claim—that feeding BART discrete tokens derived from BEATs representations improves captioning over EnCodec tokens—is established by direct benchmark experiments (Tables 1–4), not derived from a fitted constant or from a self-citation. The ART codebooks are learned by k-means on the training set (Section 4.1), which is standard tokenizer construction and is not a 'prediction' of the benchmark scores; the benchmark scores are generated by fine-tuning BART and measuring held-out captions. BEATs and EnCodec are external pretrained models; EnCLAP is an external baseline reproduced from its official implementation. The only self-citation (Koizumi et al. [4]) appears in a related-work description and is not load-bearing. The paper even includes negative results (SpeechTokenizer, HuBERT-RVQ) and ablations (w/o CLAP, RVQ layer count) that could have falsified the central hypothesis. Potential confounds such as differing token sequence length or codebook training objectives are threats to the internal validity of the attribution to 'semantic richness,' but they are not cases where an equation reduces to its own input or where a fitted parameter is renamed as a prediction. Hence no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on the semantic quality of BEATs and the assumption that quantization preserves that quality, plus a set of hyperparameters adopted from EnCLAP or tuned by grid search. No new entities are postulated.

free parameters (6)
  • Number of RVQ layers (N) = 16
    Set equal to EnCodec's layer count; no search over N is reported, and Table 3 shows performance grows with N.
  • RVQ codebook size = 1024
    Set equal to EnCodec's codebook size; the choice is not justified beyond matching the baseline.
  • Total loss weight lambda for MCM = 0.7
    Fixed following EnCLAP's setup, not tuned here.
  • Peak learning rate for AudioCaps = 6e-5
    Adopted from EnCLAP training settings.
  • Peak learning rate for Clotho = 2e-4
    Chosen by grid search; the search range and criterion are not reported (Section 5.1).
  • Pre-training epochs on AudioCaps for Clotho = 5
    Set by grid search; details not reported.
assumptions (4)
  • domain assumption BEATs iter3+ audio encoder and acoustic tokenizer provide semantically rich representations of general sounds.
    Relies on BEATs pre-training with AudioSet supervision; cited prior work, not re-verified here (Section 2.2, 5.1).
  • domain assumption Discrete tokens from RVQ preserve enough semantic information for downstream captioning.
    Central to ART; the paper tests this indirectly but does not measure semantic content directly (Section 4.1).
  • domain assumption The EnCLAP framework (BART with CLAP embedding and discrete tokens) is a valid base for comparison.
    Adopted from EnCLAP [6]; no ablation of the framework itself (Section 3).
  • domain assumption AudioCaps and Clotho benchmarks are representative of general-sound captioning.
    Standard benchmarks, but generalizability beyond these datasets is not shown (Section 5.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer." pith.science (2026). https://pith.science/paper/BPJO2KFD

@misc{pith2026250600800,
  author       = {Pith},
  title        = {Pith review of: CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPJO2KFD}},
  note         = {Machine review of arXiv:2506.00800}
}
read the original abstract

Automated Audio Captioning (AAC) aims to describe the semantic contexts of general sounds, including acoustic events and scenes, by leveraging effective acoustic features. To enhance performance, an AAC method, EnCLAP, employed discrete tokens from EnCodec as an effective input for fine-tuning a language model BART. However, EnCodec is designed to reconstruct waveforms rather than capture the semantic contexts of general sounds, which AAC should describe. To address this issue, we propose CLAP-ART, an AAC method that utilizes ``semantic-rich and discrete'' tokens as input. CLAP-ART computes semantic-rich discrete tokens from pre-trained audio representations through vector quantization. We experimentally confirmed that CLAP-ART outperforms baseline EnCLAP on two AAC benchmarks, indicating that semantic-rich discrete tokens derived from semantically rich AR are beneficial for AAC.

Figures

Figures reproduced from arXiv: 2506.00800 by the authors.

Figure 1
Figure 1. Overview of CLAP-ART. CLAP-ART utilizes ART semantic-rich discrete tokens as BART input along with a CLAP audio embedding. which are then fed into the BART language model [16] along￾side CLAP audio embeddings. This approach has validated the assertion that “pre-trained language models can leverage dis￾crete inputs better compared to their continuous inputs.” This performance enhancement is likely attributable to the… view at source ↗
Figure 2
Figure 2. Details of BEATs-RVQ. The preprocessing phase (a) obtains codebooks in advance of fine-tuning BART. The training and inference phase (b) converts input sound to discrete tokens. produces multi-layer discrete tokens, which are expected to en￾hance AAC performance by providing richer semantic infor￾mation. CLAP-ART feeds these semantic-rich discrete tokens alongside a CLAP audio embedding into BART to generate cap￾tio… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 35 canonical work pages

  1. [1]

    CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer

    Introduction Automated Audio Captioning (AAC) is a crossmodal text generation task focused on producing descriptions of diverse sounds [1–3]. Unlike traditional audio classification tasks that assign labels to the sound, AAC seeks to comprehend and artic- ulate the semantic contexts surrounding various acoustic events and scenes. To enhance the quality of...

    work page Pith review arXiv 2025

  2. [2]

    Automated Audio Captioning Methods using Pre- trained Language Model Pre-trained language models have been utilized in some stud- ies to improve AAC performance [4–11]

    Related work 2.1. Automated Audio Captioning Methods using Pre- trained Language Model Pre-trained language models have been utilized in some stud- ies to improve AAC performance [4–11]. In order to provide semantic information to language models, pre-trained AR fea- tures are used as input. Typical approaches simply pass AR fea- tures to language models ...

    work page

  3. [3]

    First, En- CLAP transforms an input audio waveform into a CLAP audio embedding and EnCodec discrete tokens

    Baseline: EnCLAP EnCLAP leverages EnCodec alongside the CLAP audio en- coder to extract input features for BART fine-tuning. First, En- CLAP transforms an input audio waveform into a CLAP audio embedding and EnCodec discrete tokens. The CLAP audio em- bedding is projected onto the BART input embedding space by using a linear layer, resulting in eclap ∈ RD...

    work page

  4. [4]

    Proposed Method CLAP-ART leverages the audio representation tokenizer (ART) proposed in this paper alongside the CLAP audio encoder to extract input features for BART fine-tuning. 4.1. Audio Representation Tokenizer ART converts input sounds into semantic-rich discrete tokens based on the knowledge of AR models. We implemented two types of ART: BEATs-AT a...

    work page

  5. [5]

    Experimental Setup We conducted experiments on two AAC datasets: Audio- Caps [23] and Clotho [24]

    Experiments 5.1. Experimental Setup We conducted experiments on two AAC datasets: Audio- Caps [23] and Clotho [24]. For the experiments on AudioCaps, we trained and evaluated models solely on AudioCaps. For the experiments on Clotho, we first pre-trained models on Audio- Caps and then fine-tuned and evaluated them on Clotho. In the setup of CLAP-ART, we u...

    work page

  6. [6]

    semantic-rich and discrete

    Conclusions This paper proposed CLAP-ART, which improves AAC per- formance by utilizing “semantic-rich and discrete” tokens, which capture fine-grained semantic contexts of general sounds. CLAP-ART is based on our hypothesis that semantic-rich dis- crete tokens are beneficial inputs for fine-tuning language mod- els. To obtain such discrete tokens, we des...

    work page

  7. [7]

    Acknowledgements This work was partially supported by JST Strategic Interna- tional Collaborative Research Program (SICORP), Grant Num- ber JPMJSC2306, Japan

    work page

  8. [8]

    Automated audio captioning with recurrent neural networks,

    K. Drossos, S. Adavanne, and T. Virtanen, “Automated audio captioning with recurrent neural networks,” in Workshop Appl. Signal Process. Audio Acoust. (WASPAA). IEEE, 2017, pp. 374– 378

    work page 2017

Show all 39 references
  1. [9]

    Automated audio captioning: An overview of recent progress and new challenges,

    X. Mei, X. Liu, M. D Plumbley, and W. Wang, “Automated audio captioning: An overview of recent progress and new challenges,” EURASIP J. Audio Speech Music Process. , vol. 2022, no. 1, pp. 26, 2022

    2022

  2. [10]

    Beyond the status quo: A contemporary survey of advances and challenges in audio cap- tioning,

    X. Xu, Z. Xie, M. Wu, and K. Yu, “Beyond the status quo: A contemporary survey of advances and challenges in audio cap- tioning,” IEEE/ACMTrans. Audio Speech Lang. Process., vol. 32, pp. 95–112, 2024

    2024

  3. [11]

    Audio captioning using pre-trained large-scale language model guided by audio-based similar caption retrieval,

    Y . Koizumi, Y . Ohishi, D. Niizumi, D. Takeuchi, and Masahiro Y ., “Audio captioning using pre-trained large-scale language model guided by audio-based similar caption retrieval,” arXiv preprint arXiv:2012.07331, 2020

    2012 arXiv

  4. [12]

    Automated audio cap- tioning by fine-tuning bart with audioset tags,

    F. Gontier, R. Serizel, and C. Cerisara, “Automated audio cap- tioning by fine-tuning bart with audioset tags,” inProc. Workshop Detect. Classif. Acoust. Scenes Events (DCASE), 2021

    2021

  5. [13]

    Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning,

    J. Kim, J. Jung, J. Lee, and S. H. Woo, “Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2024, pp. 6735–6739

    2024

  6. [14]

    Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D Plumb- ley, Y . Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Trans. Audio Speech Lang. Process., 2024

    2024

  7. [15]

    Improving audio captioning models with fine- grained audio features, text embedding supervision, and llm mix- up augmentation,

    S.-L. Wu, X. Chang, G. Wichern, J. Jung, F. Germain, J. Le Roux, and S. Watanabe, “Improving audio captioning models with fine- grained audio features, text embedding supervision, and llm mix- up augmentation,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). ...

    2024

  8. [16]

    Recap: Retrieval-augmented audio captioning,

    S. Ghosh, S. Kumar, C. K. Reddy Evuru, R. Duraiswami, and D. Manocha, “Recap: Retrieval-augmented audio captioning,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2024, pp. 1161–1165

    2024

  9. [17]

    Taming data and transformers for audio generation,

    M. Haji-Ali, W. Menapace, A. Siarohin, G. Balakrishnan, S. Tulyakov, and V . Ordonez, “Taming data and transformers for audio generation,” arXiv preprint arXiv:2406.19388, 2024

    2024 arXiv

  10. [18]

    Enhancing automated audio captioning via large language models with optimized audio encoding,

    J. Liu, G. Li, J. Zhang, H. Dinkel, Y . Wang, Z. Yan, Y . Wang, and B. Wang, “Enhancing automated audio captioning via large language models with optimized audio encoding,” in Proc. Inter- speech, 2024

    2024

  11. [19]

    BEATs: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” in Proc. Int. Conf. Mach. Learn. (ICML) , Jul. 2023, pp. 5178–5193

    2023

  12. [20]

    Audio Set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2017, pp. 776–780

    2017

  13. [21]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2023, pp. 1–5

    2023

  14. [22]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res., 2023

    2023

  15. [23]

    BART: Denoising sequence-to-sequence pre-training for natural language genera- tion, translation, and comprehension,

    Mike Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language genera- tion, translation, and comprehension,” in Proc. 58th Annu. Meet. Assoc. Comput. Linguist. (AC...

    2020

  16. [24]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu, X. Xu, Y . Yuan, M. Wu, W. Wang, and M. D Plumb- ley, “Semanticodec: An ultra low bitrate semantic audio codec for general sound,” arXiv preprint arXiv:2405.00233, 2024

    2024 arXiv

  17. [25]

    Speechtok- enizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech language models,” in Proc. Int. Conf. Learn. Represent., 2024

    2024

  18. [26]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021

    2021

  19. [27]

    SoundStream: An end-to-end neural audio codec,

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM Trans. Audio Speech Lang. Process. , vol. 30, pp. 495–507, 2022

    2022

  20. [28]

    vq- wav2vec: Self-supervised learning of discrete speech representa- tions,

    Alexei Baevski, Steffen Schneider, and Michael Auli, “vq- wav2vec: Self-supervised learning of discrete speech representa- tions,” in International Conference on Learning Representations, 2020

    2020

  21. [29]

    How should we extract discrete audio tokens from self-supervised models?,

    P. Mousavi, J. Duret, S. Zaiem, L. Della Libera, A. Ploujnikov, C. Subakan, and M. Ravanelli, “How should we extract discrete audio tokens from self-supervised models?,” in Interspeech 2024, 2024, pp. 2554–2558

    2024

  22. [30]

    Audiocaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in Proc. Conf. N. Am. Chapter Assoc. Comput. Linguist. (NAACL), 2019, pp. 119–132

    2019

  23. [31]

    Clotho: An audio cap- tioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). IEEE, 2020, pp. 736–740

    2020

  24. [32]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  25. [33]

    Meteor universal: Language spe- cific translation evaluation for any target language,

    M. Denkowski and A. Lavie, “Meteor universal: Language spe- cific translation evaluation for any target language,” in Proc. Workshop Stat. Mach. Transl., June 2014, pp. 376–380

    2014

  26. [34]

    CIDEr: Consensus-based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 4566–4575

    2015

  27. [35]

    SPICE: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “SPICE: Semantic propositional image caption evaluation,” in Eur. Conf. Comput. Vis. (ECCV), 2016, pp. 382–398

    2016

  28. [36]

    Improved image captioning via policy gradient optimization of spider,

    S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in IEEE Int. Conf. Comput.Vis. (ICCV), 2017, pp. 873–881

    2017

  29. [37]

    Can audio captions be evaluated with image caption metrics?,

    Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q Zhu, “Can audio captions be evaluated with image caption metrics?,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). IEEE, 2022, pp. 981–985

    2022

  30. [38]

    Enclap++: Ana- lyzing the enclap framework for optimizing automated audio cap- tioning performance,

    J. Kim, M. Jeon, J. Jung, S. H. Woo, and J. Lee, “Enclap++: Ana- lyzing the enclap framework for optimizing automated audio cap- tioning performance,” in Proc. Workshop Detect. Classif. Acoust. Scenes Events (DCASE),, 2024, pp. 61–65

    2024

  31. [39]

    Slam-aac: Enhancing audio captioning with paraphras- ing augmentation and clap-refine through llms,

    W. Chen, Z. Ma, X. Li, X. Xu, Y . Liang, Z. Zheng, K. Yu, and X. Chen, “Slam-aac: Enhancing audio captioning with paraphras- ing augmentation and clap-refine through llms,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.