Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

ADIFF: Explaining audio difference using natural language

T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ADIFF, a prefix-tuning model with a separator token, cross-projection layer, and three-stage training, produces audio difference explanations that outperform both the naive baseline and a much larger two-audio language model.

desk verdict Useful new benchmark task with solid baseline, but the caption-derived references make the metric partly linguistic; treat it as measuring caption-contrast reasoning, not unmediated audio understanding. read the letter →

arxiv 2502.04476 v1 pith:ULL4QEMJ submitted 2025-02-06 cs.SD cs.AIeess.AS

Soham Deshmukh , Shuo Han , Rita Singh , Bhiksha Raj This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords audiodifferenceexplanationaudio-languagemodelprefixtuningcomparativereasoningcaptioningcross-projectionhallucinationdetectionthree-tier
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish audio difference explanation as a distinct task: given two recordings, a model should say in natural language how they differ, at three levels—concise event differences, brief scene-and-signal comparisons, and detailed explanations that include listener emotions. To make the task studyable, the paper builds two datasets, ACD and CLD, by prompting a large language model with human captions from AudioCaps and Clotho, then human-verifies the test set. The proposed model, ADIFF, keeps a frozen small language model and prompts it with audio embeddings from both clips, adding a separator token, a cross-projection layer, and a three-stage training schedule. The paper reports that ADIFF beats the naive prefix-tuning baseline and a much larger two-audio audio-language model on SPIDEr and on human ratings of correctness, granularity, and readability. If the task sticks, comparative audio reasoning becomes a measurable, trainable capability instead of a manual forensic skill.

What carries the argument

The load-bearing mechanism is the cross-projection layer with a separator token: each audio is encoded by HTSAT, projected into the language model's latent space, joined by a separator embedding derived from the decoder's end-of-text token, and run through a transformer that mixes the two audio streams with the text prompt. The paper's analysis of the cross-projection output shows the text-prefix tokens come to store the comparison attributes—frequency range, pitch, loudness—that later appear in the generated explanation. Two supporting mechanisms carry the rest: position captioning (training the model to caption 'the first audio' or 'the second audio') stops it from confusing similar-sounding inputs, and the three-stage training schedule keeps pretrained audio and text knowledge intact before final finetuning.

What would settle it

Build a test set of audio pairs whose captions are identical but whose acoustics differ measurably (for example, the same dog bark with different pitch and reverberation); if ADIFF's explanations call the two clips the same while human listeners consistently report audible differences, the caption-proxy assumption fails and the model's apparent success would be an artifact of the benchmark construction.

Watch

Extended reading notes

Core claim

The central claim is that explaining audio differences reduces to a conditional language generation problem, and that the right inductive biases let a 128-million-parameter decoder do it well: a separator token in latent space keeps the two clips distinct, a cross-projection transformer reuses the text prefix to store the acoustic attributes being compared (frequency, pitch, dynamic range), and three-stage training—unimodal pretraining, frozen multimodal grounding, then low-learning-rate finetuning—preserves audio knowledge while steering generation. On the new ACD and CLD benchmarks, ADIFF's SPIDEr scores exceed both the naive baseline and Qwen-Audio finetuned variants on nearly all tiers, and human evaluators rate its explanations higher on correctness, granularity, and readability, with the largest margin in granularity.

Load-bearing premise

The benchmark's ground truth is written from human captions of the two clips, not by listening to the clips, so the task assumes caption-level differences faithfully stand in for acoustic differences.

Editorial extensions

If this is right

  • Audio difference explanation gains public benchmarks (ACD and CLD) with three tiers, so future audio-language models can be scored on comparative reasoning rather than only on captioning or question answering.
  • The three-tier design separates failure modes: Tier 1 is the hardest because almost every word carries audio content, while Tier 2 is linguistically easiest and Tier 3 tests detail and emotional or contextual description.
  • Under limited compute and data, smaller language models can be grounded in audio more effectively than larger ones, and larger models need extra training epochs to catch up.
  • Position captioning reduces confusion between two similar audio inputs, making the model more reliable on perceptually similar sounds.
  • A frozen audio event detector provides a hallucination check: predicted event probabilities over time let users spot generated events that are not present in the audio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ground-truth explanations are derived from caption pairs rather than from listening, the benchmark may reward caption-level linguistic contrast more than acoustic perception; a listening-only reference set would test whether ADIFF truly hears the differences.
  • The cross-projection finding—that text-prefix tokens end up storing comparison attributes—generalizes to other two-input generation tasks, such as comparing images, speech recordings, or medical scans, where the prompt region can act as a scratchpad for the contrast being described.
  • The hallucination-checking setup could be turned from a post-hoc debugging tool into a training objective that penalizes generated audio events whose predicted presence probability is near zero.
  • A direct stress test would pair clips that share identical captions but differ acoustically (same event, different pitch or reverberation); if the model reports 'no difference,' the caption-proxy assumption would be visible in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces the task of audio difference explanation (ADE), constructs two new datasets (ACD and CLD) by prompting an LLM with human-written captions from AudioCaps and Clotho, and proposes ADIFF, a prefix-tuning-based audio-language model with a separator token, a cross-projection layer, and a three-stage training pipeline. The authors evaluate ADIFF against a naive prefix-tuning baseline and Qwen-Audio variants using SPIDEr and human ratings of correctness, granularity, and readability, reporting consistent gains over the baseline and, on most tiers, over Qwen-Audio. The paper also presents ablations on cross-projection, language-model scaling, position captioning, and stage-3 finetuning, plus a qualitative hallucination-detection tool.

Significance. If the benchmark is valid, this is a novel and useful contribution to audio-language understanding: it formalizes a task that has practical importance (audio forensics, quality assessment, generation), provides two datasets with tiered explanations, and offers a reproducible baseline with informative ablations. The human evaluation is a strength, as are the clear architecture ablations and the explicit language-only control in Section 5.1. However, the benchmark's validity depends on an untested assumption that caption-level differences, as synthesized by an LLM, faithfully represent audio-level differences; this assumption is load-bearing for the central claim.

major comments (2)
  1. [Section 2.1, Appendix E] The ground-truth explanations in ACD and CLD are generated by prompting an LLM with two human-written captions, not by listening to the audio. The human verification of the test set is mentioned but never documented: there is no protocol, no annotator counts, no inter-annotator agreement, and no evidence that verifiers listened to the audio rather than edited the text for plausibility. As a result, the benchmark may reward caption-contrast and linguistic fluency rather than acoustic understanding. This concern is substantiated by Table 4, Experiment A, where a language-only model with a randomly initialized, frozen audio encoder reaches SPIDEr of 0.589 on CLD Tier 2. The authors should provide an explicit verification protocol, report agreement statistics, and ideally add a human study that assesses whether the reference explanations actually distinguish the two audio clips when listened to. Without this, the central claim that ADIFF improves audio difference explanation is not fully supported.
  2. [Section 2.2 and Section 5.1] The objective evaluation relies heavily on SPIDEr, but the paper shows that a language-only model can achieve substantial SPIDEr scores (e.g., 0.589 on CLD Tier 2), indicating that a large portion of the metric reflects linguistic patterns and not audio content. Although Section 5.1 identifies this bias, the main results (Table 2) are still presented without any correction or lower-bound reference. The authors should either report language-only performance as a lower bound for every metric and tier, or develop a more audio-grounded evaluation (e.g., event-level accuracy or a metric that explicitly penalizes audio-independent generation). This is necessary to support the claim that the reported improvements are due to audio understanding rather than better language modeling.
minor comments (4)
  1. [Section 2.1, Table 1] The textual description of the dataset statistics does not match Table 1: the text says the ACD Train split has 48k examples with median length 27, max 49, and vocabulary 6528 for Tier 1, but Table 1 lists 48,660 examples, median 29, max 47, and vocabulary 3,287 for ACD Train Tier 1; the text numbers actually match the CLD Train row. Please correct this discrepancy.
  2. [Table 6] The right half of Table 6 appears to repeat the ACD rows after the first two rows; the expected CLD rows are missing or mislabeled. Please fix the table so that the ACD and CLD conditions are clearly separated.
  3. [Appendix E] The description of the audio-pair sampling procedure says the randomly picked second audio excludes indices i through i+4, but the rationale for this exclusion is never explained; clarifying whether this avoids temporal proximity or some other bias would help reproducibility.
  4. [Section 3.2] Equations (1)-(3) are mathematically clear, but the text would benefit from an explicit statement of the separator token's role and the cross-projection layer's learned parameters; currently the reader must infer these details from the figure and the appendix.

Circularity Check

1 steps flagged · score 2.0 of 10

Central ADIFF results are independent; only the appendix 'tier granularity' verification is circular by construction.

  1. self definitional [Appendix Q.1 and Table 23; tier definition in Section 2.1]
    "Due to budget constraints, we use an LLM to simulate human evaluation and provide scores between 1 and 5. ... The results demonstrate that Tier-3 information density scores are the highest ... (Appendix Q.1). This follows the tier definition in Section 2.1: 'we define the tier of explanation by restricting the sources the LLM can use and the length of the explanation.'"

    Tier 3 is constructed by a prompt that explicitly restricts the LLM to longer, more detailed explanations, so the tier ordering is baked into the generation procedure. The appendix then 'demonstrates' that Tier 3 has the highest granularity by asking another LLM to rate detail and granularity, i.e., by re-measuring the same length/detail axis used to define the tiers. The high Tier-3 score is therefore entailed by construction rather than being an independent confirmation. This is a peripheral dataset-quality argument; it does not bear on the central ADIFF-vs-baseline comparison, which is evaluated on held-out test explanations and on external human ratings.

full rationale

The paper's central claim, that ADIFF improves over the naive prefix-tuning baseline and Qwen-Audio, rests on held-out test-set SPIDEr scores (Table 2) and on human ratings where annotators listened to external Studio/FSD50K/GTZAN audio (Table 3). The ACD/CLD references are generated by an LLM from human-written captions, but that LLM is not the model under test, and the test split is stated to be human-verified (Section 2.1), so the main evaluation does not assume the conclusion. The language-only baseline (Table 4, Exp. A) is presented honestly as a diagnostic of linguistic leakage, not as a prediction, and the paper explicitly attributes Tier 2 gains to linguistic simplicity. The self-citations to the authors' prior prefix-tuning work and LLM prompting setup are methodological inheritance, are accompanied by independent references such as Mokady et al. (2021), and are not used as a uniqueness theorem or as the sole justification for the central result. The only genuinely circular step is the appendix's information-density 'verification' that Tier 3 is more granular than Tier 1, which merely re-measures the length/detail restriction used to generate the tiers. Because this step is peripheral to the main contribution, the overall circularity score is low.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a synthetic data pipeline (LLM over human captions) and on frozen pretrained components. The only hand-set parameters are architectural hyperparameters; the trained weights are learned from data and are not listed as free parameters in the physical-sense ledger.

free parameters (4)
  • prefix length (audio 40 + text 40 + separator 1 = 121) = 121
    Chosen by hand for the transformer projection; affects the amount of audio context provided to the frozen LM, and the paper does not ablate it.
  • cross-projection transformer layers = 4
    Number of layers in the cross-projection module; not ablated, fixed to 4.
  • training epochs (stage 2: 30, stage 3: 10) = 30/10
    Selected to balance compute and performance; the scaling experiment shows larger LMs need more epochs, so this choice favors the base model.
  • decoding parameters top-k=3, top-p=0.8 = k=3, p=0.8
    Chosen to improve diversity; the choice affects all reported results, and greedy decoding performs worse (Appendix N).
assumptions (5)
  • domain assumption HTSAT pretrained on AudioSet provides audio embeddings that capture sound events and scenes.
    Section 3.1: HTSAT is used as frozen audio encoder; the paper relies on its generalization.
  • domain assumption Frozen GPT-2 can be steered by a prefix of projected audio embeddings.
    Sections 3.1-3.2: prefix-tuning is assumed to condition generation; supported by prior Pengi work.
  • ad hoc to paper LLM-generated difference explanations from human captions are valid training and evaluation targets.
    Section 2.1 and Appendix E: the datasets are built by prompting an LLM with captions; no audio is listened to during data generation, so the ground truth is an indirect proxy.
  • domain assumption Human verification of only the test set makes the test references reliable.
    Section 2.1: train and validation references are unverified and may contain hallucinations.
  • domain assumption Captioning metrics (BLEU, METEOR, SPIDEr) are appropriate for measuring explanation quality.
    Section 2.2: standard metrics are used despite known weaknesses in penalizing diverse correct outputs; the paper acknowledges this and adds human eval.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADIFF: Explaining audio difference using natural language." pith.science (2026). https://pith.science/paper/ULL4QEMJ

@misc{pith2026250204476,
  author       = {Pith},
  title        = {Pith review of: ADIFF: Explaining audio difference using natural language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULL4QEMJ}},
  note         = {Machine review of arXiv:2502.04476}
}
read the original abstract

Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model's ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.

Figures

Figures reproduced from arXiv: 2502.04476 by the authors.

Figure 1
Figure 1. Humans use auditory information to compare scenes and make deductions. In the Boston Marathon bombing of 2014, forensic in￾vestigators were faced with a challenge – were the var￾ious audio recordings purportedly captured of the event (and put up on social media) by various people record￾ings of the same event, or were they mistaken or fraudu￾lent uploads actually from different events? This anec￾dote highlights the … view at source ↗
Figure 2
Figure 2. A random sample from the ACD dataset is displayed across three levels of explanation. The top pane provides a concise explanation, the middle pane offers a brief explanation, and the bottom pane presents a detailed explanation. In this section, we outline the development steps for the AudioCaps Difference (ACD) and Clotho Difference (CLD) datasets. Audio recordings. We source the audio recordings from the AudioCaps … view at source ↗
Figure 3
Figure 3. ADIFF takes two audio recordings and text prompt as input and generates free-form text as output. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Change in average score across tiers with increase in LM parameters. To determine if increased computational power helps larger LMs, we train the models longer for an additional 20 epochs. The outcomes, shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Audio event presence probabilities from AD [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Using LLM to generate audio difference explanation. The process involves three key steps: sampling [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Comparison of vocabulary differences between explanatory tiers: The left figure illustrates words that [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: The plots for different ablation studies are organized as follows: The (1,1) subplot represents the [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: We compare two models: first, GPT2, the model after stage-2 multimodal grounding, Second, GPT2 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Comparing ADIFF generation against current ALMs across three scenarios: Contrasting Acoustic [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: A randomly selected example showcases the model’s detailed explanation of audio differences, [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing

    cs.SD 2025-07 conditional novelty 7.0 of 10

    MixAssist is the first audio-grounded, multi-turn conversational dataset for co-creative music mixing instruction, and fine-tuning Qwen-Audio on it yields human-comparable mixing advice.

  2. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  3. Breaking the Barriers of Text-Hungry and Audio-Deficient AI

    cs.SD 2025-06 reject novelty 4.0 of 10

    A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.

Reference graph

Works this paper leans on

68 extracted references · 43 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Getting vit in shape: Scaling laws for compute-optimal model design

    Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp.\ 382--398. Springer, 2016

    work page 2016

  4. [4]

    METEOR : An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pp.\ 65--72, Ann Arbor, Michigan,...

    work page 2005

  5. [5]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr \'e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024

    arXiv 2024

  6. [6]

    Selm: Enhancing speech emotion recognition for out-of-domain scenarios

    Hazim Bukhari, Soham Deshmukh, Hira Dhamyal, Bhiksha Raj, and Rita Singh. Selm: Enhancing speech emotion recognition for out-of-domain scenarios. In Interspeech 2024, pp.\ 2345--2349, 2024. doi:10.21437/Interspeech.2024-2257

    work page doi:10.21437/interspeech.2024-2257 2024

  7. [7]

    Audio quality assessment techniques—a review, and recent developments

    Dermot Campbell, Edward Jones, and Martin Glavin. Audio quality assessment techniques—a review, and recent developments. Signal Processing, 89 0 (8): 0 1489--1500, 2009

    work page 2009

  8. [8]

    Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 646--650. IEEE, 2022

    work page 2022

Show all 68 references
  1. [9]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  2. [10]

    Pengi: An audio language model for audio tasks

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36: 0 18090--18108, 2023 a

    2023

  3. [11]

    Audio Retrieval with WavText5K and CLAP Training

    Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio Retrieval with WavText5K and CLAP Training . In Proc. INTERSPEECH 2023, pp.\ 2948--2952, 2023 b . doi:10.21437/Interspeech.2023-1136

    2023 doi

  4. [12]

    Pam: Prompting audio-language models for audio quality assessment

    Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mahmoud Al Ismail , Rita Singh, Bhiksha Raj, and Huaming Wang. Pam: Prompting audio-language models for audio quality assessment. In Interspeech 2024, pp.\ 3320--3324, 2024 a . doi:10.21437/Interspeech.2024-325

    2024 doi

  5. [14]

    Audio entailment: Assessing deductive reasoning for audio understanding

    Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding. arXiv preprint arXiv:2407.18062, 2024 c

    2024 arXiv

  6. [15]

    Domain adaptation for contrastive audio-language models

    Soham Deshmukh, Rita Singh, and Bhiksha Raj. Domain adaptation for contrastive audio-language models. In Interspeech 2024, pp.\ 1680--1684, 2024 d . doi:10.21437/Interspeech.2024-41

    2024 doi

  7. [17]

    Automated audio captioning with recurrent neural networks

    Konstantinos Drossos, Sharath Adavanne, and Tuomas Virtanen. Automated audio captioning with recurrent neural networks. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp.\ 374--378. IEEE, 2017

    2017

  8. [18]

    Clotho: an audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. doi:10.1109/ICASSP40776.2020.9052990

    2020

  9. [19]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  10. [20]

    Natural language supervision for general-purpose audio representations

    Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general-purpose audio representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 336--340. IEEE, 2024

    2024

  11. [21]

    Fsd50k: an open dataset of human-labeled sound events

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 829--852, 2021

    2021

  12. [22]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Process...

    2017

  13. [23]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406.11768, 2024 a

    2024 arXiv

  14. [24]

    Compa: Addressing the gap in compositional reasoning in audio-language models

    Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ramaneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conferenc...

    2024

  15. [25]

    Joint audio and speech understanding

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.\ 1--8. IEEE, 2023

    2023

  16. [26]

    Liu, Leonid Karlinsky, and James R

    Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=nBZBPXdJlC

    2024

  17. [27]

    Clip4idc: Clip for image difference captioning

    Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. Clip4idc: Clip for image difference captioning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Proce...

    2022

  18. [28]

    Heller, Benjamin Elizalde, Bhiksha Raj, and Soham Deshmukh

    Laurie M. Heller, Benjamin Elizalde, Bhiksha Raj, and Soham Deshmukh. Synergy between human and machine approaches to sound/scene recognition and processing: An overview of icassp special session. arXiv preprint arXiv:2302.09719, 2023. URL https://arxiv.org/abs/2302.09719

    2023

  19. [29]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference...

    2022

  20. [30]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

  21. [31]

    Learning to describe differences between pairs of similar images

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,...

    2018 doi

  22. [32]

    Acoustic and auditory phonetics

    Keith Johnson and Keith Johnson. Acoustic and auditory phonetics. Phonetica, 61 0 (1): 0 56--58, 2004

    2004

  23. [33]

    Deductive reasoning

    Philip N Johnson-Laird. Deductive reasoning. Annual review of psychology, 50 0 (1): 0 109--135, 1999

    1999

  24. [34]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  25. [35]

    AudioCaps: Generating Captions for Audios in The Wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild . In NAACL-HLT, 2019

    2019

  26. [36]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  27. [37]

    Audio retrieval with natural language queries: A benchmark study

    A Sophia Koepke, Andreea-Maria Oncescu, Jo \ a o F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25: 0 2675--2685, 2022

    2022

  28. [39]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/foru...

    2024

  29. [40]

    Digital audio forensics: a first practical evaluation on microphone and environment classification

    Christian Kraetzer, Andrea Oermann, Jana Dittmann, and Andreas Lang. Digital audio forensics: a first practical evaluation on microphone and environment classification. In Proceedings of the 9th Workshop on Multimedia and Security, MM Sec '07, pp.\ 63–74, New York, NY, USA, 20...

    2007

  30. [41]

    Audiogen: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D \'e fossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations

  31. [42]

    Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models

    Chun-Yi Kuan, Wei-Ping Huang, and Hung yi Lee. Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models. In Interspeech 2024, pp.\ 4144--4148, 2024. doi:10.21437/Interspeech.2024-1076

    2024 doi

  32. [43]

    Clotho-aqa: A crowdsourced dataset for audio question answering

    Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pp.\ 1140--1144, 2022. doi:10.23919/EUSIPCO55093.2022.9909680

    2022

  33. [44]

    Audioldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning, pp.\ 21450--21474. PMLR, 2023

    2023

  34. [45]

    Improved image captioning via policy gradient optimization of spider

    Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pp.\ 873--881, 2017

    2017

  35. [46]

    Robert C. Maher. Audio forensic examination. IEEE Signal Processing Magazine, 26 0 (2): 0 84--94, 2009. doi:10.1109/MSP.2008.931080

    2009

  36. [47]

    Automated audio captioning: An overview of recent progress and new challenges

    Xinhao Mei, Xubo Liu, Mark D Plumbley, and Wenwu Wang. Automated audio captioning: An overview of recent progress and new challenges. EURASIP journal on audio, speech, and music processing, 2022 0 (1): 0 26, 2022 a

    2022

  37. [48]

    Diverse audio captioning via adversarial training

    Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang. Diverse audio captioning via adversarial training. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 8882--8886. IEEE, 2022 b

    2022

  38. [49]

    Towards generating diverse audio captions via adversarial training

    Xinhao Mei, Xubo Liu, Jianyuan Sun, and Mark D Plumbley. Towards generating diverse audio captions via adversarial training. IEEE/ACM transactions on audio, speech, and language processing, 2024 a

    2024

  39. [50]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Lan...

    2024

  40. [51]

    Clipcap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021

    2021 arXiv

  41. [52]

    Diversity and bias in audio captioning datasets

    Irene Martin Morato and Annamaria Mesaros. Diversity and bias in audio captioning datasets. In Detection and Classication of Acoustic Scenes and Events, pp.\ 90--94, 2021

    2021

  42. [53]

    On the audio hallucinations in large audio-video language models

    Taichi Nishimura, Shota Nakada, and Masayoshi Kondo. On the audio hallucinations in large audio-video language models. arXiv preprint arXiv:2401.09774, 2024

    2024 arXiv

  43. [54]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  44. [55]

    Robust change captioning

    Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    2019

  45. [56]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp.\ 28492--28518. PMLR, 2023

    2023

  46. [57]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  47. [58]

    Acoustic phonetics, volume 30

    Kenneth N Stevens. Acoustic phonetics, volume 30. MIT press, 2000

    2000

  48. [59]

    Audio difference captioning utilizing similarity-discrepancy disentanglement

    Daiki Takeuchi, Yasunori Ohishi, Daisuke Niizumi, Noboru Harada, and Kunio Kashino. Audio difference captioning utilizing similarity-discrepancy disentanglement. In Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023), pp....

    2023

  49. [60]

    Extending large language models for speech and audio captioning

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Extending large language models for speech and audio captioning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp....

    2024

  50. [61]

    SALMONN : Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN : Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.n...

    2024

  51. [62]

    Tzanetakis and P

    G. Tzanetakis and P. Cook. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10 0 (5): 0 293--302, 2002. doi:10.1109/TSA.2002.800560

    2002

  52. [63]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4566--4575, 2015

    2015

  53. [64]

    Beats-based audio captioning model with instructor embedding supervision and chatgpt mix-up

    Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, Fran c ois Germain, Jonathan Le Roux, and Shinji Watanabe. Beats-based audio captioning model with instructor embedding supervision and chatgpt mix-up. In Conf. Detection Classification Acoust. Scenes Events, Challenge...

  54. [65]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Sign...

    2023

  55. [66]

    Image difference captioning with pre-training and contrastive learning

    Linli Yao, Weiying Wang, and Qin Jin. Image difference captioning with pre-training and contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3108--3116, 2022

    2022

  56. [67]

    Pre-training language models for comparative reasoning

    Mengxia Yu, Zhihan Zhang, Wenhao Yu, and Meng Jiang. Pre-training language models for comparative reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 12421--12433, 2023

    2023

  57. [68]

    Narle: Natural language models using reinforcement learning with emotion feedback

    Ruijie Zhou, Soham Deshmukh, Jeremiah Greer, and Charles Lee. Narle: Natural language models using reinforcement learning with emotion feedback. arXiv preprint arXiv:2110.02148, 2021

    2021 arXiv

  58. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  59. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  60. [71]

    or ``caption the second audio

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2018

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.