{"id":"78352ac5-553b-4202-b863-1c07b684cdf9","arxiv_id":"2502.04476","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"This paper proposes the audio difference explanation task, creates two LLM-generated datasets (ACD and CLD) with three explanation tiers, and presents ADIFF, a prefix-tuning model with cross-projection that beats baselines and Qwen-Audio on the new benchmark.","lead":"Researchers introduce a new task where an AI system explains, in natural language, what is different between two audio recordings. They build two new datasets with three levels of explanation detail and a model, ADIFF, that outperforms existing audio-language models on this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth explanations are synthesized from captions, so benchmark validity depends on an untested assumption that caption-level differences equal audio-level differences; human verification of the test set is mentioned but not documented.","rationale":"Reader's weakest assumption is the same as the one I find most load-bearing: the benchmark target is generated from captions, not from listening. I agree with the reader's conditional assessment, and I do not think the concern by itself overturns the paper's empirical contribution; ADIFF still outperforms baselines on the constructed benchmark and on a separate human evaluation that involves listening. However, the benchmark's validity for 'audio difference explanation' hinges on the caption-proxy assumption. The paper's statement that the test set was human-verified is important but under-specified, and the high language-only scores in Table 4 show that the objective metrics can be gamed linguistically. A small, targeted audio-labeling study would settle whether the concern is real or merely hypothetical. Since the reader already conditions on this issue, the verdict remains conditional.","tokens_in":35918,"tokens_out":7286,"duration_ms":73939,"concrete_test":"Sample 100 ACD and 100 CLD test pairs. Have independent annotators (not involved in dataset creation) listen to each pair and (a) list the three most salient audio differences they hear, and (b) rate whether the existing ground-truth explanation is accurate and complete. Compute the proportion of annotator-listed differences covered by the ground truth, the rate of ground-truth claims judged inaudible, and inter-annotator agreement (e.g., Krippendorff's alpha). Pre-register a threshold, such as 80% coverage and below 10% inaudible claims. If the threshold is met, the caption-proxy assumption is supported; if not, the benchmark labels do not represent audio-level differences and the central comparison should be re-run on a human-audio-labeled subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is measured on ACD/CLD, whose reference explanations are produced by prompting an LLM with two human-written captions (Section 2.1, Appendix E). The model is trained and evaluated against these caption-derived outputs. For the benchmark to measure audio difference explanation, the contrast between captions must be a faithful proxy for the contrast a listener hears. The paper states that the test split was human-verified, but it provides no verification protocol, no annotator counts or agreement statistics, and no evidence that verification involved listening to the audio rather than editing text plausibility. If captions mention events that are absent or inaudible, or omit salient acoustic differences, then SPIDEr and even the human ratings of model outputs may reward linguistic fluency and caption-contrast rather than acoustic understanding. This affects all three tiers because the LLM prompt restricts length and sources but adds no acoustic evidence beyond the captions (Appendix E). The language-only baseline (Table 4, Exp. A) reaches substantial SPIDEr scores (e.g., 0.589 on CLD Tier 2) without any audio input, confirming that a large part of the metric is linguistic. The human evaluation in Table 3 uses studio/FSD50K/GTZAN audio, so it does not validate the ACD/CLD labels themselves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of audio difference explanation (ADE), constructs two new datasets (ACD and CLD) by prompting an LLM with human-written captions from AudioCaps and Clotho, and proposes ADIFF, a prefix-tuning-based audio-language model with a separator token, a cross-projection layer, and a three-stage training pipeline. The authors evaluate ADIFF against a naive prefix-tuning baseline and Qwen-Audio variants using SPIDEr and human ratings of correctness, granularity, and readability, reporting consistent gains over the baseline and, on most tiers, over Qwen-Audio. The paper also presents ablations on cross-projection, language-model scaling, position captioning, and stage-3 finetuning, plus a qualitative hallucination-detection tool.","tokens_in":36205,"tokens_out":5097,"duration_ms":49638,"significance":"If the benchmark is valid, this is a novel and useful contribution to audio-language understanding: it formalizes a task that has practical importance (audio forensics, quality assessment, generation), provides two datasets with tiered explanations, and offers a reproducible baseline with informative ablations. The human evaluation is a strength, as are the clear architecture ablations and the explicit language-only control in Section 5.1. However, the benchmark's validity depends on an untested assumption that caption-level differences, as synthesized by an LLM, faithfully represent audio-level differences; this assumption is load-bearing for the central claim.","major_comments":[{"comment":"The ground-truth explanations in ACD and CLD are generated by prompting an LLM with two human-written captions, not by listening to the audio. The human verification of the test set is mentioned but never documented: there is no protocol, no annotator counts, no inter-annotator agreement, and no evidence that verifiers listened to the audio rather than edited the text for plausibility. As a result, the benchmark may reward caption-contrast and linguistic fluency rather than acoustic understanding. This concern is substantiated by Table 4, Experiment A, where a language-only model with a randomly initialized, frozen audio encoder reaches SPIDEr of 0.589 on CLD Tier 2. The authors should provide an explicit verification protocol, report agreement statistics, and ideally add a human study that assesses whether the reference explanations actually distinguish the two audio clips when listened to. Without this, the central claim that ADIFF improves audio difference explanation is not fully supported.","section":"Section 2.1, Appendix E"},{"comment":"The objective evaluation relies heavily on SPIDEr, but the paper shows that a language-only model can achieve substantial SPIDEr scores (e.g., 0.589 on CLD Tier 2), indicating that a large portion of the metric reflects linguistic patterns and not audio content. Although Section 5.1 identifies this bias, the main results (Table 2) are still presented without any correction or lower-bound reference. The authors should either report language-only performance as a lower bound for every metric and tier, or develop a more audio-grounded evaluation (e.g., event-level accuracy or a metric that explicitly penalizes audio-independent generation). This is necessary to support the claim that the reported improvements are due to audio understanding rather than better language modeling.","section":"Section 2.2 and Section 5.1"}],"minor_comments":[{"comment":"The textual description of the dataset statistics does not match Table 1: the text says the ACD Train split has 48k examples with median length 27, max 49, and vocabulary 6528 for Tier 1, but Table 1 lists 48,660 examples, median 29, max 47, and vocabulary 3,287 for ACD Train Tier 1; the text numbers actually match the CLD Train row. Please correct this discrepancy.","section":"Section 2.1, Table 1"},{"comment":"The right half of Table 6 appears to repeat the ACD rows after the first two rows; the expected CLD rows are missing or mislabeled. Please fix the table so that the ACD and CLD conditions are clearly separated.","section":"Table 6"},{"comment":"The description of the audio-pair sampling procedure says the randomly picked second audio excludes indices i through i+4, but the rationale for this exclusion is never explained; clarifying whether this avoids temporal proximity or some other bias would help reproducibility.","section":"Appendix E"},{"comment":"Equations (1)-(3) are mathematically clear, but the text would benefit from an explicit statement of the separator token's role and the cross-projection layer's learned parameters; currently the reader must infer these details from the figure and the appendix.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the empirical work is substantial, but the benchmark-validity concern is central and should be resolved before publication. The inconsistency between the text and Table 1 is a factual error that needs correction. The manuscript is already accepted at ICLR 2025; the revision should still be genuine and address the above points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this if you are working on audio-language benchmarks. The paper defines a new task – explaining the difference between two audio clips in natural language – and ships two datasets (ACD from AudioCaps, CLD from Clotho) with three explanation tiers (concise, brief, detailed). The data work is the real contribution: thousands of test pairs across tiers, with human verification on the test split. The model, ADIFF, is a modest architectural tweak on prefix tuning (a separator token plus a cross-projection transformer between the two audio prefixes) and it beats the naive baseline and a fine-tuned Qwen-Audio on objective metrics and in a human study on out-of-distribution audio. The ablations are honest: they show smaller LMs do fine under a fixed compute budget, and they document a language-only baseline that scores surprisingly high.\n\nThe soft spot is the ground truth. The explanations are generated by prompting an LLM with human-written captions of each audio, not by listening to the audio. For the benchmark to measure audio understanding, the caption contrast has to be a faithful proxy for the audible contrast. The paper does not show that. It says human annotators verified the test explanations, but there is no protocol, no annotator counts, no agreement statistics, and no indication that they listened to the audio rather than edited text for plausibility. The language-only baseline (random audio encoder) reaches SPIDEr 0.589 on CLD Tier 2, so a large fraction of the metric is linguistic. That does not invalidate the ranking of models, but it means the benchmark is partly about matching caption-derived language patterns, not necessarily acoustic events.\n\nMinor issues: the dataset statistics in Section 2.1 swap numbers that belong to different rows of Table 1; there are no error bars or significance tests anywhere; human evaluation uses five annotators with no agreement scores.\n\nIs it worth your time? If you work on audio-language models, yes – the task is well-defined and the datasets are reusable. The core weakness (reference construction) is an inherent property of the benchmark, not a fixable bug, and the paper would be stronger if it openly characterized the task as “explaining differences between audio captions” rather than between audio. On balance, the contribution is real and the paper deserves a serious referee; the issues are the kind that should be addressed in a revision, not grounds for rejection.","headline":"Useful new benchmark task with solid baseline, but the caption-derived references make the metric partly linguistic; treat it as measuring caption-contrast reasoning, not unmediated audio understanding.","tokens_in":36747,"tokens_out":4344,"would_cite":false,"duration_ms":39640,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ADIFF, a prefix-tuning model with a separator token, cross-projection layer, and three-stage training, produces audio difference explanations that outperform both the naive baseline and a much larger two-audio language model.","keywords":["audio difference explanation","audio-language model","prefix tuning","comparative reasoning","audio captioning","cross-projection","hallucination detection","three-tier explanation"],"falsifier":"Build a test set of audio pairs whose captions are identical but whose acoustics differ measurably (for example, the same dog bark with different pitch and reverberation); if ADIFF's explanations call the two clips the same while human listeners consistently report audible differences, the caption-proxy assumption fails and the model's apparent success would be an artifact of the benchmark construction.","tokens_in":35695,"feed_emoji":"🎧","tokens_out":7232,"duration_ms":63024,"temperature":0.7,"pith_summary":"This paper tries to establish audio difference explanation as a distinct task: given two recordings, a model should say in natural language how they differ, at three levels—concise event differences, brief scene-and-signal comparisons, and detailed explanations that include listener emotions. To make the task studyable, the paper builds two datasets, ACD and CLD, by prompting a large language model with human captions from AudioCaps and Clotho, then human-verifies the test set. The proposed model, ADIFF, keeps a frozen small language model and prompts it with audio embeddings from both clips, adding a separator token, a cross-projection layer, and a three-stage training schedule. The paper reports that ADIFF beats the naive prefix-tuning baseline and a much larger two-audio audio-language model on SPIDEr and on human ratings of correctness, granularity, and readability. If the task sticks, comparative audio reasoning becomes a measurable, trainable capability instead of a manual forensic skill.","feed_headline":"Small model explains audio differences, beats 7B rival","feed_subtitle":"ADIFF adds a separator token, cross-projection, and three-stage training to win on quality and detail.","key_machinery":"The load-bearing mechanism is the cross-projection layer with a separator token: each audio is encoded by HTSAT, projected into the language model's latent space, joined by a separator embedding derived from the decoder's end-of-text token, and run through a transformer that mixes the two audio streams with the text prompt. The paper's analysis of the cross-projection output shows the text-prefix tokens come to store the comparison attributes—frequency range, pitch, loudness—that later appear in the generated explanation. Two supporting mechanisms carry the rest: position captioning (training the model to caption 'the first audio' or 'the second audio') stops it from confusing similar-sounding inputs, and the three-stage training schedule keeps pretrained audio and text knowledge intact before final finetuning.","core_discovery":"The central claim is that explaining audio differences reduces to a conditional language generation problem, and that the right inductive biases let a 128-million-parameter decoder do it well: a separator token in latent space keeps the two clips distinct, a cross-projection transformer reuses the text prefix to store the acoustic attributes being compared (frequency, pitch, dynamic range), and three-stage training—unimodal pretraining, frozen multimodal grounding, then low-learning-rate finetuning—preserves audio knowledge while steering generation. On the new ACD and CLD benchmarks, ADIFF's SPIDEr scores exceed both the naive baseline and Qwen-Audio finetuned variants on nearly all tiers, and human evaluators rate its explanations higher on correctness, granularity, and readability, with the largest margin in granularity.","pith_inferences":["Because the ground-truth explanations are derived from caption pairs rather than from listening, the benchmark may reward caption-level linguistic contrast more than acoustic perception; a listening-only reference set would test whether ADIFF truly hears the differences.","The cross-projection finding—that text-prefix tokens end up storing comparison attributes—generalizes to other two-input generation tasks, such as comparing images, speech recordings, or medical scans, where the prompt region can act as a scratchpad for the contrast being described.","The hallucination-checking setup could be turned from a post-hoc debugging tool into a training objective that penalizes generated audio events whose predicted presence probability is near zero.","A direct stress test would pair clips that share identical captions but differ acoustically (same event, different pitch or reverberation); if the model reports 'no difference,' the caption-proxy assumption would be visible in practice."],"forward_implications":["Audio difference explanation gains public benchmarks (ACD and CLD) with three tiers, so future audio-language models can be scored on comparative reasoning rather than only on captioning or question answering.","The three-tier design separates failure modes: Tier 1 is the hardest because almost every word carries audio content, while Tier 2 is linguistically easiest and Tier 3 tests detail and emotional or contextual description.","Under limited compute and data, smaller language models can be grounded in audio more effectively than larger ones, and larger models need extra training epochs to catch up.","Position captioning reduces confusion between two similar audio inputs, making the model more reliable on perceptually similar sounds.","A frozen audio event detector provides a hallucination check: predicted event probabilities over time let users spot generated events that are not present in the audio."],"supporting_citations":[{"why":"Supplies the AudioCaps recordings and human captions from which the ACD dataset and its ground-truth difference explanations are built.","marker":"Kim et al. (2019)"},{"why":"Supplies the Clotho recordings and five-caption annotations used to build the CLD dataset.","marker":"Drossos et al. (2020)"},{"why":"HTSAT is the audio encoder that extracts the audio embeddings feeding the prefix for both clips.","marker":"Chen et al. (2022)"},{"why":"AudioSet provides the pretraining data for HTSAT and the 527 audio-event probabilities used for hallucination checking.","marker":"Gemmeke et al. (2017)"},{"why":"Establishes the prefix-tuning architecture and audio projection design that ADIFF extends.","marker":"Deshmukh et al. (2023a)"},{"why":"Qwen-Audio is the comparison audio-language model that supports two audio inputs; its zero-shot, LoRA, and full-finetune versions are the baselines beaten.","marker":"Chu et al. (2023)"},{"why":"The LLM prompting setup for generating difference explanations from captions is adapted from this work.","marker":"Deshmukh et al. (2024c)"},{"why":"CLIP prefix architecture motivates the projection and prefix-tuning design for prompting a frozen language model.","marker":"Mokady et al. (2021)"},{"why":"SPIDEr, the primary objective metric combining SPICE and CIDEr, is defined here.","marker":"Liu et al. (2017)"}],"fun_headline_variants":["AI that explains audio differences in plain language","New model narrates what changes between two sounds","128M model outshines 7B rival on audio-difference explanations","First benchmarks for audio difference explanations, plus a strong model","Small model beats 7B on explaining how sounds differ"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is written from human captions of the two clips, not by listening to the clips, so the task assumes caption-level differences faithfully stand in for acoustic differences.","fun_headline_variants_meta":{"raw":{"variants":["AI that explains audio differences in plain language","New model narrates what changes between two sounds","128M model outshines 7B rival on audio-difference explanations","First benchmarks for audio difference explanations, plus a strong model","Small model beats 7B on explaining how sounds differ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1414,"prompt_tokens":1008,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":624,"tokens_out":406,"duration_ms":4334,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:35:16.800584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of audio pairs whose captions are identical but whose acoustics differ measurably (for example, the same dog bark with different pitch and reverberation); if ADIFF's explanations call the two clips the same while human listeners consistently report audible differences, the caption-proxy assumption fails and the model's apparent success would be an artifact of the benchmark construction.","supporting_citations":[{"cited_title":"AudioCaps: Generating Captions for Audios in The Wild","cited_arxiv_id":null,"evidence_quote":"Supplies the AudioCaps recordings and human captions from which the ACD dataset and its ground-truth difference explanations are built."},{"cited_title":"Clotho: an audio captioning dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the Clotho recordings and five-caption annotations used to build the CLD dataset."},{"cited_title":"Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection","cited_arxiv_id":null,"evidence_quote":"HTSAT is the audio encoder that extracts the audio embeddings feeding the prefix for both clips."},{"cited_title":"Improved image captioning via policy gradient optimization of spider","cited_arxiv_id":null,"evidence_quote":"SPIDEr, the primary objective metric combining SPICE and CIDEr, is defined here."}],"review_version":1}