Pith. sign in

REVIEW 5 major objections 4 minor 70 references

Audio-aware LLM feedback improves text-to-audio instruction following

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:15 UTC pith:MV2YMORP

load-bearing objection A useful new loop for fine-grained TTA instruction following, but the same-family judge and small human check mean the headline gains should be read with caution. the 5 major comments →

arxiv 2607.13408 v2 pith:MV2YMORP submitted 2026-07-15 eess.AS cs.AIcs.CLcs.LGcs.SD

Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models

Chun-Yi Kuan , Siwon Kim , Byeonggeun Kim , Suyoun Kim , Bo-Ru Lu , Qingming Tang , Ankur Gandhe , Hung-yi Lee
show 2 more authors
Chieh-Chi Kao Chao Wang
This is my paper
classification eess.AS cs.AIcs.CLcs.LGcs.SD
keywords text-to-audio generationaudio-aware large language modelsinstruction followingtemporal orderingpreference optimizationdirect preference optimizationsound event existenceS3Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that text-to-audio models fail at multi-event, temporally ordered instructions because they lack instruction-level training signals, and that audio-aware large language models (ALLMs) can supply such signals at scale. The authors show that ALLM judges can reliably verify whether each required sound event is present and whether events occur in the specified order, matching human judgments on a 150-clip study. They then convert these judgments into preference pairs—generations that satisfy all events and order are preferred, those that miss events or break order are rejected—and train a TangoFlux base model with direct preference optimization. Across AudioCaps, synthetic multi-event sets, and a new narrative benchmark called S3Bench, the method improves event completeness, temporal ordering, and joint instruction-following accuracy while keeping audio quality competitive. If correct, this makes scalable, instruction-level supervision available for text-to-audio generation without costly human annotation.

Core claim

The central claim is that ALLM-based fine-grained judgment is a viable and scalable training signal for text-to-audio instruction following. The paper introduces ALLM-Judged Preference Optimization (AJPO), which uses an audio-aware LLM to score candidate generations on two axes: the proportion of target sound events present (event existence score) and the Kendall's tau between predicted and required event ordering (temporal order score). Generations that achieve perfect scores on both axes become preferred samples; those that miss events or violate order become rejected samples, and the pairs are used for direct preference optimization. The authors validate the judge against AudioCaps, CompA

What carries the argument

The central mechanism is the ALLM judge that decomposes instruction following into two structured scores: event existence (average binary presence judgments per target event) and temporal order (Kendall's tau between the judged event ranking and the instruction's implied ordering). These scores are aggregated into preference pairs—anchor samples must score 1.0 on both, rejections either have incomplete event coverage or a misordered sequence—and fed into direct preference optimization. To avoid the referee-as-player problem, the paper uses one ALLM for evaluation and a smaller model of the same family as the reward model for training. S3Bench, the new evaluation set, is built by sampling two

Load-bearing premise

The paper's central claim depends on the audio-aware LLM judge actually measuring whether the generated audio follows the instruction, rather than just picking up patterns in the judge's own scoring style; if humans would disagree with the judge on the test set, the reported gains may not be real improvements in instruction following.

What would settle it

A concrete falsifying experiment: train the same TTA base model with AJPO using the same-family 3B reward model, but evaluate on a held-out set scored by a different model family (e.g., Audio Flamingo 3) or by a 500+ sample human annotation with confidence intervals. If the joint accuracy gains disappear or reverse under that independent evaluation, the claim that AJPO improves genuine instruction following rather than merely optimizing the judge's preferences would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If AJPO works as claimed, text-to-audio models can be improved for multi-event and temporal-order instructions without human preference annotation, since ALLM judgments serve as scalable verifiers.
  • The decomposition into event existence and temporal ordering provides a diagnostic evaluation protocol that could replace coarse global similarity metrics like CLAP for instruction-following tasks.
  • The gains on S3Bench, a narrative benchmark with overlapping events and partial orders, indicate the method generalizes beyond template-based instructions to more realistic sound scene descriptions.
  • Online DPO results suggest that dynamically reconstructing preference pairs as the model improves yields sustained gains, whereas static preference datasets degrade after early iterations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The core idea may transfer to other generative domains with temporal or compositional structure—such as text-to-speech prosody, music generation, or video generation—where the same ALLM judge could verify event-level conditions.
  • Because the reward model and evaluation judge share the same model family, the reported gains could partly reflect reward hacking toward that family's biases; a stronger test would use an independent judge or a larger human-annotated test set with confidence intervals.
  • S3Bench, currently generated by prompting an LLM, could be extended to include real-world recordings with annotated event orderings, making the benchmark less synthetic and more ecologically valid.
  • The proportion-based event existence score and the Kendall's tau ordering score are fundamentally rank/aggregation objectives; combining them into a single scalar reward could be explored alongside multi-objective DPO variants.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ALLM-Judged Preference Optimization (AJPO), a training framework for text-to-audio (TTA) generation that uses audio-aware large language models (ALLMs) as fine-grained judges to verify sound-event existence and temporal ordering, and converts these judgments into DPO preference pairs. The authors validate the judge on audio-understanding benchmarks and a 150-clip human study, introduce a new narrative multi-event benchmark (S3Bench), and report gains over TangoFlux-base, CLAP-DPO, static preference datasets, and ALLM-SFT across AudioCaps, MultiEvent-Temporal-2/3/4, and S3Bench, while maintaining audio quality.

Significance. If the central claim holds, AJPO is a meaningful contribution: it demonstrates that ALLM-based instruction-level feedback can serve as a scalable training signal for multi-event and temporally ordered TTA generation, going beyond global CLAP-style similarity. The paper is clearly written, the ablations are well designed (CLAP-DPO, ALLM-CAP, ALLM-SFT, online DPO), and the validation of ALLMs as judges on labeled benchmarks is a strength. The main weakness is that the primary evaluation of the optimized TTA models is performed by an ALLM from the same model family as the reward model, and the human evaluation does not directly verify the central claim on the final model's outputs. The S3Bench contribution also lacks a validity check of the generated narratives.

major comments (5)
  1. [§5.1, Tables 3–6] The main instruction-following metrics are produced by Qwen2.5-Omni-7B, while the DPO preference signal in §3.4 uses Qwen2.5-Omni-3B. The paper states the 3B model is used 'to avoid the referee-as-player issue,' but using a different size of the same model family does not establish independence: the two models share pretraining and instruction-tuning data and are likely to share systematic judgment biases. The human verification in Table 2 covers 150 clips from 'multiple representative TTA systems,' not from the optimized AJPO model, so it cannot rule out overoptimization to the judge. Please add an independent evaluation of the final model outputs—for example, human event-level and temporal-order annotation on a random sample, or scoring with a different ALLM family such as Audio Flamingo 3—and report agreement between the 7B judge and the independent judge on that output distribution.
  2. [§5.4, Table 5] The human study only rates text relevance (REL) and overall quality (OVL) on 50 clips, with no confidence intervals or significance testing. It does not measure sound-event completeness or temporal ordering for the optimized model, so the paper's central claims about these dimensions are not independently verified on the actual system. Please add a human instruction-following evaluation (event presence and temporal order) on a larger sample of AJPO-generated audio, with inter-annotator agreement and confidence intervals.
  3. [Tables 3–6] All instruction-following results are point estimates without variance, confidence intervals, or statistical tests. Since the evaluation is fully automated, bootstrapping over generated clips or running multiple seeds is feasible. Without this, it is unclear whether differences such as the AudioCaps event-existence EM tie (87.4 vs. 87.4 in Table 3) or the S3Bench Joint gain (49.9 vs. 45.4 in Table 4) are reliable. Please report bootstrap CIs or error bars for the main metrics.
  4. [§4.2] Temporal-order metrics are computed only for samples whose event-existence exact match is 1.0. This conditional subset differs across models and is typically larger for the proposed method, so cross-model temporal EM comparisons may be biased: a model with lower event completeness is evaluated on an easier, smaller subset. Please report the size of the conditional subset for each model or make Joint Accuracy the primary comparison metric.
  5. [§3.5] S3Bench is constructed by prompting an LLM to generate narrative instructions from sampled event labels, but the manuscript gives no human validation that the narratives unambiguously encode the intended event set and temporal order. Since S3Bench is used to support the generalization claim, please add a small human-annotation study of benchmark validity (e.g., whether annotators recover the intended event order from the text) or release the benchmark for inspection.
minor comments (4)
  1. [Table 5] The ALLM-SFT row reports FAD=32.43, which is an order of magnitude larger than all other FAD values (2–4). This appears to be a typo; please verify.
  2. [§3.3–3.4] The temporal-order score uses Kendall's tau, but the paper mentions partial-order constraints without defining how tau is computed for a partial order π. Please specify the handling of ties and partial orders.
  3. [§3.5] The composition of S3Bench is ambiguous: '300 with two events, 500 with three events, and 200 with four events. Additionally, 200 instances explicitly involve overlapping events.' Please clarify whether the 200 overlapping instances are included in the 1,000 or additional; the text can be read both ways.
  4. [§5.3.1] Qwen3-Embedding-8B is mentioned as the text similarity model for ALLM-CAP, but no citation or reference is provided.

Circularity Check

0 steps flagged

No circularity found: ALLM judge validation, DPO training, and evaluation form an independent chain with external human grounding.

full rationale

The paper's derivation chain is not circular under the stated criteria. The ALLM judges are first validated against ground-truth audio-understanding benchmarks (Table 1: AudioCaps, CompA, AudioTime, MultiEvent-Temporal) and against human annotations on 150 TTA-generated clips (Table 2, agreement 89.8% existence, 93.5% temporal). These are independent external checks, not definitions of the target result. The preference construction uses the validated Qwen2.5-Omni-3B judgments (Sec. 3.4), while the main evaluation uses Qwen2.5-Omni-7B (Sec. 5.1), and the paper explicitly notes it chose the 3B reward model 'to avoid the referee-as-player issue.' This does not make the evaluation equivalent to the training signal by construction: the trained model's generations are stochastic outputs, not algebraically derived from the judge, and the reported gains are measured rather than entailed. The final human evaluation (Table 5, Sec. 5.4) provides independent evidence for the central claim, albeit on 50 samples; small human sample size is a statistical-power concern, not circularity. The paper's self-citations appear only as related work or motivation (e.g., references [27], [28], [39], [45]) and are not load-bearing for the core result; no uniqueness theorem or ansatz is imported from the authors' prior work. The S3Bench construction uses LLM-generated narrative instructions from event labels, but it is used solely for evaluation and is not a renamed version of the training signal. Therefore no circular step satisfying the hard-evidence requirement can be identified.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests mainly on domain assumptions about ALLM judge reliability and transfer, not on fitted constants. The anchor threshold and training hyperparameters are design choices, and the largest circularity burden is the reliance on same-family ALLMs for both reward and evaluation.

free parameters (3)
  • Anchor criterion (sexist=1.0 and sorder=1.0) = 1.0 and 1.0
    Design threshold for preferred samples in DPO construction; changing it changes preference pairs and likely results. It is chosen by hand, not fitted to data.
  • DPO training hyperparameters = lr=1e-4, batch=128, 500 warmup steps
    Chosen by hand; standard values; central results depend on them but they are not fitted to evaluation benchmarks.
  • Number of candidate generations K
    The paper samples K candidates per instruction in §3.4 but never states K, an unstated design choice affecting preference quality.
axioms (5)
  • domain assumption ALLM judgments generalize from benchmark audio to TTA-generated audio
    The method assumes the verified judging capability in Table 1 transfers to noisy, synthetic TTA outputs; validated only on 150 human-rated clips in §4.5.
  • domain assumption Qwen2.5-Omni-3B and 7B share enough competence and bias structure to serve as train/eval judges
    The 3B reward / 7B evaluator split is meant to avoid referee-as-player, but both are the same model family; the independence of the evaluation signal is not established in §5.1.
  • standard math Direct preference optimization improves instruction-following given preference pairs
    DPO is an established training objective; the paper applies it rather than derives it in §3.4.
  • domain assumption Temporal metrics are only meaningful when all events are present
    The evaluation condition in §4.2 computes temporal metrics only on samples with existence exact-match 1.0, which conditions and changes the comparison population.
  • ad hoc to paper LLM-generated S3Bench narratives represent naturalistic multi-event instructions
    S3Bench is generated by prompting an LLM with ESC-50/AudioCaps labels and in-context examples in §3.5; no external validation that the narratives are natural or unambiguous.

pith-pipeline@v1.3.0-alltime-deepseek · 17720 in / 13380 out tokens · 126580 ms · 2026-08-02T05:15:10.682650+00:00 · methodology

0 comments
read the original abstract

Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order. This gap arises because existing evaluation and training signals mainly emphasize global similarity or perceptual quality, with limited supervision on instruction-level correctness. We propose an instruction-level framework that uses audio-aware large language models (ALLMs) as fine-grained judges to verify target event presence and temporal relations in generated audio. After validating ALLM judgments on benchmarks and through human verification, we use their feedback to construct preference pairs for direct preference optimization. We further introduce S3Bench, a narrative benchmark for evaluating multi-event temporal instruction following. Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.

Figures

Figures reproduced from arXiv: 2607.13408 by Ankur Gandhe, Bo-Ru Lu, Byeonggeun Kim, Chao Wang, Chieh-Chi Kao, Chun-Yi Kuan, Hung-yi Lee, Qingming Tang, Siwon Kim, Suyoun Kim.

Figure 1
Figure 1. Figure 1: Overview of the proposed ALLM-Judged Preference Optimization framework. Given a text instruction, a TTA model generates multiple candidate audio clips, which are then evaluated by an audio-aware large language model for sound event existence and temporal ordering. The resulting fine-grained judgments are converted into preference pairs for direct preference optimization. costly human feedback collection. I… view at source ↗
Figure 2
Figure 2. Figure 2: Online DPO progressively improves event existence, temporal reasoning, and overall joint accuracy. 5.3. Ablation Study For clarity, we denote different preference construction strate￾gies as follows. Using ALLM-based fine-grained feedback with direct preference optimization is denoted as ALLM-DPO. Re￾placing the feedback with CLAP-based similarity ranking is de￾noted as CLAP-DPO. Using ALLM-based feedback … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 19 linked inside Pith

  1. [1]

    It starts with gentle rainfall, then someone walks through the scene. After that, a wooden door slowly opens, and the clip finishes with cars passing by in the distance

    Introduction Recent progress in text-to-audio (TTA) generation [1–18] has led to substantial improvements in perceptual audio quality, enabling models to produce increasingly realistic and natural- sounding audio from textual descriptions. Modern TTA sys- tems, based on diffusion or flow-matching frameworks, achieve strong performance on widely used evalu...

    Pith/arXiv arXiv 2026

  2. [2]

    Related Work 2.1. Preference Training for Text-to-Audio Generation Preference-based training has recently been adopted to im- prove text-to-audio (TTA) generation by learning from relative comparisons rather than absolute scores [11, 12, 26]. Exist- ing methods mainly construct preference data using similarity- based ranking, heuristic pipelines, or human...

  3. [3]

    Rain is falling con- tinuously

    Method 3.1. Overview: ALLM-Judged Preference Optimization We proposeALLM-JudgedPreferenceOptimization (AJPO), a training framework for improving instruction following in text-to-audio (TTA) generation. The key idea is to leverage audio-aware large language models (ALLMs) as judges to pro- vide fine-grained, instruction-level evaluation of generated au- di...

  4. [4]

    It starts with the sound ofe 1, shifts toe 2, and ends withe 3

    Experimental Setup 4.1. Evaluation Benchmarks We use different benchmarks to support two evaluation goals: (1) verifying whether ALLMs are sufficiently capable to serve as instruction-level judges, and (2) evaluating the instruction- following performance of TTA systems under varying levels of complexity. Verifying ALLMs as Judges.To assess ALLMs, we eval...

  5. [5]

    Verifying ALLMs as Judges Table 1 reports ALLM performance on audio understanding benchmarks with ground-truth annotations

    Results and Analysis 5.1. Verifying ALLMs as Judges Table 1 reports ALLM performance on audio understanding benchmarks with ground-truth annotations. Recent ALLMs achieve strong performance in both sound event existence and temporal ordering, supporting their suitability as instruction- level evaluators. Qwen2.5-Omni-7B performs best overall, fol- lowed b...

  6. [6]

    While recent systems achieve strong percep- tual realism, our study shows that they often overlook fine- grained requirements such as sound event completeness and temporal ordering

    Conclusion We revisit TTA generation from the perspective of instruction- level correctness. While recent systems achieve strong percep- tual realism, our study shows that they often overlook fine- grained requirements such as sound event completeness and temporal ordering. By leveraging audio-aware LLMs as struc- tured judges, we demonstrate that instruc...

  7. [7]

    Disclosure of Generative AI Use The authors used Claude and ChatGPT to assist with grammar checking, language polishing, and improving the readability of the manuscript

  8. [8]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossezet al., “Audiogen: Textually guided audio generation,”arXiv preprint arXiv:2209.15352, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generating music from text,”arXiv preprint arXiv:2301.11325, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    AudioLDM: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-audio generation with latent diffusion models,”Proceedings of the International Confer- ence on Machine Learning, pp. 21 450–21 474, 2023

    2023

  11. [11]

    Text-to- audio generation using instruction tuned llm and latent diffusion model,

    D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to- audio generation using instruction tuned llm and latent diffusion model,”arXiv preprint arXiv:2304.13731, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Diffsound: Discrete diffusion model for text-to-sound genera- tion,

    D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y . Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound genera- tion,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1720–1733, 2023

    2023

  13. [13]

    Towards general-purpose text-instruction-guided voice conversion,

    C.-Y . Kuan, C.-A. Li, T.-Y . Hsu, T.-Y . Lin, H.-L. Chung, K.-W. Chang, S.-Y . Chang, and H.-y. Lee, “Towards general-purpose text-instruction-guided voice conversion,” in2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  14. [14]

    Simple and controllable music generation,

    J. Copetet al., “Simple and controllable music generation,”Ad- vances in neural information processing systems, vol. 36, pp. 47 704–47 720, 2023

    2023

  15. [15]

    Audiolm: a language modeling approach to audio gener- ation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio gener- ation,”IEEE/ACM transactions on audio, speech, and language processing, vol. 31, pp. 2523–2533, 2023

    2023

  16. [16]

    Soundstorm: Efficient parallel audio gen- eration,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio gen- eration,”arXiv preprint arXiv:2305.09636, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation,

    J. Huang, Y . Ren, R. Huang, D. Yang, Z. Ye, C. Zhang, J. Liu, X. Yin, Z. Ma, and Z. Zhao, “Make-an-audio 2: Temporal-enhanced text-to-audio generation,”arXiv preprint arXiv:2305.18474, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,

    N. Majumder, C.-Y . Hung, D. Ghosal, W.-N. Hsu, R. Mihalcea, and S. Poria, “Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,” 2024

    2024

  19. [19]

    Tangoflux: Super fast and faithful text to audio generation with flow matching and clap- ranked preference optimization,

    C.-Y . Hung, N. Majumder, Z. Kong, A. Mehrish, A. Zadeh, C. Li, R. Valle, B. Catanzaro, and S. Poria, “Tangoflux: Super fast and faithful text to audio generation with flow matching and clap- ranked preference optimization,” 2024

    2024

  20. [20]

    Ezaudio: Enhancing text-to-audio generation with efficient dif- fusion transformer,

    J. Hai, Y . Xu, H. Zhang, C. Li, H. Wang, M. Elhilali, and D. Yu, “Ezaudio: Enhancing text-to-audio generation with efficient dif- fusion transformer,”arXiv preprint arXiv:2409.10819, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Audioldm 2: Learn- ing holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learn- ing holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 32, pp. 2871–2883, 2024

    2024

  22. [22]

    Stable audio open,

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  23. [23]

    Etta: Elucidating the design space of text-to-audio models,

    S.-g. Lee, Z. Kong, A. Goel, S. Kim, R. Valle, and B. Catan- zaro, “Etta: Elucidating the design space of text-to-audio models,” inForty-second International Conference on Machine Learning, 2025

    2025

  24. [24]

    Impact: Iter- ative mask-based parallel decoding for text-to-audio generation with diffusion modeling,

    K.-P. Huang, S.-w. Yang, H. PHAN, B.-R. Lu, B. Kim, S. Macha, Q. Tang, S. Ghosh, H.-y. Lee, C.-C. Kaoet al., “Impact: Iter- ative mask-based parallel decoding for text-to-audio generation with diffusion modeling,” inForty-second International Confer- ence on Machine Learning, 2025

    2025

  25. [25]

    Gen- erative audio language modeling with continuous-valued tokens and masked next-token prediction,

    S.-w. Yang, B. Kim, K.-P. Huang, Q. Tang, H. PHAN, B.-R. Lu, H. Sundar, S. Ghosh, H.-y. Lee, C.-C. Kaoet al., “Gen- erative audio language modeling with continuous-valued tokens and masked next-token prediction,” inForty-second International Conference on Machine Learning, 2025

    2025

  26. [26]

    Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,” inProceedings of Interspeech 2019, 2019, pp. 2350–2354

    2019

  27. [27]

    Improved techniques for training gans,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,”Advances in neural information processing systems, vol. 29, 2016

    2016

  28. [28]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  29. [29]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  30. [30]

    Ritta: Model- ing event relations in text-to-audio generation,

    Y . He, Y . Jain, X. Liu, A. Markham, and V . Vineet, “Ritta: Model- ing event relations in text-to-audio generation,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 3497–3511

    2025

  31. [31]

    Aurelius: Relation aware text-to-audio generation at scale,

    Y . He, H. Liang, Y . Jain, A. Markham, and V . Vineet, “Aurelius: Relation aware text-to-audio generation at scale,” inThe Four- teenth International Conference on Learning Representations, 2026

    2026

  32. [32]

    Compa: Address- ing the gap in compositional reasoning in audio-language mod- els,

    S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. R. Evuru, S. Sakshi, O. Nieto, R. Duraiswami, D. Manochaet al., “Compa: Address- ing the gap in compositional reasoning in audio-language mod- els,” inThe Twelfth International Conference on Learning Repre- sentations, 2024

    2024

  33. [33]

    T2a-feedback: Improving basic capabilities of text-to-audio generation via fine-grained ai feed- back,

    Z. Wang, K. Lei, C. Zhu, J. Huang, S. Zhou, L. Liu, X. Cheng, S. Ji, Z. Ye, T. Jin, and Z. Zhao, “T2a-feedback: Improving basic capabilities of text-to-audio generation via fine-grained ai feed- back,” inProceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (ACL), 2025

    2025

  34. [34]

    Aqascore: Evaluating semantic alignment in text-to-audio generation via audio question answering,

    C.-Y . Kuan, K.-W. Chang, and H.-y. Lee, “Aqascore: Evaluating semantic alignment in text-to-audio generation via audio question answering,”arXiv preprint arXiv:2601.14728, 2026

    arXiv 2026

  35. [35]

    The sound of absence: Audio- language embedding models struggle with negation,

    C.-Y . Kuan and H.-y. Lee, “The sound of absence: Audio- language embedding models struggle with negation,”arXiv preprint arXiv:2607.12290, 2026

    Pith/arXiv arXiv 2026

  36. [36]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Joint audio and speech understanding,

    Y . Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  38. [38]

    Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,

    C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y . Liu, C. Zong, and J. Zhang, “Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,”arXiv preprint arXiv:2309.00916, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Audiochatllama: Towards general-purpose speech abilities for llms,

    Y . Fathullahet al., “Audiochatllama: Towards general-purpose speech abilities for llms,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 5522–5532

    2024

  40. [40]

    Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,

    C.-Y . Kuan, C.-K. Yang, W.-P. Huang, K.-H. Lu, and H.-y. Lee, “Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,” in2024 IEEE Spoken Language Technology Work- shop (SLT). IEEE, 2024, pp. 1060–1067

    2024

  41. [41]

    Speechprompt: Prompting speech language models for speech processing tasks,

    K.-W. Chang, H. Wu, Y .-K. Wang, Y .-K. Wu, H. Shen, W.- C. Tseng, I.-t. Kang, S.-W. Li, and H.-y. Lee, “Speechprompt: Prompting speech language models for speech processing tasks,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024

    2024

  42. [42]

    Blsp- emo: Towards empathetic large speech-language models,

    C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang, “Blsp- emo: Towards empathetic large speech-language models,” inPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 19 186–19 199

    2024

  43. [43]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,

    S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S.-g. Lee, C.- H. H. Yang, R. Duraiswami, D. Manocha, R. Valleet al., “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  44. [44]

    V oxtral,

    A. H. Liu, A. Ehrenberg, A. Lo, C. Denoix, C. Barreau, G. Lam- ple, J.-M. Delignon, K. R. Chandu, P. von Platen, P. R. Mud- direddyet al., “V oxtral,”arXiv preprint arXiv:2507.13264, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Qwen2.5-omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Danget al., “Qwen2.5-omni technical report,”arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Teaching audio-aware large language models what does not hear: Mitigating hallucinations through synthesized negative samples,

    C.-Y . Kuan and H.-y. Lee, “Teaching audio-aware large language models what does not hear: Mitigating hallucinations through synthesized negative samples,” inProc. Interspeech 2025, 2025, pp. 2073–2077

    2025

  47. [47]

    From alignment to advancement: Bootstrapping audio- language alignment with synthetic data,

    ——, “From alignment to advancement: Bootstrapping audio- language alignment with synthetic data,”IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 4604–4619, 2025

    2025

  48. [48]

    Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras,

    A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V . Chaudhary, C. Chenet al., “Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras,”arXiv preprint arXiv:2503.01743, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil- ities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil- ities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    On the landscape of spoken language models: A comprehensive survey,

    S. Arora, K.-W. Chang, C.-M. Chien, Y . Peng, H. Wu, Y . Adi, E. Dupoux, H.-y. Lee, K. Livescu, and S. Watanabe, “On the landscape of spoken language models: A comprehensive survey,” Transactions on Machine Learning Research, 2025

    2025

  51. [51]

    Audio flamingo 2: An audio- language model with long-audio understanding and expert reason- ing abilities,

    S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio flamingo 2: An audio- language model with long-audio understanding and expert reason- ing abilities,” inForty-second International Conference on Ma- chine Learning, 2025

    2025

  52. [52]

    Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,

    C.-Y . Kuan, W.-P. Huang, and H.-y. Lee, “Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,” inProc. Interspeech, 2024, pp. 4144–4148

    2024

  53. [53]

    Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,

    C.-Y . Kuan and H.-y. Lee, “Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  54. [54]

    Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,

    C.-y. Huanget al., “Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 136–12 140

    2024

  55. [55]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    ——, “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” inThe Thirteenth International Confer- ence on Learning Representations, 2025

    2025

  56. [56]

    Mmau: A mas- sive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Ni- eto, R. Duraiswami, S. Ghosh, and D. Manocha, “Mmau: A mas- sive multi-task audio understanding and reasoning benchmark,” inThe Thirteenth International Conference on Learning Repre- sentations, 2025

    2025

  57. [57]

    Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence,

    S. Kumar, ˇS. Sedl ´aˇcek, V . Lokegaonkar, F. L ´opez, W. Yu, N. Anand, H. Ryu, L. Chen, M. Pli ˇcka, M. Hlav ´aˇceket al., “Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence,”arXiv preprint arXiv:2508.13992, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix,

    Z. Ma, Y . Ma, Y . Zhu, C. Yang, Y .-W. Chao, R. Xuet al., “Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix,”arXiv preprint arXiv:2505.13032, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Game-time: Evaluating temporal dynamics in spoken language models,

    K.-W. Chang, E.-P. Hu, C.-Y . Kuan, W. Ren, W.-C. Chen, G.-T. Lin, Y . Tsao, S.-H. Sun, H.-y. Lee, and J. Glass, “Game-time: Evaluating temporal dynamics in spoken language models,” in ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 1–5

    2026

  60. [60]

    Aqua-bench: Beyond finding answers to knowing when there are none in audio question answering,

    C.-Y . Kuan and H.-y. Lee, “Aqua-bench: Beyond finding answers to knowing when there are none in audio question answering,” in ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 1–5

    2026

  61. [61]

    Baton: aligning text-to- audio model using human preference feedback,

    H. Liao, H. Han, K. Yang, T. Duet al., “Baton: aligning text-to- audio model using human preference feedback,” inProceedings of the Thirty-Third International Joint Conference on Artificial Intel- ligence, 2024, pp. 4542–4550

    2024

  62. [62]

    Walking through un- certainty: An empirical study of uncertainty estimation for audio- aware large language models,

    C.-Y . Kuan, W.-P. Huang, and H.-y. Lee, “Walking through un- certainty: An empirical study of uncertainty estimation for audio- aware large language models,”arXiv preprint arXiv:2604.25591, 2026

    Pith/arXiv arXiv 2026

  63. [63]

    Audio large language models can be descrip- tive speech quality evaluators,

    C. Chen, Y . Hu, S. Wang, H. Wang, Z. Chen, C. Zhang, C.-H. H. Yang, and E. Chng, “Audio large language models can be descrip- tive speech quality evaluators,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  64. [64]

    Audiojudge: Un- derstanding what works in large audio model based speech evalu- ation,

    P. Manakul, W. H. Gan, M. J. Ryan, A. S. Khan, W. Sirichotedum- rong, K. Pipatanakul, W. Held, and D. Yang, “Audiojudge: Un- derstanding what works in large audio model based speech evalu- ation,”arXiv preprint arXiv:2507.12705, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Audio-aware large language models as judges for speaking styles,

    C.-H. Chiang, X. Wang, C.-C. Lin, K. Lin, L. Li, R. Kopetz, Y . Qian, Z. Wang, Z. Yang, H.-y. Lee, and L. Wang, “Audio-aware large language models as judges for speaking styles,” inFindings of the Association for Computational Linguistics: EMNLP 2025, 2025, pp. 467–480

    2025

  66. [66]

    Instructttseval: Benchmarking complex natural-language instruction following in text-to-speech systems,

    K. Huang, Q. Tu, L. Fan, C. Yang, D. Zhang, S. Li, Z. Fei, Q. Cheng, and X. Qiu, “Instructttseval: Benchmarking complex natural-language instruction following in text-to-speech systems,” arXiv preprint arXiv:2506.16381, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    Audioeval: Automatic dual-perspective and multi-dimensional evaluation of text-to-audio-generation,

    H. Wang, J. Zhao, C. Liu, Y . Jia, H. Sun, J. Zhou, and Y . Qin, “Audioeval: Automatic dual-perspective and multi-dimensional evaluation of text-to-audio-generation,”arXiv preprint arXiv:2510.14570, 2025

    arXiv 2025

  68. [68]

    Audiocaps: Generat- ing captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generat- ing captions for audios in the wild,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), 2019, pp. 119–132

    2019

  69. [69]

    Audiotime: A temporally- aligned audio-text benchmark dataset,

    Z. Xie, X. Xu, Z. Wu, and M. Wu, “Audiotime: A temporally- aligned audio-text benchmark dataset,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  70. [70]

    ESC: Dataset for Environmental Sound Classifi- cation,

    K. J. Piczak, “ESC: Dataset for Environmental Sound Classifi- cation,” inProceedings of the 23rd Annual ACM Conference on Multimedia. ACM Press, 2015, pp. 1015–1018

    2015