Pith. sign in

REVIEW 2 major objections 5 minor 70 references

Amplifying a few audio-encoder neurons, scored by real-versus-noise activation, lifts non-semantic speech accuracy by up to 25.7 points without training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 03:05 UTC pith:UUG6LANR

load-bearing objection Sparse, label-free real-vs-noise scoring of audio-encoder FFN neurons, then amplifying ~0.12% of them, yields large Acc↑/ALA↓ gains on VoxParadox; locus and selectivity ablations cleanly support the claim. the 2 major comments →

arxiv 2607.11801 v1 pith:UUG6LANR submitted 2026-07-13 cs.SD cs.AI

Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models

Yu-Han Huang , Chih-Kai Yang , Ke-Han Lu , An-Yu Cheng , Hung-yi Lee This is my paper
classification cs.SD cs.AI
keywords large audio-language modelsaudio encoderneuron amplificationinference-time interventionparalinguistic understandingacoustic scoreVoxParadox
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large audio-language models understand the words in speech far better than fine acoustic details such as emotion, gender, pitch, or how many people are talking. Retraining is expensive, and most existing inference-time fixes act only after the audio encoder, where much of that acoustic signal may already be diluted. This paper claims that the right place and right scale of intervention is inside the encoder itself, at individual feed-forward neurons. The method, IAAN, runs the encoder once on the real clip and once on matched noise, ranks every neuron by how much more it fires on the real signal, and then multiplies the top few dozen or hundred activations by a fixed gain. On three open models the same sparse intervention raises average accuracy across ten non-semantic attributes by 9.7 to 25.7 points, also helps a model already fine-tuned for acoustic grounding, and fails when the same idea is applied after the encoder or without the acoustic ranking. The result matters because it shows that acoustic perception can be strengthened at inference with almost no compute and no labels, simply by turning up the right neurons where the waveform is first turned into features.

Core claim

A training-free, label-free procedure that ranks feed-forward neurons inside the audio encoder by the difference of their mean activations on a real waveform versus a noise (or silence) reference, then amplifies only the highest-scoring subset at inference, produces large accuracy gains on ten non-semantic speech attributes; both the encoder locus and the neuron-level selectivity are necessary for those gains.

What carries the argument

IAAN (Identifying and Amplifying Acoustic Neurons): the acoustic score si equals mean activation on the real clip minus mean activation on a matched noise reference; the top-K neurons are then scaled by gain g > 1 during the real encoder pass.

Load-bearing premise

The difference between a neuron’s response to real audio and to matched noise is assumed to isolate the neurons that carry the fine acoustic attributes the downstream tasks need, rather than generic energy or other non-task factors.

What would settle it

Replace the acoustic-score ranking with random neurons of the same budget (or amplify entire late encoder layers without selection) and measure whether the accuracy and adversarial-label drops on VoxParadox disappear; if the gains survive, the claim that the score identifies the neurons that matter is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes IAAN, a training-free, label-free inference-time intervention for large audio-language models. It scores each feed-forward neuron in the audio encoder by the difference of mean activations on the real waveform versus a matched Gaussian-noise (or silence) reference (Eq. 1), then amplifies the top-K highest-scoring neurons by a gain g during the real-audio encoder pass. On VoxParadox (ten non-semantic speech attributes), this raises average accuracy by 25.7 / 21.4 / 9.7 points on Audio-Flamingo-3, Qwen2.5-Omni and Kimi-Audio while lowering adversarial-label agreement, and still improves a model already fine-tuned for acoustic grounding (AF3-PD). Controlled comparisons (AAD, LLM-amplify, dense HS-steer, random neurons, whole-layer amplify, layer-restricted selection, reference SNR sweep) are used to argue that both the encoder locus and neuron-level selectivity are necessary for the gain.

Significance. If the result holds, the work identifies a previously underexplored site of intervention—individual feed-forward neurons inside the audio encoder—and shows that a sparse, label-free, input-specific amplification can substantially improve fine-grained acoustic perception without retraining. The gains are large, consistent across three open LALMs plus a specialized fine-tuned model, and accompanied by a joint Acc↑/ALA↓ pattern that is harder to obtain by post-encoder methods. The ablation suite (locus, granularity, selection, reference) is unusually thorough for an inference-time paper and supplies concrete evidence that the acoustic score isolates a useful sparse set rather than merely boosting energy or layer scale. This opens a practical direction for encoder-side neuron steering in multimodal models and is of clear interest to the speech and audio-language community.

major comments (2)
  1. The central claim that Eq. (1) isolates task-relevant acoustic neurons rests on the reference ablation (Table III) and selection controls (Table IV). Those controls are supportive, but the manuscript never reports variance, multiple seeds for the main IAAN runs, or statistical significance of the Acc/ALA differences. Given that K and g are chosen on a small external set (190 LISTEN clips) and that some configurations in Table II fall near or below baseline (especially Kimi), the reported point gains of +25.7 / +21.4 / +9.7 should be accompanied by at least seed-level variability or bootstrap intervals so that readers can judge robustness of the ranking.
  2. Hyper-parameter selection (Sec. IV-C) maximises listening advantage LA = Acc − ALA on LISTEN, then transfers the chosen (K, g) to VoxParadox. Table II shows that the selected configurations are near-optimal but not always the absolute best, and that sensitivity is model-dependent. The paper should either (a) demonstrate that a fixed default (e.g., K=100, g=12) already recovers most of the gain without any development set, or (b) provide a clearer automatic selection rule that does not rely on a held-out sarcastic-speech corpus whose distribution may still be closer to VoxParadox than a truly out-of-domain set.
minor comments (5)
  1. Fig. 2a reports that the K=50 threshold sits ~13 standard deviations above the mean of the acoustic-score distribution; a short note on whether this extreme tail is stable across clips or dominated by a few outliers would help interpretation.
  2. In Table I the proprietary baselines are taken from prior work; a one-sentence confirmation that the evaluation protocol (prompt, decoding, option order) matches the reproduced open-source baselines would remove any residual comparability concern.
  3. The qualitative free-text examples (Table V) are illustrative but selected; a brief quantitative check (e.g., attribute mention rate over a larger free-form sample) would strengthen the claim that descriptions become more acoustically grounded.
  4. Notation: a_i(t) is introduced for frame-wise activations, yet Eq. (1) uses only the temporal mean; a sentence clarifying whether frame-level variance or max-pooling was tried would be useful for reproducibility.
  5. Code and exact seeds for the noise reference are not mentioned; releasing them would make the label-free scoring fully reproducible.

Circularity Check

0 steps flagged

No significant circularity: the acoustic score is an independent activation contrast, accuracy is an external benchmark measurement, and hyper-parameters are selected on a held-out set.

full rationale

The paper defines the per-neuron acoustic score solely from the difference of mean feed-forward activations on a real waveform versus a matched noise/silence reference (Eq. 1, Sec. III-B), with no dependence on task labels or VoxParadox accuracy. Amplification of the top-K neurons is then applied at inference, and the resulting accuracy/ALA numbers are measured on the independent VoxParadox MCQ suite (Table I). Hyper-parameters (K, g) are chosen by maximizing listening advantage on a separate LISTEN development set (Sec. IV-C) and never on the test set; transfer is reported rather than assumed. Controlled ablations (random selection, whole-layer amplification, post-encoder interventions, reference SNR sweeps) further demonstrate that the gains are not forced by construction of the score or by the mere act of amplification. No equation equates the reported accuracy lift to the score definition, no uniqueness theorem is imported from overlapping authors, and no fitted parameter is re-labeled as a prediction of a closely related quantity. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on two free hyper-parameters tuned on a development set, a small set of standard transformer and benchmark assumptions, and two methodological constructs (the acoustic score and the resulting ‘acoustic neurons’) whose validity is established only by the paper’s own downstream gains. No new physical entities are postulated.

free parameters (2)
  • neuron budget K = 50–200 (model-dependent)
    Number of highest-scoring neurons amplified; chosen per model by maximizing listening advantage on the LISTEN development set (values 50–200). All reported gains depend on this choice.
  • gain factor g = 8–16 (model-dependent)
    Multiplicative scale applied to selected neuron activations; selected jointly with K on the same development set (values 8–16). Controls the strength of the intervention.
axioms (3)
  • domain assumption Individual coordinates of the post-nonlinearity FFN activation in a transformer audio encoder can be treated as independently scorable and amplifiable ‘neurons’.
    Definition adopted in Sec. III-A from prior LLM interpretability work and applied without further justification to Whisper-style and other audio encoders.
  • ad hoc to paper A Gaussian-noise or silence reference of matched length and approximate mean/variance lacks the acoustic attributes present in the real waveform, so the activation difference isolates acoustic sensitivity.
    Foundational to Eq. (1) and Sec. III-B; partially ablated in Table III but still an unproven modeling choice for every attribute.
  • domain assumption VoxParadox’s adversarial multiple-choice design (voice attribute vs. transcript-implied label) correctly measures genuine acoustic rather than semantic understanding.
    Benchmark adopted in Sec. IV-A; all quantitative claims rest on its validity.
invented entities (2)
  • acoustic score s_i no independent evidence
    purpose: Label-free ranking of encoder neurons by how much more they activate on real audio than on a noise reference.
    Defined by Eq. (1); no independent probe (e.g., causal ablation of known acoustic features outside the paper’s tasks) is supplied.
  • acoustic neurons (top-K set S) no independent evidence
    purpose: The sparse, input-specific subset whose amplification is claimed to strengthen acoustic representations passed to the LLM.
    Identified solely by the acoustic score; existence and functional role are evidenced only by the accuracy gains reported in this work.

pith-pipeline@v1.1.0-grok45 · 22674 in / 2956 out tokens · 42970 ms · 2026-07-14T03:05:00.355332+00:00 · methodology

0 comments
read the original abstract

Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.

Figures

Figures reproduced from arXiv: 2607.11801 by An-Yu Cheng, Chih-Kai Yang, Hung-yi Lee, Ke-Han Lu, Yu-Han Huang.

Figure 1
Figure 1. Figure 1: Overview of IAAN. The encoder first runs on the real audio [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Acoustic-score distribution and layer localization. Statistics are [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 17 linked inside Pith

  1. [1]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,

    Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” inProc. International Conference on Machine Learning (ICML), 2024, pp. 25 125–25 148

    2024

  2. [2]

    Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities,

    S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities,” in Proc. International Conference on Machine Learning (ICML), 2025, pp. 19 358–19 405

    2025

  3. [3]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,

    S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S.-g. Lee, C.-H. H. Yang, R. Duraiswami, D. Manocha, R. Valle, and B. Catanzaro, “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” inAdvances in Neural Information Processing Sys- tems (NeurIPS), 2025

    2025

  4. [4]

    Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Qwen2.5-Omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Dang, B. Zhang, X. Wang, Y . Chu, and J. Lin, “Qwen2.5-Omni technical report,”arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment,

    K.-H. Lu, Z. Chen, S.-W. Fu, H. Huang, B. Ginsburg, Y .-C. F. Wang, and H.-y. Lee, “DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment,” inInterspeech 2024, 2024, pp. 4159–4163

    2024

  8. [8]

    Developing instruction-following speech language model without speech instruction-tuning data,

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, J. Balam, B. Ginsburg, Y .-C. F. Wang, and H.-Y . Lee, “Developing instruction-following speech language model without speech instruction-tuning data,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  9. [9]

    DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, S.-F. Huang, C.-K. Yang et al., “DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,”IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 2062–2076, 2026

    2062

  10. [10]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inProc. International Conference on Learning Representations (ICLR), 2024

    2024

  11. [11]

    A preliminary exploration with GPT-4o voice mode,

    Y .-X. Lin, C.-K. Yang, W.-C. Chen, C.-A. Li, C.-y. Huang, X. Chen, and H.-y. Lee, “A preliminary exploration with GPT-4o voice mode,” arXiv preprint arXiv:2502.09940, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Building a Taiwanese Mandarin spoken language model: A first attempt,

    C.-K. Yang, Y .-K. Fu, C.-A. Li, Y .-C. Lin, Y .-X. Lin, W.-C. Chen, H. L. Chung, C.-Y . Kuan, W.-P. Huang, K.-H. Luet al., “Building a Taiwanese Mandarin spoken language model: A first attempt,”arXiv preprint arXiv:2411.07111, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Kimi-audio technical report,

    D. Ding, Z. Ju, Y . Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tanget al., “Kimi-audio technical report,”arXiv preprint arXiv:2504.18425, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” inProc. International Conference on Machine Learning (ICML), 2023

    2023

  15. [15]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 5178–5193

    2023

  16. [16]

    AudioLens: A closer look at auditory attribute perception of large audio-language models,

    C.-K. Yang, N. Ho, Y .-J. Lee, and H.-y. Lee, “AudioLens: A closer look at auditory attribute perception of large audio-language models,” in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2025, pp. 1–8

    2025

  17. [17]

    SAKE: Towards editing auditory attribute knowledge of large audio- language models,

    C.-K. Yang, Y .-T. Piao, T.-W. Hsu, S.-W. Fu, Z. Chen, K.-H. Lu, S.- F. Huang, C.-H. H. Yang, Y .-C. F. Wang, Y .-N. Chen, and H.-y. Lee, “SAKE: Towards editing auditory attribute knowledge of large audio- language models,”arXiv preprint arXiv:2510.16917, 2025

    arXiv 2025

  18. [18]

    How auditory knowledge in LLM backbones shapes audio language models: A holistic evaluation,

    K.-H. Lu, S.-W. Fu, C.-H. H. Yang, Z. Chen, S.-F. Huang, C.-K. Yang, Y .-C. Lin, C.-Y . Hsiao, W. Ren, E.-P. Hu, Y .-H. Huang, A.-Y . Cheng, C.-H. Chiang, Y . Tsao, Y .-C. F. Wang, and H.-y. Lee, “How auditory knowledge in LLM backbones shapes audio language models: A holistic evaluation,”arXiv preprint arXiv:2603.19195, 2026

    arXiv 2026

  19. [19]

    Towards holistic evaluation of large audio-language models: A comprehensive survey,

    C.-K. Yang, N. S. Ho, and H.-y. Lee, “Towards holistic evaluation of large audio-language models: A comprehensive survey,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2025, pp. 10 144–10 170

    2025

  20. [20]

    MMAU: A massive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” inProc. International Conference on Learning Representations (ICLR), 2025

    2025

  21. [21]

    Do audio LLMs listen or read? analyzing and mitigating paralinguistic failures with V oxParadox,

    J. Pang, A. Chaubey, and M. Soleymani, “Do audio LLMs listen or read? analyzing and mitigating paralinguistic failures with V oxParadox,” inProc. International Conference on Machine Learning (ICML), 2026

    2026

  22. [22]

    Speech- copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,

    C.-Y . Kuan, C.-K. Yang, W.-P. Huang, K.-H. Lu, and H.-Y . Lee, “Speech- copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,” in2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 1060–1067

    2024

  23. [23]

    Resurfacing paralinguistic awareness in large audio language models,

    H. Yang, M. Wang, T. Wu, L. Qu, E. Shareghi, and G. Haffari, “Resurfacing paralinguistic awareness in large audio language models,” arXiv preprint arXiv:2603.11947, 2026

    arXiv 2026

  24. [24]

    AudioBench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- tics (NAACL). Association for Computational Linguistics, 2025, pp. 4297–4316

    2025

  25. [25]

    SAKURA: On the multi- hop reasoning of large audio-language models based on speech and audio information,

    C.-K. Yang, N. Ho, Y .-T. Piao, and H.-y. Lee, “SAKURA: On the multi- hop reasoning of large audio-language models based on speech and audio information,” inProc. Interspeech, 2025, pp. 1788–1792

    2025

  26. [26]

    V ox-Profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,

    T. Feng, J. Lee, A. Xu, Y . Lee, T. Lertpetchpun, X. Shi, H. Wang, T. Thebaud, L. Moro-Velazquez, D. Byrd, N. Dehak, and S. Narayanan, “V ox-Profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,”arXiv preprint arXiv:2505.14648, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    EchoMind: An interrelated multi-level benchmark for evaluating empathetic speech language models,

    L. Zhou, L. Yu, Y . Lyu, Y . Lin, Z. Zhao, J. Ao, Y . Zhang, B. Wang, and H. Li, “EchoMind: An interrelated multi-level benchmark for evaluating empathetic speech language models,”arXiv preprint arXiv:2510.22758, 2025

    arXiv 2025

  28. [28]

    Reducing object hal- lucination in large audio-language models via audio-aware decoding,

    T.-w. Hsu, K.-H. Lu, C.-H. Chiang, and H.-y. Lee, “Reducing object hal- lucination in large audio-language models via audio-aware decoding,” in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2025, pp. 1–7

    2025

  29. [29]

    How contrastive decoding enhances large audio language models?

    T.-Q. Lin, W.-P. Huang, Y .-C. Lin, and H.-y. Lee, “How contrastive decoding enhances large audio language models?”arXiv preprint arXiv:2603.09232, 2026

    arXiv 2026

  30. [30]

    Adaptive perturbation selection for contrastive audio decoding,

    A. I. Grace, Z. Huo, and W. Wang, “Adaptive perturbation selection for contrastive audio decoding,”arXiv preprint arXiv:2607.00247, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models,

    T.-E. Lin, K.-Y . Lee, and H.-y. Lee, “Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models,”arXiv preprint arXiv:2510.12851, 2025

    arXiv 2025

  32. [32]

    Nudging hidden states: Training-free model steering for chain- of-thought reasoning in large audio-language models,

    L.-L. Ieong, C.-C. Chen, C.-K. Yang, Y .-H. Huang, A.-Y . Cheng, and H.- y. Lee, “Nudging hidden states: Training-free model steering for chain- of-thought reasoning in large audio-language models,”arXiv preprint arXiv:2603.14636, 2026

    arXiv 2026

  33. [33]

    Are audio-language models listening? audio-specialist heads for adaptive audio steering,

    N. Glazer, L. Aharon, and E. Fetaya, “Are audio-language models listening? audio-specialist heads for adaptive audio steering,”arXiv preprint arXiv:2603.06854, 2026

    arXiv 2026

  34. [34]

    Neuron- level emotion control in speech-generative large audio-language models,

    X. Zhao, I. R. Ulgen, P. Koehn, B. Schuller, and B. Sisman, “Neuron- level emotion control in speech-generative large audio-language models,” arXiv preprint arXiv:2603.17231, 2026

    arXiv 2026

  35. [35]

    Who wins the conflict? mechanistic interpretability of text bias in audio llms,

    H. Cho, S. Yoo, J. Jang, C. Kim, and J. S. Chung, “Who wins the conflict? mechanistic interpretability of text bias in audio llms,”arXiv preprint arXiv:2606.18924, 2026

    Pith/arXiv arXiv 2026

  36. [36]

    Anatomy of the modality gap: Dissecting the internal states of end-to-end speech LLMs,

    M.-H. Hsu, X. Zhang, X. Tian, J. Zhang, and Z. Wu, “Anatomy of the modality gap: Dissecting the internal states of end-to-end speech LLMs,” arXiv preprint arXiv:2603.01502, 2026

    arXiv 2026

  37. [37]

    Disen- tangling textual and acoustic features of neural speech representations,

    H. Mohebbi, G. Chrupała, W. Zuidema, A. Alishahi, and I. Titov, “Disen- tangling textual and acoustic features of neural speech representations,” arXiv preprint arXiv:2410.03037, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Whisper-AT: Noise- robust automatic speech recognizers are also strong general audio event taggers,

    Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-AT: Noise- robust automatic speech recognizers are also strong general audio event taggers,” inProc. Interspeech, 2023, pp. 2798–2802

    2023

  39. [39]

    Layer-wise analysis of a self- supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self- supervised speech representation model,” inProc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021

    2021

  40. [40]

    Behind the scenes: Mechanistic interpretability of LoRA-adapted Whisper for speech emo- tion recognition,

    Y . Ma, X. Lu, J. Sang, X. Jiang, and R. Li, “Behind the scenes: Mechanistic interpretability of LoRA-adapted Whisper for speech emo- tion recognition,” inProc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026

    2026

  41. [41]

    Speech-IFEval: Evaluat- ing Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models,

    K.-H. Lu, C.-Y . Kuan, and H.-y. Lee, “Speech-IFEval: Evaluat- ing Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models,” inInterspeech 2025, 2025, pp. 2078– 2082

    2025

  42. [42]

    Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning bench- mark for speech,

    C.-Y . Huang, K.-H. Lu, S.-H. Wang, C.-Y . Hsiao, C.-Y . Kuan, H. Wu, S. Arora, K.-W. Chang, J. Shi, Y . Peng, R. Sharma, S. Watanabe, B. Ramakrishnan, S. Shehata, and H.-Y . Lee, “Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning bench- mark for speech,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics,...

    2024

  43. [43]

    Dynamic-SUPERB Phase- 2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-y. Huang, W.-C. Chen, S.-w. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.- C. Tseng, A. Diwan, Y .-J. Shih, J. Shiet al., “Dynamic-SUPERB Phase- 2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” inProc. International Conference on Learning Representations (ICLR), 2025

    2025

  44. [44]

    AIR-Bench: Benchmarking large audio- language models via generative comprehension,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, Y . Leng, Y . Lv, Z. Zhao, C. Zhou, and J. Zhou, “AIR-Bench: Benchmarking large audio- language models via generative comprehension,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 1979–1998

    2024

  45. [45]

    MMAR: A challenging benchmark for deep reasoning in speech, audio, music, and their mix,

    Z. Ma, Y . Ma, Y . Zhu, C. Yang, Y .-W. Chao, R. Xu, W. Chen, Y . Chen, Z. Chen, J. Cong, K. Li, K. Li, S. Li, X. Li, X. Li, Z. Lian, Y . Liang, M. Liu, Z. Niu, T. Wang, Y . Wang, Y . Wang, Y . Wu, G. Yang, J. Yu, R. Yuan, Z. Zheng, Z. Zhou, H. Zhu, W. Xue, E. Benetos, K. Yu, E.- S. Chng, and X. Chen, “MMAR: A challenging benchmark for deep reasoning in s...

    2025

  46. [46]

    Mugen: Evaluating and improving multi-audio understanding of large audio-language models,

    C.-K. Yang, Y .-S. Tsai, Y .-K. Guo, P.-L. Tsai, Y .-T. Piao, H.-W. Chen, T.- L. Hsiao, Y .-M. Hsu, K.-H. Lu, and H.-y. Lee, “Mugen: Evaluating and improving multi-audio understanding of large audio-language models,” arXiv preprint arXiv:2603.09714, 2026

    Pith/arXiv arXiv 2026

  47. [47]

    Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,

    Y .-C. Lin, T.-Q. Lin, C.-K. Yang, K.-H. Lu, W.-C. Chen, C.-Y . Kuan, and H.-Y . Lee, “Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,” in2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 439–446

    2024

  48. [48]

    Emotional damage: Investigating safety vulnerabilities of large audio-language models under speaker emotional variations,

    B.-H. Feng, C.-F. Liu, Y .-H. Li Liang, C.-K. Yang, S.-W. Fu, Z. Chen, K.-H. Lu, S.-F. Huang, C.-H. H. Yang, Y .-C. Frank Wang, Y .-N. Chen, and H.-Y . Lee, “Emotional damage: Investigating safety vulnerabilities of large audio-language models under speaker emotional variations,” inICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and ...

    2026

  49. [49]

    Hearing the order: Investigating position bias in large audio-language models,

    Y .-X. Lin, C.-A. Li, S.-L. Wei, P.-C. Chen, H.-H. Chen, and H.-y. Lee, “Hearing the order: Investigating position bias in large audio-language models,”arXiv preprint arXiv:2510.00628, 2025

    arXiv 2025

  50. [50]

    When audio and text disagree: Benchmarking text bias in large audio-language models under cross-modal inconsistencies,

    C. Wang, G. Deng, X. Yang, H. Qiu, and T. Zhang, “When audio and text disagree: Benchmarking text bias in large audio-language models under cross-modal inconsistencies,” inProc. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025

    2025

  51. [51]

    SD-Eval: A benchmark dataset for spoken dialogue under- standing beyond words,

    J. Ao, Y . Wang, X. Tian, D. Chen, J. Zhang, L. Lu, Y . Wang, H. Li, and Z. Wu, “SD-Eval: A benchmark dataset for spoken dialogue under- standing beyond words,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024, pp. 56 898–56 918

    2024

  52. [52]

    Benchmarking contextual and paralinguistic reasoning in speech-LLMs: A case study with in-the-wild data,

    Q. Wang, H. B. Sailor, T. Liu, W. Zhang, M. Huzaifah, N. Lertcheva, S. Sun, N. F. Chen, J. Wu, and A. Aw, “Benchmarking contextual and paralinguistic reasoning in speech-LLMs: A case study with in-the-wild data,” inFindings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 2025, pp. 14 133–14 148

    2025

  53. [53]

    Do audio LLMs really LISTEN, or just transcribe? measuring lexical vs. acoustic emotion cues reliance,

    J. Chen, Z. Guo, J. Chun, P. Wang, A. Perrault, and M. Elsner, “Do audio LLMs really LISTEN, or just transcribe? measuring lexical vs. acoustic emotion cues reliance,” inProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2026, pp. 5848–5877

    2026

  54. [54]

    TAU: A benchmark for cultural sound understanding beyond se- mantics,

    Y .-C. Lin, Y .-H. Chen, J.-K. Dong, Y .-H. Huang, S.-C. Chenet al., “TAU: A benchmark for cultural sound understanding beyond se- mantics,” inICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 15 542– 15 546

    2026

  55. [55]

    Contrastive decoding: Open-ended text generation as optimization,

    X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis, “Contrastive decoding: Open-ended text generation as optimization,” inProc. Annual Meeting of the Association for Computational Linguistics (ACL), 2023, pp. 12 286–12 312

    2023

  56. [56]

    DoLa: Decoding by contrasting layers improves factuality in large language models,

    Y .-S. Chuang, Y . Xie, H. Luo, Y . Kim, J. Glass, and P. He, “DoLa: Decoding by contrasting layers improves factuality in large language models,” inProc. International Conference on Learning Representations (ICLR), 2024

    2024

  57. [57]

    Steering language models with activation engi- neering,

    A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid, “Steering language models with activation engi- neering,”arXiv preprint arXiv:2308.10248, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Steering llama 2 via contrastive activation addition,

    N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner, “Steering llama 2 via contrastive activation addition,” inProc. Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 15 504–15 522

    2024

  59. [59]

    Inference- time intervention: Eliciting truthful answers from a language model,

    K. Li, O. Patel, F. Vi ´egas, H. Pfister, and M. Wattenberg, “Inference- time intervention: Eliciting truthful answers from a language model,” in Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  60. [60]

    Representation engineering: A top-down approach to ai transparency,

    A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks, “Representation engineering: A top-down approach to ai transparency,”arXiv preprint arXiv:2310.01405, 2023

    Pith/arXiv arXiv 2023

  61. [61]

    Transformer feed-forward layers are key-value memories,

    M. Geva, R. Schuster, J. Berant, and O. Levy, “Transformer feed-forward layers are key-value memories,” inProc. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 5484– 5495

    2021

  62. [62]

    Knowledge neurons in pretrained transformers,

    D. Dai, L. Dong, Y . Hao, Z. Sui, B. Chang, and F. Wei, “Knowledge neurons in pretrained transformers,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2022, pp. 8493–8502

    2022

  63. [63]

    Locating and editing factual associations in GPT,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in GPT,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 17 359–17 372

    2022

  64. [64]

    A VCD: Mitigating hallucinations in audio-visual large language models through contrastive decoding,

    C. Jung, Y . Jang, and J. S. Chung, “A VCD: Mitigating hallucinations in audio-visual large language models through contrastive decoding,” in Advances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  65. [65]

    Temporal contrastive decoding: A training-free method for large audio-language models,

    Y . Li, Y . Liu, Z. Song, Y . Wei, M. Tak ´aˇc, and S. Lahlou, “Temporal contrastive decoding: A training-free method for large audio-language models,”arXiv preprint arXiv:2604.15383, 2026, accepted to Findings of ACL 2026

    Pith/arXiv arXiv 2026

  66. [66]

    Activation steering for accent adaptation in large audio language models,

    J. Sun, Y . Xiao, S. K. Chung, Q. Hu, G. Huang, E.-J. Holden, and T. Dang, “Activation steering for accent adaptation in large audio language models,”arXiv preprint arXiv:2603.05813, 2026

    Pith/arXiv arXiv 2026

  67. [67]

    Language-specific neurons: The key to multilingual capabilities in large language models,

    T. Tang, W. Luo, H. Huang, D. Zhang, X. Wang, X. Zhao, F. Wei, and J.- R. Wen, “Language-specific neurons: The key to multilingual capabilities in large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024, pp. 5701–5715

    2024

  68. [68]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 53 728–53 741. [Online]. Ava...

    2023

  69. [69]

    GPT-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “GPT-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  70. [70]

    Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025