Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Off-the-shelf audio and text models can be combined into an automated attack that infers a person's private attributes from sound alone, with average accuracy of 86.7 percent on the paper's benchmark.

desk verdict A well-engineered and transparent framework, but the main benchmark's label-curated audio inflates the headline accuracy and the central covert-profiling claim needs reframing. read the letter →

arxiv 2507.10016 v2 pith:BUH2UFZ3 submitted 2025-07-14 cs.CR cs.SDeess.AS

Lixu Wang , Kaixiang Yao , Xinfeng Li , Dong Yang , Haoyang Li , Xiaofeng Wang , Wei Dong This is my paper
classification cs.CRcs.SDeess.AS
keywords audioprivateattributeprofilingmultimodallargelanguagemodelsinferenceattackprivacyleakagemulti-agentframeworkbenchmarkdatasetin-contextunlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that current off-the-shelf multimodal models can already be assembled into a practical covert attack that infers a person's sensitive attributes from audio alone, a capability it calls audio private attribute profiling. To demonstrate this, it introduces AP2, a benchmark of eighty assembled real-world audio profiles and forty characters from recent TV dramas annotated with twelve sensitive attributes, and proposes Gifts, a two-agent framework in which a text LLM guides, interrogates, reviews, and consolidates the outputs of an audio-language model. On AP2-Com, Gifts reports an average accuracy of 86.7 across all twelve attributes, beating every baseline by 4.4 to 40.7 percentage points and beating human participants by 22.5 points while taking about one quarter of the time. The paper also tests two defenses, showing that model-level in-context unlearning and data-level phoneme-based noise jamming both substantially reduce the attack's accuracy, and argues the result is a current risk that warrants safety alignment of audio models.

What carries the argument

Gifts is the central mechanism: a multi-agent pipeline pairing an audio-language model (Gemini 1.5 Pro) with a general-purpose text LLM (Claude 3.5 Sonnet). It runs in five phases: the LLM generates attribute-specific guidance from audio captions and transcripts; the ALM makes a short inference; the LLM poses concise true/false/uncertain clue-validation questions that the ALM answers from the acoustic signal, avoiding the long-form reasoning that makes ALMs hallucinate; the LLM scrutinizes whether the inference is supported and, if not, orders a single re-inference with the previous answer negated; and the LLM consolidates evidence across multiple audio clips into the final profile. The AP2 dataset is the supporting object, built so that no individual maps to a real person in AP2-Com and so that AP2-TV's post-September-2024 dramas prevent memorization-based inference.

What would settle it

Run the Gifts pipeline on a fresh sample of consenting real speakers whose age, income, health, occupation, and other attributes are self-reported, using short audio clips with no speech content and no identifying context; if per-attribute accuracy falls toward the captioning-only baselines rather than the reported 86.7 percent average, the benchmark result was carried by its constructed labels rather than by acoustic inference.

Watch

Extended reading notes

Core claim

The central claim is that audio private attribute profiling is feasible today with models available through public APIs. The paper shows that neither an audio-language model alone nor a text-only large language model fed with transcripts can profile reliably, but a hybrid agent can: Gifts combines an ALM with an LLM in a guidance–inference–forensics–scrutinization–consolidation loop, and reaches an average accuracy of 86.7 on the twelve attributes of AP2-Com, with perfect gender accuracy and near-perfect scores on health condition and marital status. On AP2-TV the same pattern holds, and the gains are largest for acoustic-driven attributes such as age, gender, accent, and health, indicating that the framework lets an LLM exploit acoustic features it cannot hear directly. The paper's stated conclusion is that the privacy risk from audio is not hypothetical, that current ALMs lack adequate safety alignment, and that defenses at both model and data level are needed.

Load-bearing premise

The load-bearing premise is that the benchmark labels measure real sensitive attributes: AP2-Com's people are randomly assembled composites of audio and expert-chosen attribute values with no real-world referent, and AP2-TV's labels are three annotators' judgments about fictional characters, informed by forums, promotional material, and media coverage, so the headline accuracy is only as valid as those labels as a proxy for genuine human attributes.

Editorial extensions

If this is right

  • Anyone with API access to current audio and text models can reconstruct a twelve-attribute profile of a victim from passively captured or scraped audio, without the victim saying anything sensitive.
  • The reported 22.5-point accuracy advantage over humans, at roughly a quarter of the time, removes the cost barrier that previously confined audio profiling to trained analysts.
  • Because ALMs already leak attributes under naive prompts and do not exhibit refusal behavior, model providers cannot rely on current safety alignment to stop the attack.
  • The two defenses provide distinct mitigation paths: in-context unlearning, which providers or users can apply at inference time, and phoneme-based noise jamming, which individuals can deploy at the source.
  • The AP2 dataset and Gifts implementation enable future measurement of audio privacy leakage, giving defenders a benchmark to test alignment and countermeasures against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported accuracy transfers beyond constructed benchmarks, the practical exposure is larger than the paper states, because microphones in phones, smart speakers, and meeting software make audio capture continuous, passive, and hard to detect.
  • A natural follow-up experiment would strip the textual channel from the final consolidation phase to test whether the LLM is reasoning from acoustics or primarily from textual stereotypes in transcripts and captions; the paper leaves this decomposition untested.
  • The AP2-TV temporal-independence design is directly extensible: applying Gifts to later seasons or to sequels would reveal how much of the attack depends on memorized media knowledge versus genuinely transferable acoustic cues.
  • The defense results likely upper-bound the protection, since an adaptive adversary aware of noise jamming or in-context unlearning could adjust prompts or retrain, so the residual risk after deployed defenses is probably higher than the paper's post-defense numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AP2, a two-part audio benchmark (AP2-Com, composed from curated public audio resources, and AP2-TV, extracted from recent TV dramas) annotated with twelve sensitive attributes, and proposes Gifts, a multi-agent framework that combines an audio-language model (Gemini1.5-Pro) with a large language model (Claude3.5-Sonnet) through guidance, inference, forensics, scrutinization, and consolidation phases. The authors report that Gifts infers all twelve private attributes on AP2-Com with an average accuracy of 86.7%, outperforming all baselines and surpassing a human participant pool by 22.5% in average accuracy while using about one quarter of the time. They also evaluate two defenses, in-context unlearning and phoneme-based noise jamming, and report that both reduce profiling accuracy. The paper claims to be the first study of MLLM-based audio private attribute profiling and positions AP2 and Gifts as resources for future research and defense development.

Significance. If the reported measurements were valid, the paper would demonstrate a serious and easily assembled privacy risk: off-the-shelf ALMs and LLMs can be combined into an agentic pipeline that infers sensitive attributes from audio more accurately and faster than human listeners. The work also makes concrete contributions: a substantial annotation effort across twelve attributes, a multi-phase agent design with ablations showing each phase's contribution, and an initial exploration of model-level and data-level defenses. However, the empirical foundation currently does not support the headline claims. The construction of AP2-Com leaks the target labels into the audio itself, the human evaluation is small and procedurally inconsistent, and the AP2-TV labels are not per-clip ground truth. These issues are load-bearing because the paper's central contribution is an empirical measurement of attack feasibility rather than a theoretical derivation. The dataset and framework are potentially useful, but the reported accuracy numbers and the human-competition claim need to be re-established on a non-leaky, properly validated benchmark.

major comments (4)
  1. [Section 5.1] AP2-Com is constructed so that the attribute label is a property of the audio clip rather than of the individual. The text states that experts performed 'targeted retrieval of audio data from existing public resources ... identifying samples indicative of typical behaviors, dialogues, and activities associated with each entry' and then 'randomly assigned the validated attribute values and their corresponding audio samples to speakers.' This means, for example, HAB clips are domestic event sounds (e.g., vacuum cleaner) and OCC clips are occupation-related Sound Bible sounds; the audio itself transparently signals the label. The 86.7% average accuracy on Table 2 therefore largely measures audio-event recognition plus commonsense mapping from the curated scene to the label, not covert profiling of a person's private attributes from naturalistic audio. Because Table 4's human comparison and Tables 5-6's defense evaluations use the same benchmark, the headline risk claim is inflated. A valid evaluation would require audio clips that were not selected because they exemplify the target label, or per-clip labels that are independent of the audio content that the model is asked to interpret.
  2. [Section 7.3 and Appendix E.2] The human-vs-machine comparison is not a fair or statistically meaningful head-to-head. It uses only three individuals from AP2-Com, all subject to the label leakage described above. The appendix states that participants were forbidden from using LLMs ('the use of large language models ... to generate attribute descriptions was strictly prohibited'), while the main text says participants were 'permitted to use search engines or LLMs to retrieve relevant information.' In addition, the model pipeline receives automatically generated event descriptions and spoken-word transcriptions, whereas human participants listen only to raw audio. The 22.5% accuracy gap and the quarter-time claim in Table 4 are therefore not a valid basis for the conclusion that Gifts outperforms humans at covert audio profiling.
  3. [Section 5.2 and Appendix D.2] AP2-TV does not provide per-clip ground truth for the audio segments used in evaluation. Character-level labels are derived from the full series plus external discourse ('promotional materials, media coverage, and online forum discussions'), and Appendix D.2 explicitly instructs annotators to consider character development across the entire narrative arc. An individual audio clip extracted from one episode may therefore not match the aggregate character label, especially when the character changes over time. Moreover, using fictional characters annotated from external discourse as a proxy for real human sensitive attributes is an unvalidated assumption. Figures 3 and 4 report no numeric tables or error bars, so the actual per-attribute differences between Gifts and the baselines on AP2-TV cannot be assessed from the paper.
  4. [Section 7.1, Evaluation Metrics] For the fuzzy attributes (ACC, PER, SOP, OCC, HAB), the paper uses Claude3.7-Sonnet as the automatic judge, which is from the same model family as the Claude3.5-Sonnet LLM agent used inside Gifts. No evidence is provided that this judge agrees with human raters or that it is neutral across the compared model families. This creates an unmeasured risk of systematic bias in the main quantitative claims. The paper should validate the judge (e.g., human-judge agreement on a sample) and report results with at least one alternative judge or a human-rated subset, especially because these fuzzy attributes contribute directly to the reported averages in Tables 2-4.
minor comments (5)
  1. [Appendix A] The statement that 'AP2-Com does not include any real humans' is contradicted by the use of CommonVoice recordings and other real-human speech datasets as base profiles; the stochastic composition of attributes does not remove the fact that the audio originates from real people. The claim should be rephrased to say that no individual profile maps to a single real person.
  2. [Section 7.1 and Table 2] The three repeated runs yield very small reported variances, but no significance tests are provided; the paper should report per-attribute statistical tests (or at least confidence intervals) for the Gifts-versus-best-baseline differences.
  3. [Figures 3 and 4] The AP2-TV results are presented only as figures without numeric values or error bars; adding a table with means and standard deviations would allow the claimed consistent superiority of Gifts to be verified.
  4. [Table 4] The 'Time Spent' comparison mixes API latency for MLLMs with human self-reported total time; these quantities are not directly comparable, and no variance or per-participant breakdown is reported.
  5. [Appendix E.2] The human participant pool is highly skewed (92% aged 21-30, mean 24.2 years), which limits the generalizability of the human baseline; the paper should acknowledge this and, ideally, recruit a more diverse sample.

Circularity Check

1 steps flagged · score 6.0 of 10

AP2-Com is built by selecting audio that exemplifies each label and then assigning those pre-labeled clips to synthetic speakers, so the headline 86.7% accuracy partly measures recognition of the curator's own retrieval criterion.

  1. self definitional [Section 5.1 (Dataset Construction Process) and Appendix D.1; Figure 1]
    "we propose to follow the principle of 'first annotate then collect' to retrieve and compose existing public audio datasets ... For other attributes, we first curated an information repository for each attribute based on Wikipedia ... the aforementioned three experts performed targeted retrieval of audio data from existing public resources ... identifying samples indicative of typical behaviors, dialogues, and activities associated with each entry. ... we randomly assigned the validated attribute values and their corresponding audio samples to speakers within the CommonVoice corpus."

    The target attribute value is the retrieval criterion: each audio clip is selected because it exemplifies the label (e.g., vacuum-cleaner sounds for HAB, occupation-related Sound Bible clips for OCC), and the ground-truth profile is then assembled by randomly assigning these pre-labeled clips to synthetic speakers. Evaluating Gifts on AP2-Com therefore asks the model to recover the very criterion used to curate each clip, so high accuracy on Table 2 is partly a measurement of audio-event classification with the label entailed by construction rather than profiling a real person from covertly captured audio. Since Table 4 and the defense evaluations also use AP2-Com, the central empirical claim inherits this by-construction label leakage.

full rationale

The paper is fundamentally an empirical measurement study rather than a formal derivation, and there is no chain of equations in which an output is definitionally identical to an input; there is also no load-bearing self-citation chain. The central circularity is in the benchmark construction: AP2-Com follows 'first annotate then collect,' meaning the annotators first fix attribute values and then retrieve audio clips that are 'indicative of typical behaviors, dialogues, and activities associated with each entry.' The ground-truth label is thus the selection criterion for the test audio, and the 'randomly assigned' profiles are composites assembled from those pre-labeled clips. Consequently, the model's 86.7% average accuracy partly reflects recognition of the curator's own retrieval labels, and the same leakage inflates the human comparison in Table 4, which uses three AP2-Com individuals, and the defense evaluations, which are measured on the same benchmark. AP2-TV is more externally grounded, though its annotations are also derived from expert interpretation of narrative and online discourse; that is a validity concern rather than a by-construction reduction. The use of Claude3.7-Sonnet to score fuzzy attributes while Claude3.5-Sonnet powers Gifts is not circular, though it is a fairness consideration. Overall, the headline claim is partially forced by the benchmark's construction, so the circularity score is 6 rather than a full 10.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The paper introduces a new benchmark and a new framework, but both are self-contained: the benchmark is the basis for the evaluation, and the framework is the basis for the attack. The most important 'free parameters' are not model parameters but evaluation choices: the expert-generated ground truth, the same-family LLM judge, and the restricted human baseline. The benchmark is not fully public, so the strongest claims are conditioned on internal curation choices.

free parameters (4)
  • Expert curation of AP2-TV ground truth
    The attribute labels for 40 TV characters are selected by three expert annotators with access to external forum and promotional material. There is no audit or inter-annotator agreement metric reported, so the ground truth is, in effect, a hand-picked target set that is favorable to the model.
  • Fuzzy evaluation scoring by Claude3.7-Sonnet = Five-level similarity score
    For fuzzy attributes, the evaluation of whether a model output is 'similar' to ground truth is delegated to Claude3.7-Sonnet, which is a model from the same family as Claude3.5-Sonnet used inside the framework. This introduces a potential same-family evaluation bias and the mapping from the five-level score to accuracy is not reported.
  • Episode-to-episode attribute values for AP2-TV = Single value per character
    Character attributes can evolve across episodes, but each character receives one value per attribute. This over-simplifies and removes within-character variance, boosting the chance that the majority-vote consolidation is correct.
  • Human evaluation with only 3 individuals = 3 individuals
    The human baseline in Table 4 is based on 3 individuals from AP2-Com, which is too small to infer a reliable human-vs-model gap, and it excludes AP2-TV entirely for copyright reasons.
assumptions (4)
  • domain assumption Tone, pitch, accent, background sounds, and contextual cues in audio are stable and sufficient to infer 12 sensitive attributes.
    The whole benchmark and the attack model rest on the premise that the audio contains semantically meaningful cues for attributes like income, education, and marital status. The paper cites this as 'contextual cues' in Section 5.1, but does not provide any human labeling study or inter-annotator agreement to establish how reliable these cues are.
  • domain assumption The human baseline in Section 7.3 is a fair comparator for the model
    Humans were prohibited from using LLMs, while models use a full pipeline with transcription, captioning, guidance, forensics, and consolidation. The comparison is presented as 'MLLM agents beat humans' but the task constraints differ, so the baseline is not a like-for-like comparator.
  • domain assumption A 3-expert annotation with two-round cross-validation provides a reliable ground truth for TV characters
    Section 5.2 states that 'expert annotators' annotated with the help of online resources and viewing the complete series, but no inter-annotator agreement scores, no disagreement resolution protocol, and no reliability statistics are reported.
  • domain assumption Claude3.7-Sonnet fuzzy evaluation is a neutral judge for the same family of models used in the framework
    The fuzzy metrics for PER, SOP, OCC, HAB, ACC, and EDU are computed by Claude3.7-Sonnet, which may share bias with Claude3.5-Sonnet used in the framework. No calibration or human validation of these scores is provided.
invented entities (2)
  • AP2 dataset
    purpose: Benchmark to evaluate audio private attribute profiling
    The dataset is a central output of the paper, but it is not publicly released, and it is constructed from a mix of real and fictional individuals. Its validity is asserted, not externally verified.
  • Gifts framework
    purpose: Guide-inference-forensics-scrutinization-consolidation agent framework
    A method, not a natural entity. Its performance is only measured on AP2 and the paper's own baselines.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents." pith.science (2026). https://pith.science/paper/BUH2UFZ3

@misc{pith2026250710016,
  author       = {Pith},
  title        = {Pith review of: The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUH2UFZ3}},
  note         = {Machine review of arXiv:2507.10016}
}
read the original abstract

Our research uncovers a novel privacy risk associated with multimodal large language models (MLLMs): the ability to infer sensitive personal attributes from audio data -- a technique we term audio private attribute profiling. This capability poses a significant threat, as audio can be covertly captured without direct interaction or visibility. Moreover, compared to images and text, audio carries unique characteristics, such as tone and pitch, which can be exploited for more detailed profiling. However, two key challenges exist in understanding MLLM-employed private attribute profiling from audio: (1) the lack of audio benchmark datasets with sensitive attribute annotations and (2) the limited ability of current MLLMs to infer such attributes directly from audio. To address these challenges, we introduce AP^2, an audio benchmark dataset that consists of two subsets collected and composed from real-world data, and both are annotated with sensitive attribute labels. Additionally, we propose Gifts, a hybrid multi-agent framework that leverages the complementary strengths of audio-language models (ALMs) and large language models (LLMs) to enhance inference capabilities. Gifts employs an LLM to guide the ALM in inferring sensitive attributes, then forensically analyzes and consolidates the ALM's inferences, overcoming severe hallucinations of existing ALMs in generating long-context responses. Our evaluations demonstrate that Gifts significantly outperforms baseline approaches in inferring sensitive attributes. Finally, we investigate model-level and data-level defense strategies to mitigate the risks of audio private attribute profiling. Our work validates the feasibility of audio-based privacy attacks using MLLMs, highlighting the need for robust defenses, and provides a dataset and framework to facilitate future research.

Figures

Figures reproduced from arXiv: 2507.10016 by the authors.

Figure 1
Figure 1. The overview and building workflow of the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. An example of employing the Gifts framework to infer the social stratum of a victim from their audio data. Each answer 𝑎𝑖 in A is only one word, "True", "False" or "Uncertain". Since these questions reflect clues that support the inference result, a higher proportion of "True" indicates greater confidence. 6.4 Scrutinization The phase of Scrutinization within the Gifts framework follows the standard LLM-as-a-judge m… view at source ↗
Figure 3
Figure 3. Performance comparison between Gifts and other baselines in profiling Acoustic-driven attributes on AP2 -TV. HAB SOP PER INC OCC MAR 40 50 60 70 80 90 100 Qwen2.5-Instruct-14B Llama3-Instruct-14B GPT-4o Claude3.5-Sonnet Gifts (ours) (a) LLMs HAB SOP PER INC OCC MAR 40 50 60 70 80 90 100 Gemini1.5-Pro + Qwen2.5-Instruct-14B Gemini1.5-Pro + Llama3-Instruct-14B Gemini1.5-Pro + GPT-4o Gemini1.5-Pro + Claude3.5-Sonnet Gi… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison between Gifts and other baselines in profiling Reasoning-driven attributes on AP2 -TV [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Uncertainty-based Debiasing and Unlearning for Decontamination

    cs.CY 2026-06 unverdicted novelty 6.0 of 10

    UBD leverages ensemble uncertainty to estimate per-sample memorization and construct debiased targets for post-hoc correction or unlearning, yielding output distributions closer to uncontaminated models on MMLU-Pro an...

  2. Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    Audio language models are benchmarked on five semantic and paralinguistic reasoning tasks to reveal limitations in handling spoken audio evidence, accent variation, and domain shifts.

Reference graph

Works this paper leans on

96 extracted references · 49 canonical work pages · cited by 2 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Tony Alex, Sara Ahmed, Armin Mustafa, Muhammad Awais, and Philip JB Jack- son. 2024. DTF-AT: decoupled time-frequency audio transformer for event classification. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 17647–17655

    2024

  3. [3]

    Bizhan Alipour, Abdessamad Imine, and Michaël Rusinowitch. 2019. Gender inference for Facebook picture owners. In Trust, Privacy and Security in Digital Business: 16th International Conference, TrustBus 2019, Linz, Austria, August 26–29, 2019, Proceedings 16. Springer, 145–160

    2019

  4. [4]

    Anthropic. 2024. Claude Series. https://www.anthropic.com/news/claude-3-5- sonnet

    2024

  5. [5]

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber

  6. [6]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022)

    arXiv 2022

  7. [7]

    Jean-Pierre Briot and François Pachet. 2020. Deep learning for music generation: challenges and directions. Neural Computing and Applications 32, 4 (2020), 981– 993

    2020

  8. [8]

    Felix Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben, and Björn Schuller. 2023. Speech-based age and gender prediction with transformers. In Speech Communication; 15th ITG Conference . VDE, 46–50

    2023

Show all 96 references
  1. [9]

    Yu Chen, Tingxin Li, Huiming Liu, and Yang Yu. 2023. Hide and Seek (HaS): A Lightweight Framework for Prompt Privacy Protection. arXiv preprint arXiv:2309.03057 (2023)

    2023 arXiv

  2. [10]

    Peng Cheng and Utz Roedig. 2022. Personal voice assistant security and privacy—a survey. Proc. IEEE 110, 4 (2022), 476–507

    2022

  3. [11]

    Oubaïda Chouchane, Baptiste Brossier, Jorge Esteban Gamboa Gamboa, Thomas Lardy, Hemlata Tak, Orhan Ermis, Madhu R Kamble, Jose Patino, Nicholas WD Evans, Melek Önen, et al. 2021. Privacy-Preserving Voice Anti-Spoofing Using Secure Multi-Party Computation.. In Interspeech. 856–860

    2021

  4. [12]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 (2024)

    2024 arXiv

  5. [13]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 (2023)

    2023 arXiv

  6. [14]

    Pieter Delobelle and Bettina Berendt. 2022. Fairdistillation: mitigating stereo- typing in language models. Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2022), 638–654

    2022

  7. [15]

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems 36 (2023), 18090–18108

    2023

  8. [16]

    Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gam- per, Rita Singh, and Bhiksha Raj. 2025. Audio entailment: Assessing deductive reasoning for audio understanding. AAAI (2025)

    2025

  9. [17]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  10. [18]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

  11. [19]

    Geoffrey Hinton et al. 2024. Statement on AI Risk. https://www.safe.ai/work/ statement-on-ai-risk

    2024

  12. [20]

    European Union EU. 2021. Artificial intelligence act ai act, 2021. https://eur- lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206

    2021

  13. [21]

    Tiantian Feng, Hanieh Hashemi, Murali Annavaram, and Shrikanth S Narayanan

  14. [22]

    Tiantian Feng, Raghuveer Peri, and Shrikanth Narayanan. 2022. User-Level Differ- ential Privacy against Attribute Inference Attack of Speech Emotion Recognition on Federated Learning. In Proc. Interspeech 2022. 5055–5059

    2022

  15. [23]

    Freesound. 2025. Freesound. https://freesound.org/. Accessed: 2025-04-13

    2025

  16. [24]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal...

    2017

  17. [25]

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities. arXiv preprint arXiv:2406...

    2024 arXiv

  18. [26]

    Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. [n. d.]. CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models. In The Twelfth International ...

  19. [27]

    Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James R Glass. [n. d.]. Listen, Think, and Understand. In The Twelfth International Conference on Preprint, 2025, USA Lixu Wang et. al Learning Representations

    2025

  20. [28]

    Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al

  21. [29]

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 976–980

    2022

  22. [30]

    Hatman. [n. d.]. Hatman/audio-emotion-detection. https://huggingface.co/ Hatman/audio-emotion-detection. Accessed: 2025-4-14

    2025

  23. [31]

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. 2024. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14271–14280

    2024

  24. [32]

    Peng Huang, Yao Wei, Peng Cheng, Zhongjie Ba, Li Lu, Feng Lin, Fan Zhang, and Kui Ren. 2023. InfoMasker: Preventing Eavesdropping Using Phoneme-Based Noise.. In NDSS

    2023

  25. [33]

    Bargav Jayaraman and David Evans. 2022. Are attribute inference attacks just imputation?. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1569–1582

    2022

  26. [34]

    Panagiotis Kapetanidis, Fotios Kalioras, Constantinos Tsakonas, Pantelis Tza- malis, George Kontogiannis, Theodora Karamanidou, Thanos G Stavropoulos, and Sotiris Nikoletseas. 2024. Respiratory diseases diagnosis using audio analysis and artificial intelligence: a systematic r...

    2024

  27. [35]

    Zvi Kons, Orith Toledo-Ronen, and M Carmel. 2013. Audio event classification using deep neural networks.. In Interspeech. 1482–1486

    2013

  28. [36]

    Siddique Latif, Moazzam Shoukat, Fahad Shamshad, Muhammad Usama, Yi Ren, Heriberto Cuayáhuitl, Wenwu Wang, Xulong Zhang, Roberto Togneri, Erik Cam- bria, et al. 2023. Sparks of large audio models: A survey and outlook. arXiv preprint arXiv:2308.12792 (2023)

    2023 arXiv

  29. [37]

    Jingjin Li, Chao Chen, Mostafa Rahimi Azghadi, Hossein Ghodosi, Lei Pan, and Jun Zhang. 2023. Security and privacy problems in voice assistant applications: A survey. Computers & Security 134 (2023), 103448

    2023

  30. [38]

    Yiming Li, Shuo Shao, Yu He, Junfeng Guo, Tianwei Zhang, Zhan Qin, Pin-Yu Chen, Michael Backes, Philip Torr, Dacheng Tao, and Kui Ren. 2025. Rethinking Data Protection in the (Generative) Artificial Intelligence Era. arXiv preprint arXiv:2507.03034 (2025)

    2025 arXiv

  31. [39]

    Feiran Liu, Yuzhe Zhang, Xinyi Huang, Yinan Peng, Xinfeng Li, Lixu Wang, Yu- tong Shen, Ranjie Duan, Simeng Qin, Xiaojun Jia, et al. 2025. The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework. arXiv preprint arXiv:...

    2025 arXiv

  32. [40]

    Songting Liu. 2024. Zero-shot Voice Conversion with Diffusion Transformers. arXiv preprint arXiv:2411.09943 (2024)

    2024 arXiv

  33. [41]

    Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. 2024. Robustifying Safety-Aligned Large Language Models through Clean Data Curation. arXiv preprint arXiv:2405.19358 (2024)

    2024 arXiv

  34. [42]

    Steven R Livingstone and Frank A Russo. 2018. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PloS one 13, 5 (2018), e0196391

    2018

  35. [43]

    Daniel M Low, Kate H Bentley, and Satrajit S Ghosh. 2020. Automated assess- ment of psychiatric disorders using speech: A systematic review. Laryngoscope investigative otolaryngology 5, 1 (2020), 96–116

    2020

  36. [44]

    Samaneh Madanian, David Parry, Olayinka Adeleye, Christian Poellabauer, Farhaan Mirza, Shilpa Mathew, and Sandy Schneider. 2022. Automatic speech emotion recognition using machine learning: digital transformation of mental health. In Proceedings of the Annual Pacific Asia Conf...

    2022

  37. [45]

    Xinhao Mei, Xubo Liu, Mark D Plumbley, and Wenwu Wang. 2022. Automated audio captioning: An overview of recent progress and new challenges. EURASIP journal on audio, speech, and music processing 2022, 1 (2022), 26

    2022

  38. [46]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2024. WavCaps: A ChatGPT- Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multi- modal Research. IEEE/ACM Transactions on Audio, Speech...

    2024

  39. [47]

    Yuantian Miao, Minhui Xue, Chao Chen, Lei Pan, Jun Zhang, Benjamin Zi Hao Zhao, Dali Kaafar, and Yang Xiang. 2021. The Audio Auditor: User-Level Mem- bership Inference in Internet of Things Voice Services. Proceedings on Privacy Enhancing Technologies (2021)

    2021

  40. [48]

    Manuel Milling, Florian B Pokorny, Katrin D Bartl-Pokorny, and Björn W Schuller

  41. [49]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)

    2024 arXiv

  42. [50]

    Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, and Boris Gins- burg. 2024. Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment. In Proc. Interspeech 2024. 3425–3429

    2024

  43. [51]

    Evangelos K Oikonomou and Rohan Khera. 2024. Artificial intelligence-enhanced patient evaluation: bridging art and science. European heart journal 45, 35 (2024), 3204–3218

    2024

  44. [52]

    Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. [n. d.]. In-Context Unlearning: Language Models as Few-Shot Unlearners. InForty-first International Conference on Machine Learning

  45. [53]

    Frontiers in digital health 4 (2022), 886615

    Is speech the new blood? recent progress in ai-based disease detection from audio in a nutshell. Frontiers in digital health 4 (2022), 886615

    2022

  46. [54]

    Pixabay. 2025. Pixabay. https://pixabay.com/. Accessed: 2025-04-13

    2025

  47. [55]

    Dejan Porjazovski, Yaroslav Getman, Tamás Grósz, and Mikko Kurimo. 2023. Advancing audio emotion and intent recognition with large pre-trained models and Bayesian inference. In Proceedings of the 31st ACM International Conference on Multimedia. 9477–9481

    2023

  48. [56]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356 [eess.AS] https://arxiv.org/abs/2212.04356

    2022 arXiv

  49. [57]

    Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. 2024. Fusecap: Leveraging large language models for enriched fused image captions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5689–5700

    2024

  50. [58]

    Bizhan Alipour Pijani, Abdessamad Imine, and Michaël Rusinowitch. 2020. You are what emojis say about your pictures: language-independent gender inference attack on Facebook. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1826–1834

    2020

  51. [59]

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2025. Mmau: A massive multi-task audio understanding and reasoning benchmark. ICLR (2025)

    2025

  52. [60]

    Grace Snow and Elizabeth Guardiani. 2019. Movement disorders and voice. Otolaryngologic Clinics of North America 52, 4 (2019), 759–767

    2019

  53. [61]

    Soundbible. [n. d.]. Soundbible. https://soundbible.com/

  54. [62]

    Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. [n. d.]. Beyond Memorization: Violating Privacy via Inference with Large Language Models. In The Twelfth International Conference on Learning Representations

  55. [63]

    Frank Rudzicz, Aravind Kumar Namasivayam, and Talya Wolff. 2012. The TORGO database of acoustic and articulatory speech from speakers with dysarthria. Language resources and evaluation 46 (2012), 523–541

    2012

  56. [64]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, MA Zejun, and Chao Zhang. [n. d.]. SALMONN: Towards Generic Hearing Abilities for Large Language Models. In The Twelfth International Conference on Learning Representations

  57. [65]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  58. [66]

    Renée M Tobin, William G Graziano, Eric J Vanman, and Louis G Tassinary. 2000. Personality, emotional experience, and efforts to control emotions. Journal of personality and social psychology 79, 4 (2000), 656

    2000

  59. [67]

    Batuhan Tömekçe, Mark Vero, Robin Staab, and Martin Vechev. 2024. Private Attribute Inference from Images with Vision-Language Models. NeurIPS (2024)

    2024

  60. [68]

    Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek

  61. [69]

    IEEE Signal Processing Letters 27 (2020), 600–604

    Pre-training audio representations with self-supervision. IEEE Signal Processing Letters 27 (2020), 600–604

    2020

  62. [70]

    Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection reg- ulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10–5555

    2017

  63. [71]

    Feng Wu, Lei Cui, Shaowen Yao, and Shui Yu. 2024. Inference Attacks in Machine Learning as a Service: A Taxonomy, Review, and Promising Directions. arXiv preprint arXiv:2406.02027 (2024)

    2024 arXiv

  64. [72]

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip

  65. [73]

    Yang Xiao and Rohan Kumar Das. 2024. WildDESED: An LLM-Powered Dataset for Wild Domestic Environment Sound Event Detection System. In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2024 Workshop (DCASE2024). 196–200

    2024

  66. [74]

    Wei-Cheng Tseng, Wei-Tsung Kao, and Hung-yi Lee. 2022. Membership Inference Attacks Against Self-supervised Speech Models. In Proc. Interspeech 2022. 5040– 5044

    2022

  67. [75]

    M. A. van Wyk, M. Bekker, X. L. Richards, and K. J. Nixon. 2023. Protect Your Prompts: Protocols for IP Protection in LLM Applications. arXiv preprint arXiv:2306.06297 (2023)

    2023 arXiv

  68. [76]

    Dong Yang, Yiyi Cai, Yuki Saito, Lixu Wang, and Hiroshi Saruwatari. 2025. Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis. arXiv preprint The Man Behind the Sound : Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents Prep...

    2025

  69. [77]

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. arXiv preprint arXiv:2402.07729 (2024)

    2024 arXiv

  70. [78]

    Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. 2024. Recent advances of foundation language models-based continual learning: A survey. Comput. Surveys (2024)

    2024

  71. [79]

    In 2023 IEEE International Conference on Big Data (BigData)

    Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData) . IEEE, 2247–2256

    2023

  72. [80]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) . IEEE, 268–282

    2018

  73. [81]

    Shen Yan, Di Huang, and Mohammad Soleymani. 2020. Mitigating biases in mul- timodal personality assessment. Proceedings of the 2020 International Conference on Multimodal Interaction (2020), 361–369

    2020

  74. [82]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 Technical Report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  75. [83]

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. Findings of ACL (2024)

    2024

  76. [84]

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  77. [85]

    Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, and Jing Shao. 2024. SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model. arXiv preprint arXiv:2406...

    2024 arXiv

  78. [86]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211

    2024

  79. [87]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  80. [89]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)

    2023 arXiv

  81. [93]

    Tianyi Zhao, Liangliang Zhang, Yao Ma, and Lu Cheng. 2024. A survey on safe multi-modal learning systems. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 6655–6665

    2024

  82. [95]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States.Findings of the Association for Computational Linguistics: EMNLP (2024), 2461–2488

    2024

  83. [96]

    Juan Zuluaga-Gomez, Sara Ahmed, Danielius Visockas, and Cem Subakan

  84. [97]

    Highly Similar, Similar, Moderately Similar, Slightly Similar, and Completely Different

    CommonAccent: Exploring Large Acoustic Pretrained Models for Ac- cent Classification Based on Common Voice. Interspeech 2023 (2023). https: //arxiv.org/abs/2305.18283 Preprint, 2025, USA Lixu Wang et. al APPENDIX A ETHICAL AND OPEN SCIENCE CONSIDERATIONS Our research aims to s...

    2023 arXiv

  85. [2014]

    The distress analysis interview corpus of human and computer interviews.. In LREC. Reykjavik, 3123–3128

  86. [2020]

    In Proceedings of the Twelfth Language Resources and Evaluation Conference

    Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference . 4218–4222

  87. [2022]

    In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP)

    Enhancing privacy through domain adaptive noise injection for speech emo- tion recognition. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 7702–7706

    2022

  88. [2023]

    InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.