pith. sign in

arxiv: 2606.09966 · v1 · pith:B2VA726Bnew · submitted 2026-06-08 · 💻 cs.SD

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

Shakhrul Iman Siam , Tiantian Feng , Jiankun Zhang , Shrikanth Narayanan , Mi Zhang This is my paper

Pith reviewed 2026-06-27 15:03 UTC · model grok-4.3

classification 💻 cs.SD
keywords respiratory diseasesmultimodal foundation modelcontrastive alignmentaudio-text integrationzero-shot detectiondisease identificationAUROC evaluation
0
0 comments X

The pith

RespiraMFM aligns respiratory audio with clinical text via contrastive learning to raise disease detection accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that a multimodal foundation model can integrate respiratory sounds with patient medical history and symptoms through contrastive audio-text alignment. This produces stronger cross-modal representations than unimodal audio approaches, leading to measurable gains in identifying respiratory diseases. The evaluation covers five diseases across seven datasets in both fine-tuned and zero-shot regimes. A reader would care because respiratory conditions drive significant mortality and earlier, more accurate detection could reduce healthcare burdens.

Core claim

RespiraMFM integrates respiratory sounds with textual clinical information through an effective contrastive alignment strategy, enabling better cross-modal representations that lead to improved performance in identifying five major respiratory diseases across seven real-world datasets.

What carries the argument

The contrastive alignment strategy for audio-text multimodal integration, which learns representations between respiratory sounds and corresponding textual clinical information.

If this is right

  • Yields 9.15% higher AUROC than baselines on supervised fine-tuning tasks for five respiratory diseases.
  • Yields 20.98% higher AUROC than baselines on zero-shot tasks across the same diseases.
  • Improves generalizability of audio-based detection when textual clinical context is available.
  • Supports earlier diagnosis and better clinical decision-making for respiratory conditions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same alignment technique could be tested on other audio medical signals such as cardiac or neurological sounds.
  • Deployment in settings with electronic health records might amplify the zero-shot gains by supplying richer text context.
  • Controlled trials that isolate alignment from capacity differences would strengthen causal claims about the mechanism.

Load-bearing premise

The reported AUROC gains result from the contrastive audio-text alignment rather than from differences in model capacity, training details, or dataset choices that were not controlled.

What would settle it

An ablation that removes only the contrastive alignment module, keeps model size and training data fixed, and measures whether the 9.15% supervised and 20.98% zero-shot AUROC gains disappear.

Figures

Figures reproduced from arXiv: 2606.09966 by Jiankun Zhang, Mi Zhang, Shakhrul Iman Siam, Shrikanth Narayanan, Tiantian Feng.

Figure 1
Figure 1. Figure 1: Overview of RespiraMFM. address these limitations, we propose a contrastive alignment module that facilitates more effective in￾tegration by aligning audio and text representations in a shared semantic space. Our approach goes beyond mere dimensional alignment, aiming to es￾tablish a shared representation space that enables effective integration of multimodal information. 3 RespiraMFM 3.1 Overview [PITH_F… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of contrastive learning-based audio-text alignment. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of dataset scaling. Unobserved Respiratory Diseases: Regarding the unobserved respiratory diseases, we further compare RespiraMFM with BTS and RespLLM on the prediction of asthma (T8) and pneumonia (T9). In both tasks, the models are trained on datasets from T1 to T4, none of which include instances of asthma or pneumonia. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: AUC of zero-shot disease detection task with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: t-SNE visualization of audio embeddings for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average attention weight of top-5 tokens for [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Symptom Occurrence Distribution by COVID [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Class Distribution Across Datasets Coswara: The Coswara dataset (Bhattacharya et al., 2023) is a diverse collection of respiratory sounds and detailed metadata, recorded between April 2020 and February 2022 from 2,635 individ￾uals, including 1,819 SARS-CoV-2 negative, 674 positive, and 142 recovered cases. It features nine categories of respiratory sounds, covering varia￾tions of breathing, coughing, and s… view at source ↗
Figure 9
Figure 9. Figure 9: AUC on Task T5 under various missing input [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: LoRA vs Full Fine-tuning AUC comparison. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Per-task comparison of stage-1 projector [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
read the original abstract

Respiratory diseases remain a leading cause of global mortality, where timely and accurate diagnosis is critical to improving patient outcomes and reducing healthcare burdens. While prior work has explored audio-based models for respiratory disease detection, such unimodal approaches often suffer from limited generalizability and diagnostic precision. In this paper, we propose RespiraMFM, a Multimodal Foundation Model that integrates respiratory sounds with patient medical history and symptoms to enhance diagnostic accuracy and disease detection capabilities. We introduce an effective contrastive alignment strategy for audio-text multimodal integration, allowing the model to learn better cross-modal representations between respiratory sounds and corresponding textual clinical information. We evaluate RespiraMFM across five major respiratory diseases using seven real-world datasets in both supervised fine-tuning and zero-shot settings, achieving a 9.15% improvement in AUROC on supervised tasks and a 20.98% gain on zero-shot tasks over existing baselines. These findings underscore the potential of our framework to advance early diagnosis and improve clinical decision-making in respiratory disease management.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes RespiraMFM, a multimodal foundation model that integrates respiratory audio signals with textual patient medical history and symptoms via a contrastive audio-language alignment strategy. It claims evaluation across five respiratory diseases on seven real-world datasets, reporting 9.15% AUROC gains in supervised fine-tuning and 20.98% gains in zero-shot settings relative to existing baselines.

Significance. If the reported AUROC improvements can be causally attributed to the contrastive alignment component after proper controls for model capacity and training procedure, the work would provide evidence that multimodal audio-text representations improve generalizability in respiratory disease detection over unimodal audio baselines.

major comments (2)
  1. [Abstract] Abstract: The central performance claims (9.15% supervised and 20.98% zero-shot AUROC gains) are stated without any description of the baseline models, their architectures, parameter counts, training data volume, or optimization schedules. This prevents verification that the gains arise from the contrastive alignment rather than uncontrolled differences in model scale or compute.
  2. [Abstract] Abstract and evaluation description: No information is supplied on statistical testing, dataset characteristics (size, demographics, exclusion criteria), ablation studies isolating the alignment loss, or whether baselines were re-trained under matched conditions. These omissions make the attribution of gains to the proposed method load-bearing and unverifiable from the provided text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these comments on the clarity of our claims and evaluation. We will revise the abstract and evaluation sections to include the requested details on baselines, controls, datasets, and ablations, ensuring the attribution to contrastive alignment is verifiable.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central performance claims (9.15% supervised and 20.98% zero-shot AUROC gains) are stated without any description of the baseline models, their architectures, parameter counts, training data volume, or optimization schedules. This prevents verification that the gains arise from the contrastive alignment rather than uncontrolled differences in model scale or compute.

    Authors: We agree the abstract should provide this context to support causal attribution. In the revision we will expand the abstract to name the baseline models, note their architectures and approximate parameter counts, state that all models were trained on the same data volumes with matched optimization schedules, and clarify that the reported gains are measured against these controlled baselines. revision: yes

  2. Referee: [Abstract] Abstract and evaluation description: No information is supplied on statistical testing, dataset characteristics (size, demographics, exclusion criteria), ablation studies isolating the alignment loss, or whether baselines were re-trained under matched conditions. These omissions make the attribution of gains to the proposed method load-bearing and unverifiable from the provided text.

    Authors: We acknowledge these omissions in the current abstract and evaluation description. The revised manuscript will add: (i) results of statistical testing (e.g., paired t-tests or bootstrap confidence intervals on AUROC differences), (ii) explicit dataset sizes, demographics, and exclusion criteria for each of the seven datasets, (iii) ablation studies that isolate the contribution of the contrastive alignment loss, and (iv) explicit confirmation that baselines were re-trained under identical conditions. These additions will be placed in both the abstract and the main evaluation section. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical claims rest on external datasets and baselines

full rationale

The paper proposes RespiraMFM and reports AUROC gains on seven real-world datasets in supervised and zero-shot settings. No derivation chain, equations, or self-citations are invoked that reduce any reported metric to a fitted input or self-defined quantity by construction. The evaluation uses external benchmarks, satisfying the condition for a self-contained result against independent data.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no information on model architecture details, hyperparameters, loss function formulations, or dataset preprocessing steps, so no free parameters, axioms, or invented entities can be identified.

pith-pipeline@v0.9.1-grok · 5725 in / 1121 out tokens · 32788 ms · 2026-06-27T15:03:41.931754+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 14 canonical work pages · 4 internal anchors

  1. [1]

    Nature Machine Intelligence , volume=

    Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers , author=. Nature Machine Intelligence , volume=. 2024 , publisher=

    2024

  2. [2]

    Scientific Data , volume=

    Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection , author=. Scientific Data , volume=. 2023 , publisher=

    2023

  3. [3]

    Scientific Data , volume=

    The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms , author=. Scientific Data , volume=. 2021 , publisher=

    2021

  4. [4]

    Scientific Data , volume=

    a dataset of Solicited Cough Sound for tuberculosis triage testing , author=. Scientific Data , volume=. 2024 , publisher=

    2024

  5. [5]

    Science Advances , volume=

    TBscreen: A passive cough classifier for tuberculosis screening with a controlled dataset , author=. Science Advances , volume=. 2024 , publisher=

    2024

  6. [6]

    Physiological measurement , volume=

    An open access database for the evaluation of respiratory sound classification algorithms , author=. Physiological measurement , volume=. 2019 , publisher=

    2019

  7. [7]

    Journal of Ambient Intelligence and Humanized Computing , pages=

    Recognition of pulmonary diseases from lung sounds using convolutional neural networks and long short-term memory , author=. Journal of Ambient Intelligence and Humanized Computing , pages=. 2022 , publisher=

    2022

  8. [8]

    2017 ieee international conference on acoustics, speech and signal processing (icassp) , pages=

    CNN architectures for large-scale audio classification , author=. 2017 ieee international conference on acoustics, speech and signal processing (icassp) , pages=. 2017 , organization=

    2017

  9. [9]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Clap learning audio concepts from natural language supervision , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  10. [10]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  11. [11]

    ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection , author=. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2022 , organization=

    2022

  12. [12]

    arXiv preprint arXiv:2406.16148 , year=

    Towards open respiratory acoustic foundation models: Pretraining and benchmarking , author=. arXiv preprint arXiv:2406.16148 , year=

    work page arXiv

  13. [13]

    arXiv preprint arXiv:2410.05361 , year=

    RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction , author=. arXiv preprint arXiv:2410.05361 , year=

    work page arXiv

  14. [14]

    arXiv preprint arXiv:2403.02522 , year=

    HeAR--Health Acoustic Representations , author=. arXiv preprint arXiv:2403.02522 , year=

    work page arXiv

  15. [15]

    ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Exploring automatic COVID-19 diagnosis via voice and symptoms from crowdsourced data , author=. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2021 , organization=

    2021

  16. [16]

    arXiv preprint arXiv:2406.06786 , year=

    BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification , author=. arXiv preprint arXiv:2406.06786 , year=

    work page arXiv

  17. [17]

    Proceedings of the 18th ACM international conference on Multimedia , pages=

    Opensmile: the munich versatile and fast open-source audio feature extractor , author=. Proceedings of the 18th ACM international conference on Multimedia , pages=

  18. [18]

    arXiv preprint arXiv:2305.14032 , year=

    Patch-mix contrastive learning with audio spectrogram transformer on respiratory sound classification , author=. arXiv preprint arXiv:2305.14032 , year=

    work page arXiv

  19. [19]

    ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Whosecough: In-the-wild cougher verification using multitask learning , author=. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2020 , organization=

    2020

  20. [20]

    Nature , volume=

    Large language models encode clinical knowledge , author=. Nature , volume=. 2023 , publisher=

    2023

  21. [21]

    Communications medicine , volume=

    Making cough count in tuberculosis care , author=. Communications medicine , volume=. 2022 , publisher=

    2022

  22. [22]

    LLaMA: Open and Efficient Foundation Language Models

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    JAMA internal medicine , volume=

    Estimation of excess deaths associated with the COVID-19 pandemic in the United States, March to May 2020 , author=. JAMA internal medicine , volume=. 2020 , publisher=

    2020

  24. [24]

    2024 , publisher=

    Global tuberculosis report 2024 , author=. 2024 , publisher=

    2024

  25. [25]

    Journal of medical Internet research , volume=

    Diagnostic value of imaging modalities for COVID-19: scoping review , author=. Journal of medical Internet research , volume=. 2020 , publisher=

    2020

  26. [26]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Domain Adaptation Explainability & Fairness in AI for Medical Image Analysis: Diagnosis of COVID-19 based on 3-D Chest CT-scans , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  27. [27]

    Neurocomputing , pages=

    Automated detection and forecasting of covid-19 using deep learning techniques: A review , author=. Neurocomputing , pages=. 2024 , publisher=

    2024

  28. [28]

    Nature medicine , volume=

    Large language models in medicine , author=. Nature medicine , volume=. 2023 , publisher=

    2023

  29. [29]

    Machine Learning for Health (ML4H) , pages=

    Llms accelerate annotation for medical information extraction , author=. Machine Learning for Health (ML4H) , pages=. 2023 , organization=

    2023

  30. [30]

    LoRA: Low-Rank Adaptation of Large Language Models

    Lora: Low-rank adaptation of large language models , author=. arXiv preprint arXiv:2106.09685 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    Adventitious respiratory classification using attentive residual neural networks , author=

  32. [32]

    , author=

    LungRN+ NL: An improved adventitious lung sound classification using non-local block resnet neural network with mixup data augmentation. , author=. Interspeech , pages=

  33. [33]

    arXiv preprint arXiv:2203.16141 , year=

    Example-based explanations with adversarial attacks for respiratory sound analysis , author=. arXiv preprint arXiv:2203.16141 , year=

    work page arXiv

  34. [34]

    Nature Medicine , pages=

    A generalist vision--language foundation model for diverse biomedical tasks , author=. Nature Medicine , pages=. 2024 , publisher=

    2024

  35. [35]

    Nature Medicine , volume=

    Collaboration between clinicians and vision--language models in radiology report generation , author=. Nature Medicine , volume=. 2025 , publisher=

    2025

  36. [36]

    International Journal of Chronic Obstructive Pulmonary Disease , pages=

    Current progress of COPD Early detection: key points and novel strategies , author=. International Journal of Chronic Obstructive Pulmonary Disease , pages=. 2023 , publisher=

    2023

  37. [37]

    Respiratory Medicine , volume=

    Mortality risk attributable to classification of chronic obstructive pulmonary disease and reduced lung function: a 21-year longitudinal cohort study , author=. Respiratory Medicine , volume=. 2021 , publisher=

    2021

  38. [38]

    Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=

    Transformers: State-of-the-art natural language processing , author=. Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=

    2020

  39. [39]

    International journal of epidemiology , volume=

    Reflection on modern methods: Revisiting the area under the ROC Curve , author=. International journal of epidemiology , volume=. 2020 , publisher=

    2020

  40. [40]

    arXiv:2306.09093 (2023)

    Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration , author=. arXiv preprint arXiv:2306.09093 , year=

    work page arXiv

  41. [41]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Unibind: Llm-augmented unified and balanced representation space to bind them all , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  42. [42]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  43. [43]

    arXiv preprint arXiv:2508.06895 , year=

    BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models , author=. arXiv preprint arXiv:2508.06895 , year=

    work page arXiv

  44. [44]

    An embarrassingly simple approach for LLM with strong ASR capacity,

    An embarrassingly simple approach for llm with strong asr capacity , author=. arXiv preprint arXiv:2402.08846 , year=

    work page arXiv

  45. [45]

    Qwen2-Audio Technical Report

    Qwen2-audio technical report , author=. arXiv preprint arXiv:2407.10759 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2502.12900 , year=

    Soundwave: Less is More for Speech-Text Alignment in LLMs , author=. arXiv preprint arXiv:2502.12900 , year=

    work page arXiv

  47. [47]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  48. [48]

    Textbooks Are All You Need

    Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  49. [49]

    Microsoft Research Blog , volume=

    Phi-2: The surprising power of small language models , author=. Microsoft Research Blog , volume=