REVIEW 4 major objections 5 minor 31 references
Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fusing neural codec embeddings with spectral features via bandit-weighted cross-attention improves heart murmur classification, with the DAC+MFCC pair reaching 89.93% accuracy and 79.37% macro F1 on the CirCor subset.
desk verdict Plausible fusion recipe for heart murmur classification, but the headline results rest on an unspecified CV split that may leak patients; needs patient-stratified re-evaluation before the SOTA claim can be taken seriously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is BAOMI, a fusion framework built on a multi-head cross-attention mechanism augmented with a multi-armed bandit. The NACR and spectral features are each passed through a 1D CNN, flattened, and projected into query, key, and value spaces for two cross-attention flows (NACR attending to spectral and spectral attending to NACR). Each attention head maintains a Q-value updated by a reward proportional to how much that head reduces the cross-entropy loss; these Q-values are converted into soft weights that scale the heads before concatenation into the fused representation. This head-weighting step is what distinguishes BAOMI from the standard cross-attention baseline, and it is the mechanism the paper credits for mitigating noise from less informative heads.
What would settle it
Re-run the full BAOMI protocol on the CirCor dataset with folds split strictly by patient ID, ensuring no patient appears in both training and test, and check whether the DAC+MFCC result (89.93% accuracy, 79.37% macro F1) and its margin over the cross-attention baseline persist; if the margin shrinks or vanishes, the state-of-the-art claim rests on leakage from non-patient-disjoint splits.
Extended reading notes
Core claim
The central claim is that neural audio codec representations and spectral features are complementary for heart murmur classification, and that their fusion is best achieved by a bandit-based cross-attention mechanism that dynamically weights attention heads according to their contribution to loss reduction. On the CirCor subset, BAOMI with DAC and MFCC achieves 89.93% accuracy, 79.37% macro F1, and 89.67% weighted F1, outperforming individual NACRs, individual spectral features, and a standard cross-attention fusion baseline across all tested combinations. The paper further claims that this constitutes a new state of the art for heart murmur classification on the dataset.
Load-bearing premise
The five-fold cross-validation is stated as five folds over 3,163 recordings from 963 patients, but the paper does not say whether all recordings of a single patient are kept within one fold, so if a patient's recordings appear in both training and test sets the reported accuracy gains could be inflated by patient memorization.
Editorial extensions
If this is right
- Fusing NACRs with spectral features becomes a viable default recipe for heart murmur classification, since the best fused combination outperforms the best individual feature set by roughly 9 percentage points in accuracy and 6 points in macro F1.
- The bandit-based head-weighting mechanism should generalize to other feature-fusion and multimodal tasks where some attention heads are expected to be noisy or uninformative.
- The performance ordering across codecs (DAC and SNAC24 leading) suggests that codec choice matters and that pre-training data distribution influences transferability to heart sounds.
- If the result holds under patient-disjoint validation, the approach provides a practical pipeline for classifying murmur presence from raw PCG audio without hand-crafted segmentation.
- The improved t-SNE clustering and confusion matrices indicate the fused representation separates the Present and Absent classes more cleanly than either feature family alone.
Reading between the lines
- Editorial inference: The reported gains may depend on the unstated split design; if the five-fold splits are recording-level rather than patient-level, the accuracy advantage could shrink under patient-disjoint evaluation, so the state-of-the-art claim should be re-tested on patient-stratified folds.
- Editorial inference: The same bandit-weighted cross-attention recipe could be applied to other codec-spectral pairs (e.g., LFCC with SNAC24) and to other audio-classification domains such as respiratory sound or speech emotion classification, where similar complementary structure exists.
- Editorial inference: The paper does not ablate the number of attention heads; a natural testable extension is whether the bandit weighting still helps with 2, 8, or 16 heads, or whether the benefit is specific to the 4-head configuration used here.
- Editorial inference: Since NACRs alone underperform spectral features on this dataset, the practical message for practitioners is that codec embeddings add value primarily through fusion, not as a replacement for classical spectral descriptors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BAOMI, a bandit-based cross-attention mechanism for fusing neural audio codec representations (NACRs such as EnCodec, DAC, SNAC, SpeechTokenizer) with spectral features (MFCC, LFCC) for heart murmur classification on the CirCor DigiScope dataset. The mechanism learns per-head weights via Q-values updated from loss-reduction rewards, and the fused representation is used for classification. The authors report that BAOMI outperforms individual features and a standard multi-head cross-attention baseline across many combinations, with the best result from DAC+MFCC (accuracy 89.93%, macro F1 79.37%, weighted F1 89.67%). They claim to establish a new state of the art for heart murmur classification.
Significance. If the reported results are reliable, the paper offers a useful fusion strategy that combines complementary acoustic representations, and it makes code and checkpoints publicly available. The idea of using bandit-based head weighting in cross-attention is interesting and could transfer to other multimodal fusion tasks. However, the significance is currently limited by evaluation weaknesses: the cross-validation split unit is unspecified, no variance measures are reported, and the state-of-the-art claim is argued only indirectly. The contribution is therefore promising but not yet convincingly established.
major comments (4)
- [Section 4.1] The unit of five-fold cross-validation is unspecified. The dataset is described as 3,163 recordings from 963 patients, and the task is heart murmur classification, where the murmur label is a patient-level attribute. The text only says 'four folds are used for training and one fold for testing' without stating whether folds are stratified by patient or by recording. If the split is by recording, then multiple recordings from the same patient can appear in both training and test folds, allowing the model to memorize patient-specific recording characteristics and inflate the reported scores in Tables 1 and 2 (e.g., BAOMI DAC+MFCC accuracy 89.93%). Please specify the split unit and, if it is not patient-stratified, repeat the experiments with patient-level folds and report per-fold results.
- [Section 4.2, Tables 1 and 2] No measure of variance or statistical significance is reported. The tables show only the average over five folds for each metric. For the best combination (DAC+MFCC), the BAOMI accuracy advantage over the cross-attention baseline is only 0.12 percentage points (89.93 vs. 89.81), although the macro-F1 improvement is larger (79.37 vs. 75.06). Without per-fold scores, standard deviations, or a paired test, it is impossible to judge whether the proposed mechanism provides a stable improvement or whether the difference arises from a single favorable fold. Please report per-fold values and error bars or a significance test.
- [Section 4.2, final paragraph] The claim of setting a new state of the art is not directly supported. The authors argue that BAOMI improves over individual spectral features that were reported as state of the art in prior works [23,24,25,9], but they do not compare BAOMI against those specific methods under the same evaluation protocol. Prior works may use different data splits, feature sets, or preprocessing, so the indirect comparison does not establish a new state of the art. Please either implement and evaluate the most relevant prior methods on the same folds, or temper the claim to 'competitive with' or 'better than the individual-feature baselines considered here.'
- [Equation (5), Section 3.1] The bandit reward R_h is computed from the reduction of the cross-entropy loss, which is the same objective minimized by the whole network for classification. This makes the head weighting directly optimized toward the training loss, and the paper does not clarify whether ΔL_h is measured on the training split or a held-out split. The risk is that the bandit selects heads that overfit the training loss rather than learn a generalizable fusion. Please clarify the computation of ΔL_h and discuss this self-referential tuning risk. Also, the hyperparameters γ and ε in Eq. (5) are not given in Section 3.1 or in the training details; please state their values.
minor comments (5)
- [Abstract and Section 1] There are several typos: 'a agent' should be 'an agent', 'intially' should be 'initially', and 'calclulated' in Section 3.1 should be 'calculated'. Please proofread throughout.
- [Section 4.1] The class distribution '179, 695, and 68 samples' sums to 942, not 963 as stated for the number of patients. Please clarify whether these counts refer to patients or recordings, and reconcile the totals.
- [Section 3.1] The phrase 'we keep the modeling details as same as used in the downstream modeling' should be 'we keep the modeling details the same as those used in downstream modeling.' Also, for SpeechTokenizer, 'we add the codes and average it' should be 'average them.'
- [Figure 2] The caption 't-SNE Plots- (a) CNN (MFCC) ...' should use a colon or period instead of a dash, and the figure labels should be formatted consistently.
- [Section 2] The abbreviations NAC and NACR are used interchangeably; please define both at first use and use them consistently thereafter.
Circularity Check
No circularity: BAOMI's bandit head-weighting is trained end-to-end on the task loss, and the empirical SOTA claim does not reduce to a fitted input or to self-citation.
full rationale
The paper's derivation chain is empirical rather than analytic. The bandit head rewards in Eq. (5) use the same cross-entropy loss minimized by the overall network, but this is a training objective, not a renamed prediction: the head weights are learned parameters, and the reported evaluation metrics (accuracy, macro-F1, weighted-F1) are not identical to the reward function. The claim that BAOMI 'prioritizes the most informative heads' is a description of the mechanism's design, not an independently predicted quantity. The complementarity of NACRs and SFs is a post hoc interpretation of Table 2, not a derivation from the mechanism, so it cannot reduce the result to its inputs. There are no load-bearing self-citations: the prior work on NACRs for heart sound classification [11] is external, and no uniqueness theorem or ansatz is imported from the authors' own previous work. The SOTA claim is an indirect comparison made by showing that DAC+MFCC fusion beats the paper's own MFCC baseline, with the baseline asserted to match earlier MFCC-based SOTA references; this is a weak empirical argument but not circular. The most serious validity concern, namely that Section 4.1 does not state whether the five-fold splits are by patient or by recording, is a potential data-leakage risk in the experiments, not a circularity in the derivation, and it cannot be established from the text alone. Overall, no step of the paper's reasoning is equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- reward decay factor gamma
- number of attention heads H =
4
- epsilon in reward normalization
assumptions (4)
- standard math Softmax attention and bandit reward updates behave as described in Eqs. 3-6.
- domain assumption NACRs and spectral features capture complementary information for heart sounds.
- ad hoc to paper Loss reduction per attention head is a valid reward signal for improving classification.
- domain assumption The CirCor 963-patient subset and the five-fold CV procedure produce valid performance estimates.
Cite this review
Pith. "Pith review of Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism." pith.science (2026). https://pith.science/paper/QWRKI2PZ
@misc{pith2026250601148,
author = {Pith},
title = {Pith review of: Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism},
year = {2026},
howpublished = {\url{https://pith.science/paper/QWRKI2PZ}},
note = {Machine review of arXiv:2506.01148}
}
read the original abstract
In this study, we focus on heart murmur classification (HMC) and hypothesize that combining neural audio codec representations (NACRs) such as EnCodec with spectral features (SFs), such as MFCC, will yield superior performance. We believe such fusion will trigger their complementary behavior as NACRs excel at capturing fine-grained acoustic patterns such as rhythm changes, spectral features focus on frequency-domain properties such as harmonic structure, spectral energy distribution crucial for analyzing the complex of heart sounds. To this end, we propose, BAOMI, a novel framework banking on novel bandit-based cross-attention mechanism for effective fusion. Here, a agent provides more weightage to most important heads in multi-head cross-attention mechanism and helps in mitigating the noise. With BAOMI, we report the topmost performance in comparison to individual NACRs, SFs, and baseline fusion techniques and setting new state-of-the-art.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Cardiovascular diseases (CVDs) remain the leading cause of global mortality, claiming millions of lives each year [1]. Early diagnosis is vital to improving patient outcomes; however, tra- ditional diagnostic methods, such as manual auscultation, are dependent on clinician expertise and prone to inconsistencies. Phonocardiograms (PCGs), which...
work page Pith review arXiv 2025
-
[2]
Neural Audio Codec In this section, we discuss the SOTA NACs considered in our study. EnCodec[ 14]: It generates compact, quantized audio represen- tations through a streaming encoder-decoder architecture utiliz- ing RVQ. By incorporating multi-scale spectrogram adversarial losses and Transformer-based entropy coding, EnCodec strikes a balance between com...
-
[3]
Modeling In this section, we explain the SFs considered, the downstream modeling for individual features, and the proposed framework, BAOMIfor fusion of NACRs with SFs. Spectral Features: As SFs, we consider mel-frequency cepstral coefficients (MFCC)6 and linear-frequency cepstral coefficients (LFCC)7. We use the default parameters as given in the library...
-
[4]
Dataset We use CirCor DigiScope dataset [18], available on PhysioNet [19]
Experiments 4.1. Dataset We use CirCor DigiScope dataset [18], available on PhysioNet [19]. We specifically worked with the publicly accessible sub- set containing data from 963 patients. The dataset consists of three class labels: Present, Absent, and Unknown, with a dis- tribution of 179, 695, and 68 samples, respectively, indicating an inherent class i...
-
[5]
Conclusion In this work, we focus on HMC and hypothesize that combining NACRs with SFs will yield superior performance. To this end, we propose BAOMI, a novel framework that employs a bandit- based cross-attention mechanism to effectively fuse NACRs and SFs. By prioritizing the most important attention heads, BAOMI mitigates noise and enhances the fusion ...
-
[6]
World Health Organization, “World health organization,” https: //www.who.int/health-topics/cardiovascular-diseases, 2023, [On- line; accessed 2023-02-15]
work page 2023
-
[7]
Algorithms for automatic analysis and classification of heart sounds–a systematic review,
A. K. Dwivedi, S. A. Imtiaz, and E. Rodríguez-Villegas, “Algorithms for automatic analysis and classification of heart sounds–a systematic review,”IEEE Access, vol. 7, pp. 8316– 8345, 2019. [Online]. Available: https://api.semanticscholar.org/ CorpusID:59231261
work page 2019
-
[8]
The diagnosis of heart diseases based on pcg signals using mfcc coefficients and svm classifier,
B. Farzam and J. Shirazi, “The diagnosis of heart diseases based on pcg signals using mfcc coefficients and svm classifier,”IJISET- International Journal of Innovative Science, Engineering & Tech- nology, vol. 1, no. 10, 2014
work page 2014
Show all 31 references
-
[9]
Y . Duan, C. Yang, Z. Zhao, Y . Jiang, Y . Wang, and Y . Wang,A Comparative Study of Pre-trained Audio and Speech Models for Heart Sound Detection, 02 2024, pp. 287–301
2024
-
[10]
An efficient heart murmur recognition and cardiovascular disorders classification system,
M. S. Ahmad, J. Mir, M. O. Ullah, M. L. U. R. Shahid, and M. A. Syed, “An efficient heart murmur recognition and cardiovascular disorders classification system,”Australasian physical & engineer- ing sciences in medicine, vol. 42, pp. 733–743, 2019
2019
-
[11]
Model-driven heart rate estimation and heart murmur detection based on phonocardiogram,
J. Nie, R. Liu, B. Mahasseni, and V . Mitra, “Model-driven heart rate estimation and heart murmur detection based on phonocardiogram,” in2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), 2024, pp. 1–6
2024
-
[12]
Transformer embedded with learn- able filters for heart murmur detection,
P. Fan, Y . Shu, and Y . Han, “Transformer embedded with learn- able filters for heart murmur detection,” in2022 Computing in Cardiology (CinC), vol. 498, 2022, pp. 1–4
2022
-
[13]
Ensemble transformer-based neural networks detect heart murmur in phonocardiogram recordings,
M. Alkhodari, S. K. Azman, L. J. Hadjileontiadis, and A. H. Khan- doker, “Ensemble transformer-based neural networks detect heart murmur in phonocardiogram recordings,” in2022 Computing in Cardiology (CinC), vol. 498, 2022, pp. 1–4
2022
-
[14]
Heart murmur classification using a capsule neural network,
Y .-T. Tsai, Y .-H. Liu, Z.-W. Zheng, C.-C. Chen, and M.-C. Lin, “Heart murmur classification using a capsule neural network,”Bio- engineering, vol. 10, no. 11, p. 1237, 2023
2023
-
[15]
Intelligent cardiac auscultation for murmur detection via parallel-attentive models with uncertainty estimation,
Z. Zhang, T. Pang, J. Han, and B. W. Schuller, “Intelligent cardiac auscultation for murmur detection via parallel-attentive models with uncertainty estimation,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024,...
2024
-
[16]
Time-domain heart sound classification using neural audio codecs,
A. Mishra, J. Q. Yip, and E. S. Chng, “Time-domain heart sound classification using neural audio codecs,” in2024 11th Interna- tional Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA). IEEE, 2024, pp. 1–5
2024
-
[17]
Towards audio codec-based speech separation,
J. Q. Yip, S. Zhao, D. Ng, E. S. Chng, and B. Ma, “Towards audio codec-based speech separation,” inInterspeech 2024, 2024, pp. 2190–2194
2024
-
[18]
Dasb–discrete audio and speech benchmark,
P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli, “Dasb–discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294, 2024
2024 arXiv
-
[19]
High fidelity neural audio compression,
A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Transactions on Machine Learning Research, 2023, featured Certification, Reproducibility Certification. [Online]. Available: https://openreview.net/forum? id=ivCd8z8zR2
2023
-
[20]
Speechtokenizer: Unified speech tokenizer for speech language models,
X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtokenizer: Unified speech tokenizer for speech language models,” inThe Twelfth International Conference on Learning Representations,
-
[21]
A cross-attention layer coupled with multimodal fusion methods for recognizing depression from spon- taneous speech,
L. Ilias and D. Askounis, “A cross-attention layer coupled with multimodal fusion methods for recognizing depression from spon- taneous speech,” inInterspeech 2024, 2024, pp. 912–916
2024
-
[22]
High-fidelity audio compression with improved rvqgan,
R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Advances in Neural Information Processing Systems, vol. 36, 2024
2024
-
[23]
Snac: Multi- scale neural audio codec,
H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer, “Snac: Multi- scale neural audio codec,” inAudio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, 2024
2024
-
[24]
The circor digiscope dataset: from murmur detection to murmur classifica- tion,
J. Oliveira, F. Renna, P. D. Costa, M. Nogueira, C. Oliveira, C. Fer- reira, A. Jorge, S. Mattos, T. Hatem, T. Tavareset al., “The circor digiscope dataset: from murmur detection to murmur classifica- tion,”IEEE journal of biomedical and health informatics, vol. 26, no. 6, pp....
2021
-
[25]
Physiobank, physiotoolkit, and physionet: Compo- nents of a new research resource for complex physiologic signals,
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley, “Physiobank, physiotoolkit, and physionet: Compo- nents of a new research resource for complex physiologic signals,” Circulation [Onli...
2000
-
[26]
Heart murmur severity stages classifi- cation using multi-kernel residual cnn,
S. Das and S. Dandapat, “Heart murmur severity stages classifi- cation using multi-kernel residual cnn,”IEEE Sensors Journal, 2024
2024
-
[28]
Mul- timodal fusion for vocal biomarkers using vector cross-attention,
V . Despotovic, A. Elbéji, P. V . Nazarov, and G. Fagherazzi, “Mul- timodal fusion for vocal biomarkers using vector cross-attention,” inInterspeech 2024, 2024, pp. 1435–1439
2024
-
[29]
Heart sound classification based on improved mfcc features and convolu- tional recurrent neural networks,
M. Deng, T. Meng, J. Cao, S. Wang, J. Zhang, and H. Fan, “Heart sound classification based on improved mfcc features and convolu- tional recurrent neural networks,”Neural Networks, vol. 130, pp. 22–32, 2020
2020
-
[30]
Heart sound clas- sification based on log mel-frequency spectral coefficients features and convolutional neural networks,
H. Kui, J. Pan, R. Zong, H. Yang, and W. Wang, “Heart sound clas- sification based on log mel-frequency spectral coefficients features and convolutional neural networks,”Biomedical Signal Processing and Control, vol. 69, p. 102893, 2021
2021
-
[31]
Heart sound classification based on improved mel-frequency spectral coefficients and deep residual learning,
F. Li, Z. Zhang, L. Wang, and W. Liu, “Heart sound classification based on improved mel-frequency spectral coefficients and deep residual learning,”Frontiers in Physiology, vol. 13, p. 1084420, 2022
2022
-
[2024]
Available: https://openreview.net/forum?id= AF9Q8Vip84
[Online]. Available: https://openreview.net/forum?id= AF9Q8Vip84
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.