REVIEW 4 major objections 4 minor 28 references
Online incremental learning for audio classification using a pretrained audio model
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A frozen pretrained audio model can keep learning new sound classes and new recording domains online, in a single forward pass per task, with minimal forgetting.
desk verdict A clean, honest RanPAC adaptation to audio; the key weakness is Q chosen on the test tasks and the absence of a RanPAC baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the RanPAC recipe adapted to audio: a random nonlinear expansion followed by a ridge-regression readout. Frozen 2048-dimensional pretrained embeddings are projected by a fixed random matrix W into Q = 8192 dimensions, passed through ReLU, and accumulated into a Q×Q Gram matrix and a Q×C class-prototype matrix; inference uses Wo = (G + λI)^−1 K. The random expansion improves linear separability of classes, while the ridge inversion decorrelates the class prototypes so that updating the readout on new tasks does not overwrite the classification structure of old tasks.
What would settle it
Run the identical online protocol on a held-out sequence of sound classes or acoustic domains that are semantically far from AudioSet, such as medical or marine sounds, keeping Q = 8192 and the per-task λ search unchanged. If single-pass accuracy drops to the level of a plain linear probe, the claim that the random expansion and decorrelation are what prevent forgetting would be refuted.
Extended reading notes
Core claim
The paper demonstrates that a fixed pretrained audio encoder can be turned into an online continual learner without any base-task training. Features from the frozen PANNs CNN14 model are projected through a random matrix into 8192 dimensions, passed through ReLU, and accumulated into a Gram matrix G and an unnormalized class-prototype matrix K; the classifier weights are then computed as Wo = (G + λI)^−1 K, with λ chosen per task on a validation split. This single-pass update yields 93.4% final average accuracy with 2.5% forgetting on the 50-class ESC-50 class-incremental setup, and 61.4% accuracy with 2.0% forgetting on the 9-city TAU Urban Acoustic Scenes 2019 domain-incremental setup. The
Load-bearing premise
The whole method bets that PANNs CNN14's frozen 2048-dimensional embeddings remain informative for any sound class and domain that arrives later, and that the projection size and ridge penalty chosen on the same benchmark tasks are not silently inflating the reported accuracy.
Editorial extensions
If this is right
- One unified online framework handles both class-incremental and domain-incremental audio learning without task IDs, replay buffers, or base-task training.
- Adapting to a new task requires only a single forward pass through its training samples, so deployment on streaming or resource-constrained devices becomes practical.
- The reported accuracy exceeds even a joint linear probe that trains on all tasks together, suggesting that the nonlinear random expansion plus decorrelation recovers useful structure from frozen features.
- Forgetting stays around 2 percentage points in both setups, indicating that stability-plasticity balance is achieved without explicit memory of past data.
- Trainable parameters are limited to the Q×C readout plus the accumulated Gram matrix, far fewer than the 80.8M parameters of the frozen CNN14 backbone.
Reading between the lines
- The method's success depends on how well PANNs CNN14 embeddings separate sound classes that were not part of its training distribution; a natural test is to swap in other pretrained audio encoders and see whether accuracy tracks the quality of their embeddings.
- The authors chose Q = 8192 and the per-task λ search using validation splits on the same ESC-50 and TAU tasks; a stricter evaluation would fix these hyperparameters on a held-out sequence of new tasks to confirm the numbers are not inflated by benchmark-specific tuning.
- Because the readout is linear in random ReLU features, the method is essentially a kernel-style regressor; explicit kernel features such as random Fourier features might offer a principled way to tune the expansion dimension.
- The framework could plausibly be combined with a small memory buffer for settings where class distributions shift more abruptly, though the paper does not test that combination.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an online incremental learning method for audio classification using a frozen PANNs CNN14 model. A random ReLU projection expands the 2048-dimensional embeddings to Q=8192, and ridge regression is applied to accumulated Gram and class-sum matrices (Eqs. 1-3). The method is evaluated on class-incremental learning with ESC-50 (93.4% final average accuracy) and domain-incremental learning with TAU Urban Acoustic Scenes 2019 (61.4%), and is compared to linear-probe, joint-linear-probe, and NCM baselines. The authors conclude the approach is suitable for both CIL and DIL and outperforms all investigated methods.
Significance. If the reported results hold, the paper offers a simple, base-task-free continual audio learner that leverages a strong pretrained representation and requires no iterative training. The ablation study in Figs. 3-4 provides useful evidence that the nonlinear random projection is the key component. The method is reproducible from the public PANNs model and standard datasets. However, the evaluation protocol and the absence of a direct comparison to RanPAC, the method on which it explicitly relies, currently prevent the broad superiority claim from being accepted at face value.
major comments (4)
- [§3.2, Fig. 4] The projection dimension Q=8192 is selected via ablation on exactly the same ESC-50 and TAU tasks that are later used to report the final accuracies. Because Fig. 4 shows average accuracy after each task (i.e., on the test splits), the headline numbers 93.4% and 61.4% are selected estimates, not unbiased evaluations of a pre-specified configuration. A held-out-task protocol or nested selection is needed; at minimum, the numbers should be labeled as having Q selected on the test tasks.
- [§2.2, Eq. (3)] The paper claims adaptation 'in a single forward pass' (Abstract and §2.2). However, the λ-selection procedure requires updating G and K for 17 candidate λ values on a random 80% subset of the current task, each involving the Q×Q inverse in Eq. (3), followed by a final update over the full task. This is not a single forward pass. If λ tuning is intended as separate hyperparameter selection, that should be stated explicitly and its computational cost quantified.
- [§2, §4, Table 1] The method is described as 'relies on RanPAC [24]', yet RanPAC is never run on the audio tasks, and no comparison to prior audio continual-learning methods (e.g., [4], [7], [10]) is provided. The conclusion that the method 'outperforms all other investigated methods' therefore only covers the simple baselines implemented in this paper. A direct RanPAC comparison under the same CIL/DIL protocols is necessary to support the claimed advantage.
- [Table 1] The advantage over the offline joint linear probe is 1.9 percentage points for CIL and 1.1 for DIL, with only n=5 runs. Given the small number of runs and the overlapping standard deviations (DIL: 61.4±0.1 vs 60.3±0.7), statistical significance testing or confidence intervals should be provided before making the 'outperforms' claim.
minor comments (4)
- [Abstract and §4] The wording 'outperforms all other investigated methods' should be qualified as 'baselines implemented in this paper' to avoid overclaiming.
- [Eq. (2)] K is defined as unnormalized class sums ('CP for class y without averaging'). In balanced datasets this is harmless, but for imbalanced tasks the lack of normalization can bias Wo. This design choice should be justified or at least noted.
- [§3.1] For the NCM baseline it is not stated whether cosine similarity is computed on the original 2048-dimensional embeddings or on the projected features. Please clarify, since Figs. 3-4 suggest the projection is central to the proposed method.
- [§3.2] The term 'online' is used to mean a single pass over the task data rather than a streaming/sample-by-sample setting; consider making this explicit in the definitions.
Circularity Check
No circularity: the method is an empirical ridge-regression application with openly chosen hyperparameters.
full rationale
The paper's derivation chain is a direct application of ridge regression on random ReLU projections of frozen pretrained embeddings (Eqs. 1–3). The reported accuracies are evaluations, not predictions derived from the method's own outputs. Q is chosen from an ablation study on the same tasks, but this is transparent hyperparameter selection, not a fitted input renamed as a prediction; λ is selected on a validation split. The method explicitly relies on RanPAC [24], an external prior work, and self-citations concern related baselines rather than supporting the central claim. No step reduces an output to an input by definition or by self-citation.
Assumptions & free parameters
free parameters (2)
- Projection dimension Q =
8192
- Ridge regularization lambda =
per task from grid 1e-8 to 1e8
assumptions (4)
- domain assumption PANNs CNN14 embeddings are discriminative and stable enough for classes and domains not seen in pretraining or in prior tasks.
- domain assumption Random projection to higher dimension with ReLU preserves and enhances linear separability of classes.
- standard math Ridge regression with unnormalized sums K and Gram matrix G provides a valid classifier, and lambda selected on validation transfers to test.
- standard math Random projection matrix W drawn from an unspecified distribution is adequate.
Cite this review
Pith. "Pith review of Online incremental learning for audio classification using a pretrained audio model." pith.science (2026). https://pith.science/paper/W463TRFX
@misc{pith2026250820732,
author = {Pith},
title = {Pith review of: Online incremental learning for audio classification using a pretrained audio model},
year = {2026},
howpublished = {\url{https://pith.science/paper/W463TRFX}},
note = {Machine review of arXiv:2508.20732}
}
read the original abstract
Incremental learning aims to learn new tasks sequentially without forgetting the previously learned ones. Most of the existing incremental learning methods for audio focus on training the model from scratch on the initial task, and the same model is used to learn upcoming incremental tasks. The model is trained for several iterations to adapt to each new task, using some specific approaches to reduce the forgetting of old tasks. In this work, we propose a method for using generalizable audio embeddings produced by a pre-trained model to develop an online incremental learner that solves sequential audio classification tasks over time. Specifically, we inject a layer with a nonlinear activation function between the pre-trained model's audio embeddings and the classifier; this layer expands the dimensionality of the embeddings and effectively captures the distinct characteristics of sound classes. Our method adapts the model in a single forward pass (online) through the training samples of any task, with minimal forgetting of old tasks. We demonstrate the performance of the proposed method in two incremental learning setups: one class-incremental learning using ESC-50 and one domain-incremental learning of different cities from the TAU Urban Acoustic Scenes 2019 dataset; for both cases, the proposed approach outperforms other methods.
Reference graph
Works this paper leans on
-
[24]
Ranpac: Random projections and pre-trained models for continual learning,
M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. Van den Hengel, “Ranpac: Random projections and pre-trained models for continual learning,” in NeurIPS, 2023
work page 2023
-
[4]
Class-incremental learning for multi- label audio classification,
M. Mulimani and A. Mesaros, “Class-incremental learning for multi- label audio classification,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 916–920
work page 2024
-
[7]
Online domain-incremental learning approach to classify acoustic scenes in all locations,
M. Mulimani and A. Mesaros, “Online domain-incremental learning approach to classify acoustic scenes in all locations,” in European Signal Processing Conference (EUSIPCO). IEEE, 2024, pp. 96–100
work page 2024
-
[10]
Domain-incremental learning for audio classification,
M. Mulimani and A. Mesaros, “Domain-incremental learning for audio classification,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2025, pp. 1–5
work page 2025
-
[1]
Continual lifelong learning with neural networks: A review,
G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural networks, vol. 113, pp. 54–71, 2019
work page 2019
-
[2]
Adapter incremental continual learning of efficient audio spectrogram transformers,
N. Muthuchamy Selvaraj, X. Guo, A. Kong, B. Shen, and A. Kot, “Adapter incremental continual learning of efficient audio spectrogram transformers,” in INTERSPEECH, 2023, pp. 909–913
work page 2023
-
[3]
S. Karam, S.-J. Ruan, and Q. M. ul Haq, “Task incremental learning with static memory for audio classification without catastrophic interference,” IEEE Consumer Electronics Magazine, vol. 11, no. 5, pp. 101–108, 2022
work page 2022
-
[5]
Learning representations for new sound classes with continual self-supervised learning,
Z. Wang, C. Subakan, X. Jiang, J. Wu, E. Tzinis, M. Ravanelli, and P. Smaragdis, “Learning representations for new sound classes with continual self-supervised learning,” IEEE Signal Processing Letters , vol. 29, pp. 2607–2611, 2022
work page 2022
Show all 28 references
-
[6]
Continual learning of new sound classes using generative replay,
Z. Wang, C. Subakan, E. Tzinis, P. Smaragdis, and L. Charlin, “Continual learning of new sound classes using generative replay,” in 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019, pp. 308–312
2019
-
[8]
Acoustic scene classification across multiple devices through incremental learning of device-specific domains,
——, “Acoustic scene classification across multiple devices through incremental learning of device-specific domains,” in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE) , 2024, pp. 81–85
2024
-
[9]
Three scenarios for continual learning,
G. M. Van de Ven and A. S. Tolias, “Three scenarios for continual learning,” in Continual Learning Workshop NeurIPS , 2018
2018
-
[11]
A closer look at class-incremental learning for multi-label audio classification,
——, “A closer look at class-incremental learning for multi-label audio classification,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 1293–1306, 2025
2025
-
[12]
PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,
Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020
2020
-
[13]
Ast: Audio spectrogram transformer,
Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” in INTERSPEECH, 2021, pp. 571–575
2021
-
[14]
Efficient training of audio transformers with patchout,
K. Koutini, J. Schl ¨uter, H. Eghbal-zadeh, and G. Widmer, “Efficient training of audio transformers with patchout,” in INTERSPEECH, 2022, pp. 2753–2757
2022
-
[15]
SSAST: Self-supervised audio spectrogram transformer,
Y . Gong, C.-I. Lai, Y .-A. Chung, and J. Glass, “SSAST: Self-supervised audio spectrogram transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 10, 2022, pp. 10 699–10 709
2022
-
[16]
Enhancing acoustic scene classification with layer-wise fine-tuning on the SSAST model,
S. Hao, D. Saito, and N. Minematsu, “Enhancing acoustic scene classification with layer-wise fine-tuning on the SSAST model,” in Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) . IEEE, 2024, pp. 1–6
2024
-
[17]
CED: Consistent ensemble distillation for audio tagging,
H. Dinkel, Y . Wang, Z. Yan, J. Zhang, and Y . Wang, “CED: Consistent ensemble distillation for audio tagging,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 291–295
2024
-
[18]
Fine-tune the pretrained atst model for sound event detection,
N. Shao, X. Li, and X. Li, “Fine-tune the pretrained atst model for sound event detection,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 911–915
2024
-
[19]
Few- shot continual learning for audio classification,
Y . Wang, N. J. Bryan, M. Cartwright, J. P. Bello, and J. Salamon, “Few- shot continual learning for audio classification,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 321–325
2021
-
[20]
Few-shot class-incremental audio classification using dynamically expanded classifier with self- attention modified prototypes,
Y . Li, W. Cao, W. Xie, J. Li, and E. Benetos, “Few-shot class-incremental audio classification using dynamically expanded classifier with self- attention modified prototypes,” IEEE Transactions on Multimedia, vol. 26, pp. 1346–1360, 2023
2023
-
[21]
Few-shot class-incremental audio classification with adaptive mitigation of forgetting and overfitting,
Y . Li, J. Li, Y . Si, J. Tan, and Q. He, “Few-shot class-incremental audio classification with adaptive mitigation of forgetting and overfitting,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 32, pp. 2297–2311, 2024
2024
-
[22]
Understanding intermediate layers using linear classifier probes,
G. Alain and Y . Bengio, “Understanding intermediate layers using linear classifier probes,” arXiv preprint arXiv:1610.01644 , 2016
2016 arXiv
-
[23]
A simple episodic linear probe improves visual recognition in the wild,
Y . Liang, L. Zhu, X. Wang, and Y . Yang, “A simple episodic linear probe improves visual recognition in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 9559–9569
2022
-
[25]
ESC: Dataset for Environmental Sound Classification,
K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in Proceedings of the 23rd Annual ACM Conference on Multimedia , 2015, pp. 1015–1018
2015
-
[26]
Environmental sound classification with convolutional neural networks,
——, “Environmental sound classification with convolutional neural networks,” in 25th IEEE International Workshop on Machine Learning for Signal Processing (MLSP) . IEEE, 2015, pp. 1–6
2015
-
[27]
A multi-device dataset for urban acoustic scene classification,
A. Mesaros, T. Heittola, and T. Virtanen, “A multi-device dataset for urban acoustic scene classification,” in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE) , 2018, pp. 9–13
2018
-
[28]
SGDR: Stochastic gradient descent with warm restarts,
I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in International Conference on Learning Representations (ICLR), 2017
2017
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.