Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Vision Language Models Are Few-Shot Audio Spectrogram Classifiers

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision language models classify environmental sounds by reading spectrogram images, with GPT-4o reaching 59% on ESC-10.

desk verdict A real capability result that is currently undercut by an unaddressed contamination risk and a fitted evaluation protocol. read the letter →

arxiv 2411.12058 v1 pith:J5EIRLII submitted 2024-11-18 cs.SD eess.AS

Satvik Dixit , Laurie M. Heller , Chris Donahue This is my paper · ORCID
classification cs.SDeess.AS
keywords visionlanguagemodelsspectrogramclassificationfew-shotlearningESC-10environmentalsoundin-contextaudiovisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that vision language models (VLMs) can act as few-shot audio classifiers when audio is presented as spectrogram images, without any audio-native input. On the ESC-10 environmental sound dataset, GPT-4o reaches 59.00% cross-validated accuracy after tuning the spectrogram rendering and choosing few-shot examples by clustering; the same setup beats the commercial audio-language model Gemini-1.5 on the equivalent audio task (49.62%) and slightly exceeds ensembled human expert performance on one fold (73.75% vs. 72.50%). The authors propose visual spectrogram classification (VSC) as a new task and a benchmark for VLMs, and point toward using spectrograms to ground audio captions. A reader should care because it suggests visual pattern recognition alone can extract meaningful audio content, and because it reframes audio understanding as an image understanding problem.

What carries the argument

The load-bearing mechanism is the pairing of a deliberately rendered spectrogram image with in-context few-shot prompting. Spectrograms are computed with a 2,048-sample window and 512-sample hop, resampled to 22,050 Hz, log-scaled in frequency and amplitude, and colored with viridis; the paper then shows the model example spectrograms for each class inside the prompt and asks it to name the class of a new image. Example selection matters: choosing spectrograms closest to K-means cluster centroids of mel spectrograms outperforms random or hand-picked examples, and two examples per class gives the best first-fold accuracy (76.25%).

What would settle it

Take a set of environmental sounds that were recorded and published after GPT-4o's training data was collected, render them into spectrograms using the paper's exact pipeline, and run the same 10-shot prompt; if accuracy falls to near chance while accuracy on the original ESC-10 folds stays near 59%, the reported performance is largely memorization rather than spectrogram reasoning.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a VLM prompted with a spectrogram and a handful of labeled example spectrograms can classify environmental sounds at a level comparable to a commercial audio-native model and to human experts. The strongest result is GPT-4o's 59.00% cross-validated accuracy on ESC-10, obtained by searching spectrogram hyperparameters (logarithmic frequency and amplitude, viridis colormap, axis labels, no colorbar) and by selecting few-shot examples near K-means cluster centroids of mel spectrograms. The paper also shows that few-shot examples help far more than zero-shot prompts, and that performance collapses when the class count grows, with 14% accuracy on a 50-class ESC-50 subset.

Load-bearing premise

The load-bearing premise is that GPT-4o's correct answers come from actually reading the spectrogram patterns, not from having memorized ESC-10 spectrogram images or their class labels during pretraining, and the paper performs no contamination check to rule that out.

Editorial extensions

If this is right

  • If the central claim holds, audio classification can be performed by a model that never hears audio, only sees it, making audio tasks available to vision-language pipelines.
  • Visual spectrogram classification becomes a reusable few-shot benchmark for comparing VLM visual reasoning across models.
  • Spectrogram-plus-language prompting could be used to augment or ground audio captions, addressing vague or hallucinated captions in captioning datasets.
  • The sharp accuracy drop from 10 to 50 classes warns that current VLMs are only practical for small-class sound discrimination, not large-scale taxonomy.
  • The near-human expert result on the first fold suggests VSC is a meaningful test of human-level auditory pattern reading, not a trivial image task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result survives a contamination check, it would imply that spectrograms are a sufficiently universal visual language that pretraining on natural images transfers to time-frequency plots; a testable extension would be to render novel sounds recorded after the model's training cutoff and measure whether few-shot accuracy persists.
  • The same prompting recipe could be applied to non-environmental audio, such as music tagging or speech emotion classification, and the clustering-based example selection may transfer as a general strategy for choosing in-context exemplars.
  • The authors' comparison with Gemini-1.5 is not a matched comparison, since Gemini receives raw audio while GPT-4o receives images, so the headline numbers should be read as relative capabilities of two different input modalities, not as a direct model race.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Visual Spectrogram Classification (VSC), a task in which vision-language models classify audio content from spectrogram images, and evaluates six commercial VLMs in zero-shot and few-shot settings on ESC-10. The authors report that GPT-4o achieves 59.00% cross-validated accuracy on ESC-10, outperforming Gemini-1.5 Pro's audio-based accuracy of 49.62%, and that GPT-4o slightly exceeds ensembled human expert accuracy on the first fold (73.75% vs. 72.50%). The paper includes ablations over spectrogram representations and few-shot example-selection strategies, and it proposes VSC as a benchmark and as a route toward audio caption augmentation.

Significance. If the central result survives a contamination check, it would be a notable empirical demonstration that VLMs can extract acoustic content from spectrogram images and compete with an audio-native commercial model. The paper's strengths are its breadth (six VLMs, three human experts, one audio language model), the ablation over spectrogram hyperparameters and few-shot example selection, and the explicit limitations section. The proposed VSC task is reasonable, and the human-expert data are potentially useful. However, the current evidence does not establish that the reported accuracy comes from visual reasoning rather than from memorized training associations, and the statistical support for the headline comparisons is thin.

major comments (4)
  1. [§1, §3.5, §4] The central interpretation assumes GPT-4o classifies spectrograms by visual reasoning rather than by retrieving memorized ESC-10 image-label associations. The authors state in Section 1 that VLMs 'have seen spectrograms and associated text during pre-training,' and ESC-10 is a public dataset from 2015 that could plausibly appear in web-scale training corpora, but no contamination check is reported. The ESC-50 subset result (14%) does not rule out partial memorization of ESC-10. I would need at least one control experiment—for example, classifying spectrograms from newly recorded or private audio of the same classes, classifying frequency-inverted or scrambled versions of ESC-10 spectrograms, or comparing against a simple image-retrieval baseline—before accepting the qualitative claim that the accuracy reflects visual spectrogram understanding.
  2. [§3.3, §3.5, Table 2] The headline 59.00% cross-validated accuracy is produced by a protocol whose hyperparameters and example-selection rules were chosen on the first fold, yet the paper reports no per-fold accuracies, standard errors, or confidence intervals. For example, Table 2 shows 76.25% for '2 per class K-means Mel' on fold 1, but the cross-validated section does not state whether this same configuration was applied to all folds or whether the 59.00% is an average over single runs. The manuscript should specify the exact protocol used for each fold, report all five fold accuracies, and provide a significance test (e.g., a paired test against Gemini-1.5 audio or a binomial confidence interval on the pooled accuracy). Without this, the 59.00% versus 49.62% gap is not statistically grounded.
  3. [§3.5, Table 1] The VLM-versus-ALM comparison is under-specified. Table 1 lists Gemini-1.5 Flash/Pro (Audio) accuracies, but the appendix only documents the VSC prompts, not the audio prompts, the number of audio examples, the exact ESC-10 folds, or the API settings used for the audio-conditioned models. If the comparison is meant to support the abstract's '59.00% vs. 49.62%' claim, the audio classification protocol must be described in the same detail as the VSC protocol, including whether the same few-shot examples and folds were used. Otherwise the gap could be due to prompt design or evaluation differences rather than to the modality.
  4. [§3.4, Abstract] The claim that GPT-4o 'even perform[s] slightly better than human experts' is based on three experts and a single 80-item fold, where the ensembled human accuracy (72.5%) and GPT-4o (73.75%) differ by one correct response. No confidence interval or chance-level baseline for the human task is reported. This statement should be either softened to 'in this small comparison' or supported by additional expert evaluators and/or more folds.
minor comments (4)
  1. [§2.3, §3.3] The terms '10-shot', '20-shot', and '30-shot' refer to total examples (1, 2, or 3 per class), which is potentially confusing; define the notation at first use.
  2. [§3.1, Appendix B] Please report exact model versions (e.g., gpt-4o-2024-05-13), API call parameters, temperature, and any sampling settings, since commercial APIs can change over time and the results are otherwise not reproducible.
  3. [Figure 3] The confusion matrices are small and difficult to read; a larger figure or a table of per-class precision and recall would make the error analysis more useful.
  4. [§4] The statement that VLMs will improve as they 'potentially encounter more spectrograms in their training data' inadvertently underscores the contamination concern; this should be addressed explicitly rather than treated only as a scaling prediction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports measured accuracies from an empirical benchmark, with no derivation chain that reduces a prediction to its inputs.

full rationale

This paper is an empirical benchmark, not a derivation chain. The central claims (GPT-4o 59.00% cross-validated accuracy on ESC-10; 73.75% vs. 72.50% human expert on the first fold) are measured outcomes, and no equation or theorem is used to derive them from the inputs, so no step reduces to its own premise by construction. The only optimization in the paper is the spectrogram hyperparameter ablation and example-selection study run on the first fold; the paper explicitly acknowledges this with the asterisk in Table 1 ('the hyperparameters were tuned on GPT-4o, so it has an advantage over other models'), and the headline cross-validated figure aggregates over all five folds, so it is not logically forced to equal the first-fold tuning criterion. The 'novel task' naming of VSC restates the well-known idea of classifying spectrograms, but this is a labeling/nuance issue rather than an argument that reduces a prediction to an input. The pretraining-contamination concern is a real external-validity threat, but it is an alternative explanation for the measured accuracy, not a circularity; the paper makes no attempt to derive its numbers from prior work by the same authors and contains no load-bearing self-citations. Score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the public ESC-10 benchmark, black-box VLM APIs, and a tuned representation pipeline. The main free parameters are the spectrogram rendering choices and few-shot example selection, all tuned on the first fold. The load-bearing assumptions are no pretraining contamination, generalizability of the tuned protocol, and stability of API outputs. No new physical or mathematical entities are introduced.

free parameters (6)
  • spectrogram frequency axis = linear (vs default log)
    Tuned on first fold: linear frequency achieved 35% zero-shot accuracy vs 27.5% for log default (Appendix Table 4).
  • spectrogram style = raw amplitude STFT
    Ablation: amplitude 35%, mel 25%, MFCC 13.75%; raw amplitude selected as best (Appendix Table 4).
  • few-shot example selection method = K-means on mel spectrograms, k=3
    Chosen after comparing random, hand-picked, K-means on amplitude and mel features; best first-fold accuracy 76.25% (Table 2).
  • examples per class for few-shot = 2 for best result; 1 for human comparison
    Table 2: 2 examples per class gave 76.25%, 1 gave 73.75%, 3 gave 70.00%; the human comparison uses 1 example per class.
  • cluster count k in K-means = 3
    K-means with k=3 is reported; no search over k is shown (Table 2).
  • colormap and resolution = viridis, standard detail
    Default viridis (27.5%) beat magma (25.0%); low resolution dropped accuracy to 20.0% (Appendix Table 4).
assumptions (4)
  • domain assumption ESC-10 spectrogram images contain enough visual information to discriminate the ten classes.
    The task assumes visual patterns in spectrograms are class-discriminative; the paper does not measure an upper bound for this representation.
  • ad hoc to paper VLMs have not memorized ESC-10 spectrogram images or class labels during pretraining.
    The paper acknowledges pretraining includes spectrograms and text (Section 1) but provides no contamination analysis, so measured accuracy may include retrieval.
  • ad hoc to paper Hyperparameters and example-selection rules tuned on the first fold generalize to other folds.
    The cross-validated 59.00% is presented without specifying whether the protocol is re-tuned per fold or fixed; no per-fold results are shown.
  • domain assumption API responses are stable enough for single-run accuracy to be meaningful.
    No temperature, number of repeats, or variance across API calls is reported; commercial VLM outputs are stochastic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision Language Models Are Few-Shot Audio Spectrogram Classifiers." pith.science (2026). https://pith.science/paper/J5EIRLII

@misc{pith2026241112058,
  author       = {Pith},
  title        = {Pith review of: Vision Language Models Are Few-Shot Audio Spectrogram Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5EIRLII}},
  note         = {Machine review of arXiv:2411.12058}
}
read the original abstract

We demonstrate that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images. Specifically, we instruct VLMs to perform audio classification tasks in a few-shot setting by prompting them to classify a spectrogram image given example spectrogram images of each class. By carefully designing the spectrogram image representation and selecting good few-shot examples, we show that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset. Moreover, we demonstrate that VLMs currently outperform the only available commercial audio language model with audio understanding capabilities (Gemini-1.5) on the equivalent audio classification task (59.00% vs. 49.62%), and even perform slightly better than human experts on visual spectrogram classification (73.75% vs. 72.50% on first fold). We envision two potential use cases for these findings: (1) combining the spectrogram and language understanding capabilities of VLMs for audio caption augmentation, and (2) posing visual spectrogram classification as a challenge task for VLMs.

Figures

Figures reproduced from arXiv: 2411.12058 by the authors.

Figure 1
Figure 1. Experimental setup of the visual spectrogram classification task in the few-shot setting [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example audio spectrograms for each class in the ESC-10 dataset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Confusion matrices for GPT-4o (left) and the ensembled human expert predictions (right) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example spectrograms for the same audio using different configurations described in the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The template for prompting the VLM for zero-shot VSC. {image} is replaced by the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The template for prompting the VLM for few-shot VSC. {category-i} and {example-image [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge-Augmented Vision Language Models for Underwater Bioacoustic Spectrogram Analysis

    cs.CV 2025-09 conditional novelty 4.0 of 10

    Progressive knowledge augmentation with VLM-extracted pattern descriptions improves marine mammal spectrogram classification over vanilla VLMs (25.4% vs 13.2%) while keeping models retraining-free.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. Model card, Anthropic, March 2024

    work page 2024

  3. [3]

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Me...

    2024

  4. [4]

    Convolutional recurrent neural networks for music classification, 2016

    Keunwoo Choi, George Fazekas, Mark Sandler, and Kyunghyun Cho. Convolutional recurrent neural networks for music classification, 2016

    work page 2016

  5. [5]

    Pengi: An audio language model for audio tasks, 2024

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks, 2024

    work page 2024

  6. [6]

    A survey on in-context learning, 2024

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024

    2024

  7. [7]

    Clotho: an audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740, 2019

    work page 2020

  8. [8]

    A survey of vision-language pre-trained models

    Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In International Joint Conference on Artificial Intelligence, 2022

    work page 2022

Show all 19 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities, 2024

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities, 2024

    2024

  3. [11]

    Ast: Audio spectrogram transformer, 2021

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer, 2021

    2021

  4. [12]

    Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson. Cnn architectures for large-scale audio classification, 2017

    2017

  5. [13]

    Effectiveness assessment of recent large vision-language models, 2024

    Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, and Fahad Shahbaz Khan. Effectiveness assessment of recent large vision-language models, 2024

    2024

  6. [14]

    Esc: Dataset for environmental sound classification

    Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015

    2015

  7. [15]

    Schmidt, Andreas F

    Jordi Pons, Oriol Nieto, Matthew Prockup, Erik M. Schmidt, Andreas F. Ehmann, and Xavier Serra. End-to-end learning for music audio tagging at scale. ArXiv, abs/1711.02520, 2017

    2017 arXiv

  8. [16]

    Salmonn: Towards generic hearing abilities for large language models, 2024

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models, 2024. 6

    2024

  9. [17]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  10. [18]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  11. [19]

    "" 2 { 3

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 7 Appendix A: Spectrogram hyperparameters ablation study We varied key parameters to generate divers...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.