Audio Atlas: Visualizing and Exploring Audio Datasets

Florian Gr\"otschla; Luca A. Lanzend\"orfer; Roger Wattenhofer; Uzeyir Valizada

REVIEW 3 major objections 4 minor 22 references

Audio Atlas: Visualizing and Exploring Audio Datasets

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Audio Atlas claims that any audio dataset can be visualized as a responsive, searchable two-dimensional map, even at tens of millions of samples.

desk verdict A useful open-source demo for audio dataset exploration, with some overclaimed scale and CLAP generality; judge it as a tool, not a research result. read the letter →

arxiv 2412.00591 v1 pith:UE7F5X7Q submitted 2024-11-30 cs.SD cs.AIeess.AS

Luca A. Lanzendörfer , Florian Grötschla , Uzeyir Valizada , Roger Wattenhofer This is my paper · ORCID

classification cs.SDcs.AIeess.AS

keywords audiovisualizationCLAPembeddingscontrastivelanguage-audiopretrainingt-SNEprojectionsemanticsearchzero-shotclassificationdatasetexplorationinteractivepointcloud

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio Atlas is a web application whose stated purpose is to make any audio dataset explorable as an interactive map. The authors claim that by embedding audio clips with a contrastive audio–text model, projecting the embeddings to two dimensions with t-SNE, and rendering the result with a GPU-accelerated scatterplot engine, the tool stays responsive even at tens of millions of samples. The same embedding space enables two things that normally require labeled metadata: semantic search by text or by an uploaded audio snippet, and zero-shot classification using a user-defined list of classes. The ESC-50 demo shows that class labels form local clusters in the projection, which is the evidence offered that the map reflects meaningful audio content.

What carries the argument

The load-bearing mechanism is the text-audio embedding space produced by CLAP, a contrastive neural network trained on audio-text pairs; it makes 'semantic similarity' operational as ordinary vector distance. t-SNE then compresses those vectors to two dimensions for display. On top of this, a GPU-accelerated point-cloud renderer draws many points incrementally to keep pan and zoom responsive, while a vector database and an approximate nearest-neighbor index answer text and audio queries without scanning the dataset. The interface's zero-shot classification is the same machinery applied in reverse: user-supplied class names are embedded, and each audio point is colored by its nearest class embedding.

What would settle it

Run the same pipeline on a labeled audio dataset from a domain far from CLAP's training distribution, such as medical recordings or animal vocalizations, and measure whether the t-SNE projection separates known classes and whether text queries retrieve the correct labels. If familiar classes scatter and nearest neighbors are semantically unrelated, the claim that Audio Atlas can visualize 'any audio dataset' with meaningful clusters is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that dataset-scale audio exploration needs no bespoke audio feature engineering: a single contrastive audio-text model can supply both the spatial structure and the search index. Audio Atlas treats CLAP embeddings as the ground truth of similarity, projects them with t-SNE into a plane, and stores them in a vector database for nearest-neighbor queries. Users can then inspect clusters, click any point to hear it and see its closest neighbors, filter by zero-shot class labels, or search by describing a sound; because text and audio live in one embedding space, queries in either modality retrieve results by meaning rather than by low-level waveform similarity. The paper's demonstration on ESC-50, where all environmental sound classes fall into local pockets, is the basis for the claim that the clusters visible in the interface are semantically coherent.

Load-bearing premise

The central premise is that CLAP embeddings preserve the semantic distinctions that matter for the audio being visualized; if they do not, the spatial clusters and text-search results will be misleading even though the interface itself works.

Editorial extensions

If this is right

Users can explore unannotated audio collections by typing a description of the sound they want, because text and audio share one embedding space.
Zero-shot classification becomes a live visual operation: choose class names, and the point cloud recolors instantly, so a user can judge whether the classes are separable before trusting the labels.
The tool doubles as a qualitative evaluation surface for embedding models, since a model that has not learned relevant audio structure will produce a map without coherent clusters.
Because new datasets only require new embeddings, the same interface can be pointed at arbitrary audio and music corpora, including collections with no metadata at all.
If the responsive rendering holds at tens of millions of samples, the approach scales past the six bundled datasets to full-scale production audio libraries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper, the design implies a cheap benchmark for embedding quality: on datasets with known labels, compute cluster-label agreement (e.g., adjusted Rand index) directly on the t-SNE projection; the paper reports only the qualitative ESC-50 screenshot.
A testable extension would be to swap the static t-SNE projection for a projection method that supports out-of-sample points, so newly uploaded audio could appear on the map instead of only in a nearest-neighbor list.
The paper does not address how stable the map is across random t-SNE runs; a natural follow-up is measuring whether the visual clusters persist under different seeds and perplexities, since unstable maps would undermine cluster-based browsing.
One could also treat the tool as an active learning interface: the embedding distance plus user clicks on clusters could generate pseudo-labels for training lightweight classifiers, which the paper leaves implicit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a good demo paper for an open-source tool, and it should be judged as a demo, not as a scientific contribution. The tool does what it says on the tin for the included datasets, and the code is public, which earns real credit. The soft spots are real but minor: the 'tens of millions of samples' claim is asserted on the strength of a self-citation rather than demonstrated, and the paper leans entirely on CLAP plus one ESC-50 t-SNE screenshot to promise 'meaningful clusters' for any audio dataset. A short quantitative sanity check (a few retrieval or zero-shot numbers, or a cluster-consistency measure) would tighten this up nicely.

The genuinely new thing here is the specific combination: CLAP embeddings, t-SNE projection, DeepScatter's WebGL rendering, Milvus for vector storage, and Annoy for semantic search, all packaged for audio data with zero-shot labeling built in. That combination doesn't exist in the cited prior work, which is either image-focused or not interactive. As a practical tool, it will likely save time for people who want a quick look at an audio dataset or a qualitative audit of an embedding model. The open-sourcing makes it reproducible and useful even if the paper itself is thin.

The ESC-50 figure does show structure, and for CLAP—a model with its own benchmarks—that's plausible evidence. But the stress-test note is right that t-SNE can produce visually compact clusters for noisy data, so the screenshot alone would not convince me if the paper made a strong scientific claim. It doesn't; it makes a tool claim. Still, the Introduction's 'any audio dataset' and 'tens of millions of samples' are overstatements for what's actually shown. Those are easy fixes: soften the language or add a benchmark.

I'd take this paper as a late-breaking demo without complaint. It's the kind of contribution that belongs at ISMIR as a demo, and it deserves referee time to make sure the claims are bounded. The main thing I'd ask the authors is to state clearly what's verified and what's expected from the underlying components.

Referee Report

3 major / 4 minor

Summary. Audio Atlas is a late-breaking demo from ISMIR 2024 that presents an open-source interactive web application for visualizing and exploring audio datasets. The system computes CLAP embeddings for audio samples, projects them into two dimensions with t-SNE, stores the embeddings in Milvus, and renders the resulting point cloud with DeepScatter. It offers semantic search via text or audio queries, zero-shot classification with user-defined class label lists, and point-by-point inspection of dataset metadata and nearest neighbors. The initial implementation includes MusicCaps, YT8M-MTC, VCTK, ESC-50, MTG-Jamendo, and FMA. The authors claim that Audio Atlas can visualize any audio dataset and remains responsive even with tens of millions of samples, and they demonstrate the interface with screenshots of FMA and ESC-50.

Significance. If the tool works as described, it would be a practical contribution to the MIR community: an open-source, extensible visualization and search interface for audio datasets, combining standard components (CLAP, t-SNE, Milvus, DeepScatter) in a way that is easy to adapt to new data. The codebase is available on GitHub, which is a concrete strength and makes the tool immediately usable and verifiable. The quantitative evidence in the paper, however, is minimal: the central scaling claim is not demonstrated, and the cluster-quality claim rests on one qualitative ESC-50 screenshot. The paper is most valuable as a demo of system integration and usability; its broader claims about semantic clustering and scale need to be substantiated or carefully qualified.

major comments (3)

[Introduction; §2] The claim that Audio Atlas "can visualize any audio dataset, providing a responsive user interface even when displaying tens of millions of samples" is not demonstrated anywhere in the manuscript. Section 2 repeats this: "Audio Atlas remains responsive on large-scale datasets [19]", but reference [19] is the authors' own Disco-10m dataset paper, which does not appear to report an Audio Atlas deployment or latency/memory measurements. Since scalability is a central selling point that distinguishes Audio Atlas from the tools cited in [1–5], the paper should either include a concrete demonstration (e.g., run on Disco-10m with observed responsiveness) or explicitly rephrase this as a design goal rather than a demonstrated capability.
[§3, Figure 3] The only evidence for semantically meaningful clusters is the sentence "the t-SNE projection of the CLAP embeddings for the ESC-50 dataset has clustered all classes into local pockets." This is insufficient: t-SNE often produces seemingly compact clusters even for unstructured high-dimensional data, and a single screenshot does not establish that the visible pockets correspond to the 50 semantic classes. Because the same CLAP embedding space drives the zero-shot classification and semantic search features, a failure of the embedding model for a particular domain would make the tool misleading. I suggest adding a quantitative measure for at least one dataset, such as cluster-label agreement (NMI or ARI), zero-shot classification accuracy, or retrieval precision at a few values of k, or at least comparing the t-SNE clusters with the ground-truth labels in a table.
[Abstract; §2] The paper claims applicability to "any audio dataset" and "audio data" generally, but the only domain validation is the ESC-50 screenshot. CLAP is a model trained on particular audio-text pairs, and its embeddings may not produce meaningful groupings for speech corpora like VCTK or music datasets like MTG-Jamendo. The authors do not discuss potential domain mismatch or limitation regimes despite designing the tool for arbitrary user-uploaded audio. I would either qualify the claim to datasets whose content is similar to CLAP's training distribution, or provide a brief evaluation on at least two diverse datasets from the six listed to show the visualization and search remain semantically meaningful.

minor comments (4)

[Title] The running header shows "AUDIO A TLAS: VISUALIZING AND EXPLORING AUDIO DA TASETS", with odd spacing in "ATLAS" and "DATASETS"; this appears to be a rendering artifact and should be corrected.
[§2] The phrase "with the provided search bar in Audio Atlas" is redundant; consider simplifying to "with the search bar in Audio Atlas".
[References] Reference [7] (Wu et al., the CLAP model) lacks version or venue details; since CLAP has multiple releases, please specify the exact model checkpoint used.
[§1] The sentence "These tools are mostly unsuitable for machine learning projects" would benefit from a concrete reason (e.g., lack of programmatic access or inability to load custom data) so the reader can see the design gap.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the tool is an engineering demo built on external CLAP embeddings, with only one minor non-load-bearing self-citation.

full rationale

Audio Atlas does not fit parameters to data and then rename those fits as predictions; its central artifact is a working visualization and search system assembled from stated external components (CLAP, t-SNE, Milvus, DeepScatter). The semantic-search and zero-classification behaviors are by design nearest-neighbor operations in the CLAP embedding space, not derived results that reduce to the tool's own inputs. The only self-citation, [19] (Disco-10M), supports the claim 'Audio Atlas remains responsive on large-scale datasets' in Section 2. That is a minor engineering-support citation, not the load-bearing definition of any main result, and the present paper's contribution does not collapse into it. The assumption that CLAP embeddings are semantically meaningful is an external model assumption and a possible correctness/robustness risk, but it is not a circular step: the paper does not define CLAP's quality in terms of its own visualization. A single qualitative ESC-50 screenshot is weak validation, but weak evidence is not circularity. The paper therefore merits a low circularity score.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No fitted parameters or invented entities. The central claim rests on the external CLAP model and on the scaling capabilities of DeepScatter and Milvus, both treated as reliable inputs.

assumptions (2)

domain assumption CLAP embeddings provide semantically meaningful audio-text similarity for arbitrary audio inputs.
The tool's core functionality (clustering, semantic search, zero-shot classification) assumes CLAP generalizes across domains; stated in Section 2.
domain assumption DeepScatter and Milvus can render and query tens of millions of projected points responsively.
The scaling claim in the Introduction and the citation [19] to Disco-10M assume the underlying infrastructure handles this scale.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Audio Atlas: Visualizing and Exploring Audio Datasets." pith.science (2026). https://pith.science/paper/UE7F5X7Q

@misc{pith2026241200591,
  author       = {Pith},
  title        = {Pith review of: Audio Atlas: Visualizing and Exploring Audio Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE7F5X7Q}},
  note         = {Machine review of arXiv:2412.00591}
}

read the original abstract

We introduce Audio Atlas, an interactive web application for visualizing audio data using text-audio embeddings. Audio Atlas is designed to facilitate the exploration and analysis of audio datasets using a contrastive embedding model and a vector database for efficient data management and semantic search. The system maps audio embeddings into a two-dimensional space and leverages DeepScatter for dynamic visualization. Designed for extensibility, Audio Atlas allows easy integration of new datasets, enabling users to better understand their audio data and identify both patterns and outliers. We open-source the codebase of Audio Atlas, and provide an initial implementation containing various audio and music datasets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages

[19]

Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),

J. Yamagishi, C. Veaux, and K. MacDonald, “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID: 213060286

work page 2019
[1]

Audio Atlas: Visualiz- ing and Exploring Audio Datasets

INTRODUCTION The increasing size of machine learning datasets presents significant challenges in data visualization and analysis. Traditional tools are often insufficient for effectively man- aging and interpreting unlabeled audio datasets at scale. Having the ability to visualize large-scale datasets is cru- cial in helping to understand the structure an...

work page 2024
[2]

The application leverages the Contrastive Language-Audio Pretraining (CLAP) [7] model to gener- ate embeddings that fuse audio and text into a shared vector space

AUDIO A TLAS Audio Atlas is a visualization tool designed to help users interact with audio data through an intuitive and dynamic web interface. The application leverages the Contrastive Language-Audio Pretraining (CLAP) [7] model to gener- ate embeddings that fuse audio and text into a shared vector space. These embeddings are then projected onto a two- ...

arXiv 2024
[3]

Audio Atlas makes it easy to browse an audio dataset using text and audio queries

USE CASES We demonstrate a series of practical applications of Audio Atlas, showing its utility in audio analysis through dimen- sionality reduction and search techniques. Audio Atlas makes it easy to browse an audio dataset using text and audio queries. We can therefore also qual- itatively assess the classification capabilities of the con- trastive embe...

work page
[4]

Every noise at once,

G. McDonald, “Every noise at once,” http://everynoise. com, 2023, accessed: 2024

work page 2023
[5]

Musicmap,

Kwinten Crauwels, “Musicmap,” https://www. musicmap.info/, 2022, accessed: 2024

work page 2022
[6]

The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale,

D. Bogdanov, A. Porter, H. Schreiber, J. Urbano, and S. Oramas, “The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale,” in Proceedings of the 20th Conference of the In- ternational Society for Music Information Retrieval (ISMIR 2019): 2019 Nov 4-8; Delft, The Nether- lands.[Canada]: ISMIR; 2019. International Society f...

work page 2019
[7]

The eternal jukebox,

Paul Lamere, “The eternal jukebox,” https: //eternalbox.dev/, 2012, accessed: 2024

work page 2012

Show all 22 references

[8]

Embedding projector: Inter- active visualization and interpretation of embeddings,

D. Smilkov, N. Thorat, C. Nicholson, E. Reif, F. B. Vié- gas, and M. Wattenberg, “Embedding projector: Inter- active visualization and interpretation of embeddings,” arXiv preprint arXiv:1611.05469, 2016

2016 arXiv
[9]

Aeye: A visualization tool for image datasets,

F. Grötschla, L. A. Lanzendörfer, M. Calzavara, and R. Wattenhofer, “Aeye: A visualization tool for image datasets,” 2024. [Online]. Available: https: //arxiv.org/abs/2408.04072

2024 arXiv
[10]

Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

Y . Wu, K. Chen, T. Zhang, Y . Hui, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” 2024

2024
[11]

Fma: A dataset for music analysis,

M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,” 2017. [Online]. Available: https://arxiv.org/abs/1612.01840

2017 arXiv
[12]

Visualizing data using t-sne,

L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research , vol. 9, no. 86, pp. 2579–2605, 2008. [Online]. Available: http://jmlr.org/papers/v9/vandermaaten08a. html

2008
[13]

Milvus: A purpose-built vector data management system,

J. Wang, X. Yi, R. Guo, H. Jin, P. Xu, S. Li, X. Wang, X. Guo, C. Li, X. Xu, K. Yu, Y . Yuan, Y . Zou, J. Long, Y . Cai, Z. Li, Z. Zhang, Y . Mo, J. Gu, R. Jiang, Y . Wei, and C. Xie, “Milvus: A purpose-built vector data management system,” in Proceedings of the 2021 Internati...

2021
[14]

Deepscatter,

Nomic AI, “Deepscatter,” https://github.com/nomic-ai/ deepscatter, 2022, accessed: 2024

2022
[15]

Khronos Group, “Webgl,” http://www.khronos.org/ webgl/, 2014, accessed: 2024

2014
[16]

React: A javascript library for building user interfaces,

Meta Open Source, “React: A javascript library for building user interfaces,” https://reactjs.org/, 2013, ac- cessed: 2024

2013
[17]

Musiclm: Generating music from text,

A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “Musiclm: Generating music from text,” 2023

2023
[18]

Language-guided music recommendation for video via prompt analogies,

D. McKee, J. Salamon, J. Sivic, and B. Rus- sell, “Language-guided music recommendation for video via prompt analogies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2023, pp. 14 784–14 793

2023
[20]

ESC: Dataset for Environmental Sound Classification,

K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in Proceedings of the 23rd Annual ACM Conference on Multimedia . ACM Press, 2015, pp. 1015–1018. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2733373.2806390

2015
[21]

The mtg-jamendo dataset for automatic music tagging,

D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic music tagging,” in Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019) , Long Beach, CA, United States, 2019. [Online]. Availa...

2019
[22]

Disco-10m: a large-scale music dataset,

L. A. Lanzendörfer, F. Grötschla, E. Funke, and R. Wattenhofer, “Disco-10m: a large-scale music dataset,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [19]

Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),

J. Yamagishi, C. Veaux, and K. MacDonald, “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID: 213060286

work page 2019

[2] [1]

Audio Atlas: Visualiz- ing and Exploring Audio Datasets

INTRODUCTION The increasing size of machine learning datasets presents significant challenges in data visualization and analysis. Traditional tools are often insufficient for effectively man- aging and interpreting unlabeled audio datasets at scale. Having the ability to visualize large-scale datasets is cru- cial in helping to understand the structure an...

work page 2024

[3] [2]

The application leverages the Contrastive Language-Audio Pretraining (CLAP) [7] model to gener- ate embeddings that fuse audio and text into a shared vector space

AUDIO A TLAS Audio Atlas is a visualization tool designed to help users interact with audio data through an intuitive and dynamic web interface. The application leverages the Contrastive Language-Audio Pretraining (CLAP) [7] model to gener- ate embeddings that fuse audio and text into a shared vector space. These embeddings are then projected onto a two- ...

arXiv 2024

[4] [3]

Audio Atlas makes it easy to browse an audio dataset using text and audio queries

USE CASES We demonstrate a series of practical applications of Audio Atlas, showing its utility in audio analysis through dimen- sionality reduction and search techniques. Audio Atlas makes it easy to browse an audio dataset using text and audio queries. We can therefore also qual- itatively assess the classification capabilities of the con- trastive embe...

work page

[5] [4]

Every noise at once,

G. McDonald, “Every noise at once,” http://everynoise. com, 2023, accessed: 2024

work page 2023

[6] [5]

Musicmap,

Kwinten Crauwels, “Musicmap,” https://www. musicmap.info/, 2022, accessed: 2024

work page 2022

[7] [6]

The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale,

D. Bogdanov, A. Porter, H. Schreiber, J. Urbano, and S. Oramas, “The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale,” in Proceedings of the 20th Conference of the In- ternational Society for Music Information Retrieval (ISMIR 2019): 2019 Nov 4-8; Delft, The Nether- lands.[Canada]: ISMIR; 2019. International Society f...

work page 2019

[8] [7]

The eternal jukebox,

Paul Lamere, “The eternal jukebox,” https: //eternalbox.dev/, 2012, accessed: 2024

work page 2012