{"id":"269a6f17-7143-4174-84ed-97164e909626","arxiv_id":"2412.00591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Audio Atlas visualizes audio datasets as interactive scatter plots of CLAP embeddings, enabling semantic search and zero-shot classification.","lead":"Audio Atlas is an open-source web tool that turns audio datasets into interactive 2D maps using text-aware audio embeddings. It lets researchers browse, search, and zero-shot classify large audio collections without labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLAP embedding quality is asserted rather than validated; t-SNE screenshots alone do not establish semantically meaningful clusters or reliable search.","rationale":"The reader's weakest assumption is the semantic validity of CLAP embeddings, and I agree. The tool's entire value proposition is that visual clusters and search results reflect meaningful audio content. Without validation, the visualization is just a projection of arbitrary vectors, and the zero-shot classification is unverified. The scale claim (tens of millions of samples) is also unsupported—it cites the Disco-10M dataset paper [19] rather than a benchmark of Audio Atlas itself—but it is secondary because the architecture could in principle be stress-tested, and even a slower tool would still be useful at moderate scale. The embedding-quality issue, by contrast, undermines the tool for every dataset and every user. I would keep the reader's CONDITIONAL verdict: the open-source code and demonstrable interface justify conditional acceptance, but the paper should soften the 'any audio dataset' claim or add a validation experiment. My proposed test is directly implementable from the public repository and would settle whether the concern lands.","tokens_in":3725,"tokens_out":4577,"duration_ms":43304,"concrete_test":"Using the same CLAP checkpoint and text prompts that Audio Atlas ships with, compute leave-one-out k-NN classification accuracy (k=10) on the CLAP embeddings of ESC-50's 2,000 labeled samples, and repeat for MTG-Jamendo using its tag labels. Compare against a non-semantic baseline such as MFCC features. Additionally, issue text queries from each ESC-50 class name and compute mean precision@5 for retrieval. If k-NN accuracy is near chance or the margin over MFCC is small, the semantic-cluster claim fails; if accuracy is high, the tool's core assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §2, the authors state, 'To obtain semantically meaningful embeddings, we use CLAP,' and build the tool's cluster visualization, zero-shot labeling, and semantic search entirely on this premise. The only support offered is Fig. 3, where the t-SNE projection of ESC-50 'has clustered all classes into local pockets.' This is weak evidence: t-SNE often produces visually compact clusters even for unstructured high-dimensional data, and a single qualitative screenshot does not establish that the observed pockets correspond to the dataset's semantic categories. No quantitative measure (e.g., retrieval precision, cluster-label agreement, or zero-shot classification accuracy) is reported for any of the six datasets. Because the same CLAP space drives both the visualization and the search, a failure of the embedding model for a particular domain (music tags in MTG-Jamendo, speech in VCTK, or user-uploaded audio) would make the tool actively misleading rather than merely unhelpful. The paper also does not discuss limitation regimes for CLAP, despite claiming applicability to 'any audio dataset' in the Introduction. Thus the central utility claim rests on an unvalidated external model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Audio Atlas is a late-breaking demo from ISMIR 2024 that presents an open-source interactive web application for visualizing and exploring audio datasets. The system computes CLAP embeddings for audio samples, projects them into two dimensions with t-SNE, stores the embeddings in Milvus, and renders the resulting point cloud with DeepScatter. It offers semantic search via text or audio queries, zero-shot classification with user-defined class label lists, and point-by-point inspection of dataset metadata and nearest neighbors. The initial implementation includes MusicCaps, YT8M-MTC, VCTK, ESC-50, MTG-Jamendo, and FMA. The authors claim that Audio Atlas can visualize any audio dataset and remains responsive even with tens of millions of samples, and they demonstrate the interface with screenshots of FMA and ESC-50.","tokens_in":4038,"tokens_out":3226,"duration_ms":32325,"significance":"If the tool works as described, it would be a practical contribution to the MIR community: an open-source, extensible visualization and search interface for audio datasets, combining standard components (CLAP, t-SNE, Milvus, DeepScatter) in a way that is easy to adapt to new data. The codebase is available on GitHub, which is a concrete strength and makes the tool immediately usable and verifiable. The quantitative evidence in the paper, however, is minimal: the central scaling claim is not demonstrated, and the cluster-quality claim rests on one qualitative ESC-50 screenshot. The paper is most valuable as a demo of system integration and usability; its broader claims about semantic clustering and scale need to be substantiated or carefully qualified.","major_comments":[{"comment":"The claim that Audio Atlas \"can visualize any audio dataset, providing a responsive user interface even when displaying tens of millions of samples\" is not demonstrated anywhere in the manuscript. Section 2 repeats this: \"Audio Atlas remains responsive on large-scale datasets [19]\", but reference [19] is the authors' own Disco-10m dataset paper, which does not appear to report an Audio Atlas deployment or latency/memory measurements. Since scalability is a central selling point that distinguishes Audio Atlas from the tools cited in [1–5], the paper should either include a concrete demonstration (e.g., run on Disco-10m with observed responsiveness) or explicitly rephrase this as a design goal rather than a demonstrated capability.","section":"Introduction; §2"},{"comment":"The only evidence for semantically meaningful clusters is the sentence \"the t-SNE projection of the CLAP embeddings for the ESC-50 dataset has clustered all classes into local pockets.\" This is insufficient: t-SNE often produces seemingly compact clusters even for unstructured high-dimensional data, and a single screenshot does not establish that the visible pockets correspond to the 50 semantic classes. Because the same CLAP embedding space drives the zero-shot classification and semantic search features, a failure of the embedding model for a particular domain would make the tool misleading. I suggest adding a quantitative measure for at least one dataset, such as cluster-label agreement (NMI or ARI), zero-shot classification accuracy, or retrieval precision at a few values of k, or at least comparing the t-SNE clusters with the ground-truth labels in a table.","section":"§3, Figure 3"},{"comment":"The paper claims applicability to \"any audio dataset\" and \"audio data\" generally, but the only domain validation is the ESC-50 screenshot. CLAP is a model trained on particular audio-text pairs, and its embeddings may not produce meaningful groupings for speech corpora like VCTK or music datasets like MTG-Jamendo. The authors do not discuss potential domain mismatch or limitation regimes despite designing the tool for arbitrary user-uploaded audio. I would either qualify the claim to datasets whose content is similar to CLAP's training distribution, or provide a brief evaluation on at least two diverse datasets from the six listed to show the visualization and search remain semantically meaningful.","section":"Abstract; §2"}],"minor_comments":[{"comment":"The running header shows \"AUDIO A TLAS: VISUALIZING AND EXPLORING AUDIO DA TASETS\", with odd spacing in \"ATLAS\" and \"DATASETS\"; this appears to be a rendering artifact and should be corrected.","section":"Title"},{"comment":"The phrase \"with the provided search bar in Audio Atlas\" is redundant; consider simplifying to \"with the search bar in Audio Atlas\".","section":"§2"},{"comment":"Reference [7] (Wu et al., the CLAP model) lacks version or venue details; since CLAP has multiple releases, please specify the exact model checkpoint used.","section":"References"},{"comment":"The sentence \"These tools are mostly unsuitable for machine learning projects\" would benefit from a concrete reason (e.g., lack of programmatic access or inability to load custom data) so the reader can see the design gap.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"This is a late-breaking demo abstract, so the evaluation bar is naturally different from a full research paper. The open-source implementation is a genuine strength, and the system appears well-constructed from standard components. The main risk is overclaiming: the unsupported 'tens of millions of samples' responsiveness and the single-screenshot evidence for semantic clustering could be fixed with modest additions. If the authors add even a small quantitative evaluation and carefully qualify the 'any audio dataset' claim, I would be happy to see it accepted; in its current form the central claims outpace the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a good demo paper for an open-source tool, and it should be judged as a demo, not as a scientific contribution. The tool does what it says on the tin for the included datasets, and the code is public, which earns real credit. The soft spots are real but minor: the 'tens of millions of samples' claim is asserted on the strength of a self-citation rather than demonstrated, and the paper leans entirely on CLAP plus one ESC-50 t-SNE screenshot to promise 'meaningful clusters' for any audio dataset. A short quantitative sanity check (a few retrieval or zero-shot numbers, or a cluster-consistency measure) would tighten this up nicely.\n\nThe genuinely new thing here is the specific combination: CLAP embeddings, t-SNE projection, DeepScatter's WebGL rendering, Milvus for vector storage, and Annoy for semantic search, all packaged for audio data with zero-shot labeling built in. That combination doesn't exist in the cited prior work, which is either image-focused or not interactive. As a practical tool, it will likely save time for people who want a quick look at an audio dataset or a qualitative audit of an embedding model. The open-sourcing makes it reproducible and useful even if the paper itself is thin.\n\nThe ESC-50 figure does show structure, and for CLAP—a model with its own benchmarks—that's plausible evidence. But the stress-test note is right that t-SNE can produce visually compact clusters for noisy data, so the screenshot alone would not convince me if the paper made a strong scientific claim. It doesn't; it makes a tool claim. Still, the Introduction's 'any audio dataset' and 'tens of millions of samples' are overstatements for what's actually shown. Those are easy fixes: soften the language or add a benchmark.\n\nI'd take this paper as a late-breaking demo without complaint. It's the kind of contribution that belongs at ISMIR as a demo, and it deserves referee time to make sure the claims are bounded. The main thing I'd ask the authors is to state clearly what's verified and what's expected from the underlying components.","headline":"A useful open-source demo for audio dataset exploration, with some overclaimed scale and CLAP generality; judge it as a tool, not a research result.","tokens_in":4425,"tokens_out":2564,"would_cite":false,"duration_ms":24857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Audio Atlas claims that any audio dataset can be visualized as a responsive, searchable two-dimensional map, even at tens of millions of samples.","keywords":["audio visualization","CLAP embeddings","contrastive language-audio pretraining","t-SNE projection","semantic search","zero-shot classification","audio dataset exploration","interactive point cloud"],"falsifier":"Run the same pipeline on a labeled audio dataset from a domain far from CLAP's training distribution, such as medical recordings or animal vocalizations, and measure whether the t-SNE projection separates known classes and whether text queries retrieve the correct labels. If familiar classes scatter and nearest neighbors are semantically unrelated, the claim that Audio Atlas can visualize 'any audio dataset' with meaningful clusters is refuted.","tokens_in":3555,"feed_emoji":"🎧","tokens_out":7546,"duration_ms":69621,"temperature":0.7,"pith_summary":"Audio Atlas is a web application whose stated purpose is to make any audio dataset explorable as an interactive map. The authors claim that by embedding audio clips with a contrastive audio–text model, projecting the embeddings to two dimensions with t-SNE, and rendering the result with a GPU-accelerated scatterplot engine, the tool stays responsive even at tens of millions of samples. The same embedding space enables two things that normally require labeled metadata: semantic search by text or by an uploaded audio snippet, and zero-shot classification using a user-defined list of classes. The ESC-50 demo shows that class labels form local clusters in the projection, which is the evidence offered that the map reflects meaningful audio content.","feed_headline":"Audio Atlas puts millions of audio samples on one map","feed_subtitle":"Text and audio queries find nearby clips; zero-shot labels color the clusters.","key_machinery":"The load-bearing mechanism is the text-audio embedding space produced by CLAP, a contrastive neural network trained on audio-text pairs; it makes 'semantic similarity' operational as ordinary vector distance. t-SNE then compresses those vectors to two dimensions for display. On top of this, a GPU-accelerated point-cloud renderer draws many points incrementally to keep pan and zoom responsive, while a vector database and an approximate nearest-neighbor index answer text and audio queries without scanning the dataset. The interface's zero-shot classification is the same machinery applied in reverse: user-supplied class names are embedded, and each audio point is colored by its nearest class embedding.","core_discovery":"The central discovery is that dataset-scale audio exploration needs no bespoke audio feature engineering: a single contrastive audio-text model can supply both the spatial structure and the search index. Audio Atlas treats CLAP embeddings as the ground truth of similarity, projects them with t-SNE into a plane, and stores them in a vector database for nearest-neighbor queries. Users can then inspect clusters, click any point to hear it and see its closest neighbors, filter by zero-shot class labels, or search by describing a sound; because text and audio live in one embedding space, queries in either modality retrieve results by meaning rather than by low-level waveform similarity. The paper's demonstration on ESC-50, where all environmental sound classes fall into local pockets, is the basis for the claim that the clusters visible in the interface are semantically coherent.","pith_inferences":["Going beyond the paper, the design implies a cheap benchmark for embedding quality: on datasets with known labels, compute cluster-label agreement (e.g., adjusted Rand index) directly on the t-SNE projection; the paper reports only the qualitative ESC-50 screenshot.","A testable extension would be to swap the static t-SNE projection for a projection method that supports out-of-sample points, so newly uploaded audio could appear on the map instead of only in a nearest-neighbor list.","The paper does not address how stable the map is across random t-SNE runs; a natural follow-up is measuring whether the visual clusters persist under different seeds and perplexities, since unstable maps would undermine cluster-based browsing.","One could also treat the tool as an active learning interface: the embedding distance plus user clicks on clusters could generate pseudo-labels for training lightweight classifiers, which the paper leaves implicit."],"forward_implications":["Users can explore unannotated audio collections by typing a description of the sound they want, because text and audio share one embedding space.","Zero-shot classification becomes a live visual operation: choose class names, and the point cloud recolors instantly, so a user can judge whether the classes are separable before trusting the labels.","The tool doubles as a qualitative evaluation surface for embedding models, since a model that has not learned relevant audio structure will produce a map without coherent clusters.","Because new datasets only require new embeddings, the same interface can be pointed at arbitrary audio and music corpora, including collections with no metadata at all.","If the responsive rendering holds at tens of millions of samples, the approach scales past the six bundled datasets to full-scale production audio libraries."],"supporting_citations":[{"why":"Supplies the contrastive audio-text embedding model (CLAP) that defines semantic similarity for the whole tool.","marker":"[7]"},{"why":"Provides the t-SNE algorithm used to project embeddings into the 2D point cloud.","marker":"[9]"},{"why":"Is the vector database that stores embeddings and supports efficient similarity search.","marker":"[10]"},{"why":"Is the GPU-accelerated scatterplot renderer that keeps the interface responsive at scale.","marker":"[11]"},{"why":"Is the image-domain visualization tool whose design Audio Atlas adapts, establishing the visualization approach.","marker":"[6]"},{"why":"Provides the ESC-50 dataset used in the paper's main demonstration that classes cluster locally.","marker":"[17]"},{"why":"Provides the FMA music dataset shown in the first figure, supporting the music-browsing use case.","marker":"[8]"},{"why":"Is the large-scale music dataset cited for the claim that Audio Atlas remains responsive on big collections.","marker":"[19]"}],"fun_headline_variants":["Audio Atlas maps audio datasets by meaning, not waveform","Explore millions of audio samples on one interactive map","Text and audio search meet in one embedding space","Open-source tool turns audio datasets into visual clusters","No feature engineering: CLAP embeddings power audio exploration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that CLAP embeddings preserve the semantic distinctions that matter for the audio being visualized; if they do not, the spatial clusters and text-search results will be misleading even though the interface itself works.","fun_headline_variants_meta":{"raw":{"variants":["Audio Atlas maps audio datasets by meaning, not waveform","Explore millions of audio samples on one interactive map","Text and audio search meet in one embedding space","Open-source tool turns audio datasets into visual clusters","No feature engineering: CLAP embeddings power audio exploration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3043,"prompt_tokens":784,"completion_tokens":2259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":2186}},"tokens_in":400,"tokens_out":2259,"duration_ms":16280,"temperature":1.0,"reasoning_tokens":2186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:11:09.658763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a labeled audio dataset from a domain far from CLAP's training distribution, such as medical recordings or animal vocalizations, and measure whether the t-SNE projection separates known classes and whether text queries retrieve the correct labels. If familiar classes scatter and nearest neighbors are semantically unrelated, the claim that Audio Atlas can visualize 'any audio dataset' with meaningful clusters is refuted.","supporting_citations":[{"cited_title":"The eternal jukebox,","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive audio-text embedding model (CLAP) that defines semantic similarity for the whole tool."},{"cited_title":"Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,","cited_arxiv_id":null,"evidence_quote":"Is the vector database that stores embeddings and supports efficient similarity search."},{"cited_title":"The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale,","cited_arxiv_id":null,"evidence_quote":"Is the image-domain visualization tool whose design Audio Atlas adapts, establishing the visualization approach."},{"cited_title":"Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),","cited_arxiv_id":null,"evidence_quote":"Is the large-scale music dataset cited for the claim that Audio Atlas remains responsive on big collections."}],"review_version":1}