Information Retrieval · arXiv:2611.17070 v1 · submitted 12 Nov 2026 · 17 references
Evaluation of Music Version Retrieval Methods for Setlist Identification in Live Concerts
The paper says
“Automatic setlist identification aims to identify the songs performed in live concert recordings by matching noisy and variable-quality audio against a database of reference tracks.”
more of the abstract
This task presents challenges beyond conventional music version identification, including substantial arrangement differences, background noise, and the need to process continuous recordings efficiently at large scale. In this work, we evaluate modern music version retrieval methods for setlist identification using the ASID benchmark, comparing the established Re-MOVE system with CLEWS and NAVE, an internal neural audio version embedding model. Our results show that NAVE and CLEWS substantially outperform Re-MOVE. Notably, NAVE uses 256-dimensional embeddings, compared with 1024 dimensions for CLEWS, providing a four-fold reduction in embedding storage while achieving better retrieval performance. We further evaluate music/non-music gating and find modest improvements in detection and temporal coverage, while qualitative analysis suggests that such preprocessing can improve recognition boundaries beyond what the standard ASID metrics capture. Overall, the results demonstrate that segment-level audio embeddings provide a strong basis for scalable setlist identification and that compact representations can achieve strong accuracy under realistic live-concert conditions at much lower storage and serving cost.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Solid benchmark comparison, but the NAVE-vs-CLEWS advantage is within the noise and the paper says so itself. This line is wrong
the referee pass did not run
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 17 references say.
What would settle it
Compute 95% confidence intervals for DAP on the 62-concert evaluation set via bootstrap (resample concerts with replacement, 10,000 replicates) for NAVE recognition, NAVE pooled, and CLEWS. If the intervals overlap substantially (e.g., NAVE pooled's lower bound ≤ CLEWS's upper bound), then the claimed improvement is not statistically significant at the 5% level. Report the bootstrap means and intervals.
from the skeptic's reading
How the review went
3 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
Not run on this paper: the referee, the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper compares three version-ID systems on the ASID setlist-identification benchmark. The headline result is that both CLEWS and NAVE (an internal system from Nowon) beat the previous Re-MOVE baseline by about 12 percentage points in DAP, reaching ~0.90. That gap is real and the paper explains it well: segment-level embeddings on CQT spectrograms capture more information than Re-MOVE's single-vector chroma embeddings.
What's actually new is the application of CLEWS to setlist ID (it was published for cover-song retrieval) and the introduction of NAVE, which uses 256-D embeddings instead of CLEWS's 1024-D. The storage savings are real: 4x smaller footprint with similar accuracy. The paper also tests music/non-music gating and finds modest gains, plus a qualitative observation that the ASID metrics miss boundary precision.
The paper is honest about its limitations. It acknowledges the reconstructed dataset is missing 3 of 65 evaluation concerts, that NAVE is not publicly available, and that small DAP differences are hard to interpret. The authors explicitly say (b74) that the NAVE-vs-CLEWS difference may reflect dataset limitations rather than model gains. The stress-test concern is correct: the claimed improvement of NAVE over CLEWS is not statistically tested and is tiny (0–0.7 pp). But the paper itself flags this, so it's not a hidden flaw—it's a stated caveat that the conclusion section then slightly overstates (b76 says NAVE "improves" CLEWS). That's a minor overclaim, not a fatal error.
The soft spots: (1) NAVE is closed-source, so the comparison is not fully reproducible. (2) No significance tests for any of the differences. (3) The dataset reconstruction may introduce unknown biases. (4) The SVM filtering was not applied to CLEWS/NAVE, so the comparison is not exactly apples-to-apples with the original ASID protocol.
Who this is for: researchers and engineers working on music version identification or live-concert audio processing. It's a practical benchmark comparison, not a theoretical advance. The paper deserves a serious referee—the evaluation is careful, the limitations are stated, and the storage-efficiency result is useful. I'd recommend acceptance with a request to tone down the NAVE-vs-CLEWS claim to match the caveats, and ideally to release NAVE or at least provide more detailed architecture information.
Best, [you]
The strongest objection
The paper's central claim that NAVE 'improves the public CLEWS checkpoint in detection accuracy' (b76) rests on a DAP difference of 0.903 vs 0.903 (recognition) or 0.910 vs 0.903 (pooled). These differences are tiny (0–0.7 pp) and are reported on a reconstructed evaluation set that is missing 3 of the original 65 concerts (b17, b72). The paper itself notes that 'small DAP differences seem hard to interpret and may reflect dataset limitations, more than clear model gains' (b74). No confidence intervals, bootstrap resampling, or statistical significance tests are provided. Given that the ASID benchmark has only 62 evaluation concerts and the DAP metric is a proportion of correctly detected annotations, the variance of the estimate is non-negligible. Without a significance test, the reader cannot distinguish a genuine improvement from random variation due to the particular concert subset or the stochasticity of the embedding models. The claim that NAVE 'improves' over CLEWS is therefore unsupported by the evidence presented.
The reader's verdict is CONDITIONAL with medium correctness risk, and the weakest assumption identified is that the reconstructed ASID benchmark is representative. The reader also notes that 'the claimed improvement over CLEWS is marginal and not statistically tested.' My stress-test agrees with this assessment and sharpens it: the lack of any significance testing is the single most load-bearing concern because the paper's headline claim (NAVE improves over CLEWS) depends on DAP differences that are within the noise floor of the evaluation. The paper's own limitation statement (b74) acknowledges this, but the conclusion (b76) still asserts an improvement. The concrete test I propose (bootstrap confidence intervals) would directly settle whether the observed differences are reliable. I do not raise concerns about the Re-MOVE comparison, which is clearly larger (~12 pp) and robust. The concern is narrowly about the NAVE-vs-CLEWS comparison, which is the basis for the stronger claim. The verdict should remain CONDITIONAL because the authors can address this by adding a significance analysis; if they do, the claim may become supportable. I agree with the reader's assessment (agreement_with_reader = agree) and recommend no change to the verdict (UNCHANGED would also be reasonable, but CONDITIONAL is already the reader's verdict and I see no reason to alter it).
Pith's reading of the paper (a machine's summary, not the authors')
The paper compares three systems for automatic setlist identification: matching noisy concert audio against a database of studio reference tracks. Re-MOVE, the previous best system on the ASID benchmark, uses hand-crafted chroma features and produces a single embedding per window. CLEWS and NAVE operate on spectrogram representations and produce sequences of segment-level embeddings, enabling finer temporal matching. On the ASID dataset (62 concerts, ~100 hours of audio), both CLEWS and NAVE achieve Detection Accuracy Percentage (DAP) around 0.90, compared to Re-MOVE's 0.78. NAVE uses 256-dimensional embeddings versus CLEWS's 1024, a 4x storage reduction, while maintaining or slightly improving accuracy. The paper also tests music/non-music gating as a preprocessing step, finding modest improvements in boundary precision. The authors acknowledge limitations: the dataset is relatively small (75 concerts), three concerts were missing, and the metrics may not fully capture temporal boundary accuracy. They also note that the ASID benchmark may be approaching a ceiling for DAP, making further improvements hard to distinguish.
Core claim. NAVE and CLEWS achieve ~0.90 DAP on the ASID benchmark, a ~12 percentage point improvement over Re-MOVE (0.78 DAP), with NAVE using 256-dimensional embeddings (4x smaller than CLEWS's 1024) while maintaining or slightly improving accuracy.
The premise it rests on. The ASID benchmark, reconstructed from the original paper with 3 missing concerts (62 instead of 65 evaluation concerts), is representative of real-world live concert conditions and production deployment. The paper acknowledges this limitation but relies on it as the sole evaluation dataset.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Automatic setlist identification aims to identify the songs performed in live concert recordings by matching noisy and variable-quality audio against a database of reference tracks. This task presents challenges beyond conventional music version identification, including substantial arrangement differences, background noise, and the need to process continuous recordings efficiently at large scale. In this work, we evaluate modern music version retrieval methods for setlist identification using the ASID benchmark, comparing the established Re-MOVE system with CLEWS and NAVE, an internal neural audio version embedding model. Our results show that NAVE and CLEWS substantially outperform Re-MOVE. Notably, NAVE uses 256-dimensional embeddings, compared with 1024 dimensions for CLEWS, providing a four-fold reduction in embedding storage while achieving better retrieval performance. We further evaluate music/non-music gating and find modest improvements in detection and temporal coverage, while qualitative analysis suggests that such preprocessing can improve recognition boundaries beyond what the standard ASID metrics capture. Overall, the results demonstrate that segment-level audio embeddings provide a strong basis for scalable setlist identification and that compact representations can achieve strong accuracy under realistic live-concert conditions at much lower storage and serving cost.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-14
The authors' main claim, as the classifier picked it out 1.00
“Overall, the results demonstrate that segment-level audio embeddings provide a strong basis for scalable setlist identification and that compact representations can achieve strong accuracy under realistic live-concert conditions at much lower storage and serving cost.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.56 · human participants 0.04 · controlled comparison 0.89 · randomized assignment 0.01 · states quantitative result 0.94 · compares with prior results 0.98 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Automatic setlist identification aims to identify the songs performed in live concert recordings by matching noisy and variable-quality audio against a database of reference tracks. | background 1.00 | 0.06 | 0.02 | asserted 0.99 |
| This task presents challenges beyond conventional music version identification, including substantial arrangement differences, background noise, and the need to process continuous recordings efficiently at large scale. | background 0.97 | 0.07 | 0.02 | asserted 1.00 |
| In this work, we evaluate modern music version retrieval methods for setlist identification using the ASID benchmark, comparing the established Re-MOVE system with CLEWS and NAVE, an internal neural audio version embedding model. | contribution 0.61 | 0.89 | 0.04 | asserted 0.93 |
| Our results show that NAVE and CLEWS substantially outperform Re-MOVE. | result 1.00 | 0.96 | 0.44 | asserted 1.00 |
| Notably, NAVE uses 256-dimensional embeddings, compared with 1024 dimensions for CLEWS, providing a four-fold reduction in embedding storage while achieving better retrieval performance. | result 0.95 | 0.91 | 0.14 | asserted 1.00 |
| We further evaluate music/non-music gating and find modest improvements in detection and temporal coverage, while qualitative analysis suggests that such preprocessing can improve recognition boundaries beyond what the standard ASID metrics capture. | result 0.93 | 0.96 | 0.05 | hedged possibility 0.52 |
| Overall, the results demonstrate that segment-level audio embeddings provide a strong basis for scalable setlist identification and that compact representations can achieve strong accuracy under realistic live-concert conditions at much lower storage and serving cost. | interpretation 0.99 | 0.93 | 0.89 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 0.99 · evidence from observation or experiment 0.91
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-14: 1 request, 5,660 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (2)
References (17)
- [1] J.-C. Wang, M.-C. Yen, Y.-H. Yang, and H.-M. Wang. Automatic set list identification and song segmentation for full-length concert videos. InProc. Int. Soc. Music Inf. Retrieval Conf. (ISMIR), pp. 239–244, 2014 on Pith
- [2] F. Yesiler, E. Molina, J. Serrà, and E. Gómez. Investigating the efficacy of music version retrieval systems for setlist identification. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2021 on Pith
- [3] F. Yesiler, J. Serrà, and E. Gómez. Less is more: Faster and better music version identification with embedding distillation. InProc. Int. Soc. Music Inf. Retrieval Conf. (ISMIR), pp. 884–892, 2020 on Pith
- [4] F.Yesiler, J.Serrà, andE.Gómez. Accurateandscalableversionidentificationusingmusically-motivated embeddings. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 21–25, 2020 on Pith
- [5] J. Serrà, E. Gómez, P. Herrera, and X. Serra. Chroma binary similarity and local alignment applied to cover song identification.IEEE Trans. Audio Speech Lang. Process., 16(6):1138–1151, 2008 on Pith
- [6] D. P. W. Ellis and G. E. Poliner. Identifying ‘cover songs’ with chroma features and dynamic pro- gramming beat tracking. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), vol. 4, pp. 1429–1432, 2007 on Pith
- [7] T. Bertin-Mahieux and D. P. W. Ellis. Large-scale cover song recognition using the 2D Fourier transform magnitude. InProc. Int. Soc. Music Inf. Retrieval Conf. (ISMIR), pp. 241–246, 2012 on Pith
- [8] J.Serrà, R.O.Araz, D. Bogdanov, andY.Mitsufuji. Supervisedcontrastivelearningfromweakly-labeled audiosegmentsformusicalversionmatching. InProc. Int. Conf. Mach. Learn. (ICML),pp.53923–53939, 2025 on Pith
- [9] Y.-N. Hung, C.-W. Wu, I. Orife, A. Hipple, W. Wolcott, and A. Lerch. A large TV dataset for speech and music activity detection.EURASIP J. Audio Speech Music Process., 2022(1):21, 2022 on Pith
- [10] Z. Yu, X. Xu, X. Chen, and D. Yang. Learning a representation for cover song identification using convolutional neural network. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 541–545, 2020. 8 on Pith
- [11] X. Du, Z. Yu, B. Zhu, X. Chen, and Z. Ma. ByteCover: Cover song identification via multi-loss training. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 551–555, 2021 on Pith
- [12] X. Du, K. Chen, Z. Wang, B. Zhu, and Z. Ma. ByteCover2: Towards dimensionality reduction of latent embedding for efficient cover song identification. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 616–620, 2022 on Pith
- [13] X. Du, Z. Wang, X. Liang, H. Liang, B. Zhu, and Z. Ma. ByteCover3: Accurate cover song identification on short queries. InProc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 1–5, 2023 on Pith
- [14] F. Liu, D. Tuo, Y. Xu, and X. Han. CoverHunter: Cover song identification with refined attention and alignments. InProc. IEEE Int. Conf. Multimedia Expo (ICME), pp. 1080–1085, 2023 on Pith
- [15] F. Yesiler, C. Tralie, A. Correya, D. F. Silva, P. Tovstogan, E. Gómez, and X. Serra. Da-TACOS: A dataset for cover song identification and understanding. InProc. Int. Soc. Music Inf. Retrieval Conf. (ISMIR), pp. 327–334, 2019 on Pith
- [16] X. Xu, X. Chen, and D. Yang. Key-invariant convolutional neural network toward efficient cover song identification. InProc. IEEE Int. Conf. Multimedia Expo (ICME), pp. 1–6, 2018 on Pith
- [17] R. O. Araz, X. Serra, and D. Bogdanov. Discogs-VI: A musical version identification dataset based on public editorial metadata. InProc. Int. Soc. Music Inf. Retrieval Conf. (ISMIR), pp. 478–485, 2024. 9 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.


No comments yet.
Sign in to comment