{"id":"99816788-0db9-49f9-9389-f11ba7195e51","arxiv_id":"2611.17070","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NAVE and CLEWS achieve ~0.90 DAP on setlist identification, outperforming Re-MOVE by ~12 percentage points, with NAVE using 256-dimensional embeddings (4x smaller than CLEWS).","lead":"This paper evaluates three music version identification systems for automatically determining which songs are played in live concert recordings. The authors show that their internal system (NAVE) and the public CLEWS system substantially outperform the previous Re-MOVE baseline on the ASID benchmark, achieving around 90% detection accuracy while using smaller embeddings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"NAVE's DAP advantage over CLEWS is within the noise of the reconstructed evaluation set and no significance test is reported.","rationale":"The reader's verdict is CONDITIONAL with medium correctness risk, and the weakest assumption identified is that the reconstructed ASID benchmark is representative. The reader also notes that 'the claimed improvement over CLEWS is marginal and not statistically tested.' My stress-test agrees with this assessment and sharpens it: the lack of any significance testing is the single most load-bearing concern because the paper's headline claim (NAVE improves over CLEWS) depends on DAP differences that are within the noise floor of the evaluation. The paper's own limitation statement (b74) acknowledges this, but the conclusion (b76) still asserts an improvement. The concrete test I propose (bootstrap confidence intervals) would directly settle whether the observed differences are reliable. I do not raise concerns about the Re-MOVE comparison, which is clearly larger (~12 pp) and robust. The concern is narrowly about the NAVE-vs-CLEWS comparison, which is the basis for the stronger claim. The verdict should remain CONDITIONAL because the authors can address this by adding a significance analysis; if they do, the claim may become supportable. I agree with the reader's assessment (agreement_with_reader = agree) and recommend no change to the verdict (UNCHANGED would also be reasonable, but CONDITIONAL is already the reader's verdict and I see no reason to alter it).","tokens_in":10017,"tokens_out":675,"duration_ms":9332,"concrete_test":"Compute 95% confidence intervals for DAP on the 62-concert evaluation set via bootstrap (resample concerts with replacement, 10,000 replicates) for NAVE recognition, NAVE pooled, and CLEWS. If the intervals overlap substantially (e.g., NAVE pooled's lower bound ≤ CLEWS's upper bound), then the claimed improvement is not statistically significant at the 5% level. Report the bootstrap means and intervals.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim that NAVE 'improves the public CLEWS checkpoint in detection accuracy' (b76) rests on a DAP difference of 0.903 vs 0.903 (recognition) or 0.910 vs 0.903 (pooled). These differences are tiny (0–0.7 pp) and are reported on a reconstructed evaluation set that is missing 3 of the original 65 concerts (b17, b72). The paper itself notes that 'small DAP differences seem hard to interpret and may reflect dataset limitations, more than clear model gains' (b74). No confidence intervals, bootstrap resampling, or statistical significance tests are provided. Given that the ASID benchmark has only 62 evaluation concerts and the DAP metric is a proportion of correctly detected annotations, the variance of the estimate is non-negligible. Without a significance test, the reader cannot distinguish a genuine improvement from random variation due to the particular concert subset or the stochasticity of the embedding models. The claim that NAVE 'improves' over CLEWS is therefore unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nThis paper compares three version-ID systems on the ASID setlist-identification benchmark. The headline result is that both CLEWS and NAVE (an internal system from Nowon) beat the previous Re-MOVE baseline by about 12 percentage points in DAP, reaching ~0.90. That gap is real and the paper explains it well: segment-level embeddings on CQT spectrograms capture more information than Re-MOVE's single-vector chroma embeddings.\n\nWhat's actually new is the application of CLEWS to setlist ID (it was published for cover-song retrieval) and the introduction of NAVE, which uses 256-D embeddings instead of CLEWS's 1024-D. The storage savings are real: 4x smaller footprint with similar accuracy. The paper also tests music/non-music gating and finds modest gains, plus a qualitative observation that the ASID metrics miss boundary precision.\n\nThe paper is honest about its limitations. It acknowledges the reconstructed dataset is missing 3 of 65 evaluation concerts, that NAVE is not publicly available, and that small DAP differences are hard to interpret. The authors explicitly say (b74) that the NAVE-vs-CLEWS difference may reflect dataset limitations rather than model gains. The stress-test concern is correct: the claimed improvement of NAVE over CLEWS is not statistically tested and is tiny (0–0.7 pp). But the paper itself flags this, so it's not a hidden flaw—it's a stated caveat that the conclusion section then slightly overstates (b76 says NAVE \"improves\" CLEWS). That's a minor overclaim, not a fatal error.\n\nThe soft spots: (1) NAVE is closed-source, so the comparison is not fully reproducible. (2) No significance tests for any of the differences. (3) The dataset reconstruction may introduce unknown biases. (4) The SVM filtering was not applied to CLEWS/NAVE, so the comparison is not exactly apples-to-apples with the original ASID protocol.\n\nWho this is for: researchers and engineers working on music version identification or live-concert audio processing. It's a practical benchmark comparison, not a theoretical advance. The paper deserves a serious referee—the evaluation is careful, the limitations are stated, and the storage-efficiency result is useful. I'd recommend acceptance with a request to tone down the NAVE-vs-CLEWS claim to match the caveats, and ideally to release NAVE or at least provide more detailed architecture information.\n\nBest,\n[you]","headline":"Solid benchmark comparison, but the NAVE-vs-CLEWS advantage is within the noise and the paper says so itself.","tokens_in":10881,"tokens_out":611,"duration_ms":9537},"pith_extraction":null,"created_at":"2026-11-13T10:16:48.585577+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}