{"id":"95823aad-43f7-49e6-8cc1-3051508b5607","arxiv_id":"2608.10979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A VQ-VAE with alignment-invariant reconstruction learns stable tokens of pitch-contour shapes from unlabeled audio and shows modest signal for sigimsae categories and pansori modes.","lead":"This paper trains a VQ-VAE to chop continuous vocal pitch contours into a fixed vocabulary of small shape patterns, learned without human labels. It applies the tokens to Korean traditional music, where ornaments and modes show up as contour shapes rather than notes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 7's token-mode alignment is a post hoc max over a 256-token codebook with no null model; the wide-vibrato reading conflicts with the model's own pitch-range invariance objective.","rationale":"The method contribution is real and the segmentation-consistency results are a useful quantitative evaluation; the paper also honestly states the unvoiced-region limitation in Section 8. The sigimsae probe is reasonable as a minimal test, though the improvements over the autoencoder are small (Table 2: 0.285 vs. 0.284 F1 macro on all data; 0.331 vs. 0.324 on vocal) and no error bars or significance tests are given. The decisive weakness is Section 7: the mode alignment is based on picking the most extreme token out of 256, which requires a null model to interpret. Because the training loss explicitly includes pitch-range scaling, the interpretation of Token 54 as detecting 'wide vibrato' is also in tension with the model's intended invariance; the consistency drop in Table 1 shows the full model is not perfectly invariant. A permutation test over mode labels is cheap and would settle whether the alignment is real. Pending that test, the reader's conditional verdict remains appropriate.","tokens_in":10758,"tokens_out":10617,"duration_ms":95972,"concrete_test":"Permutation test on the mode-annotated subset: fix token assignments, shuffle mode labels 1,000 times, and each time record max_k P(mode|token) across all 256 tokens. If the observed maximum (or P(Ujo|token=54)) is not above the 95th percentile of the null distribution, the token-mode alignment is a selection artifact. As a confirmatory check, compare the 5th-to-95th-percentile pitch range of median-normalized F0 for Token 54 segments between Gyemyeonjo and Ujo; a significant difference would indicate a residual non-shape range confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in Section 7, the mode-alignment evidence for the central claim that tokens align with Gyemyeonjo and Ujo. The authors select, for each mode, the token with the highest P(mode|token) from a 256-entry codebook (Token 54 for Ujo) and then interpret its contour visually. This is a post hoc maximum over 256 random variables: under the null that tokens carry no mode information, the maximum P(mode|token) will be far above the base rate, and no permutation test or multiple-comparison correction is reported. The qualitative reading of Token 54 as separating modes by 'wide vibrato' also sits in tension with the training objective in Section 3.2, which explicitly includes pitch-range scaling (A = {0.85, 1.0, 1.15}) to make tokens invariant to amplitude and range differences. Table 1 shows that adding these pitch-domain transformations degrades segmentation-shift consistency (KLD 0.652 vs. 0.531 for the full model), so residual range sensitivity is plausible. If the apparent alignment is a selection artifact or a pitch-range confound rather than vibrato shape, the claim that individual tokens align with the two principal modes loses its evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised VQ-VAE tokenizer for fixed-length pitch-contour segments. To promote segmentation and pitch-range robustness, the reconstruction loss is minimized over candidate temporal offsets, resampling scales, pitch shifts, and pitch-range scalings. The method is evaluated on Korean traditional music through segmentation-shift consistency (Section 5), a per-token MAP probe for sigimsae categories (Section 6), and a qualitative mode analysis in pansori (Section 7). The paper claims that the learned codebook is stable, that its tokens recover label information without supervision, and that individual tokens can separate Gyemyeonjo and Ujo.","tokens_in":11059,"tokens_out":5127,"duration_ms":48000,"significance":"If substantiated, the proposed tokenization would be a valuable tool for corpus-level analysis of traditions organized around continuous pitch movement, because the discrete vocabulary is induced from unlabeled audio rather than predefined by experts. The paper has clear strengths: the transformation-minimized loss is a well-motivated architectural contribution, the code and demo are publicly available, and the deliberately weak probing classifier in Section 6 makes the sigimsae result a conservative test of label information. The segmentation-shift consistency results in Table 1 are internally coherent. However, the two strongest interpretive claims—that tokens carry mode information and that the learned invariances remove pitch-range confounds—currently rest on evidence without null-model controls or statistical support.","major_comments":[{"comment":"The mode alignment is established by selecting, for each mode, the token with the highest P(mode|token) from a 256-entry codebook. This is a post hoc maximum: under the null hypothesis that tokens carry no mode information, the maximum of 256 estimated probabilities is expected to be far above the base rate. No permutation test, null-model comparison, or multiple-comparison correction is reported. The claim that 'individual tokens align with the two principal modes' therefore lacks quantitative support. Please report the full distribution of P(mode|token) over the codebook, the counts behind the token 54 example, and the significance of the selected token relative to a null model in which segment labels are permuted.","section":"Section 7, Figure 3"},{"comment":"The paper states that because tokens are median-normalized, 'any mode-specific token usage must arise from contour shape itself.' Median subtraction removes absolute pitch offset, but not pitch range or amplitude, and Eq. (4) explicitly includes pitch-range scaling with A={0.85,1.0,1.15}, making pitch-range invariance an intended property rather than a guaranteed one. Table 1 shows that adding the pitch-domain transformations degrades segmentation-shift consistency (KLD 0.652 vs. 0.531 for temporal alignment alone), so residual sensitivity to pitch range is plausible. The wide-vibrato reading of Token 54 should be checked with a control: rescaled pitch-range versions of the same contours should be fed to the tokenizer and the token assignments compared, and Token 54's assigned contours should be compared to range-matched contours from both modes. Without this, the mode separation may be a residual range effect rather than vibrato shape.","section":"Section 7; Section 3.2, Eq. (4)"},{"comment":"The reported improvements of the full transformation loss over the autoencoder baseline on the vocal-only setting are small (F1 0.331 vs. 0.324; mAP 0.324 vs. 0.290), and all results appear to come from a single training run. No error bars, repeated seeds, or significance tests are provided. Given that the sigimsae probing experiment is the main quantitative evidence that the learned tokens 'recover information about expert-defined sigimsae categories,' the reader cannot currently distinguish a genuine improvement from training noise. Please add multiple seeds with confidence intervals or a statistical test over runs.","section":"Section 6.3, Table 2"}],"minor_comments":[{"comment":"The quantities labeled 'P=0.02', 'P=0.04', etc. in Figure 2 are conditional token probabilities P(token|label), not p-values; please relabel them to avoid statistical misinterpretation.","section":"Section 6.2, Figure 2"},{"comment":"The relationship among the 280-hour in-house pansori dataset, the 34.0/10.0-hour AI-Hub sigimsae set, and the 26.7/8.7-hour mode-annotated subset should be stated explicitly, in particular whether the mode subset is disjoint from the training segments used in Sections 5 and 6.","section":"Section 4.1"},{"comment":"The text says that only the transformation attaining the minimum error is used for backpropagation, but it would be helpful to state explicitly that the selected s, tau, a, and b are fixed when computing gradients, to rule out any confusion about differentiating through a minimum.","section":"Section 3.2, Eq. (4)"},{"comment":"The phrase 'Tokens 171' should be 'Token 171' for consistency with the surrounding text and with other token references.","section":"Section 7"},{"comment":"The model name appears as 'VQ-V AE' with a space in several places; this typography should be corrected to 'VQ-VAE'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not show problematic citation patterns, but reference [24] is an unpublished submission and should be clearly marked as such. The sigimsae probing experiment is the most convincing quantitative result; the qualitative mode analysis in Section 7 is the part most in need of control before the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper for the method, not for the mode analysis. The core idea is worth your time: learn a fixed codebook of local pitch-contour shapes from unlabeled audio, with a reconstruction loss that minimizes over temporal and pitch-range transformations. That's a genuinely new technique for a real problem, and the authors ship code and a demo.\n\nThe segmentation-shift consistency experiments are the strongest part. Temporal alignment alone drops KLD from 0.657 to 0.531 and lifts token matching accuracy from 0.462 to 0.519, a clear and internally consistent effect. The sigimsae probing is more modest: the full transformation model gets F1 0.331 vs 0.324 and mAP 0.324 vs 0.290 over the autoencoder baseline, which is suggestive but within noise without error bars. The paper is also honest about the unvoiced-region limitation.\n\nThe weakest section is 7, the pansori mode analysis. The authors select the token with the highest P(mode|token) from a 256-entry codebook and interpret its contour. That is a post hoc maximum; under a null of no mode information, the max will be far above base rate, and they report no permutation test or multiple-comparison correction. I also think the 'wide vibrato' reading of Token 54 sits in tension with their own pitch-range invariance objective: if the model minimizes over amplitude scaling, absolute vibrato width should not be a distinguishing feature. Table 1 even shows that adding these transformations degrades segmentation consistency, so residual range sensitivity is plausible. The authors need a control—comparing against a shuffled-label null or the autoencoder's prototype token—before claiming individual tokens align with modes.\n\nWho is this for? Ethnomusicologists and MIR people working on traditions where continuous pitch movement carries the musical meaning. They get a concrete, reproducible way to build discrete vocabularies for corpus statistics. For peer review: send it out. It's a novel method with real evaluation and code. I'd ask for a null model for Section 7, error bars or repeated runs for the sigimsae numbers, and a discussion of why the vibrato interpretation survives their invariance objective. Those are fixable. The core method is sound.","headline":"VQ-VAE contour tokenization is a real step forward for contour-centric MIR; the mode-alignment evidence needs a null model before it can carry the abstract's weight.","tokens_in":11562,"tokens_out":3909,"would_cite":true,"duration_ms":34582,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A VQ-VAE trained only to reconstruct fixed-length pitch-contour segments induces a finite codebook of local contour shapes that, without labels, align with expert sigimsae categories and with the two principal pansori modes.","keywords":["pitch contour tokenization","VQ-VAE","Korean traditional music","sigimsae","pansori","discrete representation learning","ornament analysis","transformation-invariant reconstruction"],"falsifier":"A synthetic control would settle the shape-invariance claim: generate contour pairs that share the same melodic shape but differ only in absolute pitch level, vibrato width, or segmentation phase, and check whether token assignment is preserved; if tokens flip under pitch shift or range scaling, the transformation-minimized loss has not removed non-shape factors. An equivalent real-data check would tokenize the same pansori phrase transposed by several semitones and compare codebook usage.","tokens_in":10580,"feed_emoji":"🎵","tokens_out":8347,"duration_ms":72867,"temperature":0.7,"pith_summary":"The paper sets out to show that the discrete units needed for computational analysis of contour-based music need not be defined in advance by experts: a VQ-VAE trained only to reconstruct fixed-length pitch-contour segments can induce a finite codebook of local contour shapes from unlabeled audio. The authors argue that these tokens are stable across segmentation positions and small variations in timing and pitch range because the reconstruction loss is minimized over candidate temporal and pitch-domain transformations. They demonstrate that the learned tokens carry label-discriminative information for expert-defined sigimsae ornament categories without any supervision, and that in pansori individual tokens align with the two principal modes, Gyemyeonjo and Ujo. If correct, the approach supplies corpus-level analysis with a unit for contour-centric traditions that plays the role the note plays for score-based music.","feed_headline":"Unlabeled audio yields a codebook of Korean ornament shapes","feed_subtitle":"Learned tokens match expert sigimsae classes and separate pansori modes by vibrato shape alone.","key_machinery":"The central mechanism is the transformation-minimized reconstruction loss, which evaluates reconstruction quality after searching over temporal scale, temporal offset, pitch-range scaling, and pitch shift and backpropagates only through the best alignment. This is what lets the codebook group contours by shape rather than by phase, timing, or range. The other load-bearing design is receptive-field separation with median subtraction: the input is pre-segmented into non-overlapping 128-frame windows so each token corresponds to exactly one fixed-length contour segment, and subtracting the segment median removes absolute pitch so tokens encode local movements such as oscillations and slides.","core_discovery":"The paper's central discovery is that a discrete vocabulary of local pitch-contour patterns can be induced directly from unlabeled F0 contours. Median normalization strips absolute pitch from each receptive-field segment, the transformation-minimized reconstruction loss absorbs segmentation phase and small timing and pitch-range differences, and the resulting 256-entry codebook assigns the same token to shape-equivalent contours. In the authors' evaluation, a single token selected per sigimsae class visually matches the canonical realization of that class, and in pansori one token separates Gyemyeonjo from Ujo by vibrato width on an otherwise identical descending motion, a distinction that note-level transcription would discard.","pith_inferences":["A direct control the paper does not run: synthesizing contour pairs identical in shape but differing only in vibrato width or absolute pitch level would test whether token 54's mode separation is caused by contour shape rather than by a residual pitch or range artifact.","Because the tokenizer discards segments with extensive unvoiced frames, corpus-level statistics built on it will systematically under-represent phrases with weak or missing voicing, a coverage gap the authors acknowledge.","The paper demonstrates per-token mode association but not that token transition statistics differ across modes; testing n-gram or transition matrices over the same codebook across Gyemyeonjo and Ujo would show whether the units support the corpus-level analyses the paper motivates."],"forward_implications":["Dense token sequences can be computed over entire voiced contours, making transition statistics and n-gram analyses available for contour-centric traditions in the way they have long been for chord and note sequences.","Without any ornament labels, the most representative token for each sigimsae class visually matches the canonical realization of that class, and the full transformation-minimized loss improves per-token classification over the autoencoder baseline.","In pansori, token 54 separates Gyemyeonjo from Ujo by the width of vibrato on an otherwise identical descending motion, preserving an ornamental distinction that note-level transcription discards.","The same codebook, queried against other label sets such as schools or performers, offers a path toward corpus-level study of ornamental practice without predefined ornament categories.","Because the method requires no labels and no note-level transcription, it applies in principle to other contour-centric traditions, including Indian art music and pop vocal singing."],"supporting_citations":[{"why":"It supplies the VQ-VAE objective and codebook mechanism the tokenizer is built on.","marker":"[12]"},{"why":"It provides the pretrained fundamental-frequency estimator that converts audio into pitch contours.","marker":"[19]"},{"why":"It isolates vocal tracks from instrumental accompaniment before pitch extraction.","marker":"[18]"},{"why":"It supplies the expert-annotated sigimsae segments used in the classification probe.","marker":"[22]"},{"why":"It provides the expert mode-annotated pansori subset used in the token-mode analysis.","marker":"[24]"},{"why":"The autoencoder baseline is adapted from this work's dimensionality-reduction and clustering tokenization pipeline.","marker":"[20]"},{"why":"The supervised temporal convolutional network gives the order-of-magnitude reference for sigimsae classification.","marker":"[23]"},{"why":"It supports the premise that vector-quantized autoencoders can discover units aligned with content categories from continuous signals.","marker":"[17]"}],"fun_headline_variants":["Pitch-shape tokens from unlabeled audio match Korean ornament classes","VQ-VAE tokenizes pitch contours to separate pansori modes by vibrato","Unlabeled audio yields discrete pitch-contour tokens for Korean music","Shape tokens from raw audio recover Korean ornament types without labels","Pitch-contour codebook from unlabeled audio reveals Korean mode structure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that median normalization plus the pitch-shift and pitch-range reconstruction transforms remove every non-shape factor, so a token that appears predominantly in Ujo must be responding to contour shape (here vibrato width) and not to residual pitch level, range, or F0-extraction artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Pitch-shape tokens from unlabeled audio match Korean ornament classes","VQ-VAE tokenizes pitch contours to separate pansori modes by vibrato","Unlabeled audio yields discrete pitch-contour tokens for Korean music","Shape tokens from raw audio recover Korean ornament types without labels","Pitch-contour codebook from unlabeled audio reveals Korean mode structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3281,"prompt_tokens":833,"completion_tokens":2448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2370}},"tokens_in":449,"tokens_out":2448,"duration_ms":19144,"temperature":1.0,"reasoning_tokens":2370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:57:00.494207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A synthetic control would settle the shape-invariance claim: generate contour pairs that share the same melodic shape but differ only in absolute pitch level, vibrato width, or segmentation phase, and check whether token assignment is preserved; if tokens flip under pitch shift or range scaling, the transformation-minimized loss has not removed non-shape factors. An equivalent real-data check would tokenize the same pansori phrase transposed by several semitones and compare codebook usage.","supporting_citations":[{"cited_title":"Soundstream: An end-to-end neural audio codec,","cited_arxiv_id":null,"evidence_quote":"It provides the expert mode-annotated pansori subset used in the token-mode analysis."},{"cited_title":"In search of the horowitz factor,","cited_arxiv_id":null,"evidence_quote":"It supplies the VQ-VAE objective and codebook mechanism the tokenizer is built on."},{"cited_title":"Mining melodic patterns in large audio collections of indian art music,","cited_arxiv_id":null,"evidence_quote":"It provides the pretrained fundamental-frequency estimator that converts audio into pitch contours."},{"cited_title":"Hierarchical generative modeling of melodic vocal contours in hindustani classical music,","cited_arxiv_id":null,"evidence_quote":"It isolates vocal tracks from instrumental accompaniment before pitch extraction."},{"cited_title":"Symbolic music loop generation with neural discrete representations,","cited_arxiv_id":null,"evidence_quote":"It supplies the expert-annotated sigimsae segments used in the classification probe."},{"cited_title":"The matrix profile for motif discovery in audio - an example application in carnatic music,","cited_arxiv_id":null,"evidence_quote":"The autoencoder baseline is adapted from this work's dimensionality-reduction and clustering tokenization pipeline."},{"cited_title":"A compact pitch and time rep- resentation for melodic contours in indian art music,","cited_arxiv_id":null,"evidence_quote":"It supports the premise that vector-quantized autoencoders can discover units aligned with content categories from continuous signals."}],"review_version":1}