{"id":"5426fe29-5236-4d8c-bda2-653b71ed227f","arxiv_id":"2411.16408","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On the Capitan mensural-notation dataset, a VICReg self-supervised encoder plus a five-layer MLP reaches 87.66% accuracy with five labelled samples per class, outperforming the earlier kNN baseline.","lead":"This paper tests a self-supervised learning pipeline that classifies musical symbols in historical manuscripts using very few labelled examples. It reports 87.66% accuracy with a small neural network classifier, beating an earlier k-nearest-neighbour baseline on the same dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvement over the [15] baseline is not yet demonstrated because the authors' own kNN reproduction diverges from the cited baseline (79.8% vs 82.0% at K=5), so the reported MLP gain may be a pipeline difference rather than a classifier effect.","rationale":"The paper is an empirical extension of [15]: same dataset, same self-supervised feature extractor, new classifiers and augmentations. The strongest claim is not that a self-supervised pipeline works—that was [15]—but that swapping kNN for MLP gives a 5.66-point gain at K=5. I looked for the condition that would make that claim true: the two classifier results must differ only in the classifier, with everything else held fixed. The paper does not satisfy that condition. Table I(a) is the key evidence: the authors' own kNN at K=5/A=0 is 79.8%, whereas the [15] baseline they compare against is 82.0%. The same pipeline configuration was not achieved, or the evaluation protocol differs, or the [15] result is not exactly reproduced. Because the advertised margin is 5.66 points, a 2.2-point discrepancy in the baseline alone is material. In addition, no variance or error bars are reported for the accuracies, and the best MLP configuration (K=5, A=10) was chosen after running the test grid; Section IV-B even states the classification augmentation set was reduced after observing overfitting in testing, which is a form of test-set selection. These issues do not prove the MLP gain is spurious, but they make the central claim unverified. The reader's weakest assumption identifies exactly this comparability problem; I agree. The fix is a controlled head-to-head with fixed crops, features, and splits, plus multiple seeds, and a rerun of the [15] baseline under its original protocol. If the matched comparison still shows an MLP edge of roughly 5.66 points with non-overlapping confidence intervals, the paper's claim would be substantially strengthened. Until then, CONDITIONAL is the appropriate verdict, and my stress test does not move it.","tokens_in":7068,"tokens_out":3324,"duration_ms":30194,"concrete_test":"Run a controlled head-to-head under one shared protocol: use identical extracted crops, identical VICReg feature extractor, identical N-way-K-shot splits, and identical augmentation (A=0 and A=10) for kNN and MLP. Compute accuracy over at least 10 random support/query splits and report mean ± 95% CI. Also rerun the kNN baseline using the exact crop-extraction and feature-training procedure of [15] to see whether the reproduced 79.8% moves to 82.0%. If the kNN and MLP accuracies overlap within CI at K=5, or if the MLP advantage over the matched kNN is not statistically significant, the 'surpassing by 5.66%' claim is not supported. If the [15] reproduction remains at 79.8%, then the comparison in Section V must be revised to use the reproduced kNN value as the baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim (Section V) is that an MLP on self-supervised VICReg features reaches 87.66% at five samples per class, 'surpassing the kNN baseline by 5.66% from Alfaro-Contreras et al. (82.0%)'. This claim is only meaningful if the comparison isolates the classifier. It does not. The authors state that they 'maintained a similar pipeline configuration' (Section IV-A), but Table I(a) shows their own kNN at K=5 with no augmentation scores 79.8%, not the 82.0% reported in [15]. That 2.2-point gap is of the same order as many of the classifier differences in Table I, so differences in crop extraction, feature training, test splits, or evaluation details could account for the advertised margin before the classifier is even considered. Moreover, the best MLP number (87.66% at K=5, A=10) is reported without error bars or variance; all entries are 'mean' values but no uncertainty or number of seeds is specified. The augmentation level A=10 was selected from the test grid, and Section IV-B describes choosing the reduced augmentation set after observing that the full set 'led to overfitting' in testing, so the headline configuration is partly selected on the test set. Therefore, the 5.66% advantage is not yet established as an effect of the MLP architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage pipeline for few-shot classification of musical symbols in historical manuscripts: sliding-window crop extraction with binarization and entropy filtering, a VICReg self-supervised CNN feature extractor trained on unlabeled crops, and a downstream classifier (kNN, SVM, MLP, or prototypical network). Using the Capitan dataset with 28 symbol classes, the authors evaluate accuracy for 1, 3, 5, and 10 samples per class and augmentation counts from 0 to 20. The headline result is an MLP accuracy of 87.66% at five samples per class, claimed to surpass the kNN baseline of Alfaro-Contreras et al. [15] by 5.66 percentage points. The paper also reports preprocessing choices and confusion-matrix analyses of difficult symbol classes.","tokens_in":7363,"tokens_out":8845,"duration_ms":80688,"significance":"If the headline comparison were clean, the result would be a useful applied contribution: it would show that a simple MLP trained on self-supervised VICReg features outperforms a nearest-neighbor baseline for low-data historical music symbol classification, with practical implications for digitizing cultural heritage. The paper's strengths are its use of an established historical-music benchmark (Capitan), its systematic sweep over classifiers and augmentation levels, and its inclusion of class-level diagnostics. The contribution is incremental rather than conceptual, and the current evidence is not yet sufficient to establish the claimed improvement because the baseline comparison is uncontrolled and no uncertainty quantification is provided.","major_comments":[{"comment":"The central claim that the MLP surpasses the Alfaro-Contreras et al. baseline by 5.66% is not supported as stated, because the comparison does not isolate the classifier. Table I(a) reports the authors' own kNN reproduction at K=5 with no augmentation as 79.8%, whereas the baseline column cites 82.0% from [15]; similar gaps appear at K=1 (60.7 vs 67.2) and K=10 (86.1 vs 86.9). These gaps are of the same magnitude as many classifier differences in Table I, so the advertised 5.66% improvement may reflect differences in crop extraction, feature training, evaluation splits, or other pipeline details rather than the MLP. The authors should reproduce the [15] baseline in their own pipeline or compare the MLP against their own kNN under identical conditions, and they should report the number of classes N in the N-way-K-shot evaluation and the support/query construction used for kNN, SVM, and MLP.","section":"§V, Table I(a)"},{"comment":"All accuracy numbers are reported only as means, with no standard deviation, number of runs or seeds, or statistical tests. In a few-shot benchmark the support set is usually resampled, and Figure 4 indicates that bootstrapping was used at least for the confusion matrix; without uncertainty estimates the reader cannot judge whether the differences between MLP values (e.g., 86.79 at A=0 vs 87.66 at A=10 for K=5) or between MLP and SVM are real. Please report mean ± std over multiple seeds or resampled support/query splits, and provide a paired test or interval for the main comparisons.","section":"Table I, §V"},{"comment":"The augmentation policy and the best augmentation count are selected using test-set information. Section IV-B states that the authors observed during testing that the full transformation set 'led to overfitting' and therefore switched to a reduced set, and the headline result of 87.66% is the best A=10 entry in Table I(c). Selecting the augmentation type and count on the test set makes the reported accuracy an optimistic selection result. The authors should make these choices on a validation split derived from the labelled training data, or present the full grid of results without designating the best test entry as the expected performance.","section":"§IV-B, Table I(c)"}],"minor_comments":[{"comment":"The text says K values of 1, 2, 5, and 10 were used, but Table I and Figure 3 show rows for 1, 3, 5, and 10; please correct the text or the table.","section":"§IV-A, Table I"},{"comment":"Section III-C states that horizontal flipping was excluded, but Figure 2 lists 'Random horizontal flip' among the transformations used prior to training; clarify which transformations were used in self-supervised pretraining and which in classification-stage augmentation.","section":"§III-C, Fig. 2"},{"comment":"References [16] and [17] are the same paper; [1], [3], [4], [5], [7], [10], [11], [13], and [14] are not cited in the text; and Section II's discussion of Reptile cites [6] (MAML) rather than [7] (Nichol et al.).","section":"References"},{"comment":"The notation for the number of samples per class switches between K (Table I, Figure 3) and L (Section V, 'L > 3'); please use one symbol throughout.","section":"§V"},{"comment":"Figure 4 reports an average over five bootstraps, but Table I's means are not described as bootstrapped; state the averaging procedure for all results.","section":"Table I, Fig. 4"},{"comment":"The CNN architecture and training details for VICReg (depth, width, epochs, batch size, optimizer, learning rate) and the MLP hyperparameters (optimizer, learning rate, dropout rate) are not specified; add these for reproducibility.","section":"§III-B, §IV-A"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well placed: Table I(a)'s reproduced kNN differs from the cited baseline, so the 5.66% claim should not be accepted as-is. I recommend major revision rather than rejection because the flaw is fixable within the manuscript's scope by redoing the comparison and adding uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a systematic, readable extension of Alfaro-Contreras et al.'s self-supervised VICReg + kNN pipeline for few-shot OMR. The authors swap the classifier (SVM, MLP, prototypical), vary augmentation levels, and evaluate on the Capitan dataset. The headline MLP accuracy of 87.66% at five samples per class is plausible, but the way it is compared to the published 82.0% baseline is the paper's main soft spot.\n\nThe paper does several things well. It is transparent about building on [15], shares full tables rather than cherry-picked numbers, and the internal comparison is actually informative: at matched augmentation levels, MLP beats their own kNN by roughly 4–7 points across K values. So the central claim—that a simple MLP on self-supervised features improves over nearest neighbours—is probably true. The augmentation findings (reduced set helps, too much hurts) are also practically useful for the OMR community.\n\nWhere it gets shaky. First, the comparison to the [15] baseline is not apples-to-apples. Their own kNN reproduction at K=5, no augmentation is 79.8%, not 82.0%, so the advertised 5.66% gain mixes pipeline differences with classifier effects. That said, the stress-test concern that the gain might vanish is too strong: the internal table shows MLP beating their own kNN even at A=0, so the classifier effect survives. The real problems are statistical. There are no error bars, no seeds, no significance tests. The best MLP setting (A=10) was selected from the test grid, and the reduced augmentation set was chosen after observing test performance, so the headline configuration is partly tuned on the test set. That is a legitimate concern for a paper whose main contribution is a number.\n\nWho is this for? People doing few-shot OMR or evaluating simple classifiers on self-supervised features. It is not a new method, but it is a useful benchmark. I would not cite it for the 87.66% figure until the authors add variance and a properly controlled baseline. Still, it deserves a serious referee: the work is coherent, the experiments are reproducible enough, and the flaws are fixable with better statistics and a redone comparison.\n\nRecommendation: send it to peer review, but ask for error bars, a direct kNN reproduction, and a clear statement that the augmentation level and recipe were selected on validation, not test.","headline":"A modest, honest extension of Alfaro-Contreras et al. that swaps in new classifiers and tunes augmentations, but the headline comparison to the published baseline is not clean and the result lacks error bars.","tokens_in":7926,"tokens_out":2889,"would_cite":false,"duration_ms":29315,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-layer MLP on VICReg self-supervised features achieves 87.66% accuracy in few-shot classification of historical music symbols, surpassing the kNN baseline by 5.66 points.","keywords":["optical music recognition","few-shot learning","self-supervised learning","VICReg","historical manuscripts","music symbol classification","multilayer perceptron"],"falsifier":"Run the MLP classifier under the exact evaluation protocol of the baseline study—same crops, same train/test splits, same augmentation settings—and compare its accuracy to the baseline's 82.0% kNN result, and to the paper's own reproduced kNN accuracy of 79.8% at K=5; if the margins shrink or reverse, the claimed improvement is an artifact of protocol differences.","tokens_in":6841,"feed_emoji":"🎼","tokens_out":7689,"duration_ms":61107,"temperature":0.7,"pith_summary":"The paper claims that a simple five-layer multilayer perceptron, trained on features extracted by a VICReg self-supervised convolutional network, classifies musical symbols in historical manuscripts at 87.66% accuracy when only five labeled examples per class are available. This would improve on the kNN baseline from the prior pipeline by 5.66 percentage points. The work is motivated by the scarcity of labeled data in historical music documents, where manual annotation is costly. If the claim holds, it suggests that self-supervised feature extraction plus a lightweight classifier is a practical route for few-shot optical music recognition.","feed_headline":"MLP on self-supervised features hits 87.66% in few-shot music symbols","feed_subtitle":"A five-layer MLP over VICReg features beats the kNN baseline by 5.66 points on the Capitan manuscript dataset.","key_machinery":"The key machinery is a two-stage pipeline. First, a convolutional neural network is trained with VICReg, a self-supervised method that distorts each crop twice and enforces variance, invariance, and covariance regularization so the distortions map to a shared feature-space point; this yields a 1600-dimensional feature vector per crop without using labels. Second, a five-layer MLP with batch normalization and dropout maps those features to the output classes. Around the pipeline sit crop preprocessing steps (Sauvola adaptive binarization, sliding-window extraction, entropy filtering) and a curated set of data augmentations that simulate historical degradation.","core_discovery":"The paper's central claim is that a five-layer MLP trained on VICReg self-supervised features classifies musical symbols in the Capitan dataset at 87.66% mean accuracy with five samples per class, beating the 82.0% kNN baseline it compares against. The authors also report that moderate augmentation at the classification stage helps, and that the MLP scales better than prototypical networks as the number of samples per class grows.","pith_inferences":["A like-for-like comparison using the paper's own reproduced kNN values (79.8% at K=5) would show a larger margin for the MLP than the 5.66 points claimed against the cited baseline, suggesting the classifier's advantage may be understated rather than overstated.","The persistent confusion among beamed notes hints that the feature extractor captures global shape but may miss fine structural cues; augmentations that specifically perturb beam connections could test whether the bottleneck is the features or the classifier.","The reduced augmentation set for classification (random resized crop, colour jitter, Gaussian blur) offers a transferable heuristic: pretrain with heavy degradation simulation, but fine-tune the head with mild augmentations to avoid overfitting."],"forward_implications":["A lightweight classifier on self-supervised features can serve as a strong baseline for few-shot optical music recognition, reducing the need for large annotated corpora.","The reported accuracy gap suggests that non-linear classifiers are preferable to kNN in the high-dimensional feature spaces produced by VICReg.","The reduced augmentation set during classification offers a practical recipe: use degradation-simulating transformations for self-supervised pretraining, but a smaller set when training the head to avoid overfitting.","The approach could be transferred to other historical document recognition tasks with scarce labels, such as ciphered manuscripts or early printed scores."],"supporting_citations":[{"why":"Supplies the self-supervised pipeline and the kNN baseline (82.0%) that the MLP result is compared against.","marker":"[15]"},{"why":"Defines VICReg, the self-supervised regularisation method used to train the feature extractor.","marker":"[21]"},{"why":"Provides the Capitan dataset of mensural notation with 17,112 labelled symbols across 28 classes.","marker":"[24]"},{"why":"Supplies the Sauvola adaptive binarisation method used in crop extraction.","marker":"[20]"},{"why":"Documents the image augmentation transformations (salt-and-pepper noise, elastic distortion, fade) used to simulate historical degradation.","marker":"[12]"},{"why":"Provides the VICReg regularisation parameter values (λ=10, μ=10, φ=1) used in the configuration.","marker":"[16]"}],"fun_headline_variants":["Self-supervised MLP classifies historical music symbols at 87.66%","Few-shot music symbol classification hits 87.66% with self-supervised features","MLP on VICReg features beats kNN by 5.66 points in few-shot music","Self-supervised learning rescues historical music classification from data scarcity","87.66% accuracy in few-shot musical symbol classification via self-supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the way crops were extracted, features were trained, and test data were split in this study matches the baseline study's protocol well enough that the 5.66-point accuracy difference is caused by the classifier choice.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised MLP classifies historical music symbols at 87.66%","Few-shot music symbol classification hits 87.66% with self-supervised features","MLP on VICReg features beats kNN by 5.66 points in few-shot music","Self-supervised learning rescues historical music classification from data scarcity","87.66% accuracy in few-shot musical symbol classification via self-supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3250,"prompt_tokens":777,"completion_tokens":2473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":393,"completion_tokens_details":{"reasoning_tokens":2367}},"tokens_in":393,"tokens_out":2473,"duration_ms":16953,"temperature":1.0,"reasoning_tokens":2367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:08:00.191784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MLP classifier under the exact evaluation protocol of the baseline study—same crops, same train/test splits, same augmentation settings—and compare its accuracy to the baseline's 82.0% kNN result, and to the paper's own reproduced kNN accuracy of 79.8% at K=5; if the margins shrink or reverse, the claimed improvement is an artifact of protocol differences.","supporting_citations":[{"cited_title":"Few-shot symbol classification via self-supervised learning and nearest neighbor","cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised pipeline and the kNN baseline (82.0%) that the MLP result is compared against."},{"cited_title":"Handwritten music recognition for mensural notation with convolutional recurrent neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the Capitan dataset of mensural notation with 17,112 labelled symbols across 28 classes."},{"cited_title":"Adaptive docu- ment binarization","cited_arxiv_id":null,"evidence_quote":"Supplies the Sauvola adaptive binarisation method used in crop extraction."},{"cited_title":"A survey on image data augmentation for deep learning","cited_arxiv_id":null,"evidence_quote":"Documents the image augmentation transformations (salt-and-pepper noise, elastic distortion, fade) used to simulate historical degradation."}],"review_version":1}