{"id":"68aba661-de4f-42c2-b328-0f8db3a35fa3","arxiv_id":"2507.18741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"KuiSCIMA v2.0 reports improved CNN classifiers that lower suzipu CER to 7.1% and lülüpu CER to 0.9% on historical Chinese music notation, with a dataset extension to all 109 pieces.","lead":"This paper improves automated recognition of symbols in Jiang Kui's 1202 Chinese music manuscript, cutting character error from 10.4% to 7.1% for suzipu notation and achieving 0.9% error for lülüpu notation. It also expands the KuiSCIMA dataset to all 109 pieces and shows the models beat naive human annotators.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported CER gains rest entirely on a contested secondary-symbol taxonomy; a direct human-expert relabeling study on a held-out subset would settle whether the 7.1% CER is meaningful recognition or partial fitting to one annotator's labels.","rationale":"The reader's weakest_assumption identified exactly the contested ground-truth taxonomy. My stress-test concurs: that is the load-bearing point. The suzipu CER numbers (6.6% aggregated, 7.1% Shanghai MS) are computed against labels whose correctness is explicitly disputed even by experts, and the human-competitor comparison uses naive participants, so it cannot independently validate the labels. The internal protocol (leave-one-edition-out, 10 model samples, validation-based model selection) is otherwise sound and disclosed transparently; model selection, augmentation, and focal loss are all reasonable for the stated setting. The lülüpu result (0.9% CER over 17 standard classes) is almost surely unaffected by the taxonomy issue because those classes are uncontroversial standard characters, and the artificial-font data augmentation further supports robustness. No internal inconsistency or fraud is alleged; the concern is about the external validity of the headline suzipu number under label uncertainty, plus the weaker-than-advertised human baseline comparison. A targeted expert-relabeling test would settle the question directly, so the correct verdict remains CONDITIONAL rather than REJECT or ACCEPT. I also note the paper's honest disclosure of dataset/code availability after acceptance and its own limitation statements, which align with the reader's framing of addressable issues.","tokens_in":13900,"tokens_out":2723,"duration_ms":23412,"concrete_test":"Have at least two independent domain experts in suzipu re-annotate a held-out stratified sample of the Shanghai MS suzipu instances (e.g., 300-500 patches, oversampling rare secondary classes) using their own taxonomy choices, then compute the model's CER against each expert's labels and the experts' inter-annotator agreement. If model CER against expert labels stays below roughly 8% and expert agreement is comparably high, the taxonomy concern is largely resolved; if model CER rises toward the 10-15% range or experts disagree at rates comparable to reported CER, the suzipu improvement is partially an artifact of a single contested label scheme.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is the suzipu CER improvement from 10.4% to 7.1% (Shanghai MS, Table 2 bottom) and the lülüpu CER of 0.9% (Table 3 bottom). The lülüpu claim is robust: its 17 classes are standard characters, the leave-one-edition-out protocol is sound, and the artificial-data variant reduces variance. The suzipu claim, however, is measured only against KuiSCIMA v2.0 ground truth built on Wu Santu's taxonomy of seven secondary classes, and Section 3 explicitly concedes that 'even domain experts do not agree about the number or semantics of these secondary symbols' and that symbol appearance is 'often ambiguous and may vary considerably'. CER is a label-matching metric: if the labels are contested, a model cannot be scored against 'the historically correct notation' — only against one annotator's choices. This is not an internal inconsistency, but it is the single condition under which the headline numbers lose their stated meaning. It is also empirically plausible to matter: Figure 6 shows many classes with near-zero F1 (e.g., Gong/Xiaozhu), and class frequencies are heavily imbalanced (e.g., only 1-6 instances for several classes); small labeling disagreements on rare classes shift CER nontrivially. The models may still be genuinely useful for assisted annotation — the authors themselves frame it that way — but the claim 'reduce CER from 10.4% to 7.1%' and the comparison to human transcribers inherit the taxonomy problem. The human comparison is also weakened by recruiting naive participants with no Chinese or suzipu training: it shows superiority over novice transcribers, not over experts, so it cannot validate the contested labels. The paper correctly discloses these limitations; they do not refute the engineering result, but they cap what the central claim can assert.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents KuiSCIMA v2.0, an extension of a historical Chinese music notation dataset, and evaluates compact factored CNNs for isolated symbol classification of suzipu and lülüpu. The evaluation uses leave-one-edition-out cross-validation over five historical editions, with data augmentation for suzipu and an additional synthetic-font training condition for lülüpu. The headline results are a suzipu CER reduction from 10.4% to 7.1% on the Shanghai MS and a 0.9% CER for lülüpu, together with temperature-scaling calibration (ECE below 0.0162) and a comparison against naive human participants, whose average CER is 15.9% and whose best CER is 7.6%. The paper also reports UMAP-based similarity visualizations for an annotation tool.","tokens_in":14327,"tokens_out":2903,"duration_ms":33055,"significance":"If the suzipu result holds as a statement about recognition quality, the paper provides a useful baseline for an under-resourced OMR area, and the lülüpu result is convincing: the 17 lülüpu classes are standard characters, the leave-one-edition-out protocol is sound, and the synthetic-font augmentation visibly reduces variance. The paper is also commendable for shipping open data and code, for using a held-out edition protocol, and for reporting calibration rather than only accuracy. The main caveat is that the suzipu headline CER is measured against a secondary-symbol taxonomy that the paper itself describes as contested among experts, so the number should be read as agreement with one annotator's labels rather than as recognition of historically correct notation.","major_comments":[{"comment":"The paper needs to reconcile this with the abstract's unqualified claim of reducing suzipu CER from 10.4% to 7.1%.","section":"§3, Table 2, Figure 6"},{"comment":"The suzipu experiments exclude 133 of 7297 instances from training because of anomalous shapes, but the paper gives no sensitivity analysis and does not state whether these instances remain in the test sets. If they are excluded from both training and test, the comparison to the 10.4% baseline in [17] may be unfair if that baseline included them. Please report the CER with and without the excluded instances, and clarify exactly which split contains them.","section":"§4.2"},{"comment":"The Tesseract comparison is internally inconsistent. Section 4.3 reports CER values of 42.4%, 42.1%, and 44.0% for page segmentation modes 6, 7, and 8, but Section 6 states that 'The tesseract CER model's performance is not comparable (CER = 51.6%)' without explaining where 51.6% comes from. Please reconcile these numbers or remove the unexplained value.","section":"§4.3 and §6"},{"comment":"The claim that the models 'outperform human transcribers' overstates what the study shows. The participants in Section 5 were naive to suzipu notation and the Chinese language, so the comparison demonstrates superiority over minimally trained non-experts, not over human transcribers in any expert sense. The best participant's CER of 7.6% is also close to the model's 7.1%, so the claim should be qualified by the participants' background and by the lack of an expert benchmark.","section":"§5 and §6"}],"minor_comments":[{"comment":"The paper is titled and framed as Optical Music Recognition, but the experiments classify pre-extracted 48x48 symbol patches; layout analysis, symbol detection, and segmentation are not evaluated. Please state more precisely that the reported results concern isolated symbol classification, and discuss what is needed for a full OMR pipeline.","section":"Title and §4"},{"comment":"There is a typo: 'Of the result, 5168 are simple symbols and 1996 are composite symbols' should read 'Of the remaining instances' or similar.","section":"§4.2"},{"comment":"The 'Aggregated' row is hard to read because the validation and test accuracy columns do not match the per-edition columns; please clarify what is being averaged and over how many model samples.","section":"Table 2"},{"comment":"The caption says 'the best two human users' while Table 4 reports only the single best user; clarify which users are shown in the heatmaps.","section":"Figure 6"},{"comment":"The ethics statement contains placeholders ('Name will be disclosed after paper acceptance (reference number 00000000)'); these need to be filled in or removed before publication.","section":"§5.3"},{"comment":"The text repeatedly states that source code and data are publicly available, but the provided links are placeholders ('The link will be published after paper acceptance'). Please include repository or DOI references.","section":"§4 and §7"},{"comment":"The phrase 'The tesseract CER model's performance is not comparable' is grammatically unclear; it should be 'Tesseract's CER performance is not comparable'.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The core evaluation protocol is sound and the lülüpu result is robust, but the suzipu claim needs either an expert-agreement study or a careful reframing; the human comparison also needs qualification. The Tesseract inconsistency must be fixed. These are addressable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a good, honest engineering paper for a genuinely underserved niche. The dataset extension to all 109 pieces of Baishidaoren Gequ is real value, and the leave-one-edition-out protocol is the right way to test robustness across handwritten editions. The lülüpu result is solid: 17 standard classes, clean evaluation, and the synthetic-font augmentation reduces variance, giving an aggregated CER of 0.9%. That part I trust. The factored CNN for suzipu is also sound, and the comparison with the prior KuiSCIMA baseline (10.4% to 7.1% on Shanghai MS) is a fair improvement claim.\n\nWhere I land close to the stress-test note: the suzipu CER is measured against the Wu Santu secondary-symbol taxonomy, and the paper itself concedes that domain experts disagree about those classes. That means the reported error rates are scores against one annotator's choices, not against \"the historically correct notation.\" I don't think this refutes the engineering result, but it caps what the paper can claim. The authors are transparent about it, which I respect. The human comparison is weakened by using naive participants with no Chinese or suzipu training—it shows the model beats novices, not experts, so it cannot validate the contested labels.\n\nOther soft spots are real but minor: the 133 excluded anomalous instances get no sensitivity analysis; the Tesseract CER numbers are inconsistent (42–44% in the experiment section, 51.6% later); and the code/data links are promised only after acceptance, so right now the paper is not reproducible as written. Those are all fixable in revision.\n\nI disagree with the reader's slight overclaim that the OMR framing is misleading. The paper is explicitly about symbol classification with patch inputs, and it future-works end-to-end OMR. That's fine for a niche venue.\n\nBottom line: the central engineering claims hold up, especially for lülüpu. The paper deserves a serious referee. I would suggest the referee push for a label-reliability study—even a small expert relabeling exercise on one edition—and a sensitivity analysis on the excluded samples. With those, the suzipu claim would be much harder to pick at.\n\nFor my own reading group, I'd probably mention it rather than assign it. I would cite it if I worked on OMR for East Asian notations, and I'd accept it for peer review without hesitation.","headline":"A solid, honest engineering paper for a niche OMR problem: the lülüpu result is robust and the dataset extension is valuable, but the suzipu headline numbers rest on a contested label taxonomy that the authors disclose but do not resolve.","tokens_in":14812,"tokens_out":1248,"would_cite":true,"duration_ms":15626,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine reading of 1202 Chinese music scores cuts errors to 7.1% for suzipu and 0.9% for lülüpu.","keywords":["Optical music recognition","suzipu","lülüpu","Jiang Kui","Baishidaoren Gequ","character error rate","class imbalance","historical Chinese music"],"falsifier":"Have an independent panel of suzipu experts re-annotate the disputed secondary symbols and recompute the CER; if the model's errors cluster on symbols whose ground-truth class changes under the panel's labels, the 7.1% claim does not measure historically correct recognition. Alternatively, run the trained models on a newly digitized manuscript edition not among the five; a CER far above the reported 4.6-9.0% range would falsify the cross-edition generalization claim.","tokens_in":13723,"feed_emoji":"🎼","tokens_out":5445,"duration_ms":53353,"temperature":0.7,"pith_summary":"The paper claims that a compact convolutional network, structured as a pair of factorized classifiers for pitch and ornament, can recognize the handwritten notations suzipu and lülüpu in Jiang Kui's Baishidaoren Gequ collection from 1202 well enough to support practical transcription. On the five historical editions tested by leaving one edition out, the best suzipu model reaches 7.1% character error rate on the Shanghai manuscript, down from the previous 10.4% baseline, while lülüpu falls to 0.9% with the help of artificial font-rendered training data. The models also beat the 15.9% average error of minimally trained human annotators studied in the paper, and their confidence is calibrated well enough that an annotation tool can rely on it. If these numbers hold, machine-assisted transcription of these historically important notations becomes feasible.","feed_headline":"Suzipu and lülüpu OCR errors drop to 7.1% and 0.9%","feed_subtitle":"A compact CNN reads Jiang Kui's 1202 music notations more accurately than the average human transcriber.","key_machinery":"The central machinery is a factored convolutional network: the suzipu classifier is a pair of small CNNs (three convolutional layers, two fully connected layers, 48x48 input) that predict pitch and secondary component separately, exploiting the notation's compositional structure rather than treating all 77 combinations as flat classes. Training uses uniform per-class sampling with replacement, aggressive geometric augmentation (random scale, rotation, crop), and focal loss to keep rare samples informative; evaluation is leave-one-edition-out across five editions. Temperature scaling then recalibrates softmax confidence, giving ECE10 below 0.0162. For lülüpu the same architecture is trained with and without synthetic training images rendered from four Chinese fonts, which reduces both error and variance.","core_discovery":"The load-bearing result is that treating a suzipu symbol as a product of an 11-way pitch component and a 7-way secondary component, and classifying the two parts with separate small CNNs, makes a scarce, imbalanced 77-class recognition problem tractable. In leave-one-edition-out evaluation the aggregated suzipu CER is 6.6% and per-edition best-model CERs lie between 4.6% and 9.0%; the Shanghai manuscript, the comparison point with the old baseline, goes from 10.4% to 7.1%. For lülüpu, a 17-class character set, the paper reports an aggregated CER of 0.9% when training is augmented with images rendered in four computer fonts, and shows that off-the-shelf Tesseract OCR fails at 42-44% on the same task. The per-class F1 analysis shows model and human errors concentrate on the same rare classes, while the Zhu edition is the hardest generalization target for every model.","pith_inferences":["Because the paper itself reports that experts disagree on suzipu secondary-symbol semantics, the 7.1% figure is a recognition rate against Wu Santu's scheme, not against a historically undisputed ground truth; re-labeling under another expert scheme could shift the measured CER even if the model is unchanged.","The Zhu edition's consistently worse accuracy across both notations suggests a distribution shift that the current one-model-per-edition scheme does not absorb; a testable extension is to fine-tune on a small Zhu-sample and measure whether the gap closes.","The font-rendering augmentation used for lülüpu could plausibly transfer to other low-resource Chinese-character notations such as gongchepu, where printed characters are available even when handwritten exemplars are scarce.","The reported CPU inference times (about 2 seconds for the full suzipu edition and 0.5 seconds for lülüpu) imply that interactive, real-time annotation is within reach; one could measure the end-to-end annotation speed-up in the tool rather than per-image latency."],"forward_implications":["At 7.1% suzipu CER on the Shanghai manuscript, a human annotator corrects roughly one in fourteen symbols instead of transcribing every symbol.","At 0.9% lülüpu CER, expert correction is needed for about one character in a hundred, making full-collection transcription practical.","The leave-one-edition-out design shows the models transfer to unseen handwritten editions, with Zhu as the consistently hardest case.","Calibrated confidence (ECE below 0.0162) lets the annotation interface route low-confidence predictions to human review rather than treating all outputs equally.","The KuiSCIMA v2.0 extension to all 109 pieces, including jianzipu instances, supplies a common benchmark for future OMR work on these notations."],"supporting_citations":[{"why":"Supplies the original KuiSCIMA dataset and the 10.4% CER baseline that the suzipu models improve upon.","marker":"[17]"},{"why":"Supplies the factored CNN product-space decomposition that the suzipu pitch/secondary pair of classifiers is modeled on.","marker":"[1]"},{"why":"Focal loss, used to keep learning from rare classes under the 77-class imbalance.","marker":"[12]"},{"why":"Defines the seven-class suzipu secondary-symbol taxonomy used as ground truth for the reported CER.","marker":"[25]"},{"why":"Supplies the user-study design and human-level performance comparison used for the suzipu annotation experiment.","marker":"[2]"},{"why":"Underpins the interpretation that the Zhu edition's worse results are due to distribution shift across editions.","marker":"[14]"},{"why":"Tesseract OCR is the out-of-the-box baseline for the lülüpu comparison, showing 42-44% CER.","marker":"[10]"}],"fun_headline_variants":["Chinese music OCR: suzipu error cut to 7.1%, lülüpu to 0.9%","Suzipu and lülüpu recognition: 7.1% and 0.9% CER, better than human","Decomposing 77-class suzipu into pitch+secondary cuts errors by 30%","Leave-one-edition-out test proves robust OCR for Jiang Kui's 1202 notation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The suzipu ground-truth labels follow one expert's seven-class scheme for secondary symbols even though the paper states that domain experts disagree on the number and semantics of those symbols; if that scheme is historically wrong, the reported error rates are not measuring recognition of the historically correct notation.","fun_headline_variants_meta":{"raw":{"variants":["Chinese music OCR: suzipu error cut to 7.1%, lülüpu to 0.9%","Suzipu and lülüpu recognition: 7.1% and 0.9% CER, better than human","Decomposing 77-class suzipu into pitch+secondary cuts errors by 30%","Leave-one-edition-out test proves robust OCR for Jiang Kui's 1202 notation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2800,"prompt_tokens":1051,"completion_tokens":1749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":1637}},"tokens_in":667,"tokens_out":1749,"duration_ms":13327,"temperature":1.0,"reasoning_tokens":1637,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:31:46.983496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent panel of suzipu experts re-annotate the disputed secondary symbols and recompute the CER; if the model's errors cluster on symbols whose ground-truth class changes under the panel's labels, the 7.1% claim does not measure historically correct recognition. Alternatively, run the trained models on a newly digitized manuscript edition not among the five; a CER far above the reported 4.6-9.0% range would falsify the cross-edition generalization claim.","supporting_citations":[{"cited_title":"The KuiSCIMA Dataset for Optical Music Recognition of Ancient Chinese Suzipu Notation","cited_arxiv_id":null,"evidence_quote":"Supplies the original KuiSCIMA dataset and the 10.4% CER baseline that the suzipu models improve upon."},{"cited_title":"Factored Convolutional Neural Network for Amharic Character Image Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the factored CNN product-space decomposition that the suzipu pitch/secondary pair of classifiers is modeled on."},{"cited_title":"Songci Yinyue Zhuanti Yanjiu 宋词音乐专题研究","cited_arxiv_id":null,"evidence_quote":"Defines the seven-class suzipu secondary-symbol taxonomy used as ground truth for the reported CER."},{"cited_title":"A Historical Handwritten Dataset for Ethiopic OCR with Baseline Models and Human-Level Performance","cited_arxiv_id":null,"evidence_quote":"Supplies the user-study design and human-level performance comparison used for the suzipu annotation experiment."},{"cited_title":"Tesseract: an open-source optical character recognition en- gine","cited_arxiv_id":null,"evidence_quote":"Tesseract OCR is the out-of-the-box baseline for the lülüpu comparison, showing 42-44% CER."}],"review_version":1}