REVIEW 4 major objections 7 minor 26 references
KuiSCIMA v2.0: Improved Baselines, Calibration, and Cross-Notation Generalization for Historical Chinese Music Notations in Jiang Kui's Baishidaoren Gequ
T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Machine reading of 1202 Chinese music scores cuts errors to 7.1% for suzipu and 0.9% for lülüpu.
desk verdict A solid, honest engineering paper for a niche OMR problem: the lülüpu result is robust and the dataset extension is valuable, but the suzipu headline numbers rest on a contested label taxonomy that the authors disclose but do not resolve. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a factored convolutional network: the suzipu classifier is a pair of small CNNs (three convolutional layers, two fully connected layers, 48x48 input) that predict pitch and secondary component separately, exploiting the notation's compositional structure rather than treating all 77 combinations as flat classes. Training uses uniform per-class sampling with replacement, aggressive geometric augmentation (random scale, rotation, crop), and focal loss to keep rare samples informative; evaluation is leave-one-edition-out across five editions. Temperature scaling then recalibrates softmax confidence, giving ECE10 below 0.0162. For lülüpu the same architecture is trained with and without synthetic training images rendered from four Chinese fonts, which reduces both error and variance.
What would settle it
Have an independent panel of suzipu experts re-annotate the disputed secondary symbols and recompute the CER; if the model's errors cluster on symbols whose ground-truth class changes under the panel's labels, the 7.1% claim does not measure historically correct recognition. Alternatively, run the trained models on a newly digitized manuscript edition not among the five; a CER far above the reported 4.6-9.0% range would falsify the cross-edition generalization claim.
Extended reading notes
Core claim
The load-bearing result is that treating a suzipu symbol as a product of an 11-way pitch component and a 7-way secondary component, and classifying the two parts with separate small CNNs, makes a scarce, imbalanced 77-class recognition problem tractable. In leave-one-edition-out evaluation the aggregated suzipu CER is 6.6% and per-edition best-model CERs lie between 4.6% and 9.0%; the Shanghai manuscript, the comparison point with the old baseline, goes from 10.4% to 7.1%. For lülüpu, a 17-class character set, the paper reports an aggregated CER of 0.9% when training is augmented with images rendered in four computer fonts, and shows that off-the-shelf Tesseract OCR fails at 42-44% on the same task. The per-class F1 analysis shows model and human errors concentrate on the same rare classes, while the Zhu edition is the hardest generalization target for every model.
Load-bearing premise
The suzipu ground-truth labels follow one expert's seven-class scheme for secondary symbols even though the paper states that domain experts disagree on the number and semantics of those symbols; if that scheme is historically wrong, the reported error rates are not measuring recognition of the historically correct notation.
Editorial extensions
If this is right
- At 7.1% suzipu CER on the Shanghai manuscript, a human annotator corrects roughly one in fourteen symbols instead of transcribing every symbol.
- At 0.9% lülüpu CER, expert correction is needed for about one character in a hundred, making full-collection transcription practical.
- The leave-one-edition-out design shows the models transfer to unseen handwritten editions, with Zhu as the consistently hardest case.
- Calibrated confidence (ECE below 0.0162) lets the annotation interface route low-confidence predictions to human review rather than treating all outputs equally.
- The KuiSCIMA v2.0 extension to all 109 pieces, including jianzipu instances, supplies a common benchmark for future OMR work on these notations.
Reading between the lines
- Because the paper itself reports that experts disagree on suzipu secondary-symbol semantics, the 7.1% figure is a recognition rate against Wu Santu's scheme, not against a historically undisputed ground truth; re-labeling under another expert scheme could shift the measured CER even if the model is unchanged.
- The Zhu edition's consistently worse accuracy across both notations suggests a distribution shift that the current one-model-per-edition scheme does not absorb; a testable extension is to fine-tune on a small Zhu-sample and measure whether the gap closes.
- The font-rendering augmentation used for lülüpu could plausibly transfer to other low-resource Chinese-character notations such as gongchepu, where printed characters are available even when handwritten exemplars are scarce.
- The reported CPU inference times (about 2 seconds for the full suzipu edition and 0.5 seconds for lülüpu) imply that interactive, real-time annotation is within reach; one could measure the end-to-end annotation speed-up in the tool rather than per-image latency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents KuiSCIMA v2.0, an extension of a historical Chinese music notation dataset, and evaluates compact factored CNNs for isolated symbol classification of suzipu and lülüpu. The evaluation uses leave-one-edition-out cross-validation over five historical editions, with data augmentation for suzipu and an additional synthetic-font training condition for lülüpu. The headline results are a suzipu CER reduction from 10.4% to 7.1% on the Shanghai MS and a 0.9% CER for lülüpu, together with temperature-scaling calibration (ECE below 0.0162) and a comparison against naive human participants, whose average CER is 15.9% and whose best CER is 7.6%. The paper also reports UMAP-based similarity visualizations for an annotation tool.
Significance. If the suzipu result holds as a statement about recognition quality, the paper provides a useful baseline for an under-resourced OMR area, and the lülüpu result is convincing: the 17 lülüpu classes are standard characters, the leave-one-edition-out protocol is sound, and the synthetic-font augmentation visibly reduces variance. The paper is also commendable for shipping open data and code, for using a held-out edition protocol, and for reporting calibration rather than only accuracy. The main caveat is that the suzipu headline CER is measured against a secondary-symbol taxonomy that the paper itself describes as contested among experts, so the number should be read as agreement with one annotator's labels rather than as recognition of historically correct notation.
major comments (4)
- [§3, Table 2, Figure 6] The paper needs to reconcile this with the abstract's unqualified claim of reducing suzipu CER from 10.4% to 7.1%.
- [§4.2] The suzipu experiments exclude 133 of 7297 instances from training because of anomalous shapes, but the paper gives no sensitivity analysis and does not state whether these instances remain in the test sets. If they are excluded from both training and test, the comparison to the 10.4% baseline in [17] may be unfair if that baseline included them. Please report the CER with and without the excluded instances, and clarify exactly which split contains them.
- [§4.3 and §6] The Tesseract comparison is internally inconsistent. Section 4.3 reports CER values of 42.4%, 42.1%, and 44.0% for page segmentation modes 6, 7, and 8, but Section 6 states that 'The tesseract CER model's performance is not comparable (CER = 51.6%)' without explaining where 51.6% comes from. Please reconcile these numbers or remove the unexplained value.
- [§5 and §6] The claim that the models 'outperform human transcribers' overstates what the study shows. The participants in Section 5 were naive to suzipu notation and the Chinese language, so the comparison demonstrates superiority over minimally trained non-experts, not over human transcribers in any expert sense. The best participant's CER of 7.6% is also close to the model's 7.1%, so the claim should be qualified by the participants' background and by the lack of an expert benchmark.
minor comments (7)
- [Title and §4] The paper is titled and framed as Optical Music Recognition, but the experiments classify pre-extracted 48x48 symbol patches; layout analysis, symbol detection, and segmentation are not evaluated. Please state more precisely that the reported results concern isolated symbol classification, and discuss what is needed for a full OMR pipeline.
- [§4.2] There is a typo: 'Of the result, 5168 are simple symbols and 1996 are composite symbols' should read 'Of the remaining instances' or similar.
- [Table 2] The 'Aggregated' row is hard to read because the validation and test accuracy columns do not match the per-edition columns; please clarify what is being averaged and over how many model samples.
- [Figure 6] The caption says 'the best two human users' while Table 4 reports only the single best user; clarify which users are shown in the heatmaps.
- [§5.3] The ethics statement contains placeholders ('Name will be disclosed after paper acceptance (reference number 00000000)'); these need to be filled in or removed before publication.
- [§4 and §7] The text repeatedly states that source code and data are publicly available, but the provided links are placeholders ('The link will be published after paper acceptance'). Please include repository or DOI references.
- [§6] The phrase 'The tesseract CER model's performance is not comparable' is grammatically unclear; it should be 'Tesseract's CER performance is not comparable'.
Circularity Check
No significant circularity: the central CER claims rest on held-out editions and validation-based model selection; self-citations to the prior KuiSCIMA paper are used only as a baseline and methodological reference.
full rationale
The paper's central quantitative claims—suzipu CER reduced from 10.4% to 7.1% and lülüpu CER of 0.9%—are evaluated with a leave-one-edition-out protocol in which each test edition is unseen during training. Training and validation sets are stratified by annotation, and the reported best-model results are selected by validation accuracy, so the test CER is not fitted or re-reported from training. The lülüpu synthetic-font data are used only as training augmentations, not as test data, so the 0.9% result is not an artifact of those synthetic inputs. The comparison to the prior baseline of 10.4% from the authors' earlier KuiSCIMA paper is a legitimate same-group benchmark, not a load-bearing self-citation: the current model's improvement is measured on a held-out edition rather than derived from the earlier paper's equations or predictions. The contested Wu Santu secondary-symbol taxonomy is a labeling-validity concern, not a circularity: the CER is computed against fixed labels, and the model is not defined in terms of that CER, nor is the CER defined in terms of the model. The self-citations to [17] and [18] provide the dataset, preprocessing pipeline, and annotation-tool context, but the central experimental results are self-contained given those resources. Thus no circular step meeting the required evidence standard was found; the minor self-citation for the baseline and preprocessing is not load-bearing.
Assumptions & free parameters
free parameters (10)
- Learning rate =
1e-3 (suzipu), 5e-4 (lülüpu)
- Weight decay =
1e-4
- Focal loss gamma =
1
- Batch size =
100
- Epochs =
80 (suzipu), 50 (lülüpu)
- Rotation augmentation range =
-9 to 9 degrees
- Training resize range =
30-42 px (suzipu), 33-46 px (lülüpu)
- Temperature scaling parameter =
not stated
- Artificial font size =
60
- Dropout fraction =
0.5
assumptions (6)
- domain assumption Ground-truth labels in KuiSCIMA v2.0 are correct and follow Wu Santu's taxonomy for suzipu secondary symbols.
- domain assumption The pre-extracted image patches in KuiSCIMA are correctly localized and cropped to individual notation symbols.
- domain assumption Training on four historical editions generalizes to the held-out fifth edition.
- ad hoc to paper Artificially rendered lülüpu characters from four computer fonts improve rather than distort recognition of handwritten historical samples.
- ad hoc to paper Excluding 133 'anomalous shapes' from suzipu training does not bias the CER comparison.
- domain assumption The 15 naive human participants are a meaningful baseline for 'human-level performance' in the abstract.
Cite this review
Pith. "Pith review of KuiSCIMA v2.0: Improved Baselines, Calibration, and Cross-Notation Generalization for Historical Chinese Music Notations in Jiang Kui's Baishidaoren Gequ." pith.science (2026). https://pith.science/paper/FNTBWLOU
@misc{pith2026250718741,
author = {Pith},
title = {Pith review of: KuiSCIMA v2.0: Improved Baselines, Calibration, and Cross-Notation Generalization for Historical Chinese Music Notations in Jiang Kui's Baishidaoren Gequ},
year = {2026},
howpublished = {\url{https://pith.science/paper/FNTBWLOU}},
note = {Machine review of arXiv:2507.18741}
}
read the original abstract
Optical Music Recognition (OMR) for historical Chinese musical notations, such as suzipu and l\"ul\"upu, presents unique challenges due to high class imbalance and limited training data. This paper introduces significant advancements in OMR for Jiang Kui's influential collection Baishidaoren Gequ from 1202. In this work, we develop and evaluate a character recognition model for scarce imbalanced data. We improve upon previous baselines by reducing the Character Error Rate (CER) from 10.4% to 7.1% for suzipu, despite working with 77 highly imbalanced classes, and achieve a remarkable CER of 0.9% for l\"ul\"upu. Our models outperform human transcribers, with an average human CER of 15.9% and a best-case CER of 7.6%. We employ temperature scaling to achieve a well-calibrated model with an Expected Calibration Error (ECE) below 0.0162. Using a leave-one-edition-out cross-validation approach, we ensure robust performance across five historical editions. Additionally, we extend the KuiSCIMA dataset to include all 109 pieces from Baishidaoren Gequ, encompassing suzipu, l\"ul\"upu, and jianzipu notations. Our findings advance the digitization and accessibility of historical Chinese music, promoting cultural diversity in OMR and expanding its applicability to underrepresented music traditions.
Figures
Reference graph
Works this paper leans on
-
[17]
The KuiSCIMA Dataset for Optical Music Recognition of Ancient Chinese Suzipu Notation
Tristan Repolusk and Eduardo Veas. “The KuiSCIMA Dataset for Optical Music Recognition of Ancient Chinese Suzipu Notation”. In: Document Analysis and Recognition - ICDAR 2024 . Ed. by Elisa H. Barney Smith, Marcus Liwicki, and Liangrui Peng. Cham, Switzerland: Springer Nature Switzerland, 2024, pp. 38–54. doi: 10.1007/978-3-031-70552-6_3
-
[1]
Factored Convolutional Neural Network for Amharic Character Image Recognition
Birhanu Belay et al. “Factored Convolutional Neural Network for Amharic Character Image Recognition”. In: 2019 IEEE International Conference on Image Processing (ICIP). 2019, pp. 2906–2910. doi: 10.1109/ICIP.2019. 8804407
-
[2]
A Historical Handwritten Dataset for Ethiopic OCR with Baseline Models and Human-Level Performance
Birhanu Hailu Belay et al. “A Historical Handwritten Dataset for Ethiopic OCR with Baseline Models and Human-Level Performance”. In: Document Analysis and Recognition - ICDAR 2024 . Ed. by Elisa H. Barney Smith, Marcus Liwicki, and Liangrui Peng. Cham: Springer Nature Switzerland, 2024, pp. 23–38. isbn: 978-3-031-70543-4
work page 2024
-
[3]
GregoBase: A database of Gregorian scores
Olivier Berten. GregoBase: A database of Gregorian scores . 2013. url: https://gregobase.selapa.net
work page 2013
-
[4]
Understand- ing Optical Music Recognition
Jorge Calvo-Zaragoza, Jan Hajic Jr., and Alexander Pacha. “Understand- ing Optical Music Recognition”. In: ACM Comput. Surv. 53.4 (July 2020). issn: 0360-0300. doi: 10 . 1145 / 3397499. url: https : / / doi . org / 10 . 1145/3397499
work page 2020
-
[5]
Two (Note) Heads Are Better Than One: Pen-Based Multimodal Interaction with Music Scores
Jorge Calvo-Zaragoza, David Rizo, and José Manuel Iñesta Quereda. “Two (Note) Heads Are Better Than One: Pen-Based Multimodal Interaction with Music Scores”. In: International Society for Music Information Re- trieval Conference . 2016. url: https : / / api . semanticscholar . org / CorpusID:621842
work page 2016
-
[6]
Gen-Fang Chen and Jia-Shing Sheu. “An optical music recognition sys- tem for traditional Chinese Kunqu Opera scores written in Gong-Che No- tation”. In: EURASIP Journal on Audio, Speech, and Music Processing (2014), pp. 7–17. doi: 10.1186/1687-4722-2014-7
-
[7]
Hiuyi Cheng et al. “SCUT-CAB: A New Benchmark Dataset of Ancient Chinese Books with Complex Layouts for Document Layout Analysis”. In: Nov. 2022, pp. 436–451. isbn: 978-3-031-21647-3. doi: 10.1007/978- 3- 031-21648-0_30
doi:10.1007/978- 2022
Show all 26 references
-
[8]
The MUSCIMA++ Dataset for Hand- written Optical Music Recognition
Jan jr. Haji and Pavel Pecina. “The MUSCIMA++ Dataset for Hand- written Optical Music Recognition”. In: 14th International Conference on Document Analysis and Recognition, ICDAR 2017. Kyoto, Japan, Novem- ber 13-15, 2017, pp. 39–46. 10 http://www.music-encoding.org KuiSCIMA v2...
2017
-
[9]
Six dragons fly again: Reviving 15th-century Ko- rean court music with transformers and novel encoding
Danbinaerin Han et al. “Six dragons fly again: Reviving 15th-century Ko- rean court music with transformers and novel encoding”. In: Proceedings of 25th International Society for Music Information Retrieval Conference (ISMIR). 2024
2024
-
[10]
Tesseract: an open-source optical character recognition en- gine
Anthony Kay. “Tesseract: an open-source optical character recognition en- gine”. In: Linux J. 2007.159 (July 2007), p. 2. issn: 1075-3583
2007
-
[11]
Ci Songs From the Song Dynasty: A Ménage à Trois of Lyrics, Music, and Performance
Joseph S. C. Lam. “Ci Songs From the Song Dynasty: A Ménage à Trois of Lyrics, Music, and Performance”. In: New Literary History 46.4 (2015), pp. 623–646. issn: 00286087, 1080661X. url: http://www.jstor.org/ stable/24772762 (visited on 08/02/2023)
2015
-
[12]
Focal Loss for Dense Object Detection
Tsung-Yi Lin et al. “Focal Loss for Dense Object Detection”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 42.2 (2020), pp. 318–327. doi: 10.1109/TPAMI.2018.2858826
2020
-
[13]
Joint Layout Analysis, Character Detection and Recog- nition for Historical Document Digitization
Weihong Ma et al. “Joint Layout Analysis, Character Detection and Recog- nition for Historical Document Digitization”. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). 2020, pp. 31–
2020
-
[14]
On the Performance of Optical Music Recognition in the Absence of Specific Training Data
Juan C. Martinez-Sevilla et al. “On the Performance of Optical Music Recognition in the Absence of Specific Training Data”. In: Proceedings of the 24th International Society for Music Information Retrieval Conference (Milan, Italy). ISMIR, Nov. 2023, pp. 319–326. url: https://d...
2023 doi
-
[15]
UMAP: Uniform Manifold Approximation and Pro- jection
Leland McInnes et al. “UMAP: Uniform Manifold Approximation and Pro- jection”. In: Journal of Open Source Software 3.29 (2018), p. 861. doi: 10.21105/joss.00861. url: https://doi.org/10.21105/joss.00861
2018 doi
-
[16]
Creating a Corpus of Jingju (Bei- jing Opera) Music and Possibilities for Melodic Analysis
Rafael Caro Repetto and Xavier Serra. “Creating a Corpus of Jingju (Bei- jing Opera) Music and Possibilities for Melodic Analysis”. In: Proceedings of the 15th International Society for Music Information Retrieval Confer- ence, ISMIR 2014, Taipei, Taiwan, October 27-31, 2014 ....
2014
-
[18]
The Suzipu Musical Annotation Tool for the Creation of Machine-Readable Datasets of Ancient Chinese Music
Tristan Repolusk and Eduardo Veas. “The Suzipu Musical Annotation Tool for the Creation of Machine-Readable Datasets of Ancient Chinese Music”. In: Proceedings of the 5th International Workshop on Reading Mu- sic Systems (WoRMS) . Ed. by Jorge Calvo-Zaragoza, Alexander Pacha, ...
2023 doi
-
[19]
ICDAR 2019 Historical Document Reading Chal- lenge on Large Structured Chinese Family Records
Rajkumar Saini et al. “ICDAR 2019 Historical Document Reading Chal- lenge on Large Structured Chinese Family Records”. In: 2019 Interna- tional Conference on Document Analysis and Recognition (ICDAR) . 2019, pp. 1499–1504. doi: 10.1109/ICDAR.2019.00241
2019
-
[20]
Semantic Recognition of Common Musical Notes in Guqin Score Based on Optimal Statistical Features
Tiantian Shen et al. “Semantic Recognition of Common Musical Notes in Guqin Score Based on Optimal Statistical Features”. In: 4th International Conference on Advances in Computer Technology, Information Science and Communications (CTISC) . 2022, pp. 1–4. doi: 10.1109/CTISC5488...
2022
-
[21]
Chinese Text Project
David Sturgeon. Chinese Text Project . 2011. url: https://ctext.org/ library.pl (visited on 06/30/2023)
2011
-
[22]
Musical Notation for Flute in Tangut Manuscripts
Andrew Christopher West. “Musical Notation for Flute in Tangut Manuscripts”. In: Tanguty v Central’noj Azii . Ed. by Irina Popova. Moskva, Russia: Vos- tonaja literatura, 2012, pp. 443–454
2012
-
[23]
The Kanseki Repository: A new online resource for Chinese textual studies
Christian Wittern. “The Kanseki Repository: A new online resource for Chinese textual studies”. In: Proceedings of the 6th Conference of Japanese Association for Digital Humanities (JADH2016) . Tokyo, Japan, 2016
2016
-
[24]
Plum Blossom on the Far Side of the Stream
Yuanzheng Yang. Plum Blossom on the Far Side of the Stream. The Re- naissance of Jiang Kui’s Lyric Oeuvre with Facsimiles and a New Critical Edition of The Songs of the Whitestone Daoist . Hong Kong, China: Hong Kong University Press, 2019
2019
-
[25]
Songci Yinyue Zhuanti Yanjiu 宋词音乐专题研究
Wu Santu 伍三土 . “Songci Yinyue Zhuanti Yanjiu 宋词音乐专题研究 ”. Dissertation. Yangzhou University, 2013
2013
-
[36]
doi: 10.1109/ICFHR2020.2020.00017
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.