REVIEW 3 major objections 3 minor 40 references
Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning
T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Zero-shot Chinese character recognition can be recast as multi-granularity image-text retrieval, with stroke and radical sequences aligned to images at several levels; the paper reports about 20-percentage-point accuracy gains over prior…
desk verdict Genuinely novel multi-granularity CLIP framework for zero-shot CCR with large, plausible gains, but the undefined TS-D decomposition and test-set-tuned hyperparameters mean the headline claims need code and clarification before they are fully reproducible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Fine-Grained Decoupled Image-Text Contrastive (FDC) loss and its two supporting operations. Text Sequence Decoupling (TS-D) splits each stroke or radical sequence representation into a detailed-description component and a structure component; the Fine-Grained Character Components Matching function then computes softmax-weighted similarities between each text token and every image token, and the contrastive loss aligns detail tokens with the corresponding stroke and radical image features while aligning structure tokens with the structure image feature. Around this loss, the Image Multi-Granularity Encoder extracts stroke, radical, and structure features and refines the first two through an image-side fusion module, while the Text Multi-Granularity Encoder applies symmetric cross-attention and global self-attention to the stroke and radical sequences. All five semantic levels are trained jointly, while inference uses the refined stroke sequence together with the refined stroke and structure image features.
What would settle it
Run an ablation that keeps Hi-GITA's encoders, fusion modules, and contrastive loss unchanged but replaces the Text Sequence Decoupling with a fixed random partition of sequence tokens into 'detail' and 'structure' groups; if zero-shot accuracy stays near the reported levels, the decoupling as described carries no information and the claimed mechanism is not what produces the gains.
Extended reading notes
Core claim
The central discovery is that zero-shot Chinese character recognition can be formulated as a multi-level contrastive retrieval problem. On the image side, a hierarchical encoder produces stroke, radical, and structure representations from a single character image; on the text side, separate encoders read the character's radical sequence and stroke sequence, and fusion modules let the two granularities refine each other. A Fine-Grained Decoupled Image-Text Contrastive loss splits each sequence representation into a detailed-description part and a structure part and aligns each part with the corresponding image features through a token-level softmax-weighted matching function. The paper reports that this setup outperforms existing zero-shot CCR methods across handwritten, printed-artistic, and scene datasets, with the largest gains of about 20 percentage points in handwritten and radical zero-shot settings.
Load-bearing premise
The method assumes every character has a pre-decoupled stroke and radical sequence and that the Text Sequence Decoupling step splits each sequence representation into a well-defined 'detailed description' part and 'structure' part, but the paper only confirms the first assumption and never specifies how the split is computed or learned; if the split is arbitrary, the multi-level alignment is not what produces the reported gains.
Editorial extensions
If this is right
- Zero-shot CCR no longer needs autoregressive decoding or edit-distance post-processing; a retrieval model over multi-granularity representations is enough.
- Recognition of characters containing unseen radicals improves by a large margin, not just recognition of unseen whole characters.
- Fusing stroke and radical information helps both modalities: removing the text fusion module costs about 9.6 accuracy points in the character zero-shot setting.
- The structure component carries layout information that matters: removing it from inference drops average accuracy by about 11 points in the character zero-shot setting.
- The same multi-granularity alignment also improves closed-set recognition, beating prior character-, stroke-, radical-, and hybrid-based recognition methods on the HWDB handwritten benchmark.
Reading between the lines
- An implication the paper leaves implicit is that if TS-D is implemented as a learned projection, the same recipe may transfer to other ideographic scripts such as Japanese kanji or Korean hanja by swapping the sequence vocabulary.
- Because the method depends on pre-decoupled stroke and radical sequences, a natural extension the paper names but does not test is predicting the character decomposition tree directly, which would remove that external dependency.
- The duplicate-positive contrastive loss could be reused in other fine-grained retrieval tasks with a small label vocabulary, where several images share one text description; the paper does not explore this transfer.
- The large gap between refined stroke and refined radical representations suggests that stroke-level features, once enriched with radical context, are the most expressive units for character identity, a hypothesis a human confusion study on visually similar radicals could probe.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hi-GITA, a contrastive retrieval framework for zero-shot Chinese character recognition (CCR). The image encoder produces hierarchical stroke, radical, and structure representations; the text encoder processes stroke and radical sequences with cross-attention fusion; and a Fine-Grained Decoupled Image-Text Contrastive (FDC) loss aligns image and text components at multiple semantic levels. The authors report large gains over previous zero-shot CCR methods on handwritten and printed artistic character datasets, smaller gains on scene text (CTW), and improved closed-set accuracy. The paper claims that the multi-granularity alignment and the decoupling of structure from detailed description are the key innovations.
Significance. If the results are reproducible, Hi-GITA represents a substantial advance for zero-shot CCR, particularly for handwritten characters, by replacing autoregressive stroke/radical decoding with multi-granularity retrieval. The paper's strengths include systematic ablations of each module (Table 2), clear attention visualizations (Figure 5), and experiments across multiple standard benchmarks. However, the main technical module (TS-D) is not specified precisely, the hyperparameters and inference representation appear to be selected on the test sets, and no error bars are reported. These issues currently prevent verification of the stated gains; the approach is plausible and the concerns are addressable, so the potential significance is high but conditional.
major comments (3)
- [Section 3.3, Eq. (6)] The Text Sequence Decoupling (TS-D) operation is never specified beyond the equation Q* = D* ∪ U*. It is not stated whether the decoupling is a fixed split by token type (leaf vs. non-leaf as suggested by Appendix A), a learned projection, or an attention-based decomposition; nor are the dimensions of D* and U* or the meaning of ∪ given. Since Eq. (9) and the inference procedure in Section 3.4 rely on D-symbol and U-symbol, and Section 4.3 reports a 10.97% accuracy drop when the structure component is removed, this omission makes the central multi-level alignment and the reported gains unreproducible as written. Please specify the exact computation, including whether ordering information within the sequences is preserved.
- [Section 4.1/4.4 and Figure 3] The hyperparameters α and β in Eq. (12) are tuned in Section 4.4 using the handwritten dataset under the character zero-shot setting (m=500), which is the same test setting (ICDAR2013) used to report the main results in Table 1. Similarly, the inference representation (refined stroke with structure) is selected based on performance on the same test sets in Figure 3. This constitutes test-set model selection and likely inflates the reported accuracies. Please introduce a validation split or report model selection on a separate validation set, then present final test results only for the selected configuration.
- [Section 4.2, Table 1] Table 1 reports single-run results with no error bars or significance tests. On the CTW radical zero-shot setting, the average improvement over the previous state of the art is 0.21%, and individual margins are small (e.g., 3.27 vs. 3.51 at n=20); without multiple seeds or statistical testing, the claim of consistent superiority in this setting is not established. Please provide variance across at least three runs and a paired test for the small-margin settings.
minor comments (3)
- [Section 3.2, Eqs. (3)-(4)] The two equations appear swapped relative to the text: if Qr attends to Qs, the output should be the refined radical representation (Qr'), not Qs'. Please correct the labels or the cross-attention argument order. There is also an extra closing parenthesis in Eq. (4).
- [Throughout] There are several typos: 'Datesets' should be 'Datasets' (Section 4.1), '60 writes' should be '60 writers' (Section 4.1), and 'sematic' should be 'semantic' (Section 3 and Figure 2 caption).
- [References] References [12] and [33] refer to the same paper (Chen et al., 'Zero-shot Chinese character recognition with stroke-level decomposition'); please unify them.
Circularity Check
No circularity found: Hi-GITA's headline accuracy is measured on external zero-shot benchmarks; the underspecified TS-D split and test-set tuning are reproducibility/selection gaps, not self-referential derivation.
full rationale
The paper's strongest claim is a held-out accuracy result on standard external benchmarks (HWDB/ICDAR2013, printed artistic characters, CTW), with a fixed evaluation protocol and comparison against published baselines. Such benchmark numbers cannot reduce by construction to the model's loss equations, and the paper does not fit any parameter to the reported accuracies and then rename that fit as a prediction; alpha and beta are stated constants (alpha=1.0, beta=0.1) and the inference representation is a fixed design choice. The line 'Q* = D* union U*' (Eq. 6) and its reliance in Eq. 9 is genuinely underspecified: D* and U* are never defined operationally, so the FDC loss is not reproducible as written, but this is a gap in specification, not an equivalence between input and output. The paper also relies on pre-decoupled stroke/radical sequences from prior work [13,16], some with overlapping authors, but these are independent data resources and standard decompositions; no load-bearing self-citation is invoked as a substitute for evidence. The ablation-driven selection of alpha, beta, and the inference representation on the test sets is an adaptive-analysis concern, not circularity. No equation in the paper reduces to its own input, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- alpha (alpha) =
1.0
- beta (beta) =
0.1
assumptions (4)
- domain assumption Chinese characters can be decomposed into canonical stroke and radical sequences with a specific order and structure tree (Appendix A).
- ad hoc to paper Text sequence representations can be decoupled into detailed description components and structure components (Eq. 6).
- domain assumption Multi-granularity alignment at stroke, radical, and structure levels improves zero-shot generalization.
- standard math ResNet-50 and transformer encoder architectures are used as standard building blocks and are assumed to work as in prior literature.
Cite this review
Pith. "Pith review of Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning." pith.science (2026). https://pith.science/paper/PIQU5VKR
@misc{pith2026250524837,
author = {Pith},
title = {Pith review of: Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning},
year = {2026},
howpublished = {\url{https://pith.science/paper/PIQU5VKR}},
note = {Machine review of arXiv:2505.24837}
}
read the original abstract
Chinese Character Recognition (CCR) is a fundamental technology for intelligent document processing. Unlike Latin characters, Chinese characters exhibit unique spatial structures and compositional rules, allowing for the use of fine-grained semantic information in representation. However, existing approaches are usually based on auto-regressive as well as edit distance post-process and typically rely on a single-level character representation. In this paper, we propose a Hierarchical Multi-Granularity Image-Text Aligning (Hi-GITA) framework based on a contrastive paradigm. To leverage the abundant fine-grained semantic information of Chinese characters, we propose multi-granularity encoders on both image and text sides. Specifically, the Image Multi-Granularity Encoder extracts hierarchical image representations from character images, capturing semantic cues from localized strokes to holistic structures. The Text Multi-Granularity Encoder extracts stroke and radical sequence representations at different levels of granularity. To better capture the relationships between strokes and radicals, we introduce Multi-Granularity Fusion Modules on the image and text sides, respectively. Furthermore, to effectively bridge the two modalities, we further introduce a Fine-Grained Decoupled Image-Text Contrastive loss, which aligns image and text representations across multiple granularities. Extensive experiments demonstrate that our proposed Hi-GITA significantly outperforms existing zero-shot CCR methods. For instance, it brings about 20% accuracy improvement in handwritten character and radical zero-shot settings. Code and models will be released soon.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
M5hisdoc: A large-scale multi-style chinese historical document analysis benchmark
Yongxin Shi, Chongyu Liu, Dezhi Peng, Cheng Jian, Jiarong Huang, and Lianwen Jin. M5hisdoc: A large-scale multi-style chinese historical document analysis benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023
work page 2023
-
[2]
Yuyi Zhang, Yongxin Shi, Peirong Zhang, Yixin Zhao, Zhenhua Yang, and Lianwen Jin. Megahan97k: A large-scale dataset for mega-category chinese character recognition with over 97k categories. Pattern Recognition, page 111757, 2025
work page 2025
-
[3]
Likun Gao, Heng Zhang, and Cheng-Lin Liu. Regularizing ctc in expectation-maximization framework with application to handwritten text recognition. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–7, 2021
work page 2021
-
[4]
Document-level relation extraction with adaptive focal loss and knowledge distillation
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. Document-level relation extraction with adaptive focal loss and knowledge distillation. arXiv preprint arXiv:2203.10900, 2022
arXiv 2022
-
[5]
Orientation-Independent Chinese Text Recognition in Scene Images
Haiyang Yu, Xiaocong Wang, Bin Li, and Xiangyang Xue. Orientation-independent chinese text recognition in scene images. arXiv preprint arXiv:2309.01081, 2023
work page Pith review arXiv 2023
-
[6]
Towards accurate video text spotting with text-wise semantic reasoning
Xinyan Zu, Haiyang Yu, Bin Li, and Xiangyang Xue. Towards accurate video text spotting with text-wise semantic reasoning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 1858–1866, 2023
work page 2023
-
[7]
Multi-column deep neural networks for offline handwritten chinese character classification
Dan Cire¸ san and Ueli Meier. Multi-column deep neural networks for offline handwritten chinese character classification. In 2015 international joint conference on neural networks (IJCNN), pages 1–6. IEEE, 2015
work page 2015
-
[8]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
Show all 40 references
-
[9]
Densely connected convolutional networks
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017
2017
-
[10]
Denseran for offline handwritten chinese character recognition
Wenchao Wang, Jianshu Zhang, Jun Du, Zi-Rui Wang, and Yixing Zhu. Denseran for offline handwritten chinese character recognition. CoRR, abs/1808.04134, 2018
2018 arXiv
-
[11]
Stroke-based autoencoders: Self-supervised learners for efficient zero-shot chinese character recognition, 2022
Zongze Chen, Wenxia Yang, and Xin Li. Stroke-based autoencoders: Self-supervised learners for efficient zero-shot chinese character recognition, 2022
2022
-
[13]
Chinese character recognition with radical- structured stroke trees
Haiyang Yu, Jingye Chen, Bin Li, and Xiangyang Xue. Chinese character recognition with radical- structured stroke trees. Machine Learning, pages 1–21, 2023
2023
-
[14]
Chinese text recognition with a pre-trained clip-like model through image-ids aligning
Haiyang Yu, Xiaocong Wang, Bin Li, and Xiangyang Xue. Chinese text recognition with a pre-trained clip-like model through image-ids aligning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11943–11952, 2023
2023
-
[15]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...
2021
-
[16]
Radical analysis network for zero-shot learning in printed chinese character recognition
Jianshu Zhang, Yixing Zhu, Jun Du, and Lirong Dai. Radical analysis network for zero-shot learning in printed chinese character recognition. In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2018
2018
-
[17]
Study of several directional feature extraction methods with local elastic meshing technology for hccr
Lian-Wen Jin, Jun-Xun Yin, Xue Gao, and Jiang-Cheng Huang. Study of several directional feature extraction methods with local elastic meshing technology for hccr. In Proceedings of the Sixth Int. Conference for Young Computer Scientist, volume 3, 2001
2001
-
[18]
Techniques for solving the large-scale classification problem in chinese handwriting recognition
Fu Chang. Techniques for solving the large-scale classification problem in chinese handwriting recognition. In Summit on Arabic and Chinese Handwriting Recognition, pages 161–169. Springer, 2006
2006
-
[19]
A novel stroke extraction method for chinese characters using gabor filters
Yih-Ming Su and Jhing-Fa Wang. A novel stroke extraction method for chinese characters using gabor filters. Pattern Recognition, 36(3):635–647, 2003
2003
-
[21]
Deep matching network for handwritten chinese character recognition
Zhiyuan Li, Qi Wu, Yi Xiao, Min Jin, and Huaxiang Lu. Deep matching network for handwritten chinese character recognition. Pattern Recognition, 107:107471, 2020. 12
2020
-
[23]
Self-information of radicals: A new clue for zero-shot chinese character recognition
Guo-Feng Luo, Da-Han Wang, Xia Du, Hua-Yi Yin, Xu-Yao Zhang, and Shunzhi Zhu. Self-information of radicals: A new clue for zero-shot chinese character recognition. Pattern Recognition, 140:109598, 2023
2023
-
[24]
Sidenet: Learning representa- tions from interactive side information for zero-shot chinese character recognition
Ziyan Li, Yuhao Huang, Dezhi Peng, Mengchao He, and Lianwen Jin. Sidenet: Learning representa- tions from interactive side information for zero-shot chinese character recognition. Pattern Recognition, 148:110208, 2024
2024
-
[25]
Ucr: A unified character-radical dual-supervision framework for accurate chinese character recognition
Qilong Li and Chongsheng Zhang. Ucr: A unified character-radical dual-supervision framework for accurate chinese character recognition. Pattern Recognition, page 111373, 2025
2025
-
[26]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...
2010 arXiv
-
[27]
Decomposition of chinese character into strokes using mathematical morphology
Jin Wook Kim, Kwang In Kim, Bong Joon Choi, and Hang Joon Kim. Decomposition of chinese character into strokes using mathematical morphology. Pattern Recognition Letters, 20(3):285–292, 1999
1999
-
[28]
Chinese character recognition with augmented character profile matching
Xinyan Zu, Haiyang Yu, Bin Li, and Xiangyang Xue. Chinese character recognition with augmented character profile matching. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 6094–6102, New York, NY , USA, 2022. Association for Computing Machinery
2022
-
[29]
Learning deconvolution network for semantic segmentation
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pages 1520–1528, 2015
2015
-
[30]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[31]
Online and offline handwritten chinese character recognition: benchmarking on new databases
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Online and offline handwritten chinese character recognition: benchmarking on new databases. Pattern Recognition, 46(1):155–162, 2013
2013
-
[32]
Icdar 2013 chinese handwriting recognition competition
Fei Yin, Qiu-Feng Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Icdar 2013 chinese handwriting recognition competition. In 2013 12th international conference on document analysis and recognition, pages 1464–
2013
-
[33]
Zero-shot chinese character recognition with stroke-level decomposition
Jingye Chen, Bin Li, and Xiangyang Xue. Zero-shot chinese character recognition with stroke-level decomposition. arXiv preprint arXiv:2106.11613, 2021
2021 arXiv
-
[34]
Chinese text in the wild
Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, and Shi-Min Hu. Chinese text in the wild. arXiv preprint arXiv:1803.00085, 2018
2018 arXiv
-
[35]
Online and offline handwritten chinese character recognition: Benchmarking on new databases
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Online and offline handwritten chinese character recognition: Benchmarking on new databases. Pattern Recognition, 46(1):155–162, 2013
2013
-
[36]
Zero-shot handwritten chinese character recognition with hierarchical decomposition embedding
Zhong Cao, Jiang Lu, Sen Cui, and Changshui Zhang. Zero-shot handwritten chinese character recognition with hierarchical decomposition embedding. Pattern Recognition, 107:107488, 2020
2020
-
[37]
Template-instance loss for offline handwritten chinese character recognition
Yao Xiao, Dan Meng, Cewu Lu, and Chi-Keung Tang. Template-instance loss for offline handwritten chinese character recognition. In 2019 International conference on document analysis and recognition (ICDAR), pages 315–322. IEEE, 2019
2019
-
[38]
Radical analysis network for learning hierarchies of chinese characters
Jianshu Zhang, Jun Du, and Lirong Dai. Radical analysis network for learning hierarchies of chinese characters. Pattern Recognition, 103:107305, 2020
2020
-
[39]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[40]
Online and offline handwritten chinese character recognition: A comprehensive study and new benchmark
Xu-Yao Zhang, Yoshua Bengio, and Cheng-Lin Liu. Online and offline handwritten chinese character recognition: A comprehensive study and new benchmark. Pattern Recognition, 61:348–360, 2017
2017
-
[41]
Dropsample: A new training method to enhance deep convolutional neural networks for large-scale unconstrained handwritten chinese character recognition
Weixin Yang, Lianwen Jin, Dacheng Tao, Zecheng Xie, and Ziyong Feng. Dropsample: A new training method to enhance deep convolutional neural networks for large-scale unconstrained handwritten chinese character recognition. Pattern Recognition, 58:190–203, 2016
2016
-
[42]
Rzcr: Zero-shot character recognition via radical-based reasoning
Xiaolei Diao, Daqian Shi, Hao Tang, Qiang Shen, Yanzeng Li, Lei Wu, and Hao Xu. Rzcr: Zero-shot character recognition via radical-based reasoning. arXiv preprint arXiv:2207.05842, 2022
2022 arXiv
-
[43]
Recognition ways
Jinshan Zeng, Ruiying Xu, Yu Wu, Hongwei Li, and Jiaxing Lu. Zero-shot chinese character recognition with stroke-and radical-level decompositions. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2023. 13 A Preliminaries Chinese characters ar...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.