REVIEW 4 major objections 5 minor 23 references
The Return of Structural Handwritten Mathematical Expression Recognition
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Structural math recognition returns, hitting 74.14% on CROHME-2023
desk verdict A genuinely useful dataset and a clean modular structural pipeline, but the auto-labeling loop makes the headline gains conditional on labeler consistency rather than independent ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Stroke Label Graph (SLG) is the central object: a directed graph whose nodes are trace groups (symbols) and whose edges are spatial relations (right, sup, sub, over, under, line_start), directly convertible to LaTeX. The argument is carried by two machines: (1) an automatic annotation network—a BiLSTM with fully connected layers that takes raw trace coordinates plus structural info extracted from the LaTeX label and outputs a binary segmentation mask per symbol—which produces the SLG training data; and (2) the five-stage modular recognition pipeline that independently optimizes segmentation, classification, and relation prediction, with a transformer correcting symbol classes using globa
What would settle it
Take a random sample of 1,000 auto-annotated expressions from the released CROHME+ dataset, have independent human annotators create trace-level MathML labels, and compare. If the disagreement rate is high for uncommon symbols or unusual writing styles, and if training the pipeline on only human-verified samples drops expression accuracy far below 74.14%, then the reported gains partly reflect labeler consistency rather than true structural understanding.
Extended reading notes
Core claim
The central discovery is that the perceived ceiling of structural HMER was largely a data problem, not an architecture problem. By training a neural network to align LaTeX labels with raw traces, the authors generate structural annotations (symbol segmentation, classification, and spatial relations) for hundreds of thousands of expressions that previously had only LaTeX labels. Trained on these auto-labeled annotations, a modular pipeline—graph-based trace sorting, dual-modal symbol classification, relation prediction, transformer-based correction, and a final relation refinement—achieves state-of-the-art results among structurally interpretable methods and narrows the gap to black-box encod
Load-bearing premise
The whole pipeline depends on the auto-generated structural labels being correct enough to serve as training targets, and the cross-checking filters use a classifier trained in the same label ecosystem rather than independent ground truth.
Editorial extensions
If this is right
- If the central claim holds, structural HMER is no longer relegated to simple formulas: with enriched data, it can match or beat graph-based baselines on all five CROHME-2023 metrics (Seg, Sym, Rel, Exp, Stru).
- Systems that need selective content updates or transparent error analysis—such as interactive math editors and grading tools—can use the SLG output to localize mistakes to specific strokes or symbols.
- The released CROHME+ and MathWriting+ datasets give the community trace-level annotations for 374,000 expressions, removing the manual-labeling bottleneck that previously limited structural methods.
- The auto-annotation network's cross-dataset generalization (trained on CROHME, labeling MathWriting with 254 classes) suggests that annotation transfer may work for other structured handwritten content.
- Because all modules are trained independently, individual stages can be swapped or improved without retraining the whole system.
Reading between the lines
- The paper's strongest implicit claim is that labeler consistency—not ground-truth correctness—is what drives the downstream gains; a direct comparison against human-verified labels would separate these two explanations.
- If the auto-annotation approach transfers beyond math, the same LaTeX-to-trace alignment idea could provide structural supervision for handwritten chemical formulas, musical notation, or diagrams where a symbolic description already exists.
- A testable extension: feeding the SLG structure (rather than only LaTeX text) into an LLM-based post-corrector could combine the interpretability of structural recognition with the fluency of modern language models.
- The 1:1 trace-to-symbol constraint noted in the paper suggests that relaxing it—allowing shared or fragmented traces—is the natural next boundary; verifying the pipeline on cursive-style or multi-letter-symbol handwriting would show whether the approach generalizes beyond the CROHME symbol set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an automatic annotation network that takes raw traces and a LaTeX label as input and predicts trace groups and spatial relations, generating MathML and Trace-Group annotations for CROHME synthetic data and MathWriting. Using these enriched CROHME+ labels, the authors train a modular structural recognition pipeline with five stages: trace segmentation, dual-modal symbol classification, primary relation prediction, transformer-based classification correction, and revised relation prediction. The system outputs a Stroke Label Graph (SLG) that links traces to symbols and is convertible to LaTeX. The central empirical claim is 74.14% expression accuracy on the CROHME-2023 benchmark, presented as competitive with encoder-decoder/LLM approaches while retaining full structural interpretability. The authors release CROHME+ and MathWriting+ datasets.
Significance. If validated, the work would be a notable result: it challenges the prevailing view that structural, modular HMER is inherently inferior to end-to-end encoder-decoder models, and it provides a scalable way to generate trace-level structural annotations without manual labeling. The explicitly interpretable SLG output is a practical advantage for error analysis and interactive applications, and the released datasets are potentially valuable community resources. The paper also gives a clear ablation story showing that synthetic structural annotations improve each stage of the pipeline. However, the central benefit depends on the correctness of auto-generated labels, and the current manuscript does not provide independent evidence of label quality for the majority of the newly annotated data. The reported Stage-wise versus final metrics are also internally inconsistent, and the validation/test numbers are presented without a clear split, so the headline accuracy is not yet fully supported.
major comments (4)
- [Section 3.2 and 3.3] The filtering procedure is not independent of the label-generation process. For synthetic CROHME, a symbol is accepted if its class appears in the classifier's top-10 outputs; for MathWriting, an expression is retained only if reclassification matches the LaTeX label. Both classifiers are trained in the same label ecosystem as the auto-annotation network, so systematic labeler errors on uncommon symbols or unusual writing styles can be propagated rather than detected. The paper reports 99.45% per-symbol accuracy on CROHME validation, but the bulk of the new training data (synthetic CROHME and MathWriting) has no ground truth, and no accuracy estimate is given for the retained labels there. Since these labels are training targets for the downstream recognizer, the 74.14% test result could partly reflect learning labeler-consistent mappings rather than human-ground-truth structure. Please
- [Table 1 vs. Table 2 and Section 4.2/4.3] The paper reports validation Exp = 82.77% in Table 1 and benchmark Exp = 74.14% in Table 2 without explicitly stating that one is validation and the other is test. That difference is plausible, but the manuscript should state the split clearly and discuss the generalization gap. More seriously, Table 1 lists Sym = 99.30% and Rel = 98.37% for the full system, while the text in Section 4.2 reports Sym = 96.73% after the correction module and Section 4.3 reports Rel = 96.98% for the primary relation stage. No explanation reconciles these numbers. Please provide a single end-to-end result table with each metric, the exact evaluation split, and the precise pipeline configuration, or explicitly state that the stage-by-stage numbers are before final integration.
- [Section 3.1] The annotation network is described only at a high level: a BiLSTM plus fully connected layers predicts a binary segmentation mask per 'step', using reference symbols, next symbols, neighbours, and relations. The input representation is not specified in enough detail to reproduce the method: how are LaTeX labels tokenized, aligned to trace steps, and converted into the relation context? How is the network trained on CROHME-2023 when, as the paper notes, 92% of the training set lacks MathML and many trace-group labels are incorrect? The paper should specify the exact training subset, the input feature vector, and the decoding/merging procedure from per-step binary masks to final Trace Groups. This is load-bearing because the entire CROHME+ and MathWriting+ resources, and hence all downstream gains, depend on this component.
- [Section 4.2 / 4.4] The classification module uses a dual-stream network with a BiLSTM pathway and a CNN pathway, and the correction transformer, yet the reported gain from adding the CNN is large (Sym 95.01% to 96.55%, Exp 64.08% to 72.60%) and from the correction module is small. However, the paper never reports whether the test-set 74.14% corresponds to the full system with corrected relations or to the 'classification correction' stage alone. The 'Integrated Pipeline' section implies all five stages are used, but the exact inference-time configuration for the Table 2 result is not stated. Please specify the exact inference configuration for the headline number and report all component metrics on the same test split.
minor comments (5)
- [Section 3.3 / Section 5] The retention rate for MathWriting is stated as 66% in Section 3.3 and 67% in the same paragraph / Section 5. Please make the numbers consistent.
- [References / Table 1] Table 1 cites 'G2G (2021) [4]', but reference [4] is an 'Edge-featured graph attention network' paper, not the G2G paper; the G2G paper appears as references [15] and [16] (duplicate). Please fix the citation mapping and remove the duplicate reference.
- [Section 4.4] Typo: 'Raw tracs are partitioned' should be 'Raw traces are partitioned'.
- [Figures 5 and 6] The example equations in Figures 5 and 6 are shown as raw LaTeX strings in the caption. If these are rendered outputs, it would help to show them as images so the reader can actually assess the segmentation and relation quality; otherwise, the figures are hard to interpret.
- [General] No variance or multiple-run information is given for any of the reported numbers. Given that several claims rely on small differences (e.g., 0.14% gain from revised relation prediction), please report the number of runs and standard deviations, or at least state that all numbers are from a single training run.
Circularity Check
No significant circularity: the benchmark claim is measured on an external test set, and generated annotations are auxiliary training targets rather than the predicted outputs.
full rationale
The central claim (74.14% expression accuracy on CROHME-2023, Table 2) is an external benchmark result, not a fitted value. The auto-labeling system (Section 3) is trained on CROHME-2023 training data and generates trace-group/MathML annotations for synthetic CROHME and MathWriting; these annotations are used only as training supervision for the modular recognition system (Section 4). The recognizer is then evaluated on held-out CROHME test data whose ground-truth labels are independent of the annotation network. No equation defines the predicted label in terms of the input label; the LaTeX labels used by the annotation network are input information, not outputs being predicted. The cross-checking filters in Section 3.2 use a reclassification model that shares the CROHME label ecosystem, which is a real annotation-quality limitation (the paper itself notes the filters 'may exclude rare patterns'), but it does not make the downstream test accuracy true by construction. The paper contains no load-bearing self-citations and no imported uniqueness theorem. The difference between validation Exp (82.77%, Table 1) and benchmark Exp (74.14%, Table 2) reflects validation-versus-test reporting, not circularity. The derivation is therefore self-contained with respect to the benchmark claim.
Assumptions & free parameters
free parameters (6)
- Segmentation foreground loss weight w=5 =
5
- MST nearest-neighbor count k=20 =
20
- Augmentation hyperparameters (scaling sigma=0.2/0.4 clamped [0.2,5], shear 0.1, rotation 8 deg, translation 0.15) =
as listed
- Classification BiLSTM hidden size h=256 and three layers =
256/3 layers
- Correction transformer d_model=256, 8 heads, 3 layers =
256/8/3
- Relation network hidden size 64 and 8 attention heads =
64/8
assumptions (5)
- domain assumption CROHME-2023 and MathWriting ground-truth LaTeX labels are correct and sufficiently complete for training and evaluation.
- domain assumption The auto-labeler's validation accuracy on CROHME transfers to synthetic CROHME data and to MathWriting, which has no structural ground truth.
- domain assumption All symbols in the target domain consist of disjoint trace groups with a 1:1 trace-to-symbol mapping.
- domain assumption The relation set R = {right, sup, sub, over, under, line_start} is sufficient to express the structural content of CROHME-2023 expressions.
- domain assumption Processing (x,y) trace coordinates without timestamps preserves enough temporal structure for the BiLSTM models.
Cite this review
Pith. "Pith review of The Return of Structural Handwritten Mathematical Expression Recognition." pith.science (2026). https://pith.science/paper/5OL6LW2Q
@misc{pith2026250819773,
author = {Pith},
title = {Pith review of: The Return of Structural Handwritten Mathematical Expression Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/5OL6LW2Q}},
note = {Machine review of arXiv:2508.19773}
}
read the original abstract
Handwritten Mathematical Expression Recognition is foundational for educational technologies, enabling applications like digital note-taking and automated grading. While modern encoder-decoder architectures with large language models excel at LaTeX generation, they lack explicit symbol-to-trace alignment, a critical limitation for error analysis, interpretability, and spatially aware interactive applications requiring selective content updates. This paper introduces a structural recognition approach with two innovations: 1 an automatic annotation system that uses a neural network to map LaTeX equations to raw traces, automatically generating annotations for symbol segmentation, classification, and spatial relations, and 2 a modular structural recognition system that independently optimizes segmentation, classification, and relation prediction. By leveraging a dataset enriched with structural annotations from our auto-labeling system, the proposed recognition system combines graph-based trace sorting, a hybrid convolutional-recurrent network, and transformer-based correction to achieve competitive performance on the CROHME-2023 benchmark. Crucially, our structural recognition system generates a complete graph structure that directly links handwritten traces to predicted symbols, enabling transparent error analysis and interpretable outputs.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Syntax-directed recognition of hand-printed two-dimensional mathematics
Robert H Anderson. Syntax-directed recognition of hand-printed two-dimensional mathematics. InSymposium on interactive systems for experimental applied math- ematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pages 436–459, 1967
work page 1967
-
[2]
A syntactic approach for handwritten mathematical formula recognition
Abdelwaheb Belaid and Jean-Paul Haton. A syntactic approach for handwritten mathematical formula recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1):105–111, 1984
work page 1984
-
[3]
A method for the structural analysis of two-dimensional mathe- matical expressions
Shi-Kuo Chang. A method for the structural analysis of two-dimensional mathe- matical expressions. information sciences, 2(3):253–272, 1970
work page 1970
-
[4]
Edge-featured graph attention network
Jun Chen and Haopeng Chen. Edge-featured graph attention network. arXiv preprint arXiv:2101.07671, 2021
arXiv 2021
-
[5]
Mathwriting: A dataset for handwritten mathematical expression recognition
Philippe Gervais, Asya Fadeeva, and Andrii Maksai. Mathwriting: A dataset for handwritten mathematical expression recognition. arXiv preprint arXiv:2404.10690, 2024
arXiv 2024
-
[6]
Iamondo-database: an online handwritten document database with non-uniform contents
Emanuel Indermühle, Marcus Liwicki, and Horst Bunke. Iamondo-database: an online handwritten document database with non-uniform contents. pages 97–104, 2010
work page 2010
-
[7]
International Organization for Standardization. Iso/iec 40314:2016 - information technology — mathematical markup language (mathml) version 3.0 2nd edition,
work page 2016
-
[8]
Detexify: Erkennung handgemalter latex-symbole, 2010
Daniel Kirsch. Detexify: Erkennung handgemalter latex-symbole, 2010
work page 2010
Show all 23 references
-
[9]
Visual parsing with query-driven global graph attention (qd-gga): preliminary results for handwritten math formula recog- nition
Mahshad Mahdavi and Richard Zanibbi. Visual parsing with query-driven global graph attention (qd-gga): preliminary results for handwritten math formula recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 570–571, 2020
2020
-
[10]
Icdar 2019 crohme+ tfd: Competition on recognition of hand- written mathematical expressions and typeset formula detection
Mahshad Mahdavi, Richard Zanibbi, Harold Mouchere, Christian Viard-Gaudin, and Utpal Garain. Icdar 2019 crohme+ tfd: Competition on recognition of hand- written mathematical expressions and typeset formula detection. In2019 Interna- tional Conference on Document Analysis and R...
2019
-
[11]
Line-of-sight with graph attention parser (lgap) for math formulas
Ayush Kumar Shah and Richard Zanibbi. Line-of-sight with graph attention parser (lgap) for math formulas. InInternational Conference on Document Analysis and Recognition, pages 401–419. Springer, 2023
2023
-
[12]
A survey on handwritten mathematical expression recog- nition: The rise of encoder-decoder and gnn models
Thanh-Nghia Truong, Cuong Tuan Nguyen, Richard Zanibbi, Harold Mouchère, and Masaki Nakagawa. A survey on handwritten mathematical expression recog- nition: The rise of encoder-decoder and gnn models. Pattern Recognition, page 110531, 2024
2024
-
[13]
Watt and Tom Underhill
Stephen M. Watt and Tom Underhill. Ink markup language (inkml). W3C Rec- ommendation, 2011. Accessed: 2023-01-08
2011
-
[14]
Hand- written mathematical expression recognition via paired adversarial learning.In- ternational Journal of Computer Vision, 128:2386–2401, 2020
Jin-Wen Wu, Fei Yin, Yan-Ming Zhang, Xu-Yao Zhang, and Cheng-Lin Liu. Hand- written mathematical expression recognition via paired adversarial learning.In- ternational Journal of Computer Vision, 128:2386–2401, 2020
2020
-
[15]
Graph- to-graph: towards accurate and interpretable online handwritten mathematical ex- pression recognition
Jin-WenWu,FeiYin,Yan-MingZhang,Xu-YaoZhang,andCheng-LinLiu. Graph- to-graph: towards accurate and interpretable online handwritten mathematical ex- pression recognition. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 2925–2933, 2021. The Re...
2021
-
[16]
Graph- to-graph: towards accurate and interpretable online handwritten mathematical ex- pression recognition
Jin-WenWu,FeiYin,Yan-MingZhang,Xu-YaoZhang,andCheng-LinLiu. Graph- to-graph: towards accurate and interpretable online handwritten mathematical ex- pression recognition. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 2925–2933, 2021
2021
-
[17]
Icdar 2023 crohme: Competition on recognition of handwritten mathematical expres- sions
Yejing Xie, Harold Mouchère, Foteini Simistira Liwicki, Sumit Rakesh, Rajkumar Saini, Masaki Nakagawa, Cuong Tuan Nguyen, and Thanh-Nghia Truong. Icdar 2023 crohme: Competition on recognition of handwritten mathematical expres- sions. In International Conference on Document An...
2023
-
[18]
Local and global graph mod- eling with edge-weighted graph attention network for handwritten mathematical expression recognition
Yejing Xie, Richard Zanibbi, and Harold Mouchère. Local and global graph mod- eling with edge-weighted graph attention network for handwritten mathematical expression recognition. arXiv preprint arXiv:2410.18555, 2024
2024 arXiv
-
[19]
Track, attend, and parse (tap): An end-to- end framework for online handwritten mathematical expression recognition.IEEE Transactions on Multimedia, 21(1):221–233, 2019
Jianshu Zhang, Jun Du, and Lirong Dai. Track, attend, and parse (tap): An end-to- end framework for online handwritten mathematical expression recognition.IEEE Transactions on Multimedia, 21(1):221–233, 2019
2019
-
[20]
Watch, attend and parse: An end-to-end neural network based ap- proach to handwritten mathematical expression recognition.Pattern Recognition, 71:196–206, 2017
Jianshu Zhang, Jun Du, Shiliang Zhang, Dan Liu, Yulong Hu, Jinshui Hu, Si Wei, and Lirong Dai. Watch, attend and parse: An end-to-end neural network based ap- proach to handwritten mathematical expression recognition.Pattern Recognition, 71:196–206, 2017
2017
-
[21]
Comer: Modeling coverage for transformer-based handwritten mathematical expression recognition
Wenqi Zhao and Liangcai Gao. Comer: Modeling coverage for transformer-based handwritten mathematical expression recognition. InEuropean conference on com- puter vision, pages 392–408. Springer, 2022
2022
-
[22]
Handwritten mathematical expression recognition with bidirectionally trained transformer
Wenqi Zhao, Liangcai Gao, Zuoyu Yan, Shuai Peng, Lin Du, and Ziyin Zhang. Handwritten mathematical expression recognition with bidirectionally trained transformer. In Document analysis and recognition–ICDAR 2021: 16th interna- tional conference, Lausanne, Switzerland, Septembe...
2021
-
[2016]
Published in February 2016; last reviewed and confirmed in 2024
2016
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.