REVIEW 3 major objections 3 minor 30 references
MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition
T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Pairing a Transformer branch with a CNN counting branch improves handwritten math expression recognition on three CROHME benchmarks.
desk verdict A plausible but weakly supported extension: combining PosFormer's transformer branch with CAN-style counting improves CROHME results modestly, but the evidence is undercut by test-set-driven tuning and missing stats. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the multi-viewer architecture itself: two parallel decoders—a Position-Forest Transformer viewer and a CNN-GRU counting viewer—sharing a DenseNet encoder. The load-bearing identity is the combined loss \(L_{\mathrm{all}} = \lambda_1 L_{\mathrm{rec}} + \lambda_2 L_{\mathrm{pos}} + \lambda_3 L_{\mathrm{counting}}\), which ties the Transformer's symbol recognition and position-forest objectives to the CNN branch's symbol-counting regression (a smooth-L1 loss). The Multi-Scale Counting Module in the CNN branch, with 3x3 and 5x5 convolutional kernels, channel attention, and sum pooling, is what produces the counting signal; the position forest produces the structural signal. The argument is that these two views—local counting and global structure—push the shared backbone to learn features neither decoder would learn alone.
What would settle it
Train the same architecture with \(\lambda_3\) selected on a held-out validation split drawn from the CROHME training data, then evaluate once on the official test sets; if the resulting margins over PosFormer shrink to near zero or become negative, the reported improvement is an artifact of test-set-driven selection. A second check: run the full recipe with several random seeds and report the variance of the ExpRate margins.
Extended reading notes
Core claim
MMHMER's core claim is that a CNN viewer and a Transformer viewer, trained jointly as two observers of the same input, produce better recognition than the Transformer alone. Concretely, a DenseNet backbone feeds two branches: a Transformer decoder equipped with a position forest and implicit attention correction, and a GRU-based counting-aware decoder. The training objective is \(L_{\mathrm{all}} = \lambda_1 L_{\mathrm{rec}} + \lambda_2 L_{\mathrm{pos}} + \lambda_3 L_{\mathrm{counting}}\), with \(\lambda_1=1\), \(\lambda_2=0.5\), and \(\lambda_3=0.1\) chosen by experiment. On CROHME14/16/19 with scale augmentation the model reports 63.96%, 62.51%, and 65.55% ExpRate, exceeding PosFormer by 1.28, 1.48, and 0.58 percentage points; on the MNE nested-expression set it reports gains of 1.44%, 0.65%, and 0.21% on N1/N2/N3. The paper also reports that adding a third task (Task2, text prediction) degrades performance markedly, which it interprets as evidence that task balance, not task count, is what matters.
Load-bearing premise
The reported gain is only convincing if the CROHME test sets were used solely for final evaluation and not for choosing the loss weight \(\lambda_3\), the task set, or other hyperparameters; the paper does not report a held-out validation split, repeated runs, or significance tests.
Editorial extensions
If this is right
- If the reported numbers hold, fusing a position-forest Transformer with a counting CNN gives a new state of the art on CROHME14, CROHME16, and CROHME19 under scale augmentation.
- Symbol counting (Task1) alone gives only small or negative gains on the Transformer baseline, so the multi-viewer joint training—not the counting loss per se—is what the paper credits for the improvement.
- Adding an additional text-prediction task (Task2) drops accuracy by 5.4–8.0 percentage points, implying that multi-task HMER frameworks must limit the number and type of auxiliary tasks.
- The gains extend to nested expressions: the paper reports consistent increases on the MNE N1/N2/N3 subsets, suggesting the multi-viewer setup helps structured complexity, not just overall accuracy.
Reading between the lines
- An implicit consequence the paper does not test: the same two-viewer recipe could transfer to other image-to-sequence problems (e.g., formula detection or scene text recognition) where a counting branch and a structure-aware branch can be defined.
- The paper motivates a consistency constraint between the two viewers but the loss function shown contains no explicit consistency term; a testable extension is to add one and measure whether the reported margins grow.
- Because the gains are reported without variance or significance testing, a cautious reading is that the true effect may be smaller than the point estimates; a re-run with multiple seeds on a held-out validation split would settle the size of the effect.
- The MNE results suggest the method helps most on shallow nesting (N1: +1.44%) and least on deep nesting (N3: +0.21%), which, if confirmed, points to where the CNN counting view stops adding information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MMHMER, a multi-viewer, multi-task framework for handwritten mathematical expression recognition (HMER). It combines a Transformer-based branch following PosFormer with a CNN/GRU branch following CAN that performs symbol counting. The two branches are trained jointly with a combined loss, and the paper claims improved ExpRate over PosFormer on CROHME 2014, 2016, and 2019 (63.96%, 62.51%, 65.55% vs. 62.68%, 61.03%, 64.97%) and on the MNE nested-expression test set. The main contribution is stated as an online collaborative multi-viewer strategy that lets CNN and Transformer complement each other.
Significance. If the reported gains are reliable, the paper would demonstrate a simple and effective way to combine a counting-based CNN decoder with a position-forest Transformer decoder for HMER, a direction that the authors note is underexplored. The work uses standard public benchmarks, reports comparisons with several recent methods, and includes ablations of the counting task and the loss weight. However, the core evidence is empirical and the reported improvements are small (0.58–1.48 percentage points on test sets of roughly 1,000–1,200 expressions). The manuscript does not currently provide the statistical or methodological support needed to distinguish these gains from test-set-driven selection or random variation, so the central claim should be treated as unverified until that support is added.
major comments (3)
- [§IV, Tables III and IV] The final configuration appears to be selected on the same CROHME test sets that are then used to report the headline results. Table IV varies λ3 over {0.01, 0.1, 0.5} and reports the best value (0.1) on each test set; Table III adds Task2 and then removes it after observing large drops (5.75, 5.40, and 8.01 points) on those same test sets. No held-out validation split, no multiple-seed runs, and no significance tests are reported. The claimed gains over PosFormer (1.28, 1.48, and 0.58 points) are within the range that could arise from test-set-driven selection, especially given the small test-set sizes. The central claim that MMHMER outperforms PosFormer is therefore not established by the evidence as presented.
- [Abstract vs. Table I] The abstract reports 65.46% ExpRate on CROHME19 and an absolute gain of 0.58% over PosFormer, while Table I reports 65.55% for CROHME19 (a 0.58% gain relative to the 64.97% PosFormer result). The abstract numbers are internally inconsistent: 65.46% would imply a 0.49% gain. This discrepancy must be resolved before the reported results can be relied upon.
- [§IV, Table III] The ablation shows that the counting task alone ('+Task1') reduces CROHME16 ExpRate from 61.03 to 60.85, while the text states that symbol counting 'contributes to a slight enhancement' in HMER performance. No explanation is given for the negative result on CROHME16. Since the multi-task contribution is a central part of the paper's claim, the interaction between the counting task and the multi-viewer mechanism needs explicit analysis rather than a summary statement over inconsistent numbers.
minor comments (3)
- [§III-C, Eq. (7)] Only λ3 is ablated in Table IV; λ1 and λ2 are fixed at 1 and 0.5, respectively, with no sensitivity analysis. A brief discussion or supplementary ablation of these weights would strengthen the paper.
- [General] The manuscript contains several typographical and formatting errors, including an author email with an embedded space ('zhong lifan@yahoo.co.jp'), duplicate reference [23] identical to [11], and a repeated value in Table I for PosFormer on CROHME19 (87.24 appears twice in the ≤3 column).
- [Figure 2] Figure 2 is referenced in the text but no actual images are visible in the provided manuscript; the caption 'Sample Images from CROHME 2014/2016/2019 Datasets' appears but the figure content is missing.
Circularity Check
Headline CROHME accuracies are selected, not predicted: λ3 and the task set (Task2 dropped) were chosen on the same test sets that produce the Table I gains over PosFormer.
-
fitted input called prediction
[Section IV, Table IV and adjacent ablation text (λ3 selection)]
"As shown in Table IV, the symbol λ is used to balance the trade-off between tasks for CNNs and transformers in the experiments above. In our setting, λ=0.1 performs the best, whereas a smaller λ would reduce the learning capability of the CNN. On the other hand, a larger λ presents issues and can lead to performance degradation because the network might converge in the wrong direction."
λ3 was varied over {0.01, 0.1, 0.5} and the value yielding the highest ExpRate on the same CROHME14/16/19 test sets used in Table I was chosen (λ3=0.1). The headline numbers 63.96/62.51/65.55 are therefore the maximum over the tried λ values on those exact test sets. Reporting these as MMHMER's predicted performance is equivalent to reporting the result of fitting a hyperparameter to the evaluation set; the gains over PosFormer are part of the selection criterion, not an independent prediction.
-
fitted input called prediction
[Section IV, Table III and ablation discussion (Task2 removal)]
"As we integrate additional tasks into our model, such as text prediction designated as Task2, we notice a marked deterioration in the overall performance metrics. Specifically, the inclusion of this extra task results in a decrease in performance by 5.75% on the 2014 dataset, 5.40% on the 2016 dataset, and a more pronounced drop of 8.01% on the 2019 dataset."
The final architecture reported in Table I was obtained by first adding Task2, observing large drops on the same CROHME14/16/19 test sets, then removing it. The reported configuration is thus selected to maximize performance on the very test sets that later serve as the evidence for the claimed 0.58-1.48% gains over PosFormer. The ablation table is the selection record, and the same numbers are reused as the prediction, so the headline comparison is statistically forced rather than an independent evaluation.
full rationale
No load-bearing self-citation was found: reference [33] (SemiHMER) is a related-work self-citation only and does not justify the central architecture. The CNN/Transformer fusion and the losses in Eq. (7) are not defined in terms of the target ExpRate, so the method has independent empirical content. The circular component is confined to test-set-driven selection: Table IV chooses λ3 because it 'performs the best' on CROHME14/16/19, and Table III removes Task2 after observing drops on those same sets; Table I then reuses those selected configurations as the headline result. There is no held-out validation split, repeated-seed statistics, or significance test, and the abstract's CROHME19 value (65.46%) differs from Table I (65.55%). These issues make the reported gains over PosFormer a selected optimum rather than a verified prediction, giving partial circularity (6/10) rather than a fully circular derivation.
Assumptions & free parameters
free parameters (5)
- λ3 (counting loss weight) =
0.1
- λ1 (recognition loss weight) =
1 (default)
- λ2 (position loss weight) =
0.5 (default)
- Transformer decoder layers =
3
- Counting convolution kernel sizes =
3x3 and 5x5
assumptions (3)
- domain assumption The CROHME training/test splits and ground truth LaTeX labels are correct and representative of handwritten mathematical expression recognition.
- domain assumption The PosFormer baseline results in Table I are reproduced under the same training conditions.
- standard math Standard cross-entropy and smooth L1 losses are appropriate for the recognition, position, and counting tasks.
Cite this review
Pith. "Pith review of MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition." pith.science (2026). https://pith.science/paper/DC7SZCBZ
@misc{pith2026250205557,
author = {Pith},
title = {Pith review of: MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/DC7SZCBZ}},
note = {Machine review of arXiv:2502.05557}
}
read the original abstract
Handwritten Mathematical Expression Recognition (HMER) methods have made remarkable progress, with most existing HMER approaches based on either a hybrid CNN/RNN-based with GRU architecture or Transformer architectures. Each of these has its strengths and weaknesses. Leveraging different model structures as viewers and effectively integrating their diverse capabilities presents an intriguing avenue for exploration. This involves addressing two key challenges: 1) How to fuse these two methods effectively, and 2) How to achieve higher performance under an appropriate level of complexity. This paper proposes an efficient CNN-Transformer multi-viewer, multi-task approach to enhance the model's recognition performance. Our MMHMER model achieves 63.96%, 62.51%, and 65.46% ExpRate on CROHME14, CROHME16, and CROHME19, outperforming Posformer with an absolute gain of 1.28%, 1.48%, and 0.58%. The main contribution of our approach is that we propose a new multi-view, multi-task framework that can effectively integrate the strengths of CNN and Transformer. By leveraging the feature extraction capabilities of CNN and the sequence modeling capabilities of Transformer, our model can better handle the complexity of handwritten mathematical expressions.
Figures
Reference graph
Works this paper leans on
-
[1]
Winkler H J. HMM-based handwritten symbol recognition using on- line and off-line features[C]//1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings. IEEE, 1996, 6: 3438-3441
work page 1996
-
[2]
Hu L, Zanibbi R. HMM-based recognition of online handwritten math- ematical symbols using segmental k-means initialization and a modified pen-up/down feature[C]//2011 International conference on Document analysis and Recognition. IEEE, 2011: 457-462
work page 2011
-
[4]
Keshari B, Watt S. Hybrid mathematical symbol recognition using sup- port vector machines[C]//Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). IEEE, 2007, 2: 859-863
work page 2007
-
[5]
Densely connected convo- lutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition
Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convo- lutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 4700-4708
2017
-
[6]
Liu J, Sun H, Katto J. Learned image compression with mixed transformer-cnn architectures[C]//Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. 2023: 14388-14397
work page 2023
-
[7]
CNN-transformer mixed model for object detection
Li W. CNN-transformer mixed model for object detection[J]. arXiv preprint arXiv:2212.06714, 2022
work page Pith review arXiv 2022
-
[8]
Zhu J, Luo Y , Zheng X, et al. A good student is cooperative and reliable: CNN-transformer collaborative learning for semantic segmen- tation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11720-11730
work page 2023
-
[9]
Zhang, Z., Zhang, Y . (2022). Combining CNN and Transformer as En- coder to Improve End-to-End Handwritten Mathematical Expression Recognition Accuracy. In: Porwal, U., Forn´es, A., Shafait, F. (eds) Fron- tiers in Handwriting Recognition. ICFHR 2022. Lecture Notes in Com- puter Science, vol 13639. Springer, Cham. https://doi.org/10.1007/978- 3-031-21648-0 13
doi:10.1007/978- 2022
Show all 30 references
-
[10]
Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition[J]
Zhang J, Du J, Zhang S, et al. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition[J]. Pattern Recognition, 2017, 71: 196-206
2017
-
[12]
When counting meets HMER: counting-aware network for handwritten mathematical expression recog- nition[C]//European conference on computer vision
Li B, Yuan Y , Liang D, et al. When counting meets HMER: counting-aware network for handwritten mathematical expression recog- nition[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 197-214
2022
-
[13]
Semantic graph representation learning for handwritten mathematical expression recognition[C]//International conference on document analysis and recognition
Liu Z, Yuan Y , Ji Z, et al. Semantic graph representation learning for handwritten mathematical expression recognition[C]//International conference on document analysis and recognition. Cham: Springer Nature Switzerland, 2023: 152-166
2023
-
[14]
Zhang J, Du J, Dai L. A gru-based encoder-decoder approach with attention for online handwritten mathematical expression recogni- tion[C]//2017 14th IAPR international conference on document analysis and recognition (ICDAR). IEEE, 2017, 1: 902-907
2017
-
[15]
Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recogni- tion[J]
Zhang J, Du J, Dai L. Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recogni- tion[J]. IEEE Transactions on Multimedia, 2018, 21(1): 221-233
2018
-
[16]
Zhao W, Gao L, Yan Z, et al. Handwritten mathematical expres- sion recognition with bidirectionally trained transformer[C]//Document analysis and recognition–ICDAR 2021: 16th international conference, Lausanne, Switzerland, September 5–10, 2021, proceedings, part II 16. Spring...
2021
-
[17]
Comer: Modeling coverage for transformer-based hand- written mathematical expression recognition[C]//European conference on computer vision
Zhao W, Gao L. Comer: Modeling coverage for transformer-based hand- written mathematical expression recognition[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 392- 408
2022
-
[18]
PosFormer: recognizing com- plex handwritten mathematical expression with position forest trans- former[C]//European Conference on Computer Vision
Guan T, Lin C, Shen W, et al. PosFormer: recognizing com- plex handwritten mathematical expression with position forest trans- former[C]//European Conference on Computer Vision. Springer, Cham, 2025: 130-147
2025
-
[19]
In: Proc
Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr 2014 competition on recognition of on-line handwritten mathematical expres- sions (crohme 2014). In: Proc. of International Conference on Frontiers in Handwriting Recognition. pp. 791–796 (2014)
2014
-
[20]
In: ICFHR
Mouch `ere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr2016 crohme: Competition on recognition of online handwritten mathematical expressions. In: ICFHR. pp. 607–612 (2016)
2016
-
[21]
In: ICDAR
Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: Icdar 2019 crohme+ tfd: Competition on recognition of handwritten mathematical expressions and typeset formula detection. In: ICDAR. pp. 1533–1538 (2019)
2019
-
[23]
Multi-scale attention with dense encoder for hand- written mathematical expression recognition[C]//2018 24th international conference on pattern recognition (ICPR)
Zhang J, Du J, Dai L. Multi-scale attention with dense encoder for hand- written mathematical expression recognition[C]//2018 24th international conference on pattern recognition (ICPR). IEEE, 2018: 2245-2250
2018
-
[24]
Kosmala A, Rigoll G, Lavirotte S, et al. On-line handwritten formula recognition using hidden Markov models and context dependent graph grammars[C]//Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR’99 (Cat. No. PR00318). IEEE, 1999: 107-110
1999
-
[25]
Elastic structural matching for online handwritten alphanumeric character recognition[C]//Proceedings
Chan K F, Yeung D Y . Elastic structural matching for online handwritten alphanumeric character recognition[C]//Proceedings. Fourteenth Interna- tional Conference on Pattern Recognition (Cat. No. 98EX170). IEEE, 1998, 2: 1508-1511
1998
-
[26]
Towards a web-based progressive hand- writing recognition environment for mathematical problem solving[J]
Vuong B Q, He Y , Hui S C. Towards a web-based progressive hand- writing recognition environment for mathematical problem solving[J]. Expert Systems with Applications, 2010, 37(1): 886-893
2010
-
[27]
Learning phrase represen- tations using RNN encoder-decoder for statistical machine translation[J]
Cho K, Van Merri ¨enboer B, Gulcehre C, et al. Learning phrase represen- tations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014
2014 arXiv
-
[28]
A tree-based structure-aware transformer decoder for image-to-markup generation[C]//Proceedings of the 30th ACM International Conference on Multimedia
Zhong S, Song S, Li G, et al. A tree-based structure-aware transformer decoder for image-to-markup generation[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 5751-5760
2022
-
[29]
NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition[C]//European Conference on Computer Vision
Liu C, Pan J, Hu J, et al. NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition[C]//European Conference on Computer Vision. Springer, Cham, 2025: 273-291
2025
-
[30]
Attention is all you need[J]
Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017
2017
-
[31]
The pascal visual object classes (voc) challenge[J]
Everingham M, Van Gool L, Williams C K I, et al. The pascal visual object classes (voc) challenge[J]. International journal of computer vision, 2010, 88: 303-338
2010
-
[32]
Faster R-CNN: Towards real-time object detection with region proposal networks[J]
Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149
2016
-
[33]
SemiHMER: Semi-supervised Handwritten Mathe- matical Expression Recognition using pseudo-labels[J]
Chen K, Shen H. SemiHMER: Semi-supervised Handwritten Mathe- matical Expression Recognition using pseudo-labels[J]. arXiv preprint arXiv:2502.07172, 2025
2025 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.