REVIEW 4 major objections 4 minor 1 cited by
Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead
T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read A temporary text-alignment branch, used only during training and discarded afterward, lets an IMU-based handwriting recognizer cut character error by up to 10.4% with zero extra inference cost.
desk verdict A genuinely new contrastive training loss with zero inference overhead, but the headline gains are split-dependent and lack error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a temporary auxiliary branch consisting of a character-level Transformer text encoder and an attention-pooling module that compresses the CNN's time-series features into a fixed vector. Two contrastive objectives are applied during training: the in-batch contrastive loss, which treats other pairs in the batch as negatives and enforces a soft alignment between sensor and text embeddings, and the error-based contrastive loss, which builds hard negatives by applying single-character delete/insert/substitute edits to the ground-truth transcript and demands a larger distance to those. The auxiliary branch is removed after training, so the original model—with its original
What would settle it
Measure CER on a writer-dependent split whose validation words share no character n-grams with the training words. If ECHWR does not beat the CTC-only baseline on that split, the claimed zero-overhead gain on novel vocabulary is falsified.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a sequence-to-sequence handwriting recognizer can be trained to encode a margin between correct and near-correct transcripts. The framework, ECHWR, adds two contrastive objectives to the usual CTC objective: an in-batch loss that aligns each sensor embedding with the embedding of its matching transcript, and an error-based loss that contrasts the sensor embedding against transcripts generated by single-character deletion, insertion, or substitution. The auxiliary text encoder and attention pooling that produce these embeddings are discarded after training, so inference sees exactly the same architecture as the baseline. Evaluations on OnHW-Word
Load-bearing premise
The paper assumes that the text encoder trained from scratch on the training split can produce stable anchor embeddings for word combinations it has never seen; its own writer-dependent results show this assumption breaks down (adding the error-based loss raises CER from 12.95% to 14.04%), and the qualitative section states the text encoder 'struggles to generalize its embeddings to the unseen word combinations.'
Editorial extensions
If this is right
- Because the auxiliary branch is removed after training, the deployed recognizer has exactly the same architecture, memory footprint, and inference latency as the baseline, so the accuracy gains come at zero runtime cost.
- On the writer-independent split, where the vocabulary is fixed but writers are unseen, the error-based contrastive loss provides the largest gain, indicating that hard negatives help when the input distribution shifts.
- On the writer-dependent split, where writers are known but vocabulary is unseen, the in-batch alignment loss alone gives the largest gain, indicating that semantic alignment helps when the label distribution shifts.
- Adding error-based discrimination to a model that already has good alignment can hurt if the text encoder cannot embed unseen words; thus the benefit of hard-negative contrastive learning depends on the stability of the label anchor.
Reading between the lines
- If a pretrained or far larger text encoder were used, the error-based loss might also help the writer-dependent split, where the current model's text embeddings fail on unseen words; this would likely unify the two configurations the paper reports.
- Hard negatives could be generated from the recognizer's own confusions (e.g., characters that the model often mispredicts) rather than random edits, which would make the margin task better aligned with the model's actual error modes.
- The reported gains on OnHW-Words500 come with a training-time memory and compute overhead for the auxiliary branch; a practical follow-up would measure whether the accuracy gain survives when the training budget is fixed and the extra compute is instead spent on longer training of the baseline.
- The task-dependent dichotomy suggests a testable hypothesis: on a dataset with both unseen writers and unseen vocabulary, a single configuration with moderate hard-negative weight may be optimal—neither of the two extremes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ECHWR, a training framework for IMU-based online handwriting recognition. An auxiliary text-encoder branch is trained alongside the primary CTC-based recognizer using two contrastive objectives: an in-batch contrastive loss (BC) aligning sensor and text embeddings, and an error-based contrastive loss (EC) that separates the ground-truth transcript from synthetic single-edit hard negatives. The auxiliary branch is removed after training, so inference uses the unmodified REWI architecture. Experiments on the right-handed OnHW-Words500 dataset report CER/WER for writer-dependent (WD) and writer-independent (WI) splits, with headline relative CER reductions of up to 10.4% (WD) and 7.4% (WI). The paper also includes ablations on normalization, gated attention, registers, and the number of synthetic error sets.
Significance. If the reported gains were robust, this would be a useful practical contribution: the zero-inference-overhead property is attractive for edge-device deployment, and the idea of using a temporary semantic alignment branch as a regularizer is clearly formulated. The release of code and the explicit loss definitions are also strengths. However, the current evidence is a proof of concept rather than a rigorous state-of-the-art claim: the results come from a single dataset, the best WD number is obtained with a configuration that excludes the error-based contrastive loss, and no uncertainty quantification or statistical testing is reported. With small absolute improvements and per-split architecture selection, the central quantitative claims are not yet established.
major comments (4)
- [Section 4.3, Table 1] The abstract's WD improvement of 10.4% is not achieved by the full ECHWR method as defined by Eq. (1). On the WD split, the full objective CTC+BC+EC gives CER 14.04%, while CTC+BC alone gives 12.95%. Thus the headline WD gain comes from the BC-only variant, not from the proposed error-based contrastive loss. In fact, adding EC degrades WD CER by about 8% relative to BC-only. The authors should clearly separate claims for the BC-only and full frameworks, and either report the full-framework WD result as the main result or justify why the BC-only variant is part of ECHWR.
- [Sections 4.3-4.5] All CER/WER numbers appear to be single point estimates over 5-fold cross-validation with no standard deviations, confidence intervals, or statistical tests. Given that the full ECHWR/B WD gain is only 0.41 pp (14.45 to 14.04) and the WI gain is 0.54 pp, these differences could easily be within run-to-run variation. Moreover, the architecture (RMSNorm+GA for WD, LayerNorm+GA+Registers for WI) and the error-set size (Fig. 2) appear to be selected after inspecting test-fold results. The authors should report fold-level means and standard deviations, state how many random seeds were used, and clarify whether validation folds or test folds were used for model selection.
- [Section 4.6 and Appendix A] The paper's own qualitative analysis states that the text encoder 'struggles to generalize its embeddings to the unseen word combinations' on the WD split, and Appendix A shows that the WD train/validation character distributions differ substantially (some characters, e.g., 'q' and 'ä', are absent from the validation set). Since both contrastive losses rely on the auxiliary text branch, this is a direct limitation of the proposed framework's stability on exactly the split where the largest headline gain is reported. The authors should connect this limitation to their quantitative claims and provide evidence that the WD BC-only gain is not an artifact of the misaligned text embeddings.
- [Section 4.4, Table 2] There is no single ECHWR configuration that performs best on both splits: WD favors RMSNorm+GA, while WI favors LayerNorm+GA+Registers. The current framing suggests that ECHWR is a unified method, but the experiments actually provide per-split recipes. This is acknowledged in the conclusion, but the abstract and introduction nevertheless state that 'ECHWR significantly outperforms' baselines. The authors should either present a unified configuration, or explicitly frame the contribution as a family of contrastive regularizers whose optimal architecture is task-dependent, and adjust the claims accordingly.
minor comments (4)
- [Table 1 caption] The terms ECHWR/S and ECHWR/B are not defined in the main text. Please define the small and base variants before the table.
- [Section 4.5, Fig. 2] The sensitivity plots would benefit from numerical values and error bars; as shown, it is difficult to judge whether the differences between error-set sizes are meaningful. Also, the central legend applies to all subplots, but the legend text is small and hard to read.
- [Abstract/Introduction] The phrase 'zero inference overhead' is repeated several times. The fact that the auxiliary branch is discarded is clear; avoid over-emphasizing a property that follows trivially from the design.
- [Appendix A] The appendix reference appears as 'Appx. A' in Section 4.1; please use 'Appendix A' for consistency.
Circularity Check
No significant circularity; the training objective is a supervised regularizer evaluated on a public benchmark.
full rationale
The paper's central claim is an empirical comparison on the public OnHW-Words500 dataset. The composite objective L_total = L_CTC + L_BC + L_EC is a supervised training loss: the auxiliary text branch is trained from ground-truth transcripts and then discarded, so the deployed model is unchanged at inference. Reported CER/WER values come from evaluating the trained recognizer on held-out folds against the published REWI baseline. No predicted quantity is defined in terms of fitted values, and no load-bearing argument reduces to a self-citation: the REWI baseline is an independently published prior architecture used as a comparison point, not as the evidence for the gains. The paper's own negative result for the EC objective on the writer-dependent split (CER worsens from 12.95% to 14.04% when EC is added) further shows that the reported improvements are not forced by the loss construction. Concerns about test-fold architecture selection and missing error bars pertain to statistical robustness, not circularity.
Assumptions & free parameters
free parameters (4)
- error set size S =
2
- per-split architecture selection =
RMSNorm+GA (WD); LayerNorm+GA+Reg (WI)
- contrastive temperature tau =
learnable
- loss weights =
1, 1, 1
assumptions (4)
- standard math InfoNCE contrastive loss induces useful alignment between modalities
- domain assumption Aligning sensor features to the ground-truth text embedding improves the primary CTC objective
- ad hoc to paper Synthetic single-edit (Levenshtein distance 1) negatives are effective hard negatives
- domain assumption Character-level tokenization of text is the right granularity for handwriting alignment
Cite this review
Pith. "Pith review of Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead." pith.science (2026). https://pith.science/paper/I4OZIV3B
@misc{pith2026260207049,
author = {Pith},
title = {Pith review of: Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead},
year = {2026},
howpublished = {\url{https://pith.science/paper/I4OZIV3B}},
note = {Machine review of arXiv:2602.07049}
}
read the original abstract
Online handwriting recognition using inertial measurement units opens up handwriting on paper as input for digital devices. Doing it on edge hardware improves privacy and lowers latency, but entails memory constraints. To address this, we propose Error-enhanced Contrastive Handwriting Recognition (ECHWR), a training framework designed to improve feature representation and recognition accuracy without increasing inference costs. ECHWR utilizes a temporary auxiliary branch that aligns sensor signals with semantic text embeddings during the training phase. This alignment is maintained through a dual contrastive objective: an in-batch contrastive loss for general modality alignment and a novel error-based contrastive loss that distinguishes between correct signals and synthetic hard negatives. The auxiliary branch is discarded after training, which allows the deployed model to keep its original, efficient architecture. Evaluations on the OnHW-Words500 dataset show that ECHWR significantly outperforms state-of-the-art baselines, reducing character error rates by up to 7.4% on the writer-independent split and 10.4% on the writer-dependent split. Finally, although our ablation studies indicate that solving specific challenges require specific architectural and objective configurations, error-based contrastive loss shows its effectiveness for handling unseen writing styles.
Figures
Forward citations
Cited by 1 Pith paper
-
Beyond Decision Boundaries: Relational Geometry Attacks on Contrastive Embedding Manifolds
A feed-forward generator trained offline can corrupt and invert the pairwise similarity structure of contrastive verification systems using small bounded image perturbations.
Reference graph
Works this paper leans on
-
[1]
In: 2012 16th International Symposium on Wearable Computers
Amma, C., Georgi, M., Schultz, T.: Airwriting: Hands-free mobile text input by spotting and continuous recognition of 3d-space handwriting with inertial sensors. In: 2012 16th International Symposium on Wearable Computers. pp. 52–59 (2012). https://doi.org/10.1109/ISWC.2012.21
-
[2]
In: Proceedings of the 37th International Conference on Machine Learning
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. ICML’20, JMLR.org (2020)
2020
-
[3]
In: 2006 IEEE International Conference on Information Acquisition
Choi, S.d., Lee, A.S., Lee, S.y.: On-line handwritten character recognition with 3d accelerometer. In: 2006 IEEE International Conference on Information Acquisition. pp. 845–850 (2006). https://doi.org/10.1109/ICIA.2006.305842
arXiv 2006
-
[4]
In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=2dnO3LLiJ1
Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need regis- ters. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=2dnO3LLiJ1
2024
-
[5]
In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C.K., Li, X., Guan, C.: Time-series representation learning via temporal and contextual contrasting. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. pp. 2352–2359 (2021)
2021
-
[6]
Faundez-Zanuy, M., Mekyska, J.: Privacy of online handwriting biometrics re- lated to biomedical analysis, chap. Chapter 2, pp. 17–39. IET (2017). https: //doi.org/10.1049/PBSE004E_ch2, https://digital-library.theiet.org/doi/abs/10. 1049/PBSE004E_ch2
-
[7]
Guo, P., Nakayama, M.: Towards user-generalizable wearable-sensor-based hu- man activity recognition: A multi-task contrastive learning approach. Sen- sors25(22) (2025). https://doi.org/10.3390/s25226988, https://www.mdpi.com/ 1424-8220/25/22/6988
-
[8]
IEEE Sensors Journal 15(1), 154–163 (2015)
Hsu, Y.L., Chu, C.L., Tsai, Y.J., Wang, J.S.: An inertial pen with dynamic time warping recognizer for handwriting and gesture recognition. IEEE Sensors Journal 15(1), 154–163 (2015). https://doi.org/10.1109/JSEN.2014.2339843
arXiv 2015
Show all 29 references
-
[9]
In: Proceedings of the 2nd International Conference on Advances in Computer Science and Engineering (CSE 2013)
Jeen-Shing, W., Yu-Liang, H., Cheng-Ling, C.: Online handwriting recognition using an accelerometer-based pen device. In: Proceedings of the 2nd International Conference on Advances in Computer Science and Engineering (CSE 2013). pp. 231–234. Atlantis Press (2013). https://doi...
2013 doi
-
[10]
In: 2024 International Joint Conference on Neural Networks (IJCNN)
Jiang, R., Nguyen, T., Ishwar, P., Aeron, S.: Supervised contrastive learning with hard negative samples. In: 2024 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2024). https://doi.org/10.1109/IJCNN60899.2024.10650863
2024
-
[11]
In: Proceedings of the 34th International Confer- ence on Neural Information Processing Systems
Kalantidis,Y.,Sariyildiz,M.B.,Pion,N.,Weinzaepfel,P.,Larlus,D.:Hardnegative mixing for contrastive learning. In: Proceedings of the 34th International Confer- ence on Neural Information Processing Systems. NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)
2020
-
[12]
In: 2022 IEEE International Joint Conference on Biometrics (IJCB)
Khaertdinov, B., Asteriadis, S.: Temporal feature alignment in contrastive self- supervised learning for human activity recognition. In: 2022 IEEE International Joint Conference on Biometrics (IJCB). pp. 1–9 (2022). https://doi.org/10.1109/ IJCB54206.2022.10007984
2022
-
[13]
Kiyasseh, D., Zhu, T., Clifton, D.A.: Clocs: Contrastive learning of cardiac signals acrossspace,time,andpatients.In:InternationalConferenceonMachineLearning. pp. 5606–5615. PMLR (2021) Enhancing IMU-Based OnHWR via CL with Zero Inference Overhead 15
2021
-
[14]
In: Durmaz Incel, Ö., Qin, J., Bieber, G., Kuijper, A
Li, J., Hamann, T., Barth, J., Kämpf, P., Zanca, D., Eskofier, B.: Robust and efficient writer-independent imu-based handwriting recognition. In: Durmaz Incel, Ö., Qin, J., Bieber, G., Kuijper, A. (eds.) Sensor-Based Activity Recognition and Artificial Intelligence. pp. 261–28...
2026
-
[15]
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., Wei, F.: Trocr: transformer-based optical character recognition with pre-trained mod- els. In: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelli- gence and Thirty-Fifth Conferenc...
2023
-
[16]
Pattern Recognition158, 110967 (2025)
Li, Y., Chen, D., Tang, T., Shen, X.: Htr-vt: Handwritten text recognition with vi- sion transformer. Pattern Recognition158, 110967 (2025). https://doi.org/https: //doi.org/10.1016/j.patcog.2024.110967, https://www.sciencedirect.com/science/ article/pii/S0031320324007180
2025
-
[17]
SIGSPATIAL ’22, Association for Computing Machinery, New York, NY, USA (2022)
Liu, X., Liang, Y., Huang, C., Zheng, Y., Hooi, B., Zimmermann, R.: When do con- trastive learning signals help spatio-temporal graph forecasting? In: Proceedings of the 30th International Conference on Advances in Geographic Information Sys- tems. SIGSPATIAL ’22, Association ...
2022
-
[18]
Journal of Open Source Software3(29), 861 (2018)
McInnes, L., Healy, J., Saul, N., Großberger, L.: Umap: Uniform manifold ap- proximation and projection. Journal of Open Source Software3(29), 861 (2018). https://doi.org/10.21105/joss.00861, https://doi.org/10.21105/joss.00861
2018 doi
-
[19]
Ott, F., Rügamer, D., Heublein, L., Hamann, T., Barth, J., Bischl, B., Mutschler, C.: Benchmarking online sequence-to-sequence and character-based handwriting recognition from imu-enhanced pens. Int. J. Doc. Anal. Recognit.25(4), 385– 414 (Dec 2022). https://doi.org/10.1007/s1...
2022 doi
-
[20]
In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025), https: //openreview.net/forum?id=1b7whO4SfY
Qiu, Z., Wang, Z., Zheng, B., Huang, Z., Wen, K., Yang, S., Men, R., Yu, L., Huang, F., Huang, S., Liu, D., Zhou, J., Lin, J.: Gated attention for large lan- guage models: Non-linearity, sparsity, and attention-sink-free. In: The Thirty- ninth Annual Conference on Neural Infor...
2025
-
[21]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021), https://arxiv.org/abs/ 2103.00020
2021 arXiv
-
[22]
Robinson, J.D., Chuang, C.Y., Sra, S., Jegelka, S.: Contrastive learning with hard negativesamples.In:InternationalConferenceonLearningRepresentations(2021), https://openreview.net/forum?id=CR1XOQ0UTh-
2021
-
[23]
In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processi...
2017
-
[24]
In: Lladós, J., Lopresti, D., Uchida, S
Wehbi, M., Hamann, T., Barth, J., Kaempf, P., Zanca, D., Eskofier, B.: Towards an imu-based pen online handwriting recognizer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) Document Analysis and Recognition – ICDAR 2021. pp. 289– 16 J. Li et al
2021
-
[25]
In: 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)
Yousef, M., Bishop, T.E.: Origaminet: Weakly-supervised, segmentation-free, one- step, full page text recognition by learning to unfold. In: 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 14698–14707 (2020). https://doi.org/10.1109/CVPR42600....
2020
-
[26]
In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R
Zhang, B., Sennrich, R.: Root mean square layer normalization. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran As- sociates, Inc. (2019), https://proceedings.neuri...
2019
-
[27]
IEEE Transactions on Pattern Analysis and Machine Intelligence47(4), 3049–3066 (2025)
Zhang, P., Liu, Y., Lai, S., Li, H., Jin, L.: Privacy-preserving biometric verification with handwritten random digit string. IEEE Transactions on Pattern Analysis and Machine Intelligence47(4), 3049–3066 (2025). https://doi.org/10.1109/TPAMI. 2025.3529022
2025
-
[28]
In: Proceedings of the 36th International Conference on Neural Information Processing Systems
Zhang, X., Zhao, Z., Tsiligkaridis, T., Zitnik, M.: Self-supervised contrastive pre- training for time series via time-frequency consistency. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, Curran Associates Inc., Red Ho...
2022
-
[303]
https://doi.org/10.1007/ 978-3-030-86334-0_19
Springer International Publishing, Cham (2021). https://doi.org/10.1007/ 978-3-030-86334-0_19
2021
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.