Pith. sign in

REVIEW 5 major objections 6 minor 29 references

WriteViT: Handwritten Text Generation with Vision Transformer

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WriteViT claims a Vision Transformer pipeline can imitate a writer's style from one reference image and improve low-resource handwritten text recognition.

desk verdict Solid engineering with a genuine new configuration, but the paper's own OOV-U numbers undercut the headline one-shot generalization claim and the evaluation protocol needs tightening before the results can be taken at face value. read the letter →

arxiv 2505.13235 v1 pith:V5BWPSGB submitted 2025-05-19 cs.CV cs.LG

Dang Hoai Nam , Huynh Tong Dang Khoa , Vo Nguyen Le Duy This is my paper · ORCID
classification cs.CVcs.LG
keywords handwrittentextsynthesisVisionTransformerone-shotlearningVietnamesehandwritingmulti-scalegenerationgenerativeadversarialnetworksHTRdataaugmentationstyletransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces WriteViT, a one-shot handwriting synthesis system that uses Vision Transformers in the generator, the recognizer, and the writer identifier. The authors claim it produces handwriting that is more faithful to real samples than prior methods on both English (IAM) and Vietnamese (VNOnDB) datasets, measured by FID and KID. They further claim that augmenting a 5,000-image recognition training set with 25,000 WriteViT-generated images gives the best word error rate among all compared generators. If these claims hold, low-resource scripts could obtain useful training data without manual annotation.

What carries the argument

The central object is the ViT-based Writer Identifier, which encodes a real handwriting sample into a compact style embedding that conditions the generator; it is trained as a writer classifier on real images and frozen during generation. Around it, the generator stacks Transformer encoder blocks at multiple spatial scales with Conditional Positional Encoding (CPE), and a lightweight ViT-based recognizer transcribes generated images to enforce content fidelity. The asymmetric attention pairing, with character embeddings as keys and style embeddings as queries and values, is what ties textual content to visual style.

What would settle it

Test the model on a held-out set of writers and words that never appeared in training, with one reference image per writer, and compare FID against the same model given 15 references; if FID rises steeply with content novelty or reference count matters more than style, the one-shot cross-content transfer claim is refuted.

Watch

Extended reading notes

Core claim

WriteViT reports the lowest FID and KID on IAM ($11.102$ and $0.37$) and on VNOnDB ($6.1785$ and $0.51$) among all compared methods, and its synthetic images improve a Transformer-based HTR model to a word error rate of $5.76$, beating all baselines. The paper attributes this to replacing CNN/CRNN components with Vision Transformers throughout, adding a multi-scale generator with conditional positional encodings, and training a ViT-based writer identifier to produce style embeddings from a single reference image. The same framework is shown to handle Vietnamese diacritics, which the authors highlight as a first for this line of work.

Load-bearing premise

The style embedding learned by the ViT writer classifier on training writers must transfer to unseen words and unseen writers from a single reference image; if it captures identity without generalizing across content, the one-shot and out-of-vocabulary cases fail.

Editorial extensions

If this is right

  • Handwriting synthesis for low-resource languages could be built from just one reference image per writer instead of per-writer corpora.
  • HTR systems with a few thousand labeled images can be improved by synthetic augmentation without altering the recognizer architecture.
  • The multi-scale generator with CPE suggests that transformer-based handwriting generators can be made smaller than prior HWT-based designs while retaining quality.
  • The framework's reported success on Vietnamese, a diacritic-heavy script, suggests it may transfer to other under-resourced writing systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test is fine-tuning the writer embedding on a handful of new-writer samples to see whether quality improves beyond one-shot; the paper does not train per-writer adapters.
  • Because the recognizer loss couples content fidelity to a ViT trained alongside the generator, evaluating generated images with an independent HTR model would reveal how much of the WER gain is tied to the recognizer itself.
  • Since style transfer flows through a single embedding vector, interpolating between two writers' embeddings might produce blended styles—an operation the architecture supports but the paper does not explore.
  • The reported compactness of the generation modules suggests on-device deployment, but inference speed and memory usage during generation are not measured, so practical latency remains open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces WriteViT, a one-shot handwritten text generation framework built around Vision Transformers. The architecture combines a ViT-based writer identifier that extracts style embeddings from reference images, a multi-scale Transformer generator with conditional positional encodings, a lightweight ViT recognizer, and a GAN discriminator. The method is evaluated on the IAM (English) and HANDS-VNOnDB (Vietnamese) datasets against HiGAN, HiGAN+, HWT, VATr, and ScrabbleGAN. The paper reports the lowest FID and KID on both datasets (11.102/0.37 on IAM; 6.1785/0.51 on VNOnDB) and the best WER/CER when augmenting a 5,000-image HTR training set on IAM, with NED tied with VATr. The text further claims robust one-shot generalization to out-of-vocabulary words and unseen writers, and presents Vietnamese handwriting synthesis as a novel application. The paper includes an ablation study, model-size comparison, and qualitative figures.

Significance. If the results are reproducible, the paper offers a useful step for transformer-based handwriting synthesis: it extends the task to Vietnamese, presents multi-scale ViT generation with conditional positional encodings, evaluates several strong baselines, and demonstrates downstream HTR gains with an external TrOCR model. Strengths include the public implementation, the use of external feature extractors for FID/KID and for HTR evaluation (which mitigates circularity from the in-training recognizer), and the inclusion of low-resource HTR augmentation experiments. However, the central one-shot and generalization claims are not consistently supported by the paper's own tables: the OOV-U FID is worse than two baselines, the number of style reference images used in each experiment is not stated, and all metrics lack error bars. These issues need to be resolved before the stronger claims can be accepted.

major comments (5)
  1. [§4.2, Table 2] Table 2 reports WriteViT OOV-U FID = 30.87, which is worse than HWT (29.68) and VATr (29.50); WriteViT's OOV-S FID = 27.56 is also worse than both baselines (26.47 and 26.82). The text in §4.2 nevertheless states that 'our model maintains robust generalization when both content and style are novel' and highlights OOV-U as a scenario where the model is particularly strong. This is an internal contradiction. The generalization claim should be revised or supported by additional experiments; as it stands, the most challenging condition is exactly where the proposed method underperforms the two strongest baselines.
  2. [§3.2 and Tables 1–3, 6] Section 3.2 defines P=15 as the default number of style reference images and P=1 for the one-shot variant, but none of the quantitative tables (Tables 1, 2, 3, 6) states which P was used. If the headline results were obtained with P=15, the paper's central 'one-shot' claim is not tested; if they were obtained with P=1, the OOV-U result in Table 2 suggests the style embedding does not generalize as advertised. Every reported experiment must specify P, and the one-shot configuration should be evaluated in a dedicated table.
  3. [§4.1–4.3, Tables 1–3] All FID, KID, WER, CER, and NED values are reported as single point estimates with no error bars or confidence intervals. This makes small differences—e.g., WER 5.76 vs 6.21 and NED 3.09 vs 3.09 in Table 3—unassessable. The authors should report mean and standard deviation over multiple training runs, or at least provide bootstrapped confidence intervals for FID/KID and paired evaluation uncertainty for HTR, before claiming superiority over baselines.
  4. [§4, dataset splits and baselines] The baseline protocol in §4 is asymmetric: HWT and VATr are used through their publicly available pretrained checkpoints, while HiGAN, HiGAN+, and ScrabbleGAN are retrained from scratch on the same partitions as HWT/VATr. The paper does not state whether the pretrained checkpoints were trained on the same training writers/splits as the retrained baselines, nor how they were adapted to VNOnDB (e.g., zero-shot evaluation of English-pretrained models versus fine-tuning). Without this information, Tables 1–3 and 6 cannot be interpreted as a controlled comparison, and the claim of consistently superior quality is not yet supported.
  5. [§3.3.4, §4.2] The paper's novelty rests on the writer identifier W producing a style embedding from a single reference that transfers to arbitrary content and unseen writers, yet there is no quantitative evaluation of style consistency on generated images (e.g., writer identification or retrieval accuracy on synthetic samples). The OOV-U condition in Table 2 is the only quantitative proxy for this capability, and WriteViT does not win there. The authors should add a direct writer-consistency metric on one-shot and unseen-writer generations, or temper the generalization claim to match the available evidence.
minor comments (6)
  1. [Eq. (5)] Equation (5) uses σS in the denominator of the ∇W update, but S is not defined; it should presumably be σW. Please correct the notation.
  2. [§4.3, Table 3] The sentence 'our method outperforms all baselines across the three key HTR metrics' is inaccurate because NED for Ours and VATr are both 3.09; the claim should say 'matches or outperforms.'
  3. [§4.4, Table 4] The FID of the 'Base (CNN + CRNN)' row is 13.615, identical to the HWT number in Table 1. Please state explicitly whether the base model is HWT; otherwise the ablation appears to reuse a baseline number without explanation.
  4. [§4.1] The protocol says all images are resized to 32×128 with right-side padding/truncation, while §4 also says the width is 16 pixels per character; the handling of words longer than eight characters by truncation should be described, because it could distort FID/KID comparisons for long OOV words.
  5. [References] References [5] and [8] appear to be the same Graves arXiv technical report but are cited as separate works; please merge or disambiguate.
  6. [Figures 2–4] The qualitative figures are small and difficult to inspect; please provide high-resolution crops of representative characters and diacritics, especially for the Vietnamese comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all losses are supervised by external labels and headline metrics use independent pre-trained models.

full rationale

The paper's derivation chain is self-contained. The generator is trained with adversarial, recognition, and writer-identification losses (Eqs. 1–4); the recognition loss L_R is supervised by ground-truth transcriptions and the writer loss L_W by ground-truth writer identities, so the objectives are not defined in terms of the generated images' own outputs. The headline quality metrics (FID/KID) use a pre-trained Inception network, and the HTR evaluation uses TrOCR, an external pre-trained recognizer; neither is a module fitted in this paper. The Writer Identifier W is trained on real images and used as a feature extractor for style, which is a standard transfer procedure, not a fitted parameter renamed as a prediction. No self-citation chain or uniqueness theorem is invoked. Although the OOV-U FID in Table 2 (30.87) is worse than HWT (29.68) and VATr (29.50) despite the text claiming robust generalization, that is an internal-consistency/correctness concern, not circularity. Hence score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about style transfer, metric validity, recognizer loss behavior, and baseline comparability. The fixed hyperparameters alpha, beta, and the image preprocessing are chosen by hand. No new theoretical entities are introduced.

free parameters (4)
  • alpha (gradient balancing weight for recognizer) = 0.7
    Fixed hyperparameter in gradient balancing (Eq. 5), chosen by hand and not tuned per dataset; the central training dynamics depend on it.
  • beta (gradient balancing weight for writer identifier) = 0.7
    Fixed hyperparameter in gradient balancing (Eq. 5), chosen by hand; together with alpha it controls the relative influence of recognizer and writer losses.
  • Number of style reference samples P = 1 or 15, unclear which is used in main experiments
    Section 3.2 defines P=15 as common practice and P=1 for one-shot, but the experimental section does not state which P was used for the main results, despite the one-shot claim in the abstract.
  • Image resolution and character width = 32 pixels high, 16 pixels per character (32x128 after padding)
    All images are resized and padded to 32x128 for evaluation; this preprocessing choice affects FID/KID comparisons across methods that natively output different resolutions.
assumptions (4)
  • domain assumption A ViT-based writer identifier can extract a style embedding from a single reference image that transfers to arbitrary unseen content.
    Core to the one-shot claim; if the embedding does not generalize across content and writers, the generator cannot produce style-consistent OOV text (Sections 3.2 and 3.3.4).
  • domain assumption FID and KID computed with Inception features on resized 32x128 handwriting images are valid proxies for perceptual handwriting quality.
    All quantitative quality claims rely on these metrics; the Inception network was not trained on handwriting, and the fixed resizing may mask or introduce artifacts (Section 4.1).
  • domain assumption The recognizer loss on generated images improves content fidelity without causing the generator to produce images that are optimized for the recognizer rather than for human legibility.
    L_R is backpropagated through G; if the recognizer R has blind spots, the generator may exploit them, which would inflate downstream HTR gains (Section 3.4).
  • domain assumption The IAM and VNOnDB test splits, together with the pretrained HWT/VATr checkpoints, are directly comparable to models retrained from scratch.
    The baseline protocol mixes pretrained and retrained models; if the pretrained checkpoints used different data splits, the comparisons are not controlled (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of WriteViT: Handwritten Text Generation with Vision Transformer." pith.science (2026). https://pith.science/paper/V5BWPSGB

@misc{pith2026250513235,
  author       = {Pith},
  title        = {Pith review of: WriteViT: Handwritten Text Generation with Vision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5BWPSGB}},
  note         = {Machine review of arXiv:2505.13235}
}
read the original abstract

Humans can quickly generalize handwriting styles from a single example by intuitively separating content from style. Machines, however, struggle with this task, especially in low-data settings, often missing subtle spatial and stylistic cues. Motivated by this gap, we introduce WriteViT, a one-shot handwritten text synthesis framework that incorporates Vision Transformers (ViT), a family of models that have shown strong performance across various computer vision tasks. WriteViT integrates a ViT-based Writer Identifier for extracting style embeddings, a multi-scale generator built with Transformer encoder-decoder blocks enhanced by conditional positional encoding (CPE), and a lightweight ViT-based recognizer. While previous methods typically rely on CNNs or CRNNs, our design leverages transformers in key components to better capture both fine-grained stroke details and higher-level style information. Although handwritten text synthesis has been widely explored, its application to Vietnamese -- a language rich in diacritics and complex typography -- remains limited. Experiments on Vietnamese and English datasets demonstrate that WriteViT produces high-quality, style-consistent handwriting while maintaining strong recognition performance in low-resource scenarios. These results highlight the promise of transformer-based designs for multilingual handwriting generation and efficient style adaptation.

Figures

Figures reproduced from arXiv: 2505.13235 by the authors.

Figure 1
Figure 1. Overview of the proposed WriteViT architecture. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Qualitative comparison of generated handwriting from different models given [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Qualitative reconstruction results. Each row corresponds to a different model, [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of Vietnamese handwriting generation on the VNOnDB [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 8 canonical work pages

  1. [1]

    H. T. Nguyen, C. T. Nguyen, P. T. Bao, M. Nakagawa, A database of unconstrained vietnamese online handwriting and recognition exper- iments by recurrent neural networks, Pattern Recognition 78 (2018) 291–306.doi:https://doi.org/10.1016/j.patcog.2018.01.013. URLhttps://www.sciencedirect.com/science/article/pii/ S0031320318300141

    doi:10.1016/j.patcog.2018.01.013 2018

  2. [2]

    Kleber, S

    F. Kleber, S. Fiel, M. Diem, R. Sablatnig, Cvl-database: An off-line database for writer retrieval, writer identification and word spotting, in: 18 2013 12th International Conference on Document Analysis and Recog- nition, 2013, pp. 560–564.doi:10.1109/ICDAR.2013.117

    work page doi:10.1109/icdar.2013.117 2013

  3. [3]

    Pratikakis, K

    I. Pratikakis, K. Zagori, P. Kaddas, B. Gatos, Icfhr 2018 competition on handwritten document image binarization (h-dibco 2018), in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp. 489–493.doi:10.1109/ICFHR-2018.2018.00091

    arXiv 2018

  4. [4]

    R. D. Lins, Nabuco - two decades of document processing in latin amer- ica, J. Univers. Comput. Sci. 17 (2011) 151–161. URLhttps://api.semanticscholar.org/CorpusID:2896293

    2011

  5. [6]

    A. K. Bhunia, S. H. Khan, H. Cholakkal, R. M. Anwer, F. S. Khan, M. Shah, Handwriting transformers, 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021) 1066–1074. URLhttps://api.semanticscholar.org/CorpusID:233181822

    work page 2021

  6. [7]

    Marti, H

    U.-V. Marti, H. Bunke, The iam-database: an english sentence database for offline handwriting recognition, International Journal on Document Analysis and Recognition 5 (2002) 39–46. URLhttps://api.semanticscholar.org/CorpusID:29622813

    work page 2002

  7. [8]

    Graves, Generating sequences with recurrent neural networks (2014)

    A. Graves, Generating sequences with recurrent neural networks (2014). arXiv:1308.0850. URLhttps://arxiv.org/abs/1308.0850

    arXiv 2014

  8. [9]

    Hochreiter, J

    S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Com- put. 9 (8) (1997) 1735–1780.doi:10.1162/neco.1997.9.8.1735. URLhttps://doi.org/10.1162/neco.1997.9.8.1735

    doi:10.1162/neco.1997.9.8.1735 1997

Show all 29 references
  1. [10]

    D. Ha, D. Eck, A neural representation of sketch drawings (2017). arXiv:1704.03477

    2017 arXiv

  2. [11]

    Kotani, S

    A. Kotani, S. Tellex, J. Tompkin, Generating Handwriting via De- coupled Style Descriptors, Springer International Publishing, 2020, p. 764–780.doi:10.1007/978-3-030-58610-2_45. URLhttp://dx.doi.org/10.1007/978-3-030-58610-2_45 19

    2020 doi

  3. [12]

    L. Kang, P. Riba, Y. Wang, M. Rusi˜ nol, A. Fornés, M. Villegas, Gan- writing: Content-conditioned generation of styled handwritten word images, in: Computer Vision – ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII, Springer-Verl...

    2020 doi

  4. [13]

    Mattick, M

    A. Mattick, M. Mayr, M. Seuret, A. Maier, V. Christlein, Smart- Patch: Improving Handwritten Word Imitation with Patch Discrim- inators, Springer International Publishing, 2021, p. 268–283.doi: 10.1007/978-3-030-86549-8_18. URLhttp://dx.doi.org/10.1007/978-3-030-86549-8_18

    2021 doi

  5. [14]

    Fogel, H

    S. Fogel, H. Averbuch-Elor, S. Cohen, S. Mazor, R. Litman, Scrabble- gan: Semi-supervised varying length handwritten text generation, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  6. [15]

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks (2014).arXiv:1406.2661

    2014 arXiv

  7. [16]

    L. Kang, P. Riba, M. Rusinol, A. Fornes, M. Villegas, Content and style aware generation of text-line images for handwriting recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (12) (2022) 8846–8860.doi:10.1109/tpami.2021.3122572. URLhttp://dx.doi.or...

    2022

  8. [17]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017,...

    2017

  9. [18]

    X. Liu, G. Meng, S. Xiang, C. Pan, Handwritten text generation via disentangled representations, IEEE Signal Processing Letters 28 (2021) 1838–1842.doi:10.1109/LSP.2021.3109541. 20

    2021

  10. [19]

    J. Gan, W. Wang, Higan: Handwriting imitation conditioned on arbitrary-length texts and disentangled styles, in: AAAI Conference on Artificial Intelligence, 2021, pp. 7484–7492

    2021

  11. [20]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale (2021).arXiv:2010.11929. URLhttps://arxiv.org/...

    2021 arXiv

  12. [21]

    B. Shi, X. Bai, C. Yao, An end-to-end trainable neural network for image-basedsequencerecognitionanditsapplicationtoscenetextrecog- nition (2015).arXiv:1507.05717. URLhttps://arxiv.org/abs/1507.05717

    2015 arXiv

  13. [22]

    X. Chu, Z. Tian, B. Zhang, X. Wang, C. Shen, Conditional positional encodings for vision transformers (2021).arXiv:2102.10882

    2021 arXiv

  14. [23]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition (2015).arXiv:1512.03385

    2015 arXiv

  15. [24]

    J. H. Lim, J. C. Ye, Geometric gan (2017).arXiv:1705.02894. URLhttps://arxiv.org/abs/1705.02894

    2017 arXiv

  16. [25]

    Alonso, B

    E. Alonso, B. Moysset, R. Messina, Adversarial generation of hand- written text images conditioned on sequences, in: 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 481–486.doi:10.1109/ICDAR.2019.00083

    2019

  17. [26]

    Heusel, H

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S.Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equi- librium (2018).arXiv:1706.08500. URLhttps://arxiv.org/abs/1706.08500

    2018 arXiv

  18. [27]

    Bi´ nkowski, D

    M. Bi´ nkowski, D. J. Sutherland, M. Arbel, A. Gretton, Demystifying mmd gans (2021).arXiv:1801.01401. URLhttps://arxiv.org/abs/1801.01401

    2021 arXiv

  19. [28]

    J. Gan, W. Wang, J. Leng, X. Gao, Higan+: Handwriting imitation gan with disentangled representations, ACM Trans. Graph. 42 (1) (2022). 21 doi:10.1145/3550070. URLhttps://doi.org/10.1145/3550070

    2022 doi

  20. [29]

    Pippi, S

    V. Pippi, S. Cascianelli, R. Cucchiara, Handwritten Text Generation from Visual Archetypes, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  21. [30]

    M. Li, T. Lv, J. Chen, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li, F. Wei, Trocr: Transformer-based optical character recognition with pre-trained models (2022).arXiv:2109.10282. URLhttps://arxiv.org/abs/2109.10282 22

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.