{"id":"76ff8c02-d7c4-49a1-ae01-003316249f06","arxiv_id":"2412.15853","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A style-embedding masked autoencoder and a semi-supervised training scheme let a handwritten-text diffusion model generate training images for a new, unlabeled handwriting dataset, improving downstream recognition error.","lead":"This paper extends a latent diffusion model for handwritten text generation so it can imitate writers it never saw during training, using a masked autoencoder to encode style. The authors also add a semi-supervised training step that uses unlabeled target-domain images, and report lower recognition error when synthetic RIMES data trains a handwriting recognizer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RIMES semi-supervised gain may reflect memorization: the unlabeled adaptation set and the CER-train replication set are the same RIMES train images; no disjoint-split or quantitative OOV test is reported.","rationale":"I read the paper in good faith: the architecture is a reasonable extension of Wordstylist, the IAM-only versus semi-supervised comparisons are consistent across style inclusions in Table 5b, and retraining DiffusionPen under the same data split is an appropriate control. The central claim, however, is about generation for a previously unseen dataset, and the RIMES evaluation as designed cannot distinguish between (a) learning a transferable writer-style/content representation and (b) memorizing or heavily borrowing from the unlabeled RIMES images seen during adaptation. Since CER-train replicates the same train split used for adaptation, the 9.99% to 6.61% comparison is not a clean test of generalization to new words or new writers. The qualitative OOV failure in Sec. 5.6 adds weight to this concern. The reader's weakest assumption, that the MAE averaged patch embedding encodes calligraphic style, is plausible but also unverified; my concern is more immediate because it targets the evaluation design rather than an internal representation. The conditional verdict remains appropriate: the method is promising, but the specific RIMES numbers should not be taken at face value until a disjoint adaptation/generation split or a quantitative OOV evaluation is provided.","tokens_in":27749,"tokens_out":6947,"duration_ms":69831,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the RIMES CER-train improvement from 9.99% (IAM-only) to 6.61% (semi-supervised) in Table 9. But the evaluation protocol in Sec. 5.2 defines CER-train as replicating the train set, i.e. generating the same text/style pairs as the original train set. The unlabeled RIMES images used for semi-supervised adaptation in Sec. 4.5 are exactly those same train images. Thus the diffusion model has seen the target images during training, and the measured improvement may reflect direct exposure to, or memorization of, the adaptation set rather than a transferable ability to generate new content in the target style. The paper reports no RIMES out-of-vocabulary metric and no split between the images used for adaptation and the images generated for evaluation. Qualitative evidence in Sec. 5.6 is consistent with this concern: the model fails to render 'cartes', a RIMES word not seen during training even as unlabeled data, for an unknown writer. This does not disprove the method, but it means the headline claim about generating usable training images for a previously unseen dataset is only demonstrated on the adaptation set's own vocabulary and writers. The reader's MAE style-fidelity concern is related: both concerns test whether the learned representation generalizes, but the overlap between adaptation and evaluation data is the more immediate threat to the specific numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the Wordstylist latent diffusion model for handwritten text generation (HTG) by introducing a masked-autoencoder-based style encoder, several content-encoder variants for combining text and style conditioning, classifier-free guidance, and a semi-supervised training scheme for adapting the model to a new unlabeled dataset. The method is evaluated on IAM as the fully labeled source dataset and RIMES as the target dataset, using downstream HTR character error rates on generated training images as the main metric. The paper reports that semi-supervised adaptation reduces RIMES CER-train from about 9.99% (IAM-only) to 6.61% (semi-supervised), and concludes that the approach enables generation of usable training images for previously unseen datasets.","tokens_in":27886,"tokens_out":7972,"duration_ms":69698,"significance":"If the central claim holds, the work is significant for practical HTG applications: it would allow adapting a diffusion-based handwriting generator to a new corpus using only unlabeled writer-grouped images, with downstream HTR training as the payoff. The paper's strengths include a clear architectural description, a broad ablation of style-conditioning placements and guidance scales, and quantitative comparisons against two public reference implementations (Wordstylist and DiffusionPen), which is more extensive than many HTG reports. The main limitation is that the headline RIMES improvement is measured on the same image set used for adaptation, so the external validity of the 'unseen dataset' claim is not yet established. The MAE style-fidelity assumption is also not directly validated. These gaps are fixable with additional experiments, but they are load-bearing for the paper's main claim.","major_comments":[{"comment":"The central RIMES improvement may reflect overlap between the adaptation set and the evaluation set. CER-train is defined in Sec. 5.2 as replicating the train set, i.e., generating the same text/style conditioning pairs as the original train set. In Sec. 4.5, the semi-supervised training uses RIMES train images without transcriptions; in Sec. 5.5 these are exactly the images whose text/style pairs are later replicated for CER-train. The model therefore sees the target images during adaptation (albeit without paired text labels), and the reported gain from 9.99% to 6.61% in Table 9 may be attributable to exposure to the adaptation images rather than to a transferable ability to generate new content in RIMES style. The authors should evaluate on a disjoint split of the target dataset, report a RIMES out-of-vocabulary metric, or otherwise separate the images used for adaptation from those used for CER-train. The qualitative failure on 'cartes' in Sec. 5.6 is consistent with the need for such a test.","section":"Sec. 5.2/4.5, Table 9"},{"comment":"There is an internal inconsistency in the reported RIMES CER-train values for the same configuration. Table 9 lists 'Ours (IAM only)' as 10.12 at wgs=2 and 9.99 at wgs=5, while the TS column of Table 5a, which corresponds to the same setup (DM trained only on IAM, CFG with puncond=0.1, TS style inclusion), lists 9.58 and 9.86. Since Table 9 uses the reported IAM-only result as the baseline against which the semi-supervised improvement is measured, the authors must reconcile these numbers and clarify which model and MAE training set produced each entry.","section":"Sec. 5.7, Table 9 vs Sec. 5.5, Table 5a"},{"comment":"The claim that the MAE style embedding captures calligraphic style is not directly tested. The paper assumes that the unmasking objective 'pushes the model to encode information about the strokes' and that reconstruction forces the encoder to store style, but no writer-retrieval, writer-identification, or style-similarity metric is reported. The unseen-writer capability, the semi-supervised RIMES gains, and the comparison to DiffusionPen all depend on the averaged MAE embedding actually conveying per-writer style rather than mostly content. I recommend adding a quantitative style-fidelity evaluation, e.g., writer retrieval accuracy on held-out writers or a style-consistency measure on generated images.","section":"Sec. 4.4"},{"comment":"All results appear to come from single runs without error bars or multiple seeds. Many of the comparisons that support the paper's claims differ by less than one CER point (for example, several entries in Tables 5 and 6 differ by 0.1-0.5 percentage points), so the reader cannot assess whether these differences are significant. At minimum, the key semi-supervised comparison in Table 9 should be repeated with multiple seeds and reported with mean and standard deviation.","section":"Tables 3-6, 9"}],"minor_comments":[{"comment":"The text refers to an 'HRT model' in the paragraph defining Diff-IV; this should be 'HTR model'.","section":"Sec. 5.2"},{"comment":"The phrase 'genration of online and offline handwriting' contains a typo; it should read 'generation'.","section":"Sec. 2.2"},{"comment":"The notation SE is overloaded: SE denotes the encoder in Eq. (33) and the writer embedding function in Eq. (34). Rename one of these, e.g., use a separate symbol for the averaged writer embedding.","section":"Eq. (34)"},{"comment":"The table caption states that MAEs were trained on different datasets and refers to a column 'MAE DS', but Table 5a has no such column. Please clarify which MAE training set was used for Table 5a or restructure the table.","section":"Table 5"},{"comment":"The sentence 'we replace the sequence of embedding vectors for the text conditioning with the same mask tokens as for classifier-free guidance' is vague; it should specify whether the mask token is appended to or substituted for the text sequence and how this interacts with the timestep embedding in the TS variant.","section":"Sec. 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a single-author technical report with a useful architecture and a generally fair comparison to related work. The main issue is not circularity in the sense of conditioning on the evaluation labels, but rather an evaluation-set overlap that weakens the headline claim. The internal inconsistency between Tables 5a and 9 also needs to be resolved before the results can be trusted. I do not see any indication of misleading intent, but the reproducibility bar for this kind of claim is high without multiple runs or code release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper extends Wordstylist with an MAE-based style encoder and a semi-supervised training scheme for DDPM-based handwritten text generation. The genuinely new piece is using a masked autoencoder trained only with the unmasking objective to derive writer embeddings from example images, plus the semi-supervised adaptation that incorporates unlabeled target-domain data into diffusion training. The evaluation is thorough: multiple style-inclusion strategies, classifier-free guidance sweeps, and a downstream HTR CER metric that makes sense for the application. The internal consistency across Table 5b (semi-supervised) versus 5a (IAM-only) is real evidence that the method does something.\n\nThe soft spots are serious, and one is more serious than the reader flagged. The reader worried about MAE style fidelity, and that concern stands: the paper states the unmasking task pushes the model to encode strokes, but never measures whether averaged patch embeddings actually carry per-writer style. The paper itself admits the style embedding may be inferior. But the stress-test note hits the headline claim harder. CER-train replicates the train set, and the semi-supervised training uses the exact same RIMES train images as unlabeled data. So the improvement from 9.99% to 6.61% may be memorization or direct exposure, not a transferable ability to generate new words in the target style. The paper reports no RIMES OOV metric and no disjoint split between adaptation and evaluation. The qualitative example where \"cartes\" fails for an unknown writer is consistent with the memorization concern. Also, all tables come from single runs, and guidance scales are selected from the same test tables, which makes the specific numbers fragile.\n\nFor a serious revision, the author should rerun with a proper split: adapt on a subset of RIMES train, evaluate on held-out RIMES words/writers, and report OOV CER. A style-fidelity metric, such as writer classification accuracy on generated images, would also help verify the MAE assumption. The idea is relevant, the writing is clear, and the comparison to DiffusionPen on the same split is fair. But the central quantitative claim is not yet trustworthy.\n\nThis deserves peer review because the subfield needs this kind of adaptation method, and the flaws are fixable. I would not cite the headline numbers yet, but I would read a revised version carefully.","headline":"Useful extension of Wordstylist to unseen writers via MAE style embeddings, but the headline RIMES gain is suspect because the semi-supervised adaptation set and the CER-train evaluation set are the same RIMES train images.","tokens_in":701,"tokens_out":934,"would_cite":false,"duration_ms":28410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A latent diffusion model can learn a new, unseen writer's style from unlabeled images via masked-autoencoder embeddings; semi-supervised training on the new dataset lowers generated-data character error rate from 9.99% to 6.61%.","keywords":["handwritten text generation","latent diffusion models","masked autoencoder","style embedding","semi-supervised learning","domain adaptation","classifier-free guidance","handwriting recognition"],"falsifier":"Train the masked autoencoder exactly as described, then measure writer-identification accuracy on the averaged embeddings for held-out writers: if a linear classifier trained on writer labels performs near chance, or if the embeddings cluster by word content rather than by writer, the central mechanism is not carrying style information and the unseen-writer and semi-supervised results would need another explanation.","tokens_in":27357,"feed_emoji":"✍️","tokens_out":13423,"duration_ms":92495,"temperature":0.7,"pith_summary":"The paper aims to make latent diffusion models for handwritten text generation (HTG) usable for new, previously unseen handwriting datasets that come without word transcriptions. It claims that a masked autoencoder trained only to reconstruct randomly masked patches produces writer style embeddings when its patch vectors are averaged, so the diffusion model can imitate writers it never saw during training. Building on the Wordstylist architecture, the paper adds a content encoder that injects the style embedding into the text conditioning in several ways, uses classifier-free guidance, and proposes a semi-supervised training scheme that adds unlabeled target-domain images with masked text conditioning. The payoff would be practical: a handwriting recognition system could be adapted to a new corpus (here RIMES, with its own language and distribution) using only writer-grouped, unlabeled images, cutting the character error rate of synthetic training data from 9.99% to 6.61%.","feed_headline":"Cut synthetic handwriting error from 9.99% to 6.61%","feed_subtitle":"Unlabeled images teach new writer styles, improving synthetic training data for handwriting recognition.","key_machinery":"The load-bearing component is the masked autoencoder (MAE) style encoder, which computes a writer embedding by averaging the patch-level latent vectors of $K$ example images: $\\mathrm{SE}(w) = \\frac{1}{K N}\\sum_{i=1}^{K}\\sum_{j=1}^{N}\\mathrm{SE}({}^i x_p^j)$. The paper assumes the unmasking reconstruction objective forces the encoder to store stroke and calligraphic information, not just letter content. Around this encoder, the content encoder offers six ways to inject the style vector into the text conditioning (TP, TPL, CP, TA, CA, TS), with TS—adding the projected style vector to the timestep embedding—performing best. The semi-supervised trick is to replace the text-conditioning sequence with the same mask tokens used for classifier-free guidance, so unlabeled target images contribute style information while the model keeps learning from labeled examples.","core_discovery":"The paper's central claim is that a latent diffusion model conditioned on a masked-autoencoder style embedding can generate readable word images in the style of writers never seen during training, and that the same model can be adapted to an entirely new dataset without any transcriptions. In the RIMES experiment, a model trained only on IAM achieves a character error rate of 9.99% when asked to replicate RIMES training data, whereas the semi-supervised model—trained on labeled IAM plus unlabeled RIMES images whose text conditioning is replaced by mask tokens—reaches 6.61%, approaching the 3.25% of a model trained with full RIMES labels. The paper presents this as the first demonstration of semi-supervised training for DDPM-based HTG and as a step toward using generated images to adapt downstream recognition models to new domains.","pith_inferences":["The paper never directly measures whether the MAE embeddings encode style; a writer-discrimination test on the averaged embeddings would validate or undermine the mechanism without retraining the diffusion model.","Part of the semi-supervised gain may come from matching the target dataset's image-level statistics (ink, background, preprocessing) rather than from true calligraphic transfer; if so, a simpler domain-adaptation module might achieve part of the improvement.","The semi-supervised scheme could be extended to cases where even the writer grouping is unknown, replacing the style embedding with a nearest-neighbor style estimate, which the paper leaves as future work.","Retraining the frozen VAE on handwritten documents, rather than using a natural-image VAE, could combine with this semi-supervised scheme and potentially close the remaining gap to fully supervised generation."],"forward_implications":["An HTR system can be adapted to a new handwriting dataset, even in a different language, using only writer-grouped images and no transcriptions.","Semi-supervised training improves the synthetic training data for the new dataset, cutting CER-train from 9.99% to 6.61%, approaching the 3.25% achieved with full labels.","Classifier-free guidance with a moderate scale improves generation fidelity for known datasets, while for unseen datasets too-large guidance can hurt.","Unseen writers can be imitated from around ten example images, which is practical when one page implies one writer.","The choice of MAE training data (IAM-only vs. IAM+RIMES) has little influence on the final RIMES generation quality, suggesting the style representation transfers across datasets."],"supporting_citations":[{"why":"Supplies the Wordstylist latent diffusion architecture that this work extends and the main in-domain baseline.","marker":"[63]"},{"why":"Provides the masked autoencoder whose encoder is adapted into the style encoder computing writer embeddings.","marker":"[76]"},{"why":"Defines the masked-autoencoder unmasking objective that the style encoder is trained with, the mechanism assumed to encode strokes.","marker":"[27]"},{"why":"Introduces classifier-free guidance, the conditioning-drop scheme used for training and the guidance used at sampling.","marker":"[31]"},{"why":"Provides the latent diffusion model and the frozen pre-trained VAE that the generative backbone is built on.","marker":"[69]"},{"why":"The IAM database is the fully labeled training and evaluation dataset.","marker":"[54]"},{"why":"The RIMES database is the unseen, unlabeled target dataset in the semi-supervised adaptation experiments.","marker":"[23]"},{"why":"The HTR model used to compute CER-train, the metric that measures generation quality as training data.","marker":"[67]"},{"why":"DiffusionPen is the concurrent style-encoder baseline that this work compares against for unseen-writer generation.","marker":"[64]"}],"fun_headline_variants":["Diffusion models learn new handwriting styles from unlabeled images","Semi-supervised diffusion models cut handwriting generation error by a third","Adapting diffusion models to unseen handwriting styles with unlabeled data","Unlabeled images help diffusion models mimic unfamiliar handwriting","Semi-supervised training improves synthetic handwriting for recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method collapses if a masked autoencoder trained only to reconstruct masked patches does not actually encode calligraphic style in its averaged patch embeddings; the paper states this as an assumption and never directly measures whether the embeddings distinguish writers better than they distinguish content.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion models learn new handwriting styles from unlabeled images","Semi-supervised diffusion models cut handwriting generation error by a third","Adapting diffusion models to unseen handwriting styles with unlabeled data","Unlabeled images help diffusion models mimic unfamiliar handwriting","Semi-supervised training improves synthetic handwriting for recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3072,"prompt_tokens":931,"completion_tokens":2141,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2060}},"tokens_in":547,"tokens_out":2141,"duration_ms":12666,"temperature":1.0,"reasoning_tokens":2060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:01:50.458719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the masked autoencoder exactly as described, then measure writer-identification accuracy on the averaged embeddings for held-out writers: if a linear classifier trained on writer labels performs near chance, or if the embeddings cluster by word content rather than by writer, the central mechanism is not carrying style information and the unseen-writer and semi-supervised results would need another explanation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Wordstylist latent diffusion architecture that this work extends and the main in-domain baseline."},{"cited_title":"Brandenbusch invariant autoencoder for text recognition and document enhancement","cited_arxiv_id":null,"evidence_quote":"Provides the masked autoencoder whose encoder is adapted into the style encoder computing writer embeddings."},{"cited_title":"In: IEEE/CVF Conf","cited_arxiv_id":null,"evidence_quote":"Provides the latent diffusion model and the frozen pre-trained VAE that the generative backbone is built on."},{"cited_title":"In: Int.Conf.onDocumentAnalysisandRecognition.pp.1398–1402.Barcelona,Spain (2009)","cited_arxiv_id":null,"evidence_quote":"The RIMES database is the unseen, unlabeled target dataset in the semi-supervised adaptation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The HTR model used to compute CER-train, the metric that measures generation quality as training data."},{"cited_title":"In: European Conf","cited_arxiv_id":null,"evidence_quote":"DiffusionPen is the concurrent style-encoder baseline that this work compares against for unseen-writer generation."}],"review_version":1}