{"id":"f52abedb-6694-42d6-bae5-125865a53a37","arxiv_id":"2411.17332","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 336-case out-of-distribution benchmark of eight handwritten text recognition models shows a roughly 30 to 40 point CER drop, and the authors argue that textual mismatch, not visual style, drives the drop.","lead":"Handwritten text recognition models trained on one manuscript corpus transcribe unseen historical or foreign-language domains with character error rates around 37 to 54 percent, far above their in-domain rates. This study maps which domain differences matter and argues that textual or linguistic mismatch is the main driver, with visual writing style playing a weaker role.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section 5.2 estimator's label-free status is ambiguous: if it uses ΔGT, OOD error prediction leaks target transcripts; if it uses ΔL, no evidence supports the 70% accuracy claim.","rationale":"The reader's weakest_assumption is exactly the spot I would attack: Section 5.2's estimator is described as label-free, but the KL-divergence input is not specified, and one of the two defined divergences (ΔGT) uses target transcripts. The benchmark itself—336 OOD evaluations, eight architectures, seven datasets—is valuable and the raw OOD degradation finding is credible; I am not questioning the experiments. What is load-bearing is the abstract's quantitative claim about estimating OOD error. If the reported 10.9 MAE / 70% figure depends on ΔGT, it is partly an artifact of label leakage; if it depends on ΔL, the paper has not shown that this proxy alone or with the other metrics achieves that accuracy. The factor-ordering claim is secondary and already hedged in the conclusion, but the estimator ambiguity directly undermines the paper's strongest claimed contribution. A minimal re-run excluding ΔGT settles it. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":87,"tokens_out":5657,"duration_ms":112706,"concrete_test":"Re-run the Section 5.2 prediction experiment twice: once with the full metric set including ΔGT, once with ΔGT removed (retaining ΔL, ID error, ID ECE, parameter count, and reconstruction errors). If the MAE/70%-within-10-points figure degrades materially without ΔGT, the label-free estimator is not supported; if ΔGT's regression coefficient dominates, the claim is circular. Also report a leave-one-source-out split to check out-of-sample predictive accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central predictive claim—that OOD error can be estimated with discrepancies below 10 CER points in 70% of cases (Abstract; Section 5.2)—depends on which 'KL divergence' is fed to the estimator. Section 5.1 defines two textual metrics: ΔGT(YS,YT), the divergence between source text and the target's ground-truth transcriptions, and ΔL(YS,YT), the divergence between source text and a synthetic corpus in a known language L. Section 5.2 states that the estimator uses 'metrics that do not require target labels... and KL divergence' without specifying which metric. If ΔGT is included, the estimator uses the very target transcripts used to compute OOD error, so the 70% figure partly measures leakage, not prediction. If instead only ΔL is used, the paper reports no ablation or standalone result for a ΔL-only estimator, and no evidence that the within-10-point accuracy survives without ΔGT. Moreover, ΔL is not label-free in the domain-generalization sense: selecting the synthetic corpus requires knowing the target language, which is target-domain knowledge the DG setup (Section 3.3) claims to exclude.\n\nThe factor-analysis ordering is also weaker than the abstract states: Factor 2, labeled visual divergence, loads negatively (-0.3) on OOD error in Fig. 6, so the data do not show visual divergence as a second positive driver; the conclusion itself hedges ('We emphasize that more research studying this last factor has to be done'). The primary concern remains the unspecified estimator input, because the quantitative headline claim is either circular or unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale empirical study of out-of-distribution (OOD) generalization for handwritten text recognition (HTR). Eight HTR architectures are trained on seven real datasets across five languages, yielding 336 OOD evaluations, plus additional experiments with synthetic text rendered in five languages. The authors report that OOD error is much higher than in-distribution error, that CTC-based alignment generalizes best, that synthetic data give a modest OOD improvement, and that model selection and capacity have little effect. They then perform a factor analysis over features that include visual and textual divergence proxies, concluding that textual divergence is the most significant factor and visual divergence is second. Finally, they claim that OOD error can be estimated from label-free proxies with a mean absolute error of 10.9 CER points and that about 70% of predictions fall below 10 points.","tokens_in":25227,"tokens_out":5521,"duration_ms":51830,"significance":"If the claims are supported, the paper would be a valuable first systematic study of HTR under the domain-generalization setting, with practical guidance for practitioners and a useful public benchmark. The raw empirical finding—that OOD CER is dramatically higher than ID CER across all architectures and datasets—is robust and consistently presented in Tables 3–7, and the release of code is a concrete asset. The synthetic-data comparison and the analysis of model-selection strategies are also informative. However, the two headline claims—the factor ordering and the label-free OOD-error estimator—are not yet established by the evidence as presented. The factor-ordering conclusion conflicts with the displayed loadings, and the estimator section leaves unspecified whether the KL-divergence input uses ground-truth target transcriptions or a synthetic proxy, which is decisive for the label-free claim. These issues affect the central contributions and require a revision.","major_comments":[{"comment":"The estimator described in Section 5.2 is said to use 'metrics that do not require target labels... and KL divergence', but the manuscript does not specify which KL divergence is used. Section 5.1 defines two textual divergences: ΔGT(YS,YT), computed from ground-truth target transcriptions, and ΔL(YS,YT), computed against a synthetic corpus. If ΔGT enters the estimator, the label-free claim is false and the reported 70%-below-10-points result partly measures leakage of the target labels that are also used to compute the OOD CER. If ΔL is used instead, the paper provides no ablation or standalone result showing that a ΔL-only estimator achieves the claimed accuracy; moreover, selecting the synthetic corpus requires knowing the target language, which the DG setup in Section 3.3 explicitly excludes. Please specify which divergence is used and provide a separate evaluation for the truly label-free version.","section":"Section 5.2"},{"comment":"The abstract's ordering claim—'textual divergence... followed by visual divergence'—is not supported by the displayed factor loadings. Factor 2, labeled visual divergence, loads 0.80 on ΔS and 0.92 on ΔT but has a negative loading of -0.30 on OOD error, so the visual factor is not a positive driver of OOD error in this analysis. Furthermore, because OOD error is itself an input to the factor analysis, the loadings describe shared variance rather than a causal or predictive ordering; no regression or variance decomposition is reported that would quantify the relative contribution of textual versus visual divergence to OOD error. The conclusion in Section 6 appropriately hedges ('We emphasize that more research studying this last factor has to be done'), but the abstract does not, and the two should be made consistent.","section":"Section 5.1, Fig. 6"},{"comment":"The OOD-error estimator is described only as 'we calculated the expected error'; the manuscript does not state what model is fit, how the coefficients are obtained, or whether the evaluation is held-out or cross-validated. If the estimator is fit and evaluated on the same 336 cases, the reported MAE of 10.9 and the cumulative residual distribution are training-set diagnostics, and the phrase 'can be reliably estimated' overstates predictive validity. Please describe the fitting procedure and report a held-out or cross-validated evaluation.","section":"Section 5.2, Fig. 7"},{"comment":"The claimed threshold is numerically inconsistent with the cumulative plot. In Fig. 7 (right), the cumulative percentages at x-axis positions 0, 5, 10, 15 are approximately 29.1, 50.8, 68.3, 85.2, which implies that about 50.8% of residuals are below 10 CER points and about 68.3% are below 15 points. The abstract's 'discrepancies falling below 10 points in 70% of cases' and the Fig. 7 caption's 'below 10 points of CER' do not match this plot. Please correct the text or clarify the binning; if the intended statement is 'below 15 points', the abstract and caption must be revised accordingly.","section":"Abstract, Section 5.2, Fig. 7"}],"minor_comments":[{"comment":"The averages in Table 3 are computed after filtering out non-converged runs marked with a star, but no sensitivity analysis is provided. Since non-converged runs are concentrated in particular architectures, please report whether the qualitative conclusions (e.g., the ranking of alignment types) are robust to including or excluding these runs.","section":"Table 3 and Section 4.1"},{"comment":"The figure caption reports 'MSE of 10.9 on average' while the main text reports the same numeric value as Mean Absolute Error (MAE); both cannot be correct for the same quantity. Please correct the metric name and value.","section":"Fig. 7 caption and Section 5.2 text"},{"comment":"The paragraph after Table 5 ends with an incomplete sentence beginning 'Additionally,'; this appears to be a leftover fragment and should be removed or completed.","section":"Appendix 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical contribution, but the headline predictive and factor-ordering claims currently outrun the evidence. The main risk is the Section 5.2 ambiguity: if the KL term is ΔGT, the 'label-free' estimator leaks target labels, and if it is ΔL, the claimed accuracy is unsupported. I would ask for a revision that (i) specifies and ablates the estimator inputs, (ii) adds a held-out evaluation, and (iii) either corrects the Factor 2 interpretation or softens the abstract. I do not see grounds for rejection if these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful benchmark paper. 336 OOD evaluations, 8 architectures, 7 datasets, code released. The raw finding is solid: OOD CER jumps to 37–54% compared to 2–12% ID, and this holds across architectures and datasets. The synthetic data results are also honest and informative—synthetic data helps a bit even with a mismatched language. I'd want this in the literature as a reference point.\n\nThe problems are in the interpretive layer. The abstract says textual divergence is the most significant factor, followed by visual. The factor loadings in Fig. 6 show Factor 1 (textual) correlating 0.62 with OOD error, which is moderate. But Factor 2 (visual) correlates -0.3 with OOD error. That is not a positive second driver; the paper's own text admits 'more research studying this last factor has to be done.' So the ordering claim overstates the evidence.\n\nThe bigger issue is the Section 5.2 estimator. The text says it uses metrics that don't require target labels 'and KL divergence.' Section 5.1 defines two KL variants: delta_GT against ground-truth target transcriptions, and delta_L against synthetic corpora. If the estimator uses delta_GT, the 70% within-10-CER-point claim is partly circular—the target transcriptions are used to compute both the prediction and the error being predicted. If it uses delta_L, there is no reported ablation showing delta_L alone achieves that accuracy. Either way, the headline predictive claim is not supported as written. Also, the estimator appears to be fit and evaluated on the same 336 cases; no held-out split is described. That makes 'prediction' closer to interpolation.\n\nI don't think any of this sinks the empirical core. The raw OOD gap, the model comparisons, and the synthetic-data results are real contributions. But the paper needs a major revision: specify exactly which KL input the estimator uses, add a delta_L-only result or drop the label-free claim, validate on held-out domain pairs, and calibrate the abstract to what Fig. 6 actually shows.\n\nWho is this for: anyone working on HTR robustness or domain generalization in document analysis. It deserves a serious referee—the benchmark alone justifies the time. My recommendation: send to peer review, with the expectation of substantial revision rather than acceptance as-is.","headline":"A valuable OOD benchmark for HTR with a robust raw gap, but the factor-ordering and label-free estimator claims outrun the evidence.","tokens_in":78,"tokens_out":1900,"would_cite":true,"duration_ms":47280,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Handwritten text recognition fails out of distribution mainly because the target words and characters differ from the source, not because the handwriting looks unfamiliar.","keywords":["handwritten text recognition","domain generalization","out-of-distribution error","textual divergence","visual divergence","factor analysis","synthetic data","character error rate"],"falsifier":"Re-run the Section 5.2 estimator on the same 336 cases with Δ_L, the synthetic language-matched divergence, as the only textual input and compute the mean absolute error; if it exceeds roughly 11 CER points, or fewer than 70% of residuals fall below 10 points, the headline predictive claim fails. As a simpler check, inspect the estimator's feature list: if Δ_GT appears among the inputs, the 70%-below-10-points statistic is not a label-free prediction.","tokens_in":24661,"feed_emoji":"✍️","tokens_out":7502,"duration_ms":61652,"temperature":0.7,"pith_summary":"Handwritten text recognition (HTR) models are usually trained and tested on the same distribution, but real documents arrive from unknown scripts, eras, and languages. This paper runs the first large-scale domain-generalization study for HTR, with 336 out-of-distribution evaluations across eight architectures, seven datasets, and five languages, and asks what actually breaks these models. Its central claim is that the dominant cause of out-of-distribution failure is textual divergence between source and target — the characters and words the model must emit — with visual divergence playing a secondary role. The paper further claims that out-of-distribution error can be estimated in advance from proxies that do not require target labels, with an average error of about 10.9 CER points and roughly 70% of predictions falling within 10 points. If this is right, practitioners can forecast which domains will break their recognizer before deploying it.","feed_headline":"Text divergence, not handwriting style, breaks HTR models","feed_subtitle":"Across 336 tests, word-level mismatch predicts out-of-distribution error far better than visual difference.","key_machinery":"The argument rides on two novel divergence metrics and a factor analysis. Visual divergence Δ(X_S, X_T) is the average reconstruction error (MSE) of a convolutional autoencoder trained on source images when it is fed target images, a reconstruction-based anomaly-detection score. Textual divergence Δ(Y_S, Y_T) is the averaged KL divergence, over n = 1 to 5, between the character n-gram distributions of source and target text; the paper distinguishes Δ_L, the divergence between the source text and synthetic text of a known language, from Δ_GT, the divergence between the source text and the ground-truth target transcriptions. These metrics feed a factor analysis (eigenvalues ≥ 1, oblimax rotation) producing four latent factors, and the resulting factor structure, together with model capacity and calibration metrics, forms the input to the out-of-distribution error estimator.","core_discovery":"Trained from scratch on a single source domain and evaluated on every other domain, eight state-of-the-art HTR models collapse out of distribution: average CER rises from roughly 5–10% in distribution to 37–54% out of distribution. A factor analysis over model parameters, in-distribution and out-of-distribution CER and calibration error, visual divergence, and textual divergence extracts four latent factors, and the factor loaded on textual divergence — averaged KL divergence over 1-to-5-gram character distributions between source and target text — correlates most strongly with out-of-distribution error (0.62). The visual-divergence factor, measured as autoencoder reconstruction error, shows only a weak secondary influence, with a negative loading on OOD error (−0.3). The paper concludes that textual divergence is the most significant factor for generalization, followed by visual divergence, and demonstrates that OOD error can be estimated from proxy metrics without target labels, reaching a mean absolute error of 10.9 CER points.","pith_inferences":["The negative loading of the visual factor on OOD error (−0.3) sits awkwardly with the abstract's ordering of the two factors; if it holds, a recognizer fails most where the text differs even when the handwriting looks similar, meaning visual novelty alone is a poor basis for predicting failure.","The estimator's accuracy across 336 cases in five languages suggests the same proxy recipe — in-distribution error, calibration, reconstruction error, and n-gram KL divergence — could transfer to other sequence-recognition tasks such as automatic speech recognition or OCR, where target transcripts are also unavailable at deployment.","A direct testable extension is to feed the estimator only Δ_L, the synthetic language-matched divergence, and check whether the 70%-below-10-points accuracy survives; if it does, the predictor is genuinely label-free, and if not, the headline predictive claim needs an explicit label-free baseline.","The finding that autoregressive models are more biased by source language implies that hybrid CTC models, or decoders whose language model can be swapped at test time, are the more promising route for cross-lingual HTR."],"forward_implications":["No current HTR architecture generalizes acceptably: the best out-of-distribution CER is 37.4%, and the average in-distribution-to-out-of-distribution gap is 37.6 CER points, so benchmark gains under the i.i.d. assumption do not transfer to unseen domains.","CTC-aligned models are the safer choice for generalization (average OOD CER 40.4 vs. 48.7 for autoregressive decoders), and the internal language model of autoregressive models biases them more strongly toward the source language.","Synthetic training data improves out-of-distribution performance even when rendered in a different language — about 6 CER points on average, and over 11 when the synthetic language matches the target — making synthetic data a cheap lever for generalization.","Model selection strategy and model capacity have negligible impact on out-of-distribution performance, so tuning those will not close the generalization gap in HTR.","Because textual divergence dominates, out-of-distribution error can be estimated in advance from proxies, with roughly 70% of estimates within 10 CER points, allowing risky target domains to be flagged before deployment."],"supporting_citations":[{"why":"Supplies the domain-generalization benchmark setup and the three model-selection strategies (no-selection, heldout, oracle) that the paper adapts.","marker":"[33]"},{"why":"The accuracy-on-the-line result that motivates estimating out-of-distribution error from in-distribution proxies.","marker":"[55]"},{"why":"Agreement-on-the-line method for predicting model performance under distribution shift, the template for the paper's OOD error estimator.","marker":"[8]"},{"why":"Supplies the averaged KL-divergence-over-n-grams recipe used to quantify textual divergence.","marker":"[30]"},{"why":"Reconstruction-based autoencoder anomaly detection that underlies the visual-divergence metric.","marker":"[14]"},{"why":"Provides the factor-analysis method used to extract the latent factors from the collected metrics.","marker":"[74]"},{"why":"Supplies the IAM dataset, which the results identify as the best-performing source domain in nearly 60% of out-of-distribution cases.","marker":"[52]"}],"fun_headline_variants":["HTR models fail when text differs, not style","Text divergence predicts HTR failure, not visuals","Why HTR models break: text, not handwriting","OOD error in HTR: text divergence is key","Handwriting style? No, text mismatch breaks HTR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that out-of-distribution error can be forecast to roughly 11 CER points depends on which textual-divergence measure feeds the estimator: if it is the ground-truth divergence Δ_GT, the forecast smuggles in the very target transcriptions that domain generalization forbids access to, and the paper never shows that the label-free divergence Δ_L alone achieves the claimed accuracy.","fun_headline_variants_meta":{"raw":{"variants":["HTR models fail when text differs, not style","Text divergence predicts HTR failure, not visuals","Why HTR models break: text, not handwriting","OOD error in HTR: text divergence is key","Handwriting style? No, text mismatch breaks HTR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1367,"prompt_tokens":952,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":568,"tokens_out":415,"duration_ms":3731,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:16:46.224260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Section 5.2 estimator on the same 336 cases with Δ_L, the synthetic language-matched divergence, as the only textual input and compute the mean absolute error; if it exceeds roughly 11 CER points, or fewer than 70% of residuals fall below 10 points, the headline predictive claim fails. As a simpler check, inspect the estimator's feature list: if Δ_GT appears among the inputs, the 70%-below-10-points statistic is not a label-free prediction.","supporting_citations":[{"cited_title":"In search of lost do- main generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the domain-generalization benchmark setup and the three model-selection strategies (no-selection, heldout, oracle) that the paper adapts."},{"cited_title":"Carmon, and Ludwig Schmidt","cited_arxiv_id":null,"evidence_quote":"The accuracy-on-the-line result that motivates estimating out-of-distribution error from in-distribution proxies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the averaged KL-divergence-over-n-grams recipe used to quantify textual divergence."},{"cited_title":"Spearman","cited_arxiv_id":null,"evidence_quote":"Provides the factor-analysis method used to extract the latent factors from the collected metrics."},{"cited_title":"The iam-database: an en- glish sentence database for offline handwriting recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the IAM dataset, which the results identify as the best-performing source domain in nearly 60% of out-of-distribution cases."}],"review_version":1}