{"id":"268007cd-d87f-4d24-ac01-8e8d296a3b2b","arxiv_id":"2412.14596","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A language-decoupled pre-training method, using a text-editing diffusion model to remove readable text from document images, improves cross-lingual visual information extraction from English-only data.","lead":"The paper shows that removing readable text from document images during pre-training, while keeping layout and visual cues, lets a model trained on English-only data generalize to seven other languages in form-understanding tasks. It introduces a diffusion-based step to scramble text into a fake language and a SAM-based model that is then fine-tuned with multilingual sentence embeddings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-lingual gains are not cleanly attributed to language decoupling: the frozen multilingual Sentence-BERT in LKI and the omitted ESP zero-shot baseline are confounds not addressed by the current ablations.","rationale":"Read in good faith, the paper identifies a real problem and supplies suggestive evidence: Figure 2 shows the decoupling operator drives OCR and language-classification surrogates toward chance, and Table 7 shows a monotonic trend in cross-lingual F1 as decoupling resolution decreases. Those are genuine checks. The weakness is not the existence of an effect but its attribution. LKI injects a frozen multilingual Sentence-BERT at the classifier, so the model is not purely vision-layout-based in the strong sense claimed; language transfer can flow through the text branch. The omission of ESP from the zero-shot setting is particularly damaging because ESP is the closest prior control: same data, same pseudo-labeling, no decoupling, no LKI, and it nearly matches LDM in per-language fine-tuning. The paper needs an interaction ablation to separate decoupling from LKI and a zero-shot ESP row to bound the baseline. I would keep the reader's CONDITIONAL verdict: the direction is plausible, but the central causal claim is not yet cleanly supported.","tokens_in":13002,"tokens_out":6779,"duration_ms":50235,"concrete_test":"Run a single 2x2 ablation on XFUND zero-shot: pre-train LDM with original images versus AnyText-decoupled images, and fine-tune on FUNSD with LKI either enabled or disabled, using the same number of pre-training epochs and multiple seeds. If decoupled pre-training improves XFUND by at least 3 F1 in both LKI conditions, the decoupling effect is established; if the gain appears only with LKI enabled, the multilingual text encoder is the source. As a supporting check, add ESP to the zero-shot comparison under identical fine-tuning to bound the non-decoupled baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that decoupling language bias from document images is what enables cross-lingual VIE. The decisive evidence would be an ablation that isolates decoupling while holding all other mechanisms fixed. That ablation is missing. The final LDM uses LKI, a frozen multilingual Sentence-BERT fused at the classifier; in zero-shot XFUND evaluation, non-English OCR text is embedded by this multilingual model, so cross-lingual transfer can enter through the text branch even if the decoupled images contribute nothing. Table 8 shows LKI itself is worth about 1.5 F1 on XFUND, and Table 7, the only decoupling-versus-original comparison, is run for one pre-training epoch with no error bars. Moreover, the directly comparable ESP baseline, which uses the same DocBank and RVL-CDIP pre-training without decoupling and without LKI, is absent from the zero-shot Table 1; in per-language fine-tuning (Table 2) ESP reaches 86.76 non-English average versus LDM's 88.21, a small margin that could be within noise. Without a 2x2 ablation crossing decoupled versus original pre-training with LKI on and off, the headline result could be driven by the multilingual text encoder or by the architecture rather than by language decoupling per se.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LDP (Language Decoupled Pre-training), a paradigm that pre-trains a vision-layout document model on English-only images whose readable text has been scrambled by the AnyText diffusion editor, with the goal of removing language bias while retaining layout and visual appearance. The resulting model, LDM, is built on SAM with a Multi-Token Information Merging (MTIM) module and, at fine-tuning time, a Language Knowledge Inserting (LKI) module that fuses frozen multilingual Sentence-BERT embeddings. The authors report state-of-the-art cross-lingual results on XFUND under zero-shot, per-language, and multitask fine-tuning, plus strong results on SIBR, FUNSD, and CORD. The central claim is that decoupling language from document images enables monolingual pre-training to transfer to unseen languages.","tokens_in":13274,"tokens_out":5157,"duration_ms":38626,"significance":"If the central claim holds, the paper offers a practical and conceptually attractive way to build multilingual VIE models without multilingual pre-training corpora: a monolingual English corpus is sufficient once language bias is removed from images. The paper deserves credit for formulating a clear hypothesis about visual/layout invariance, for proposing concrete surrogate metrics (OCR recognition ratio and language classification accuracy) to measure decoupling, and for grounding the approach in extensive experiments across multiple benchmarks. However, the evidence as presented does not cleanly isolate the effect of language decoupling from other ingredients of the final model, especially the frozen multilingual text encoder inserted at fine-tuning and the architectural differences with the closest baseline. The paper's significance therefore rests on a plausible but not yet established attribution.","major_comments":[{"comment":"The final LDM fuses frozen multilingual Sentence-BERT embeddings into the classifier through LKI, so cross-lingual transfer can enter through the text branch even if the decoupled images contribute nothing. Table 8 shows that LKI itself improves XFUND from 61.36 to 62.90 (+1.54) in the zero-shot setting, which is a substantial fraction of the reported margins over LiLT and LayoutXLM in Table 1. To support the paper's central claim that language decoupling is what enables cross-lingual generalization, the authors should provide a 2x2 ablation crossing decoupled vs. original pre-training with LKI on vs. off, ideally with multiple random seeds. The current ablations do not separate these factors.","section":"Language Knowledge Inserting (LKI), Eq. (2), and Table 8"},{"comment":"Table 7 is the only direct comparison of pre-training on decoupled vs. original images, and it is reported for a single pre-training epoch without error bars or significance tests. The claimed benefit at decouple_resolution=1024 is 60.68 vs. 57.65 XFUND F1, a 3-point gap that could be within run-to-run noise at one epoch. Moreover, the hyperparameter decouple_resolution is selected based on Table 7's XFUND results, meaning the test benchmark is used for model selection; this should be stated explicitly. The authors should report multi-seed mean and variance, and ideally verify the trend with a longer pre-training budget or on a validation split.","section":"Ablation Study, Table 7"},{"comment":"The surrogate metrics support that AnyText removes readable text, but they do not directly demonstrate that layout and visual features are preserved at the chosen decouple_resolution=1024. The text reports language classification accuracy at 768 (21.67%) and 512 (20.33%) but does not report the value at 1024, which is the setting used in all main experiments. If language bias remains substantial at 1024, the mechanism underlying the claimed improvement is unclear. The authors should report the language classification accuracy at 1024 and, ideally, add a layout-preservation metric (e.g., layout detection consistency before/after decoupling) to support the claim that the decoupled images retain the cues on which VIE relies.","section":"Quantitative Evaluation, Figure 2"},{"comment":"ESP is the most direct control because it uses the same pre-training corpora (DocBank and RVL-CDIP), the same pseudo-label generation, and no language decoupling. Yet ESP is absent from the zero-shot Table 1, where its inclusion is necessary to interpret LDM's cross-lingual gains. In the per-language fine-tuning setting (Table 2), LDM beats ESP on the non-English average by only 1.45 points (88.21 vs. 86.76), and the two models differ in architecture, so this margin cannot be attributed to decoupling alone. The paper should include ESP in Table 1 and, ideally, a decoupled variant of ESP or an LDM variant pre-trained on original images without LKI to isolate the effect.","section":"Comparison with ESP, Tables 1 and 2"}],"minor_comments":[{"comment":"The text states that 'all other parameters are randomly initialized' but also that SAM's pre-trained parameters are inherited. Please clarify exactly which modules inherit SAM weights (image encoder, prompt encoder, decoder) and which modules are randomly initialized (MTIM, prediction head).","section":"Implementation Details"},{"comment":"The figure is referenced but not fully described in the text; please ensure axis labels, legends, and the values at decouple_resolution=1024 are visible and stated, since the main experiments use 1024.","section":"Figure 2"},{"comment":"In Eq. (1), the index k in F^SAM_nk is not precisely defined beyond 'k ∈ [0, K]'; please clarify whether K is the number of prompt tokens from the SAM decoder and whether the concatenation order of k matters.","section":"Notation, Eq. (1)"},{"comment":"The captions use 'F1 accuracy'; F1 is not an accuracy measure, so 'F1 score' or 'F1' would be more precise.","section":"Tables 2 and 3 captions"},{"comment":"The claim of being 'the first to systematically study visual invariance in multilingual VIE' should be qualified by a discussion of prior vision-based multilingual work, or supported by a focused literature comparison, to avoid overclaiming.","section":"Introduction"},{"comment":"The term 'decouple resolution' is used interchangeably with 'decouple_resolution'; please unify the terminology in the text and figures.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, and the surrogate metrics in Figure 2 are a good start. The main weakness is attribution: the final model includes a frozen multilingual Sentence-BERT through LKI, and the most direct control (ESP) is missing from the zero-shot table and is only 1.45 points behind in per-language fine-tuning despite architectural differences. I would strongly encourage the editor to request the 2x2 ablation (decoupling × LKI) and the ESP zero-shot result before acceptance; without these, the headline claim that language decoupling drives the cross-lingual gains is not demonstrated. The use of XFUND for choosing decouple_resolution should also be disclosed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe one thing to know: this paper has a genuinely fresh idea—decouple language from document images via a text-editing diffusion model (AnyText) so pre-training on English transfers to other languages—and the authors actually measure the decoupling with two sensible surrogates (OCR edit distance and language classification accuracy). That part is well executed, and the central ablation in Table 7 shows decoupled images improve zero-shot XFUND by roughly 3 F1 while costing little on FUNSD. This is a real signal.\n\nThe soft spot is attribution. The final model fuses a frozen multilingual Sentence-BERT into the classifier (LKI), so cross-lingual information can enter through the text branch even if the visual branch learned nothing language-independent. Table 8 shows LKI alone contributes about 1.5 F1 on XFUND. The paper never presents the clean 2x2 ablation that would isolate decoupling from LKI. Additionally, ESP—the most comparable vision-only baseline, same pre-training data—is missing from the zero-shot table; in per-language fine-tuning it trails LDM by only 1.45 F1, a margin that could be noise without error bars. The authors also tune decouple_resolution on the same downstream benchmarks, which is a minor circularity. And they admit AnyText distorts small text in dense documents, which the surrogate metrics confirm (OCR ratio drops to near zero at low resolutions); at the chosen 1024 resolution, some residual text likely remains, so the model may still see partial language cues.\n\nNone of this makes the paper weak. The idea is new, the surrogate metrics are thoughtful, and the direction is credible. But the headline claim needs a cleaner isolation. I'd send it to peer review with a request for the 2x2 ablation, ESP in Table 1, and variance estimates.\n\nWho is this for? Document-AI researchers working on multilingual transfer. A general CV reader will find it narrower than the abstract suggests. I'd cite the decoupling idea if I worked in that space, but I wouldn't treat the current numbers as definitive.","headline":"Fresh idea, plausible direction, but the cross-lingual gain isn't cleanly isolated from the multilingual text encoder.","tokens_in":13762,"tokens_out":3272,"would_cite":true,"duration_ms":22905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training on language-scrambled English document images lets a vision-layout model generalize to non-English form understanding, beating multilingual pre-trained models on XFUND and staying competitive on English benchmarks.","keywords":["visual information extraction","multilingual transfer","language decoupling","document pre-training","vision-layout models","form understanding","diffusion text editing","XFUND"],"falsifier":"Measure the OCR word accuracy of the decoupled images at the chosen resolution: if readable words remain at levels far above the reported ~3.5%, language bias is not removed and the cross-lingual result could come from residual English text. The sharper test is to pre-train the same model on the original, non-decoupled English images and zero-shot evaluate on XFUND; if that version matches or exceeds the 62.90 average F1, the decoupling operation is not what produces the generalization.","tokens_in":12809,"feed_emoji":"🌐","tokens_out":8499,"duration_ms":63187,"temperature":0.7,"pith_summary":"This paper tries to establish that the visual appearance and spatial layout of a document are language-invariant, so a model does not need readable text in every language to learn how to extract information from forms. It proposes Language Decoupled Pre-training (LDP): scramble the text in English document images with a text-editing diffusion model, pre-train a vision-layout model on those language-independent images, then fine-tune on downstream tasks with the original images and injected text embeddings. If the claim is right, multilingual visual information extraction no longer depends on large, balanced multi-language corpora. The reported numbers support the claim: fine-tuned on English FUNSD, the proposed LDM reaches 62.90 average F1 on seven non-English XFUND languages, compared with 57.24 for the best prior layout-only baseline, and reaches 88.21 average F1 when fine-tuned per language.","feed_headline":"Scrambled-text pretraining lifts cross-lingual form F1 to 62.9","feed_subtitle":"Pretraining on garbled English text lets a vision-layout model beat multilingual baselines on seven languages.","key_machinery":"The load-bearing mechanism is the language-decoupling operation: a text-editing diffusion model is asked to edit a 1x1 pixel corner with an empty prompt, so its learned editing behavior scrambles the small, dense text across the whole document image while leaving fonts, colors, backgrounds, ruling, and layout geometry intact. A 'decouple resolution' hyperparameter controls how much detail survives. The model that consumes these images, LDM, is built on a pre-trained image-segmentation backbone: an image encoder plus prompt encoder fed with bounding boxes, with the segmentation head replaced by an MLP classification head. Its Multi-Token Information Merging (MTIM) module exchanges features among the bounding boxes of one image so neighboring boxes can influence each other's labels, and in fine-tuning the Language Knowledge Inserting (LKI) module adds a frozen multilingual sentence embedding of each OCR text to the final features. Pre-training on the decoupled images, with pseudo-labels generated from the same English corpora, is what teaches the cross-lingual invariance.","core_discovery":"The central discovery is that vision and layout alone—bold fonts, gray background blocks, relative positions, ruling lines—carry enough structure to identify semantic roles such as question, answer, and header across languages, while the identity of the glyphs themselves is a bias that hurts transfer. When the text is scrambled so thoroughly that OCR reads almost none of it (text recognition ratio near 3.5%) and a classifier cannot tell which language an image came from (21.67% vs 16.66% random), a vision-layout model becomes better at zero-shot cross-lingual extraction, not worse. Pre-trained on English-only corpora and fine-tuned on English FUNSD, the model outperforms multilingual models on non-English XFUND; fine-tuned per language or on all languages jointly, it also sets the best multilingual numbers while remaining competitive on English FUNSD and CORD.","pith_inferences":["The paper does not test this, but a natural extension is to apply the same decoupling to other visually rich document tasks such as table structure recognition or document question answering; the invariance claim would predict similar cross-lingual gains there.","The paper's surrogate metrics suggest a testable design rule: pick the decouple resolution by watching when a language classifier approaches chance while a perceptual similarity metric between original and decoupled images stays high, rather than fixing 1024 across corpora.","Because fine-tuning still needs OCR text through the frozen sentence encoder, the approach inherits OCR errors in low-resource languages; an OCR-free variant that conditions on visual text features only would be a stronger test of the invariance claim.","If the invariance generalizes beyond these seven languages, the method could be applied to under-resourced languages with no labeled forms at all, using English pre-training plus zero-shot transfer."],"forward_implications":["Multilingual VIE can be built from English-only pre-training data, removing the need for large balanced corpora in every target language.","Scrambling text at a controlled resolution is a tunable data augmentation: cross-lingual F1 rises as more language bias is removed, until the resolution gets low enough that useful visual detail is destroyed too.","The model retains strong English performance (88.23 F1 on FUNSD, 95.80 on CORD), so the decoupling does not sacrifice the language it was pre-trained on.","In per-language and multitask settings the model also beats prior multilingual pre-trained models, with the largest margins on non-English languages.","Injecting language knowledge at fine-tuning (LKI) is complementary to the language-independent pre-training: adding frozen multilingual text embeddings improves downstream accuracy."],"supporting_citations":[{"why":"supplies the AnyText text-editing diffusion model that the paper uses to scramble text while preserving layout and visual features.","marker":"Tuo et al. 2024"},{"why":"provides the pseudo-label generation algorithm and the SIBR benchmark, and is the closest vision-based baseline the paper builds from.","marker":"Yang et al. 2023"},{"why":"LiLT is the prior layout-only decoupling approach that LDM is compared against and that motivates adding vision to layout.","marker":"Wang, Jin, and Ding 2022"},{"why":"XFUND is the multilingual benchmark whose seven non-English subsets measure cross-lingual generalization.","marker":"Xu et al. 2022"},{"why":"FUNSD is the English fine-tuning set used for zero-shot cross-lingual evaluation.","marker":"Jaume, Ekenel, and Thiran 2019"},{"why":"SAM supplies the image encoder and prompt-encoder backbone that LDM inherits before replacing the mask head.","marker":"Kirillov et al. 2023"},{"why":"provides the frozen multilingual sentence encoder used to inject language knowledge in the LKI module during fine-tuning.","marker":"Reimers and Gurevych 2019, 2020"},{"why":"DocBank is one of the two English-only pre-training corpora from which the language-independent images are made.","marker":"Li et al. 2020"},{"why":"RVL-CDIP supplies the second English pre-training corpus used in the LDP pipeline.","marker":"Harley, Ufkes, and Derpanis 2015"}],"fun_headline_variants":["Scrambled text unlocks cross-lingual form parsing","Layout-only model with scrambled text beats multilingual baselines","Language-free pretraining: English only, works in 7 languages","Scrambled English text yields 62.9 F1 on multilingual forms","Vision-layout pretraining generalizes across languages without text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the diffusion text-edit operation destroys readable text in dense document images while preserving exactly the visual and layout cues a VIE model needs; the paper's own evidence is surrogate OCR and language-classification metrics plus one-epoch ablations, not a mechanism that guarantees preservation.","fun_headline_variants_meta":{"raw":{"variants":["Scrambled text unlocks cross-lingual form parsing","Layout-only model with scrambled text beats multilingual baselines","Language-free pretraining: English only, works in 7 languages","Scrambled English text yields 62.9 F1 on multilingual forms","Vision-layout pretraining generalizes across languages without text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000955,"raw_usage":{"total_tokens":4063,"prompt_tokens":931,"completion_tokens":3132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":3046}},"tokens_in":547,"tokens_out":3132,"duration_ms":19544,"temperature":1.0,"reasoning_tokens":3046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:04:57.652647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the OCR word accuracy of the decoupled images at the chosen resolution: if readable words remain at levels far above the reported ~3.5%, language bias is not removed and the cross-lingual result could come from residual English text. The sharper test is to pre-train the same model on the original, non-decoupled English images and zero-shot evaluate on XFUND; if that version matches or exceeds the 62.90 average F1, the decoupling operation is not what produces the generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the AnyText text-editing diffusion model that the paper uses to scramble text while preserving layout and visual features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the pseudo-label generation algorithm and the SIBR benchmark, and is the closest vision-based baseline the paper builds from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LiLT is the prior layout-only decoupling approach that LDM is compared against and that motivates adding vision to layout."},{"cited_title":"K.; and Thiran, J.-P","cited_arxiv_id":null,"evidence_quote":"FUNSD is the English fine-tuning set used for zero-shot cross-lingual evaluation."},{"cited_title":"C.; Lo, W.-Y.; et al","cited_arxiv_id":null,"evidence_quote":"SAM supplies the image encoder and prompt-encoder backbone that LDM inherits before replacing the mask head."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the frozen multilingual sentence encoder used to inject language knowledge in the LKI module during fine-tuning."},{"cited_title":"W.; Ufkes, A.; and Derpanis, K","cited_arxiv_id":null,"evidence_quote":"RVL-CDIP supplies the second English pre-training corpus used in the LDP pipeline."}],"review_version":1}