{"id":"178e13c9-87d0-4135-aecd-88bc67f95c0a","arxiv_id":"2412.09341","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Training LayoutLM from scratch on DOCILE invoices beats the official IIT-CDIP checkpoint on a new anonymized payslip NER dataset, and a 6-layer variant halves inference time.","lead":"Pre-training LayoutLM from scratch on 900,000 invoices instead of the generic IIT-CDIP corpus improves named-entity recognition on a new set of 611 insurance payslips, lifting F1 from 62.3 to 64.7. A 6-layer version runs about twice as fast while staying within one F1 point of the standard model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The missing size-matched generic-corpus control means the paper's causal claim that DOCILE's domain similarity drives the F1 gain on PAYSLIPS is not yet established.","rationale":"The paper's title, abstract, and Section 4.2 all claim that using domain-relevant documents improves NER, and that causal claim is the reason the recipe would transfer to other insurance subdomains. The comparison as run cannot distinguish corpus domain from other confounds because the two compared models differ in many ways. The absence of a size-matched generic pre-training control is the single point on which the paper's central inference rests. A fair control is feasible: IIT-CDIP and RVL-CDIP are public, and the authors already have the code and compute used for DOCILE. The reader's weakest_assumption identified the same confound, including the tokenizer question, which is part of the same issue. The Appendix A significance test is flawed and should be redone, but the mean-over-seeds gap is likely robust; the larger problem is attribution rather than chance. Since the concern is addressable and the empirical resources (dataset, code) have value, the conditional verdict remains appropriate and no change to the reader's verdict is needed.","tokens_in":8791,"tokens_out":7128,"duration_ms":77128,"concrete_test":"Pretrain LayoutLM-base from scratch on a ~900k-page random sample of IIT-CDIP (or RVL-CDIP) using the exact DOCILE recipe (same tokenizer/vocabulary, batch 80, 5 epochs, lr 5e-5, cosine warmup, no image embeddings), then fine-tune on PAYSLIPS with the same 100 seeds and hyperparameters. If this generic control reaches ~64.7 F1, domain similarity is not the active ingredient; if it remains near ~62.3, the domain-corpus explanation is supported. As a precondition, verify from the released checkpoints that the DOCILE-pretrained model uses the same tokenizer/vocabulary as the official LayoutLM; if it does not, the comparison is confounded regardless of the control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 2 (64.74 vs 62.31 F1) contrasts a model pre-trained from scratch on DOCILE (~900k invoice pages, 5 epochs, lr 5e-5, batch 80) with the released LayoutLM-base checkpoint pre-trained on IIT-CDIP (~11M documents, unknown protocol). These conditions differ simultaneously in corpus domain, corpus size, OCR noise, pre-training length, and possibly tokenizer/vocabulary: the paper never states whether the DOCILE-pretrained model reuses the official LayoutLM tokenizer. The paper attributes the 2.43-point gain to the 'semantically and structurally similar' nature of DOCILE, supported only by visual inspection (Figure 2). No control pre-trains with the same recipe on a size-matched generic corpus, so the result is equally compatible with cleaner OCR, shorter pre-training causing less catastrophic forgetting, or less domain diversity. The Appendix A significance test does not repair this: it selects one pair of fine-tuned models after the fact and pools predictions in a non-item-level shuffle, so it does not test the pre-training-corpus effect. Without the control, the paper's transferable recipe ('use domain-relevant pre-training data') is not isolated from these confounds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PAYSLIPS, a novel NER dataset of 611 anonymized insurance payslip pages with nine entity types. The authors pre-train LayoutLM from scratch on DOCILE, an invoice corpus, and compare it with the official LayoutLM pre-trained on IIT-CDIP. They report that the DOCILE-pretrained model achieves higher F1 on PAYSLIPS (64.74 vs 62.31) with lower variance, and that a 6-layer variant retains competitive performance at roughly half the inference time. The paper argues that pre-training on domain-relevant documents improves downstream NER and proposes a practical recipe for building efficient in-house models.","tokens_in":8958,"tokens_out":6845,"duration_ms":60681,"significance":"If the causal claim were validated, the result would be practically significant: it would show that a tenfold-smaller, domain-specific corpus can outperform a massive generic corpus for specialized document NER, and that smaller encoders can be fast enough for production. The paper also ships a new public dataset and code, and uses 100 fine-tuning runs per condition, which is commendable. The central comparison, however, is confounded by differences in corpus size, pre-training length, and possibly tokenizer, and the significance test used to support the headline difference is statistically invalid. The dataset release is a contribution, but the empirical support for the main claim needs strengthening.","major_comments":[{"comment":"The headline comparison contrasts a from-scratch pre-training on DOCILE (~900k pages, 5 epochs, batch 80, lr 5e-5) with the released LayoutLM-base checkpoint pre-trained on IIT-CDIP (11M documents, unknown protocol). These conditions differ in corpus domain, corpus size, pre-training length, and possibly tokenizer/vocabulary — the paper never states whether the from-scratch model reuses the original LayoutLM tokenizer. The 2.43-point F1 gain on PAYSLIPS is therefore not attributable specifically to 'semantically and structurally similar' pre-training data, especially since that similarity is supported only by visual inspection in Figure 2. A control pre-training with the same recipe on a size-matched generic corpus (e.g., a random subset of IIT-CDIP or RVL-CDIP invoices) is needed to isolate the effect of domain relevance. Without such a control, the paper's central claim and the transferable recipe 'use domain-relevant pre-training data' are not established.","section":"§4.2, Table 2"},{"comment":"The approximate randomization test is not a valid test of the difference between the two systems. Algorithm 1 concatenates the N predictions of the two systems, shuffles them, and splits them into two halves, thereby destroying the pairing of predictions on the same test items; this tests whether random partitions of the pooled prediction set can mimic the observed F1 gap, not whether the two systems differ on individual items. A valid test would randomly swap, for each test item, the prediction of the two systems (or shuffle system labels per item) and recompute the F1 difference. Additionally, Section 5.2 states that 'we took two models with a F1-score difference below 1.00 for the test'; this post-hoc selection from 100 runs makes the reported significance value of 0.0019 unreliable. The significance of the headline result is therefore unsupported as reported.","section":"§5.2, Appendix A, Algorithm 1"},{"comment":"The PAYSLIPS evaluation is performed on the original, non-public version of the dataset; the released version has manually altered amounts and other sensitive information. The paper states that 'the results presented in this paper are based on this version' and does not report results on the released, altered version. Since the public data differ from the evaluated data, the exact numbers in Table 2 cannot be reproduced from the released artifacts. The paper should either report results on the released version or provide a clear statement of the expected change, together with an evaluation on the altered data.","section":"Appendix C"}],"minor_comments":[{"comment":"State explicitly whether the from-scratch model reuses the original LayoutLM tokenizer; if a new tokenizer is trained, report its vocabulary size and training data, as this is a potential confound.","section":"§4.2"},{"comment":"Report all pre-training hyperparameters (max sequence length, masking probability, weight decay, warmup steps, etc.) beyond batch size, learning rate, and epochs, to allow exact reproduction.","section":"§4.2"},{"comment":"Clarify the sentence 'we took two models with a F1-score difference below 1.00 for the test'; the selection criterion is ambiguous and should be described precisely.","section":"§5.2"},{"comment":"Report standard deviations for precision and recall as well, since F1 is presented with ± values; the current table gives only point estimates for these metrics.","section":"Table 4"},{"comment":"The visual comparison in Figure 2 is suggestive but qualitative; consider adding a quantitative measure of layout or semantic similarity (e.g., distribution of spatial box positions or field-name overlap) to support the claim that DOCILE and PAYSLIPS are closer than IIT-CDIP and PAYSLIPS.","section":"Figure 2"},{"comment":"Clarify the relationship between the evaluated version and the released version, and state whether the released version was validated with the NER pipeline; if not, discuss the expected impact of the modifications.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation on a non-public version of the released dataset is a reproducibility concern that the editor may want to weigh. The central comparison needs a size-matched generic-corpus control to support the causal claim, and the statistical test in Appendix A should be replaced with a proper paired randomization test. The strengths are the new dataset, the 100-run fine-tuning experiments, and the clear presentation of efficiency benefits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you: this is a small but honest empirical paper. The new thing is PAYSLIPS, 611 annotated payslip pages, plus a comparison showing that a LayoutLM pre-trained from scratch on DOCILE invoices beats the official IIT-CDIP checkpoint by 2.43 F1 on that dataset (64.74 vs 62.31, 100 fine-tuning runs). The dataset is the main contribution; the result is useful evidence for practitioners who want to pre-train in-house models under privacy constraints.\n\nWhat the paper does well: it releases the dataset and code, runs 100 fine-tuning seeds, reports variance, and is upfront in the Limitations section that F1 is below 65 and the dataset is small and specialized. The writing is clear, and the comparison on DOCILE labeled (no improvement) is a useful sanity check. The 6-layer model at roughly half inference time with a 0.5 F1 deficit is a plausible efficiency trade-off, though \"comparable\" is slightly generous.\n\nThe soft spots are real but addressable. The headline claim — that domain similarity drives the gain — is not isolated. The DOCILE pre-training differs from the IIT-CDIP checkpoint in corpus domain, corpus size (900k pages vs 11M documents), pre-training length, and possibly tokenizer/vocabulary, and the paper never says whether it reuses the official LayoutLM tokenizer. Without a size-matched generic control, the result is equally compatible with other pre-training differences. The statistical test in Appendix A is also not a valid paired test: Algorithm 1 pools the predictions of both systems and randomly splits them, which tests whether the two prediction sets are exchangeable, not whether the paired per-item difference is significant. The correct approximate randomization would swap predictions between systems on each test item and recompute F1 on the paired data. That said, the observed 2.43-point gap with lower variance is plausibly real; the test just doesn't support the stated p=0.0019. A reviewer should ask for a corrected test and a control pre-training on a generic corpus of matched size.\n\nWho is this for? Anyone building document NER in finance or insurance under privacy constraints, and anyone interested in domain-adaptive pre-training as a practical recipe. It deserves referee time, but with major revisions. I would send it out rather than desk-reject.","headline":"Useful new payslip NER dataset; the DOCILE-pre-training gain is plausible but the causal claim and the significance test both need work.","tokens_in":9576,"tokens_out":2688,"would_cite":true,"duration_ms":28260,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training LayoutLM from scratch on an invoice corpus that structurally resembles insurance payslips improves NER on the new Payslips dataset beyond the official IIT-CDIP-pre-trained model, while a six-layer variant keeps performance at…","keywords":["named entity recognition","LayoutLM","pre-training from scratch","domain adaptation","DOCILE","PAYSLIPS","document understanding","insurance documents"],"falsifier":"Pre-train LayoutLM from scratch with the same protocol on a random subset of IIT-CDIP matched in size to DOCILE; if the F1 on PAYSLIPS is indistinguishable from the DOCILE-pre-trained model's 64.74, the domain-similarity explanation collapses.","tokens_in":8518,"feed_emoji":"🧾","tokens_out":8377,"duration_ms":66755,"temperature":0.7,"pith_summary":"This paper tests whether pre-training a document model from scratch on domain-relevant material can beat a generic off-the-shelf model when the downstream task is named-entity recognition on confidential financial documents. It introduces Payslips, a 611-page annotated dataset of anonymized insurance pay statements, and pre-trains LayoutLM on the public DOCILE invoice corpus instead of the 11-million-document IIT-CDIP corpus used for the official model. On the Payslips NER task the DOCILE-pre-trained model reaches 64.74 F1 against 62.31 for the official LayoutLM, with lower variance across 100 fine-tuning runs and a significance value of 0.0019 by approximate randomization. The paper also finds that a six-layer version keeps roughly the same performance while cutting inference time nearly in half. The implication is that in-house, commercially usable models can be built without relying on private data, as long as a structurally similar public corpus exists.","feed_headline":"Invoice pre-training beats generic docs on payslip NER","feed_subtitle":"From-scratch LayoutLM trained on DOCILE invoices scores 64.7 F1 versus 62.3 for the official model on Payslips.","key_machinery":"The central object is LayoutLM, a BERT-style transformer whose input embeddings combine token embeddings with 2D positional embeddings (a discretized 6-tuple of coordinates, height, and width) so that self-attention can exploit document layout. The paper's intervention is to pre-train this architecture from scratch with a masked-language-modeling loss on DOCILE, an unlabeled invoice corpus, and then fine-tune it on the new PAYSLIPS dataset. The pre-training corpus is the only variable changed: architecture, MLM objective, and fine-tuning protocol are kept fixed while the corpus changes from IIT-CDIP to DOCILE. The paper also uses layer count as a dial on the same machinery, replacing the 12-layer base with 6-, 2-, and 1-layer versions to trade accuracy for speed.","core_discovery":"The central claim is that pre-training LayoutLM from scratch on DOCILE, a corpus of roughly 900k invoices that resemble payslips in layout and semantics, transfers better to insurance-domain NER than the official LayoutLM pre-trained on IIT-CDIP, even though DOCILE is more than ten times smaller. The numbers are 64.74 ± 2.92 F1 versus 62.31 ± 5.13 on PAYSLIPS, with the gain attributed mainly to higher precision (71.47 versus 65.70) rather than recall. The same pre-training choice leaves DOCILE-labeled NER performance essentially unchanged, which the paper reads as evidence that the benefit is not a general pre-training artefact but a domain-match effect. Reducing the model to six layers drops F1 to 61.80, close to the official base model, while halving inference time. The authors state that the result is highly significant under an approximate randomization test with significance value 0.0019.","pith_inferences":["A controlled next step would be pre-training from scratch on a similarly sized random sample of IIT-CDIP; if the PAYSLIPS F1 is indistinguishable from the DOCILE-pre-trained result, corpus similarity is not the active ingredient.","The comparison may be confounded if the from-scratch model used a different tokenizer than the official LayoutLM, since vocabulary differences alone can change extraction behaviour; the paper does not state which tokenizer was used.","The same recipe could transfer to other sparse financial documents such as bank statements or tax forms whenever a structurally similar public corpus exists, but that transfer is not demonstrated in this paper."],"forward_implications":["Domain-relevant public corpora can substitute for private in-house data when pre-training document models for regulated industries.","The F1 gain on PAYSLIPS is mainly a precision gain, meaning the DOCILE-pre-trained model produces fewer spurious extractions on payslip-like documents.","A six-layer LayoutLM pre-trained on DOCILE matches the official 12-layer model on PAYSLIPS while cutting per-page inference time roughly in half, making it a candidate for high-throughput document processing.","The released PAYSLIPS dataset gives document-understanding research a benchmark where the dominant label is O, so sparse-extraction behaviour becomes directly measurable."],"supporting_citations":[{"why":"Defines the LayoutLM architecture and supplies the official IIT-CDIP-pre-trained baseline model that the experiments compare against.","marker":"Xu et al., 2020"},{"why":"Provides the DOCILE invoice corpus used for the from-scratch pre-training that the paper claims improves PAYSLIPS NER.","marker":"Šimsa et al., 2023"},{"why":"Documents the IIT-CDIP collection, the generic pre-training corpus behind the baseline model.","marker":"Schmidt et al., 2002"},{"why":"Describes the IIT-CDIP test collection and its construction, grounding the baseline corpus used by the official model.","marker":"Lewis et al., 2006"},{"why":"Establishes the MUC significance-testing practice that the paper follows for comparing F1 differences.","marker":"Chinchor, 1992"},{"why":"Provides the evaluation analysis and the approximate randomization procedure used to report the significance value of 0.0019.","marker":"Chinchor et al., 1993"},{"why":"Supplies the approximate randomization testing method that the paper applies to the fine-tuning runs.","marker":"Noreen, 1989"}],"fun_headline_variants":["From-scratch LayoutLM on invoices outperforms generic pretraining","Domain-matched invoices beat bigger generic pre-training for NER","Smaller from-scratch LayoutLM with invoices matches official base F1","Precision gain from invoice pretraining drives payslip NER win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that DOCILE invoices are similar enough to insurance payslips that the measured improvement on PAYSLIPS comes from that domain match, and not from other differences between the two pre-training setups, such as corpus size, how words are split, or training details.","fun_headline_variants_meta":{"raw":{"variants":["From-scratch LayoutLM on invoices outperforms generic pretraining","Domain-matched invoices beat bigger generic pre-training for NER","Smaller from-scratch LayoutLM with invoices matches official base F1","Precision gain from invoice pretraining drives payslip NER win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1708,"prompt_tokens":838,"completion_tokens":870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":795}},"tokens_in":454,"tokens_out":870,"duration_ms":6607,"temperature":1.0,"reasoning_tokens":795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:31.537401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train LayoutLM from scratch with the same protocol on a random subset of IIT-CDIP matched in size to DOCILE; if the F1 on PAYSLIPS is indistinguishable from the DOCILE-pre-trained model's 64.74, the domain-similarity explanation collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the IIT-CDIP collection, the generic pre-training corpus behind the baseline model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the MUC significance-testing practice that the paper follows for comparing F1 differences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the evaluation analysis and the approximate randomization procedure used to report the significance value of 0.0019."}],"review_version":1}