Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-training LayoutLM from scratch on an invoice corpus that structurally resembles insurance payslips improves NER on the new Payslips dataset beyond the official IIT-CDIP-pre-trained model, while a six-layer variant keeps performance at…

desk verdict Useful new payslip NER dataset; the DOCILE-pre-training gain is plausible but the causal claim and the significance test both need work. read the letter →

arxiv 2412.09341 v1 pith:2HZDKP4X submitted 2024-12-12 cs.CL

Benno Uthayasooriyar , Antoine Ly , Franck Vermet , Caio Corro This is my paper · ORCID
classification cs.CL
keywords namedentityrecognitionLayoutLMpre-trainingfromscratchdomainadaptationDOCILEPAYSLIPSdocumentunderstandinginsurancedocuments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether pre-training a document model from scratch on domain-relevant material can beat a generic off-the-shelf model when the downstream task is named-entity recognition on confidential financial documents. It introduces Payslips, a 611-page annotated dataset of anonymized insurance pay statements, and pre-trains LayoutLM on the public DOCILE invoice corpus instead of the 11-million-document IIT-CDIP corpus used for the official model. On the Payslips NER task the DOCILE-pre-trained model reaches 64.74 F1 against 62.31 for the official LayoutLM, with lower variance across 100 fine-tuning runs and a significance value of 0.0019 by approximate randomization. The paper also finds that a six-layer version keeps roughly the same performance while cutting inference time nearly in half. The implication is that in-house, commercially usable models can be built without relying on private data, as long as a structurally similar public corpus exists.

What carries the argument

The central object is LayoutLM, a BERT-style transformer whose input embeddings combine token embeddings with 2D positional embeddings (a discretized 6-tuple of coordinates, height, and width) so that self-attention can exploit document layout. The paper's intervention is to pre-train this architecture from scratch with a masked-language-modeling loss on DOCILE, an unlabeled invoice corpus, and then fine-tune it on the new PAYSLIPS dataset. The pre-training corpus is the only variable changed: architecture, MLM objective, and fine-tuning protocol are kept fixed while the corpus changes from IIT-CDIP to DOCILE. The paper also uses layer count as a dial on the same machinery, replacing the 12-layer base with 6-, 2-, and 1-layer versions to trade accuracy for speed.

What would settle it

Pre-train LayoutLM from scratch with the same protocol on a random subset of IIT-CDIP matched in size to DOCILE; if the F1 on PAYSLIPS is indistinguishable from the DOCILE-pre-trained model's 64.74, the domain-similarity explanation collapses.

Watch

Extended reading notes

Core claim

The central claim is that pre-training LayoutLM from scratch on DOCILE, a corpus of roughly 900k invoices that resemble payslips in layout and semantics, transfers better to insurance-domain NER than the official LayoutLM pre-trained on IIT-CDIP, even though DOCILE is more than ten times smaller. The numbers are 64.74 ± 2.92 F1 versus 62.31 ± 5.13 on PAYSLIPS, with the gain attributed mainly to higher precision (71.47 versus 65.70) rather than recall. The same pre-training choice leaves DOCILE-labeled NER performance essentially unchanged, which the paper reads as evidence that the benefit is not a general pre-training artefact but a domain-match effect. Reducing the model to six layers drops F1 to 61.80, close to the official base model, while halving inference time. The authors state that the result is highly significant under an approximate randomization test with significance value 0.0019.

Load-bearing premise

The load-bearing premise is that DOCILE invoices are similar enough to insurance payslips that the measured improvement on PAYSLIPS comes from that domain match, and not from other differences between the two pre-training setups, such as corpus size, how words are split, or training details.

Editorial extensions

If this is right

  • Domain-relevant public corpora can substitute for private in-house data when pre-training document models for regulated industries.
  • The F1 gain on PAYSLIPS is mainly a precision gain, meaning the DOCILE-pre-trained model produces fewer spurious extractions on payslip-like documents.
  • A six-layer LayoutLM pre-trained on DOCILE matches the official 12-layer model on PAYSLIPS while cutting per-page inference time roughly in half, making it a candidate for high-throughput document processing.
  • The released PAYSLIPS dataset gives document-understanding research a benchmark where the dominant label is O, so sparse-extraction behaviour becomes directly measurable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled next step would be pre-training from scratch on a similarly sized random sample of IIT-CDIP; if the PAYSLIPS F1 is indistinguishable from the DOCILE-pre-trained result, corpus similarity is not the active ingredient.
  • The comparison may be confounded if the from-scratch model used a different tokenizer than the official LayoutLM, since vocabulary differences alone can change extraction behaviour; the paper does not state which tokenizer was used.
  • The same recipe could transfer to other sparse financial documents such as bank statements or tax forms whenever a structurally similar public corpus exists, but that transfer is not demonstrated in this paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PAYSLIPS, a novel NER dataset of 611 anonymized insurance payslip pages with nine entity types. The authors pre-train LayoutLM from scratch on DOCILE, an invoice corpus, and compare it with the official LayoutLM pre-trained on IIT-CDIP. They report that the DOCILE-pretrained model achieves higher F1 on PAYSLIPS (64.74 vs 62.31) with lower variance, and that a 6-layer variant retains competitive performance at roughly half the inference time. The paper argues that pre-training on domain-relevant documents improves downstream NER and proposes a practical recipe for building efficient in-house models.

Significance. If the causal claim were validated, the result would be practically significant: it would show that a tenfold-smaller, domain-specific corpus can outperform a massive generic corpus for specialized document NER, and that smaller encoders can be fast enough for production. The paper also ships a new public dataset and code, and uses 100 fine-tuning runs per condition, which is commendable. The central comparison, however, is confounded by differences in corpus size, pre-training length, and possibly tokenizer, and the significance test used to support the headline difference is statistically invalid. The dataset release is a contribution, but the empirical support for the main claim needs strengthening.

major comments (3)
  1. [§4.2, Table 2] The headline comparison contrasts a from-scratch pre-training on DOCILE (~900k pages, 5 epochs, batch 80, lr 5e-5) with the released LayoutLM-base checkpoint pre-trained on IIT-CDIP (11M documents, unknown protocol). These conditions differ in corpus domain, corpus size, pre-training length, and possibly tokenizer/vocabulary — the paper never states whether the from-scratch model reuses the original LayoutLM tokenizer. The 2.43-point F1 gain on PAYSLIPS is therefore not attributable specifically to 'semantically and structurally similar' pre-training data, especially since that similarity is supported only by visual inspection in Figure 2. A control pre-training with the same recipe on a size-matched generic corpus (e.g., a random subset of IIT-CDIP or RVL-CDIP invoices) is needed to isolate the effect of domain relevance. Without such a control, the paper's central claim and the transferable recipe 'use domain-relevant pre-training data' are not established.
  2. [§5.2, Appendix A, Algorithm 1] The approximate randomization test is not a valid test of the difference between the two systems. Algorithm 1 concatenates the N predictions of the two systems, shuffles them, and splits them into two halves, thereby destroying the pairing of predictions on the same test items; this tests whether random partitions of the pooled prediction set can mimic the observed F1 gap, not whether the two systems differ on individual items. A valid test would randomly swap, for each test item, the prediction of the two systems (or shuffle system labels per item) and recompute the F1 difference. Additionally, Section 5.2 states that 'we took two models with a F1-score difference below 1.00 for the test'; this post-hoc selection from 100 runs makes the reported significance value of 0.0019 unreliable. The significance of the headline result is therefore unsupported as reported.
  3. [Appendix C] The PAYSLIPS evaluation is performed on the original, non-public version of the dataset; the released version has manually altered amounts and other sensitive information. The paper states that 'the results presented in this paper are based on this version' and does not report results on the released, altered version. Since the public data differ from the evaluated data, the exact numbers in Table 2 cannot be reproduced from the released artifacts. The paper should either report results on the released version or provide a clear statement of the expected change, together with an evaluation on the altered data.
minor comments (6)
  1. [§4.2] State explicitly whether the from-scratch model reuses the original LayoutLM tokenizer; if a new tokenizer is trained, report its vocabulary size and training data, as this is a potential confound.
  2. [§4.2] Report all pre-training hyperparameters (max sequence length, masking probability, weight decay, warmup steps, etc.) beyond batch size, learning rate, and epochs, to allow exact reproduction.
  3. [§5.2] Clarify the sentence 'we took two models with a F1-score difference below 1.00 for the test'; the selection criterion is ambiguous and should be described precisely.
  4. [Table 4] Report standard deviations for precision and recall as well, since F1 is presented with ± values; the current table gives only point estimates for these metrics.
  5. [Figure 2] The visual comparison in Figure 2 is suggestive but qualitative; consider adding a quantitative measure of layout or semantic similarity (e.g., distribution of spatial box positions or field-name overlap) to support the claim that DOCILE and PAYSLIPS are closer than IIT-CDIP and PAYSLIPS.
  6. [Appendix C] Clarify the relationship between the evaluated version and the released version, and state whether the released version was validated with the NER pipeline; if not, discuss the expected impact of the modifications.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DOCILE-pretrained model is compared against an external off-the-shelf checkpoint on a held-out NER test set, so the central result is not forced by construction.

full rationale

The paper's central claim is that pre-training LayoutLM from scratch on DOCILE yields F1 64.74 versus 62.31 on PAYSLIPS. This is an empirical comparison between two models, one from an external IIT-CDIP checkpoint and one trained from scratch on DOCILE, evaluated on a held-out test set. No parameter is fitted to the PAYSLIPS result, and the choice of DOCILE is not used to define the metric. The paper does not rely on a self-citation chain or a uniqueness theorem; the cited prior work is standard external literature. The main limitations are experimental confounds: the comparison changes corpus domain, corpus size, pre-training duration, OCR pipeline, and potentially the tokenizer simultaneously, and the Appendix A significance test selects a favorable pair of runs and uses a nonstandard shuffle. These are threats to validity, not cases where the result reduces to its input by construction. Therefore, under the required standard, no circularity step is identifiable.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that DOCILE is a valid proxy for insurance payslips and that the only meaningful difference between the two compared models is the pre-training corpus. The paper does not provide a size-matched control, a quantitative domain-similarity measure, or a description of tokenizer handling, so the comparison is not fully controlled. The listed hyperparameters are hand-chosen but are fixed across the compared models.

free parameters (6)
  • pre-training epochs = 5
    Chosen by hand for DOCILE pre-training; the comparison against the official IIT-CDIP checkpoint does not match pre-training steps.
  • pre-training batch size = 80
    Hand set, stated as similar to Xu et al. (2020).
  • pre-training learning rate = 5e-5
    Hand set, with a 5% warmup cosine schedule.
  • fine-tuning epochs = 10
    Fixed for all model comparisons and not tuned per model.
  • fine-tuning learning rate = 5e-5
    Fixed for all model comparisons.
  • reduced layer count = 6 layers
    Chosen as half of the 12-layer base; not optimized, and 2-layer and 1-layer variants are also reported.
assumptions (4)
  • domain assumption DOCILE invoices are semantically and structurally similar to PAYSLIPS payslips
    The paper's central hypothesis; supported only by visual inspection (Figure 2) and not by any quantitative domain-distance or layout statistics.
  • domain assumption MLM pre-training on 900k DOCILE documents is sufficient for a from-scratch LayoutLM
    The paper asserts this based on its experimental results, but there is no ablation with other corpus sizes or a size-matched generic corpus.
  • domain assumption The tokenizer and vocabulary are identical between the IIT-CDIP and DOCILE models
    Never stated in Section 4.2; if a DOCILE-specific tokenizer was trained, the comparison confounds vocabulary coverage with pre-training data.
  • domain assumption The OCR layout coordinates from DOCILE and PAYSLIPS are normalized consistently to 0-1000
    LayoutLM expects normalized coordinates; the paper does not describe how the two corpora are aligned, which could affect the transfer result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain." pith.science (2026). https://pith.science/paper/2HZDKP4X

@misc{pith2026241209341,
  author       = {Pith},
  title        = {Pith review of: Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HZDKP4X}},
  note         = {Machine review of arXiv:2412.09341}
}
read the original abstract

Generic pre-trained neural networks may struggle to produce good results in specialized domains like finance and insurance. This is due to a domain mismatch between training data and downstream tasks, as in-domain data are often scarce due to privacy constraints. In this work, we compare different pre-training strategies for LayoutLM. We show that using domain-relevant documents improves results on a named-entity recognition (NER) problem using a novel dataset of anonymized insurance-related financial documents called Payslips. Moreover, we show that we can achieve competitive results using a smaller and faster model.

Figures

Figures reproduced from arXiv: 2412.09341 by the authors.

Figure 1
Figure 1. Sample of the newly introduced PAYSLIPS dataset for named-entity recognition in the insurance domain. of their models, or to change the license to for￾bid commercial use.1 As such, it is increasingly important to ensure replicability and robustness to changes in training data (including for domain transfer) not only for scientific reasons, but also to ensure widespread commercial deployment. In this work, we study L… view at source ↗
Figure 2
Figure 2. Samples from IIT-CDIP (first column), DOCILE (second column) and PAYSLIPS (third column) datasets. Invoices from DOCILE and pay statements from PAYSLIPS are closer visually and semantically. ing the useful financial information. To this end, we worked with insurance professionals and iden￾tified nine specific fields, as detailed in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 13 canonical work pages

  1. [1]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. http://arxiv.org/abs/1409.0473 Neural machine translation by jointly learning to align and translate . In ICLR

    arXiv 2015

  2. [2]

    Nancy Chinchor. 1992. https://aclanthology.org/M92-1003 The statistical significance of the MUC -4 results . In F ourth M essage U nderstanding C onference ( MUC -4): Proceedings of a Conference Held in M c L ean, V irginia, J une 16-18, 1992

    work page 1992

  3. [3]

    Nancy Chinchor, Lynette Hirschman, and David D. Lewis. 1993. https://aclanthology.org/J93-3001 Evaluating message understanding systems: An analysis of the third M essage U nderstanding C onference ( MUC -3) . Computational Linguistics, 19(3):409--450

    work page 1993

  4. [4]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    2023

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    Thibault Douzon, Stefan Duffner, Christophe Garcia, and J \'e r \'e my Espinas. 2023. https://link.springer.com/chapter/10.1007/978-3-031-41501-2_4#citeas Long-range transformer architectures for document understanding . In Document Analysis and Recognition -- ICDAR 2023 Workshops, pages 47--64, Cham. Springer Nature Switzerland

    work page doi:10.1007/978-3-031-41501-2_4 2023

  7. [7]

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. 2015. https://adamharley.com/icdar15/< Evaluation of deep convolutional nets for document image classification and retrieval . In International Conference on Document Analysis and Recognition ( ICDAR )

    work page 2015

  8. [8]

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. https://doi.org/10.1145/3503161.3548112 Layoutlmv3: Pre-training for document ai with unified text and image masking . In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, page 4083–4091, New York, NY, USA. Association for Computing Machinery

    arXiv 2022

Show all 30 references
  1. [9]

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. https://doi.org/10.1109/ICDAR.2019.00244 Icdar2019 competition on scanned receipt ocr and information extraction . In 2019 International Conference on Document Analysis and...

    2019

  2. [10]

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. https://doi.org/10.1109/ICDARW.2019.10029 Funsd: A dataset for form understanding in noisy scanned documents . In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume ...

    2019

  3. [11]

    Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. https://arxiv.org/abs/1602.02410 Exploring the limits of language modeling . Preprint, arXiv:1602.02410

    2016 arXiv

  4. [12]

    Lewis, G

    D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. 2006. https://doi.org/10.1145/1148170.1148307 Building a test collection for complex document information processing . In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Dev...

    2006

  5. [13]

    Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021. https://doi.org/10.18653/v1/2021.acl-long.493 S tructural LM : Structural pre-training for form understanding . In Proceedings of the 59th Annual Meeting of the Association for Computational...

    2021 doi

  6. [14]

    Mahmoud Limam, Marwa Dhiaf, and Yousri Kessentini. 2023. https://doi.org/10.5281/zenodo.8261508 Fatura dataset

    2023 doi

  7. [15]

    Laura Nguyen, Thomas Scialom, Jacopo Staiano, and Benjamin Piwowarski. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.207 Skim-attention: Learning to focus via document layout . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2413--2427, ...

    2021 doi

  8. [16]

    E.W. Noreen. 1989. https://books.google.fr/books?id=kinvAAAAMAAJ Computer- Intensive Methods for Testing Hypotheses : An Introduction . Wiley

    1989

  9. [17]

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsing

    2019

  10. [18]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  11. [19]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  12. [20]

    Lance Ramshaw and Mitch Marcus. 1995. https://aclanthology.org/W95-0107 Text chunking using transformation-based learning . In Third Workshop on Very Large Corpora

    1995

  13. [21]

    Heidi Schmidt, Karen Butter, and Cynthia Rider. 2002. https://www.dlib.org/dlib/september02/schmidt/09schmidt.html Building digital tobacco industry document libraries at the university of california, san francisco library/center for knowledge management . D-Lib Magazine, 8(9)...

    2002

  14. [22]

    Hinrich Sch \"u tze. 1998. https://aclanthology.org/J98-1004 Automatic word sense discrimination . Computational Linguistics, 24(1):97--123

    1998

  15. [23]

    Holger Schwenk and Jean-Luc Gauvain. 2005. https://aclanthology.org/H05-1026 Training neural network language models on very large corpora . In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 201--20...

    2005

  16. [24]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  17. [25]

    Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. https://doi.org/10.18653/v1/2022.acl-long.534 L i LT : A simple yet effective language-independent layout transformer for structured document understanding . In Proceedings of the 60th Annual Meeting of the Association for Computa...

    2022 doi

  18. [26]

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. https://doi.org/10.18653/v1/2021.acl-long.201 L ayout LM v2: Multi-modal pre-training for visually-rich document understandin...

    2021 doi

  19. [27]

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. https://doi.org/10.1145/3394486.3403172 Layoutlm: Pre-training of text and layout for document image understanding . In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discov...

    2020

  20. [28]

    Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalický, Jiří Matas, Antoine Doucet, Mickaël Coustaty, and Dimosthenis Karatzas. 2023. https://link.springer.com/chapter/10.1007/978-3-031-41679-8_9#citeas DocILE Benchmark for Document Inf...

    2023 doi

  21. [29]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.