Pith. sign in

REVIEW 3 major objections 4 minor 27 references

Seeing an invoice beats reading a text-parsed version of it: for eight large language models, feeding the raw image usually outperforms converting to markdown first.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across three invoice datasets, multimodal LLMs extract fields more accurately from raw images than from markdown converted by a parsing tool, with Gemini 2.5 Pro leading.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful, honestly-scoped benchmark of native-vs-parsed invoice extraction; the headline result is consistent, but contamination and naming sloppiness keep it from being definitive. the 3 major comments →

arxiv 2509.04469 v1 pith:54A4POHL submitted 2025-08-29 cs.CL cs.AI

Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing

David Berghaus , Armin Berger , Lars Hillebrand , Kostadin Cvejoski , Rafet Sifa This is my paper
classification cs.CL cs.AI
keywords invoice processingmulti-modal large language modelsdocument understandingOCR vs native visionfield extractionzero-shot evaluationDocling parsingscanned documents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to settle a practical question: when an LLM must pull fields out of an invoice, should it be shown the document image or a text version produced by an OCR-to-markdown parser? On three invoice/receipt datasets and eight models from three model families, the answer is consistent: direct image input wins on every dataset and for nearly every model. The gaps are large—on scanned receipts the best native-image model reached 87.46% accuracy versus 47.00% for the best parsed-text result. The authors read this as evidence that visual layout and spatial context carry information that a markdown conversion discards, and that the conversion step, not the LLM's reasoning, becomes the bottleneck.

Core claim

The paper's central claim is that native image processing is the superior strategy for zero-shot invoice field extraction with current multi-modal LLMs. Across all three datasets (clean synthetic invoices, noisy scanned receipts, and scanned invoices with real-world artifacts), direct image analysis produced higher overall accuracy than the Docling markdown pipeline in nearly every model comparison. For example, Gemini 2.5 Pro scored 96.50% on the clean invoices and 92.71% on scanned invoices natively, versus 85.14% and 63.94% through the parser. The paper also identifies a distinctive bottleneck signature: on the clean invoices most parsed-text models cluster in a narrow 84–85% band, sugges

What carries the argument

The machinery is a paired two-strategy benchmark: identical zero-shot prompts and ground-truth scoring, with the only intended difference being the input representation—raw document image versus Docling-generated markdown. Accuracy is field-level exact match after minimal normalization (dates standardized, whitespace collapsed, but commas and dots in numbers left untouched). The contrast between these two branches is what isolates the effect of losing visual context, and the field-level metric is what exposes which data types suffer most.

Load-bearing premise

The benchmark assumes the publicly available invoice datasets were not seen by the models during training, although the authors note this is possible; memorization would inflate the zero-shot accuracies and could create the native-versus-parsing gap artificially.

What would settle it

Run the same eight models on a private, never-published set of invoice images with the same field types and layouts; if the native-image advantage over Docling shrinks or disappears, training-data memorization inflated the reported gap.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Organizations building invoice automation on current LLMs should feed document images directly rather than pre-converting them to markdown.
  • The parser bottleneck means improving the LLM alone will not fix parsed-text pipelines; gains must come from better layout-aware conversion.
  • Gemini 2.5 Pro is the strongest all-round choice in this benchmark, while gemma-3-12b-it is the open-source option worth considering.
  • Alphanumeric identifier fields like IBAN remain error-prone; systems that depend on them need additional validation or specialized handling.
  • Smaller models show a capability threshold on direct vision: gemma-3-4b-it collapses to 45.69% on clean invoices, so model size matters more in the image branch than in the parsed branch.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the advantage is attributed to spatial layout, a testable extension is to compare the two strategies on other tabular documents (financial statements, forms) where layout carries meaning; the gap should generalize.
  • The authors tested SmolDocling but do not report numbers because they were almost identical to Docling; reporting that near-equivalence would strengthen the claim that the bottleneck is the markdown representation itself, not the specific parser.
  • If the public datasets overlap with pretraining, the zero-shot numbers may overstate real-world generalization; a private held-out invoice set would settle how much of the native-image edge is genuine understanding versus memorization.
  • The exact-match, no-currency-correction metric likely understates real-world value: a system that gets the digits right but swaps comma and point may still be operationally useful, so practical accuracy could be higher than reported.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper presents a benchmark of eight multimodal LLMs (GPT-5, Gemini 2.5, Gemma 3) on three publicly available invoice/receipt datasets (Donut, SROIE, inv-cdip). For each model and dataset, the authors compare two processing strategies: native image input, where the model receives the document image directly, and a structured parsing strategy, where the document is first converted to markdown using Docling/SmolDocling and then fed to the model as text. The paper reports overall and field-level extraction accuracies in zero-shot settings. The central claim (Sections IV.A and V) is that native image processing consistently and substantially outperforms the structured parsing approach, suggesting that visual context and layout understanding are crucial for invoice processing. The authors also examine model-family differences, challenging fields (e.g., IBANs), and limitations such as scanned-document noise.

Significance. If the result holds, the benchmark provides a useful, reproducible comparison (code is released) of current multimodal LLMs for document automation, with a clear practical takeaway: direct image input beats OCR-to-markdown conversion for this task. The consistent directional signal across 24 model-dataset combinations is a strength. However, the paper's central interpretive claim — that the gap reflects genuine visual understanding — rests on an unaddressed training-data contamination risk, and the reported 'significance' is not backed by any statistical analysis. The manuscript also contains a major inconsistency between the text and Table I regarding which parsing tool was actually used. Because these issues affect the validity of the main conclusion, the work is not yet ready for publication without revision.

major comments (3)
  1. [Section III.D, IV.A, V] The paper concedes that all three datasets are open source and 'it is possible that they have been used as training data for the large language models.' This creates an asymmetric contamination risk: the native condition presents the exact original document image, which a model could memorize (image-to-annotation mappings), while the Docling condition presents a converted markdown string that is unlikely to have been seen in training. Thus memorization would selectively inflate native-image accuracy, potentially explaining part of the headline gap (e.g., 87.46% vs 47.00% on SROIE, 96.50% vs 85.14% on Donut). The conclusion in Section V that 'visual context and layout understanding are crucial' is not established without ruling out this contamination pathway. Please provide a contamination check (e.g., held-out invoices, membership-inference analysis, or a comparison with results on novel
  2. [Section III.C, Table I] There is a direct contradiction between the text and Table I. Section III.C states that SmolDocling was tested 'but the results were almost identical so we do not report them here'; the text consistently refers to the parsing pipeline as 'Docling.' However, Table I labels every parsed-strategy row as 'smoldocling.' This is not a minor typo: if the reported numbers are actually from SmolDocling, then the paper does not actually evaluate the Docling pipeline it claims to compare in the abstract and conclusion; if they are from Docling, the table is mislabeled. The authors must clarify which tool produced the reported results and, if SmolDocling was used, revise the text and title accordingly.
  3. [Section IV.A, Table I] The paper repeatedly uses the word 'significantly' (e.g., 'direct image analysis yielded significantly higher accuracy') without any error bars, confidence intervals, or statistical tests. All numbers are point estimates from a single run (or an unspecified aggregation). Because the central claim is the consistency and size of the native-vs-parsing gap, the absence of any variance measure is a load-bearing gap. Please report per-document or per-split results, bootstrap confidence intervals for the overall accuracies, or at minimum a clear statement that no statistical significance is claimed. This is essential for a benchmark paper whose headline is a performance comparison.
minor comments (4)
  1. [Section III.E] The normalization procedure is described as 'minimal normalization' with 'appropriate normalization' but the details are thin. For example, are monetary values with/without currency symbols treated as equal? Are dates normalized across formats (e.g., MM/DD/YYYY vs DD.MM.YYYY)? Please specify the exact normalization rules to make the results reproducible.
  2. [Appendix Figures 1-3] The field-level accuracy figures are dense and difficult to read; the legend lists all 16 model/strategy pairs, and the heatmap cells contain small numbers. Consider grouping by model family or using a clearer layout. Error bars per field would also help interpret the differences.
  3. [Section III.C] The sentence 'All datasets are stored as images, meaning that OCR techniques have to be used by the models in order to convert them to text' is imprecise. Multimodal models do not necessarily perform explicit OCR; they process the image directly via visual encoders. Please rephrase to avoid implying that native processing uses OCR.
  4. [References] Reference [16] is a blog post. If it is used to support the claim of Docling's 'SOTA performance,' please cite the Docling technical report [14] or a peer-reviewed evaluation instead.

Circularity Check

0 steps flagged

No circular derivation: the benchmark compares direct model outputs; the central claim is an empirical observation, not a reduction to fitted inputs or self-cited premises.

full rationale

This paper does not contain a derivation chain in the sense of fitting parameters and then predicting them back, nor does it import a uniqueness theorem or ansatz from the authors' prior work. The central claim—that native image processing outperforms Docling-based parsing (Section IV.A)—is supported directly by measured model accuracies on three external datasets. There are no fitted parameters, no equations that define the outcome in terms of the input, and no 'prediction' that is statistically forced by earlier fitting. The authors' self-citations in Related Work [2], [3], [4] describe prior business/auditing applications but are not load-bearing for the benchmark conclusion. The only substantive validity concern is training-data contamination: Section III.D notes 'the datasets are open source, so it is possible that they have been used as training data for the large language models.' This is an honest disclosure of a potential confound, but it is a limitation about generalization, not a circularity of the kind this pass targets. The comparison itself remains an externally observable benchmark result rather than a self-referential construction. Therefore, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No numeric parameters are fitted in this benchmark; the reported accuracies are direct model outputs. The only tuning is prompt design per dataset and the choice of normalization rules (exact match, date/whitespace normalization), which are methodological choices, not fitted values. No new entities are introduced; the paper relies on existing models, datasets, and parsing tools.

axioms (4)
  • domain assumption Ground truth annotations in Donut, SROIE, and inv-cdip are correct and reliable after the authors' single annotation fix.
    The benchmark evaluates against these labels without independent verification; the authors report and correct one inconsistency ('total net worth' vs 'item net worth' in 14 inv-cdip samples), implying other errors may remain.
  • domain assumption The open-source invoice images have not been used to train the tested models (zero-shot assumption).
    Authors explicitly flag this uncertainty in Section III.D: 'it is possible that they have been used as training data for the large language models.' If violated, reported accuracies measure memorization rather than generalization.
  • domain assumption Exact match after minimal normalization is a valid measure of extraction correctness.
    The paper does not correct for comma/dot differences in numbers, which can penalize semantically equivalent outputs; no human evaluation or partial-credit scoring is used.
  • domain assumption Docling/SmolDocling markdown conversion preserves the document structure faithfully enough for fair comparison.
    The parsing strategy's performance is bottlenecked by the converter; the choice of converter is a fixed pipeline component and not varied beyond the mentioned test.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing." pith.science (2026). https://pith.science/paper/54A4POHL

@misc{pith2026250904469,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54A4POHL}},
  note         = {Machine review of arXiv:2509.04469}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper benchmarks eight multi-modal large language models from three families (GPT-5, Gemini 2.5, and open-source Gemma 3) on three diverse openly available invoice document datasets using zero-shot prompting. We compare two processing strategies: direct image processing using multi-modal capabilities and a structured parsing approach converting documents to markdown first. Results show native image processing generally outperforms structured approaches, with performance varying across model types and document characteristics. This benchmark provides insights for selecting appropriate models and processing strategies for automated document systems. Our code is available online.

Figures

Figures reproduced from arXiv: 2509.04469 by Armin Berger, David Berghaus, Kostadin Cvejoski, Lars Hillebrand, Rafet Sifa.

Figure 1
Figure 1. Figure 1: Field extraction performance on the Scanned Receipts (ICDAR [18]) dataset [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Field extraction performance on the Clean Invoices (Donut [17]) dataset [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Field extraction performance on the Scanned Invoices (inv-cdip [19]) dataset [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages · 1 internal anchor

  1. [1]

    From Theory to Practice: Real-World Use Cases on Trustworthy LLM-Driven Process Modeling, Prediction and Automation

    P. Pfeiffer, A. Rombach, M. Majlatow, and N. Mehdiyev, “From theory to practice: Real-world use cases on trustworthy llm-driven process modeling, prediction and automation,” 2025. [Online]. Available: https://arxiv.org/abs/2506.03801

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Towards automated auditing with machine learning,

    R. Sifa, A. Ladi, M. Pielka, R. Ramamurthy, L. Hillebrand, B. Kirsch, D. Biesner, R. Stenzel, T. Bell, M. L ¨ubbering, U. N ¨utten, C. Bauckhage, U. Warning, B. F ¨urst, T. D. Khameneh, D. Thom, I. Huseynov, R. Kahlert, J. Schlums, H. Ismail, B. Kliem, and R. Loitz, “Towards automated auditing with machine learning,” in Proceedings of the ACM Symposium on...

    work page 2019

  3. [3]

    Advancing risk and quality assurance: A rag chatbot for improved regulatory compliance,

    L. Hillebrand, A. Berger, D. Uedelhoven, D. Berghaus, U. Warning, T. Dilmaghani, B. Kliem, T. Schmid, R. Loitz, and R. Sifa, “Advancing risk and quality assurance: A rag chatbot for improved regulatory compliance,” in 2024 IEEE International Conference on Big Data (BigData), 2024, pp. 8668–8670

    work page 2024

  4. [4]

    Fine-tuning large language models for compliance checks,

    T. Bell, D. Leonhard, A. H. Bashir, T. Dilmaghani, M. Khaled, U. Warn- ing, R. Loitz, S. Halscheidt, J. Birr, A. Berger, R. Sifa, and D. Berghaus, “Fine-tuning large language models for compliance checks,” in 2024 IEEE International Conference on Big Data (BigData) , 2024, pp. 8790– 8792

    work page 2024

  5. [5]

    An overview of data extraction from invoices,

    T. Saout, F. Lardeux, and F. Saubion, “An overview of data extraction from invoices,” IEEE Access, vol. 12, pp. 19 872–19 886, 2024

    work page 2024

  6. [6]

    A survey of deep learning approaches for ocr and document understanding,

    N. Subramani, A. Matton, M. Greaves, and A. Lam, “A survey of deep learning approaches for ocr and document understanding,” ArXiv, vol. abs/2011.13534, 2020. [Online]. Available: https://api.semanticscholar. org/CorpusID:227209404

    Pith/arXiv arXiv 2011

  7. [7]

    Deep learning based visually rich document content understanding: A survey,

    Y . Ding, J. Lee, and S. C. Han, “Deep learning based visually rich document content understanding: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2408.01287

    Pith/arXiv arXiv 2024

  8. [8]

    Memory-augmented agent training for business document understanding,

    J. Liu, Y . Zeng, M. Højmark-Bertelsen, M. N. Gadeberg, H. Wang, and Q. Wu, “Memory-augmented agent training for business document understanding,” 2024. [Online]. Available: https://arxiv.org/abs/2412. 15274

    work page 2024

  9. [9]

    Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks,

    M. Wornow, A. Narayan, B. Viggiano, I. S. Khare, T. Verma, T. Thompson, M. A. F. Hernandez, S. Sundar, C. Trujillo, K. Chawla, R. Lu, J. Shen, D. Nagaraj, J. Martinez, V . Agrawal, A. Hudson, N. H. Shah, and C. Re, “Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks,”

    work page

  10. [10]

    MMLONGBENCH-DOC: Benchmarking long-context document understanding with visualizations,

    Y . Ma, Y . Zang, L. Chen, M. Chen, Y . Jiao, X. Li, X. Lu, Z. Liu, Y . Ma, X. Dong, P. Zhang, L. Pan, Y .-G. Jiang, J. Wang, Y . Cao, and A. Sun, “MMLONGBENCH-DOC: Benchmarking long-context document understanding with visualizations,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024. [Online]. A...

    work page 2024

  11. [11]

    M-longdoc: A benchmark for multimodal super- long document understanding and a retrieval-aware tuning framework,

    Y . K. Chia, L. Cheng, H. P. Chan, C. Liu, M. Song, S. M. Aljunied, S. Poria, and L. Bing, “M-longdoc: A benchmark for multimodal super- long document understanding and a retrieval-aware tuning framework,”

    work page

  12. [12]

    Docbench: A benchmark for evaluating llm-based document reading systems,

    A. Zou, W. Yu, H. Zhang, K. Ma, D. Cai, Z. Zhang, H. Zhao, and D. Yu, “Docbench: A benchmark for evaluating llm-based document reading systems,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10701

    Pith/arXiv arXiv 2024

  13. [13]

    Gemma 3 technical report,

    G. Team, “Gemma 3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2503.19786

    Pith/arXiv arXiv 2025

  14. [14]

    Available: https://arxiv.org/abs/2411.06176

    [Online]. Available: https://arxiv.org/abs/2411.06176

    Pith/arXiv arXiv

  15. [15]

    Smoldocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion,

    A. Nassar, A. Marafioti, M. Omenetti, M. Lysak, N. Livathinos, C. Auer, L. Morin, R. T. de Lima, Y . Kim, A. S. Gurbuz, M. Dolfi, M. Farr´e, and P. W. J. Staar, “Smoldocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion,” 2025. [Online]. Available: https://arxiv.org/abs/2503.11576

    Pith/arXiv arXiv 2025

  16. [16]

    Pdf data extraction benchmark 2025: Comparing docling, unstructured, and llamaparse for document processing pipelines,

    A. Javanmard, “Pdf data extraction benchmark 2025: Comparing docling, unstructured, and llamaparse for document processing pipelines,” 2025. [Online]. Available: https://procycons.com/en/blogs/ pdf-data-extraction-benchmark/

    work page 2025

  17. [17]

    Docling technical report,

    C. Auer, M. Lysak, A. Nassar, M. Dolfi, N. Livathinos, P. Vagenas, C. B. Ramis, M. Omenetti, F. Lindlbauer, K. Dinkla, L. Mishra, Y . Kim, S. Gupta, R. T. de Lima, V . Weber, L. Morin, I. Meijer, V . Kuropiatnyk, and P. W. J. Staar, “Docling technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2408.09869

    Pith/arXiv arXiv 2024

  18. [18]

    Icdar 2019 robust reading challenge on scanned receipts ocr and information extraction,

    Niansong Zhang, “Icdar 2019 robust reading challenge on scanned receipts ocr and information extraction,” Online resource, 2019. [Online]. Available: https://github.com/zzzDavid/ICDAR-2019-SROIE

    work page 2019

  19. [19]

    Field extraction from forms with unlabeled data,

    M. Gao, Z. Chen, N. Naik, K. Hashimoto, C. Xiong, and R. Xu, “Field extraction from forms with unlabeled data,” ACL Spa-NLP Workshop , 2022

    work page 2022

  20. [20]

    Ocr-free document understanding transformer,

    G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park, “Ocr-free document understanding transformer,” in Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII . Berlin, Heidelberg: Springer-Verlag, 2022, p. 498–517. [Online]. Available: https://doi.org/10.10...

    doi:10.1007/978-3-031-19815-1 2022

  21. [21]

    Layoutlm: Pre-training of text and layout for document image understanding,

    Y . Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, “Layoutlm: Pre-training of text and layout for document image understanding,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , ser. KDD ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 1192–1200. [Online]. Available: https://doi...

    work page arXiv 2020

  22. [22]

    Layoutlmv3: Pre-training for document ai with unified text and image masking,

    Y . Huang, T. Lv, L. Cui, Y . Lu, and F. Wei, “Layoutlmv3: Pre-training for document ai with unified text and image masking,” in Proceedings of the 30th ACM International Conference on Multimedia, ser. MM ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 4083–4091. [Online]. Available: https://doi.org/10.1145/3503161.3548112

    work page arXiv 2022

  23. [23]

    Truth tobacco industry documents (formerly legacy tobacco documents library),

    University of California, San Francisco, “Truth tobacco industry documents (formerly legacy tobacco documents library),” Online resource, 2002, digital archive of 14+ million internal tobacco industry documents. Includes collections from major companies (e.g., Philip Morris, British American Tobacco) and spans advertising, manufacturing, and scientific re...

    work page 2002

  24. [26]

    Lilt: A simple yet effective language- independent layout transformer for structured document understanding,

    J. Wang, L. Jin, and K. Ding, “Lilt: A simple yet effective language- independent layout transformer for structured document understanding,” in Annual Meeting of the Association for Computational Linguistics ,

    work page

  25. [2019]

    Available: https://doi.org/10.1145/3342558.3345421

    [Online]. Available: https://doi.org/10.1145/3342558.3345421

    work page arXiv

  26. [2022]

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 247158521 APPENDIX A FIELD EXTRACTION PERFORMANCE Address Company Date T otal gemini-2.5-pro (native) gemini-2.5-pro (docling) gemini-2.5-flash (native) gemini-2.5-flash (docling) gemini-2.5-flash-lite (native) gemini-2.5-flash-lite (docling) gpt-5-chat (native) gpt-5-chat (docling) gpt-5-mini...

    work page

  27. [2024]

    Available: https://arxiv.org/abs/2406.13264

    [Online]. Available: https://arxiv.org/abs/2406.13264

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.