Pith. sign in

REVIEW 4 major objections 5 minor 41 references

The paper argues that classifying a document before prompting a frozen vision-language model with type-specific instructions yields more accurate field extraction than a trained OCR pipeline, with zero fine-tuning of the model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:53 UTC pith:LV5DOP25

load-bearing objection Useful engineering paper with an overstated 'zero-shot' claim; the private-dataset gain mostly reflects hand-crafted per-type prompts, not a general method. the 4 major comments →

arxiv 2607.22723 v1 pith:LV5DOP25 submitted 2026-07-22 cs.CV

Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models

Huafu Li , Guo Chen , Jia Xia , Lei Wang , Wei Du , Yun Yao , Weijun Peng , Liming Li This is my paper
classification cs.CV
keywords visual information extractiondocument understandinglarge vision-language modelsin-context learningprompt engineeringdocument-type classificationkey information extractionzero-shot extraction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the hard part of extracting fields from many visually different document types is not the extraction itself but telling the model what to look for. Its framework splits the task in two: a fast classifier decides which of 16 document types is in the image, and a prompt assembler builds a short, type-specific prompt containing the fields to extract, layout hints, and two to four worked examples. This prompt is then handed to a frozen vision-language model, which never sees any task-specific training. On a real-world bidding dataset, the zero-shot pipeline reaches 86.43% F1, beating a supervised OCR-plus-extraction baseline by 18.35 percentage points; optional fine-tuning pushes F1 to 93.65%. The claim matters because, if correct, it means high-accuracy multi-type document extraction can be achieved with almost no labeled extraction data, shifting the bottleneck to document-type classification and prompt design.

Core claim

Decoupling document-type classification from content extraction and using the predicted type to assemble a short, type-specific prompt turns multi-type document extraction into conditional generation. The authors formalize this as hard routing: a classifier selects one of several per-type prompts, approximating a sum over document types and narrowing the model's hypothesis space before generation. Removing irrelevant instructions raises the prompt's signal-to-noise ratio and prevents attention dilution. On a private 16-type bidding dataset, the zero-shot pipeline reaches 86.43% F1 and 0.90 NED, beating a trained OCR-and-extraction pipeline by 18.35 points; removing classification alone drops

What carries the argument

The mechanism is classification-guided dynamic prompt construction. A document-type classifier (either training-free feature-index matching or a trained image classifier, optionally fused with OCR keyword counts) predicts the document type; a prompt assembler then concatenates a shared instruction block with a type-specific landmark list, document background and layout hints, and 2-4 in-context demonstrations. This dynamically assembled prompt is the only task-specific knowledge the frozen LVLM receives. The paper also supplies an information-theoretic interpretation: conditional prompting is a hard-routing approximation to P(Y|I)=sum_d P(Y|I,P(d))P(d|I), which reduces prompt entropy, raises

Load-bearing premise

The load-bearing premise is that a person with knowledge of the target dataset hand-crafts the per-type landmarks, layout hints, and 2-4 demonstration examples; the headline zero-shot result is measured on the same private dataset those prompts were written for, so the gain could partly reflect prompt engineering rather than routing itself.

What would settle it

Apply the exact pipeline to a document type whose type-specific prompts, landmarks, and demonstrations were not written by the authors, for instance a public dataset with a schema the paper did not see, and compare the zero-shot F1 to the monolithic-prompt baseline. If routing alone adds no more than a few points once the prompt content is controlled, the central claim of conditional computation is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim is correct, high-accuracy multi-type VIE no longer requires task-specific fine-tuning of a generation model; a frozen LVLM plus a classifier and per-type prompt templates is enough to beat a trained OCR pipeline.
  • The 18.35-point zero-shot gain implies that most of the difficulty in this task is task ambiguity, not perception: the same model with a monolithic prompt loses most of the advantage.
  • Because the 7-billion-parameter model outperforms a 72-billion-parameter one on the private dataset at 2.5x lower latency, the framework suggests model scale is secondary to prompt conditioning for this class of documents.
  • The public-benchmark results staying close to the base model indicate the routing mechanism does not destroy the LVLM's general document understanding, so the method should carry over to new document types with modest prompt adaptation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same routing idea should transfer to other structured-output tasks, such as tables, forms, receipts, and medical records, wherever the document category is predictable and the output schema differs by category; a natural test is learning the landmark lists and demonstrations automatically from a few labeled examples.
  • Beyond the paper: the ablation pins most of the gain on the classification step, so the framework's ceiling is set by classifier accuracy and prompt quality rather than by LVLM capacity; improving classifier confidence estimates or allowing soft/hypothesis-based prompting should raise extraction accuracy further.
  • Beyond the paper: the reported cross-dataset numbers are hard to interpret without knowing how the 16-type classifier and prompts were mapped onto datasets with different schemas; an independent evaluation on a document type whose prompts were not hand-crafted by the same authors is the open test of generality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a classification-guided LVLM framework for multi-type visual information extraction (multi-VIE). A document-type classifier (feature matching or a trained ConvNeXt combined with an OCR keyword filter) selects a type-specific prompt containing landmarks, layout/background hints, and 2–4 in-context demonstrations, which is then fed to a frozen Qwen2.5-VL-7B model; optional LoRA fine-tuning is also explored. On a private real-world bidding dataset with 16 certificate types, the authors report that the zero-shot configuration reaches 86.43% F1 and 0.9012 NED, outperforming an OCR&UIE pipeline (68.08% F1, 0.67 NED), and that fine-tuning improves to 93.65% F1 / 0.9348 NED. They also present a theoretical framing based on conditional computation, information-theoretic SNR, and attention dilution, plus experiments on eight public benchmarks.

Significance. If the headline zero-shot result were robust, the paper would make a useful practical contribution: a simple classification step plus dynamic ICL prompting could avoid LVLM fine-tuning for multi-type VIE while retaining high accuracy, with meaningful latency savings (2.6 s vs 6.5 s per image for the 7B vs 72B model). The authors provide code, model weights, and a transparent ablation study (Table 6), and the 16-type real-world testbed with seals, watermarks, and low contrast is a valuable stress test for VIE systems. However, the evidence presented does not support the strong zero-shot generalization claim as stated: the gain appears largely attributable to manually crafted per-type prompt content on the same private dataset, and the public-benchmark results show no consistent advantage over the base LVLM.

major comments (4)
  1. [Results and Discussion / Multi-VIE Performance and Ablation Study (Tables 3, 5, 6)] The central zero-shot result is not cleanly traceable. Table 3 labels a row "Qwen2.5-VL-7B" with 86.43% F1 and 0.9012 NED, but the text says the table compares the fine-tuned model against two baseline versions; Table 5's "LVLM-based" column is the fine-tuned model (93.65%), and no per-type zero-shot column is given. The abstract's 18.35-point advantage is thus an inference across tables rather than a directly reported comparison. More substantively, the ablation in Table 6 shows that removing classification collapses the full zero-shot framework from 86.43 to 68.42 F1, nearly identical to the OCR&UIE baseline (68.08), and removing ICL drops it to 79.65. Since the classification-guided prompt includes per-type landmarks, layout hints, and ICL demonstrations authored with knowledge of the same 16 types and annotation schema (Figure 2), the observed gap is better explained as task-specific
  2. [Cross-Dataset Generalization Analysis (Table 4)] The public-benchmark results contradict the claim that the framework "generalizes well across datasets" and "preserves the general reasoning capability of LVLMs." Compared to base Qwen2.5-VL-7B, the proposed zero-shot method is worse on COLD-CELL (-1.67 F1), COLD-SIBR (-1.08), EPHOIE (-3.69), FUNSD (-0.56), POIE (-1.01), and SROIE (-0.68), with gains only on ICDAR (+0.99) and XFUND-zh (+2.00); NED also degrades on most datasets. Moreover, the paper does not describe how the 16-type classifier and per-type landmarks/demonstrations are adapted to datasets with different categories and annotation schemas (e.g., FUNSD vs SROIE). Without this protocol and per-dataset prompt details, the cross-dataset generality claim is not substantiated.
  3. [Alternative OCR&UIE Pipeline as Baseline (Table 5)] The comparison baseline is described as requiring "only a small number of annotated samples (e.g., 10 per type)" for unified information extraction, yet the abstract calls the OCR&UIE pipeline a "strong supervised baseline." This overstates the baseline. To support a zero-shot claim, the comparison should include strong zero-shot LVLM baselines with generic prompts (e.g., Qwen2.5-VL-7B without classification-guided prompting, GPT-4V) and strong supervised VIE models (LayoutLMv3, DocLLM, Donut, etc.) on the same data, matched in supervision and prompt information. The current design does not disentangle the contribution of classification-guided dynamic prompting from the LVLM's pretrained knowledge or the manual task-specific prompt content.
  4. [Theoretical Perspective (Eqs. 5–18)] The theoretical section is post hoc and rests on unverified assumptions: Eq. (9) assumes irrelevant prompt tokens are independent of the output, Eq. (15) asserts that attention mass to relevant tokens decreases monotonically with prompt length, and Eq. (16) restates the routing trade-off essentially tautologically. The information-theoretic and attention-dilution claims are not empirically tested. If the theory is intended as a contribution, please provide direct measurements (e.g., attention mass allocated to relevant tokens, or mutual information estimates under different prompt compositions); otherwise it should be explicitly framed as intuition, not as a proven mechanism, and separated from the empirical claims.
minor comments (5)
  1. [Evaluation Metrics, Eq. (23)] The NED definition is non-standard: I_k + D_k + M_k can exceed L_{g,k}, making the term negative, and the symbols are under-specified. Please either use a standard normalized edit distance or clarify the formula and its range.
  2. [Tables 3 and 4] The column/row label "Ours" is ambiguous: in Table 3 it denotes the LoRA-fine-tuned model, while in Table 4 it appears to denote the zero-shot classification-guided model. Use distinct names (e.g., "Ours-ZS" and "Ours-SFT") throughout.
  3. [Methodology, Classification Options] The term "zero-shot" is used even though classification Option 2 trains a ConvNeXt classifier on labeled data and Option 1 builds a reference index from labeled samples. Please specify which option is used in each reported result and clarify what "zero-shot" means precisely (no task-specific model training, but with prompt-level task knowledge).
  4. [Table 1 and Figure 3] Abbreviations such as PILPC, CSCRC, TNSSCC, and PCI are not expanded in the text; please provide full names at first mention. Figure 3 contains a typo: "Traing Set" should read "Training Set."
  5. [Benchmark Protocol (Table 4)] For public benchmarks, the authors should report how landmarks and demonstrations were defined for each dataset, including any filtering or mapping of entity schemas, so that the results are reproducible.

Circularity Check

2 steps flagged

Central zero-shot claim reduces to hand-fitted per-type prompt engineering on the same private dataset; theoretical SNR justification is tautological.

specific steps
  1. fitted input called prediction [Methodology: ICL-Based Prompt Engineering; Experiment: Dataset; Results: OCR&UIE vs. Classification-Guided LVLM (Table 5/Abstract)]
    "The predicted document type then injects the corresponding predefined entity list (landmarks) and appends 2–4 carefully selected in-context demonstrations. ... The dataset comprises 98,600 images of 16 common certificate types ... The data were split into training and test sets in a 7:3 ratio."

    The zero-shot F1 (86.43%) is reported on the test split of the same private 16-type bidding dataset whose type names, landmark lists, layout hints, and 2–4 ICL demonstrations were hand-authored. The 'prediction' therefore measures how well the manually fitted prompt content matches the annotation schema of the evaluation data, not zero-shot generalization to unseen document types. Table 4 shows the routing mechanism alone does not beat the base LVLM on most public benchmarks, isolating the private gain to the hand-fitted prompt content.

  2. self definitional [Theoretical Perspective: Information-Theoretic Analysis (Eqs. 8-12)]
    "Z=Z rel ∪Z irr. (8) ... Assuming irrelevant tokens are independent of the target output: I(Y;Z irr |I)≈0, (9) ... SNR= I(Y;Z rel |I) / H(Z) . (11) Classification-guided prompting removesZ irr, yielding: SNRguided ≫SNR all, (12)"

    The improvement SNR_guided >> SNR_all is not derived from any measured quantity; it is true by construction because Z_irr was defined as output-independent (Eq. 9) and SNR was defined as I(Y;Z_rel|I)/H(Z), while the guided prompt is defined to remove Z_irr. Equations 8-12 restate the design decision as a 'theoretical foundation' rather than proving it from first principles; the result is tautological.

full rationale

The paper's central comparison is not algebraically circular: the 18.35-point F1 gap is an empirical benchmark number, not a quantity derived from the method's equations. However, the 'zero-shot' label conceals a hand-fitted component: per-type landmarks, background/layout hints, and 2–4 ICL demonstrations were authored for the same 16 certificate types and annotation schema on which the F1 is reported. The gain over the OCR&UIE baseline is therefore partially attributable to this prompt fitting, not to the classification-routing mechanism per se; Table 4 shows the same mechanism does not reliably beat the base LVLM on public benchmarks. The theoretical section (Eqs. 8-12) is tautological: it defines irrelevant tokens as output-independent and then 'shows' removing them improves SNR. There is no load-bearing self-citation or imported uniqueness theorem, and the method is benchmarked on public data, so the circularity is partial rather than total. Score 5 reflects one prediction-like claim that reduces by construction (manual prompt fit on the evaluation dataset) plus a self-definitional theoretical step, while independent content remains in the fine-tuned results (93.65% F1) and in some public-benchmark transfers.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The system's empirical success rests on many hand-designed components: keyword sets, landmark lists, in-context examples, thresholds, and prompt structures tuned to the private dataset. The theoretical analysis is post hoc and not load-bearing for the reported numbers.

free parameters (5)
  • similarity threshold tau = 0.9
    Used in classification Option 1 to filter nearest neighbors; chosen by hand and not justified by a tuning curve.
  • retrieval neighbors K = 5
    Top-5 nearest neighbors are retrieved for feature-matching classification; arbitrary choice.
  • per-type ICL demonstrations = 2-4 hand-selected examples
    The number and content of in-context demonstrations are manually chosen per type and materially affect performance according to Table 6.
  • per-type landmark lists and layout hints = hand-authored for 16 types
    The shared prompt and type-specific landmarks are manually constructed using knowledge of the evaluation dataset; no release of the full templates.
  • OCR keyword sets K(d_i) = hand-authored
    The keyword-based classifier relies on manually defined keyword sets per document type.
axioms (5)
  • domain assumption Qwen2.5-VL's pretrained weights provide reliable OCR and entity extraction from Chinese documents when given type-specific prompts
    The entire framework relies on the frozen LVLM's ability to extract fields from visually rich Chinese certificates; no local training provides this capability.
  • domain assumption The 16 document types form a complete and mutually exclusive taxonomy for the deployment environment
    The classifier and per-type prompts assume every input belongs to one of the 16 categories; unknown types are not handled.
  • ad hoc to paper The conditional probability factorization and hard routing approximation (Equations 5-6) faithfully model LVLM behavior
    The paper treats P(Y|I) = sum_d P(Y|I,P(d))P(d|I) and the hard-routing approximation as a valid theoretical model without empirical verification.
  • ad hoc to paper Irrelevant prompt tokens are independent of the output, so removing them does not change I(Y;Z|I)
    Equation 9 assumes irrelevant tokens carry no mutual information with the output; this is asserted, not proven, and is unlikely to hold exactly in complex prompts.
  • ad hoc to paper Attention mass assigned to relevant tokens decreases monotonically as prompt length increases
    The attention-dilution argument (Equation 15) is presented as a general property without formal proof or measurement.

pith-pipeline@v1.3.0-alltime-deepseek · 13252 in / 11661 out tokens · 130795 ms · 2026-08-01T11:53:47.117998+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models." pith.science (2026). https://pith.science/paper/LV5DOP25

@misc{pith2026260722723,
  author       = {Pith},
  title        = {Pith review of: Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LV5DOP25}},
  note         = {Machine review of arXiv:2607.22723}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Visual information extraction (VIE) from visually rich documents remains challenging due to high layout variability and real-world impairments. Existing methods typically rely on sequential OCR pipelines or end-to-end models requiring extensive labeled data and layout-specific training, limiting their scalability.We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision. The approach decouples document-type classification from content extraction and employs in-context learning (ICL)-based dynamic prompt engineering to inject task-specific knowledge, enabling robust zero-shot inference across diverse layouts. From a theoretical perspective, the proposed method can be viewed as a form of conditional computation that reduces task uncertainty and improves information efficiency during prompt-based inference. Evaluated on a real-world bidding dataset with 16 certificate types, our zero-shot method (based on Qwen2.5-VL-7B) outperforms a strong supervised baseline by 18.35 percentage points in F1-score (86.43\% vs. 68.08\%) and 0.23 in normalized edit distance (0.90 vs. 0.67). Optional domain-specific fine-tuning further improves performance to 93.65\% F1 and 0.93 NED, demonstrating superior robustness against seals, watermarks, and low contrast. The framework offers an efficient, scalable solution for complex document understanding in office automation. Code is available at https://github.com/FairmeHIT/Multi-VIE, and fine-tuned models at https://huggingface.co/fairme/Qwen2.5-VL-7B-SFT.

Figures

Figures reproduced from arXiv: 2607.22723 by Guo Chen, Huafu Li, Jia Xia, Lei Wang, Liming Li, Wei Du, Weijun Peng, Yun Yao.

Figure 1
Figure 1. Figure 1: Overview of the classification-guided LVLM framework for multi-VIE. The design leverages a parameter-frozen LVLM to support minimally supervised, scalable deployment. Option 1 (training-free) classifies images via feature similarity matching. Features are extracted from the input image using a pre-trained model, then matched against a reference feature index with cosine similarity, thresholding, and majori… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the dynamic prompt engineering process, showing shared instructions, type-specific landmarks, and in-context demonstrations. ICL-Based Prompt Engineering Once the document type is determined, we dynamically assemble a concise, task-specific prompt by combining a shared instruction block with type-specific components, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the OCR&UIE baseline pipeline, incorporating a keyword-based classifier, pre-trained OCR, and per-type UIE models. Dataset Public benchmarks such as SROIE39 and SCID40 are limited in document variety, layout complexity, and entity types, and may have been exposed during large vision-language model pretraining. To enable rigorous evaluation under realistic conditions, we construct a new domain-s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 7 canonical work pages

  1. [1]

    & Xiao, R

    Yu, W., Lu, N., Qi, X., Gong, P. & Xiao, R. PICK: Processing key information extraction from documents using improved graph learning-convolutional networks. In2020 25th International Conference on Pattern Recognition, 4363–4370, DOI: 10.1109/ICPR48806.2021.9412927 (2021)

    arXiv 2021

  2. [2]

    1016/j.neucom.2015.09.116 (2016)

    Guo, Y .et al.Deep learning for visual understanding: A review.Neurocomputing187, 27–48, DOI: https://doi.org/10. 1016/j.neucom.2015.09.116 (2016). 12/14

    2015

  3. [3]

    Intell.55, 311, DOI: https://doi.org/10.1007/s10489-024-05937-6 (2025)

    Ren, Y .et al.Tablegpt: a novel table understanding method based on table recognition and large language model collaborative enhancement.Appl. Intell.55, 311, DOI: https://doi.org/10.1007/s10489-024-05937-6 (2025)

    work page doi:10.1007/s10489-024-05937-6 2025

  4. [4]

    5.Lu, Y .et al.Unified structure generation for universal information extraction.arXiv preprint arXiv:2203.12277(2022)

    Wei, H.et al.General OCR theory: Towards OCR-2.0 via a unified end-to-end model.arXiv e-printsarXiv–2409 (2024). 5.Lu, Y .et al.Unified structure generation for universal information extraction.arXiv preprint arXiv:2203.12277(2022)

    Pith/arXiv arXiv 2024

  5. [6]

    InInternational Conference on Document Analysis and Recognition, 36–53, DOI: https://doi.org/10.1007/978-3-031-41731-3_3 (2023)

    Kuang, J.et al.Visual information extraction in the wild: practical dataset and end-to-end solution. InInternational Conference on Document Analysis and Recognition, 36–53, DOI: https://doi.org/10.1007/978-3-031-41731-3_3 (2023)

    work page doi:10.1007/978-3-031-41731-3_3 2023

  6. [7]

    InProceedings of the AAAI Conference on Artificial Intelligence, vol

    Wang, J.et al.Towards robust visual information extraction in real world: new dataset and novel solution. InProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2738–2745, DOI: https://doi.org/10.1609/aaai.v35i4.16378 (2021)

    work page doi:10.1609/aaai.v35i4.16378 2021

  7. [8]

    & Luo, X

    Zhang, J., Wang, H. & Luo, X. Dual-VIE: Dual-level graph attention network for visual information extraction. InPacific Rim International Conference on Artificial Intelligence, 422–434, DOI: https://doi.org/10.1007/978-3-031-20862-1_31 (2022)

    work page doi:10.1007/978-3-031-20862-1_31 2022

  8. [9]

    Shi, Y .et al.Exploring OCR capabilities of GPT-4V (ision): A quantitative and in-depth evaluation.arXiv preprint arXiv:2310.16809(2023)

    Pith/arXiv arXiv 2023

  9. [10]

    Wang, P.et al.Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  10. [11]

    He, Z.et al.Seeing is believing? mitigating ocr hallucinations in multimodal large language models.arXiv preprint arXiv:2506.20168(2025)

    arXiv 2025

  11. [12]

    & Zhang, C

    Chen, Q., Zhang, X., Guo, L., Chen, F. & Zhang, C. Dianjin-ocr-r1: Enhancing ocr capabilities via a reasoning-and-tool interleaved vision-language model.arXiv preprint arXiv:2508.13238(2025)

    arXiv 2025

  12. [13]

    In2013 12th International Conference on Document Analysis and Recognition, 101–105, DOI: 10.1109/ICDAR.2013.28 (2013)

    Schuster, D.et al.Intellix–end-user trained information extraction for document archiving. In2013 12th International Conference on Document Analysis and Recognition, 101–105, DOI: 10.1109/ICDAR.2013.28 (2013). 14.Katti, A. R.et al.Chargrid: Towards understanding 2D documents.arXiv preprint arXiv:1809.08799(2018)

    Pith/arXiv arXiv 2013

  13. [15]

    Denk, T. I. & Reisswig, C. Bertgrid: Contextualized embedding for 2D document representation and understanding.arXiv preprint arXiv:1909.04948(2019)

    Pith/arXiv arXiv 1909

  14. [16]

    InDocument Analysis and Recognition, 548–563 (2021)

    Lin, W.et al.Vibertgrid: a jointly trained multi-modal 2D document representation for key information extraction from documents. InDocument Analysis and Recognition, 548–563 (2021). 17.Qian, Y ., Santus, E., Jin, Z., Guo, J. & Barzilay, R. Graphie: A graph-based framework for information extraction.arXiv preprint arXiv:1810.13083(2018)

    Pith/arXiv arXiv 2021

  15. [18]

    Tang, G.et al.Matchvie: Exploiting match relevancy between entities for visual information extraction.arXiv preprint arXiv:2106.12940(2021)

    Pith/arXiv arXiv 2021

  16. [19]

    & Zhang, J

    Zhang, Z., Ma, J., Du, J., Wang, L. & Zhang, J. Multimodal pre-training based on graph attention network for document understanding.IEEE Transactions on Multimed.25, 6743–6755, DOI: https://doi.org/10.1109/TMM.2022.3214102 (2022)

    arXiv 2022

  17. [20]

    & Wei, F

    Huang, Y ., Lv, T., Cui, L., Lu, Y . & Wei, F. Layoutlmv3: Pre-training for document ai with unified text and image masking. InProceedings of the 30th ACM International Conference on Multimedia(2022)

    2022

  18. [21]

    InProceedings of the 29th ACM International Conference on Multimedia, 1912–1920, DOI: https://doi.org/10.1145/3474085.34753 (2021)

    Li, Y .et al.Structext: Structured text understanding with multi-modal transformers. InProceedings of the 29th ACM International Conference on Multimedia, 1912–1920, DOI: https://doi.org/10.1145/3474085.34753 (2021)

    arXiv 1912

  19. [22]

    Neural Inf

    Gu, J.et al.Unidoc: Unified pretraining framework for document understanding.Adv. Neural Inf. Process. Syst.34, 39–50 (2021)

    2021

  20. [23]

    Xu, Y .et al.Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding.arXiv preprint arXiv:2104.08836(2021)

    Pith/arXiv arXiv 2021

  21. [24]

    & Ding, K

    Wang, J., Jin, L. & Ding, K. Lilt: A simple yet effective language-independent layout transformer for structured document understanding.arXiv preprint arXiv:2202.13669(2022)

    Pith/arXiv arXiv 2022

  22. [25]

    In2019 International Conference on Document Analysis and Recognition (ICDAR), 254–259, DOI: 10.1109/ICDAR.2019.00049 (2019)

    Guo, H.et al.Eaten: Entity-aware attention for single shot visual text extraction. In2019 International Conference on Document Analysis and Recognition (ICDAR), 254–259, DOI: 10.1109/ICDAR.2019.00049 (2019)

    arXiv 2019

  23. [26]

    InProceedings of the 28th ACM International Conference on Multimedia, 1413–1422, DOI: https://doi.org/10.1145/3394171.341390 (2020)

    Zhang, P.et al.TRIE: end-to-end text reading and information extraction for document understanding. InProceedings of the 28th ACM International Conference on Multimedia, 1413–1422, DOI: https://doi.org/10.1145/3394171.341390 (2020). 13/14

    arXiv 2020

  24. [27]

    InEuropean Conference on Computer Vision, 498–517, DOI: https://doi.org/10.1007/978-3-031-19815-1_29 (2022)

    Kim, G.et al.OCR-free document understanding transformer. InEuropean Conference on Computer Vision, 498–517, DOI: https://doi.org/10.1007/978-3-031-19815-1_29 (2022)

    doi:10.1007/978-3-031-19815-1_29 2022

  25. [28]

    & Lin, W

    Cheng, M., Qiu, M., Shi, X., Huang, J. & Lin, W. One-shot text field labeling using attention and belief propagation for structure information extraction. InProceedings of the 28th ACM International Conference on Multimedia, 340–348, DOI: https://doi.org/10.1145/3394171.341351 (2020)

    arXiv 2020

  26. [29]

    & Shang, J

    Wang, Z. & Shang, J. Towards few-shot entity recognition in document images: a label-aware sequence-to-sequence framework.arXiv preprint arXiv:2204.05819(2022)

    Pith/arXiv arXiv 2022

  27. [30]

    & Feng, Y

    Chen, F. & Feng, Y . Chain-of-thought prompt distillation for multimodal named entity and multimodal relation extraction. arXiv preprint arXiv:2306.14122(2023)

    Pith/arXiv arXiv 2023

  28. [31]

    InFindings of the Association for Computational Linguistics: EMNLP 2023, 2969–2979, DOI: 10.18653/v1/2023.findings-emnlp.196 (2023)

    Cai, C.et al.In-context learning for few-shot multimodal named entity recognition. InFindings of the Association for Computational Linguistics: EMNLP 2023, 2969–2979, DOI: 10.18653/v1/2023.findings-emnlp.196 (2023)

    work page doi:10.18653/v1/2023.findings-emnlp.196 2023

  29. [32]

    Peng, K.et al.Mitigating label noise using prompt-based hyperbolic meta-learning in open-set domain generalization.Int. J. Comput. Vis.134, DOI: 10.1007/s11263-025-02643-9 (2026)

    work page doi:10.1007/s11263-025-02643-9 2026

  30. [33]

    In Ku, L.-W., Martins, A

    Wang, D.et al.DocLLM: A layout-aware generative language model for multimodal document understanding. In Ku, L.-W., Martins, A. & Srikumar, V . (eds.)Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8529–8548, DOI: 10.18653/v1/2024.acl-long.463 (Association for Computational Linguistics, Bang...

    doi:10.18653/v1/2024.acl-long.463 2024

  31. [34]

    Liu, Y .et al.Textmonkey: An ocr-free large multimodal model for understanding document.IEEE transactions on pattern analysis machine intelligence(2026)

    2026

  32. [35]

    In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26753–26763, DOI: 10.1109/CVPR52733.2024.02527 (2024)

    Li, Z.et al.Monkey: Image resolution and text label are important things for large multi-modal models. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26753–26763, DOI: 10.1109/CVPR52733.2024.02527 (2024)

    arXiv 2024

  33. [36]

    Li, Z.et al.Monkeyocr: Document parsing with a structure-recognition-relation triplet paradigm.arXiv preprint arXiv:2506.05218(2025)

    arXiv 2025

  34. [37]

    Tan, M. & Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational conference on machine learning, 6105–6114, DOI: 10.48550/arXiv.1905.11946 (2019)

    Pith/arXiv arXiv doi:10.48550/arxiv.1905.11946 1905

  35. [38]

    In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11966–11976, DOI: 10.1109/CVPR52688.2022.01167 (2022)

    Liu, Z.et al.A convnet for the 2020s. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11966–11976, DOI: 10.1109/CVPR52688.2022.01167 (2022)

    arXiv 2022

  36. [39]

    In2019 International Conference on Document Analysis and Recognition (ICDAR), 1516–1520, DOI: 10.1109/ICDAR.2019.00244 (2019)

    Huang, Z.et al.ICDAR2019 competition on scanned receipt OCR and information extraction. In2019 International Conference on Document Analysis and Recognition (ICDAR), 1516–1520, DOI: 10.1109/ICDAR.2019.00244 (2019)

    arXiv 2019

  37. [40]

    Qiao, L., Li, Z., Cheng, Z. & Li, X. SCID: a chinese characters invoice-scanned dataset in relevant to key information extraction derived of visually-rich document images.J. Image Graph.28, 2298–2313, DOI: 10.11834/jig.220911 (2023)

    work page doi:10.11834/jig.220911 2023

  38. [41]

    Li, C.et al.Pp-OCRv3: More attempts for the improvement of ultra lightweight OCR system.arXiv preprint arXiv:2206.03001(2022)

    Pith/arXiv arXiv 2022

  39. [42]

    Paddlelabel, an effective and flexible tool for data annotation

    Authors, P. Paddlelabel, an effective and flexible tool for data annotation. https://github.com/PaddleCV-SIG/PaddleLabel (2022)

    2022

  40. [43]

    & Liang, X

    Nakayama, H., Kubo, T., Kamura, J., Taniguchi, Y . & Liang, X. doccano: Text annotation tool for human (2018). Software available from https://github.com/doccano/doccano

    2018

  41. [44]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16133–16142, DOI: 10.1109/CVPR52729.2023.01548 (2023)

    Woo, S.et al.Convnext V2: Co-designing and scaling convnets with masked autoencoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16133–16142, DOI: 10.1109/CVPR52729.2023.01548 (2023). 14/14

    arXiv 2023