Pith. sign in

REVIEW 4 major objections 6 minor 18 references

Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read BLOCKIE claims that splitting visually rich documents into self-contained semantic blocks lets LLM-based extraction reach state-of-the-art F1 on CORD, FUNSD, and SROIE while also generalizing to unseen formats and inferring absent values.

desk verdict Genuinely new block-based LLM pipeline for document IE with credible F1 gains, but the central causal claim about block self-containment is never directly tested and the combiner's full-document access muddies the attribution. read the letter →

arxiv 2505.13535 v1 pith:HKK3J4QO submitted 2025-05-18 cs.IR cs.AI

classification cs.IRcs.AI
keywords informationextractionvisuallyrichdocumentssemanticblockslargelanguagemodelslayoutunderstandingreceiptparsingformvalue-absentinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BLOCKIE is an information-extraction method for visually rich documents that first cuts a document's OCR text into self-contained semantic blocks, parses each block independently, and then lets an LLM combine the partial parses into the requested schema. The paper claims this block-first organization is what drives accuracy, not the raw power of the language model: with a 32-billion-parameter open model it beats direct few-shot use of a much larger proprietary model, and with a 7-billion-parameter model it far surpasses comparably sized layout-aware fine-tuned models. On the public benchmarks CORD, FUNSD, and SROIE, BLOCKIE reports 98.83%, 92.15%, and 98.52% F1 respectively, 1 to 3 points above prior state-of-the-art methods. The same organization is claimed to make extraction resilient to new document formats and able to infer values that are not explicitly printed, such as counting line items on a receipt.

What carries the argument

The load-bearing object is the semantic block, defined by the self-containment identity v(B,B)=v(B,D): parsing the block alone yields exactly the values it would yield with the whole document as context. Blocks are assembled from semantic atoms, indivisible spatially coherent text fragments, connected by attribute-value and hierarchical linkages; a block is valid when all linkages for its atoms are internal to it. This converts a single hard whole-document parsing problem into many smaller, format-agnostic ones, which is what lets similar blocks be matched across different templates and lets a relatively small LLM outperform a much larger one used directly.

What would settle it

Run BLOCKIE on the same receipts with deliberately broken block boundaries, for example placing every attribute and value in separate blocks, and check final F1; if extraction stays near 98.83%, the self-containment premise is not essential, while a large drop would confirm that it is.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the context needed to extract an entity from a visually rich document is usually local: a small group of spatially and semantically linked tokens contains everything required to read that entity correctly. It formalizes this as the semantic block, a segment B satisfying v(B,B)=v(B,D), meaning the values extracted from B alone equal the values extracted when B is considered with the whole document as context. Semantic blocks are built from semantic atoms, indivisible spatially coherent text units, joined by attribute-value or hierarchy linkages, and a block is valid only when every linkage of every atom inside it is also inside it. BLOCKIE then runs three LLM stages: block creation, independent block parsing with similar-block few-shot examples, and a combining judge that reconstructs the full schema from partial parses. The paper attributes the reported state-of-the-art accuracy, resilience to unseen formats, and value-absent inference to this decomposition rather than to the choice or size of the underlying LLM.

Load-bearing premise

The pipeline assumes the block creator reliably splits unseen documents into self-contained blocks, because any linkage that crosses a block boundary is missed and cannot be fully recovered at the combination stage.

Editorial extensions

If this is right

  • On CORD, FUNSD, and SROIE, BLOCKIE reports F1 of 98.83%, 92.15%, and 98.52%, beating the previous best methods by 1 to 3 points, so block-level organization is a competitive alternative to layout-aware pretraining for visually rich document extraction.
  • Model scale stops being the main driver: BLOCKIE with Qwen 2.5 32B reaches 96.14% F1 on CORD, above direct few-shot Sonnet at 95.72% and LMDX-Gemini Pro at 95.57%, while the 7B variant reaches 87.72%, far above similarly sized fine-tuned DocLLM and LayoutLLM.
  • With only 100 format-diverse training samples, BLOCKIE retains 94.47% F1 on CORD where LayoutLMV3 falls to 78.79%, and a CORD-trained model transfers to SROIE total-amount extraction at 97.06%, so format heterogeneity and unseen templates are handled better.
  • Value-absent inference is demonstrated: on 20 sampled CORD receipts, BLOCKIE infers the correct line-item count 18 times, including cases requiring subtraction and multi-step addition.
  • Block creation quality is the main performance bottleneck: replacing test blocks with ground-truth blocks lets a 7B parser reach 94.38% F1, close to the 96.14% of the 32B model, so further gains should come from better block boundaries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The self-containment condition suggests a direct diagnostic: measure how often parsing a block alone disagrees with parsing it with full-document context on a held-out set; if such disagreements are common in blocks the creator judged valid, the claimed bottleneck is even more critical than the F1 numbers show.
  • Because block quality is the binding constraint, a dedicated lightweight model trained to predict block boundaries from the block creator's outputs could reduce the cost and latency of the current sequential pipeline while preserving accuracy.
  • The same decompose-then-combine pattern might transfer to other structured layout tasks, such as table extraction or document question answering, where local cell and header linkages play the role of semantic atoms.
  • The value-absent-inference claim rests on a 20-sample evaluation; running the same counting task on the full CORD test set would show how often the claimed reasoning survives at scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. BLOCKIE is an LLM-based information extraction system for visually rich documents. The paper defines semantic blocks as localized segments for which parsing in isolation equals parsing with full-document context (Eq. 2), and proposes a three-stage pipeline: an LLM block creator segments OCR text into blocks using the document schema and retrieved few-shot demonstrations; an LLM block parser annotates each block independently; and an LLM combiner receives the full OCR text, bounding boxes, and partial parses to assemble the document-level output. Training labels are converted into block-level reasoning examples. The authors report state-of-the-art F1 on CORD (98.83), FUNSD entity linking (92.15), and SROIE (98.52) with Claude 3.5 Sonnet, and consistent results with Qwen 2.5 models from 7B to 72B. Additional experiments cover format heterogeneity, cross-dataset transfer, the effect of block creation quality, and value-absent inference. The paper acknowledges latency, dependence on LLM reasoning, and block-creation errors as limitations.

Significance. The reported results, if reproducible, are a significant practical advance for document IE: BLOCKIE outperforms published layout-aware and LLM-based methods on three standard benchmarks and shows that a 32B open model can beat much larger proprietary models when combined with block-level organization. The paper's strengths include held-out test evaluation, a broad LLM sweep, an informative ground-truth-block ablation (Table 5), a 100-sample heterogeneity test, and a cross-dataset transfer experiment. The main gaps are that the central causal claim - that self-contained semantic blocks, rather than the full-context combiner, drive the gains - is not directly tested, and that baseline comparisons rely on published numbers with no statistical significance information. These gaps are addressable and do not invalidate the empirical contribution, but they currently temper the strength of the paper's theoretical and causal claims.

major comments (4)
  1. [§5.2.1, Table 1] The comparison in Table 1 is against published F1 values rather than baselines re-run under BLOCKIE's exact OCR, prompting, and schema settings; for generative LLM baselines in particular, differences of 1-3% can easily be within prompt-engineering or OCR variation. Please re-run the main competitors under identical conditions, or state clearly which numbers are quoted from prior papers and which are measured in this work.
  2. [§3, Eq. (2), and §5.2.4] The defining property of a semantic block, v(B,B)=v(B,D), is never measured directly. The block-level F1 in Table 4 compares created blocks with ground-truth blocks generated from labels, which tests boundary agreement rather than context independence. Since the combiner prompt in Table 9 supplies all document words and instructs the model to use the complete document text, the combiner can in principle repair any context-dependence of the blocks; the Limitations section even concedes that missed linkages are hard to recover, but no experiment quantifies this failure mode. A direct test is required: parse each created block in isolation and with full-document context, and measure the divergence; or ablate the combiner's access to full text to quantify how much end-to-end F1 depends on the final full-context pass.
  3. [§5.2.2, Tables 1-2] The paper reports single-point F1 values with no variance or significance information. Differences such as BLOCKIE with Qwen-32B (96.14) versus Qwen-72B (96.01) are likely not meaningful without confidence intervals; please report means and standard deviations over repeated runs or bootstrap intervals over the test sets for the headline results.
  4. [§3, semantic atoms and Eq. (2)] The statement that a collection of semantic atoms containing all of its linkages is 'a sufficient condition for equation 2' is presented as a mathematical claim but no proof is given. Because Eq. (2) is the formal basis for the method, the authors should either prove the implication under their definitions or explicitly demote it to a heuristic motivation.
minor comments (6)
  1. [§5.2.5] The cross-reference 'Figure 5.2.5' is broken; it should point to the actual figure showing the inference examples.
  2. [§5.2.3] The phrase 'the enity total amount' should read 'the entity total amount'.
  3. [Appendix B.1] The text 'FUNSD Datatet' should read 'FUNSD Dataset'.
  4. [References] The entry for Harley et al. is incomplete (missing year and venue), and the CORD reference has a formatting issue in the author list ('V Y'); please fix the bibliography.
  5. [Table 5] The caption says 'Semantic Block F1-scores' but the table reports end-to-end entity F1 under different block sources; the caption should be corrected to match the content.
  6. [§5.2.5] The value-absent inference experiment uses only 20 hand-selected cases with no stated selection protocol or error analysis; please document how the 20 cases were chosen and include a failure analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BLOCKIE's headline F1 results are held-out benchmark evaluations, and its internal bottleneck claims are tested by oracle ablations rather than by construction.

full rationale

The paper's central empirical claims are F1 scores on held-out test sets of CORD, FUNSD, and SROIE, computed against published baselines; no test labels are used to fit the method's prompts or parameters. The train-labeling step uses ground-truth labels to construct few-shot demonstrations for block creation and parsing, which is standard in-context learning and does not force the held-out predictions. The claim that block creation is the performance bottleneck is supported by measured correlations (Table 4) and by an oracle experiment in which test blocks are replaced with ground-truth blocks (Table 5), so the bottleneck claim is tested rather than assumed. Equation (2) defines a semantic block via v(B,B)=v(B,D), and the paper states a sufficient practical condition: 'all linkages for each atom in the collection is present inside the collection itself.' This is a definitional framework with an acknowledged sufficient condition, not an empirical result derived from itself; the paper does not claim to have proven that every LLM-created block satisfies Eq. (2), and indeed the Limitations section concedes that 'missed linkages can be hard to recover,' which is a correctness risk rather than circularity. The only self-citation, Bhattacharyya and Tripathi (2024), appears in a general related-work remark about LLaVa/CogVLM and is not load-bearing for the method's claims. Overall, the derivation chain is self-contained against external benchmarks, and no prediction reduces by construction to its inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 2 invented entities

The method contributes a prompting pipeline rather than a mathematical derivation; the main components pulled from prior literature are the LLMs themselves (Claude, Qwen) and the benchmark datasets. The few-shot count is the only obvious hand-set hyperparameter. The conceptual constructs (blocks, atoms) are not independently evidenced.

free parameters (1)
  • few_shot_examples = 5
    The paper states 'We used 5 few-shot examples in the prompts for both block creator and parser' and reports tuning the number for Sonnet on the CORD validation set; the same count is used for all models and datasets without per-model tuning.
assumptions (3)
  • domain assumption A collection of semantic atoms containing all attribute:value and hierarchical linkages internally is sufficient for the independence condition v(B,B)=v(B,D) (equation 2).
    Stated in Section 3 without proof; the paper relies on it to justify independent block parsing.
  • domain assumption In-context learning with 5 few-shot examples selected by cosine similarity of OCR text generalizes across document formats and schema variations.
    The method's cross-format claims depend on this transfer assumption; no analysis of similarity thresholds or failure cases is provided.
  • domain assumption The OCR text and bounding boxes from CORD, FUNSD, and SROIE are accurate enough for block creation and parsing.
    The pipeline consumes OCR output directly; errors in OCR would propagate.
invented entities (2)
  • semantic block
    purpose: A self-contained group of text tokens used as the unit of independent LLM parsing.
    No external falsifiable handle; it is a methodological construct evaluated only through downstream F1 and block-level agreement with ground-truth blocks.
  • semantic atom
    purpose: An indivisible visual region of text that forms a complete semantic unit, used to build blocks.
    Defined operationally; no independent measurement outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments." pith.science (2026). https://pith.science/paper/HKK3J4QO

@misc{pith2026250513535,
  author       = {Pith},
  title        = {Pith review of: Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKK3J4QO}},
  note         = {Machine review of arXiv:2505.13535}
}
abstract

Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLM-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called $\textit{semantic blocks}$, which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.

Figures

Figures reproduced from arXiv: 2505.13535 by the authors.

Figure 1
Figure 1. The Information Extraction Task, illustrated using sample images from ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Sample image with document schema and value In this section, we define the concept of semantic blocks, and show how these are created practically. Let us consider a set of documents D with a common set of hierarchical entities of interest E, which we refer to as the document schema. Let V denote the set of all possible instantiations of E. Given a document D ∈ D, let VE(D) ∈ V denote the actual values of the entitie… view at source ↗
Figure 3
Figure 3. Sample image with document schema and value In other words, a semantic block must be inter￾pretable independently without any additional con￾text - the values extracted from B in isolation must match those extracted with full document context. To illustrate, consider [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustrative flow with a simulated receipt and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Two documents with different formats (a fax [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 5
Figure 5. Figure 5: Motivating example for the conceptualization [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Block Parser on Figure [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Some challenging inferences made by BLOCKIE. In test_30, the single line item does not have a quantity mentioned. In test_29, the LLM has to reason to leave out sub-items from the count. In test_20, it has to perform a multi-step addition. datasets. The framework is de…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 14 canonical work pages

  1. [1]

    A LL THE WORDS IN THE DOCUMENT

  2. [2]

    U TILIZE THE PARTIAL PARSES FROM BLOCKS TO HELP CONSTRUCT THE FINAL DICTIONARY - REMEMBER - THE PARTIAL PARSES MAY NOT HAVE FULL CONTEXT

  3. [3]

    I NDIVIDUAL BLOCKS OF TEXT AND THEIR PARTIAL PARSES

  4. [4]

    U SE THESE TO HELP CONSTRUCT THE COMPLETE DICTIONARY : {blocks_and_parses} INSTRUCTIONS:

    T HE REQUIRED DICTIONARY SCHEMA <SCHEMA AND FIELD DESCRIPTIONS> <Linked and Hierarchical entity identification rules> ALL WORDS IN THE DOCUMENT: {text} ALL BOUNDING BOXES IN THE DOCUMENT: {bboxes} PARSED BLOCKS: BELOW ARE THE INDIVIDUAL BLOCKS AND THEIR PARTIAL PARSES ALONG WITH REASON . U SE THESE TO HELP CONSTRUCT THE COMPLETE DICTIONARY : {blocks_and_p...

  5. [11]

    U SE THE COMPLETE DOCUMENT TEXT TO UNDERSTAND THE FULL CONTEXT

  6. [13]

    E NSURE ALL INFORMATION IS CORRECTLY CATEGORIZED ACCORDING TO THE SCHEMA

  7. [14]

    M AINTAIN CONSISTENCY WITH NUMERICAL FORMATS FROM THE ORIGINAL TEXT <Verification Process> YOUR FINAL DICTIONARY SHOULD CONTAIN TWO KEYS :

  8. [15]

    U SE THE REASON FROM PARTIAL PARSES , CHECK IF IT MENTIONS EXACT MATCH

    REASON - JUSTIFY STEP BY STEP WHY YOU CHOSE PARTICULAR VALUES . U SE THE REASON FROM PARTIAL PARSES , CHECK IF IT MENTIONS EXACT MATCH

Show all 18 references
  1. [16]

    INVOICE - SHARE THE INVOICE DICTIONARY RETURN ONLY THE FINAL JSON DICTIONARY WITHOUT ANY ADDITIONAL EXPLANATION WITH PROPER FORMAT . B Datasets and Benchmarks B.1 Datasets CORD Dataset CORD (Park et al., 2019) con- tains 1000 Indonesian receipts, divided into train, validation...

  2. [18]

    It incorporates layout information us- ing cross-attention between bounding boxes and text, and through masked image modeling

    is a state-of-the-art information extraction benchmark. It incorporates layout information us- ing cross-attention between bounding boxes and text, and through masked image modeling. It shows competitive performance on all three benchmark datasets. Note that while (Luo et al.,...

  3. [100]

    30 hierarchical entities are annotated manually under top-level entities menu, subtotal and total

    Along with the images, CORD also contains crowdsourced labels, and OCR output with bound- ing boxes. 30 hierarchical entities are annotated manually under top-level entities menu, subtotal and total. The associated task is to assign the words in the OCR output to these entitie...

  4. [2013]

    Association for Computational Linguistics

    Rule-based information extraction is dead! long live rule-based information extraction systems! In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing , pages 827–832, Seattle, Washington, USA. Association for Computational Linguistics. Tim...

  5. [2016]

    In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 287–292

    A table detection method for pdf documents based on convolutional neural networks. In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 287–292. Adam W Harley, Alex Ufkes, and Konstantinos G Der- panis. Evaluation of deep convolutional nets for document image c...

  6. [2017]

    Preprint, arXiv:1708.07403

    Cloudscan - a configuration-free invoice analy- sis system using recurrent neural networks. Preprint, arXiv:1708.07403. Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsi...

  7. [2019]

    In 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE. Wonseok Hwang, Jinyeong Yim, Seunghyun Park, So- hee Yang, and Minjoon Seo. 2021. Spatial depen- dency parsing for semi...

  8. [2020]

    Preprint, arXiv:2002.12804

    Unilmv2: Pseudo-masked language models for unified language model pre-training. Preprint, arXiv:2002.12804. Aniket Bhattacharyya and Anurag Tripathi. 2024. In- formation extraction from heterogeneous documents without ground truth labels using synthetic label generation and kn...

  9. [2022]

    {query_reason}

    Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document understand- ing. Preprint, arXiv:2210.06155. Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Zifeng Wang, Jiaqi Mu, Hao Zhang, Chen-Yu Lee, and Na...

  10. [2024]

    Preprint, arXiv:2401.10825

    Recent advances in named entity recogni- tion: A comprehensive survey and comparative study. Preprint, arXiv:2401.10825. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. A systematic study and comprehe...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.