REVIEW 4 major objections 6 minor 18 references
Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read BLOCKIE claims that splitting visually rich documents into self-contained semantic blocks lets LLM-based extraction reach state-of-the-art F1 on CORD, FUNSD, and SROIE while also generalizing to unseen formats and inferring absent values.
desk verdict Genuinely new block-based LLM pipeline for document IE with credible F1 gains, but the central causal claim about block self-containment is never directly tested and the combiner's full-document access muddies the attribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the semantic block, defined by the self-containment identity v(B,B)=v(B,D): parsing the block alone yields exactly the values it would yield with the whole document as context. Blocks are assembled from semantic atoms, indivisible spatially coherent text fragments, connected by attribute-value and hierarchical linkages; a block is valid when all linkages for its atoms are internal to it. This converts a single hard whole-document parsing problem into many smaller, format-agnostic ones, which is what lets similar blocks be matched across different templates and lets a relatively small LLM outperform a much larger one used directly.
What would settle it
Run BLOCKIE on the same receipts with deliberately broken block boundaries, for example placing every attribute and value in separate blocks, and check final F1; if extraction stays near 98.83%, the self-containment premise is not essential, while a large drop would confirm that it is.
Extended reading notes
Core claim
The paper's central discovery is that the context needed to extract an entity from a visually rich document is usually local: a small group of spatially and semantically linked tokens contains everything required to read that entity correctly. It formalizes this as the semantic block, a segment B satisfying v(B,B)=v(B,D), meaning the values extracted from B alone equal the values extracted when B is considered with the whole document as context. Semantic blocks are built from semantic atoms, indivisible spatially coherent text units, joined by attribute-value or hierarchy linkages, and a block is valid only when every linkage of every atom inside it is also inside it. BLOCKIE then runs three LLM stages: block creation, independent block parsing with similar-block few-shot examples, and a combining judge that reconstructs the full schema from partial parses. The paper attributes the reported state-of-the-art accuracy, resilience to unseen formats, and value-absent inference to this decomposition rather than to the choice or size of the underlying LLM.
Load-bearing premise
The pipeline assumes the block creator reliably splits unseen documents into self-contained blocks, because any linkage that crosses a block boundary is missed and cannot be fully recovered at the combination stage.
Editorial extensions
If this is right
- On CORD, FUNSD, and SROIE, BLOCKIE reports F1 of 98.83%, 92.15%, and 98.52%, beating the previous best methods by 1 to 3 points, so block-level organization is a competitive alternative to layout-aware pretraining for visually rich document extraction.
- Model scale stops being the main driver: BLOCKIE with Qwen 2.5 32B reaches 96.14% F1 on CORD, above direct few-shot Sonnet at 95.72% and LMDX-Gemini Pro at 95.57%, while the 7B variant reaches 87.72%, far above similarly sized fine-tuned DocLLM and LayoutLLM.
- With only 100 format-diverse training samples, BLOCKIE retains 94.47% F1 on CORD where LayoutLMV3 falls to 78.79%, and a CORD-trained model transfers to SROIE total-amount extraction at 97.06%, so format heterogeneity and unseen templates are handled better.
- Value-absent inference is demonstrated: on 20 sampled CORD receipts, BLOCKIE infers the correct line-item count 18 times, including cases requiring subtraction and multi-step addition.
- Block creation quality is the main performance bottleneck: replacing test blocks with ground-truth blocks lets a 7B parser reach 94.38% F1, close to the 96.14% of the 32B model, so further gains should come from better block boundaries.
Reading between the lines
- The self-containment condition suggests a direct diagnostic: measure how often parsing a block alone disagrees with parsing it with full-document context on a held-out set; if such disagreements are common in blocks the creator judged valid, the claimed bottleneck is even more critical than the F1 numbers show.
- Because block quality is the binding constraint, a dedicated lightweight model trained to predict block boundaries from the block creator's outputs could reduce the cost and latency of the current sequential pipeline while preserving accuracy.
- The same decompose-then-combine pattern might transfer to other structured layout tasks, such as table extraction or document question answering, where local cell and header linkages play the role of semantic atoms.
- The value-absent-inference claim rests on a 20-sample evaluation; running the same counting task on the full CORD test set would show how often the claimed reasoning survives at scale.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. BLOCKIE is an LLM-based information extraction system for visually rich documents. The paper defines semantic blocks as localized segments for which parsing in isolation equals parsing with full-document context (Eq. 2), and proposes a three-stage pipeline: an LLM block creator segments OCR text into blocks using the document schema and retrieved few-shot demonstrations; an LLM block parser annotates each block independently; and an LLM combiner receives the full OCR text, bounding boxes, and partial parses to assemble the document-level output. Training labels are converted into block-level reasoning examples. The authors report state-of-the-art F1 on CORD (98.83), FUNSD entity linking (92.15), and SROIE (98.52) with Claude 3.5 Sonnet, and consistent results with Qwen 2.5 models from 7B to 72B. Additional experiments cover format heterogeneity, cross-dataset transfer, the effect of block creation quality, and value-absent inference. The paper acknowledges latency, dependence on LLM reasoning, and block-creation errors as limitations.
Significance. The reported results, if reproducible, are a significant practical advance for document IE: BLOCKIE outperforms published layout-aware and LLM-based methods on three standard benchmarks and shows that a 32B open model can beat much larger proprietary models when combined with block-level organization. The paper's strengths include held-out test evaluation, a broad LLM sweep, an informative ground-truth-block ablation (Table 5), a 100-sample heterogeneity test, and a cross-dataset transfer experiment. The main gaps are that the central causal claim - that self-contained semantic blocks, rather than the full-context combiner, drive the gains - is not directly tested, and that baseline comparisons rely on published numbers with no statistical significance information. These gaps are addressable and do not invalidate the empirical contribution, but they currently temper the strength of the paper's theoretical and causal claims.
major comments (4)
- [§5.2.1, Table 1] The comparison in Table 1 is against published F1 values rather than baselines re-run under BLOCKIE's exact OCR, prompting, and schema settings; for generative LLM baselines in particular, differences of 1-3% can easily be within prompt-engineering or OCR variation. Please re-run the main competitors under identical conditions, or state clearly which numbers are quoted from prior papers and which are measured in this work.
- [§3, Eq. (2), and §5.2.4] The defining property of a semantic block, v(B,B)=v(B,D), is never measured directly. The block-level F1 in Table 4 compares created blocks with ground-truth blocks generated from labels, which tests boundary agreement rather than context independence. Since the combiner prompt in Table 9 supplies all document words and instructs the model to use the complete document text, the combiner can in principle repair any context-dependence of the blocks; the Limitations section even concedes that missed linkages are hard to recover, but no experiment quantifies this failure mode. A direct test is required: parse each created block in isolation and with full-document context, and measure the divergence; or ablate the combiner's access to full text to quantify how much end-to-end F1 depends on the final full-context pass.
- [§5.2.2, Tables 1-2] The paper reports single-point F1 values with no variance or significance information. Differences such as BLOCKIE with Qwen-32B (96.14) versus Qwen-72B (96.01) are likely not meaningful without confidence intervals; please report means and standard deviations over repeated runs or bootstrap intervals over the test sets for the headline results.
- [§3, semantic atoms and Eq. (2)] The statement that a collection of semantic atoms containing all of its linkages is 'a sufficient condition for equation 2' is presented as a mathematical claim but no proof is given. Because Eq. (2) is the formal basis for the method, the authors should either prove the implication under their definitions or explicitly demote it to a heuristic motivation.
minor comments (6)
- [§5.2.5] The cross-reference 'Figure 5.2.5' is broken; it should point to the actual figure showing the inference examples.
- [§5.2.3] The phrase 'the enity total amount' should read 'the entity total amount'.
- [Appendix B.1] The text 'FUNSD Datatet' should read 'FUNSD Dataset'.
- [References] The entry for Harley et al. is incomplete (missing year and venue), and the CORD reference has a formatting issue in the author list ('V Y'); please fix the bibliography.
- [Table 5] The caption says 'Semantic Block F1-scores' but the table reports end-to-end entity F1 under different block sources; the caption should be corrected to match the content.
- [§5.2.5] The value-absent inference experiment uses only 20 hand-selected cases with no stated selection protocol or error analysis; please document how the 20 cases were chosen and include a failure analysis.
Circularity Check
No significant circularity: BLOCKIE's headline F1 results are held-out benchmark evaluations, and its internal bottleneck claims are tested by oracle ablations rather than by construction.
full rationale
The paper's central empirical claims are F1 scores on held-out test sets of CORD, FUNSD, and SROIE, computed against published baselines; no test labels are used to fit the method's prompts or parameters. The train-labeling step uses ground-truth labels to construct few-shot demonstrations for block creation and parsing, which is standard in-context learning and does not force the held-out predictions. The claim that block creation is the performance bottleneck is supported by measured correlations (Table 4) and by an oracle experiment in which test blocks are replaced with ground-truth blocks (Table 5), so the bottleneck claim is tested rather than assumed. Equation (2) defines a semantic block via v(B,B)=v(B,D), and the paper states a sufficient practical condition: 'all linkages for each atom in the collection is present inside the collection itself.' This is a definitional framework with an acknowledged sufficient condition, not an empirical result derived from itself; the paper does not claim to have proven that every LLM-created block satisfies Eq. (2), and indeed the Limitations section concedes that 'missed linkages can be hard to recover,' which is a correctness risk rather than circularity. The only self-citation, Bhattacharyya and Tripathi (2024), appears in a general related-work remark about LLaVa/CogVLM and is not load-bearing for the method's claims. Overall, the derivation chain is self-contained against external benchmarks, and no prediction reduces by construction to its inputs.
Assumptions & free parameters
free parameters (1)
- few_shot_examples =
5
assumptions (3)
- domain assumption A collection of semantic atoms containing all attribute:value and hierarchical linkages internally is sufficient for the independence condition v(B,B)=v(B,D) (equation 2).
- domain assumption In-context learning with 5 few-shot examples selected by cosine similarity of OCR text generalizes across document formats and schema variations.
- domain assumption The OCR text and bounding boxes from CORD, FUNSD, and SROIE are accurate enough for block creation and parsing.
invented entities (2)
-
semantic block
-
semantic atom
Cite this review
Pith. "Pith review of Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments." pith.science (2026). https://pith.science/paper/HKK3J4QO
@misc{pith2026250513535,
author = {Pith},
title = {Pith review of: Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments},
year = {2026},
howpublished = {\url{https://pith.science/paper/HKK3J4QO}},
note = {Machine review of arXiv:2505.13535}
}
abstract
Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLM-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called $\textit{semantic blocks}$, which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
A LL THE WORDS IN THE DOCUMENT
-
[2]
U TILIZE THE PARTIAL PARSES FROM BLOCKS TO HELP CONSTRUCT THE FINAL DICTIONARY - REMEMBER - THE PARTIAL PARSES MAY NOT HAVE FULL CONTEXT
-
[3]
I NDIVIDUAL BLOCKS OF TEXT AND THEIR PARTIAL PARSES
-
[4]
U SE THESE TO HELP CONSTRUCT THE COMPLETE DICTIONARY : {blocks_and_parses} INSTRUCTIONS:
T HE REQUIRED DICTIONARY SCHEMA <SCHEMA AND FIELD DESCRIPTIONS> <Linked and Hierarchical entity identification rules> ALL WORDS IN THE DOCUMENT: {text} ALL BOUNDING BOXES IN THE DOCUMENT: {bboxes} PARSED BLOCKS: BELOW ARE THE INDIVIDUAL BLOCKS AND THEIR PARTIAL PARSES ALONG WITH REASON . U SE THESE TO HELP CONSTRUCT THE COMPLETE DICTIONARY : {blocks_and_p...
-
[11]
U SE THE COMPLETE DOCUMENT TEXT TO UNDERSTAND THE FULL CONTEXT
-
[13]
E NSURE ALL INFORMATION IS CORRECTLY CATEGORIZED ACCORDING TO THE SCHEMA
-
[14]
M AINTAIN CONSISTENCY WITH NUMERICAL FORMATS FROM THE ORIGINAL TEXT <Verification Process> YOUR FINAL DICTIONARY SHOULD CONTAIN TWO KEYS :
-
[15]
U SE THE REASON FROM PARTIAL PARSES , CHECK IF IT MENTIONS EXACT MATCH
REASON - JUSTIFY STEP BY STEP WHY YOU CHOSE PARTICULAR VALUES . U SE THE REASON FROM PARTIAL PARSES , CHECK IF IT MENTIONS EXACT MATCH
Show all 18 references
-
[16]
INVOICE - SHARE THE INVOICE DICTIONARY RETURN ONLY THE FINAL JSON DICTIONARY WITHOUT ANY ADDITIONAL EXPLANATION WITH PROPER FORMAT . B Datasets and Benchmarks B.1 Datasets CORD Dataset CORD (Park et al., 2019) con- tains 1000 Indonesian receipts, divided into train, validation...
2019
-
[18]
It incorporates layout information us- ing cross-attention between bounding boxes and text, and through masked image modeling
is a state-of-the-art information extraction benchmark. It incorporates layout information us- ing cross-attention between bounding boxes and text, and through masked image modeling. It shows competitive performance on all three benchmark datasets. Note that while (Luo et al.,...
2023
-
[100]
30 hierarchical entities are annotated manually under top-level entities menu, subtotal and total
Along with the images, CORD also contains crowdsourced labels, and OCR output with bound- ing boxes. 30 hierarchical entities are annotated manually under top-level entities menu, subtotal and total. The associated task is to assign the words in the OCR output to these entitie...
2019
-
[2013]
Association for Computational Linguistics
Rule-based information extraction is dead! long live rule-based information extraction systems! In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing , pages 827–832, Seattle, Washington, USA. Association for Computational Linguistics. Tim...
2013 arXiv
-
[2016]
In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 287–292
A table detection method for pdf documents based on convolutional neural networks. In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 287–292. Adam W Harley, Alex Ufkes, and Konstantinos G Der- panis. Evaluation of deep convolutional nets for document image c...
2016 arXiv
-
[2017]
Preprint, arXiv:1708.07403
Cloudscan - a configuration-free invoice analy- sis system using recurrent neural networks. Preprint, arXiv:1708.07403. Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsi...
2019 arXiv
-
[2019]
In 2019 International Conference on Document Analysis and Recognition (ICDAR)
Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE. Wonseok Hwang, Jinyeong Yim, Seunghyun Park, So- hee Yang, and Minjoon Seo. 2021. Spatial depen- dency parsing for semi...
2019 arXiv
-
[2020]
Preprint, arXiv:2002.12804
Unilmv2: Pseudo-masked language models for unified language model pre-training. Preprint, arXiv:2002.12804. Aniket Bhattacharyya and Anurag Tripathi. 2024. In- formation extraction from heterogeneous documents without ground truth labels using synthetic label generation and kn...
2002 arXiv
-
[2022]
{query_reason}
Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document understand- ing. Preprint, arXiv:2210.06155. Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Zifeng Wang, Jiaqi Mu, Hao Zhang, Chen-Yu Lee, and Na...
-
[2024]
Preprint, arXiv:2401.10825
Recent advances in named entity recogni- tion: A comprehensive survey and comparative study. Preprint, arXiv:2401.10825. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. A systematic study and comprehe...
2023 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.