{"id":"322b3c79-16e5-4954-8381-829e62f7f0dd","arxiv_id":"2505.13535","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BLOCKIE, a prompt-based LLM pipeline that splits documents into semantic blocks, reports state-of-the-art F1 scores on CORD, FUNSD, and SROIE.","lead":"This paper presents BLOCKIE, a method that splits receipts and forms into small self-contained text blocks, parses each block with a large language model, and combines the partial answers into a final structured output. The reported improvements matter because document understanding is widespread in business, and the method avoids fine-tuning while claiming robustness to new layouts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never directly verifies Eq. 2—that created blocks are context-independent—so the central claim that block organization, not the full-text combiner, drives the gains is not yet established.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: the whole method rests on Eq. (2), yet no evidence directly verifies it. The paper's own Table 4 shows block creation F1 between 73% and 87% across models, and the Limitations section admits missed linkages are hard to recover. This makes the independence assumption both central and unverified. I agree that the verdict should remain CONDITIONAL: the method is plausible and well engineered, but the strongest claim about the source of the gains needs a direct test. I do not see an internal logical contradiction in the pipeline; the concern is empirical validation, not soundness. The concrete test above would settle whether block self-containment is genuinely load-bearing or whether the full-text combiner is the real engine.","tokens_in":14227,"tokens_out":5928,"duration_ms":63207,"concrete_test":"Run a direct independence audit on 50 CORD test documents: for every block created by the Sonnet block creator, extract the affected schema fields twice—once from the block alone and once with the block placed inside the full document text—and record per-field disagreements. Then compare end-to-end F1 against the disagreement rate. As a control, rerun BLOCKIE with deliberately corrupted blocks (e.g., split an attribute-value pair across two blocks). If end-to-end F1 barely moves when blocks violate Eq. (2), the combiner is doing the load-bearing work; if F1 drops sharply, the block-independence assumption is confirmed and the bottleneck claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that decomposing documents into self-contained semantic blocks is what produces the reported F1 gains. This requires created blocks to satisfy Eq. (2), v(B,B)=v(B,D). The paper never tests that condition directly. Table 4's block-level F1 compares LLM-produced blocks to ground-truth blocks generated from training labels; that measures boundary or label agreement, not whether a block yields the same parse in isolation as with full document context. Moreover, the combiner prompt (Table 9) receives the complete OCR text and bounding boxes, so it can in principle repair errors caused by non-independent blocks. The Limitations section concedes that missed linkages are 'hard to recover,' but no experiment quantifies how often such misses occur or how much end-to-end F1 depends on the combiner's access to the full document. Without a direct independence measurement, the ablation in Table 5 cannot separate the contribution of block self-containment from the contribution of the multi-stage pipeline's later full-context pass.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"BLOCKIE is an LLM-based information extraction system for visually rich documents. The paper defines semantic blocks as localized segments for which parsing in isolation equals parsing with full-document context (Eq. 2), and proposes a three-stage pipeline: an LLM block creator segments OCR text into blocks using the document schema and retrieved few-shot demonstrations; an LLM block parser annotates each block independently; and an LLM combiner receives the full OCR text, bounding boxes, and partial parses to assemble the document-level output. Training labels are converted into block-level reasoning examples. The authors report state-of-the-art F1 on CORD (98.83), FUNSD entity linking (92.15), and SROIE (98.52) with Claude 3.5 Sonnet, and consistent results with Qwen 2.5 models from 7B to 72B. Additional experiments cover format heterogeneity, cross-dataset transfer, the effect of block creation quality, and value-absent inference. The paper acknowledges latency, dependence on LLM reasoning, and block-creation errors as limitations.","tokens_in":14388,"tokens_out":7655,"duration_ms":76802,"significance":"The reported results, if reproducible, are a significant practical advance for document IE: BLOCKIE outperforms published layout-aware and LLM-based methods on three standard benchmarks and shows that a 32B open model can beat much larger proprietary models when combined with block-level organization. The paper's strengths include held-out test evaluation, a broad LLM sweep, an informative ground-truth-block ablation (Table 5), a 100-sample heterogeneity test, and a cross-dataset transfer experiment. The main gaps are that the central causal claim - that self-contained semantic blocks, rather than the full-context combiner, drive the gains - is not directly tested, and that baseline comparisons rely on published numbers with no statistical significance information. These gaps are addressable and do not invalidate the empirical contribution, but they currently temper the strength of the paper's theoretical and causal claims.","major_comments":[{"comment":"The comparison in Table 1 is against published F1 values rather than baselines re-run under BLOCKIE's exact OCR, prompting, and schema settings; for generative LLM baselines in particular, differences of 1-3% can easily be within prompt-engineering or OCR variation. Please re-run the main competitors under identical conditions, or state clearly which numbers are quoted from prior papers and which are measured in this work.","section":"§5.2.1, Table 1"},{"comment":"The defining property of a semantic block, v(B,B)=v(B,D), is never measured directly. The block-level F1 in Table 4 compares created blocks with ground-truth blocks generated from labels, which tests boundary agreement rather than context independence. Since the combiner prompt in Table 9 supplies all document words and instructs the model to use the complete document text, the combiner can in principle repair any context-dependence of the blocks; the Limitations section even concedes that missed linkages are hard to recover, but no experiment quantifies this failure mode. A direct test is required: parse each created block in isolation and with full-document context, and measure the divergence; or ablate the combiner's access to full text to quantify how much end-to-end F1 depends on the final full-context pass.","section":"§3, Eq. (2), and §5.2.4"},{"comment":"The paper reports single-point F1 values with no variance or significance information. Differences such as BLOCKIE with Qwen-32B (96.14) versus Qwen-72B (96.01) are likely not meaningful without confidence intervals; please report means and standard deviations over repeated runs or bootstrap intervals over the test sets for the headline results.","section":"§5.2.2, Tables 1-2"},{"comment":"The statement that a collection of semantic atoms containing all of its linkages is 'a sufficient condition for equation 2' is presented as a mathematical claim but no proof is given. Because Eq. (2) is the formal basis for the method, the authors should either prove the implication under their definitions or explicitly demote it to a heuristic motivation.","section":"§3, semantic atoms and Eq. (2)"}],"minor_comments":[{"comment":"The cross-reference 'Figure 5.2.5' is broken; it should point to the actual figure showing the inference examples.","section":"§5.2.5"},{"comment":"The phrase 'the enity total amount' should read 'the entity total amount'.","section":"§5.2.3"},{"comment":"The text 'FUNSD Datatet' should read 'FUNSD Dataset'.","section":"Appendix B.1"},{"comment":"The entry for Harley et al. is incomplete (missing year and venue), and the CORD reference has a formatting issue in the author list ('V Y'); please fix the bibliography.","section":"References"},{"comment":"The caption says 'Semantic Block F1-scores' but the table reports end-to-end entity F1 under different block sources; the caption should be corrected to match the content.","section":"Table 5"},{"comment":"The value-absent inference experiment uses only 20 hand-selected cases with no stated selection protocol or error analysis; please document how the 20 cases were chosen and include a failure analysis.","section":"§5.2.5"}],"recommendation":"major_revision","confidential_remarks":"The main empirical result is credible enough to warrant a revision rather than rejection. My recommendation is driven by the missing direct test of Eq. (2) and the reliance on quoted baselines; both are fixable within the paper's scope. I would not require re-running every old baseline if the authors instead clearly delimit the claim to 'reported' numbers and add the direct independence experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: BLOCKIE is a genuinely new take on LLM-based document IE — split the document into \"semantic blocks,\" parse each block with few-shot examples, then recombine. The F1 numbers are credible and the ablation linking block quality to final accuracy is the most convincing part. But the paper's central causal claim, that block self-containment is what drives the gains, is never directly tested. The combiner receives the full OCR text and boxes, so it can paper over missing context. That makes the headline interpretation shakier than the results themselves.\n\nWhat's actually new: the formal definition of a semantic block (Eq. 2) and the practical scheme to create blocks from labeled training data using LLM reasoning. None of the cited prior work organizes documents into independent semantic units parsed separately; they do token classification or whole-document prompting. The train-time labeling of blocks with reasoning and partial annotations is a sensible way to build few-shot examples. The results across three benchmarks are good, and the experiments with smaller Qwen models show the method helps weaker LLMs, which is a nice practical result.\n\nSoft spots, in order of importance. First, Eq. 2 is never measured. Table 4's block-level F1 compares created blocks to ground-truth blocks — that's boundary agreement, not evidence that a block yields the same parse in isolation as in context. The paper's own Limitations section admits missed linkages are hard to recover, and the combiner gets the full document. Without a direct test of context independence, the ablation in Table 5 can't separate the contribution of block self-containment from the contribution of the later full-context pass. Second, the SOTA comparison uses published F1 values for baselines rather than re-running them under identical conditions, and there are no error bars or significance tests. That's standard in the field but worth noting. Third, the robustness claims rest on thin evidence: one 100-sample heterogeneity experiment and a 20-sample value-absent test. The cross-dataset CORD-to-SROIE result is suggestive but with no variance.\n\nNone of this sinks the paper. The method is practical and the honest Limitations section is a point in its favor. The main fix is to test Eq. 2 directly and to re-run at least one baseline with the same OCR and prompt protocol.\n\nThe paper is for practitioners building LLM-based document IE pipelines and for researchers working on layout-aware document understanding. It deserves serious peer review. The editor should not desk reject it; the reviewers should ask for the direct independence test and more careful robustness evaluation.","headline":"Genuinely new block-based LLM pipeline for document IE with credible F1 gains, but the central causal claim about block self-containment is never directly tested and the combiner's full-document access muddies the attribution.","tokens_in":14945,"tokens_out":2220,"would_cite":true,"duration_ms":22466,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BLOCKIE claims that splitting visually rich documents into self-contained semantic blocks lets LLM-based extraction reach state-of-the-art F1 on CORD, FUNSD, and SROIE while also generalizing to unseen formats and inferring absent values.","keywords":["information extraction","visually rich documents","semantic blocks","large language models","layout understanding","receipt parsing","form understanding","value-absent inference"],"falsifier":"Run BLOCKIE on the same receipts with deliberately broken block boundaries, for example placing every attribute and value in separate blocks, and check final F1; if extraction stays near 98.83%, the self-containment premise is not essential, while a large drop would confirm that it is.","tokens_in":1906,"feed_emoji":"🧾","tokens_out":2197,"duration_ms":93141,"temperature":0.7,"pith_summary":"BLOCKIE is an information-extraction method for visually rich documents that first cuts a document's OCR text into self-contained semantic blocks, parses each block independently, and then lets an LLM combine the partial parses into the requested schema. The paper claims this block-first organization is what drives accuracy, not the raw power of the language model: with a 32-billion-parameter open model it beats direct few-shot use of a much larger proprietary model, and with a 7-billion-parameter model it far surpasses comparably sized layout-aware fine-tuned models. On the public benchmarks CORD, FUNSD, and SROIE, BLOCKIE reports 98.83%, 92.15%, and 98.52% F1 respectively, 1 to 3 points above prior state-of-the-art methods. The same organization is claimed to make extraction resilient to new document formats and able to infer values that are not explicitly printed, such as counting line items on a receipt.","feed_headline":"Block-first parsing drives document extraction to 98.8% F1","feed_subtitle":"Organizing OCR text into independent blocks beats layout-tuned models and stays accurate on unseen formats.","key_machinery":"The load-bearing object is the semantic block, defined by the self-containment identity v(B,B)=v(B,D): parsing the block alone yields exactly the values it would yield with the whole document as context. Blocks are assembled from semantic atoms, indivisible spatially coherent text fragments, connected by attribute-value and hierarchical linkages; a block is valid when all linkages for its atoms are internal to it. This converts a single hard whole-document parsing problem into many smaller, format-agnostic ones, which is what lets similar blocks be matched across different templates and lets a relatively small LLM outperform a much larger one used directly.","core_discovery":"The paper's central discovery is that the context needed to extract an entity from a visually rich document is usually local: a small group of spatially and semantically linked tokens contains everything required to read that entity correctly. It formalizes this as the semantic block, a segment B satisfying v(B,B)=v(B,D), meaning the values extracted from B alone equal the values extracted when B is considered with the whole document as context. Semantic blocks are built from semantic atoms, indivisible spatially coherent text units, joined by attribute-value or hierarchy linkages, and a block is valid only when every linkage of every atom inside it is also inside it. BLOCKIE then runs three LLM stages: block creation, independent block parsing with similar-block few-shot examples, and a combining judge that reconstructs the full schema from partial parses. The paper attributes the reported state-of-the-art accuracy, resilience to unseen formats, and value-absent inference to this decomposition rather than to the choice or size of the underlying LLM.","pith_inferences":["The self-containment condition suggests a direct diagnostic: measure how often parsing a block alone disagrees with parsing it with full-document context on a held-out set; if such disagreements are common in blocks the creator judged valid, the claimed bottleneck is even more critical than the F1 numbers show.","Because block quality is the binding constraint, a dedicated lightweight model trained to predict block boundaries from the block creator's outputs could reduce the cost and latency of the current sequential pipeline while preserving accuracy.","The same decompose-then-combine pattern might transfer to other structured layout tasks, such as table extraction or document question answering, where local cell and header linkages play the role of semantic atoms.","The value-absent-inference claim rests on a 20-sample evaluation; running the same counting task on the full CORD test set would show how often the claimed reasoning survives at scale."],"forward_implications":["On CORD, FUNSD, and SROIE, BLOCKIE reports F1 of 98.83%, 92.15%, and 98.52%, beating the previous best methods by 1 to 3 points, so block-level organization is a competitive alternative to layout-aware pretraining for visually rich document extraction.","Model scale stops being the main driver: BLOCKIE with Qwen 2.5 32B reaches 96.14% F1 on CORD, above direct few-shot Sonnet at 95.72% and LMDX-Gemini Pro at 95.57%, while the 7B variant reaches 87.72%, far above similarly sized fine-tuned DocLLM and LayoutLLM.","With only 100 format-diverse training samples, BLOCKIE retains 94.47% F1 on CORD where LayoutLMV3 falls to 78.79%, and a CORD-trained model transfers to SROIE total-amount extraction at 97.06%, so format heterogeneity and unseen templates are handled better.","Value-absent inference is demonstrated: on 20 sampled CORD receipts, BLOCKIE infers the correct line-item count 18 times, including cases requiring subtraction and multi-step addition.","Block creation quality is the main performance bottleneck: replacing test blocks with ground-truth blocks lets a 7B parser reach 94.38% F1, close to the 96.14% of the 32B model, so further gains should come from better block boundaries."],"supporting_citations":[{"why":"Supplies the CORD dataset, its hierarchical receipt schema, and ground-truth labels used for the main training, evaluation, and value-absent inference tests.","marker":"Park et al., 2019"},{"why":"Supplies FUNSD, the form-understanding benchmark used for the entity-linking evaluation.","marker":"Jaume et al., 2019"},{"why":"Supplies SROIE, the receipt-extraction benchmark used for the third evaluation.","marker":"Huang et al., 2019"},{"why":"Provides the LMDX baseline and the practice of selecting similar documents for in-context examples, which BLOCKIE adapts to selecting similar blocks.","marker":"Perot et al., 2024"},{"why":"Provides LayoutLMV3, the layout-aware baseline BLOCKIE compares against and the model used to demonstrate the limits of token-classification methods on new formats.","marker":"Huang et al., 2022"},{"why":"Provides GeoLayoutLM, a strong layout-aware baseline on all three datasets that BLOCKIE must surpass.","marker":"Luo et al., 2023"},{"why":"Provides the Qwen 2.5 model family used to show that BLOCKIE sustains strong performance across model sizes down to 7B parameters.","marker":"Qwen et al., 2025"},{"why":"Motivates the step-by-step reasoning used in the block creation and parsing prompts, a mechanism the paper credits for its gains.","marker":"Wei et al., 2023"}],"fun_headline_variants":["Semantic blocks make document extraction local and generalizable","BLOCKIE decomposes documents into reusable semantic blocks for better IE","Local context beats layout-tuning: semantic blocks lift F1 by 1-3%","Block-wise reasoning improves extraction accuracy on unseen formats"],"cache_read_input_tokens":17152,"weakest_assumption_plain":"The pipeline assumes the block creator reliably splits unseen documents into self-contained blocks, because any linkage that crosses a block boundary is missed and cannot be fully recovered at the combination stage.","fun_headline_variants_meta":{"raw":{"variants":["Semantic blocks make document extraction local and generalizable","BLOCKIE decomposes documents into reusable semantic blocks for better IE","Local context beats layout-tuning: semantic blocks lift F1 by 1-3%","Block-wise reasoning improves extraction accuracy on unseen formats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1518,"prompt_tokens":942,"completion_tokens":576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":558,"tokens_out":576,"duration_ms":6078,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:33:27.923503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BLOCKIE on the same receipts with deliberately broken block boundaries, for example placing every attribute and value in separate blocks, and check final F1; if extraction stays near 98.83%, the self-containment premise is not essential, while a large drop would confirm that it is.","supporting_citations":[],"review_version":1}