{"id":"167d7e07-6406-4488-85d3-b94671555b3d","arxiv_id":"2412.20072","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A retrieve-summarize-extract pipeline with simple table-to-text serialization improves LLM extraction from hybrid long documents, and a new financial KPI dataset (FINE) is introduced to support evaluation.","lead":"The paper presents AIE, a pipeline that splits long mixed text-and-table documents into chunks, retrieves the chunks most relevant to a query, summarizes them, and asks a large language model to output the requested number. It also introduces FINE, a financial-report dataset, and reports that this retrieve-summarize-extract approach beats a naive truncation baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'naive' baseline is undefined in Section IV, so AIE's central claim of consistent outperformance is not interpretable; the reported margins may reflect a weak comparison.","rationale":"The paper's strongest claim is the empirical superiority of AIE over the naive method, so the comparison must be well-defined. In the full text, the naive method is never described; the figures simply label a bar 'Naive'. Without knowing whether the baseline truncates the document, ignores tables, or uses a different prompt, the reader cannot assess whether the 50-point gap at RETA 1% comes from the framework or from an artificially weak baseline. This is more load-bearing than the retrieval-module concern raised by the reader, because even if retrieval is imperfect, a meaningful baseline would still allow the framework's end-to-end value to be estimated; conversely, a perfect retrieval module cannot rescue an undefined comparison. The reader's retrieval concern is real and partially overlaps with the paper's own ambiguity experiments, but the baseline specification is a prerequisite for any quantitative conclusion. The test-set tuning of R@3, Refine, TD-RSP, KTC, and 1-shot is also concerning: Tables III–VII report accuracy on the same FINE test set used for the final comparison, so the reported numbers are optimistic. Neither issue is fatal; both are addressable in revision by defining the baseline and adding a validation split. The reader's CONDITIONAL verdict already captures this need, so I recommend leaving the verdict unchanged. My agreement with the reader is partial because I prioritize the undefined baseline over retrieval as the weakest point, and I would require the baseline specification as the first condition for acceptance.","tokens_in":8054,"tokens_out":3292,"duration_ms":33459,"concrete_test":"Inspect the released code and prompt templates for the exact 'naive' baseline used in Figures 2–4; if no baseline is defined, re-run the FINE comparison with a documented baseline (e.g., truncate each document to a fixed token budget and apply the same extraction prompt), then report whether AIE's RETA 1% margin (63.89% vs 13.92%) persists under that baseline with all hyperparameters selected on a held-out validation set rather than on the test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV states AIE is compared with 'a naive LLM-based approach' but never defines what that approach is—how many tokens are processed, whether the same extraction prompt is used, what happens to tables, or whether it is identical to AIE minus retrieval and summarization. Figures 2–4 report 'Naive' bars with no specification. Because the paper's stated conclusion ('AIE consistently outperforms the naive method') rests entirely on this comparison, an undefined baseline makes the central quantitative claim unfalsifiable and unreproducible. A secondary but related issue is that the final AIE configuration uses settings (R@3, Refine, TD-RSP, KTC, 1-shot) identified by the very tables (Tables III–VII) that report performance on the same FINE test set, so the reported accuracy is selected on the test set. This does not by itself invalidate the framework, but it means the specific numbers are upper bounds until a validation/test split is enforced. Both issues are fixable, but the baseline definition is the more load-bearing: without it, even a clean validation split cannot interpret the headline margin.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AIE, a four-module framework (Segmentation, Retrieval, Summarization, Extraction) for enabling LLMs to perform information extraction from hybrid long documents (HLDs) that exceed the LLM token limit. It introduces FINE, a new dataset of financial key performance indicators extracted from SEC filings, and evaluates AIE against a 'naive' LLM-based approach on FINE, WIKIR, and MPP. The experiments show large improvements for AIE on FINE (63.89% vs 13.92% at RETA 1% with GPT-3.5) and analyze several design choices: table serialization formats, number of retrieved segments, summarization strategies, numerical precision prompts, keyword completion, and few-shot counts. The paper concludes that a retrieve-summarize-extract pipeline with plain-table serialization and refine summarization is effective for HLD information extraction.","tokens_in":8213,"tokens_out":4006,"duration_ms":40862,"significance":"If the results are reproducible, the AIE framework offers a practical recipe for LLM-based extraction from long, mixed text-table documents, and the FINE dataset is a valuable new resource for the community. The paper also provides a useful systematic comparison of multiple design decisions, including table serialization, retrieval granularity, and prompt engineering. The main claims are concrete and testable, and the dataset/code release is a positive feature. However, the significance is currently tempered by two methodological issues: an undefined baseline and test-set-driven configuration selection, both of which affect the credibility of the quantitative claims.","major_comments":[{"comment":"The 'naive LLM-based approach' is never defined. Please specify exactly how the naive method processes an HLD: whether the document is truncated, how tables are serialized, which prompt is used for extraction, whether the same extraction module is used, and how the final answer is produced. Without this definition, the comparison in Figures 2–4 and the central claim that 'AIE consistently outperforms the naive method' are not interpretable or reproducible.","section":"Section IV, Figures 2–4"},{"comment":"The final AIE configuration is selected by running all variants on the FINE test set and picking the winner in each dimension (R@3 in Table III, PLAIN in Table II, Refine in Table IV, TD-RSP in Table V, KTC in Table VI, 1-shot in Table VII). The headline AIE accuracy in Figure 2 is then reported under this configuration. This is test-set selection, so the reported numbers are upper bounds rather than unbiased estimates of performance. Please enforce a validation split (or cross-validation) for model selection and report the chosen configuration's accuracy on a held-out test set, or at least discuss the magnitude of the selection effect.","section":"Section IV, Tables II–VII"}],"minor_comments":[{"comment":"The phrase 'The naive AIE has adaptability in many complex scenarios' appears to be a typo; it should likely read 'The AIE has adaptability' or 'The proposed AIE is adaptable.'","section":"Abstract"},{"comment":"The RETA metric is described only verbally. Please provide the explicit formula and specify how predictions with missing or non-numeric outputs are scored.","section":"Section III"},{"comment":"The y-axis label 'Average Accuracy' is confusing because the x-axis already shows RETA levels. Clarify that the bars represent the average accuracy over all queries at the corresponding RETA threshold.","section":"Figure 2"},{"comment":"The heatmaps lack a legend and a clear description of the color scale. Add a legend and explain what the colors represent for the RPD values.","section":"Figure 4"},{"comment":"The paper states that the dataset and code are 'publicly available in the attachments,' but no link or repository identifier is provided. Include a working URL for reproducibility.","section":"Section I and Section VI"},{"comment":"No variance or significance tests are reported for any of the comparisons. Given the large margins this is not fatal, but including error bars or a significance test for the headline FINE result would strengthen the claim.","section":"Section IV"},{"comment":"For the FINE dataset, please also report the number of documents and the number of queries, not only token length statistics, so readers can gauge the dataset size.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The core framework and the FINE dataset are potentially useful contributions, but the undefined baseline and the test-set-based selection of hyperparameters are load-bearing flaws that prevent the quantitative comparisons from being interpreted as they stand. The paper also appears to be an extension of the authors' earlier work (arXiv:2305.16344); the introduction should state clearly what is new in this manuscript relative to that preprint. I would be willing to review a revised version that addresses the baseline definition and the validation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a compose-the-obvious-pieces paper with one potentially useful new artifact, the FINE dataset for financial KPI extraction from hybrid long documents. The empirical story, however, leans on a naive baseline that is never defined, and the final AIE configuration is selected on the same test set used to report headline numbers. So the specific accuracies (63.89% vs 13.92% at RETA 1%) are upper bounds, not final estimates.\n\nWhat the paper does well: it runs a clean set of ablations. The serialization comparison (PLAIN over CSV/XML/HTML), the retrieval count sweep, Refine vs. Map-Reduce, numerical-precision prompts, keyword completion, and few-shot counts are all systematically varied and reported. The finding that a simple serialization is enough and that refinement beats map-reduce is useful practical guidance. If the FINE dataset is actually released, it could serve as a shared benchmark for long-document financial extraction; the construction details are thin here (only token statistics), so that remains conditional.\n\nWhere it's soft, in order of severity.\n\nFirst, the baseline. Section IV says 'a naive LLM-based approach' and the figures label bars 'Naive,' but no definition appears anywhere in the paper. We don't know whether it is truncation, full-document input, or a zero-shot extraction without retrieval and summarization. The paper's central claim—consistent outperformance—is unfalsifiable until this is pinned down.\n\nSecond, test-set selection. Tables II through VII all report accuracy on what appears to be the same FINE test set, and the final AIE numbers are exactly the best combination (R@3, PLAIN, Refine, TD-RSP, KTC, 1-shot) from those tables. No held-out validation or cross-validation is reported. This makes the magnitudes optimistic, though the qualitative trends may survive.\n\nThird, novelty overlap with the authors' own arXiv:2305.16344 is substantial; the framework and most findings appear in that earlier preprint. The paper needs to state explicitly what FINE adds beyond that work.\n\nFourth, minor: no error bars or significance tests anywhere, and the cross-dataset adaptability claim rests on two small external datasets. The ambiguity analysis (Figure 4) is interesting but would benefit from error bars.\n\nWho should read this: practitioners building LLM pipelines for long documents, and anyone looking for a financial KPI benchmark. I'd send it to review, but I'd require the baseline definition, a validation/test split, and a dataset construction section before I'd trust the numbers.\n\nRecommendation: engage with it, but treat the results as preliminary.","headline":"Systematic ablations and a new financial dataset, but the undefined baseline and test-set tuning make the headline numbers provisional; worth reviewing with major revisions.","tokens_in":8843,"tokens_out":2852,"would_cite":false,"duration_ms":28167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can pull numbers from hybrid long documents by retrieving, summarizing, and extracting only the relevant segments.","keywords":["hybrid long documents","information extraction","large language models","table serialization","embedding retrieval","financial reports","few-shot prompting","RETA metric"],"falsifier":"Build a hybrid document in which the target value sits in a table with paraphrased headers and no exact keyword match, then check whether embedding retrieval selects the segment containing the value; if retrieval recall on such tables drops sharply, the central claim fails at its first step. A second check is to evaluate the R@n choice on a held-out split rather than the analyzed test set, since the R@3 advantage could shrink or reverse on a new split.","tokens_in":7775,"feed_emoji":"📊","tokens_out":7794,"duration_ms":70437,"temperature":0.7,"pith_summary":"Large language models struggle with hybrid long documents—reports where prose and tables are woven together and the whole text exceeds the model's context window. This paper proposes AIE, a four-stage pipeline that splits the document into segments, retrieves the segments most similar to a query keyword using embedding similarity, condenses them with an LLM, and extracts the target number. On the new FINE financial dataset with GPT-3.5, AIE reaches 63.89% accuracy at a 1% relative-error tolerance, versus 13.92% for naive truncation, and it also beats the naive approach on Wikipedia and materials-science documents. The paper further reports that plain-text table serialization outperforms hierarchical XML and HTML, that retrieving three segments is the best setting, and that one well-chosen few-shot example beats zero or multiple shots.","feed_headline":"Pipeline lifts LLM extraction accuracy to 64% on mixed docs","feed_subtitle":"Retrieve-summarize-extract beats naive truncation on financial, Wikipedia, and materials documents.","key_machinery":"The central object is the AIE pipeline with four modules: Segmentation, Retrieval, Summarization, and Extraction. Segmentation serializes tables into plain text, splits oversized elements, and merges adjacent small ones; Retrieval scores each segment against the query keyword using a sentence-embedding model (SentenceTransformer) and keeps the top K=3 segments; Summarization applies the Refine strategy, iteratively updating a running summary with each retrieved segment; Extraction uses a tailored prompt to return the precise value. The evaluation machinery is the RETA metric, which counts a prediction as correct when its relative error stays within a chosen tolerance, and the new FINE dataset of financial KPI values extracted from SEC filings.","core_discovery":"The paper's central claim is that information extraction from hybrid long documents does not require processing the entire document or preserving the hierarchical structure of tables. A pipeline that segments the document, retrieves keyword-similar segments by embedding similarity, summarizes them iteratively with an LLM, and extracts the answer from the summary consistently outperforms a naive method that truncates the document, across financial reports, Wikipedia pages, and materials-science papers. On FINE, AIE with GPT-3.5 achieves an average accuracy of 69.93% versus 15.99% for the naive method. The paper also claims that a simple serialization of tables into plain text is sufficient for LLM table comprehension, and that the framework benefits from retrieving three segments, using the Refine summarization strategy, completing query keywords with company and time context, and using a single precision-aware example.","pith_inferences":["If embedding retrieval is the bottleneck, then better embeddings for table-heavy text, or hybrid retrieval that also uses table structure, should directly raise extraction accuracy within the same pipeline.","The R@3 optimum found on FINE may not transfer to other genres; a production version could select the number of retrieved segments per query based on retrieval confidence rather than a global constant.","The framework's modularity suggests it could be applied to legal, medical, or scientific records, but the paper demonstrates only finance, Wikipedia, and materials-science domains, so cross-domain generality remains an open extension.","The plain-serialization result hints that LLMs may understand tables through token co-occurrence between headers and values rather than through structural nesting, which could be probed by shuffling table layout while keeping content identical."],"forward_implications":["A retrieval-first pipeline removes the context-window ceiling: LLMs can work on arbitrarily long hybrid documents by reading only the segments that match the query.","Plain-text table serialization is enough for LLM table understanding, so expensive structure-preserving encodings such as XML or HTML are unnecessary and can even hurt accuracy.","The number of retrieved segments has a sweet spot: R@3 outperforms R@1, R@5, and R@7 on FINE, so retrieval quantity should be tuned rather than maximized.","Keyword completion with document context (company and time) materially improves extraction, raising average accuracy from 0.3922 with the bare keyword to 0.6993 with full context.","A single well-designed few-shot example gives better accuracy than zero-shot or multiple shots, indicating that prompt consistency matters more than example count."],"supporting_citations":[{"why":"Supplies the SentenceTransformer embedding model used to score similarity between each document segment and the query keyword.","marker":"[13]"},{"why":"Grounds the embedding-based retrieval approach that the Retrieval Module adopts for selecting relevant segments.","marker":"[10]"},{"why":"Provides the WIKIR dataset used to evaluate AIE versus the naive method on Wikipedia key-value extraction.","marker":"[14]"},{"why":"Provides the MPP dataset used to evaluate AIE on material-property extraction from scientific papers.","marker":"[15]"},{"why":"Supports the use of LLMs as few-shot table reasoners, motivating the one-shot setting that the paper finds optimal.","marker":"[7]"}],"fun_headline_variants":["LLM pipeline lifts hybrid doc extraction to 70% accuracy","Retrieve-summarize-extract beats naive truncation on long docs","Simple table serialization suffices for LLM comprehension","AIE framework: 70% accuracy on financial reports, beats truncation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retrieval step trusts that embedding similarity between the query keyword and a document segment points to the segment containing the answer; if the keyword is phrased differently from the table or the table serializes poorly, retrieval can select the wrong segments and the later summarization and extraction steps cannot recover the value.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline lifts hybrid doc extraction to 70% accuracy","Retrieve-summarize-extract beats naive truncation on long docs","Simple table serialization suffices for LLM comprehension","AIE framework: 70% accuracy on financial reports, beats truncation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2530,"prompt_tokens":907,"completion_tokens":1623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1549}},"tokens_in":523,"tokens_out":1623,"duration_ms":12596,"temperature":1.0,"reasoning_tokens":1549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:35:27.002048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a hybrid document in which the target value sits in a table with paraphrased headers and no exact keyword match, then check whether embedding retrieval selects the segment containing the value; if retrieval recall on such tables drops sharply, the central claim fails at its first step. A second check is to evaluate the R@n choice on a held-out split rather than the analyzed test set, since the R@3 advantage could shrink or reverse on a new split.","supporting_citations":[{"cited_title":"Embedding-based product retrieval in taobao search,","cited_arxiv_id":null,"evidence_quote":"Grounds the embedding-based retrieval approach that the Retrieval Module adopts for selecting relevant segments."},{"cited_title":"From Dataset Recycling to Multi-Property Extraction and Beyond","cited_arxiv_id":"2011.03228","evidence_quote":"Provides the WIKIR dataset used to evaluate AIE versus the naive method on Wikipedia key-value extraction."}],"review_version":1}