Pith. sign in

REVIEW 4 major objections 7 minor 54 references

Scout: Scalable Document Extraction via Data Similarity

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A program-learning pipeline matches full-document LLM extraction accuracy while reading only answer-located spans, cutting cost by 61x to over 1000x on a 1,000-document collection.

desk verdict A genuinely useful systems result, but the headline approximation guarantee for rule refinement is not a bound against the stated objective and needs a fix before acceptance. read the letter →

arxiv 2608.08261 v1 pith:GHX4G5UA submitted 2026-08-08 cs.DB

Yiming Lin , Chiyu Hao , Shreya Shankar , Aditya G. Parameswaran This is my paper · ORCID
classification cs.DB
keywords documentextractionlargelanguagemodelsdatasimilarityrulesynthesisrefinementsetcoverapproximationPareto-optimalsubsetselectioncascadefallback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Scout is built on a simple observation: within a collection of similar documents, the answer to a given query tends to sit in the same kind of location, so an LLM should not need to read every token. The paper's claim is that one can synthesize small Python 'rules' that locate that span on any document, feed only the span to an expensive oracle LLM, and still match the accuracy of an agent that reads the full document, at 61x to over 1000x lower cost on a 1,000-document collection. This claim is supported by a two-stage design: an agent first generates a broad pool of high-recall rules, then a greedy algorithm selects a cheap Pareto-optimal subset, a problem proven NP-hard (by reduction from Set Cover) and approximated within $O(\ln n)$ of optimal. Scout also handles collections that are only partially similar by sampling documents across clusters with an LLM-free farthest-point strategy, and by applying rules in a cascade with a cheap proxy check and a fallback to the full rule pool. On six real-world datasets the full pipeline averages 0.898 accuracy against the golden baseline's 0.895, and it is 61 points more accurate than the strongest prior program-based extractor.

What carries the argument

The load-bearing object is the 'rule': a self-contained Python function synthesized by an LLM agent that maps a document's enriched span representation to a short subsequence thought to contain the answer. Three mechanisms carry the argument: (1) accuracy monotonicity (Property 1) — the assumption that adding text to a sufficient span never destroys the oracle's answer, which makes rule-set correctness monotone; (2) rule refinement, formalized as finding a Pareto-optimal subset (one no other subset beats in both cost and accuracy) within accuracy tolerance $\alpha$, proven NP-hard by reduction from Set Cover and solved by a greedy cost-effectiveness rule that mimics weighted set cover, with Chvátal's $O(\ln n)$ approximation bound; (3) farthest-point sampling over a cosine distance between documents' chunked query-similarity vectors, with cluster-coverage guarantees (Theorems 5.1–5.2), followed by a cascade application that uses a cheap proxy model to verify each prefix of refined rules and falls back to the full pool if none suffices.

What would settle it

Take a yes/no query on a financial filing where the minimal sufficient span is 'management concluded that internal control over financial reporting was effective' (answer: No material weakness). Append a sentence from elsewhere in the document that says 'However, management reported a material weakness in the titanium alloy used in turbine blades' and observe whether a frontier LLM changes its answer. If such span-superset reversals occur on more than about 6% of documents in a real collection, then monotonicity fails more often than the cited study suggests and Scout's accuracy guarantee would be measurably below the golden baseline on that collection.

Watch

Extended reading notes

Core claim

The central discovery is that the expensive step in LLM document extraction—reading the whole document—can be replaced by a learned 'rule': a Python program that returns the subsequence where the answer recurs, after which an oracle LLM answers from that short span alone. The paper proves that choosing the cheapest accurate subset of such rules is NP-hard and gives a greedy solution whose cost is within a logarithmic factor of optimal, the same algorithm as weighted set cover. The accuracy of the resulting rule set is protected by a monotonicity assumption (Property 1: if a span suffices, any superset suffices), which lets the pipeline fall back safely to a larger rule pool when a refined rule fails. Experiments on six real-world datasets show that Scout matches or exceeds the accuracy of a frontier per-document agent (0.898 vs 0.895 average) while cutting amortized cost by one to two orders of magnitude at measured sizes and by 61x to over 1000x at 1,000 documents.

Load-bearing premise

The entire accuracy argument relies on the oracle being monotone: if a span of text is enough to produce the correct answer, any larger span containing it is also enough; the paper cites a study finding this holds on over 94% of real-world documents, so on the remaining documents the greedy refinement and the fallback can silently lose the answer.

Editorial extensions

If this is right

  • If the claim holds, large-scale extraction jobs that previously cost hundreds to thousands of dollars per query can run for cents to a few dollars: the one-time rule-generation cost amortizes over the whole collection, so the total cost stays nearly flat as the collection grows.
  • The $O(\ln n)$ approximation guarantee means a user can set an accuracy tolerance $\alpha$ and know that the refined rule set's cost is within a logarithmic factor of the cheapest subset that meets that tolerance, making cost-accuracy tradeoffs predictable.
  • The cascade fallback means the system degrades gracefully on heterogeneous collections: when refined rules miss the answer on an unseen document, the proxy detects it and the full high-recall rule pool (which alone matches golden-baseline accuracy) is used instead.
  • Because the full rule pool alone matches the accuracy of reading every document, the paper's numbers imply that answer-location learning, not question-answering ability, is the main bottleneck for scalable extraction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If monotonicity fails more often than the cited 6%, particularly on yes/no or negation-heavy queries, the accuracy guarantees would degrade proportionally; a stress test on adversarial or contradictory documents would reveal whether the cited bound is conservative.
  • The reported cost savings are computed at 1,000 documents; because offline costs are fixed, the advantage should widen on larger collections, but this is an extrapolation in the paper, not a measurement, and it assumes the rule pool generalizes without needing more samples.
  • The rule generation step's diversity hints (page, section, keyword, table cells, typography) suggest that a richer coverage objective over hint categories might reduce fallback rates further; the paper does not test that directly.
  • The span-retrieval abstraction is general enough that the same rules could be compiled to layout-aware or vision-based locators for scanned documents, but Scout leaves that extension unstated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. Scout is a document-extraction system that exploits inter-document similarity to make LLM-based extraction over large collections affordable. Given a query and a collection, Scout (i) samples a small representative subset via embedding-based farthest-point sampling (Algorithm 2); (ii) uses an LLM agent to generate a pool of 'rules'—Python programs that return candidate answer-bearing spans (Section 3); (iii) greedily selects a cheap, accurate subset of rules, a problem it proves NP-hard (Section 4); and (iv) applies the refined rules to unseen documents using a cheap proxy to verify whether the retrieved span suffices, with fallback to the full rule pool (Section 5.2). Experiments on six real-world datasets report that Scout matches the accuracy of a per-document agent (~0.90 vs ~0.89) while being 61x to over 1000x cheaper at 1,000 documents, and is 61 points more accurate than the program-based Evaporate baseline. The paper also proves cluster-coverage guarantees for the sampling strategy (Theorems 5.1 and 5.2).

Significance. The problem is well chosen and the solution is concrete: if the reported numbers are stable, a matched-accuracy cost reduction of two to three orders of magnitude over per-document agents would be a substantial contribution to the LLM-powered data-systems literature. Strengths of the paper include the evaluation design (six domains, two external benchmarks with existing ground truth, an explicit amortized-cost model, and a stated artifact URL), the clean and correctly argued analysis of farthest-point sampling, which avoids relying on the triangle inequality that cosine distance does not satisfy, and the clear separation of an accuracy-first rule-generation stage from a cost-focused refinement stage. The main reservations are that the stated approximation guarantee for rule refinement (Theorem 4.3) is not established for the union-cost objective of Problem 1, and that the empirical comparison rests on single runs without variance estimates. Neither issue invalidates the empirical accuracy/cost comparison in my reading, and both are addressable in revision.

major comments (4)
  1. [Section 4.2, Theorem 4.3] Theorem 4.3 establishes cr(R', D_s) <= H_n * sum_{r in R*} cr(r, D_s), where the right-hand side evaluates the sum of the per-rule average cost ratios of the optimal subset, not the union-cost objective cr(R*, D_s) that Problem 1 asks to minimize. Since overlapping rules are counted once in cr(R*, D_s) but once per rule in the sum, the sum can exceed cr(R*, D_s) by an arbitrarily large factor (up to |R*|), so the theorem does not show that Algorithm 1 is within a logarithmic factor of the Problem 1 optimum. The 'provable approximation guarantee' in the abstract and the 'within a logarithmic factor of optimal' claim in the conclusion consequently overstate the result. The fix is either to state the guarantee explicitly for the sum-cost surrogate objective, or to change Algorithm 1 to use marginal union costs and analyze it as a submodular cover problem.
  2. [Section 6.2] Every headline number (Table 1, Figures 9 and 10, Table 6) comes from a single run of a pipeline containing stochastic components (GPT-5.4-based rule generation and refinement, and a single draw of the sample), and no variance or repeat information is reported. Because the central claim is that Scout matches the per-document agent's accuracy while being one to three orders of magnitude cheaper, and several dataset-level differences are only a few points (average 0.898 vs 0.895, within 1.6 points on FinanceBench, OfficeQA, and Product), run-to-run variability could in principle change the direction of some of these comparisons. Please report multiple runs with standard deviations, at minimum for rule generation and the sample draw, or provide evidence that the reported numbers are stable.
  3. [Section 2.2 (Property 1) and Section 4.2 (Algorithm 1)] The reduction to set cover and the greedy's coverage bookkeeping require exact accuracy monotonicity: a document is marked covered when any selected rule is correct on it, and the union of individually correct rules is assumed correct. The paper acknowledges that monotonicity holds only on 'over 94% of documents' according to the external study [31], and it does not measure monotonicity on its own six datasets. The stated degradation ('only that document is affected... guarantees degrade by at most the fraction of documents on which monotonicity fails') bounds the direct accuracy effect but not the effect on Algorithm 1's selection: on a failing document, a rule that is correct alone can be incorrect when its span is unioned with another rule's span, which can mislead the coverage labels used in Line 6 on subsequent documents as well. Please measure monotonicity on the six datasets or provide a bound on the resulting selection error.
  4. [Section 4.2 (proxy labeling) and Section 5.2 (cascade)] Algorithm 1's per-rule correctness labels a(r, d_j) and the cascade's sufficiency checks (Figure 8) both use an unspecified 'cheap proxy model,' and the paper asserts without measurement that on short spans the proxy matches the oracle's estimate. Because the refined and cascaded rule sets whose costs and accuracies are reported (Table 6, Figure 9) are selected and applied using these proxy decisions, the paper should report per-dataset proxy-vs-oracle agreement on the rule spans and on the proxy-verification task; currently the reliability of the proxy is an unsupported premise of the experimental pipeline.
minor comments (7)
  1. [Section 1, Table 1] Table 1 presents the headline accuracy and cost results inside the Introduction, before the experimental setup is described; consider labeling it as a teaser and deferring the full comparison to Section 6.
  2. [Sections 2.1 and 6.1] The oracle is introduced as 'a top-of-the-line LLM (e.g., GPT-5.5)' in Section 2.1, but the experiments use GPT-5.4 for the oracle and judge and Claude Opus 4.7 for labeling; please state the exact oracle model in Section 6.1 so the accuracy definition of Section 2.2 can be checked against the evaluation.
  3. [Section 6.1.2] In the ablation prompts, the paid verify_accuracy calls are bounded by a '[Budget]' placeholder, but no budget values are reported anywhere in Section 6; without these values, the comparison between Scout and Scout-AgentRef, Scout-AgentGen, and Scout-AgentE2E is under-specified.
  4. [Section 5.1] The sample-size cap min(|S|, 0.1|D|) is applied after Algorithm 2 terminates, while Theorems 5.1 and 5.2 are proved for the uncapped sample; please clarify whether the cluster-coverage guarantees survive truncation by the cap.
  5. [Sections 2.2 and 6.1.3] Accuracy is defined as agreement with the oracle's per-document answer O(d_j, Q) in Section 2.2, but Section 6.1.3 measures accuracy against ground-truth labels using an LLM-as-a-judge (GPT-5.4); the paper should state how documents on which the oracle and the labelers disagree are treated, and should acknowledge that using the same model family for oracle and judge may be optimistic.
  6. [Reference [31]] Reference [31], which carries the empirical burden for the monotonicity assumption (Property 1), is cited as a Google Drive link to a technical report rather than a published venue; please provide a stable, citable reference.
  7. [Throughout] There are several small typos and spacing errors (e.g., 'consecutivespans' in Section 3.1, the duplicated 'H_n = O(ln n)' near Theorem 4.3, and inconsistent notation d^i_j vs o_ji for rule outputs); a copyediting pass is needed.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity; one minor load-bearing self-citation for the monotonicity assumption; Theorem 4.3 is overstated but not circular.

  1. self citation load bearing [Section 2.2, Property 1 (also used in Section 4.1 and Algorithm 1)]
    "Even a top-of-the-line LLM (e.g., GPT-5.5) is not a true oracle, so monotonicity may not always hold. Such failures are rare in practice: a recent study [31] (published at VLDB 2026) reports that monotonicity holds on over 94% of documents across real-world datasets, and we therefore adopt it as an assumption."

    Property 1 underwrites the refinement guarantees: Section 4.1 uses it to assert that 'no subset of R can exceed a(R,Ds)', and Theorem 4.2 plus Algorithm 1 depend on a(R',Ds)=a(R,Ds) being the target and on adding rules never hurting accuracy. The only cited support is [31], a prior study by the same authors (Lin, Zeighami, Parameswaran), referenced via a Google-Drive technical report rather than an independently checkable artifact. Because the paper explicitly labels it an assumption and quantifies degradation when it fails, this is a minor self-citation issue rather than a derivation that equates the conclusion to its input.

full rationale

Scout's main empirical derivation is self-contained: rules are generated from a small sample, and accuracy is measured on held-out documents (Table 3, Table 6) and on external benchmarks (FinanceBench, OfficeQA). The cost comparison amortizes measured offline costs and is not a renamed fit. The theoretical results are conditional on Property 1, an explicit monotonicity assumption whose only cited empirical support is the authors' own earlier study [31]; because the paper labels it an assumption and states the failure mode, this is minor self-citation rather than circular derivation. One non-circular correctness issue surfaced by the derivation chain: Theorem 4.3 bounds the greedy subset against the sum of per-rule costs, not against the union-cost optimum of Problem 1, so the 'logarithmic factor of optimal' phrasing is overstated for the stated objective; this does not make the empirical accuracy/cost claims circular. LLM-generated labels and LLM-as-judge on four datasets are evaluation-validity concerns, not construction-level circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or ontological entity; 'rules' are Python programs and 'clusters' are defined subsets of documents. Both are operational rather than postulated. The free parameters listed above are hand-chosen constants that influence sampling, rule generation, and refinement, and the central empirical results may shift with different values.

free parameters (6)
  • chunk count m = 50
    Used to build query-document similarity vectors for sampling; chosen empirically in Section 5.1.
  • stop ratio rho = 0.5
    Farthest-point sampling stops when the gap falls below half the previous gap (Algorithm 2); chosen by hand.
  • sample size cap = min(|S|, 0.1|D|)
    Caps the cost of rule generation; justified empirically in Section 5.1.
  • accuracy tolerance alpha = 0.05
    Default tolerance in Problem 1 for rule refinement, set in Section 6.1.3.
  • rule generation accuracy and cost targets = 0.95 accuracy, 0.1 cost ratio
    Prompt targets for the agent in Section 3.1; these are soft requirements that shape the generated rule pool.
  • cluster cost threshold delta = 0.1
    Defines a cluster as documents where a rule is correct with cost ratio below 0.1, given in Section 5.
assumptions (5)
  • domain assumption Oracle monotonicity: a superset of a sufficient span remains sufficient (Property 1).
    Central to accuracy monotonicity and the rule refinement/cascade design; supported only by the authors' prior study [31] claiming 94% of documents.
  • domain assumption Answers to the same query recur in similar locations across similar documents.
    Motivating premise for the entire rule-based approach in Section 1; if false, rules cannot generalize.
  • domain assumption The embedding-vector cosine distance approximates whether two documents share a rule-level answer pattern.
    Used for cluster sampling in Section 5.1; the paper states the metric is approximate and may over-sample.
  • domain assumption LLM-as-a-judge correctly decides whether an extracted span reproduces the ground-truth answer.
    Used for all accuracy measurements in Section 6.1.3; the judge is GPT-5.4 while ground truth for four datasets is LLM-labeled.
  • standard math Standard NP-hardness and approximation results of Karp, Chvatal, Slavik, and Gonzalez hold.
    Invoked in Theorems 4.2, 4.3, 5.1, and 5.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scout: Scalable Document Extraction via Data Similarity." pith.science (2026). https://pith.science/paper/GHX4G5UA

@misc{pith2026260808261,
  author       = {Pith},
  title        = {Pith review of: Scout: Scalable Document Extraction via Data Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHX4G5UA}},
  note         = {Machine review of arXiv:2608.08261}
}
read the original abstract

Extracting values from large document collections powers data analysis across many domains. Frontier LLMs extract such values accurately, but processing an entire collection with one is prohibitively costly. Yet this cost is largely avoidable: real-world collections exhibit rich similarity, so for the same query over similar documents, the answer tends to recur in similar locations; an LLM need only read that small span, not the whole document. Prior methods that exploit this similarity fall short: they either assume a rigid document structure, or assume the answer is a set of substrings of the input and use an LLM-generated program to return it directly. Even a frontier agent fails to generate effective programs to directly locate the answer's span, as the search space is large and programs learned from a small sample tend to overfit. We present Scout, a tool that generates accurate and cost-effective programs (that we call rules) to extract data at scale. From a few sampled documents, Scout generates a broad rule set and refines it by selecting a pareto-optimal subset with low cost without sacrificing accuracy. We prove rule refinement is NP-hard and give a greedy solution with a provable approximation guarantee. Scout handles collections that are only partly similar, where similarity holds within clusters of documents. In this setting, a sampling strategy, using no LLM, draws samples from each cluster; and a cascade strategy selects a subset of refined rules, falling back to the unrefined rule set when the selected rules don't contain the answer. Experiments on six real-world datasets show that Scout matches the accuracy of the strongest baseline, a frontier LLM agent that reads each full document, while being 61x to over 1000x cheaper on a collection of 1,000 documents, and is 61% more accurate than the strongest prior program-based approach.

Figures

Figures reproduced from arXiv: 2608.08261 by the authors.

Figure 1
Figure 1. Extraction queries over a 10-K. Each panel shows the query, its source span (answer-relevant parts in blue italics), and the program Evaporate generates with the answer it returns. these analyses is extracting structured fields from the documents [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Motivating example to illustrate Scout’s pipeline on financial datasets. instead be inferred from the context; for Q2, a program matching “material weakness” returns a false positive. Beyond the above, a large body of work studies extraction from each document individ￾ually, ignoring data similarity, and is thus either inaccurate at scale or not scalable; we defer detailed discussion to Section 7. Frontier agent for… view at source ↗
Figure 3
Figure 3. Pipeline of Scout. • We formalize document similarity as rules, Python pro￾grams synthesized by LLMs that retrieve portions of doc￾uments to be fed into an oracle LLM. Such rules serve as primitives for scalable document extraction. (Section 2) • We formalize the selection of an accurate, cost-efficient rule set as a pareto-optimal optimization problem, prove it to be NP-hard, and provide a greedy algorithm with a p… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Python script of 𝑟2 in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Rule refinement as a Pareto problem. that is at least as accurate. The two labeled rule sets 𝑅 ′ 1 and 𝑅 ′ 2 are both pareto-optimal, yet neither dominates the other: 𝑅 ′ 1 is more accurate while 𝑅 ′ 2 is cheaper. Problem 1 (Rule Refinement). Given the set of rules 𝑅, …
Figure 7
Figure 7. Figure 7: Documents cluster by where their answers to 𝑄 reside, and each rule covers one cluster of similar documents. 𝑎(𝑅, 𝐷𝑠 ) is met. Chvátal’s theorem [16] bounds the total weight of the greedy’s cover by 𝐻𝑛 times that of any feasible cover. At 𝛼 = 0 the constraint of Proble…
Figure 8
Figure 8. Figure 8: The proxy-verification prompt. [Question] and [Text] are placeholders for the task 𝑄 and the union of retrieved spans. them to 𝑑𝑖 and forms the union of their returned spans,𝑈𝑗 = Ð𝑗 𝑙=1 𝑑 𝑙 𝑖 . It then asks a cheap proxy 𝑃 (e.g., GPT-5.4 mini) whether 𝑈𝑗 can answer 𝑄, …
Figure 9
Figure 9. Figure 9: Accuracy versus amortized cost ratio across the six datasets. Up and to the left is better (higher accuracy, lower cost). agent performing the same steps. For the agents used in these abla￾tions, we provide tools that compute the accuracy and cost of any rule set, as d…
Figure 10
Figure 10. Figure 10: Total cost as the document collection grows. Only strategies within 5% of the per-document agent’s accuracy are shown. reduces cost while preserving accuracy through rule refinement. When the refined rules don’t contain the answers, Scout falls back to the initial rul…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 43 canonical work pages

  1. [31]

    Parameswaran

    Yiming Lin, Sepanta Zeighami, and Aditya G. Parameswaran. 2025. Bolt-on, Verifiable Provenance for LLM-Powered Data Processing. https://drive.google. com/file/d/16thcosiFUVvwiMr3pgzZRHNejOV52lNp/view

    work page 2025

  2. [1]

    CourtListener: U.S

    2026. CourtListener: U.S. Federal Court Opinions. https://www.courtlistener.com. Free Law Project. 2026

    work page 2026

  3. [2]

    Medicines: European Public Assessment Reports (EPAR)

    2026. Medicines: European Public Assessment Reports (EPAR). https://www. ema.europa.eu/en/medicines. European Medicines Agency. 2026

    work page 2026

  4. [3]

    PHMSA Pipeline Safety Enforcement: Notices of Probable Violation

    2026. PHMSA Pipeline Safety Enforcement: Notices of Probable Violation. https: //primis.phmsa.dot.gov/comm/reports/enforce/. U.S. Pipeline and Hazardous Materials Safety Administration. 2026

    work page 2026

  5. [4]

    Tropical Cyclone Reports

    2026. Tropical Cyclone Reports. https://www.nhc.noaa.gov/data/tcr/. U.S. National Hurricane Center, NOAA. 2026

    work page 2026

  6. [5]

    Milan Aggarwal, Hiresh Gupta, Mausoom Sarkar, and Balaji Krishnamurthy

  7. [6]

    Amazon Web Services. [n. d.]. Amazon Textract. https://aws.amazon.com/ textract/. 2026

    work page 2026

  8. [7]

    Arvind Arasu and Hector Garcia-Molina. 2003. Extracting Structured Data from Web Pages. InProceedings of the 2003 ACM SIGMOD International Conference on Management of Data. ACM, 337–348

    work page 2003

Show all 54 references
  1. [8]

    Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes.Proc. VLDB Endow.17, 2 (2023), 92–105

    2023

  2. [9]

    Christoph Auer, Maksym Lysak, Ahmed S. Nassar, Michele Dolfi, Nikolaos Li- vathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindl- bauer, Kasper Dinkla, Lokesh Mishra, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valéry Weber, Lucas Morin, Ingmar Me...

    2024 arXiv

  3. [10]

    Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Mark Dredze, and Alan Ritter. 2024. Schema-Driven Information Extraction from Heterogeneous Tables. InFindings of the Association for Computational Linguistics: EMNLP. 10252–10273

    2024

  4. [11]

    Robert Baumgartner, Sergio Flesca, and Georg Gottlob. 2001. Visual Web Infor- mation Extraction with Lixto. InProceedings of the 27th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann, 119–128

    2001

  5. [12]

    Chengliang Chai, Jiajun Li, Yuhao Deng, Yuanhao Zhong, Ye Yuan, Guoren Wang, and Lei Cao. 2025. Doctopus: Budget-aware Structural Table Extraction from Unstructured Documents.Proc. VLDB Endow.18, 11 (2025), 3695–3707

    2025

  6. [13]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.Trans. Mach. Learn. Res. (TMLR)(2024)

    2024

  7. [14]

    Zhijia Chen, Weiyi Meng, and Eduard C. Dragut. 2022. Web Record Extraction with Invariants.Proceedings of the VLDB Endowment16, 4 (2022), 959–972

    2022

  8. [15]

    Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, Yves- Laurent Kom Samo, Pushkar Khadilkar, Xianshun Chen, Sam Idicula, Fatma Ozcan, et al. 2026. 100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models:[Exp...

    2026

  9. [16]

    V. Chvátal. 1979. A Greedy Heuristic for the Set-Covering Problem.Mathematics of Operations Research4, 3 (1979), 233–235

    1979

  10. [17]

    Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. 2001. RoadRunner: Towards Automatic Data Extraction from Large Web Sites. InProceedings of the 27th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann, 109–118

    2001

  11. [18]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gard- ner. 2021. A dataset of information-seeking questions and answers anchored in research papers. InProceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computation...

    2021

  12. [19]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. InThe Twelfth International Conference on Learning Representations (ICLR)

    2024

  13. [20]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Shaohan Wang, Pengyu Wang, Xiaorui Wang, and Zhendong Mao. 2026. A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces. arXiv preprint arXiv:2602.03442. arXiv:2602.03442 [cs.CL]

    2026

  14. [21]

    Gonzalez

    Teofilo F. Gonzalez. 1985. Clustering to Minimize the Maximum Intercluster Distance.Theoretical Computer Science38 (1985), 293–306

    1985

  15. [22]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2026. A survey on llm-as-a-judge.The Innovation7, 6 (2026)

    2026

  16. [23]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944(2023)

    2023 arXiv

  17. [24]

    Saehan Jo and Immanuel Trummer. 2024. ThalamusDB: Approximate Query Processing on Multi-Modal Data.Proc. ACM Manag. Data2, 3, Article 186 (2024)

    2024

  18. [25]

    Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li wei H. Lehman, Leo A. Celi, and Roger G. Mark. 2023. MIMIC-IV, a freely accessible electronic health record dataset.Scientif...

    2023 doi

  19. [26]

    Richard M. Karp. 1972. Reducibility among Combinatorial Problems. InCom- plexity of Computer Computations. Plenum Press, New York, 85–103

    1972

  20. [27]

    Mohammed Kayed and Chia-Hui Chang. 2010. FiVaTech: Page-Level Web Data Extraction from Template Pages.IEEE Transactions on Knowledge and Data Engineering22, 2 (2010), 249–263

    2010

  21. [28]

    Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. 2024. GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models. InFindings of the Association for Computatio...

    2024

  22. [29]

    Yiming Lin, Mawil Hasan, Rohan Kosalge, Alvin Cheung, and Aditya G Parameswaran. 2025. Visual template inference for data extraction from docu- ments.Proceedings of the ACM on Management of Data3, 6 (2025), 1–27

    2025

  23. [30]

    Parameswaran, and Eugene Wu

    Yiming Lin, Madelon Hulsebos, Ruiying Ma, Shreya Shankar, Sepanta Zeighami, Aditya G. Parameswaran, and Eugene Wu. 2025. Querying Templatized Docu- ment Collections with Large Language Models. In41st IEEE International Confer- ence on Data Engineering (ICDE). IEEE, 2422–2435

    2025

  24. [32]

    Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael J

    Chunwei Liu, Matthew Russo, Michael J. Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael J. Franklin, Tim Kraska, Samuel Madden, Rana Shahout, and Gerardo Vitagliano. 2025. Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing. In15th Conference o...

    2025

  25. [33]

    Parameswaran

    Ruiying Ma, Yiming Lin, and Aditya G. Parameswaran. 2025. Robust Hierarchical Structures for Document Analytics. https://people.eecs.berkeley.edu/~adityagp/ papers/shed-tr.pdf. Technical report

    2025

  26. [34]

    Microsoft. [n. d.]. Azure AI Document Intelligence. https://azure.microsoft.com/ en-us/products/ai-services/ai-document-intelligence. 2026

    2026

  27. [35]

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations

    2025

  28. [36]

    OpenAI. [n. d.]. GPT-4 Vision (GPT-4V). https://developers.openai.com/api/docs/ guides/images-vision. 2026

    2026

  29. [37]

    OpenAI. 2025. Codex. https://openai.com/codex/

    2025

  30. [38]

    Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen, et al

  31. [39]

    Suresh Parthasarathy, Lincy Pattanaik, Anirudh Khatry, Arun Iyer, Arjun Rad- hakrishna, Sriram K Rajamani, and Mohammad Raza. 2022. Landmarks and regions: a robust approach to data extraction. InProceedings of the 43rd ACM SIGPLAN International Conference on Programming Langua...

    2022

  32. [40]

    Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic operators and their optimization: En- abling llm-based data processing with accuracy guarantees in lotus.Proceedings of the VLDB Endowment18, 11 (2025), 4171–4184

    2025

  33. [41]

    Ritesh Sarkhel and Arnab Nandi. 2021. Improving Information Extraction from Visually Rich Documents using Visual Span Representations.Proceedings of the VLDB Endowment14, 5 (2021), 822–834

    2021

  34. [42]

    Parameswaran, and Eugene Wu

    Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing.Proc. VLDB Endow.18, 9 (2025), 3035–3048

    2025

  35. [43]

    Petr Slavík. 1997. Improved Performance of the Greedy Algorithm for Partial Cover.Inform. Process. Lett.64, 5 (1997), 251–254

    1997

  36. [44]

    Li Sun, Liu He, Shuyue Jia, Yangfan He, and Chenyu You. 2025. Docagent: An agentic framework for multi-modal long-context document understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 17712–17727

    2025

  37. [45]

    Zhaoze Sun, Chengliang Chai, Qiyan Deng, Kaisen Jin, Xinyu Guo, Han Han, Ye Yuan, Guoren Wang, and Lei Cao. 2025. QUEST: Query Optimization in Unstructured Document Analysis.Proc. VLDB Endow.18, 11 (2025), 4560–4573

    2025

  38. [46]

    Wendt, Lauro Beltr ao Costa, Marc Najork, and Beliz Gunel

    Sandeep Tata, Navneet Potti, James B. Wendt, Lauro Beltr ao Costa, Marc Najork, and Beliz Gunel. 2021. Glean: Structured Extractions from Templatic Documents. Proceedings of the VLDB Endowment14, 6 (2021), 997–1005

    2021

  39. [47]

    Vazirani

    Vijay V. Vazirani. 2001.Approximation Algorithms. Springer, Berlin, Heidelberg

    2001

  40. [48]

    Jiayi Wang and Guoliang Li. 2025. AOP: Automated and Interactive LLM Pipeline Orchestration for Answering Complex Queries. In15th Conference on Innovative 16 Data Systems Research (CIDR)

    2025

  41. [49]

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutLM: Pre-training of Text and Layout for Document Image Understanding. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, 1192–1200

    2020

  42. [50]

    Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran. 2025. Cut costs, not accuracy: Llm-powered data processing with guarantees.Proceedings of the ACM on Management of Data3, 6 (2025), 1–26

    2025

  43. [51]

    Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2024. LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration. InProceedings of the 2024 Conference on Empirical Methods in Natural Lang...

    2024

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing LLM-as-a-Judge with MT-Bench and Chatbot Arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623. 17

    2023

  45. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    Form2Seq: A Framework for Higher-Order Form Structure Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 3830–3840

    2020

  46. [2026]

    arXiv preprint arXiv:2603.08655(2026)

    Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning. arXiv preprint arXiv:2603.08655(2026)

    2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.