Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

BARGAIN claims a model cascade can meet an accuracy, precision, or recall target with probability 1−δ while avoiding up to 86% more expensive-LLM calls than prior methods — via adaptive sampling plus an anytime-valid betting test.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 11:16 UTC pith:SHCITBJ4

load-bearing objection New adaptive cascade-threshold selection with betting-based tests and real utility gains; one unresolved sample-reuse issue in the anytime-valid argument needs a fix before the guarantees are bulletproof. the 2 major comments →

arxiv 2509.02896 v2 pith:SHCITBJ4 submitted 2025-09-02 cs.DB cs.AI

Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees

Sepanta Zeighami , Shreya Shankar , Aditya Parameswaran This is my paper
classification cs.DB cs.AI
keywords model cascadesLLM data processingfinite-sample quality guaranteesanytime-valid hypothesis testingadaptive samplingprecision and recall targetscost-quality trade-offbetting-based confidence bounds
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

BARGAIN takes on a simple, expensive practice — calling a top-tier LLM on every record of a large text dataset — and tries to make it mostly unnecessary without forfeiting quality. The paper's claim is that a model cascade, which lets a cheap LLM answer records where its own confidence is high and saves the expensive model for the rest, can be tuned so that the final output meets a user-specified accuracy, precision, or recall target with probability at least 1−δ, in finite samples and not just asymptotically. The technical core is replacing the previous approach's central-limit estimates with an anytime-valid betting-based hypothesis test, combined with sampling that adapts to the target and reuses labels across thresholds. Empirically the paper reports up to 86% more oracle calls avoided than the state of the art on accuracy-target queries, and large recall and precision gains on filtering queries, across eight real-world datasets. A sympathetic reader would care because it suggests the cost of LLM-powered data processing can be cut substantially while keeping a formal, checkable quality promise.

Core claim

BARGAIN's central claim: cascade-threshold selection can be strict and data-adaptive at once — every threshold it certifies meets the accuracy, precision, or recall target with probability at least 1−δ in finite samples, while sampling and estimation adapt to the target and label distribution. The method walks candidate thresholds downward, drawing records above each until an anytime-valid betting test certifies it, reusing labels across thresholds; the test's false-positive probability is bounded at every stopping time, unlike prior CLT-based guarantees that hold only asymptotically. For recall queries, an impossibility result forces the guarantee onto a dense positive subset, with a parame

What carries the argument

The load-bearing object is the estimation function E_BARGAIN: an anytime-valid betting-based hypothesis test (the paper's Lemma 2.1, drawn from its reference [43]) that decides whether a candidate cascade threshold ρ has true quality at least the target T. Its defining property is that the chance of certifying a threshold whose true quality is below T is bounded by α at every stopping time, even under sequential sampling without replacement and reuse of labels across lower thresholds; this turns the adaptive sampling loop into a 1−δ guarantee. Supporting the test are a selection rule (Algorithm 2 and the tolerance η of Lemma 3.5) that keeps the failure probability from growing with the candi

Load-bearing premise

The load-bearing premise is that the anytime-valid false-positive bound of the betting hypothesis test continues to hold when sampling is sequential and without replacement and the same oracle labels are reused across decreasing thresholds — a one-paragraph reduction (Appx. B.3.1–B.3.2) on which every BARGAIN variant's guarantee rests; separately, for recall-target queries the guarantee is proven on the dense subset dense_β(D), and the gap to full-dataset recall is only contr

What would settle it

Adversarially flip the labels of the lowest-scoring records on a benchmark dataset (the paper's own Imagenet probe), run BARGAIN_P-A and BARGAIN_R-U over many seeds with δ=0.1, and count failures — thresholds whose true quality is below T; the observed failure rate must never exceed 0.1 at any sampling horizon, since the bound is anytime-valid. Exceeding it at an intermediate stopping time, or on a dataset whose precision oscillates around T, would falsify the reduction the guarantees rest on.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Accuracy-target queries can avoid up to 86% more oracle calls than SUPG across the eight benchmark datasets while replacing an asymptotic promise with a finite-sample bound P(accuracy ≥ T) ≥ 1−δ.
  • Under a fixed oracle budget, precision-target queries roughly double recall on sparse-positive datasets (up to +118% over SUPG) and recall-target queries raise precision (up to +19%), because adaptive sampling concentrates labels where they inform the threshold choice.
  • The anytime-valid bound means no sample size needs to be fixed in advance: BARGAIN can keep drawing labels for a threshold until the test certifies it, or stop when the budget runs out, without weakening the guarantee.
  • For recall targets, the impossibility result implies a fundamental limit: on datasets with very few true positives, no monotone-sampling cascade can guarantee full-dataset recall and still return high precision; BARGAIN's β-relaxation quantifies exactly how much guarantee scope is traded for utility.
  • A multiclass variant (BARGAIN_A-M) extends the same guarantee to per-class thresholds, sharing the allowed failure probability δ across classes by union bound, which helps when the cheap model is unevenly calibrated across classes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My extension: the anytime-valid estimation core is not specific to LLM cascades — any pipeline that must verify a quality claim about a large batch against an expensive gold labeler (human review, fact-checking APIs, slow retrieval re-rankers) could reuse the same stop-when-certified pattern, provided a cheap confidence score is available.
  • My extension: the guarantee is clean, but the savings hinge on calibration — the paper's own noise experiments show utility collapses to the Naive baseline when proxy scores are scrambled, so a deployment should first measure score-quality correlation; the paper does not propose such a pre-check.
  • My extension: the paper sets η=0 because real precision curves look monotone, yet nothing in the guarantee requires monotonicity; constructing a dataset whose precision oscillates around T would expose exactly how much utility the worst-case analysis leaves on the table, which the paper does not quantify.
  • My extension: for RT queries the guarantee lives on dense_β(D), and the paper only empirically bounds the gap to full-dataset recall; a direct extension would report recall on the ignored low-density region separately, making the β trade-off auditable on new datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents BARGAIN, a model-cascade threshold selection method for LLM-based data processing under accuracy (AT), precision (PT), and recall (RT) targets. The approach combines adaptive sampling, hypothesis-testing estimators based on Waudby-Smith and Ramdas' anytime-valid betting bounds, and a threshold selection rule with a tolerance parameter. The paper claims finite-sample (1-delta) quality guarantees, reports large empirical gains over the SUPG baseline on eight datasets, and includes an impossibility result motivating a beta-relaxed RT guarantee. The main technical proofs are in Appendix B, with Lemma B.4 as the key guarantee for the adaptive PT/AT variants.

Significance. If the guarantees are correct, this is a practically important contribution: it directly addresses a known weakness of SUPG (asymptotic-only guarantees), brings modern anytime-valid inference into cascade selection, and the empirical study is broad. The paper also deserves credit for openly disclosing the beta-relaxation for RT queries and for providing an impossibility result. However, the central proof for the adaptive variants contains a load-bearing gap: the sample-reuse argument in Appendix B.3.2 does not establish the required anytime-valid bound. The uniform-sampling variants (BARGAIN_P-U, BARGAIN_R-U) and the naive baseline appear less affected, but the headline empirical improvements are driven by the adaptive variants whose guarantees are currently unsupported.

major comments (2)
  1. [Appx. B.3.2 / Lemma B.4] The proof that sample reuse preserves the anytime-valid bound is not valid. Lemma B.3 (and WSR Theorem 4) requires X1,...,Xk to be the initial segment of a uniform random permutation of D_rho, with no prior conditioning on the labels of these same records. In BARGAIN_P-A, the algorithm reaches a smaller threshold rho only after tests at larger thresholds returned 1; those tests used records in D_rho (since D_rho' is a subset of D_rho for rho'>rho). Conditional on reaching rho, the first elements of the sequence for rho are biased toward the records that made the earlier tests cross the 1/delta threshold. Thus the K_WR statistic in Eq. (19) can start with capital already inflated even when P_D(rho)<T. The statement in the proof of Lemma B.4 that 'sampling elements of D_rho in order forms a uniform sample set without replacement' is true only unconditionally; the theorem invoked does not c
  2. [Sec. 4.2 / Lemma B.12] The stated RT guarantee is not for full-dataset recall. The text says BARGAIN_R-A guarantees R_beta_D(rho)>=T (recall on the dense subset dense_beta(D)), while Lemma B.12 in Appx. B.6.3 states P(R_D(rho_S)<T)<=delta for datasets 'with dense positive labels', without defining this class in the lemma statement. If 'dense positive labels' is intended to mean R_D = R_beta_D, it must be defined and proved; otherwise the lemma is inconsistent with Sec. 4.2. Because the abstract and contributions say BARGAIN supports guarantees on recall, this caveat needs to be stated prominently and the formal statements aligned. This is not fatal given the explicit beta relaxation, but it is load-bearing for the RT claim.
minor comments (5)
  1. [Appx. B.1] Lemma B.1 statements (14) and (16) contain a stray '=1' inside the indicator expression, and the notation 'P(...=1)' is confusing. Please clean up the formal statements to match the simplified presentation in Sec. 2.3.
  2. [Appx. C.1] The proof overview refers to 'Theorem B.1' and 'Theorem B.2' where the actual statements are Lemmas B.1 and B.2, and 'Theorems B.8 and 3.4' should be 'Lemmas B.8 and 3.4'. This makes the appendix harder to follow.
  3. [Appx. B.5.2 / proof of Lemma B.9] The text says 'we can take the minimum over rho in C', but Eq. (13) and the surrounding argument use a maximum. Please fix the wording to 'maximum'.
  4. [Figure 3] The caption and the pink/orange markings are dense and difficult to parse; clarifying the relationship between cumulative samples, cumulative records, and oracle calls avoided would improve readability.
  5. [Table 9 / Appx. G.2] The standard deviations are informative, but no test of significance is reported for the main utility comparisons in Table 5. Adding paired tests or confidence intervals would strengthen the empirical claims.

Circularity Check

0 steps flagged

No significant circularity: the quality guarantees are derived from external WSR betting supermartingales plus union bounds, not from fitted values or load-bearing self-citations. The only flagged gap is an unproven sample-reuse assertion in Appendix B.3.2, which is a correctness risk rather than circularity.

full rationale

The central claim (Lemmas 3.6, B.4, B.6, B.9, B.12) is a probabilistic guarantee that the returned cascade threshold meets the user target with probability at least 1−δ. The derivation chain is: define EBARGAIN as the Waudby-Smith–Ramdas betting test (Lemma 2.1 / B.1 / B.2, external references [43]); show false-positive control for fixed and anytime sampling (Lemmas 3.4, B.3, B.5, B.8, B.10); combine with union-bound selection (Lemma 3.5) or monotonicity of recall (Lemma B.9). No parameter is fitted to the data and then renamed as a prediction; the variance-aware estimator is taken as a black box from external work, and the η/β relaxations are explicit user parameters, not hidden fitted values. Self-citations in the paper (related-work references to the authors' own systems) are not load-bearing for the guarantees. The one passage that merits flagging is Appendix B.3.2: 'We show our theoretical guarantees still hold when performing such sample reuse,' followed by Lemma B.4's proof which only says 'sampling elements of ˚D_rho in order forms a uniform sample set without replacement from D_rho.' If samples from earlier, larger thresholds are reused as the prefix for a smaller threshold, that prefix is not generally the initial segment of a uniform random permutation of D_rho, so the reduction to WSR Theorem 4 (Lemma B.2/B.3) is not established by the proof. This is a potential invalidation of the anytime-valid guarantee (a correctness risk), not an equivalence-by-construction or fitted-input circularity, so it does not raise the circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The guarantees lean on an external published result (Waudby-Smith and Ramdas theorems), a clear data-access model (oracle labels, negligible proxy cost), and an empirically motivated but unproven density concentration assumption for the RT variant. The main user-set parameters (M, c, eta, beta, r) are chosen by hand and validated by sensitivity experiments; none are fitted to force the utility results.

free parameters (5)
  • M (number of candidate thresholds) = 20 (default)
    Controls precision vs sampling budget; authors show diminishing returns for M>20 (Sec. 5, Appx. G.3).
  • c (minimum samples per threshold before failing) = 1%-5% of dataset size (default)
    Stops premature rejection of thresholds; sensitivity low across tested range (Sec. 5, Appx. G.3).
  • eta (tolerance parameter) = 0 (default)
    Set to 0 because precision/accuracy appear monotone in proxy score in real data (Sec. 3.2.2, Fig. 6).
  • beta (minimum positive density) = 0.02 (default)
    For RT queries, trades guarantee strength against utility; larger beta can cause the method to ignore positives in low-density regions and miss the user's recall target (Sec. 4.2, Fig. 16).
  • r (positive-density resolution) = 150 (default)
    Window size defining D_rho^r; affects the quality of the density estimate used in BARGAIN_R-A (Sec. 4.2, Sec. 6.1).
axioms (3)
  • standard math The Waudby-Smith and Ramdas betting-based hypothesis test (Theorems 3 and 4) bounds the false-positive probability of the anytime-valid estimator.
    The estimation functions EBARGAIN, EBARGAIN^A, EBARGAIN^R, and EBARGAIN^d all invoke this external result as an unproved primitive (Lemma 2.1, Lemmas B.1 and B.2).
  • domain assumption Oracle labels are ground truth, and every sampled record is labeled exactly; proxy score is available for every record at negligible cost.
    The whole sampling procedure (Sec. 2.1) and the cost model (proxy cost negligible) rely on this; if proxy cost is non-negligible, savings estimates change (acknowledged in Appx. D).
  • domain assumption For RT queries, positive labels are densely distributed at high proxy scores, so restricting guarantees to dense-beta(D) approximately preserves the user's recall target.
    BARGAIN_R-A's guarantee is on R^beta_D, not R_D, and the gap is only small when density is concentrated at high scores (Fig. 9, Fig. 16(a)); the paper says this is an optional relaxation.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 39952 in / 11522 out tokens · 124972 ms · 2026-08-05T11:16:53.767619+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees." pith.science (2026). https://pith.science/paper/SHCITBJ4

@misc{pith2026250902896,
  author       = {Pith},
  title        = {Pith review of: Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHCITBJ4}},
  note         = {Machine review of arXiv:2509.02896}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) are being increasingly used as a building block in data systems to process large text datasets. To do so, LLM model providers offer multiple LLMs with different sizes, spanning various cost-quality trade-offs when processing text at scale. Top-of-the-line LLMs (e.g., GPT-4o, Claude Sonnet) operate with high accuracy but are prohibitively expensive when processing many records. To avoid high costs, more affordable but lower quality LLMs (e.g., GPT-4o-mini, Claude Haiku) can be used to process records, but we need to ensure that the overall accuracy does not deviate substantially from that of the top-of-the-line LLMs. The model cascade framework provides a blueprint to manage this trade-off, by using the confidence of LLMs in their output (e.g., log-probabilities) to decide on which records to use the affordable LLM. However, existing solutions following this framework provide only marginal cost savings and weak theoretical guarantees because of poor estimation of the quality of the affordable LLM's outputs. We present BARGAIN, a method that judiciously uses affordable LLMs in data processing to significantly reduce cost while providing strong theoretical guarantees on the solution quality. BARGAIN employs a novel adaptive sampling strategy and statistical estimation procedure that uses data and task characteristics and builds on recent statistical tools to make accurate estimations with tight theoretical guarantees. Variants of BARGAIN can support guarantees on accuracy, precision, or recall of the output. Experimental results across 8 real-world datasets show that BARGAIN reduces cost, on average, by up to 86% more than state-of-the-art, while providing stronger theoretical guarantees on accuracy of output, with similar gains when guaranteeing a desired level of precision or recall.

Figures

Figures reproduced from arXiv: 2509.02896 by Aditya Parameswaran, Sepanta Zeighami, Shreya Shankar.

Figure 1
Figure 1. Figure 1: Overview of Model Cascade users can often tolerate some marginal quality degradation as long as we can reduce cost substantially, e.g., if the system is guaranteed to match the top-of-the-line LLM’s output 90% of the time but at half the cost. The system then needs to decide when to use which LLM to minimize cost while guaranteeing this desired answer quality. A common paradigm to decide when to use the af… view at source ↗
Figure 2
Figure 2. Figure 2: Summary of AT Query Results [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: An example of cascade approaches for PT queries [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of dif￾ferent estimation methods 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Cascade Threshold, 0.00 0.25 0.50 0.75 1.00 D( ) Tacred NS Reviews Court Screenplay Imagenet [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Estimation with a fixed uniform sample 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Cascade Threshold, 0.00 0.25 0.50 0.75 1.00 P r e cisio n E stim ate s at D( ) for Court D( ) for Reviews BARGAIN for Court BARGAIN for Reviews [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: BARGAIN𝑅-A ex￾ample Algorithm 4 BARGAIN𝑅-A 1: 𝑘1, 𝑘2 ← 𝑘/2 2: 𝛿1, 𝛿2 ← 𝛿/2 3: 𝜌𝑃 ← 0, 𝜌 ← 0.5 4: while 𝑘1 ≥ 0 do 5: 𝑆 ← ∅ 6: while E BARGAIN 𝑑 (𝑆, 𝛽, 𝜌, 𝛿1 ) = 0 and 𝑘1 ≥ 0 do 7: Sample a record uniformly from 𝐷 𝜌 𝑟 and add to 𝑆 8: 𝑘1− = 1 9: if E BARGAIN 𝑑 (𝑆, 𝛽, 𝜌, 𝛿1 ) = 0 then 10: break 11: 𝜌𝑃 ← 𝜌, 𝜌 ← (1 + 𝜌)/2 12: return BARGAIN𝑅-U sampling 𝑘2 points over [𝜌𝑃 , 1] with 𝛿2 estimated to be in the rang… view at source ↗
Figure 11
Figure 11. Figure 11: Meeting target in Onto Dataset 100 200 300 400 500 k 0.00 0.25 0.50 0.75 1.00 Observed Recall (a) Reviews 100 200 400 800 1600 k 0.00 0.25 0.50 0.75 1.00 (b) Onto Naive SUPG BARGAINP-U BARGAINP-A [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Impact of 𝑘 in RT Queries 0.75 0.80 0.85 0.90 0.95 T 0.0 0.2 0.4 0.6 % of Oracle Calls Avoided (a) Reviews 0.75 0.80 0.85 0.90 0.95 T 0.00 0.25 0.50 0.75 1.00 (b) Onto BARGAINA-A BARGAINA-M SUPG Naive [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗
Figure 20
Figure 20. Figure 20: Impact of 𝜂 and 𝑐 on AT Queries You must respond with ONLY True or False: Screenplay Classification I will give you a screenplay of a movie. Your task is to determine if the protagonist makes a critical decision based on false information. - True if the protagonist makes an important decision based on information that is incorrect or misleading - False if the protagonist's key decisions are based on accur… view at source ↗
Figure 21
Figure 21. Figure 21: Impact of 𝑀 increases further, since spending too many samples on thresholds that does not meet the target causes wasting samples. Nonetheless, [PITH_FULL_IMAGE:figures/full_fig_p022_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kalypso: Relational LLM Serving

    cs.DB 2026-07 conditional novelty 7.0

    Query-aware LLM serving with adaptive memory-aware pipelining reuses cross-operator KV-cache state and speeds semantic queries by up to 4.57×.

  2. Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

    cs.DB 2026-06 unverdicted novelty 7.0

    An adaptive two-phase semantic filter using clustering then a hybrid proxy trained on LLM confidence achieves 1.6-2.0x speedup over prior methods at 90% accuracy on 10K document corpora.

Reference graph

Works this paper leans on

50 extracted references · 26 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    Eric Anderson, Jonathan Fritz, Austin Lee, Bohou Li, Mark Lindblad, Henry Lindeman, Alex Meyer, Parth Parmar, Tanvi Ranade, Mehul A Shah, et al. 2024. The Design of an LLM-powered Unstructured Analytics System.arXiv preprint arXiv:2409.00847(2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Michael R Anderson, Michael Cafarella, German Ros, and Thomas F Wenisch

    work page

  3. [3]

    Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language models enable simple systems for generating structured views of heterogeneous data lakes.arXiv preprint arXiv:2304.09433(2023)

    Pith/arXiv arXiv 2023

  4. [4]

    Favyen Bastani and Samuel Madden. 2022. OTIF: Efficient tracker pre-processing over large video datasets. InProceedings of the 2022 International Conference on Management of Data. 2091–2104

    work page 2022

  5. [5]

    Jiashen Cao, Karan Sarkar, Ramyad Hadidi, Joy Arulraj, and Hyesoon Kim. 2022. Figo: Fine-grained query optimization in video analytics. InProceedings of the 2022 International conference on management of data. 559–572

    work page 2022

  6. [6]

    Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for the Analysis of Conversations. InProceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 57–60

    work page 2020

  7. [7]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176(2023)

    Pith/arXiv arXiv 2023

  8. [8]

    courtlistener. [n.d.]. Court Opinion. Retrieved April, 2025 from courtlistener.com

    work page 2025

  9. [9]

    Dujian Ding, Sihem Amer-Yahia, and Laks Lakshmanan. 2022. On Efficient Approximate Queries over Machine Learning Models.Proceedings of the VLDB Endowment (PVLDB)16, 4 (2022), 918–931

    work page 2022

  10. [10]

    Raul Castro Fernandez, Aaron J Elmore, Michael J Franklin, Sanjay Krishnan, and Chenhao Tan. 2023. How large language models will disrupt data management. Proceedings of the VLDB Endowment16, 11 (2023), 3302–3309

    work page 2023

  11. [11]

    Juliana Freire, Grace Fan, Benjamin Feuer, Christos Koutras, Yurong Liu, Ed- uardo Pena, Aécio Santos, Cláudio Silva, and Eden Wu. [n.d.]. Large Language Models for Data Discovery and Integration: Challenges and Opportunities.Data Engineering([n. d.]), 3

    work page

  12. [12]

    Wassily Hoeffding. 1994. Probability inequalities for sums of bounded random variables.The collected works of Wassily Hoeffding(1994), 409–426

    1994

  13. [13]

    Keke Huang, Yimin Shi, Dujian Ding, Yifei Li, Yang Fei, Laks Lakshmanan, and Xiaokui Xiao. 2025. ThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries.arXiv preprint arXiv:2501.04901(2025)

    arXiv 2025

  14. [14]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering.Transactions of the Association for Computational Linguistics 9 (2021), 962–977

    work page 2021

  15. [15]

    Saehan Jo and Immanuel Trummer. 2024. SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees.arXiv preprint arXiv:2403.13835(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    Kaggle. [n.d.]. Screenplay. Retrieved April, 2025 from https://www.kaggle.com/ datasets/gufukuro/movie-scripts-corpus

    work page 2025

  17. [17]

    Kaggle. [n.d.]. Steam game reviews. Retrieved April, 2025 from https://www. kaggle.com/datasets/najzeko/steam-reviews-2021

    work page 2025

  18. [18]

    Daniel Kang, Peter Bailis, and Matei Zaharia. [n.d.]. BlazeIt: Optimizing Declara- tive Aggregation and Limit Queries for Neural Network-Based Video Analytics. Proceedings of the VLDB Endowment13, 4 ([n. d.])

    work page

  19. [19]

    Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, and Matei Zaharia. 2017. NoScope: Optimizing Neural Network Queries over Video at Scale.Proceedings of the VLDB Endowment10, 11 (2017)

    2017

  20. [20]

    Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia. [n.d.]. SUPG code and datasets. Retrieved Mar, 2025 from https://github.com/ stanford-futuredata/supg

    work page 2025

  21. [21]

    Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia

  22. [22]

    Daniel Kang, John Guibas, Peter Bailis, Tatsunori Hashimoto, Yi Sun, and Matei Zaharia. 2021. Accelerating Approximate Aggregation Queries with Expensive Predicates.Proc. VLDB Endow.14 (2021), 2341–2354. https://api.semanticscholar. org/CorpusID:237012342

    work page 2021

  23. [23]

    Daniel Kang, John Guibas, Peter D Bailis, Tatsunori Hashimoto, and Matei Za- haria. 2022. Tasti: Semantic indexes for machine learning-based queries over unstructured data. InProceedings of the 2022 International Conference on Man- agement of Data. 1934–1947

    work page 2022

  24. [24]

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. Calibration-tuning: Teaching large lan- guage models to know what they don’t know. InProceedings of the 1st Workshop on Uncertainty-A ware NLP (UncertaiNLP 2024). 1–14

    work page 2024

  25. [25]

    Alexander W Lee, Justin Chan, Michael Fu, Nicolas Kim, Akshay Mehta, Deepti Raghavan, and Ugur Cetintemel. 2025. Semantic Integrity Constraints: Declar- ative Guardrails for AI-Augmented Data Processing Systems.arXiv preprint arXiv:2503.00600(2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Yiming Lin, Madelon Hulsebos, Ruiying Ma, Shreya Shankar, Sepanta Zeigham, Aditya G Parameswaran, and Eugene Wu. 2024. Towards accurate and efficient document analytics with large language models.arXiv preprint arXiv:2405.04674 (2024)

    Pith/arXiv arXiv 2024

  27. [27]

    Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano

  28. [28]

    Yao Lu, Aakanksha Chowdhery, Srikanth Kandula, and Surajit Chaudhuri. 2018. Accelerating machine learning inference with probabilistic predicates. InPro- ceedings of the 2018 International Conference on Management of Data. 1493–1508

    2018

  29. [29]

    2017.Probability and computing: Random- ization and probabilistic techniques in algorithms and data analysis

    Michael Mitzenmacher and Eli Upfal. 2017.Probability and computing: Random- ization and probabilistic techniques in algorithms and data analysis. Cambridge university press

    work page 2017

  30. [30]

    Zan Ahmad Naeem, Mohammad Shahmeer Ahmad, Mohamed Eltabakh, Mourad Ouzzani, and Nan Tang. 2024. RetClean: Retrieval-Based Data Cleaning Using LLMs and Data Lakes.Proceedings of the VLDB Endowment17, 12 (2024), 4421– 4424

    work page 2024

  31. [31]

    Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré

  32. [32]

    openai. 2025. OpenAI pricing. Retrieved Mar, 2025 from https://openai.com/ api/pricing/

    work page 2025

  33. [33]

    Liana Patel, Siddharth Jha, Carlos Guestrin, and Matei Zaharia. 2024. Lotus: Enabling semantic queries with llms over tables of unstructured and structured data.arXiv preprint arXiv:2407.11418(2024)

    Pith/arXiv arXiv 2024

  34. [34]

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik

  35. [35]

    Matthew Russo, Tatsunori Hashimoto, Daniel Kang, Yi Sun, and Matei Zaharia

  36. [36]

    Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G Parameswaran, and Eugene Wu. 2024. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing.arXiv preprint arXiv:2410.12189(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Matthias Urban and Carsten Binnig. 2024. Demonstrating CAESURA: Language Models as Multi-Modal Query Planners. InCompanion of the 2024 International Conference on Management of Data. 472–475

    2024

  38. [38]

    Matthias Urban and Carsten Binnig. 2024. ELEET: Efficient Learned Query Execution over Text and Tables.Proc. VLDB Endow17 (2024), 13

    2024

  39. [39]

    Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql.arXiv preprint arXiv:2410.01943(2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Cheng Wang. 2023. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222(2023)

    Pith/arXiv arXiv 2023

  41. [41]

    Jiayi Wang and Guoliang Li. 2025. Aop: Automated and interactive llm pipeline orchestration for answering complex queries. CIDR

    2025

  42. [42]

    Qiming Wang and Raul Castro Fernandez. 2023. Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach.Proceedings of the ACM on Management of Data1, 4 (2023), 1–27

    work page 2023

  43. [43]

    Ian Waudby-Smith and Aaditya Ramdas. 2024. Estimating means of bounded random variables by betting.Journal of the Royal Statistical Society Series B: Statistical Methodology86, 1 (2024), 1–27

    work page 2024

  44. [44]

    Sepanta Zeighami, Yiming Lin, Shreya Shankar, and Aditya Parameswaran. 2025. LLM-Powered Proactive Data Systems.arXiv preprint arXiv:2502.13016(2025). A Overview This appendix is organized as follows. • Section B presents formal statements and additional details for theoretical results discussed in the paper. • Section C present proofs of all results, bot...

    Pith/arXiv arXiv 2025

  45. [45]

    David Vos, Till Döhmen, and Sebastian Schelter. 2022. Towards parameter- efficient automation of data wrangling tasks with prefix-tuning. InNeurIPS 2022 First Table Representation Workshop

    work page 2022

  46. [2019]

    In2019 IEEE 35th International Conference on Data Engineering (ICDE)

    Physical representation-based predicate optimization for a visual analytics database. In2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 1466–1477

    work page

  47. [2020]

    Approximate Selection with Guarantees using Proxies.Proceedings of the VLDB Endowment13, 11 (2020). 14

    work page 2020

  48. [2022]

    Can foundation models wrangle your data?arXiv preprint arXiv:2205.09911 (2022)

    Pith/arXiv arXiv 2022

  49. [2023]

    Accelerating Aggregation Queries on Unstructured Streams of Data.Pro- ceedings of the VLDB Endowment16, 11 (2023), 2897–2910

    work page 2023

  50. [2024]

    A declarative system for optimizing ai workloads.arXiv preprint arXiv:2405.14696(2024)

    Pith/arXiv arXiv 2024