Pith. sign in

REVIEW 3 major objections 25 references

When the database and workload are known, synthesizing a cardinality estimator as executable code beats generic statistics and turns better estimates into faster plans.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-14 18:09 UTC pith:NTCUE5XA

load-bearing objection Real third path for cardinality estimation: LLM-synthesized executable estimators with staged feedback deliver large JOB/JOB-Complex gains, but headline numbers rest on single synthesis runs and a couple of write-up inconsistencies. the 3 major comments →

arxiv 2606.09361 v2 pith:NTCUE5XA submitted 2026-06-08 cs.DB

Bespoke-Card: Why Tune When You Can Generate? Synthesizing Workload-Specific Cardinality Estimators

classification cs.DB
keywords cardinality estimationquery optimizationcode synthesisworkload specializationagent-driven systemsjoin-order benchmarkq-errorPostgreSQL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Generic cardinality estimators must work for any schema and workload, so they rely on compact, assumption-heavy statistics that often fail on realistic joins and filters. This paper argues that if the database and a representative workload are known in advance, the right move is not to retune a fixed estimator but to generate a new one: program code whose statistics and estimation logic are specialized to that contract. Bespoke-Card uses a planning agent to design the statistics strategy, a coding agent to implement it, and a deterministic validator that scores estimates against true cardinalities and a mature baseline, feeding structured error signals back in stages that isolate joins, filters, and their interaction. On standard hard join benchmarks, injecting the synthesized estimates into a production optimizer cuts total runtime by about a third to two thirds and shrinks the heavy tail of estimation error by orders of magnitude, for under an hour and under ten dollars of synthesis cost. The claim is that executable, workload-specific synthesis is a third path beside classical generic estimators and fixed learned model families.

Core claim

Bespoke-Card shows that a planner/coder agent loop, closed by deterministic subplan-level feedback against true cardinalities and a production baseline, can synthesize an inspectable Python cardinality estimator specialized to a fixed database and tuned for a declared workload. When those estimates are injected into PostgreSQL’s optimizer, total runtime falls 33% on JOB and 68% on JOB-Complex, and median q-error over all JOB subplans falls from 190.5 to 11.5, with the upper tail compressed by roughly two orders of magnitude.

What carries the argument

The structured synthesis harness: a planning agent that designs workload-specific statistics and estimation strategy, a coding agent that emits a card_estimator module over structured SPAJ requests (not raw SQL), and a deterministic evaluator that returns staged, structured feedback—join-only, filter-only, then full subplans—including q-error percentiles, regression rates versus PostgreSQL, and concrete outlier subplans, with archival selection of the best candidate.

Load-bearing premise

A representative, relatively stable workload is known up front, and true cardinalities can be collected once for the subplans used to score and repair the estimator.

What would settle it

Resynthesize an estimator for JOB or JOB-Complex under the same harness, inject its estimates into the same optimizer, and check whether end-to-end total runtime and the full subplan q-error distribution fail to beat the production baseline by the reported margins; a null result on either accuracy or plan quality would undercut the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For databases with known schemas and recurring workloads, optimizers can ship a regenerable, inspectable estimator artifact instead of only tuning a general statistics catalog.
  • Staged isolation of join errors, filter errors, and their interaction becomes a reusable repair curriculum for any synthesized estimator under measurable feedback.
  • Estimator quality can be improved without choosing a fixed neural architecture or collecting large labeled training sets for every model family.
  • When data, schema, or workload shifts, the same loop can rebuild or re-specialize the estimator rather than maintaining a permanent universal component.
  • The same feedback-driven code-synthesis pattern is proposed as applicable to other optimizer pieces that admit deterministic scoring, such as cost models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If true-cardinality labeling for feedback remains the expensive step, cheaper approximate oracles or partial labeling schedules would become the main lever for wider adoption.
  • The largest gains on the harder complex-join workload suggest the method’s value is highest where independence and uniformity assumptions break most, not on simple single-table filters.
  • Human audit of the modest-size generated code could become part of production governance, since the artifact is readable Python rather than an opaque model checkpoint.
  • Joint synthesis of cardinality logic with neighboring optimizer components could compound plan-quality gains beyond cardinality alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. Bespoke-Card reframes cardinality estimation as LLM-driven synthesis of an executable, workload-specific estimator for a fixed database. A planner designs statistics and estimation strategies; a coder implements them as Python; a deterministic harness scores candidates against true subplan cardinalities and PostgreSQL, with a staged curriculum (join-only, filter-only, full subplans), structured q-error/regression feedback, outlier cases, and archival selection. On IMDb, injecting the synthesized estimates into PostgreSQL reduces total runtime by 33% on JOB and 68% on JOB-Complex versus PostgreSQL’s own estimator, with large median and tail q-error reductions (Table 3), small statistics footprints, and synthesis cost under one hour and $10.

Significance. If the results hold under modest robustness checks, the paper opens a genuine third path beside classical catalogs and fixed learned architectures: treating an optimizer component as code to be synthesized under measurable feedback. Strengths include end-to-end plan injection (not only offline q-error), a clear staged-feedback ablation (Figure 5), an inspectable strategy inventory (Table 4), artifact availability, and explicit cost/size reporting. The database-and-workload contract is a coherent specialization thesis rather than an unstated limitation. That combination is significant for query optimization and for the broader “bespoke systems” agenda.

major comments (3)
  1. Headline runtime and q-error claims (§5.2–5.4, Fig. 1, Table 3) rest on a single GPT-5.4 synthesis trajectory per workload (§5.1, §5.8: 88 turns, $9.74 on JOB). LLM coding is stochastic; different seeds can change the statistics plan, join/filter logic, and residual regressions. Without multi-seed runs (or at least variance on median/tail q-error and total runtime), the 33% and 68% deltas are not yet established as stable properties of the method. Please report several independent synthesis runs, or justify why a single trajectory is sufficient and add a sensitivity discussion.
  2. Accuracy and runtime evaluation use the same workload subplans that drive the staged feedback curriculum (§4.3, §5.3–5.4). The paper also claims support for arbitrary SPAJ requests over the target database (§3.1, §4.2). Those claims are not separated: there is no hold-out query set, template shift, or literal perturbation. At minimum, evaluate on held-out JOB/JOB-Complex queries or a second SPAJ sample over IMDb that was not used in feedback, and report whether plan quality and q-error degrade.
  3. Numerical reporting of the JOB median q-error is inconsistent in a load-bearing way. Table 3 and the arXiv-style abstract give PostgreSQL q50 = 190.5 and Bespoke-Card = 11.5 (~94% reduction). §5.4 states “from 19.5 to 11.5 (a 41% reduction),” and the workshop abstract claims a 41% median reduction. Please reconcile abstract, §5.4, and Table 3 so the central accuracy claim is unambiguous.

Circularity Check

0 steps flagged

No circular derivation: runtime and q-error claims are measured against external ground truth and an independent baseline after the estimator is frozen.

full rationale

Bespoke-Card is an engineering synthesis system, not a first-principles derivation that claims to predict quantities already used as definitional inputs. The load-bearing results (JOB/JOB-Complex end-to-end runtime with injected estimates; subplan q-error quantiles) are obtained by (i) freezing a generated estimator, (ii) comparing estimates to true cardinalities from EXPLAIN ANALYZE, (iii) comparing to PostgreSQL’s built-in estimator, and (iv) measuring wall-clock execution under PG-Lab injection with join enumeration and cost model held fixed. None of these outcomes is algebraically forced by the synthesis inputs: the planner’s design hypothesis and the coder’s patches are scored by a deterministic external harness, not by re-labeling a fitted parameter as a prediction. Self-citations to the authors’ Bespoke DBMS / Bespoke-OLAP vision and to JOB-Complex supply research context and a harder workload, but they do not underwrite uniqueness theorems, forbid alternatives, or substitute for the empirical measurements. Specializing to a declared workload and using that same workload’s subplans in the feedback curriculum is a stated design contract and a possible overfitting concern, not a by-construction identity between claim and input. No self-definitional step, fitted-input-as-prediction, load-bearing self-citation chain, uniqueness import, smuggled ansatz, or renamed known result is present in the derivation of the headline claims.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard empirical-systems assumptions plus a small number of design choices that are free parameters of the synthesis harness. No physical constants or fitted scientific parameters appear; the free parameters are engineering knobs (LLM model, curriculum stages, outlier count). The invented entities are the system architecture itself and the database-and-workload “contract.” All axioms are either standard database practice or explicit methodological choices of the paper.

free parameters (4)
  • LLM backbone (GPT-5.4)
    Choice of model and temperature directly affects planner and coder quality; results are reported for one commercial snapshot only.
  • Feedback curriculum stages and grouping dimensions
    The three-stage join/filter/full schedule and the particular grouping keys (Table 2) are hand-designed; different curricula could yield different estimators.
  • Outlier count (10 worst over- and under-estimates)
    The number of concrete failing subplans returned to the coder is an arbitrary but load-bearing hyper-parameter of the repair signal.
  • Archival selection criterion
    Which checkpoint is retained after non-monotonic agent behavior is a free design choice that can alter the final reported numbers.
axioms (5)
  • domain assumption A representative workload and fixed schema are known a priori and form a stable “contract” for specialization.
    Stated in Section 3.1; without it the specialization premise collapses.
  • domain assumption True cardinalities of all evaluation subplans can be obtained once (via EXPLAIN ANALYZE) and reused as ground truth.
    Section 4.3; the entire feedback loop is defined relative to these labels.
  • ad hoc to paper PostgreSQL’s cardinality estimates serve as a useful “easy-to-beat” baseline that distinguishes fixable from intrinsically hard errors.
    Explicit design choice in Section 4.3; the regression-rate signal depends on it.
  • domain assumption SPAJ queries with complex predicates are a sufficient query class for the prototype.
    Section 3.2 scope statement.
  • standard math Standard relational algebra and q-error (max(est/true, true/est)) are the correct accuracy metrics.
    Used throughout Sections 4–5; conventional in the cardinality-estimation literature.
invented entities (2)
  • Bespoke-Card multi-agent synthesis loop (planner + coder + staged deterministic harness) no independent evidence
    purpose: To generate executable, workload-specialized cardinality estimators rather than tune a fixed architecture.
    The system itself is the primary contribution; no independent existence outside the paper.
  • Database-and-workload contract no independent evidence
    purpose: To justify shedding generality and specializing statistics and estimation logic.
    Conceptual framing introduced in Section 3; not an independently measured object.

pith-pipeline@v1.1.0-grok45 · 20225 in / 3470 out tokens · 35469 ms · 2026-07-14T18:09:32.151060+00:00 · methodology

0 comments
read the original abstract

Cardinality estimators are built to support arbitrary schemas and workloads, forcing them to rely on generic statistics even when the schema and workload is known in advance, leaving optimizers prone to large errors and poor plans. We present Bespoke-Card, an agent-driven system that synthesizes workload-specific cardinality estimators as executable code: a planning agent designs the estimators' strategies, a coding agent implements them, and a validator scores the estimates against true cardinalities and PostgreSQL estimates, forming a robust and deterministic harness. Going beyond naive prompting, Bespoke-Card uses structured q-error feedback, regression analysis, concrete outlier subplans, a curriculum isolating join-only, filter-only, and full-subplan errors, and archival selection of the best implementation. Injecting its estimates into the optimizer cuts total PostgreSQL runtime on JOB by 33% and reduces median q-error over all JOB subplans from 190.5 to 11.5 (-94%), while synthesizing a strong estimator in under one hour for less than $10. Bespoke-Card is opening a new avenue for cardinality estimation next to classical generic estimators and learned estimator architectures.

Figures

Figures reproduced from arXiv: 2606.09361 by Anton Winter, Carsten Binnig, Johannes Wehrstein, Timo Eckmann.

Figure 1
Figure 1. Figure 1: Bespoke-Card reduces total PostgreSQL runtime by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Bespoke-Card synthesizes a cardinality estimator given a dataset and tunes it for a given workload. Two agents, a [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Quality of cardinality estimates for multi-join queries compared to the true cardinalities. Bespoke-Card [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Q-error distribution of cardinality estimates com [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Q-error distribution over all subplans of the JOB [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references

  1. [1]

    Gibbons, Viswanath Poosala, and Sridhar Ra- maswamy

    Swarup Acharya, Phillip B. Gibbons, Viswanath Poosala, and Sridhar Ra- maswamy. 1999. Join Synopses for Approximate Query Answering. InSIGMOD. 275–286

  2. [2]

    Rico Bergmann, Claudio Hartmann, Dirk Habich, and Wolfgang Lehner. 2025. An Elephant Under the Microscope: Analyzing the Interaction of Optimizer Components in PostgreSQL.SIGMOD3, 1 (2025), 9:1–9:28

  3. [3]

    Garofalakis, Peter J

    Graham Cormode, Minos N. Garofalakis, Peter J. Haas, and Chris Jermaine

  4. [4]

    Foundations and Trends in Databases4, 1-3 (2012), 1–294

    Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches. Foundations and Trends in Databases4, 1-3 (2012), 1–294

  5. [5]

    Timo Eckmann, Matthias Jasny, Johannes Wehrstein, and Carsten Binnig. 2026. The Future Is Bespoke: Synthesizing One-Size-Fits-One DBMSs with LLM Coding Agents.IEEE Data Engineering Bulletin50, 1 (2026), 88–103

  6. [6]

    Ullman, and Jennifer Widom

    Hector Garcia-Molina, Jeffrey D. Ullman, and Jennifer Widom. 2009.Database Systems - The Complete Book (2. ed.)

  7. [7]

    Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kris- tian Kersting, and Carsten Binnig. 2020. DeepDB: Learn from Data, not from Queries!VLDB13, 7 (2020), 992–1005

  8. [8]

    Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. InCIDR

  9. [9]

    Oleksii Kliukin. 2014. PgTune – Tuning PostgreSQL Config by Your Hardware

  10. [10]

    Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How Good Are Query Optimizers, Really?VLDB9, 3 (2015), 204–215

  11. [11]

    Viktor Leis, Bernhard Radke, Andrey Gubichev, Alfons Kemper, and Thomas Neumann. 2017. Cardinality Estimation Done Right: Index-Based Join Sampling. InCIDR

  12. [12]

    Yao Lu, Srikanth Kandula, Arnd Christian König, and Surajit Chaudhuri. 2021. Pre-training Summarization Models of Structured Datasets for Cardinality Esti- mation.VLDB15, 3 (2021), 414–426

  13. [13]

    Ioannidis

    Viswanath Poosala and Yannis E. Ioannidis. 1997. Selectivity Estimation Without the Attribute Value Independence Assumption. InVLDB. 486–495

  14. [14]

    Ioannidis, Peter J

    Viswanath Poosala, Yannis E. Ioannidis, Peter J. Haas, and Eugene J. Shekita

  15. [15]

    In SIGMOD

    Improved Histograms for Selectivity Estimation of Range Predicates. In SIGMOD. 294–305

  16. [16]

    Selinger, Morton M

    Patricia G. Selinger, Morton M. Astrahan, Donald D. Chamberlin, Raymond A. Lorie, and Thomas G. Price. 1979. Access Path Selection in a Relational Database Management System. InSIGMOD. 23–34

  17. [17]

    One Size Fits All

    Michael Stonebraker and Ugur Çetintemel. 2005. "One Size Fits All": An Idea Whose Time Has Come and Gone (Abstract). InICDE. 2–11

  18. [18]

    Alexander van Renen, Dominik Horn, Pascal Pfeil, Kapil Vaidya, Wenjian Dong, Murali Narayanaswamy, Zhengchun Liu, Gaurav Saxena, Andreas Kipf, and Tim Kraska. 2024. Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet.VLDB17, 11 (2024), 3694–3706

  19. [19]

    Johannes Wehrstein, Carsten Binnig, Fatma Özcan, Shobha Vasudevan, Yu Gan, and Yawen Wang. 2025. Towards Foundation Database Models. InCIDR

  20. [20]

    Johannes Wehrstein, Timo Eckmann, Roman Heinrich, and Carsten Binnig

  21. [21]

    JOB-Complex: A Challenging Benchmark for Traditional & Learned Query Optimization.VLDB(2025)

  22. [22]

    Johannes Wehrstein, Timo Eckmann, Matthias Jasny, and Carsten Binnig. 2026. Bespoke OLAP: Synthesizing Workload-Specific One-size-fits-one Database En- gines.arXiv preprint arXiv:2603.02001(2026)

  23. [23]

    Peizhi Wu and Gao Cong. 2021. A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation. InSIGMOD. 2009–2022

  24. [24]

    Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica. 2020. NeuroCard: One Cardinality Estimator for All Tables.VLDB14, 1 (2020), 61–73

  25. [25]

    Tianjing Zeng, Junwei Lan, Jiahong Ma, Wenqing Wei, Rong Zhu, Pengfei Li, Bolin Ding, Defu Lian, Zhewei Wei, and Jingren Zhou. 2024. PRICE: A Pretrained Model for Cross-Database Cardinality Estimation.VLDB18, 3 (2024), 637–650. 10