Pith. sign in

REVIEW 3 major objections 4 minor 14 references

PromptDB makes prompts first-class database values, then rewrites them the way a query optimizer rewrites plans, and shows that database context improves output validity and cost-quality trade-offs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:45 UTC pith:VEG7QSQC

load-bearing objection Clean model, weak empirical proof: the static baseline is the load-bearing flaw, but the idea is worth referee time. the 3 major comments →

arxiv 2607.21756 v1 pith:VEG7QSQC submitted 2026-07-23 cs.DB cs.LG

Prompt as a Data Type: In-Database LLM Prompt Management and Rewriting

Denis Mayr Lima Martins , Gottfried Vossen This is my paper
classification cs.DB cs.LG
keywords prompt as data typedatabase-guided prompt rewritingLLM integrationprompt optimizationrelational query optimizationsemantic operatorsvalidity constraints
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PromptDB treats prompts as typed, tuple-level database values rather than opaque application strings. By storing template, bindings, model, and output domain inside the database, the system can rewrite prompts using schema, constraints, query context, and statistics. The paper argues that this creates a new optimization space for LLM-backed database work, analogous to classical query optimization. Experiments on classification, semantic filtering, and value normalization show that database-guided rewrite rules improve validity and yield favorable cost-quality trade-offs compared to static prompts.

Core claim

Prompts can be represented as a self-describing PROMPT value—template, attribute bindings, model metadata, and output domain—and stored directly in relational tables. Rendering is separated from rewriting: before a prompt is filled in, the database can apply rewrite rules such as constraint injection, column projection, output format minimization, and few-shot example selection. A lightweight optimizer (PromptOpt) then chooses a rewrite plan by balancing an estimated quality score against an estimated token cost. The central claim is that database metadata is the right source of prompt-rewrite signals, and that this approach outperforms static application-level prompting on constrained or st

What carries the argument

The PROMPT datatype: a structured value P = ⟨T,B,M,D⟩ storing a template, bindings to tuple attributes, model family, and output domain, plus a database-derived context Ω = ⟨S,C,Q,Ts⟩ of schema, constraints, query, and statistics. Prompt rewriting rules transform P using Ω, and PromptOpt selects a plan π by maximizing Q̂ − λĈ. This mirrors how a relational optimizer rewrites SQL plans using metadata.

Load-bearing premise

The empirical comparison assumes the manually written static prompt is a fair application-level baseline; if that baseline is degenerate (for instance, omits output-domain guidance), the measured gains from database-guided rewriting may be inflated.

What would settle it

Re-run the attribute extraction task with a static prompt that explicitly includes the label domain (e.g., 'Return one of: refund, delivery, technical, other'). If static accuracy rises from 0.00 to near the rewritten-prompt level, the paper's headline quality advantage collapses to a baseline artifact.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Prompt-valued relations make LLM calls visible to query planning, enabling cost-based decisions about when and how to invoke a model.
  • Constraint injection and output format minimization improve validity for tasks with controlled vocabularies, such as semantic filtering and attribute extraction.
  • Query-aware column projection reduces token cost by rendering only relevant tuple attributes, analogous to projection pushdown.
  • The quality of rewriting is task-dependent; no single rule dominates, motivating optimizer variants that adapt to task requirements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this design is right, the next step is learned or calibrated quality estimators: PromptOpt's current heuristic is intentionally lightweight, but a model trained on execution logs could choose rewrite plans with much finer granularity.
  • The same rewrite-rule framework could be extended to multi-table joins and foreign-key paths, where prompt context is assembled across relations rather than from a single tuple.
  • The cost model could incorporate output tokens and latency, not just input length, to better capture real LLM API economics.
  • A direct implication for LLM application development: if prompts live as database values, prompt engineering becomes a metadata-management and optimization activity rather than an application-code concern.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes PromptDB, a database system that treats prompts as first-class tuple-level data values. The authors define a logical PROMPT datatype with template, bindings, model metadata, and output domain; store these values in relations; evaluate them through generated views and an EVAL operator; and rewrite/optimize them using database context (schema, constraints, statistics, query context). The central idea is to apply query-optimizer thinking to prompts: rules such as constraint injection, column projection, output-format minimization, and database-selected few-shot examples are applied as rewrite plans, and PromptOpt selects among plans using a cost-quality objective. The paper includes a formal model, a DuckDB-based prototype, and an evaluation on three datasets across three tasks (attribute extraction, semantic filtering, semantic value normalization) comparing static prompting, all-rules rewriting, and PromptOpt.

Significance. If the empirical claims are substantiated, the paper introduces a genuinely new optimization space for LLM-in-the-loop database systems: making prompts visible to the DBMS enables metadata-driven rewriting and cost-aware plan selection, analogous to relational query optimization. The formal model is simple, internally consistent, and the rewrite rules are clearly specified. The prototype demonstrates feasibility. The main novel contribution is the conceptual framework, which is potentially influential for AI-native databases. However, the evaluation is thin and the headline empirical claim—improvement over static prompts—rests on a possibly degenerate baseline. The paper also ships no code or data, and the optimizer heuristic is not validated. The strength of the contribution is the architecture and formalization, not the current empirical demonstration.

major comments (3)
  1. [Section 4] The Static baseline scores 0.00 on Attribute Extraction. This is a red flag. The paper's motivating example itself uses a minimal prompt ('Classify this customer support ticket: {message}') with no output-domain guidance, and the two rules that most directly address that deficiency (RCI and ROFM) are exactly what All Rules and PromptOpt apply. If the Static strategy in the experiment is similarly under-specified, then the measured gains simply reflect adding information (valid labels, output format) that a competent developer would include in a real prompt. The paper does not report the exact static prompts used, nor any comparison against a best-practice prompt baseline. This is load-bearing: the abstract claim of 'improves output validity ... compared with static, manually written prompts' is not established if the static prompts are degenerate. Please provide the full static prompts f
  2. [Section 4.3, Eq. (5)] The empirical evaluation reports point estimates only. There are no confidence intervals, standard deviations, or per-dataset breakdowns. The text says temperature is 0.0, but with database-selected few-shot examples, the example set can vary with the tuple or with query context, so results are likely not deterministic across runs. For example, Figure 2 shows Semantic Filtering at 0.61 for All Rules and 0.37 for Static; without any measure of variance, we cannot judge whether this is a meaningful difference. The paper also does not state how many tuples are in each dataset, how many prompts are evaluated, or whether metrics are aggregated over the entire dataset or a sample. Please report these details, include error bars or repeated runs (e.g., different example selections), and provide per-dataset tables in addition to the aggregate figures.
  3. [Section 4.3] PromptOpt's objective contains a free parameter λ, but the paper never reports its value, how it was chosen, or a sensitivity analysis. The quality estimator Qhat is described as a lightweight heuristic, yet there is no validation that Qhat correlates with actual task quality, and no comparison against alternatives (e.g., exhaustive enumeration of plans or random selection). Figure 4 shows PromptOpt occupies an intermediate cost-quality region, but this could be trivially true if it always selects a cheap, low-quality plan. The paper itself states that 'the current heuristic estimator is incomplete,' which is honest, but it also makes the claim of 'favorable cost-quality trade-offs' premature. At minimum, report λ, show how PromptOpt's plan choices compare to the oracle best plan, and demonstrate that the heuristic does not systematically under-select rich plans for tasks that need them.
minor comments (4)
  1. [Section 4] There is a stray comma in the tuple definition: '⟨𝑇,𝐵,𝑀,𝐷,⟩' should be '⟨𝑇,𝐵,𝑀,𝐷⟩'.
  2. [Table 1] The text says 'The LLM backend is also replaceble' — typo for 'replaceable'. Also, the conclusion says 'We also intent to investigate' — should be 'intend'.
  3. [Section 4.1 Figures 2-4] The table claims PROMPT is 'Versioned', but the paper does not describe any versioning mechanism or experiment for prompts. Please either add support for versioning or remove that checkmark.
  4. [Section 3] The figures aggregate across three datasets but do not distinguish them. It is impossible to see whether the reported averages are driven by one dataset or are consistent across synthetic, Car Evaluation, and TPC-H. A supplementary table with per-dataset results would substantially improve clarity.

Circularity Check

0 steps flagged

No significant circularity: the derivation is self-contained and empirical claims rest on hand-built rewrite rules, not on fitted targets or self-citation.

full rationale

PromptDB's formal model defines prompt values, rewriting contexts, and an EVAL operator, but it does not define the evaluation outcome into the input. The optimization objective in Eq. (5) uses Qhat, explicitly described as a lightweight heuristic based on rule applicability, and Chat estimated from lengths — neither is fitted to the experimental results, so the cost-quality trade-off is not circular. The rewrite rules (RCI, RQCP, ROFM, RFSE) are presented as database-guided transformations and are then empirically evaluated; their benefit is an empirical question, not an artifact of definition. The paper cites prior work by one of the authors [12] only as philosophical motivation for treating commands as data, not as load-bearing evidence for the central claim, so this self-citation is not circular. The known weakness of the Static baseline (scoring 0.00 on Attribute Extraction in Figure 2) is an experimental-design concern about whether the baseline is representative, not a circularity in the derivation chain. The paper also candidly states that PromptOpt's heuristic estimator is incomplete and that no single rule dominates, which further supports that the reported effects are measured rather than assumed. Therefore, no circular step is present.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The paper's contributions are design and engineering artifacts, not fitted physical models. The main free parameters are the optimizer's trade-off coefficient and the hand-built quality heuristic; the main assumptions concern the usefulness of database context for prompt rewriting and the completeness of the rule set.

free parameters (3)
  • λ (PromptOpt trade-off coefficient)
    In Eq. (5), λ balances estimated quality vs cost; no value or sensitivity analysis is given, so it is a hand-set parameter affecting which rewrite plans are selected.
  • max_tokens = 128
    Execution hyperparameter set for all LLM calls; not central but affects cost and output-length estimates.
  • Rule applicability heuristics for Qhat
    Qhat is a hand-designed heuristic based on rule applicability, not learned or calibrated; it determines PromptOpt selections.
axioms (3)
  • domain assumption LLM output can be treated as a scalar SQL value returned by EVAL.
    Section 2, Prompt Evaluation: EVAL(P,t,Ω) -> y where y is a string/label; assumes model outputs can be mapped to database values.
  • domain assumption Database metadata (schema, constraints, query, statistics) provides signal that can improve prompt quality and reduce cost.
    Section 1 and Prompt Rewriting Context; the core premise of PromptDB, validated only by the paper's limited experiments.
  • ad hoc to paper The set of rewrite rules (RCI, RQACP, ROFM, RFSE) and the candidate plans cover the useful optimization space.
    Section 3 defines a fixed candidate set; no evidence this set is complete or optimal.
invented entities (3)
  • PROMPT logical datatype no independent evidence
    purpose: Store templates, bindings, model metadata, and output domain as tuple-level values
    Introduced and implemented only in the paper's prototype; no external benchmark or artifact verifies it independently.
  • EVAL operator no independent evidence
    purpose: Render, rewrite, execute, and log prompt execution inside the DBMS
    Internal UDF; only evidenced by the paper's own experiments.
  • PromptOpt optimizer no independent evidence
    purpose: Select rewrite plan maximizing Qhat - λ Chat
    Heuristic, uncalibrated; paper acknowledges incompleteness in Section 4.4.

pith-pipeline@v1.3.0-alltime-deepseek · 8290 in / 11686 out tokens · 119460 ms · 2026-08-01T06:45:58.054918+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly used in database-backed applications to classify tuples, filter records using semantic predicates, extract structured attributes, and enrich query results. Yet the prompt that start these computations are typically stored outside the DBMS in unstructured formats, making them invisible to query execution, metadata management, and optimization. Drawing on Stonebraker's QUEL as a Data Type and the principles of reflective programming, this paper introduces PromptDB, a database system that treats prompts as tuple-level database values. PromptDB provides a logical PROMPT datatype whose values store a template, bindings to tuple attributes, model metadata, and task metadata. Relations may contain PROMPT attributes directly in base tables, or expose them through views over joined tuples. Users query prompt-valued attributes through generated evaluation views, while the system internally renders, rewrites, optimizes, and executes prompts through an EVAL operator. Making prompts database-visible creates a new optimization space. The key idea is to bring query-optimizer thinking to prompts: just as query optimizers exploit database metadata to rewrite SQL plans, PromptDB exploits database metadata to rewrite prompts. We evaluate PromptDB on synthetic and real-world data workloads across different tasks. The results show how database-guided rewriting improves output validity and yields favorable cost-quality trade-offs compared with static, manually written prompts.

Figures

Figures reproduced from arXiv: 2607.21756 by Denis Mayr Lima Martins, Gottfried Vossen.

Figure 1
Figure 1. Figure 1: Motivating contrast between traditional prompt handling (left) and PromptDB (right). A traditional system fetches [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end task quality. Applying all rules yields [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Rule ablation. The results show that rewrite rules [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cost-quality trade-off between PromptOpt and fixed [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 3 linked inside Pith

  1. [1]

    Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes.Proc. VLDB Endow.17, 2 (Oct. 2023), 92–105. doi:10.14778/3626292.3626294

    arXiv 2023

  2. [2]

    Stephen Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al

  3. [3]

    Lee, Deepti Raghavan, Duo Lu, and Andrew Crotty

    Ugur Çetintemel, Shu Chen, Alexander W. Lee, Deepti Raghavan, Duo Lu, and Andrew Crotty. 2026. Making Prompts First-Class Citizens for Adaptive LLM Pipelines. In16th Conference on Innovative Data Sys- tems Research, CIDR 2026, Chaminade, CA, USA, January 18-21, 2026. www.cidrdb.org. https://vldb.org/cidrdb/2026/making-prompts-first-class- citizens-for-ada...

    2026

  4. [4]

    Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. Openprompt: An open-source framework for prompt- learning. InProceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations. 105–113

    2022

  5. [5]

    Yixiong Fang, Tianran Sun, Yuling Shi, and Xiaodong Gu. 2025. Attention- RAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation. arXiv:2503.10720 [cs.CL] https://arxiv.org/abs/2503.10720

    arXiv 2025

  6. [6]

    Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. 2025. JSON- SchemaBench: A Rigorous Benchmark of Structured Outputs for Language Mod- els. arXiv:2501.10868 [cs.CL] https://arxiv.org/abs/2501.10868

    Pith/arXiv arXiv 2025

  7. [7]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav San- thanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al. 2024. DSPy: compiling declarative language model calls into state- of-the-art pipelines. InInternational Conference on Learning Representations, Vol. 2024. 54928–54958

    2024

  8. [8]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What Makes Good In-Context Examples for GPT-3? arXiv:2101.06804 [cs.CL] https://arxiv.org/abs/2101.06804

    Pith/arXiv arXiv 2021

  9. [9]

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024. In- context Learning with Retrieved Demonstrations for Language Models: A Survey. arXiv:2401.11624 [cs.CL] https://arxiv.org/abs/2401.11624

    Pith/arXiv arXiv 2024

  10. [10]

    Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators and Their Optimization: Enabling LLM-Based Data Processing with Accuracy Guarantees in LOTUS.Proc. VLDB Endow.18, 11 (July 2025), 4171–4184. doi:10.14778/3749646.3749685

    arXiv 2025

  11. [11]

    Hanson, and W

    Michael Stonebraker, Erika Anderson, Eric N. Hanson, and W. Bradley Rubenstein

  12. [12]

    Jan Van den Bussche, Dirk Van Gucht, and Gottfried Vossen. 1993. Reflective programming in the relational algebra. InProceedings of the Twelfth ACM SIGACT- SIGMOD-SIGART Symposium on Principles of Database Systems(Washington, D.C., USA)(PODS ’93). Association for Computing Machinery, New York, NY, USA, 17–25. doi:10.1145/153850.153852

    arXiv 1993

  13. [1984]

    InSIGMOD’84, Proceedings of Annual Meeting, Boston, Massachusetts, USA, June 18-21, 1984, Beatrice Yormark (Ed.)

    Quel as a Data Type. InSIGMOD’84, Proceedings of Annual Meeting, Boston, Massachusetts, USA, June 18-21, 1984, Beatrice Yormark (Ed.). ACM Press, 208–214. doi:10.1145/602259.602287

    arXiv 1984

  14. [2022]

    InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations

    Promptsource: An integrated development environment and repository for natural language prompts. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 93–104