Pith. sign in

REVIEW 3 major objections 3 minor 32 references

AI Query Compilation for Unified and Optimized Execution

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A single compiled tensor program can run SQL and LLM inference together, eliminating the CPU-to-accelerator data movement penalty.

desk verdict The unified SQL+LLM compilation idea is a worthwhile vision, but the throughput numbers are internally inconsistent and the missing output-equivalence check makes the speedups untrustworthy as written. read the letter →

arxiv 2608.10139 v1 pith:YWXF5SFV submitted 2026-08-10 cs.DB

classification cs.DB
keywords AIquerycompilationhybridSQL/LLMqueriestensorcomputegraphJAX/XLAautomaticshardingsemanticoperatorsTPUexecutionSemBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hybrid AI queries—SQL statements that call an LLM for classification or text transformation—are normally split between a CPU database engine and a separate accelerator, with rows and logits crossing a PCIe bus at each step. The paper's core claim is that the entire hybrid query can be lowered into a single tensor compute graph and JIT-compiled for the accelerator, so filters, LLM inference, and aggregation all execute on-device in one program. If this holds, the data-movement penalty disappears and the compiler can automatically shard the combined relational and neural work across devices, which the paper exhibits with near-linear scaling from one to four TPUs. On SemBench-style movie-review queries the paper reports up to 5.34x latency speedup and 9.8x throughput speedup over a CPU-plus-TPU split baseline, while also noting the split baseline wins when an early relational filter removes most rows.

What carries the argument

The central object is the unified tensor compute graph: a JAX program in which database rows, boolean masks, model parameters, and reductions are all tensor arrays, compiled by XLA into a single accelerator executable. The graph carries the argument through three mechanisms: it fuses relational predicates with the LLM forward pass so no intermediate results leave the device; it uses XLA/GSPMD-style automatic sharding to split rows and model weights across devices without separate CPU/TPU parallelization logic; and it uses structured control flow (jax.lax.scan) to stream micro-batches through the model while keeping intermediate state on the accelerator. Named mappings from SQL AST nodes to JAX implementations (table scans become input arrays, WHERE becomes boolean masks, AI_IF becomes an LLM forward pass with logit comparison, aggregation becomes jnp.where-masked sum) are what make the compilation mechanical.

What would settle it

Run the five benchmark queries from Table 2 over identical data through both the baseline split execution and the unified compiled program, and compare every returned SUM, COUNT, and AI_TRANSFORM string; any difference in those outputs would refute the semantic-equivalence premise even if the latency and throughput speedups reproduce.

Watch

Extended reading notes

Core claim

By compiling the hybrid AI query as a whole—mapping SQL clauses to JAX tensor primitives and keeping the LLM forward pass inside the same traced program—the authors show that a filter like AI_IF('is this review positive?') plus SUM can run as one @jax.jit-compiled function on TPUs. The result is that no logits or intermediate rows are copied back to the host; only the final scalar aggregate or generated tokens return. The paper reports measured speedups of 5.34x latency and 5.30x throughput for semantic filtering (Q1), 2.28x latency and 5.31x throughput for a hybrid relational-plus-semantic filter (Q2), throughput gains up to 9.8x on a selectivity sweep for Q3, and 4.00x throughput scaling on four devices via automatic data-parallel sharding. The claim is that this unified compiled execution, not specialized kernels, is what removes the CPU-TPU boundary penalty and enables global optimizations like on-device cross-join expansion and fused string matching.

Load-bearing premise

The entire argument assumes the compiled unified program produces exactly the same model outputs and therefore exactly the same query answers (sums, counts, generated strings) as the split CPU-plus-accelerator execution; no experiment in the paper checks that equivalence.

Editorial extensions

If this is right

  • AI queries whose outputs are small—a sum, a count, a handful of generated tokens—can run end-to-end on accelerators, shrinking per-query host-device transfers from gigabytes of logits to kilobytes.
  • Compiler-driven auto-sharding replaces hand-written CPU/TPU parallelization; the reported 1-to-2-to-4 device scaling for Q3 is roughly 1x, 2x, 4x throughput without manual calibration.
  • The performance of unified compilation depends on selectivity: the baseline split execution wins for Q3 at threshold 10 (0.02x latency), so the optimal choice between compiled and split execution is query-dependent.
  • Under concurrent tenants the current compiled single-program design saturates at about 1377 rows/sec aggregate throughput because TPU execution is FIFO-serialized, identifying runtime scheduling of compiled kernels as the next bottleneck.
  • Long-generation queries with debugging outputs benefit most because on-device string matching means only matching essays are copied back, not the full logits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same lowering suggests that any relational operator expressible as tensor algebra—joins, grouping, ordering—could be fused into the model program, so the five queries here are a small slice of a much larger design space.
  • Because the paper reports no comparison of final query results between the unified program and the split baseline, the most direct next experiment is to assert semantic equivalence: run the same queries both ways and check that sums, counts, and generated strings match exactly.
  • The selectivity results imply a cost-based optimizer should choose between unified compiled and split execution per query, much as traditional optimizers choose whether to inline a UDF based on estimated cost.
  • The fixed-throughput multi-tenancy result points toward compiling independent queries into a shared batched program or adding a scheduler that interleaves tokens, rather than compiling each query as a standalone executable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a vision for executing hybrid AI queries (SQL plus LLM-based semantic operators) by compiling the entire query into a single JAX/XLA tensor program that runs on TPUs, thereby avoiding PCIe data movement between a CPU database engine and a separate ML serving system. The authors describe a translation from SQL AST to JAX constructs, report preliminary latency and throughput measurements for five queries on SemBench-based movie review data with a Gemma-2B model, and claim up to 5.34x latency and 9.87x throughput speedups, near-linear horizontal scaling on 4 TPU devices, and outline open research challenges.

Significance. If the results hold, this is a useful feasibility demonstration that relational operators and LLM inference can coexist in one compiled accelerator program, enabling automatic sharding and elimination of cross-boundary copies. The paper is honest about limitations, including high-selectivity cases and multi-tenancy serialization, and it articulates a concrete research roadmap. The main contributions are the architectural vision and the preliminary performance characterization, not a full production system. However, the quantitative claims are not yet reliable because the paper omits an output-equivalence validation and contains internally inconsistent speedup metrics.

major comments (3)
  1. [Section 4, Tables 3-5; Section 3.1] The paper never verifies that the unified compiled program returns the same query results as the split-execution baseline. The headline speedups are only meaningful if the fused SQL+LLM program yields exactly the same aggregates, classifications, and transformations as the baseline, but no experiment compares outputs. For Q1/Q2 the SUM depends on the binary classification from the LLM; with padding to a static maximum sequence length, the attention computation near sequence boundaries could alter logits and flip classifications. For Q5 the COUNT(*) and the LIKE '%masterpiece%' predicate depend on the generated essay text; unless decoding is seeded/argmax and length-capped identically in both executions, the two pipelines can disagree. For Q4 the cross-join with A.id = B.id AND A.review_id < B.review_id relies on row order and duplicate handling that a tensorized program may not preserve. Add a direct fused-vs-split output-equivalence check (or an explicit correctness argument) for each query.
  2. [Section 4.1, Table 3; Section 4.2, Table 4] The throughput speedup numbers are internally inconsistent with the latency speedups. In Table 3, Q3 reports latency speedup 0.47x (unified is slower) yet throughput speedup 4.29x. In Table 4, at Thresh=10.0 the latency speedup is 0.02x but throughput speedup is 9.87x; at Thresh=50.0 they are 1.00x and 4.24x, respectively. For a fixed workload, throughput is N/latency, so the throughput ratio should equal the inverse latency ratio of the two systems; the reported combinations would require the baseline to have both lower latency and lower throughput than the unified system, which is impossible. Clarify the metric definitions (end-to-end query latency vs per-micro-batch latency, steady-state throughput vs batch-inclusive throughput) and recompute the speedups.
  3. [Section 4.1] The baseline is described as an eager mixture execution model (CPU DBMS plus TPU inference), but the text also refers to a 'JIT-compiled baseline' and reports very tight standard deviations (e.g., ±0.00 ms). The paper should specify precisely what the baseline does: whether the LLM serving side uses a production server (e.g., vLLM) or a simple Python loop, whether batching is identical between baseline and unified execution, and how the CPU-side SQL execution is timed. This matters because the reported gains may conflate the unification benefit with differences in baseline engineering maturity.
minor comments (3)
  1. [Throughout] The manuscript contains numerous typos and residual LaTeX fragments, including 'subesquently', 'classsificaiton', 'intermedidate', 'executnion', 'databse', 'textitaudienceScore', 'qureis', 'achi-tectures', 'wokrld', 'oeprators', and 'hybird'. A careful proofreading pass is needed.
  2. [Section 3.3] The text says the results are 'demonstrated in Section??' — an unresolved cross-reference. Also, the figure numbering is confusing: Figure 1 is mentioned as 'High-level view' but the generated JAX program is Figure 2, and the frontend compilation flow is Figure 3; the narrative in Section 3.3 refers to 'Figure 3' for the consolidation, though the intended figure may be Figure 1.
  3. [Section 3.3] The phrase 'completely removes PCIe bug bottlenecks' should read 'PCIe bus bottlenecks'. In addition, the paper claims 'completely alleviate PCIe data movement bottlenecks' in the abstract, but Section 4.2 shows that the unified execution can be much slower at high selectivity; the claim should be qualified accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the speedup claims rest on measured execution times against a split-execution baseline, not on a self-referential derivation or fitted target.

full rationale

The paper is an experimental vision paper: it translates SQL AST constructs to JAX primitives (Table 1), embeds a Flax causal LM forward pass (Figure 2), and reports measured latencies/throughputs on TPUs against an eager CPU-SQL plus TPU-LLM baseline. The central claim (up to 5.3x latency and 9.8x throughput speedup) is an empirical measurement, not the output of a derivation whose inputs include that claim or whose parameters are fitted to that target. No parameter is fitted to a target result, and no prediction is obtained by renaming fitted inputs as conclusions. The only self-citations are SemBench [16] and the authors' AI-query approximation paper [5]; SemBench is used as a query/dataset workload, not as a proof or uniqueness argument, and the related-work citation [5] is not load-bearing for the compilation architecture. Section 4 explicitly scopes the evaluation to feasibility and extra-data-copy elimination, which further separates the measured speedups from any imported conclusion. The absence of a fused-vs-split output-equivalence check is a correctness/robustness risk, not a circularity step, because output equivalence is never defined in terms of, or assumed equivalent to, the measured speedups. No circular step can be exhibited from the paper's equations or construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central performance claims rest on a handful of experimental choices (micro-batch size, padding length, baseline configuration) and on assumptions that the fused program preserves LLM semantics and that the baseline is representative. No new theoretical entities are introduced.

free parameters (3)
  • micro-batch size (bsz) = 128
    Chosen by hand for the TPU streaming loop (Section 3.2 and Figure 2); throughput and latency results depend on this value, and no sensitivity analysis is given.
  • maximum padded sequence length = not reported
    The paper notes inputs are padded to a static maximum sequence length for JIT compilation (Section 5, Dynamic Shape Compilation), but does not report the value; it affects memory and compute.
  • baseline configuration = single-process CPU database engine plus separate TPU inference
    The comparison baseline is chosen as an unoptimized single-process CPU engine (Section 4.1), which the authors admit is less efficient; the speedup numbers depend on this baseline choice.
assumptions (3)
  • domain assumption The compiled JAX program produces the same LLM outputs as the original model, preserving query semantics.
    Section 3.1 translates SQL to JAX and Section 3.3 consolidates boundaries, but no experiment verifies that the fused execution yields the same AI_IF classifications or AI_TRANSFORM texts as the split baseline.
  • domain assumption The baseline split execution with CPU database and TPU inference is representative of contemporary AI query engines.
    Stated in Section 4.1; the paper compares against a single-process CPU engine without the cross-engine parallelization and tuning that production systems like vLLM and LOTUS employ, so speedups may be overstated.
  • domain assumption XLA's GSPMD/pmap auto-sharding correctly and efficiently partitions the fused program across TPU devices.
    Section 3 and Section 4.4 rely on the compiler's sharding for the scaling results; the paper does not analyze whether the auto-sharding choices are optimal or how they interact with LLM tensor parallelism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Query Compilation for Unified and Optimized Execution." pith.science (2026). https://pith.science/paper/YWXF5SFV

@misc{pith2026260810139,
  author       = {Pith},
  title        = {Pith review of: AI Query Compilation for Unified and Optimized Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWXF5SFV}},
  note         = {Machine review of arXiv:2608.10139}
}
read the original abstract

In this vision paper, we propose a novel architectural paradigm for accelerated AI query execution via a unified compiled execution strategy. By compiling the hybrid AI Query as a whole -- integrating both standard SQL relational constructs and LLM inference layers into a single, unified tensor compute graph -- we completely alleviate PCIe data movement bottlenecks across execution boundaries and enable global compiler optimizations and efficient automatic sharding. We demonstrate the viability of this unified execution paradigm on select and extended AI queries on SemBench Reviews and Movies datasets, achieving up to 5.3x latency speedup and 9.8x throughput speedup on TPUs, and outline a research roadmap of open technical challenges to realize this vision.

Figures

Figures reproduced from arXiv: 2608.10139 by the authors.

Figure 1
Figure 1. High-level view of the unified AI Query compilation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Actual generated JAX program containing the vec [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Frontend compilation flow from SQL AST to unified [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The cartesian (cross-join) Host-to-Device (H2D) data [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Device-to-Host (D2H) data transfer overheads and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages

  1. [1]

    Samuel Arch, Yuchen Liu, Todd C Mowry, Jignesh M Pate, and Andrew Pavlo

  2. [2]

    Matthias Boehm, Matteo Interlandi, and Chris Jermaine. 2023. Optimizing tensor computations: From applications to compilation and runtime techniques. In Companion of the 2023 International Conference on Management of Data. 53–59

  3. [3]

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. 2024. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247(2024)

  4. [4]

    Periklis Chrysogelos, Panagiotis Sioulas, and Anastasia Ailamaki. 2019. Hardware-conscious query processing in gpu-accelerated analytical engines. InProceesings of the 9th Biennial Conference on Innovative Data Systems Research

  5. [5]

    Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, Yves- Laurent Kom Samo, Pushkar Khadilkar, Xianshun Chen, Sam Idicula, Fatma Özcan, et al. 2026. 100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models. InProceedings of the 2026 ACM SIGMOD International Conference on Management of Data...

  6. [6]

    Yeounoh Chung, Thibaud Hottelier, Cosmin Arad, Brenton Milne, Per Jacobsson, Sam Idicula, Fatma Özcan, and Alon Halevy. 2026. Architecting the AI-Powered Agentic Data Cloud.IEEE Data Engineering Bulletin49, 1 (March 2026), 22–31

  7. [7]

    Roy Frostig, Matthew James Johnson, and Chris Leary. 2019. Compiling machine learning programs via high-level tracing. InSysML conference 2018

  8. [8]

    Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer

Show all 32 references
  1. [9]

    2026.TPU v5e

    Google Cloud. 2026.TPU v5e. Google. https://docs.cloud.google.com/tpu/docs/ v5e Accessed: 2026-05-11

  2. [10]

    Google Cloud. 2026. Vertex AI: Google Cloud’s Unified Machine Learning Platform. https://docs.cloud.google.com/vertex-ai/docs. Accessed: May 8, 2026

  3. [11]

    Dong He, Supun Nakandala, Dalitso Banda, Rathijit Sen, Karla Saur, Kwanghyun Park, Carlo Curino, Jesús Camacho-Rodríguez, Konstantinos Karanasos, and Matteo Interlandi. 2022. Query processing on tensor computation runtimes. arXiv preprint arXiv:2203.01877(2022)

  4. [12]

    Pedro Holanda and Hannes Mühleisen. 2019. Relational queries with a ten- sor processing unit. InProceedings of the 15th International Workshop on Data Management on New Hardware. 1–3

  5. [13]

    Yu-Ching Hu, Yuliang Li, and Hung-Wei Tseng. 2022. TCUDB: Accelerating data- base with tensor processors. InProceedings of the 2022 International Conference on Management of Data. 1360–1374

  6. [14]

    Wentao Huang, Mian Lu, and Kian-Lee Tan. 2026. Hash Joins Meet CXL: A Fresh Look. InProceedings of the 2026 Conference on Innovative Data Systems Research (CIDR ’26). https://www.vldb.org/cidrdb/papers/2026/p1-huang.pdf

  7. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAtten- tion. InProceedings of the ACM SIGOPS 29th Symposium on Operating ...

  8. [16]

    Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Özcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel, Sebastian Schelter, Andreas Kipf, and Im- manuel Trummer. 2026. SemBench: A Benchmark for Se...

  9. [17]

    Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Olek- sandr Zinenko. 2021. MLIR: Scaling compiler infrastructure for domain specific computation. In2021 IEEE/ACM International Sympo...

  10. [18]

    Guanghua Li, Hao Zhang, Xibo Sun, Qiong Luo, and Yuanyuan Zhu. 2024. Ten- graph: A tensor-based graph query engine.Proceedings of the VLDB Endowment 17, 13 (2024), 4571–4584

  11. [19]

    Yinan Li, Bailu Ding, Ziyun Wei, Lukas M Maas, Momin Al-Ghosien, Spyros Blanas, Nicolas Bruno, Carlo Curino, Matteo Interlandi, Craig Peeper, et al. 2025. Scaling GPU-Accelerated Databases beyond GPU Memory Size.Proceedings of the VLDB Endowment18, 11 (2025), 4518–4531

  12. [20]

    Paweł Liskowski, Benjamin Han, Paritosh Aggarwal, Bowei Chen, Boxin Jiang, Nitish Jindal, Zihan Li, Aaron Lin, Kyle Schmaus, Jay Tayade, et al. 2026. Cortex AISQL: A Production SQL Engine for Unstructured Data. InProceedings of the 2026 ACM SIGMOD International Conference on M...

  13. [21]

    Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano

  14. [22]

    Vasilis Mageirakos, Joel André, Marko Kabić, Bowen Wu, Yannis Chronis, and Gustavo Alonso. 2026. To GPU or Not to GPU: Vector Search in Relational Engines.arXiv preprint arXiv:2605.15957(2026). arXiv:2605.15957 [cs.DB] https: //arxiv.org/abs/2605.15957

  15. [23]

    Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators and Their Optimization: Enabling LLM-Based Data Processing with Accuracy Guarantees in LOTUS. Proceedings of the VLDB Endowment (PVLDB)18, 11 (2025...

  16. [24]

    A declarative system for optimizing ai workloads.arXiv preprint arXiv:2405.14696(2024)

  17. [25]

    Peter Benjamin Volk, Dirk Habich, and Wolfgang Lehner. 2010. GPU-Based Speculative Query Processing for Database Operations.. InADMS@ VLDB. 51– 60

  18. [26]

    Kaibo Wang, Kai Zhang, Yuan Yuan, Siyuan Ma, Rubao Lee, Xiaoning Ding, and Xiaodong Zhang. 2014. Concurrent Analytical Query Processing with GPUs. Proc. VLDB Endow.7, 11 (2014), 1011–1022

  19. [27]

    Viktor Rosenfeld, Sebastian Breß, and Volker Markl. 2022. Query processing on heterogeneous CPU/GPU systems.ACM Computing Surveys (CSUR)55, 1 (2022), 1–38. 7

  20. [28]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538

  21. [29]

    Shuhao Zhang, Jiong He, Bingsheng He, and Mian Lu. 2013. OmniDB: Towards portable and efficient query processing on parallel CPU/GPU architectures. Proceedings of the VLDB Endowment6, 12 (2013), 1374–1377. 8

  22. [30]

    Wulf and Sally A

    Wm A. Wulf and Sally A. McKee. 1995. Hitting the Memory Wall: Implications of the Obvious.ACM SIGARCH Computer Architecture News23, 1 (1995), 20–24

  23. [2024]

    AI and Memory Wall.IEEE Micro44, 3 (2024), 20–32

  24. [2026]

    Partial UDF Inlining.ACM SIGMOD Record55, 1 (2026), 74–83

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.