REVIEW 4 major objections 6 minor 18 references
LLM-generated storage readers bypass Postgres and MySQL engines for up to 27× faster analytics
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-09 07:10 UTC pith:PTNJINBW
load-bearing objection LLM-synthesized Postgres/MySQL storage readers that bypass the engine into Arrow are a real systems contribution; the snapshot-only eval is the main soft spot, not a load-bearing collapse. the 4 major comments →
Breaking Database Lock-in: Agentic Regeneration of High Performance Storage Readers for Database Bypass
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Database file formats can be turned into directly queryable columnar artifacts by LLM-assisted code synthesis: given source and documentation, an LLM regenerates correct, operator-specific storage readers for PostgreSQL and MySQL that bypass the engine, materialize Apache Arrow, match full TPC-H results against JDBC/ODBC baselines, and yield up to 27× end-to-end analytical speedups on snapshot workloads.
What carries the argument
Jailbreak: an LLM-driven synthesis pipeline that ingests database storage format specifications (source and docs), emits operator-specific table readers, and materializes in-memory Apache Arrow buffers consumable by external analytical engines without human-written parsers.
Load-bearing premise
Public source code and documentation specify storage formats completely and stably enough for an LLM to regenerate correct readers across versions, configurations, and production edge cases without hand-written parsing logic.
What would settle it
A TPC-H or production snapshot where the generated PostgreSQL or MySQL reader either fails to match JDBC/ODBC result sets on any query or falls below baseline throughput once real compression, TOAST, indexes, or page-layout variants are present.
If this is right
- Analytical engines can read Postgres and MySQL data files directly as Arrow without standing up a live database process or paying driver overhead.
- Read-replica and offline pipeline workloads that today are bottlenecked on JDBC/ODBC can see large end-to-end throughput gains.
- Any database whose on-disk format is available to an LLM becomes a candidate for the same bypass pattern.
- Operator-specific readers can be regenerated when formats change, reducing the need for permanent hand-maintained parsers.
Where Pith is reading between the lines
- The same synthesis loop could target other open formats (e.g., other RDBMS page layouts or columnar stores) once their sources are in the model context.
- Production viability likely hinges on automated regression harnesses that re-validate regenerated readers against live engines after every version bump.
- If format ambiguity remains high, hybrid approaches that keep a thin verification path through the original engine may still be required.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents Jailbreak, a system that bypasses database engines by reading PostgreSQL and MySQL storage files directly and materializing them as Apache Arrow buffers for analytical engines (DuckDB, Spark, cuDF, Spark RAPIDS). The core technical claim is that LLMs can ingest public source code and documentation of database file formats and synthesize operator-specific storage readers without human-engineered parsers. Evaluation targets analytical snapshot workloads: correctness is checked by comparing TPC-H query results against JDBC/ODBC baselines, and end-to-end throughput improvements of up to 27× are reported. The authors position the approach as a general methodology for breaking data lock-in for any system whose on-disk format is available to an LLM from docs or source.
Significance. If the result holds under the claimed scope, the work is significant for analytical data systems: it reframes opaque proprietary or engine-guarded storage as a regenerable, queryable artifact and demonstrates large practical speedups by removing driver and engine layers from the bulk-read path. Strengths that should be credited include external correctness validation against JDBC/ODBC on full TPC-H result sets (not self-defined metrics), empirical end-to-end throughput measurements, and Arrow-based interoperability with multiple query engines. The LLM-assisted synthesis framing is a concrete contribution relative to hand-written open-format readers, provided the correctness and generality claims are scoped and evidenced carefully.
major comments (4)
- [Abstract; Evaluation; Conclusion] The abstract and conclusion claim a 'viable and generalizable methodology for breaking data lock-in across database systems' and applicability 'for any system whose file format is available to the LLM from documentation or source code.' The evaluation, however, is restricted to analytical snapshot scenarios on TPC-H for PostgreSQL and MySQL. Production format features that the source/docs describe—TOAST/out-of-line values, compression variants, multi-version/heap page layouts, configuration-dependent encodings, and cross-version format drift—are not systematically exercised. This is load-bearing for both the correctness guarantee and the lock-in-breaking methodology: a reader that matches JDBC/ODBC on clean TPC-H heap snapshots can still mis-decode or fail when those features appear. Either (a) add targeted correctness tests for these features with explicit pass/fail evidence, or (b) sub
- [Evaluation / Correctness validation (TPC-H vs JDBC/ODBC)] Correctness is reported as matching TPC-H query results against JDBC/ODBC baselines. That is a necessary but incomplete oracle for a storage reader: query-level agreement can mask silent mis-decodes on columns or pages not stressed by the workload, partial type coverage, or null/encoding edge cases. The manuscript should specify which storage features and type combinations were present in the evaluated files, whether page-level or row-level bit-identical checks (or equivalent structural checks against known layouts) were performed, and how failures during synthesis were detected and repaired. Without that, the claim that regenerated readers are 'correct' rests on an underspecified validation surface relative to the full decode path.
- [Method / Jailbreak synthesis pipeline] The pipeline depends on LLM model choice, prompts, and iterative synthesis, listed as free parameters. For a systems contribution claiming regeneration without human-engineered parsing logic, the paper needs a clearer account of: (i) how much human intervention remains (prompt engineering, failure triage, schema/operator scoping), (ii) reproducibility of the generated readers (are artifacts fixed and shipped, or re-synthesized?), and (iii) sensitivity of correctness/performance to model and prompt. If readers are tuned until they match baselines, that process must be described so the result is not an unreproducible one-off. This does not invalidate the empirical speedups, but it is load-bearing for the 'LLM-assisted synthesis is a viable methodology' claim.
- [Evaluation / Performance results] Performance claims of up to 27× end-to-end analytical throughput are central. The report should make explicit the baseline configuration (JDBC/ODBC fetch sizes, serialization format, whether the engine was cold/warm, concurrent load, and whether comparison is fair against best-practice bulk export such as COPY/OUTFILE or parallel drivers). Snapshot-only bypass vs. a live engine path can inflate speedups for reasons orthogonal to reader quality. Table or section-level breakdowns separating decode time, Arrow materialization, and query execution would ground the 27× figure and show how much is truly attributable to storage bypass.
minor comments (6)
- [Introduction / Scope] Clarify the threat model and operational assumptions early: offline snapshots and read replicas are mentioned, but production constraints (file locking, concurrent writers, WAL/visibility, permissions) should be listed so readers do not over-interpret 'bypass the database engine entirely.'
- [Method] Define 'operator-specific table reading components' more precisely: which operators, projection/filter pushdown into the reader, and whether the synthesized code is per-table, per-schema, or per-engine version.
- [Related Work] Related work should more explicitly contrast with existing open storage readers, foreign data wrappers, and tools that already read Postgres/MySQL files or export to Arrow/Parquet, so the incremental contribution of LLM synthesis is sharp.
- [Experimental setup] Report LLM model versions, prompt structure (even at high level), and whether generated code is released; this aids reproducibility without requiring full prompt dumps in the main text.
- [Evaluation figures/tables] Figures and tables reporting the 27× result should include absolute throughputs and variance across runs, not only relative speedups.
- [Throughout] Minor terminology consistency: 'Jailbreak' as system name vs. informal 'jailbreak' usage; keep capitalization consistent throughout.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The four major comments correctly identify places where our claims outrun the evaluation surface and where methodological detail is underspecified. We agree that the abstract/conclusion generality language, the correctness oracle, the synthesis process, and the performance baseline all need tighter scoping and more evidence. We will revise the manuscript accordingly: narrow the lock-in claims to the evaluated analytical-snapshot regime (with an explicit limitations section on unexercised format features), expand the correctness and synthesis methodology sections, and add baseline configuration detail plus a time breakdown for the reported speedups. We address each major comment below.
read point-by-point responses
-
Referee: [Abstract; Evaluation; Conclusion] The abstract and conclusion claim a 'viable and generalizable methodology for breaking data lock-in across database systems' and applicability 'for any system whose file format is available to the LLM from documentation or source code.' The evaluation, however, is restricted to analytical snapshot scenarios on TPC-H for PostgreSQL and MySQL. Production format features (TOAST/out-of-line values, compression variants, multi-version/heap page layouts, configuration-dependent encodings, cross-version format drift) are not systematically exercised. Either (a) add targeted correctness tests for these features with explicit pass/fail evidence, or (b) substantially narrow the claims.
Authors: We agree that the current abstract and conclusion overstate generality relative to the evaluation. Our experiments target analytical snapshot workloads on TPC-H for PostgreSQL and MySQL; they do not systematically exercise TOAST/out-of-line values, compression variants, multi-version/heap page layouts under concurrent update, configuration-dependent encodings, or cross-version format drift. A reader that matches JDBC/ODBC on clean TPC-H heaps can still fail on those features, so the lock-in-breaking methodology claim must be scoped. In revision we will (b) narrow the abstract, introduction, and conclusion to the evaluated regime—analytical snapshots / read-replica and offline pipelines for the formats and versions we actually regenerated—and add an explicit Limitations subsection listing unexercised production features and the risk of silent mis-decode. Where feasible within revision time, we will also (a) add targeted micro-tests for features already present or easily enabled in our dumps (e.g., TOAST for large text/bytea, null bitmaps, common type encodings) with pass/fail against JDBC, and report failures honestly. We will not claim systematic coverage of compression variants, live multi-version concurrency, or cross-version drift without evidence. The methodological framing (LLM ingestion of public format specs/source to synthesize Arrow readers) remains, but as a demonstrated approach for the scoped setting rather than a proven universal decoder. revision: yes
-
Referee: [Evaluation / Correctness validation (TPC-H vs JDBC/ODBC)] Correctness is reported as matching TPC-H query results against JDBC/ODBC baselines. That is a necessary but incomplete oracle for a storage reader: query-level agreement can mask silent mis-decodes on columns or pages not stressed by the workload, partial type coverage, or null/encoding edge cases. The manuscript should specify which storage features and type combinations were present, whether page-level or row-level bit-identical checks (or equivalent structural checks) were performed, and how failures during synthesis were detected and repaired.
Authors: The referee is right: full TPC-H result agreement against JDBC/ODBC is necessary but not a complete oracle for a storage decoder. Query-level equality can hide mis-decodes on unstressed columns, pages, nulls, or encodings. In revision we will: (1) document the concrete storage surface of the evaluated dumps—PostgreSQL/MySQL versions, heap/page layout assumptions, which TPC-H types and nullability patterns appear, and whether TOAST or compressed pages were present; (2) describe validation beyond query results, including row-count and schema checks, per-column type/null checks where implemented, and any structural or sample-level comparisons against known layouts or against engine-exported ground truth (e.g., COPY/CSV or Arrow from a trusted path); (3) state clearly what we did not do (full page-level bit-identical decode of every page against an independent reference implementation) so the validation surface is not oversold; (4) describe how synthesis failures were detected (compile/runtime errors, schema mismatches, result mismatches on held-out checks) and how they were repaired (prompt iteration, scoped re-synthesis, human triage). We will not claim bit-identical page fidelity unless we add such checks. This keeps the TPC-H end-to-end oracle as primary evidence while making its limits explicit. revision: yes
-
Referee: [Method / Jailbreak synthesis pipeline] The pipeline depends on LLM model choice, prompts, and iterative synthesis, listed as free parameters. For a systems contribution claiming regeneration without human-engineered parsing logic, the paper needs a clearer account of: (i) how much human intervention remains (prompt engineering, failure triage, schema/operator scoping), (ii) reproducibility of the generated readers (are artifacts fixed and shipped, or re-synthesized?), and (iii) sensitivity of correctness/performance to model and prompt. If readers are tuned until they match baselines, that process must be described so the result is not an unreproducible one-off.
Authors: We agree this is load-bearing for the 'LLM-assisted synthesis is a viable methodology' claim and is underspecified in the current draft. Revision plan: (i) Human intervention—we will enumerate remaining human roles: choosing model and context (docs/source excerpts), writing/refining the synthesis prompts and operator/schema scope, triaging compile and runtime failures, and deciding when a reader is 'done' against the oracle. We will state explicitly that we do not hand-write format parsers, but that prompt engineering and failure triage are part of the methodology, not zero-touch automation. (ii) Reproducibility—we will clarify whether evaluation uses fixed, shipped generated artifacts (preferred for the paper's numbers) or live re-synthesis; we will commit to releasing the generated readers used in experiments (and prompts/context where license allows) so results do not depend on a one-off chat session. (iii) Sensitivity—we will report the model(s) used, note that correctness was achieved under iterative synthesis against the oracle, and, where space allows, add a short note or appendix on alternative models/prompt variants tried and outcomes (success, failure modes). We will not claim model-agnostic robustness without data. These additions do not change the empirical speedups but make the methodology auditable. revision: yes
-
Referee: [Evaluation / Performance results] Performance claims of up to 27× end-to-end analytical throughput are central. The report should make explicit the baseline configuration (JDBC/ODBC fetch sizes, serialization format, whether the engine was cold/warm, concurrent load, and whether comparison is fair against best-practice bulk export such as COPY/OUTFILE or parallel drivers). Snapshot-only bypass vs. a live engine path can inflate speedups for reasons orthogonal to reader quality. Table or section-level breakdowns separating decode time, Arrow materialization, and query execution would ground the 27× figure.
Authors: We agree the 27× figure needs a fairer, more transparent baseline and a breakdown of where time is spent. In revision we will: (1) fully specify the JDBC/ODBC baseline—driver versions, fetch size / batch settings, result serialization path, single- vs multi-connection use, cold vs warm cache, and whether the DB process was idle or under concurrent load; (2) discuss fairness relative to best-practice bulk export (PostgreSQL COPY, MySQL OUTFILE/SELECT INTO, and parallel or bulk drivers where applicable), and either add such a comparison or clearly state that the primary baseline is the interactive analytical path via JDBC/ODBC (the lock-in path we target), not the fastest possible dump tool; (3) add a breakdown separating storage decode, Arrow materialization, and downstream query execution (and transfer where relevant) so readers can see how much of the speedup is attributable to engine/driver bypass versus other factors; (4) restate that snapshot-only bypass omits live concurrency and recovery costs, which can inflate speedups relative to a production primary—consistent with our scoped analytical-snapshot setting. We will keep the end-to-end numbers but ground them so the 27× claim is not read as a universal win over every export path. revision: yes
Circularity Check
No load-bearing circular derivation; claims are empirical systems results checked against external JDBC/ODBC and TPC-H baselines.
full rationale
Jailbreak is an empirical systems paper: LLM-assisted synthesis of storage readers for PostgreSQL/MySQL, materializing Apache Arrow, validated by matching TPC-H query results against JDBC/ODBC drivers and by measuring end-to-end analytical throughput (up to 27x). Correctness and performance are not derived from self-defined quantities, fitted parameters renamed as predictions, uniqueness theorems imported from the same authors, or ansatzes smuggled via self-citation. The derivation chain is methodological (ingest public source/docs → synthesize readers → validate externally), not algebraic or definitional. The only minor self-referential risk is iterative generation/tuning until outputs match external baselines; that is ordinary validation methodology, not a circular reduction of the central claim to its inputs. Score 1 (not 0) only to acknowledge that residual risk; no CircularStep meets the quote-and-exhibit bar for kinds 1–6. Paper is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (3)
- LLM model and prompt configuration
- Target schema / operator-specific reader scope
- Evaluation snapshot and configuration settings
axioms (5)
- domain assumption Database on-disk formats are fully recoverable from available source code and documentation for correct third-party reading.
- domain assumption Analytical snapshot / read-replica / offline pipeline workloads do not require live transactional concurrency through the engine.
- domain assumption Apache Arrow is a sufficient interchange format for major analytical engines (DuckDB, Spark, cuDF, RAPIDS).
- domain assumption TPC-H result equivalence to JDBC/ODBC is an adequate correctness oracle for the regenerated readers.
- standard math Standard code-generation and systems evaluation methodology (LLM synthesis + empirical benchmarks).
invented entities (1)
-
Jailbreak agentic storage-reader synthesis pipeline
independent evidence
read the original abstract
Analytical workloads operating on data stored in external database systems face a fundamental bottleneck: data access is guarded entirely by the database driver, like JDBC or ODBC, forcing all reads through query execution and other driver layers that are not designed for bulk columnar analytics. We present Jailbreak, an approach that bypasses the database engine entirely by reading storage files directly and materializing data as in-memory columnar buffers. Jailbreak's key insight is that database file formats, while complex, are fully specified by their source code and documentation, artifacts that Large Language Models (LLMs) can ingest to regenerate operator-specific table reading components without human-engineered parsing logic. Jailbreak leverages LLM-assisted code synthesis for database storage decoding, turning a traditionally opaque format into a directly queryable artifact. We evaluate Jailbreak on PostgreSQL and MySQL storage files, targeting analytical snapshot scenarios common in read replicas and offline processing pipelines. The generated reader produces Apache Arrow buffers consumable directly by most of the widely known query engines, including DuckDB, Apache Spark, and GPU-accelerated frameworks such as cuDF and Spark RAPIDS. We validate correctness against JDBC/ODBC-based baselines using the TPC-H benchmark across all query results, and demonstrate significant performance improvements in end-to-end analytical throughput, achieving up to 27x speedups. Our results showcase that LLM-assisted storage reader synthesis is a viable and generalizable methodology for breaking data lock-in across database systems, with applications beyond PostgreSQL and MySQL for any system whose file format is available to the LLM from documentation or source code.
Figures
Reference graph
Works this paper leans on
- [1]
- [2]
-
[3]
[n.d.]. cuDF. https://github.com/rapidsai/cudf
-
[4]
Kaustubh Beedkar, Bertty Contreras-Rojas, Haralampos Gavriilidis, Zoi Kaoudi, Volker Markl, Rodrigo Pardo-Meza, and Jorge-Arnulfo Quiané-Ruiz. 2023. Apache wayang: A unified data analytics framework.ACM SIGMOD Record52, 3 (2023), 30–35
work page 2023
-
[5]
Haralampos Gavriilidis, Kaustubh Beedkar, Matthias Boehm, and Volker Markl
-
[6]
Fast and Scalable Data Transfer Across Data Systems.Proceedings of the ACM on Management of Data3, 3 (2025), 1–28
work page 2025
-
[7]
Haralampos Gavriilidis, Leonhard Rose, Joel Ziegler, Kaustubh Beedkar, Jorge- Arnulfo Quiané-Ruiz, and Volker Markl. 2023. XDB in Action: Decentralized Cross-Database Query Processing for Black-Box DBMSes.Proc. VLDB Endow.16, 12 (2023), 4078–4081
work page 2023
-
[8]
Victor Giannakouris, Nikolaos Papailiou, Dimitrios Tsoumakos, and Nectarios Koziris. 2016. MuSQLE: Distributed SQL query execution over multiple engine environments. In2016 IEEE International Conference on Big Data (Big Data). IEEE, 452–461
work page 2016
-
[9]
Victor Giannakouris and Immanuel Trummer. [n.d.]. Rethinking Pluggable Federated Query Optimization: From Laptops to Data Warehouses.Proceedings of the VLDB Endowment. ISSN2150 ([n. d.]), 8097
-
[10]
Manos Karpathiotakis, Avrilia Floratou, Fatma Özcan, and Anastasia Ailamaki
-
[11]
In Proceedings of the 2017 Symposium on Cloud Computing
No data left behind: real-time insights from a complex data ecosystem. In Proceedings of the 2017 Symposium on Cloud Computing. 108–120
work page 2017
- [12]
-
[13]
Pedro Pedreira, Orri Erling, Maria Basmanova, Kevin Wilfong, Laith Sakka, Krishna Pai, Wei He, and Biswapesh Chattopadhyay. 2022. Velox: Meta’s Unified Execution Engine.Proc. VLDB Endow.15, 12 (2022), 3372–3384
work page 2022
-
[14]
Mark Raasveldt and Hannes Mühleisen. 2017. Don’t hold my data hostage: a case for client protocol redesign.Proceedings of the VLDB Endowment10, 10 (2017), 1022–1033
work page 2017
-
[15]
Mark Raasveldt and Hannes Mühleisen. 2019. Duckdb: an embeddable analytical database. InProceedings of the 2019 international conference on management of data. 1981–1984
work page 2019
-
[16]
Xiaoying Wang, Weiyuan Wu, Jinze Wu, Yizhou Chen, Nick Zrymiak, Changbo Qu, Lampros Flokas, George Chow, Jiannan Wang, Tianzheng Wang, et al. 2022. ConnectorX: accelerating data loading from databases to dataframes.Proceedings of the VLDB Endowment15, 11 (2022), 2994–3003
work page 2022
-
[17]
Johannes Wehrstein, Timo Eckmann, Matthias Jasny, and Carsten Binnig. 2026. Bespoke OLAP: Synthesizing Workload-Specific One-size-fits-one Database En- gines.arXiv preprint arXiv:2603.02001(2026)
work page internal anchor Pith review arXiv 2026
-
[18]
Matei Zaharia, Reynold S Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J Franklin, et al. 2016. Apache spark: a unified engine for big data processing. Commun. ACM59, 11 (2016), 56–65. 8
work page 2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.