{"id":"e5cfe2e2-72f9-42d4-aa9d-f43e9afe718a","arxiv_id":"2607.07696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-assisted synthesis of database storage readers that bypass engines and materialize PostgreSQL/MySQL data as Apache Arrow for analytical engines.","lead":"Jailbreak uses LLMs to regenerate high-performance storage readers that read PostgreSQL and MySQL files directly into Apache Arrow, bypassing JDBC/ODBC. Smart generalists should care because it attacks database lock-in and can cut analytical read latency by up to 27x for snapshot workloads.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Snapshot-only evaluation and incomplete edge-case coverage leave open whether LLM-regenerated readers remain correct under real storage complexity (TOAST, compression, indexes, multi-version layouts).","rationale":"The Reader already identified the load-bearing soft spot: that publicly available source and documentation are sufficient for an LLM to regenerate correct operator-specific readers across versions, configurations, and production edge cases beyond the evaluated TPC-H snapshot setting. My read of the abstract and the paper’s stated scope confirms the same gap. The contribution is real as a methodology for snapshot/offline analytics, validation against JDBC/ODBC on TPC-H is external and non-circular, and the speedups are plausible for a pure storage-to-Arrow path. Nothing in the available text shows systematic coverage of TOAST, compression, multi-version pages, or cross-version robustness, so the generalization claim remains conditional on fuller artifact release and edge-case evidence. That is exactly why CONDITIONAL is the right verdict; I do not see a stronger internal inconsistency that would push toward REJECT, nor enough evidence to move to ACCEPT. Agreement with the Reader is therefore full on the weakest assumption and on the verdict. The concrete test above is a minimal, reproducible way to settle whether the concern lands.","tokens_in":2161,"tokens_out":678,"duration_ms":9929,"concrete_test":"Take a PostgreSQL TPC-H scale-factor database, enable TOAST on large text/blob columns, apply page-level compression if available, and introduce concurrent updates so that multi-version heap pages and free-space maps appear; freeze a physical snapshot of the data directory. Run the Jailbreak-generated reader on that snapshot and compare full table materializations (and TPC-H query results) byte-for-byte / row-for-row against a JDBC baseline taken from a consistent read. If any column values, nulls, or row counts diverge, or if the reader fails, the load-bearing premise that source/docs alone yield correct readers beyond clean snapshots does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM-assisted synthesis can regenerate correct, high-performance storage readers that fully bypass the engine, match JDBC/ODBC TPC-H results, and deliver large analytical speedups. That claim rests on the premise that public source/docs suffice for the LLM to produce operators that correctly decode the on-disk format. The paper evaluates only analytical snapshot scenarios on TPC-H for PostgreSQL and MySQL, materializing Arrow and checking query results against driver baselines. It does not systematically demonstrate correctness under production format features that the weakest-assumption flags: TOAST/out-of-line values, compression variants, index-organized or multi-version page layouts, configuration-dependent encodings, and cross-version format drift. If those features are present in real files, an LLM-generated reader that works on clean TPC-H heap snapshots can silently mis-decode or crash, breaking both the correctness guarantee and the lock-in-breaking methodology. The reported 27x gains and “generalizable” claim therefore depend on an untested extrapolation from the snapshot regime to the full format surface that the source/docs describe.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript presents Jailbreak, a system that bypasses database engines by reading PostgreSQL and MySQL storage files directly and materializing them as Apache Arrow buffers for analytical engines (DuckDB, Spark, cuDF, Spark RAPIDS). The core technical claim is that LLMs can ingest public source code and documentation of database file formats and synthesize operator-specific storage readers without human-engineered parsers. Evaluation targets analytical snapshot workloads: correctness is checked by comparing TPC-H query results against JDBC/ODBC baselines, and end-to-end throughput improvements of up to 27× are reported. The authors position the approach as a general methodology for breaking data lock-in for any system whose on-disk format is available to an LLM from docs or source.","tokens_in":2378,"tokens_out":1511,"duration_ms":20912,"significance":"If the result holds under the claimed scope, the work is significant for analytical data systems: it reframes opaque proprietary or engine-guarded storage as a regenerable, queryable artifact and demonstrates large practical speedups by removing driver and engine layers from the bulk-read path. Strengths that should be credited include external correctness validation against JDBC/ODBC on full TPC-H result sets (not self-defined metrics), empirical end-to-end throughput measurements, and Arrow-based interoperability with multiple query engines. The LLM-assisted synthesis framing is a concrete contribution relative to hand-written open-format readers, provided the correctness and generality claims are scoped and evidenced carefully.","major_comments":[{"comment":"The abstract and conclusion claim a 'viable and generalizable methodology for breaking data lock-in across database systems' and applicability 'for any system whose file format is available to the LLM from documentation or source code.' The evaluation, however, is restricted to analytical snapshot scenarios on TPC-H for PostgreSQL and MySQL. Production format features that the source/docs describe—TOAST/out-of-line values, compression variants, multi-version/heap page layouts, configuration-dependent encodings, and cross-version format drift—are not systematically exercised. This is load-bearing for both the correctness guarantee and the lock-in-breaking methodology: a reader that matches JDBC/ODBC on clean TPC-H heap snapshots can still mis-decode or fail when those features appear. Either (a) add targeted correctness tests for these features with explicit pass/fail evidence, or (b) sub","section":"Abstract; Evaluation; Conclusion"},{"comment":"Correctness is reported as matching TPC-H query results against JDBC/ODBC baselines. That is a necessary but incomplete oracle for a storage reader: query-level agreement can mask silent mis-decodes on columns or pages not stressed by the workload, partial type coverage, or null/encoding edge cases. The manuscript should specify which storage features and type combinations were present in the evaluated files, whether page-level or row-level bit-identical checks (or equivalent structural checks against known layouts) were performed, and how failures during synthesis were detected and repaired. Without that, the claim that regenerated readers are 'correct' rests on an underspecified validation surface relative to the full decode path.","section":"Evaluation / Correctness validation (TPC-H vs JDBC/ODBC)"},{"comment":"The pipeline depends on LLM model choice, prompts, and iterative synthesis, listed as free parameters. For a systems contribution claiming regeneration without human-engineered parsing logic, the paper needs a clearer account of: (i) how much human intervention remains (prompt engineering, failure triage, schema/operator scoping), (ii) reproducibility of the generated readers (are artifacts fixed and shipped, or re-synthesized?), and (iii) sensitivity of correctness/performance to model and prompt. If readers are tuned until they match baselines, that process must be described so the result is not an unreproducible one-off. This does not invalidate the empirical speedups, but it is load-bearing for the 'LLM-assisted synthesis is a viable methodology' claim.","section":"Method / Jailbreak synthesis pipeline"},{"comment":"Performance claims of up to 27× end-to-end analytical throughput are central. The report should make explicit the baseline configuration (JDBC/ODBC fetch sizes, serialization format, whether the engine was cold/warm, concurrent load, and whether comparison is fair against best-practice bulk export such as COPY/OUTFILE or parallel drivers). Snapshot-only bypass vs. a live engine path can inflate speedups for reasons orthogonal to reader quality. Table or section-level breakdowns separating decode time, Arrow materialization, and query execution would ground the 27× figure and show how much is truly attributable to storage bypass.","section":"Evaluation / Performance results"}],"minor_comments":[{"comment":"Clarify the threat model and operational assumptions early: offline snapshots and read replicas are mentioned, but production constraints (file locking, concurrent writers, WAL/visibility, permissions) should be listed so readers do not over-interpret 'bypass the database engine entirely.'","section":"Introduction / Scope"},{"comment":"Define 'operator-specific table reading components' more precisely: which operators, projection/filter pushdown into the reader, and whether the synthesized code is per-table, per-schema, or per-engine version.","section":"Method"},{"comment":"Related work should more explicitly contrast with existing open storage readers, foreign data wrappers, and tools that already read Postgres/MySQL files or export to Arrow/Parquet, so the incremental contribution of LLM synthesis is sharp.","section":"Related Work"},{"comment":"Report LLM model versions, prompt structure (even at high level), and whether generated code is released; this aids reproducibility without requiring full prompt dumps in the main text.","section":"Experimental setup"},{"comment":"Figures and tables reporting the 27× result should include absolute throughputs and variance across runs, not only relative speedups.","section":"Evaluation figures/tables"},{"comment":"Minor terminology consistency: 'Jailbreak' as system name vs. informal 'jailbreak' usage; keep capitalization consistent throughout.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the external TPC-H/JDBC validation is a real strength relative to many LLM-for-systems papers. My major_revision recommendation is driven by overclaim of generality and underspecified correctness surface, not by disbelief in the snapshot speedups. If the authors tightly scope to snapshot analytics and add feature-level decode tests (or honestly limit claims), this could become a solid systems paper. Fit for cs.DB is appropriate; watch for novelty overlap with prior direct-storage and Arrow export work when assessing final contribution."},"author_rebuttal":{"model":"grok-4.5","summary":"We thank the referee for a careful and constructive review. The four major comments correctly identify places where our claims outrun the evaluation surface and where methodological detail is underspecified. We agree that the abstract/conclusion generality language, the correctness oracle, the synthesis process, and the performance baseline all need tighter scoping and more evidence. We will revise the manuscript accordingly: narrow the lock-in claims to the evaluated analytical-snapshot regime (with an explicit limitations section on unexercised format features), expand the correctness and synthesis methodology sections, and add baseline configuration detail plus a time breakdown for the reported speedups. We address each major comment below.","responses":[{"response":"We agree that the current abstract and conclusion overstate generality relative to the evaluation. Our experiments target analytical snapshot workloads on TPC-H for PostgreSQL and MySQL; they do not systematically exercise TOAST/out-of-line values, compression variants, multi-version/heap page layouts under concurrent update, configuration-dependent encodings, or cross-version format drift. A reader that matches JDBC/ODBC on clean TPC-H heaps can still fail on those features, so the lock-in-breaking methodology claim must be scoped.\n\nIn revision we will (b) narrow the abstract, introduction, and conclusion to the evaluated regime—analytical snapshots / read-replica and offline pipelines for the formats and versions we actually regenerated—and add an explicit Limitations subsection listing unexercised production features and the risk of silent mis-decode. Where feasible within revision time, we will also (a) add targeted micro-tests for features already present or easily enabled in our dumps (e.g., TOAST for large text/bytea, null bitmaps, common type encodings) with pass/fail against JDBC, and report failures honestly. We will not claim systematic coverage of compression variants, live multi-version concurrency, or cross-version drift without evidence. The methodological framing (LLM ingestion of public format specs/source to synthesize Arrow readers) remains, but as a demonstrated approach for the scoped setting rather than a proven universal decoder.","revision_made":"yes","referee_comment":"[Abstract; Evaluation; Conclusion] The abstract and conclusion claim a 'viable and generalizable methodology for breaking data lock-in across database systems' and applicability 'for any system whose file format is available to the LLM from documentation or source code.' The evaluation, however, is restricted to analytical snapshot scenarios on TPC-H for PostgreSQL and MySQL. Production format features (TOAST/out-of-line values, compression variants, multi-version/heap page layouts, configuration-dependent encodings, cross-version format drift) are not systematically exercised. Either (a) add targeted correctness tests for these features with explicit pass/fail evidence, or (b) substantially narrow the claims."},{"response":"The referee is right: full TPC-H result agreement against JDBC/ODBC is necessary but not a complete oracle for a storage decoder. Query-level equality can hide mis-decodes on unstressed columns, pages, nulls, or encodings.\n\nIn revision we will: (1) document the concrete storage surface of the evaluated dumps—PostgreSQL/MySQL versions, heap/page layout assumptions, which TPC-H types and nullability patterns appear, and whether TOAST or compressed pages were present; (2) describe validation beyond query results, including row-count and schema checks, per-column type/null checks where implemented, and any structural or sample-level comparisons against known layouts or against engine-exported ground truth (e.g., COPY/CSV or Arrow from a trusted path); (3) state clearly what we did not do (full page-level bit-identical decode of every page against an independent reference implementation) so the validation surface is not oversold; (4) describe how synthesis failures were detected (compile/runtime errors, schema mismatches, result mismatches on held-out checks) and how they were repaired (prompt iteration, scoped re-synthesis, human triage). We will not claim bit-identical page fidelity unless we add such checks. This keeps the TPC-H end-to-end oracle as primary evidence while making its limits explicit.","revision_made":"yes","referee_comment":"[Evaluation / Correctness validation (TPC-H vs JDBC/ODBC)] Correctness is reported as matching TPC-H query results against JDBC/ODBC baselines. That is a necessary but incomplete oracle for a storage reader: query-level agreement can mask silent mis-decodes on columns or pages not stressed by the workload, partial type coverage, or null/encoding edge cases. The manuscript should specify which storage features and type combinations were present, whether page-level or row-level bit-identical checks (or equivalent structural checks) were performed, and how failures during synthesis were detected and repaired."},{"response":"We agree this is load-bearing for the 'LLM-assisted synthesis is a viable methodology' claim and is underspecified in the current draft.\n\nRevision plan: (i) Human intervention—we will enumerate remaining human roles: choosing model and context (docs/source excerpts), writing/refining the synthesis prompts and operator/schema scope, triaging compile and runtime failures, and deciding when a reader is 'done' against the oracle. We will state explicitly that we do not hand-write format parsers, but that prompt engineering and failure triage are part of the methodology, not zero-touch automation. (ii) Reproducibility—we will clarify whether evaluation uses fixed, shipped generated artifacts (preferred for the paper's numbers) or live re-synthesis; we will commit to releasing the generated readers used in experiments (and prompts/context where license allows) so results do not depend on a one-off chat session. (iii) Sensitivity—we will report the model(s) used, note that correctness was achieved under iterative synthesis against the oracle, and, where space allows, add a short note or appendix on alternative models/prompt variants tried and outcomes (success, failure modes). We will not claim model-agnostic robustness without data. These additions do not change the empirical speedups but make the methodology auditable.","revision_made":"yes","referee_comment":"[Method / Jailbreak synthesis pipeline] The pipeline depends on LLM model choice, prompts, and iterative synthesis, listed as free parameters. For a systems contribution claiming regeneration without human-engineered parsing logic, the paper needs a clearer account of: (i) how much human intervention remains (prompt engineering, failure triage, schema/operator scoping), (ii) reproducibility of the generated readers (are artifacts fixed and shipped, or re-synthesized?), and (iii) sensitivity of correctness/performance to model and prompt. If readers are tuned until they match baselines, that process must be described so the result is not an unreproducible one-off."},{"response":"We agree the 27× figure needs a fairer, more transparent baseline and a breakdown of where time is spent.\n\nIn revision we will: (1) fully specify the JDBC/ODBC baseline—driver versions, fetch size / batch settings, result serialization path, single- vs multi-connection use, cold vs warm cache, and whether the DB process was idle or under concurrent load; (2) discuss fairness relative to best-practice bulk export (PostgreSQL COPY, MySQL OUTFILE/SELECT INTO, and parallel or bulk drivers where applicable), and either add such a comparison or clearly state that the primary baseline is the interactive analytical path via JDBC/ODBC (the lock-in path we target), not the fastest possible dump tool; (3) add a breakdown separating storage decode, Arrow materialization, and downstream query execution (and transfer where relevant) so readers can see how much of the speedup is attributable to engine/driver bypass versus other factors; (4) restate that snapshot-only bypass omits live concurrency and recovery costs, which can inflate speedups relative to a production primary—consistent with our scoped analytical-snapshot setting. We will keep the end-to-end numbers but ground them so the 27× claim is not read as a universal win over every export path.","revision_made":"yes","referee_comment":"[Evaluation / Performance results] Performance claims of up to 27× end-to-end analytical throughput are central. The report should make explicit the baseline configuration (JDBC/ODBC fetch sizes, serialization format, whether the engine was cold/warm, concurrent load, and whether comparison is fair against best-practice bulk export such as COPY/OUTFILE or parallel drivers). Snapshot-only bypass vs. a live engine path can inflate speedups for reasons orthogonal to reader quality. Table or section-level breakdowns separating decode time, Arrow materialization, and query execution would ground the 27× figure."}],"tokens_in":2220,"tokens_out":1888,"duration_ms":21185,"standing_objections":[]},"desk_editor":{"model":"grok-4.5","letter":"The one thing to know: Jailbreak shows LLM-assisted synthesis can regenerate operator-specific storage readers for Postgres and MySQL that skip JDBC/ODBC, materialize Arrow, match TPC-H results against driver baselines, and report large end-to-end analytical speedups (up to 27x) on snapshot workloads. That is a concrete artifact and a usable methodology for lock-in, not a rehash of “LLMs write code.”\n\nWhat is actually new is the framing and the pipeline: treat public source and docs as a complete enough format spec, regenerate table readers without hand-written parsers, and feed modern columnar engines (DuckDB, Spark, cuDF/RAPIDS) from raw DB files. Direct file readers and Arrow materialization already exist; LLM code synthesis already exists. Combining them for engine bypass on real storage formats is a solid systems contribution. Correctness is checked externally against JDBC/ODBC TPC-H results, so circularity is low. The evaluation design for the stated regime—analytical snapshots, read replicas, offline pipelines—is defensible.\n\nSoft spots, in proportion: the paper is snapshot-only on two engines. The stress-test concern lands: TOAST/out-of-line values, compression variants, multi-version page layouts, indexes, and config-dependent encodings are not systematically shown. If those appear in real files, a reader that works on clean TPC-H heaps can mis-decode. That weakens the “generalizable across any documented format” claim more than it weakens the snapshot result itself. LLM model/prompt choices and snapshot config are free parameters; fuller artifact release would help. None of that makes the core snapshot result look fake—it makes the extrapolation soft.\n\nThis is for data-systems and lakehouse people who care about I/O path and driver lock-in. It deserves a serious referee, not a desk reject. I would send it to review and ask for edge-case coverage, version robustness, and synthesis artifacts. Worth a reading-group slot if the group does systems or LLM-for-systems; I would cite the methodology if I were writing on storage bypass or agentic systems tooling.","headline":"LLM-synthesized Postgres/MySQL storage readers that bypass the engine into Arrow are a real systems contribution; the snapshot-only eval is the main soft spot, not a load-bearing collapse.","tokens_in":3067,"tokens_out":549,"would_cite":false,"duration_ms":13874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM-generated storage readers bypass Postgres and MySQL engines for up to 27× faster analytics","keywords":["database bypass","LLM code synthesis","storage readers","Apache Arrow","PostgreSQL","MySQL","analytical workloads","TPC-H"],"falsifier":"A TPC-H or production snapshot where the generated PostgreSQL or MySQL reader either fails to match JDBC/ODBC result sets on any query or falls below baseline throughput once real compression, TOAST, indexes, or page-layout variants are present.","tokens_in":2993,"feed_emoji":"🗄️","tokens_out":782,"duration_ms":11343,"temperature":0.7,"pith_summary":"Analytical tools that pull data from external databases normally must go through JDBC or ODBC drivers, so every bulk read pays the cost of the database engine, its query layer, and row-oriented transport. Jailbreak argues that this lock-in is unnecessary for snapshot-style workloads: database storage formats are fully specified by source code and documentation that large language models can read, so an LLM can regenerate operator-specific table readers that open the files directly, decode pages, and emit Apache Arrow columnar buffers. Those buffers are then consumable by DuckDB, Spark, cuDF, and similar engines without ever contacting the original database process. On PostgreSQL and MySQL storage files under TPC-H, the generated readers match JDBC/ODBC query results and deliver up to 27× higher end-to-end analytical throughput. If the method generalizes, any system whose on-disk format is public becomes a directly queryable artifact rather than a gated service.","feed_headline":"LLM readers bypass Postgres and MySQL for up to 27× analytics","feed_subtitle":"Generated storage decoders emit Arrow directly, match TPC-H results, and skip JDBC/ODBC entirely.","key_machinery":"Jailbreak: an LLM-driven synthesis pipeline that ingests database storage format specifications (source and docs), emits operator-specific table readers, and materializes in-memory Apache Arrow buffers consumable by external analytical engines without human-written parsers.","core_discovery":"Database file formats can be turned into directly queryable columnar artifacts by LLM-assisted code synthesis: given source and documentation, an LLM regenerates correct, operator-specific storage readers for PostgreSQL and MySQL that bypass the engine, materialize Apache Arrow, match full TPC-H results against JDBC/ODBC baselines, and yield up to 27× end-to-end analytical speedups on snapshot workloads.","pith_inferences":["The same synthesis loop could target other open formats (e.g., other RDBMS page layouts or columnar stores) once their sources are in the model context.","Production viability likely hinges on automated regression harnesses that re-validate regenerated readers against live engines after every version bump.","If format ambiguity remains high, hybrid approaches that keep a thin verification path through the original engine may still be required."],"forward_implications":["Analytical engines can read Postgres and MySQL data files directly as Arrow without standing up a live database process or paying driver overhead.","Read-replica and offline pipeline workloads that today are bottlenecked on JDBC/ODBC can see large end-to-end throughput gains.","Any database whose on-disk format is available to an LLM becomes a candidate for the same bypass pattern.","Operator-specific readers can be regenerated when formats change, reducing the need for permanent hand-maintained parsers."],"fun_headline_variants":["LLM regenerates storage readers to bypass Postgres MySQL for 27× analytics","Jailbreak: LLM code synthesis turns DB files into direct Arrow buffers","Agentic regeneration of Postgres MySQL readers yields 27× end-to-end speedups","Bypass JDBC entirely: LLM-built readers materialize Arrow from storage files","LLMs ingest DB formats to synthesize high-performance columnar readers"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Public source code and documentation specify storage formats completely and stably enough for an LLM to regenerate correct readers across versions, configurations, and production edge cases without hand-written parsing logic.","fun_headline_variants_meta":{"raw":{"variants":["LLM regenerates storage readers to bypass Postgres MySQL for 27× analytics","Jailbreak: LLM code synthesis turns DB files into direct Arrow buffers","Agentic regeneration of Postgres MySQL readers yields 27× end-to-end speedups","Bypass JDBC entirely: LLM-built readers materialize Arrow from storage files","LLMs ingest DB formats to synthesize high-performance columnar readers"]},"model":"grok-4.5","cost_usd":0.013118,"raw_usage":{"total_tokens":2791,"prompt_tokens":859,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":131180000,"prompt_tokens_details":{"text_tokens":859,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1849,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":859,"tokens_out":83,"duration_ms":58559,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T07:10:46.639949+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A TPC-H or production snapshot where the generated PostgreSQL or MySQL reader either fails to match JDBC/ODBC result sets on any query or falls below baseline throughput once real compression, TOAST, indexes, or page-layout variants are present.","supporting_citations":[],"review_version":2}