{"id":"b9e4f7c8-1ced-4ec5-8e57-70f562337f71","arxiv_id":"2602.10387","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-proposed JSON-patch edits to Apache DataFusion physical plans, refined by evolutionary search, yield median ~1.1x speedups and up to 4.78x on generated TPC-H/TPC-DS queries.","lead":"This paper builds a harness that shows an LLM the internal execution plan of a database query, lets it suggest small JSON edits, and keeps the fastest variants through an evolutionary search. Over generated TPC-H and TPC-DS workloads, the approach finds median speedups around 1.1x and occasional 4.78x wins against DataFusion's built-in optimizer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speedups are measured on GPT-5-generated, runtime-filtered queries, not canonical TPC-H/TPC-DS; representativeness is unestablished, and no random-patch control isolates LLM semantic reasoning.","rationale":"The paper's internal verification is unusually careful: min-of-50 timings, output-equivalence checks, external bare-metal re-execution of top-30, and footprint analysis all support that the reported speedups are real on the tested queries. The conditional verdict is appropriate. My primary concern is external validity: the workload is generated by GPT-5 and filtered to a 0.15–15s window; it is not canonical TPC-H/TPC-DS. Since the same LLM family writes the queries and the patches, the suite may be enriched for semantic selectivity patterns the LLM can exploit. The absence of a random-patch control means semantic reasoning is not isolated from evolutionary search over valid plans. I would not reject the paper: the experiment is reproducible in principle and the conditions are testable. I also note Section 5's median prose (1.15–1.2×) conflicts with Table 1 (1.10–1.12×); this should be corrected but is not the load-bearing issue. Hence UNCHANGED from the reader's CONDITIONAL.","tokens_in":25291,"tokens_out":9010,"duration_ms":89692,"concrete_test":"Run the identical DBPlanBench pipeline (same DataFusion version, K=5, T=4, R=50 min-of-50 protocol, output-equivalence check) on the official TPC-H 22-query and TPC-DS 99-query suites at SF3, without GPT-5 query generation or runtime-based filtering. Compare median P50, P75, P90 speedups and fastQ,1 to Table 1; also re-measure the top-30 on the bare-metal machine. If medians drop below ~1.05×, the headline 'on TPC-H/TPC-DS' overstates practical gains; if they remain ~1.1×, the workload concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claims (abstract: 1.10–1.12× TPC-H, 1.05–1.07× TPC-DS, up to 4.78×) are computed on a workload that is not the standard TPC-H/TPC-DS query suite. Section 4 ('Query Suite Construction') states that GPT-5 generates 120 queries per dataset at complexity levels 5–8, and that only queries executing in 0.15–15 s under DataFusion are kept. Nothing shows this filtered, self-generated set has the same distribution of join graphs, predicate selectivity, and optimizer blind spots as the official 22/99 TPC queries. Because the generator and optimizer are the same model family, the workload may be enriched for semantic patterns (e.g., d_year=2001, date-dim joins) that the LLM can exploit but that may be rarer in real workloads. The absence of a random-patch/evolution-only baseline further means the speedups cannot be attributed to LLM semantic reasoning rather than to broad search over valid plan variants plus execution feedback. Section 5's prose median (1.15–1.2× TPC-H) also disagrees with Table 1 (P50 1.10–1.12×), a reporting inconsistency. These are addressable, so the result is conditional rather than rejected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DBPlanBench, a harness for Apache DataFusion that serializes physical query plans into a compact JSON representation, lets an LLM (GPT-5) propose localized JSON-patch edits, and uses evolutionary search (PST-Evol and BoL-Evol) with execution feedback to refine candidates. The evaluation is on GPT-5-generated SQL workloads over TPC-H and TPC-DS schemas, with correctness enforced by output-equivalence checks and timings aggregated as min-of-50 sandbox runs. The main claim is that LLM-proposed, execution-verified plan patches can outperform DataFusion's default optimizer, with median speedups of 1.10–1.12x on TPC-H and 1.05–1.07x on TPC-DS, up to 4.78x, and that optimizations found at scale factor 3 transfer to scale factor 10.","tokens_in":25630,"tokens_out":4841,"duration_ms":53556,"significance":"If the result holds, the paper makes a useful empirical contribution to test-time query-plan optimization: it provides a practical serialization and patching interface for DataFusion plans, a careful execution protocol, and evidence that an LLM-guided search can discover non-obvious join reorderings and build-side selections. The strongest parts are the measurement discipline—50 sandbox runs per plan, min-of-R timing, output-equivalence checks, an external bare-metal re-execution of the top-30 plans, and a join-footprint analysis—which make the reported speedups credible within the tested workload. The work is also reproducible in principle through the promised open-source release. However, the significance for real OLAP workloads is currently limited by the use of LLM-generated, runtime-filtered queries rather than canonical TPC-H/TPC-DS query suites, and by the absence of a non-semantic control baseline.","major_comments":[{"comment":"The central numerical claims are computed on 120 GPT-5-generated queries per dataset over TPC-H/TPC-DS schemas, filtered to queries running between 0.15s and 15s. This is not the official TPC-H/TPC-DS query suite, yet the abstract and Section 5 phrase the results as 'on TPC-H' and 'on TPC-DS'. The filter and generator may select for queries with particularly LLM-friendly semantic patterns (e.g., selective date predicates and dimension/fact joins) and for optimizer blind spots. Please evaluate at least the official 22 TPC-H and 99 TPC-DS queries, or alternatively provide a distributional comparison (join graphs, predicate selectivities, plan shapes) between the generated workload and the canonical suites, and hedge the claims accordingly.","section":"Section 4, Query Suite Construction"},{"comment":"There is no random-patch or mutation-only baseline. The Best-of baseline controls for the number of LLM samples but not for whether the LLM's semantic reasoning contributes anything beyond a broad search over valid plan variants. To support the Section 5 claim that 'LLMs use semantic knowledge to optimize plans', add a control that performs the same evolutionary search using randomized but valid plan mutations (e.g., random join swaps and side flips) with the same budget, output-equivalence checks, and timing protocol. If random mutations achieve comparable speedups, the semantic-cardinality explanation is unsupported; if they do not, the causal claim would be substantially strengthened.","section":"Section 3.4 and Table 1"},{"comment":"There is a direct inconsistency in the reported medians. Section 5 states that 'median speedups are around 1.15x–1.2x for TPC-H and around 1.1x for TPC-DS', but Table 1 reports P50 values of 1.10–1.12x for TPC-H and 1.05–1.07x for TPC-DS for the same final round. The abstract follows Table 1. This must be reconciled; if the prose refers to a different statistic, algorithm, or subset, that should be stated explicitly.","section":"Section 5 vs. Table 1"}],"minor_comments":[{"comment":"The wording 'on TPC-H and TPC-DS' should be replaced with 'on queries generated over TPC-H/TPC-DS schemas' or a similar qualifier, to avoid implying evaluation on the official benchmark query sets.","section":"Abstract and Section 4"},{"comment":"The caption shows BoL-Evol for TPC-H and PST-Evol for TPC-DS without explaining why different algorithms are used for the two datasets. Clarify the selection rule, or report both algorithms on both datasets.","section":"Figure 4"},{"comment":"The caption should state the number of queries that remain after the 0.15s–15s filtering step, since 120 per dataset was the pre-filter target. The table headers also use Q and T without defining them in the caption; define Q as the query set and T as the evolution horizon.","section":"Table 1"},{"comment":"The external verification only re-executes the top-30 queries per dataset. State explicitly that this validates the head of the distribution and does not verify the full set of speedups, and discuss whether the untested tail could be affected by selection bias.","section":"Section B.2"},{"comment":"Typo: 'derivated TPC-H and TPC-DS benchmarks' should be 'derived TPC-H and TPC-DS benchmarks'.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a solid empirical systems paper with an unusually careful timing protocol. The main gap is external validity: the workload is generated by the same model family that performs the optimization, and the absence of a random-mutation control means the 'semantic reasoning' claim is not yet isolated. If the authors can add the official TPC query evaluation or a convincing workload-representativeness argument, and a random-patch baseline, the paper would be considerably stronger. The internal median discrepancy should also be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read it the same way you did, with one adjustment: the measurement protocol is better than the stress-test gives it credit for. Min-of-50, output-equivalence checks, bare-metal re-run of top-30, join-footprint analysis—that's real work, and the external verification is the right kind of evidence. The speedups are not artifacts of noisy sandboxes.\n\nWhat's new: DBPlanBench itself. A token-efficient serialization of DataFusion physical plans, JSON-patch editing, and a search loop with actual execution as fitness. That's a concrete artifact others can build on, and it's distinct from SQL-level rewriting or hint steering in prior work. The evolutionary framing is standard, but the patch-based plan editor is the contribution.\n\nThe soft spots are exactly where the stress-test points. The workload is GPT-5-generated, filtered to 0.15–15s, and nothing shows it resembles real OLAP or even the canonical TPC query suites. The median 1.1x and the 4.78x outlier are conditional on that workload. There's also no random-patch baseline, so you can't tell whether the gains come from LLM semantic knowledge or just from broad search over valid plan variants with execution feedback. That matters because the paper's title claims 'with LLMs'—if random patches do just as well, the semantic story collapses.\n\nOne reporting inconsistency: Section 5 says median speedups are around 1.15–1.2x for TPC-H, but Table 1's P50 is 1.10–1.12x. That's a small error, but it should be fixed.\n\nThe transfer experiment is nice, though the deterministic transfer script is hand-built from one in-context example; the limitations paragraph in Appendix A is honest about that. The complexity-vs-speedup analysis shows TPC-DS speedups actually decrease with complexity, which is a useful counterpoint to any 'more complex = more gain' narrative.\n\nBottom line: this deserves a serious referee. The harness is real, the measurement is careful, but the headline claims generalize only if the workload gap is closed. I'd send it to review with a request for canonical query sets or at least a clear argument for why the generated set is representative, plus a random-patch control. The prose/table median mismatch is trivial to fix.","headline":"A solid empirical contribution with a real workload-representativeness gap; the speedups are measured carefully but on queries the LLM itself generated, so the headline numbers should be read as upper bounds.","tokens_in":26131,"tokens_out":1761,"would_cite":true,"duration_ms":20159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-proposed, execution-verified plan patches beat DataFusion's optimizer, with median 1.1× speedups and up to 4.78×.","keywords":["query optimization","large language models","physical query plans","evolutionary search","DataFusion","join reordering","cardinality estimation","test-time optimization"],"falsifier":"Run the identical pipeline on the official TPC-H and TPC-DS query suites: if median speedup over DataFusion is at or below 1.0, the claimed out-performance is an artifact of the generated workload rather than a general property.","tokens_in":25185,"feed_emoji":"⚡","tokens_out":2805,"duration_ms":28455,"temperature":0.7,"pith_summary":"The paper tries to establish that LLMs, given a compact serialized physical plan and execution feedback, can discover faster query plans than an industrial-grade rule-based optimizer. It shows that evolutionary sampling over LLM-proposed JSON patches yields median speedups around 1.1× on TPC-H-derived and TPC-DS-derived workloads, occasional large speedups, and plans that transfer from small to larger data scales. If true, this would make test-time LLM optimization a practical complement to traditional cost-based optimizers, especially for repeated OLAP queries.","feed_headline":"LLM patches beat database optimizer: up to 4.78× speedup","feed_subtitle":"Execution-verified evolutionary edits find faster join orders on TPC-H and TPC-DS workloads.","key_machinery":"DBPlanBench, a harness that serializes DataFusion physical plans into a compact, token-efficient JSON representation, applies LLM-generated RFC-6902 JSON patches to rewire the plan, and executes the patched plans to verify correctness and measure runtime. Two evolutionary sampling strategies (Best-of-Last and Parallel Single-Thread) refine candidate patches across iterations, using execution-based fitness to select the fastest valid plan.","core_discovery":"The paper claims that a physical execution plan produced by DataFusion's optimizer can be improved after the fact: serialize the plan, have an LLM propose localized JSON-patch edits guided by semantic cardinality estimates, and evolve the best candidates using measured execution time as the fitness function. Across 240 synthetic TPC-H/TPC-DS queries, median speedups reach 1.10–1.12× (TPC-H) and 1.05–1.07× (TPC-DS), with up to 4.78× on one TPC-DS query. The mechanism is join reordering and build-side swapping that reduce intermediate cardinalities.","pith_inferences":["The speedup estimates may not extend to canonical TPC-H/TPC-DS queries or real production workloads, since the eval suite is LLM-generated and filtered by runtime.","A random-patch control would isolate whether the gains come from semantic reasoning or from blind search through valid plan mutations; the paper does not include one.","The same harness could be applied to other engines or to other plan-level decisions (e.g., join algorithms, parallelism) beyond the two edit types studied.","Because the approach is test-time and query-specific, it could be combined with learned cardinality models or adaptive query processing without modifying the core optimizer."],"forward_implications":["Even modest per-query speedups compound for repeated OLAP workloads, making test-time optimization economically worthwhile.","Optimizations found at a small scale factor transfer to larger scale factors, supporting a low-cost discovery-then-lift workflow.","Semantic cardinality estimation by LLMs can catch systematic errors that statistical heuristics make, e.g., when correlated predicates such as year = 2001 are highly selective.","The approach requires only localized structural edits, reducing token cost and preserving plan correctness through invariants and output-equivalence checking."],"fun_headline_variants":["LLM-tuned query plans: up to 4.78x faster","Evolve LLM patches for database speed: 4.78x","Test-time LLM edits boost query speed 4.78x","Database optimizer upgrade: LLM patch, 4.78x gain"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The generated benchmark queries contain the kind of selective predicates and dimension/fact join patterns where semantic reasoning helps; if actual workloads are less structured, the median gains could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLM-tuned query plans: up to 4.78x faster","Evolve LLM patches for database speed: 4.78x","Test-time LLM edits boost query speed 4.78x","Database optimizer upgrade: LLM patch, 4.78x gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1126,"prompt_tokens":773,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":517,"tokens_out":353,"duration_ms":4019,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:07:14.713797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline on the official TPC-H and TPC-DS query suites: if median speedup over DataFusion is at or below 1.0, the claimed out-performance is an artifact of the generated workload rather than a general property.","supporting_citations":[],"review_version":1}