{"id":"da86cc74-7415-45df-b0c9-32ba2fed83bf","arxiv_id":"2501.16544","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PLANSIEVE uses a transformer and L1-error to classify query plans as suboptimal during optimization, using surrogate cardinalities refined with observed execution results.","lead":"PLANSIEVE is a machine learning system that tries to detect when a database's query plan will be slow because its row-count guesses are wrong, before the query runs. It ranks subplans by estimated versus true counts and refines guesses over time; tests report up to 88.7% suboptimal-plan detection accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online evaluation appears to include training queries, so the reported real-time accuracy (up to 88.7%) may reflect memorization rather than detection on unseen queries.","rationale":"The reader's verdict identified online evaluation methodology as a conditional concern, noting that online experiments run over the full workload including training queries. My stress-test confirms this is the single most load-bearing issue: the paper's online confusion matrices sum to the full dataset sizes, not to the held-out test sizes, and no statement restricts the online phase to unseen queries. The offline held-out comparison (Section 6.3) is more credible and shows strong suboptimal-plan detection when true cardinalities are available, which is why I do not recommend outright rejection. However, the real-time claim depends on the online phase, and the reported accuracy numbers are compromised if training queries are included. A test-only online rerun, ideally with a temporal cache simulation, would settle the question. I partially agree with the reader: the reader's stated weakest assumption was surrogate-cardinality transfer, but the reader's rationale also flagged the full-workload online protocol. My concern is that protocol issue rather than surrogate transfer per se, although the two interact because poor surrogates amplify any leakage effect.","tokens_in":28094,"tokens_out":3514,"duration_ms":35620,"concrete_test":"Rerun the Section 6.4 online protocol using only the held-out test queries (JOB-LIGHT-SCALE: 408 queries; STATS-CEB-SCALE: 715 queries), with the same five surrogate/true cardinality mixtures. Report suboptimal-plan accuracy separately for the test-only set. If DeepDB-only suboptimal accuracy drops substantially below the reported 82.3-88.7% range, the real-time claim is unsupported. As a secondary check, simulate a temporal cache: order the test queries by time, warm the cache on earlier queries, and evaluate only on later, unseen queries, to test the incremental-refinement claim without train/test overlap.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The offline protocol (Section 5.4.1 and Section 6.3) splits each workload into 70% training and 30% testing. The online protocol in Section 6.4, however, reports confusion matrices whose row/column totals equal the full workload size: 1,357 for JOB-LIGHT-SCALE (Figure 10) and 2,380 for STATS-CEB-SCALE (Figure 11). The paper never states that the online evaluation is restricted to the held-out 30% of queries. Because the classification model was trained on true cardinalities for about 70% of each workload, evaluating online on the full workload means those training queries are scored again with surrogate or mixed cardinality inputs. Accuracy on previously seen queries does not establish that PLANSIEVE detects suboptimal plans for queries the model has not encountered. This is load-bearing because the paper's central claim is real-time detection during query optimization in deployment, not classification of training instances. The 'up to 88.7% suboptimal-plan accuracy' headline comes from this full-workload online evaluation. If the online numbers were computed only on the unseen 30%, the real-time claim could be materially weaker, especially with poor surrogates; Table 3 already shows suboptimal accuracy falling to 57.7-70% for Rand-Est and Reversed-TC, and those numbers may themselves be inflated by training-query leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PLANSIEVE, a framework for detecting suboptimal query plans during query optimization rather than after execution. The core idea is to learn, with a transformer-based classifier, the relationship between (i) the discrepancy in the relative order of subplans sorted by estimated versus true cardinalities (encoded as position vectors plus the aggregate L1-error) and (ii) whether the optimizer's plan is suboptimal. A cardinality collector supplies surrogates from a third-party estimator and progressively replaces them with true cardinalities observed from execution. The paper evaluates on augmented JOB-LIGHT-SCALE and STATS-CEB-SCALE workloads, reporting high offline classification accuracy on held-out 30% and online accuracy up to 88.7% for suboptimal-plan detection across five simulated scenarios of surrogate/true cardinality mixing.","tokens_in":28398,"tokens_out":4003,"duration_ms":40587,"significance":"If the real-time detection claim holds, the work addresses a well-known pain point: existing suboptimality metrics such as L1-error and P-error require true cardinalities and are only usable post-execution. The position-vector framing is a natural and interesting extension of the authors' earlier L1-error work, and the cross-validated offline evaluation is a genuine strength: it uses a strict train/test split, compares against an L1-decision-tree baseline, reports confusion matrices with absolute counts, and includes an ablation of estimator quality (DeepDB, Rand-Est, Reversed-TC) and a generalization probe (JOB-LIGHT-RANGES). The public artifacts and explicit statement of the suboptimality threshold (c=1) also improve reproducibility. However, the central real-time claim rests on an online evaluation protocol that, as described, appears to include training queries, which materially weakens the support for generalization to unseen queries.","major_comments":[{"comment":"The online evaluation appears to use the full workloads, not the held-out 30% used in the offline phase. Section 6.3 states that each dataset is partitioned into 70% training and 30% testing, but Section 6.4 does not state that the online phase is restricted to the held-out portion. The confusion matrices in Figure 10 sum to 1,357 and those in Figure 11 sum to 2,380, which equal the total sizes of JOB-LIGHT-SCALE and STATS-CEB-SCALE. Since the model was trained on roughly 70% of these queries with true cardinalities, the online results, including the headline \"up to 88.7% suboptimal-plan accuracy\" in Table 3, are likely inflated by scoring queries seen during training. Please rerun the online protocol on the held-out 30% only (or explicitly report separate held-out online numbers) and state clearly which split each figure and table uses.","section":"Section 6.4, Figures 10 and 11"},{"comment":"The online experiments do not directly validate the incremental-refinement mechanism. The five scenarios (100%, 75%, 50%, 25%, 0% surrogates) apply a globally fixed mixture of surrogate and true cardinalities to all queries at once, rather than simulating a chronological stream in which the cache fills as queries execute and later queries benefit from earlier executions, as described in the workflow of Section 3.2. The observed accuracy trend is consistent with refinement, but it does not demonstrate that the cache-based incremental process achieves this improvement in a real workload order. A time-ordered or cache-filling evaluation, or an explicit statement that these are ablation scenarios rather than a temporal simulation, is needed to support the \"incremental refinements\" part of the central claim.","section":"Section 6.4 and Section 3.2"}],"minor_comments":[{"comment":"The JOB-LIGHT-SCALE row lists 1,357 training queries, but Section 6.3.1 and the training confusion matrix show 949 training queries (731+177+10+31). The table should list 949 (70% of the total) and 408 for testing to match the text.","section":"Table 2"},{"comment":"The framework name is spelled inconsistently as both \"PLANSIEVE\" and \"PLANSEIVE\" (e.g., Sections 3.3, 3.4, and 6.3.3). Please standardize to one spelling.","section":"Throughout"},{"comment":"The example query is referred to as \"qry_68_9\" in Section 2.2 but as \"t_68_9\" in the caption of Figure 3; these should be aligned.","section":"Figure 3 and Section 2.2"},{"comment":"The formula for aggregating cardinalities in partial-match patterns is garbled (\"NX 1 Yp = Yp1 + ...\"). The notation should be cleaned up and the averaging formula stated unambiguously.","section":"Section 4.2"},{"comment":"The text in Section 6.5 says Reversed-TC improves \"from 70.8% to 85.5%\" for JOB-LIGHT-SCALE, but Table 3 reports the lower bound as 70%. Please make the numbers consistent.","section":"Section 6.5, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the offline evaluation is well done, but the online evaluation protocol as written includes training queries, which undermines the headline real-time generalization claim. The fix is straightforward in principle—rerun the online scenarios on the held-out 30%—but it is essential before publication. I also encourage the authors to clarify whether the online scenarios are meant as static ablations or as evidence for temporal cache refinement; if the latter, an explicitly chronological experiment is needed. The paper is otherwise within the scope of the journal and builds sensibly on prior L1-error work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of PLANSIEVE (arXiv:2501.16544). The new piece is real: they take the L1-error idea from their earlier paper and make it predictive at optimization time by feeding estimated-vs-true subplan orderings into a GPT-2-style transformer plus MLP, with surrogate cardinalities from a third-party estimator and a cache to refine them. The offline evaluation is the paper's best part. On a clean 70/30 split, their classifier beats the L1-decision-tree baseline clearly—JOB-LIGHT-SCALE test set: 84 true negatives vs 28; STATS-CEB-SCALE: 241 vs 204. They also report training/prediction overhead and test robustness with bad estimators (Rand-Est, Reversed-TC) and a different workload (JOB-LIGHT-RANGES), where accuracy drops—that honesty is to their credit.\n\nThe problem is the online phase. The confusion matrices in Figures 10 and 11 have row/column totals equal to the full workload sizes (1,357 and 2,380), not the 30% held-out sets. The paper never says the online evaluation was restricted to unseen queries. Since the model was trained on true cardinalities for 70% of each workload, the 'surrogates only' and mixed scenarios are scoring training queries again with different inputs. That can inflate accuracy substantially through memorization, and the low suboptimal accuracy with poor surrogates (57.7–70%) is consistent with the model recognizing queries it has seen. So the headline 'up to 88.7%' is not evidence for real-time generalization.\n\nSecond, the incremental-refinement claim is not actually tested as a process. The online experiments are static mixes (75/25, 50/50, etc.) of surrogate and true cardinalities, not a temporal simulation where a cache accumulates over a query sequence. So we get no measurement of hit rates or learning curves. That is a weaker but real gap.\n\nThird, the paper says PLANSIEVE can prevent costly plans but never measures execution impact; it reports classification accuracy only. That is a limitation, not a fatal one, but it should be stated.\n\nThe circularity concern (label and features both derive from true/estimated cardinalities) does not bother me much: the offline held-out split gives an honest generalization test, and the model is learning a mapping from positional discrepancy to cost-based suboptimality, not deriving it.\n\nBottom line: this deserves a serious referee, but with a clear revision demand. Send it to review; require an online protocol restricted to the held-out 30% (or both full and held-out), and ideally a temporal cache simulation. If the held-out online numbers hold up, this is a useful systems contribution. If they collapse, the paper reduces to an offline detector, which is still publishable but more modest.","headline":"A credible offline classifier for suboptimal-plan detection, but the online evaluation appears to score training queries again—so the real-time claims need a held-out rerun before I'd trust them.","tokens_in":28890,"tokens_out":3727,"would_cite":true,"duration_ms":35924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PLANSIEVE claims it can identify suboptimal query plans during optimization, before execution, by comparing the estimated and true-cardinality orderings of subplans and refining estimates online.","keywords":["query optimization","cardinality estimation","suboptimal query plan","L1-error","transformer","join order","surrogate cardinality","incremental refinement"],"falsifier":"A direct test would take queries whose optimizer-chosen estimated order exactly matches the true-cardinality order for every join size, so L1-error is zero, yet whose plan is still suboptimal by the paper's P-error threshold; if PLANSIEVE labels them optimal, the claim that positional order plus L1-error captures suboptimality is refuted. The paper's own Reversed-TC experiment gives the complementary check: when order is inverted, accuracy should degrade and then recover as true cardinalities are cached, and failure to recover would falsify the incremental-refinement claim.","tokens_in":27930,"feed_emoji":"🗄️","tokens_out":6094,"duration_ms":58697,"temperature":0.7,"pith_summary":"PLANSIEVE is a framework that tries to identify suboptimal query execution plans during the optimization phase, before the database runs the query. It is built on the idea that cardinality estimation errors matter through the relative order they impose on subplans: when the optimizer's estimated order diverges from the true-cardinality order, the chosen plan is more likely to be suboptimal. The system trains a transformer-based classifier offline on pairs of position vectors plus an L1-error measure, then applies it online using surrogate cardinalities from a third-party estimator, refining those surrogates with true cardinalities cached from query execution. On augmented JOB-LIGHT and STATS-CEB workloads, the paper reports up to 88.7% accuracy in predicting suboptimal plans, with accuracy improving as more true cardinalities are accumulated. If it works, database optimizers could act on the flag and re-optimize or correct estimates before a slow plan executes.","feed_headline":"Catches suboptimal query plans up to 88.7% of the time","feed_subtitle":"PLANSIEVE learns from subplan ordering and refines cardinality estimates as queries execute, before slow plans run.","key_machinery":"The load-bearing object is the pair of position vectors ρ and ρ̂ for each join size k, encoding the relative order of subplans under true and estimated cardinalities, together with the query-level L1-error derived from those vectors. Subplans are featurized as one-hot join encodings, tokenized with a delimiter separating the true from the estimated order, and processed by a GPT-2-style transformer whose output is concatenated with the L1-error and passed to an MLP for binary classification. The Cardinality Collector supplies true cardinalities from a cache refined over time, and a third-party estimator provides surrogates on cache misses.","core_discovery":"The paper's central claim is that suboptimality of a query plan can be predicted in real time from the positional discrepancy between subplan orderings by estimated versus true cardinalities, without waiting for execution. For each join size k, subplans are sorted by true cardinalities Y to form position vector ρ and by estimated cardinalities to form ρ̂; the L1-error aggregates absolute positional differences, weighted toward smaller joins. A transformer reads the concatenated sequence of true-order tokens, a delimiter, and estimated-order tokens, and together with the L1-error an MLP classifies the plan as optimal or suboptimal. The paper claims the model transfers from offline training with true cardinalities to online operation using surrogate estimates, and that incremental caching of true cardinalities improves accuracy over time. Experiments report classification accuracy up to 88.7% for suboptimal-plan prediction.","pith_inferences":["The core idea suggests an optimizer could use PLANSIEVE's prediction not just as a flag but to rank candidate plans by their estimated-order divergence, making robust plan selection a classification-guided search rather than a one-shot binary warning.","Because the model learns from relative order rather than absolute cardinality magnitudes, it may transfer across databases where absolute estimates differ but order structures are similar; this is testable but not claimed in the paper.","The benchmark augmentation framework, described as a way to obtain balanced training data, could also serve as a general stress-test for other learned query components by generating selective variations of base queries.","A natural extension is to measure end-to-end wall-clock savings when PLANSIEVE's flags trigger re-optimization or estimate correction, rather than stopping at classification accuracy."],"forward_implications":["If a plan is flagged suboptimal during optimization, the optimizer can explore alternative plans, prefer physical operators less sensitive to cardinality errors, or trigger targeted estimate correction before execution.","Accuracy improves as the system caches true cardinalities from executed queries: on JOB-LIGHT-SCALE overall accuracy moves from 93.15% with surrogates alone to 95.58% with true cardinalities only, and suboptimal-plan accuracy rises correspondingly.","Starting with a better surrogate estimator yields better initial suboptimal-plan accuracy, but the paper reports that even random or reversed-order surrogates recover to the same accuracy ceiling once true cardinalities accumulate.","On a distributionally different unseen workload, JOB-LIGHT-RANGES, suboptimal-plan prediction accuracy drops to 48% to 52.5%, indicating that significant workload shifts may require retraining or fine-tuning.","PLANSIEVE's classifier consistently identifies suboptimal plans better than the L1-error decision-tree baseline, especially by reducing the false positives that would let bad plans run."],"supporting_citations":[{"why":"Supplies the L1-error metric and the definition of plan sub-optimality that PLANSIEVE builds on.","marker":"[22]"},{"why":"DeepDB is the third-party estimator used to generate surrogate cardinalities in the online phase.","marker":"[20]"},{"why":"Provides neurocard, the estimator used in the generalization experiments and the JOB-LIGHT-RANGES workload.","marker":"[48]"},{"why":"Defines the STATS-CEB benchmark and supplies one of the two base workloads for evaluation.","marker":"[17]"},{"why":"Defines JOB-LIGHT, the other base workload, and motivates the need for augmented query sets.","marker":"[23]"},{"why":"Motivates targeted estimate correction and provides the prior approach that flags subplans by error probability.","marker":"[47]"},{"why":"Supplies the cardinality aggregation strategy that PLANSIEVE adapts for cache pattern refinement.","marker":"[19]"},{"why":"Supplies the transformer architecture that PLANSIEVE adapts for subplan sequence classification.","marker":"[44]"},{"why":"PostgreSQL is the DBMS whose optimizer and execution logs PLANSIEVE uses for subplan extraction and true-cardinality collection.","marker":"[52]"}],"fun_headline_variants":["Real-time suboptimal query plan detection hits 88.7% accuracy","PLANSIEVE catches suboptimal plans before execution, up to 88.7%","Incremental refinements spot suboptimal query plans in real time","Predict suboptimal query plans in real time with up to 88.7% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that surrogate cardinalities, as progressively refined by cached true values, preserve enough of the true subplan ordering that a classifier trained on true-cardinality orderings still recognizes suboptimal plans; when surrogates are random or reversed, the paper itself reports suboptimal-plan accuracy dropping to 57.7% to 70%.","fun_headline_variants_meta":{"raw":{"variants":["Real-time suboptimal query plan detection hits 88.7% accuracy","PLANSIEVE catches suboptimal plans before execution, up to 88.7%","Incremental refinements spot suboptimal query plans in real time","Predict suboptimal query plans in real time with up to 88.7% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1759,"prompt_tokens":904,"completion_tokens":855,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":520,"tokens_out":855,"duration_ms":7932,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:28:24.165566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would take queries whose optimizer-chosen estimated order exactly matches the true-cardinality order for every join size, so L1-error is zero, yet whose plan is still suboptimal by the paper's P-error threshold; if PLANSIEVE labels them optimal, the claim that positional order plus L1-error captures suboptimality is refuted. The paper's own Reversed-TC experiment gives the complementary check: when order is inverted, accuracy should degrade and then recover as true cardinalities are cached, and failure to recover would falsify the incremental-refinement claim.","supporting_citations":[{"cited_title":"Izenov, A","cited_arxiv_id":null,"evidence_quote":"Supplies the L1-error metric and the definition of plan sub-optimality that PLANSIEVE builds on."},{"cited_title":"Hilprecht, A","cited_arxiv_id":null,"evidence_quote":"DeepDB is the third-party estimator used to generate surrogate cardinalities in the online phase."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides neurocard, the estimator used in the generalization experiments and the JOB-LIGHT-RANGES workload."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the STATS-CEB benchmark and supplies one of the two base workloads for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines JOB-LIGHT, the other base workload, and motivates the need for augmented query sets."},{"cited_title":"PARQO: Penalty-Aware Robust Plan Selection in Query Optimization","cited_arxiv_id":"2406.01526","evidence_quote":"Motivates targeted estimate correction and provides the prior approach that flags subplans by error probability."},{"cited_title":"Hertzschuch, C","cited_arxiv_id":null,"evidence_quote":"Supplies the cardinality aggregation strategy that PLANSIEVE adapts for cache pattern refinement."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer architecture that PLANSIEVE adapts for subplan sequence classification."},{"cited_title":"www.postgresql.org, 2024","cited_arxiv_id":null,"evidence_quote":"PostgreSQL is the DBMS whose optimizer and execution logs PLANSIEVE uses for subplan extraction and true-cardinality collection."}],"review_version":1}