{"id":"a6e46457-5d95-4064-975d-403efe896f26","arxiv_id":"2412.00749","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CONCERTO predicts query latency on parallel OLAP databases by estimating per-operator costs, calibrating resource contention with graph attention networks, and aggregating calibrated costs with a tree convolutional network.","lead":"CONCERTO is a machine-learning method that predicts how long complex analytical database queries will take on modern systems that run queries in parallel with changing pipelines. It reports lower prediction error than four existing methods on standard TPC-H and TPC-DS benchmarks running on ClickHouse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe-phase completeness is the load-bearing assumption: dynamic modifications after the probe are invisible to CONCERTO's data-flow tree, and the paper offers no measurement of how often or how late such modifications occur in its TPC-H/DS evaluation.","rationale":"I read the paper's central claim as: CONCERTO, by combining serial-executor operator cost models, a probe-based data-flow tree, GAT-based resource-competition calibration, and TCN aggregation, predicts query latency more accurately than existing learned QPP methods for high-performance OLAP DBMSs such as ClickHouse. The architecture is coherent and the ablation study gives some internal support: removing the resource competition attention raises mean Q-Error from 1.46 to 2.09 on TPC-H, and removing OCPs raises it to 1.70. The reported comparisons to GCN, TCN, QPPNet, QueryFormer, and RAAL are consistent across two benchmarks. However, the evaluation is built on a single, unverified bridge: the probe phase must reveal the dynamic pipeline structure that is actually executed for the rest of the query. The paper's own limitation statement admits this is not guaranteed for load-driven modifications and asserts without evidence that cardinality-error-driven modifications happen early. The data-flow tree construction (Algorithm 1) merges chunk paths observed only during the probe; any later modification is absent from the model input. This is not an internal inconsistency, but it is the least secure condition for the central claim. The reader identified the same assumption as weakest. I do not see a stronger objection: the baselines are adapted to the same data format, the Q-Error metric is standard, and the method is deterministic in description. I would keep the conditional verdict, adding the recommended modification-timing instrumentation as a condition for acceptance. This is a verification gap rather than demonstrated falsity, so no change in verdict direction is warranted.","tokens_in":18950,"tokens_out":4879,"duration_ms":47772,"concrete_test":"Instrument ClickHouse's pipeline builder and execution engine to log, for every test query, the set of dynamic pipeline modifications (algorithm changes, inserted operators) together with the fraction of total chunks processed before each modification. Classify each query by whether any modification occurs after the probe phase (define the probe as the first few chunks or the ~155ms probe duration reported in Section V.B). Then compute CONCERTO's mean Q-Error separately for the 'probe-complete' and 'late-modification' groups. If the late-modification group is non-empty and its Q-Error is substantially higher, or if late modifications occur in a nontrivial fraction of TPC-H/DS queries, the probe assumption fails and the data-flow-tree input is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim depends on the data-flow tree constructed by Algorithm 1 being a faithful representation of the query's actual execution. That tree is built exclusively from chunk execution paths observed during probe execution mode (Section V.B), which 'only launches a few chunks.' The paper explicitly concedes this mode 'cannot capture all dynamic modifications... particularly those stemming from system load variations,' and justifies the limitation by asserting that cardinality-error-driven modifications 'typically occur during the early stages of query execution.' This assertion is not measured. If a join algorithm switch or sort insertion happens after the probe, chunks in the remainder of the query traverse a different pipeline, and Algorithm 1's merged tree contains only the pre-modification paths; the calibrated cost vectors and TCN aggregation then describe a query that was not actually executed. The reported accuracy gains are modest (mean Q-Error 1.46 vs 1.60 on TPC-H; 1.84 vs 2.00 on TPC-DS), so even a small fraction of late-modification queries could erase the advantage. The paper's own robustness experiment (Table IV) shows CONCERTO deteriorates sharply when cardinality perturbations are larger (mean Q-Error 2.59 at sigma=1.5), which is precisely the regime in which late modifications would be triggered. Without a measurement of modification timing in the evaluated workloads, the probe-completeness assumption is unverified, and the headline claim is conditional on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CONCERTO proposes a multi-stage learned query performance prediction (QPP) method aimed at modern OLAP DBMSs with vectorized operators, parallel DAG pipelines, and dynamic pipeline modifications. The system decouples per-operator resource-cost prediction (Stage 1, via lightweight MLPs called OCPs), calibration of parallel-execution resource contention (Stage 2, via a data-flow tree, hand-constructed resource-competition matrices, an attention mechanism, and a GAT), and bottom-up latency aggregation (Stage 3, via a TCN). A Runtime Tracker collects chunk-level execution paths in a probe execution mode and supports serial-executor training data. Experiments on TPC-H and TPC-DS (scale factor 1, implemented on ClickHouse) report mean Q-Errors of 1.46 and 1.84, respectively, improving over adapted baselines, with ablations showing the resource-attention and OCP components contribute. The paper candidly states that the probe phase cannot capture dynamic modifications arising from system-load variations, and the robustness experiment shows large accuracy degradation under strong cardinality perturbations.","tokens_in":19257,"tokens_out":5830,"duration_ms":59287,"significance":"If the claims are substantiated, CONCERTO addresses a genuine gap: most learned QPP methods target static tree-shaped plans, while ClickHouse-style engines use dynamic parallel DAG pipelines. The strengths are the clean decomposition of operator-cost prediction, contention calibration, and latency aggregation; the design of a Runtime Tracker that exposes chunk execution paths; and ablations indicating that the attention-based calibration and OCPs both matter. The reported model size and inference latency are attractive. However, the central accuracy claim is conditional on two yet-unverified premises: that the probe execution phase captures the dynamic modifications relevant to the evaluated workloads, and that the baseline comparisons are fair when all baselines are adapted to CONCERTO's own data-collection format. The robustness result in Table IV also materially qualifies the headline claim.","major_comments":[{"comment":"The probe execution mode is the sole source of the data-flow tree used at inference, yet the manuscript explicitly concedes that it 'cannot capture all dynamic modifications... particularly those stemming from system load variations' and justifies this by asserting that cardinality-error-driven modifications 'typically occur during the early stages of query execution.' No measurement of modification timing or frequency is reported for TPC-H or TPC-DS. If a join-algorithm switch or sort insertion occurs after the probe, Algorithm 1 constructs a tree containing only the pre-modification paths, so the calibrated cost vectors and TCN aggregation describe a query that was not actually executed. This concern is reinforced by Table IV, where CONCERTO's mean Q-Error degrades sharply from 1.46 to 2.59 as cardinality perturbation increases, exactly the regime in which late modifications would be triggered. Please instrument ClickHouse to log all dynamic modifications and report (i) how often modifications occur in the test workloads, (ii) their timing relative to the probe window, and (iii) sensitivity to the number of probe chunks, or otherwise demonstrate that the evaluated workloads contain no late modifications that affect the predictions.","section":"Section V.B and Algorithm 1"},{"comment":"All baselines are evaluated on data collected by CONCERTO's Runtime Tracker: the paper states 'We used their open-source code and modified it to adapt the data collected by CONCERTO's Runtime Tracker from ClickHouse.' Methods such as QPPNet and QueryFormer are designed for planner plan trees with cardinality estimates, not for chunk-level runtime execution paths; feeding them CONCERTO's feature set can disadvantage them independently of their modeling quality. In addition, MB2 [9], the closest prior two-level decomposed QPP method, is discussed in Related Work but is not included in the comparison. Please add MB2 and either evaluate baselines with their native plan representations or carefully justify and validate the adaptation, for example by ablating input-feature choices on at least one baseline to show that the reported gaps are not artifacts of feature adaptation.","section":"Section VI.A and Table II"},{"comment":"The robustness experiment contradicts the unqualified abstract claim that 'CONCERTO achieves higher prediction accuracy than existing methods.' At sigma=1.5, CONCERTO's mean Q-Error is 2.59, whereas QueryFormer achieves 1.61 and RAAL 2.30; the paper itself concludes that 'its robustness remains to be improved under larger cardinality estimation errors.' The abstract and conclusion should be restricted to the unperturbed setting, or the robustness boundary should be characterized and explained. Moreover, the improvements over TCN and QueryFormer on TPC-H are modest (mean Q-Error 1.46 versus 1.60), and no confidence intervals, seed counts, or significance tests are reported for Tables II and IV, so it is difficult to assess whether the differences are stable.","section":"Table IV and Abstract"},{"comment":"The workload-split description is internally inconsistent: the paper says 4 held-out TPC-H templates are '23.5%' of the workload, but 4/22 is 18.2%, and the TPC-DS list contains 10 template indices while the text says 9 templates (15.52%). This prevents a reader from reproducing the exact train/test split and should be corrected, with the full list of training and test template indices provided.","section":"Section VI.A (workload split)"}],"minor_comments":[{"comment":"In the pseudocode, line 19 creates a node for 'child operator' but it should create a node for the current operator; line 2 contains the typo 'spilt' for 'split'.","section":"Algorithm 1"},{"comment":"The abbreviation for Operator Cost Predictor is introduced as OCP, but the text later uses 'OPCs' in several places; the abbreviation should be used consistently.","section":"Section IV.A"},{"comment":"The construction of the meta resource-competition matrices Mc, Mm, and Mio is described only qualitatively ('based on both code implementation and the number of their CPU instructions, memory cost, and I/O times'); the exact procedure, normalization, and threshold choices should be specified so the experiment can be reproduced.","section":"Section IV.B"},{"comment":"The performance comparison in Table III reports only model inference time, not end-to-end prediction overhead including the probe execution phase; the authors state the TPC-DS probe overhead is about 155ms, but the corresponding TPC-H figure and its share of total query latency are not reported.","section":"Section VI.C"},{"comment":"No code or data availability statement is provided; given the substantial system-engineering component (Runtime Tracker, serial executor, ClickHouse modifications), a release or detailed artifact description would significantly improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the decoupled architecture is well motivated, but the acceptance decision hinges on whether the probe-phase completeness assumption can be verified and whether the baseline comparison can be made convincingly fair. I do not see circularity in the training setup, and the ablations are a genuine strength. The omission of MB2 and the unmeasured late-modification risk are the main reasons I cannot recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: CONCERTO is the first learned QPP I've seen aimed squarely at ClickHouse-style dynamic parallel DAG pipelines, and the data-flow tree construction plus resource-competition attention is a genuine, coherent design. The paper reports consistent accuracy improvements over four baselines on TPC-H and TPC-DS, with mean Q-Error 1.46 vs 1.60 and 1.84 vs 2.00 respectively, and ablations show both the OCP module and the resource attention mechanism earn their keep. The Runtime Tracker is real engineering, and the probe overhead (~155ms) is plausibly acceptable for long OLAP queries.\n\nThe soft spots are real, though not fatal. The most load-bearing is the probe-completeness assumption: the data-flow tree is built only from chunk paths observed during a brief probe, and the paper explicitly concedes (Section V.B) that dynamic modifications after the probe are missed, especially those from system load variations. The justification that cardinality-error-driven modifications \"typically occur during the early stages\" is asserted, not measured. Given that the accuracy gains are modest, even a small fraction of late-modification queries could erase the advantage. The robustness experiment in Table IV actually supports this worry: at sigma=1.5, CONCERTO's mean Q-Error jumps to 2.59, worse than QueryFormer's 1.61 and RAAL's 2.30, which is precisely the regime where late modifications would be triggered.\n\nAlso worth flagging: MB2, the closest prior work (operator-level cost models plus an interference model), is cited but not included as a baseline. The other baselines were adapted to CONCERTO's own runtime data collection format, which may not be apples-to-apples. No error bars or statistical significance tests are reported, and no code or data are released. These are addressable issues, not design flaws.\n\nWho this is for: anyone working on query performance prediction, learned cost models, or adaptive execution in columnar/parallel OLAP engines. It deserves a serious referee. I'd send it to review, but the majors would be: add MB2, measure when and how often dynamic modifications actually occur in the TPC-H/DS workloads, report error bars across multiple seeds, and release the tracker code.\n\nMy bottom line: engage with it. The architecture is thoughtful, the limitations are honestly stated, but the central claim is currently conditional on an unverified assumption that a probe captures the modifications that matter.","headline":"A solid, well-ablated multi-stage QPP for dynamic parallel DAG pipelines, but the headline accuracy gain is modest and the central claim depends on an unverified probe-completeness assumption.","tokens_in":19781,"tokens_out":1571,"would_cite":true,"duration_ms":16897,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CONCERTO predicts OLAP query latency more accurately than existing learned methods by decoupling operator cost, resource-competition calibration, and latency aggregation.","keywords":["query performance prediction","learned cost estimation","AI4DB","graph attention network","temporal convolutional network","parallel query execution","dynamic pipeline","OLAP"],"falsifier":"A direct falsifier is a workload in which pipeline modifications are triggered only after the probe window closes, for example by injecting a cardinality underestimate that the optimizer discovers mid-query. If CONCERTO's mean Q-Error then rises to or above the best baseline's, the probe-driven data-flow tree is not sufficient for general dynamic execution and the accuracy claim only holds for early modifications.","tokens_in":18773,"feed_emoji":"⏱️","tokens_out":11194,"duration_ms":79593,"temperature":0.7,"pith_summary":"The paper claims that query performance prediction on modern OLAP databases—where operators are vectorized, pipelines run in parallel, and the execution plan is dynamically rewritten at runtime—fails when done with tree-shaped, static-plan models, and that a multi-stage method aware of these mechanisms does better. To show this, CONCERTO first predicts each physical operator's standalone resource cost, then calibrates those costs for parallel resource contention using a graph attention network over a data-flow tree, and finally aggregates the calibrated costs into a total latency prediction with a temporal convolutional network. The design is backed by a Runtime Tracker that collects chunk-level execution paths, including dynamic pipeline modifications, through a probe execution. Experiments on TPC-H and TPC-DS report mean Q-Error of 1.46 and 1.84 respectively, better than the compared learned baselines.","feed_headline":"CONCERTO cuts OLAP query cost error to 1.46 mean Q-Error","feed_subtitle":"Splitting operator cost, parallel-contention calibration, and latency aggregation beats tree-based predictors on TPC-H/TPC-DS.","key_machinery":"The central object is the data-flow tree: a tree built by merging the execution paths of data chunks collected during probe execution, so that both pre- and post-modification pipeline states coexist as branches. Around this backbone, the Graph Constructor adds resource-competition edges among operators in the same CPU, memory, or I/O resource class, with an attention mechanism that reweights competition intensity from the tree structure. A Graph Attention Network (GAT), a neural architecture for graph-structured data, then recalibrates each operator's predicted cost vector, and a Temporal Convolutional Network (TCN), a convolution over tree structures, aggregates the recalibrated vectors bottom-up into a query latency prediction. The whole pipeline from graph construction to latency is differentiable, so the calibrator and aggregator are trained jointly on query-level latency.","core_discovery":"CONCERTO's central discovery is that the accuracy bottleneck for learned query cost estimation on high-performance OLAP systems is not operator cost itself but the interaction between concurrent operators in a parallel DAG pipeline, and that this interaction can be modeled separately. The authors demonstrate that decomposing prediction into three differentiable stages—an operator cost predictor, a graph-attention resource-competition calibrator, and a tree-convolution aggregator—yields substantially lower Q-Error than end-to-end tree or graph models such as QPPNet, QueryFormer, GCN, TCN, and RAAL. The data-flow tree, built from probe-phase chunk execution paths, is the key representation: it preserves both the original and dynamically modified pipelines by expanding the time dimension of execution into the spatial tree structure, which is what lets the model see modifications that a static EXPLAIN plan would miss.","pith_inferences":["If the probe assumption holds, the same data-flow-tree representation could serve other dynamic-engine tasks besides QPP, such as plan debugging or feeding an adaptive optimizer with the actual executed paths rather than the planned ones.","The robustness experiment shows CONCERTO's mean Q-Error rising from 1.46 to 2.59 under σ=1.5 cardinality noise while QueryFormer stays near 1.6, so CONCERTO's lead depends on early access to accurate cardinality; a system with worse early estimates would likely erode that advantage.","Because the attention weights that set competition intensity are learned from the tree structure, a natural test is whether those learned intensities transfer to different hardware configurations (fewer cores, smaller cache); if not, they would need per-hardware re-training.","The paper's stated future direction is a cross-DBMS plugin; a concrete extension would be to replace the ClickHouse-specific tracer with a generic profiler hook so that the same three-stage predictor works on SparkSQL and DuckDB without redesign."],"forward_implications":["On TPC-H and TPC-DS, CONCERTO reports the lowest mean, median, and tail Q-Error among the compared methods, with the ablation showing the resource attention mechanism contributing the largest accuracy gain.","Because prediction is decoupled from operator cost, the method is claimed to generalize across query templates and cope with workload drift, while keeping model size (0.094 MB) and inference time (4.2 ms) smaller than the baselines.","The probe-based data-flow tree is claimed to capture cardinality-error-driven dynamic modifications that occur early in execution, which is where the authors argue such pipeline changes concentrate.","The design is intended to be portable to other DAG-pipeline OLAP systems such as SparkSQL and DuckDB, since the Runtime Tracker needs only a serial executor or a DOP knob plus access to chunk execution paths.","The documented probe overhead (about 155 ms on TPC-DS) is small relative to OLAP query times, making the approach practical for production admission control and resource management."],"supporting_citations":[{"why":"Supplies the Graph Attention Network used to calibrate operator costs under resource competition.","marker":"[21]"},{"why":"Supplies the Temporal Convolutional Network used to aggregate calibrated cost vectors into query latency.","marker":"[30]"},{"why":"ClickHouse is the target high-performance OLAP DBMS and its probe phase motivates CONCERTO's probe execution mode.","marker":"[20]"},{"why":"QPPNet is a primary operator-level baseline whose accuracy CONCERTO must exceed.","marker":"[11]"},{"why":"RAAL is the closest resource-aware baseline, providing the comparison that isolates CONCERTO's intra-query resource modeling.","marker":"[17]"},{"why":"Prior work modeling concurrent queries as DAGs with resource-competition edges; CONCERTO adapts this to resource contention within a single parallel pipeline.","marker":"[12]"},{"why":"QueryFormer is a transformer-based tree-plan baseline used in the accuracy comparison.","marker":"[10]"}],"fun_headline_variants":["CONCERTO: Cost estimation that tracks parallel operator contention","Modeling concurrent operators cuts Q-Error for OLAP queries","Dynamic pipeline-aware cost model beats tree predictors","CONCERTO: Graph attention for query cost estimation","How CONCERTO sees dynamic pipeline changes in cost prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Dynamic pipeline changes that materially affect a query's latency all happen early enough for a short probe execution of the first few data chunks to reveal them, so the data-flow tree built from probe paths represents the full execution.","fun_headline_variants_meta":{"raw":{"variants":["CONCERTO: Cost estimation that tracks parallel operator contention","Modeling concurrent operators cuts Q-Error for OLAP queries","Dynamic pipeline-aware cost model beats tree predictors","CONCERTO: Graph attention for query cost estimation","How CONCERTO sees dynamic pipeline changes in cost prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1586,"prompt_tokens":915,"completion_tokens":671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":531,"tokens_out":671,"duration_ms":6550,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:02:51.995762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is a workload in which pipeline modifications are triggered only after the probe window closes, for example by injecting a cardinality underestimate that the optimizer discovers mid-query. If CONCERTO's mean Q-Error then rises to or above the best baseline's, the probe-driven data-flow tree is not sufficient for general dynamic execution and the accuracy claim only holds for early modifications.","supporting_citations":[{"cited_title":"& Jin, Z","cited_arxiv_id":null,"evidence_quote":"Supplies the Temporal Convolutional Network used to aggregate calibrated cost vectors into query latency."},{"cited_title":"& Milovidov, A","cited_arxiv_id":null,"evidence_quote":"ClickHouse is the target high-performance OLAP DBMS and its probe phase motivates CONCERTO's probe execution mode."},{"cited_title":"& Papaemmanouil, O","cited_arxiv_id":null,"evidence_quote":"QPPNet is a primary operator-level baseline whose accuracy CONCERTO must exceed."},{"cited_title":"& Peng, Z","cited_arxiv_id":null,"evidence_quote":"RAAL is the closest resource-aware baseline, providing the comparison that isolates CONCERTO's intra-query resource modeling."},{"cited_title":"& Feng, J","cited_arxiv_id":null,"evidence_quote":"Prior work modeling concurrent queries as DAGs with resource-competition edges; CONCERTO adapts this to resource contention within a single parallel pipeline."}],"review_version":1}