{"id":"77a2485f-58cd-4147-be3f-0d5a0f898af7","arxiv_id":"2507.00188","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A modular lifelong-learning wrapper for learned cost prediction that reduces catastrophic forgetting and improves execution-time stability under dynamic workloads.","lead":"LIMAO is a framework that helps learned database query optimizers keep their skills when the database or workload changes over time. It splits query plans into reusable pieces, stores them as small neural modules, and recombines them so old knowledge is not forgotten when new queries arrive.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Volume-switch task encodings are scale-invariant, so a single module cannot represent two different data volumes; the claimed data-volume adaptation is not supported by the architecture.","rationale":"The reader's weakest assumption concerns unreliable module routing when selectivity estimates degrade under Volume Switch. My concern is more fundamental: even if the selectivities are perfectly estimated, the normalized features in Section 5 are scale-invariant, so the routing and the module input carry no information about which volume is active. This is a representational limitation of the architecture, not just an evaluation gap, and it directly affects a core advertised capability: adapting to data-volume shifts while retaining prior knowledge. The concern is testable by inspecting the artifact and does not attack the authors' integrity; it targets the mapping between the feature design and the claimed scenario. Other dynamic scenarios, such as workload switches, may still support part of the central claim, so I would not reject the paper outright. I retain the reader's CONDITIONAL verdict, but with a sharper condition: the authors must either show that volume information is present in the encodings, add an explicit volume/cardinality context feature, or withdraw the volume-switch retention claims until the representational issue is resolved.","tokens_in":24785,"tokens_out":8052,"duration_ms":107130,"concrete_test":"Using the released GitHub artifact, log the full four-part task encoding for a fixed TPC-H query template executed at sf=1 and sf=10, and compute the resulting K-prototype cluster assignments from Eq. (1). If the encodings and assignments are identical (or statistically indistinguishable) while measured execution times differ by roughly an order of magnitude, the model input cannot distinguish the two volumes. Then take one selected module and train it on interleaved cost labels from both volumes, measuring mean absolute error separately for each volume before and after training; if the module cannot achieve low error on both volumes simultaneously, the Volume Switch results must be re-explained without claiming retained knowledge.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5 encodes every task using Feature A = table selectivity divided by table size, normalized to [0,1], plus operator counts, preorder indices, and query flags. None of these features carries absolute table cardinality, data volume, or any other volume-dependent context. Section 9.1 defines the TPC-H Volume Switch as switching between databases with scale factors 10 and 1. For the same query template on sf=1 and sf=10, the normalized selectivities, operator counts, preorder indices, and query flags are identical or nearly identical: the encoding is invariant to a uniform scaling of the data. Consequently, the K-prototype module selector in Section 6 (Eq. 1) routes the same task to the same module in both volumes, and the composed cost predictor receives the same input vector in both cases. But the actual cost of the plan differs by roughly an order of magnitude between the two volumes. A feedforward module cannot map one input vector to two different cost outputs; the offline replay training over B_all in Algorithm 1 (line 7) cannot fit contradictory input-output pairs. The only way the model can track the current volume is to overwrite the previous volume's mapping, which is precisely catastrophic forgetting rather than mitigation. This makes the reported Volume Switch improvements (Tables 5-6, Figures 10c/10d and 12a) hard to credit, and it directly challenges the central claim that LIMAO retains prior knowledge while adapting to shifting data distributions. The paper needs to demonstrate either that Volume Switch encodings actually differ across volumes, or that volume is injected as an explicit context feature; otherwise the architecture cannot represent both volumes simultaneously.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LIMAO, a framework that turns a learned cost prediction model in an LQO into a modular lifelong learner. Query plans are decomposed into sub-plans ('tasks') via break operators (hash/nested-loop/merge joins). Tasks are encoded using normalized selectivities, operator counts, preorder indices, and query flags, and assigned to neural modules via K-prototype clustering. An attention-based merger composes the module outputs to predict plan cost, and a two-phase training scheme with episodic updates and experience replay is used to update the model. LIMAO is integrated with Balsa and Bao, and evaluated on IMDB and TPC-H under static, workload-switch, volume-switch, and combined-switch scenarios, reporting improvements in execution time and stability over the base LQOs.","tokens_in":24991,"tokens_out":7717,"duration_ms":80749,"significance":"If the central claims are correct, LIMAO would be one of the first general frameworks for lifelong learned query optimization, with a modular approach to knowledge retention that goes beyond simple replay or EWC-style regularization. The paper's strengths include integrating the framework with two independent LQOs (Balsa and Bao), covering a wide range of dynamic scenarios, and providing an artifact repository. However, the volume-switch encoding problem, the in-sample hyperparameter selection, and the missing replay-only baseline mean that the empirical evidence for the core claims is currently incomplete, and the contradiction between the abstract's '0 bad plans' claim and Table 6 must be resolved.","major_comments":[{"comment":"The task encoding in Section 5 is scale-invariant: Feature A is table selectivity divided by table size, and Features B, C, and D are operator counts, preorder indices, and query flags, none of which carry absolute data volume. Consequently, for the same TPC-H query template under sf=1 and sf=10, the K-prototype selector (Eq. 1) routes to the same module and the composed cost predictor receives the same input vector, yet the true execution cost differs by roughly an order of magnitude. A feedforward module cannot map one input to two different cost outputs, and training on the replay buffer B_all (Algorithm 1, line 7) cannot fit contradictory input-output pairs; the only way to track the current volume is to overwrite the previous mapping, which is precisely catastrophic forgetting. This undermines the reported Volume Switch improvements in Table 5, Table 6, and Figures 10c/10d and 12a/12c, and the central claim that LIMAO retains prior knowledge while adapting to data-distribution shifts. Please add volume-dependent features (e.g., absolute cardinality estimates or table sizes) to the encoding, or otherwise condition the model on the data volume, and rerun the Volume Switch experiments.","section":"Sections 5, 6, and 9.1 (Volume Switch)"},{"comment":"Module hub sizes (K=2 for HJ and K=3 for NL on IMDB) were selected by comparing variants S1, S2, S3 on the same IMDB Workload Switch scenario that is later used as the main evaluation (Section 9.3.2 states that the three settings are evaluated over 20 iterations in the IMDB Workload Switch scenario). The default hyperparameters are therefore in-sample, so the reported improvements in Tables 2-5 over Balsa are partially attributable to this tuning. Please report hub-size selection on a validation set disjoint from all evaluation scenarios, or provide a sensitivity analysis across all dynamic scenarios.","section":"Section 9.3.2 and Section 9.1"},{"comment":"The ablation study does not isolate the contribution of modular decomposition from the contribution of the experience-replay/episodic-training mechanism. The variant 'Balsa + decomposition + Modular RL training' includes replay-style training but no module hubs, while the full LIMAO includes both; no baseline uses replay with the original monolithic Balsa LCP. Without such a baseline, it is unclear whether the performance gains come from modularity or simply from the replay buffer and episodic updates. Please add a 'Balsa + replay buffer' (or 'LIMAO without decomposition/modules') condition to the ablation.","section":"Section 9.3.3 and Section 8"},{"comment":"The introduction states 'LIMAO can reduce the number of bad plans to 0, while Balsa has a few hundred,' but Table 6 reports 120 timeouts for LIMAO-Balsa in IMDB Workload Switch and IMDB Both Switch, and 98 in TPC-H Both Switch. Please correct this claim to match the reported results, e.g., by specifying which scenarios achieve zero timeouts.","section":"Introduction and Table 6"}],"minor_comments":[{"comment":"The claim of 'up to a 4× speedup' is not supported by any of the reported tables; the largest speedup versus Postgres shown is 3.20× in Table 4, and Tables 2 and 3 show 2.44–2.90×. Please reconcile the abstract and introduction with the reported numbers.","section":"Abstract"},{"comment":"The Workload Switch panels in Figures 10e and 10f show 50 iterations on the x-axis, whereas Section 9.2.2 states the challenging Workload Switch runs for 120 iterations; please unify the time horizon or clarify which experiment the panels depict.","section":"Figure 10"},{"comment":"The algorithm references drift detection, but the paper never specifies the drift-detection method used; please provide the algorithm or citation so the 'Drift Detected' branch is reproducible.","section":"Algorithm 1, line 6"},{"comment":"The derivative of the smoothed execution-time curve is used as a stability metric, but the smoothing method and the threshold for declaring convergence are not defined; please define them.","section":"Section 9.3.1"},{"comment":"Feature A is described as a numerical vector of length n where each entry contains 'table selectivity divided by table size'; please clarify whether 'table size' refers to the row count or a fixed reference and specify how division by zero is handled for absent tables.","section":"Section 5, Feature A"},{"comment":"Feature C (preorder index) is a sequence of integers, not a typical categorical feature; please clarify how it is mapped to a categorical value for the K-prototype dissimilarity computation.","section":"Section 6, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The volume-switch encoding issue is the most serious concern: if the task encodings are truly scale-invariant, the reported Volume Switch results are not credible, and the core claim of adaptation to data-distribution shifts is unsupported. The in-sample hub-size selection compounds this. The paper has a promising architecture and a broad evaluation, but these issues require substantive changes to the feature representation and experimental methodology before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI read LIMAO expecting the usual 'deep learning fixes your optimizer' story, and there is a real idea here: decompose query plan trees at the first occurrence of selected join operators, assign reusable modules via K-prototype clustering, compose them with attention, and train in two phases with experience replay. The break-operator decomposition is genuinely new, the writing is clear, and the integration with both Balsa and Bao shows the framework is engine-agnostic. The Workload Switch experiments (switching between JOB/CEB/BaoQs) show LIMAO reduces variance and timeouts compared to Balsa, and those results are plausible.\n\nThe soft spots are real, though. The one that worries me most is the Volume Switch. Feature A is table selectivity divided by table size, normalized to [0,1]; Features B, C, D are operator counts, tree indices, and query flags. None of these carries absolute data volume or cardinality. For TPC-H sf=1 vs sf=10, the same query template produces near-identical encodings but execution times differ by roughly an order of magnitude. A feedforward module cannot map one input to two outputs, so training on the replay buffer B_all gives contradictory targets. The model can only settle on a compromise, which would lower variance by refusing to adapt. That would make the TPC-H Volume Switch variance reduction look good while actually being a failure to adapt. The authors need to show either that their encodings differ across volumes or inject an explicit volume context, otherwise the volume-switch claims are misleading.\n\nSecondary issues: module hub sizes were tuned on the same IMDB Workload Switch used in evaluation, so part of the gain is in-sample; there is no replay-only baseline to show modularity adds something over plain replay; no error bars or multiple seeds; and the introduction's claim that LIMAO reduces bad plans to zero is contradicted by Table 6, which shows 120 timeouts in some scenarios.\n\nThat said, the workload-switch results survive the volume critique. The modular decomposition idea is worth discussing and the paper is clearly written. It deserves a serious referee, but the authors must address the volume representation problem before those specific claims can be credited.","headline":"Useful modular lifelong-learning idea for LQOs, but the volume-switch claims rest on an encoding that cannot tell two data volumes apart.","tokens_in":25642,"tokens_out":5768,"would_cite":false,"duration_ms":64269,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular lifelong-learning framework wraps existing learned query optimizers so they adapt to shifting workloads and data without forgetting old skills.","keywords":["learned query optimization","lifelong learning","catastrophic forgetting","modular neural networks","cost prediction","plan decomposition","attention mechanism","workload drift"],"falsifier":"The most direct test is an ablation that keeps LIMAO's training and composition intact but replaces the cluster-based module selector with random module assignment; if random routing matches the reported execution-time and variance results, then clustering is not what carries the performance. A more targeted experiment would alternate a workload with data-volume switches and compare the chosen module clusters against an exhaustive search over all modules, checking whether routing errors grow monotonically as selectivities and data sizes drift; if they do, the reuse guarantee breaks.","tokens_in":24472,"feed_emoji":"🧩","tokens_out":8694,"duration_ms":88480,"temperature":0.7,"pith_summary":"This paper argues that learned query optimizers fail in dynamic settings because retraining from scratch loses prior knowledge, and proposes LIMAO, a framework that turns an existing learned cost predictor into a lifelong learner. LIMAO decomposes query plans into sub-plan tasks at selected join operators, clusters similar tasks into shared neural modules, and composes the chosen modules with an attention-weighted merger, so a query's cost is predicted by recombining existing skills rather than relearning them. A two-phase training scheme (lightweight online updates to a working copy plus offline replay of stored experiences) is designed to keep old knowledge intact while absorbing new workload patterns. The authors report up to 40% faster query execution and up to 60% lower execution-time variance under dynamic workloads, with larger stability gains on a second benchmark, and show the framework can be layered onto two different existing learned optimizers.","feed_headline":"Lifelong learning for query optimizers cuts runtime up to 40%","feed_subtitle":"A modular wrapper lets learned optimizers keep old skills while adapting to shifting workloads and data.","key_machinery":"The framework rests on three linked mechanisms. First, the plan decomposer: a top-down traversal of the plan tree that cuts the tree at the first occurrence of each selected break operator type, producing tasks rooted at those operators plus one residual task; break operators are chosen to be performance-critical joins so that sub-plans are neither too coarse nor too fine. Second, the module hub: a K-prototype clustering of task encodings (a table selectivity vector, a join and scan operator count vector, a pre-order index vector, and a binary query-flag vector) into K representative modules per break-operator type, with a dissimilarity threshold that spawns a new cluster when an incoming task is far from all existing ones. Third, the two-phase training loop: online episodes update only a copy of the composed network using freshly executed queries, while offline replay refreshes the original network from an experience buffer, using drift detection to choose between the last-iteration buffer and the full history.","core_discovery":"The central claim is that catastrophic forgetting in learned cost prediction can be avoided by making the cost model modular and compositional. A query plan is split at the first occurrence of designated break operators, normally hash joins and nested-loop joins, yielding sub-plan tasks, and each task is encoded with table selectivities from traditional estimators, operator counts, pre-order tree indices, and query flags. A variation of the K-prototype clustering algorithm maintains a small set of representative neural modules per break-operator type, each specialized for a cluster of similar tasks; the selected modules for a query are combined by a softmax attention layer whose weights are learned jointly with the modules, producing one cost estimate for the whole plan. Training proceeds in two phases: during the online phase only a private copy of the model is updated in short episodes, and during the offline phase the original model is replayed against stored experiences, with drift detection choosing between recent and full history. The paper reports that this design improves execution time by up to 40% and reduces execution-time variance by up to 60% under dynamic workloads, and on a second benchmark it claims a more than two-orders-of-magnitude stability gain and a reduction of severely bad plans to near zero.","pith_inferences":["The two-phase training with a private working copy plus offline replay acts as a safety margin: bad plans produced during online exploration are not committed to long-term modules, a design choice other continual-learners could adopt even outside query optimization.","The attention weights over modules amount to a per-query explanation of which sub-plan patterns are driving the cost estimate, so administrators could use LIMAO's internals to identify performance-critical join patterns in their workloads.","Because routing still relies on traditional selectivity estimates, a natural upgrade would be to replace that one feature with an updatable learned selectivity module; the paper leaves this extension unstated.","The break-operator choice and hub sizes are set manually, so the framework's 'seamless' integration depends on a practitioner knowing which operators drive their workload; an automatic configurator is the obvious follow-up."],"forward_implications":["Learned query optimizers can be updated incrementally on live workloads, removing the expensive step of full retraining from scratch.","Temporary workload reversions no longer erase previously learned plan quality, because old knowledge lives in modules that are recombined rather than overwritten.","The framework wraps around an existing optimizer's cost predictor, so the underlying plan search algorithm does not need to be redesigned to benefit.","The same decomposition-and-composition recipe is portable to other learned database components, such as cardinality estimators, scheduler policies, or index structures."],"supporting_citations":[{"why":"Supplies the reinforcement-learning-based query optimizer that becomes LIMAO's primary prototype and the main baseline for most experiments.","marker":"[68]"},{"why":"Supplies the hint-based learned query optimizer used as the second integration target to demonstrate engine-agnostic reuse.","marker":"[39]"},{"why":"Provides the K-prototype clustering algorithm for mixed numerical and categorical data, which LIMAO adapts to build its module hubs.","marker":"[26]"},{"why":"Establishes the modular lifelong reinforcement learning and neural composition paradigm that LIMAO transfers to plan cost prediction.","marker":"[44]"},{"why":"Defines the catastrophic forgetting problem and the way it is measured, which motivates the stability metrics and the two-phase training design.","marker":"[27]"},{"why":"Provides the movie-information schema and the join-order benchmark workloads used to construct several of the dynamic evaluation sets.","marker":"[33]"},{"why":"Provides the standardized decision-support benchmark and its query templates, used for the second set of dynamic workloads.","marker":"[6]"},{"why":"Supplies the table-selectivity feature idea, originally developed for cardinality estimation under workload drift, that LIMAO reuses in its task encoding.","marker":"[51]"}],"fun_headline_variants":["Query optimizer learns forever, forgets nothing","Modular optimizer cuts query times 40% under shifts","Lifelong learning keeps query optimizers fast and stable","LIMAO: lifelong optimizer that remembers all workloads","Never relearn: modular query optimizer adapts on the fly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design depends on the assumption that splitting query plans at first join operators produces sub-plans that are stable, reusable units, and that the K-prototype clusters built from hand-crafted features, especially table selectivities from traditional estimators, continue to route each new task to the right module even as workloads and data volumes shift.","fun_headline_variants_meta":{"raw":{"variants":["Query optimizer learns forever, forgets nothing","Modular optimizer cuts query times 40% under shifts","Lifelong learning keeps query optimizers fast and stable","LIMAO: lifelong optimizer that remembers all workloads","Never relearn: modular query optimizer adapts on the fly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1623,"prompt_tokens":1052,"completion_tokens":571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":668,"tokens_out":571,"duration_ms":5647,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:21:59.673633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The most direct test is an ablation that keeps LIMAO's training and composition intact but replaces the cluster-based module selector with random module assignment; if random routing matches the reported execution-time and variance results, then clustering is not what carries the performance. A more targeted experiment would alternate a workload with data-volume switches and compare the chosen module clusters against an exhaustive search over all modules, checking whether routing errors grow monotonically as selectivities and data sizes drift; if they do, the reuse guarantee breaks.","supporting_citations":[{"cited_title":"Yang et al","cited_arxiv_id":null,"evidence_quote":"Supplies the reinforcement-learning-based query optimizer that becomes LIMAO's primary prototype and the main baseline for most experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hint-based learned query optimizer used as the second integration target to demonstrate engine-agnostic reuse."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the K-prototype clustering algorithm for mixed numerical and categorical data, which LIMAO adapts to build its module hubs."},{"cited_title":"Modular Lifelong Reinforcement Learning via Neural Composition","cited_arxiv_id":"2207.00429","evidence_quote":"Establishes the modular lifelong reinforcement learning and neural composition paradigm that LIMAO transfers to plan cost prediction."},{"cited_title":"Hayes, and Christo- pher Kanan","cited_arxiv_id":null,"evidence_quote":"Defines the catastrophic forgetting problem and the way it is measured, which motivates the stability metrics and the two-phase training design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standardized decision-support benchmark and its query templates, used for the second set of dynamic workloads."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the table-selectivity feature idea, originally developed for cardinality estimation under workload drift, that LIMAO reuses in its task encoding."}],"review_version":1}