{"id":"a871904c-9099-40be-b8f0-666e7a4c865b","arxiv_id":"2505.02010","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Q-Mamba trains a Mamba-based Q-function controller for evolutionary algorithm configuration on an offline dataset and matches or slightly exceeds online baselines on BBOB benchmarks.","lead":"This paper introduces Q-Mamba, an offline training method that learns to configure evolutionary optimization algorithms from pre-collected data instead of training online. It reports competitive optimization performance with less directly-measured training time than online reinforcement-learning baselines, though the cost of building the offline dataset is not counted in that time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claim omits dataset-collection cost: pre-training RLPSO/LDE/GLEET is required to build the E&E dataset, so Q-Mamba's total budget exceeds the online baselines, contradicting 'at most half training budget'.","rationale":"After reading the paper and the reader's verdict, the single most load-bearing issue is the efficiency accounting. The reader identified coverage of the E&E dataset as the weakest assumption, and that is a legitimate concern for the neuroevolution transfer, but it is harder to falsify from the paper alone. The efficiency problem is directly visible from the paper's own text: Section 4.2 describes pre-training RLPSO, LDE, and GLEET to collect the exploitation portion of the dataset, and Table 1 shows those pre-training runs take 25-28h each. The paper reports only Q-Mamba's 13h as its 'training time,' which is not the cost of producing the policy. Since the offline dataset is generated by the very online baselines being compared, the claimed 'less than half training budget' is an artifact of excluding the major cost. This undermines the central claim of attaining efficiency via offline learning. The concern can be settled by re-accounting the budget or running a matched-budget comparison, so it is appropriate to keep the verdict CONDITIONAL: the authors should either report full pipeline cost or revise the efficiency claim. I therefore do not change the reader's verdict. The coverage concern remains valid as a secondary limitation, especially for OOD neuroevolution states, but it is not the primary reason the central claim is at risk.","tokens_in":23237,"tokens_out":8726,"duration_ms":85734,"concrete_test":"Compute the total wall-clock cost of the Q-Mamba pipeline as the sum of the reported pre-training times of RLPSO, LDE, and GLEET (Table 1 last row: 28h, 28h, 25h) plus Q-Mamba's own training time (13h) and the time for random rollouts, for each low-level algorithm. If this total exceeds the online baseline time (e.g., 25h for GLEET), the 'at most half training budget' claim fails. Stronger: re-run the comparison under a matched total budget—give online baselines the same wall-clock budget as the full Q-Mamba pipeline including data collection—and compare final Perf values. Also report dataset-collection time separately in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline efficiency claim (Section 5.2, item i; abstract) is that Q-Mamba consumes at most half the training budget of online baselines. The reported 13h in Table 1, however, includes only Q-Mamba's own policy training. Section 4.2 states that the E&E dataset is built by pre-training RLPSO, LDE, and GLEET on the target algorithm/problem distribution, then rolling them out to collect µ*D exploitation trajectories. Those pre-training runs are exactly the online training runs whose times are listed in Table 1 (28h, 28h, 25h). For each of the three low-level algorithms, the Q-Mamba pipeline must therefore pay roughly 28+28+25=81h of pre-training plus 13h of Q-Mamba training (plus random-rollout time), i.e., over three times the 25-28h of a single online baseline. Even if a single behavior policy were used, 28h+13h=41h already exceeds the half-budget bound for GLEET (12.5h). The comparison is thus between a single online training run and an offline pipeline that implicitly includes multiple online runs to generate data. Unless the data-collection cost is amortized across many downstream tasks or provided by an external budget, 'at most half training budget' is not supported by the paper's own numbers. This is load-bearing because the central contribution is 'both effectiveness and efficiency' through offline learning; if the efficiency disappears under full-cost accounting, the motivation for the offline framework is substantially weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Q-Mamba, an offline meta-black-box-optimization (MetaBBO) framework for dynamic algorithm configuration (DAC). The method transforms DAC into a long-sequence decision process, decomposes the Q-function over the low-level algorithm's hyperparameters, and learns the decomposed Q-functions from a pre-collected dataset of DAC trajectories using a Mamba-based sequence model with a conservative Q-learning loss. The authors compare Q-Mamba with online MetaBBO baselines (RLPSO, LDE, GLEET) and offline RL baselines (DT, DeMa, QDT, QT, Q-Transformer) on BBOB training/test instances, and also evaluate zero-shot transfer to neuroevolution tasks. They report competitive or superior optimization performance and claim that Q-Mamba consumes at most half the training budget of the online baselines.","tokens_in":23563,"tokens_out":6791,"duration_ms":69287,"significance":"If the efficiency claim is corrected to account for all pipeline costs, the paper would still be a valuable first step in offline MetaBBO: it introduces a plausible combination of Q-function decomposition, conservative regularization, and a Mamba sequence model, and it provides code, ablations, and transfer experiments. The in-distribution results show that an offline policy can roughly match the performance of online baselines, which is a useful existence proof. However, the headline efficiency contribution is currently undercut by an incomplete cost accounting and by an arithmetic inconsistency even within the paper's own Table 1, so the significance as stated is not yet established.","major_comments":[{"comment":"The claim that Q-Mamba consumes at most half the training budget of online baselines omits the cost of building the E&E dataset. Section 4.2 states that the dataset is collected by pre-training RLPSO, LDE, and GLEET on the target algorithm/problem distribution and then rolling them out; these pre-training runs are the same online runs whose times are listed in Table 1 (28h, 28h, 25h). A full-cost accounting therefore gives roughly 28+28+25+13=94h for the Q-Mamba pipeline, which is over three times the cost of a single online baseline. Even if only one behavior policy were used, 28h+13h=41h exceeds the half-budget bound of 12.5h for GLEET. Moreover, the arithmetic fails even with Q-Mamba's own reported 13h, since 13h is not at most half of GLEET's 25h. The efficiency comparison should either include behavior-policy training time, with the amortization scenario explicitly stated, or the claims in the abstract and Section 5.2 should be restricted to 'policy-optimizer training time' rather than total training budget.","section":"§4.2, Table 1, §5.2(i), Abstract"},{"comment":"The results support 'competitive' but not 'superior' performance, because the reported differences between Q-Mamba and the best online baselines are within the reported error bars on all three low-level algorithms. For example, on Alg1 the mean performance of Q-Mamba is 9.973E-01 ± 2.441E-03 while RLPSO is 9.953E-01 ± 3.322E-03, and on Alg2 the interval for Q-Mamba (9.950E-01 ± 9.981E-03) fully overlaps that of RLPSO (9.914E-01 ± 4.497E-03). No significance tests or effect sizes are reported. The paper should add a paired or per-instance significance test across the 19 runs, or soften the superiority claim to a competitiveness claim.","section":"Table 1, §5.2"},{"comment":"Equation (5) uses the same network parameters for the prediction and the target in the decomposed Bellman backups. The targets max_j Q^t_{i+1,j} and r_t + γ max_j Q^{t+1}_{1,j} are outputs of the same Q-function being trained, and the paper does not state whether stop-gradient or a target network is applied. Without stop-gradient, the loss for i<K is not a standard Bellman backup for Q^t_i but a consistency constraint that could in principle be minimized by driving Q^t_i and max_j Q^t_{i+1,j} to a common constant, with the reward signal entering only through the final action dimension. The convergence argument in Appendix A assumes an exact optimal Q-function update, not the gradient dynamics of Eq. (5). Please state explicitly whether gradients are blocked on the target terms, and if they are not, explain why the network does not exhibit the collapse suggested by the loss structure.","section":"§4.3, Eq. (5), Appendix A"}],"minor_comments":[{"comment":"There is a typo: 'his is especially appealing' should read 'This is especially appealing'.","section":"§5.2(i)"},{"comment":"The description of s7 contains 'potion' instead of 'portion', and the definition of s8 uses s_t both as a stagnation counter and as the state symbol, which is confusing; please rename one of the two quantities.","section":"Table 4, rows s7–s8"},{"comment":"The random strategy used to collect the exploration trajectories is not specified beyond 'randomly control the hyper-parameters'; please state whether hyperparameter values are drawn uniformly over their ranges, over the discretized bins, or in some other way, since this affects the coverage of the offline dataset.","section":"§4.2"},{"comment":"The timing comparison would be more meaningful if the hardware and software environment (GPU/CPU, batch size, number of workers) were reported, and if the same timing protocol were used for all baselines (including any evaluation or checkpointing overhead).","section":"Table 1, last row"},{"comment":"The architecture details for the Mamba-based Q-learner are minimal ('we use the mamba-block in Mamba repo, with default settings'); for reproducibility, please report the number of layers, hidden dimension, state dimension, and total parameter count.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim of 'at most half training budget' is not supported by the paper's own numbers when the cost of pre-training the behavior policies is included, and it is arithmetically false even for Q-Mamba's own reported time for the GLEET baseline. This is the main load-bearing issue. The effectiveness results are plausible but currently not statistically validated. If the authors add full-cost accounting, significance tests, and a stop-gradient clarification for Eq. (5), the contribution could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first offline MetaBBO paper, the experiments are honestly done, and the decomposition-plus-conservative-loss story is sensible. But the efficiency number the paper sells is 13 hours of Q-Mamba training versus 28/28/25 hours of online baselines, while the ~81 hours needed to pre-train the behavior policies that build its own E&E dataset never enter the account.\n\nWhat is genuinely good: treating DAC as a long-sequence decision process, decomposing the Q-function per hyperparameter, adding CQL regularization, and using a Mamba backbone is a coherent design. The paper tests on three algorithms with 3, 10, and 16 configurable parameters, held-out BBOB instances, and a zero-shot neuroevolution setting, and it ablates the mixing ratio and the loss coefficients. The appendix proof of the Q-function decomposition is correct in spirit. Code is promised. The limitation about per-algorithm retraining is acknowledged. Credit where due: this is a real first for the subfield.\n\nNow the soft spots, in proportion.\n\nThe efficiency claim is load-bearing and, as written, unsupported. Section 4.2 says the E&E dataset is built by pre-training RLPSO, LDE, and GLEET, then rolling them out. Their listed training times are 28h, 28h, and 25h, so the full Q-Mamba pipeline costs about 81h of pre-training plus 13h of Q-Mamba training, over three times a single online baseline. Even using just one behavior policy, 28h + 13h already exceeds half of GLEET's 25h. The claim 'at most half training budget' only holds if dataset collection is amortized across many downstream tasks or supplied by an external budget, and the paper should say that plainly. This matters because efficiency is half of the motivation.\n\nSecond, the performance margins over the best online baselines are within the reported error bars, and there are no significance tests. The honest claim is 'competitive,' not 'superior.' Third, the explanation in Section 5.2(iv) that Transformer has 'linear time invariance' is technically confused; LTI is an SSM property, not a property of attention. It is an explanatory mistake, not a load-bearing one. Fourth, ConfigX is discussed and used as the source of the algorithm space but never compared; the paper owes a sentence on why.\n\nThe citation pattern is fine; the self-citations are to GLEET and ConfigX, which are the methods actually in play.\n\nBottom line: this paper deserves serious refereeing. A careful revision that fixes the efficiency accounting and softens the comparative claims would make it a solid contribution for the MetaBBO and learning-to-optimize community. I would send it to reviewers.","headline":"First offline MetaBBO paper with honest experiments and a sensible decomposition-plus-CQL design; the headline efficiency claim, though, only works if you ignore the cost of pre-training the behavior policies that generate its own dataset.","tokens_in":24101,"tokens_out":2623,"would_cite":true,"duration_ms":29950,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that dynamic configuration of a black-box optimizer can be meta-learned entirely offline, matching or beating online reinforcement-learning baselines on unseen problems while using less than half the training budget.","keywords":["meta black-box optimization","offline reinforcement learning","dynamic algorithm configuration","Q-function decomposition","conservative Q-learning","Mamba state space model","BBOB benchmark","neuroevolution"],"falsifier":"Two checks would settle the claim. Directly: count the distinct discretized configuration bins (out of 16^K for K=16) that appear in the exploitation half of the dataset, then compare how often the deployed policy selects bins absent from the dataset; if unvisited bins are chosen frequently and systematically underperform visited alternatives in brute-force evaluation, the coverage premise fails. Experimentally: train Q-Mamba on trajectories restricted to one sub-region of a continuous hyper-parameter (for example DE scale factor F <= 0.5), then test on the complementary region; if its argmax bin there does not match the best bin found by exhaustive evaluation, the offline Q-function has not learned the missing region.","tokens_in":23045,"feed_emoji":"🧬","tokens_out":14903,"duration_ms":132065,"temperature":0.7,"pith_summary":"Meta-black-box-optimization normally learns its controller by interacting online with the optimizer-environment, which is slow because a single run takes hundreds of optimization steps. This paper claims that the same quality of control can be learned offline: given a fixed dataset of past configuration trajectories, an agent can match or slightly beat online-trained agents on unseen benchmark problems while consuming at most half the training budget. The proposed system, Q-Mamba, rewrites the configuration choice as a long sequence of per-parameter decisions, learns a decomposed Q-function for each parameter with a conservative regularization term that keeps out-of-distribution actions from being overvalued, and processes the whole sequence with a Mamba state-space model. The practical bet is that expensive online data collection in algorithm configuration can be replaced by a single offline dataset that mixes expert and random trajectories, with no loss in downstream optimization quality.","feed_headline":"Q-Mamba: offline RL matches online baselines at half the budget","feed_subtitle":"Trained offline, it matches or beats online baselines on unseen BBOB problems and transfers to neuroevolution.","key_machinery":"The load-bearing object is the decomposed, discretized Q-function: each of the K hyper-parameters of the low-level optimizer is mapped to M=16 action bins (5-bit tokens), and the Q-learner predicts the value of bin j for parameter i autoregressively, conditioned on the optimization state and the previously selected bins, Q(a_i | s, a_{1:i}); the per-dimension Bellman backup ties dimension i to the max over dimension i+1, and the last dimension closes the loop with the reward and next state. Around this, three mechanisms carry the argument: the E&E dataset, 10,000 trajectories split evenly between rollouts of three pre-trained MetaBBO policies and a random configuration policy, which supplies both exploitation quality and exploration coverage; the compositional Q-loss, whose three branches are the TD error for the selected bin, a weighted (beta=10) TD error on the last dimension, and a CQL-style conservative term (lambda=1) pushing unselected bins toward zero; and the Mamba block, a selective state-space model whose input-dependent parameters and hardware-aware parallel scan make long-sequence Q-learning efficient. The appendix's proof that the decomposed Bellman updates equal the full-action Bellman update is what legitimizes treating configuration as a sequence problem.","core_discovery":"The paper's central claim is that dynamic algorithm configuration does not need online interaction: a meta-level policy trained purely by offline Q-learning on a pre-collected dataset matches or exceeds online RL baselines (RLPSO, LDE, GLEET) on held-out BBOB instances, and does so at roughly half the training time (about 13 hours versus 25-28 hours). The discovery that makes this work is that the joint configuration action should be decomposed into a sequence of per-hyperparameter Q-functions, one prediction step per parameter in a fixed order, so that learning the value of the massive joint action space is replaced by learning K small autoregressive classification problems over 16 discretized bins each. Training uses a compositional loss that combines the decomposed Bellman backup with a conservative Q-learning term that drives Q-values of unselected bins toward zero, and the data comes from an exploration-exploitation balanced dataset in which half the trajectories are rollouts of pre-trained MetaBBO policies and half are random configurations. A proof in the appendix shows the decomposed backups are consistent with the full Bellman backup, so optimizing the per-dimension Q-functions optimizes the original MDP. The same policy transfers zero-shot to neuroevolution on Mujoco control tasks, where it performs comparably to the online baselines despite having been trained only on synthetic benchmark functions.","pith_inferences":["The efficiency gain is amortized, not free: the exploitation half of the dataset is produced by pre-training three online MetaBBO policies, so an end-to-end cost accounting that includes that upstream training is the paper's implicit next step; the paper also notes Q-Mamba must be retrained from scratch for a new optimizer because no algorithm-identity features are provided to the Q-learner.","The decomposition recipe is generic beyond evolutionary computation: any control problem with a factored action space and a long decision horizon (scheduling, circuit or prompt configuration) could use the same offline Q-decomposition and Mamba backbone, but the paper only demonstrates it on MetaBBO and explicitly leaves validation on other RL domains to future work.","The optimal data mix likely tracks behavior-policy quality: because mu=0.5 beats both extremes, a dataset-quality curve varying mu against the strength of the exploitation policies would reveal whether the 50/50 choice is intrinsic to dynamic algorithm configuration or an artifact of these three baselines.","Discretization caps the achievable control: the 16-bin uniform grid makes parameter values inside a bin indistinguishable to the policy, so for hyper-parameters with sharp performance cliffs a post-hoc refinement of the selected bin's interior would be a natural extension the paper does not test."],"forward_implications":["If the results hold, offline data alone yields a meta-level configuration policy whose mean accumulated performance improvement on held-out BBOB instances is competitive with or higher than every online and offline baseline across all three optimizer complexities (K=3, 10, 16).","Meta-training time drops to roughly half that of the online baselines (about 13 hours versus 25-28 hours), with no extra inference cost, making learned configuration practical when each optimization run is itself expensive.","Zero-shot transfer works: a controller trained only on synthetic BBOB functions of at most 50 dimensions can configure the evolution of MLP policies with thousands of parameters on Mujoco continuous-control tasks at a level comparable to the online baselines.","The two loss coefficients carry weight: removing the conservative regularization (lambda=0) or unweighting the last action dimension (beta=1) both measurably reduce performance, and a 50/50 mix of expert and random trajectories beats either pure exploitation or pure exploration data.","Control granularity is not the bottleneck: increasing action bins from 16 up to 512 does not improve performance, consistent with the tested evolutionary operators being insensitive to small parameter changes."],"supporting_citations":[{"why":"Supplies the autoregressive Q-function decomposition scheme (per-action-dimension Bellman backups) that Q-Mamba adapts to algorithm configuration.","marker":"Chebotar et al., 2023"},{"why":"Supplies conservative Q-learning, the regularization branch of the Q-loss that pushes Q-values of unselected bins toward zero.","marker":"Kumar et al., 2020"},{"why":"Supplies the Mamba block, the selective state-space model used as the Q-Learner backbone.","marker":"Gu & Dao, 2023"},{"why":"Supplies the COCO/BBOB problem distribution used for training and held-out testing.","marker":"Hansen et al., 2021"},{"why":"Supplies the ConfigX algorithm construction space from which the three low-level optimizers (K=3, 10, 16) are sampled.","marker":"Guo et al., 2024b"},{"why":"RLPSO is one of the three pre-trained policies whose rollouts form the exploitation half of the offline dataset, and an online comparison baseline.","marker":"Wu & Wang, 2022"},{"why":"LDE is one of the three pre-trained policies whose rollouts form the exploitation half of the offline dataset, and an online comparison baseline.","marker":"Sun et al., 2021"},{"why":"GLEET is one of the three pre-trained policies generating exploitation trajectories, an online baseline, and the source of the 9-dimension optimization-state design.","marker":"Ma et al., 2024b"},{"why":"Decision Transformer is the conditional-imitation offline baseline that Q-Mamba must outperform to justify value-based offline learning.","marker":"Chen et al., 2021"},{"why":"Prefix-sum parallel scan underpins Mamba's hardware-aware computation, the basis of the claimed training-efficiency advantage.","marker":"Blelloch, 1990"}],"fun_headline_variants":["Offline MetaBBO: Q-Mamba matches online at half the budget","Decomposed Q-functions make offline MetaBBO efficient","Q-Mamba: offline RL for meta black-box optimization","Half the training, same results: offline MetaBBO with Q-Mamba"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The offline dataset must actually cover enough of the configuration space, the regions expert policies seldom visit as well as the ones they favor, for the learned Q-values to rank actions correctly on out-of-distribution tasks like neuroevolution; if the behavior policies under-cover the space, the offline Q-function cannot recover good decisions there.","fun_headline_variants_meta":{"raw":{"variants":["Offline MetaBBO: Q-Mamba matches online at half the budget","Decomposed Q-functions make offline MetaBBO efficient","Q-Mamba: offline RL for meta black-box optimization","Half the training, same results: offline MetaBBO with Q-Mamba"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2523,"prompt_tokens":1088,"completion_tokens":1435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":704,"tokens_out":1435,"duration_ms":10922,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:05:18.694472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two checks would settle the claim. Directly: count the distinct discretized configuration bins (out of 16^K for K=16) that appear in the exploitation half of the dataset, then compare how often the deployed policy selects bins absent from the dataset; if unvisited bins are chosen frequently and systematically underperform visited alternatives in brute-force evaluation, the coverage premise fails. Experimentally: train Q-Mamba on trajectories restricted to one sub-region of a continuous hyper-parameter (for example DE scale factor F <= 0.5), then test on the complementary region; if its argmax bin there does not match the best bin found by exhaustive evaluation, the offline Q-function has not learned the missing region.","supporting_citations":[],"review_version":1}