{"id":"b42a4241-3bb7-4739-9b34-cb4d63db8c06","arxiv_id":"2412.07507","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified RL policy can configure modular evolutionary algorithms within a family, but the claimed universal zero-shot generalization across algorithm families is not supported.","lead":"ConfigX trains a single Transformer-based reinforcement learning policy to configure modular evolutionary algorithms across many optimization tasks. The paper claims zero-shot generalization to unseen problems and algorithms, but its own results show it fails to generalize to new algorithm families without fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ConfigX's reward and state features require the global optimum f* of every problem (Eq. 8, Appendix C Eqs. 9–16); in genuine black-box optimization f* is unknown, so the claimed universal agent cannot be deployed on unseen real problems. This is the load-bearing gap.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the reward and state features require the global optimum f* (and the true optimum location x*) for every problem instance, which is unavailable in genuine black-box optimization. My independent reading of the full text confirms this: Eq. (8) in Section 3.3 and Appendix C features 1, 2, 3, 6, 7, and 8 all use f* or x*. This is not merely a training-time normalization detail; the state features are computed at inference, so the policy cannot be applied to an unseen real problem at all without an oracle or a proxy that the paper never defines. The paper's own experiments also concede weak zero-shot transfer to PSO/GA (Figure 4, left panel), which further narrows the 'universal' claim, but the f* dependence is more fundamental because it breaks deployability even within the trained DE family. I credit the paper for a clean modularization framework, a sensible multitask RL formulation, in-family generalization evidence, and ablations showing the transformer/positional-encoding components matter. None of these rescue the central claim as stated. Rejection remains appropriate: the abstract promises robust zero-shot generalization to unseen tasks and an all-purpose configuration agent, but the method as presented cannot operate without knowledge that black-box settings do not provide. The proposed concrete test would settle the concern by showing whether the reported results survive when f* is replaced with a causal estimator.","tokens_in":29656,"tokens_out":2155,"duration_ms":25827,"concrete_test":"Re-run the Ttest,in and T(1)test,out evaluations with f* replaced by a causal proxy in Eq. (8) and features (9)–(16), e.g., the best objective value observed so far or the current population minimum, keeping all other training and evaluation settings identical. If the reported large margin over SMAC3 shrinks or inverts, the headline results are contingent on oracle knowledge of f* and the practical black-box claim fails. A smaller positive control is to recompute the normalized state features on BBOB with f* deliberately shifted by a small epsilon; if the policy's recommended configurations change materially, the learned policy is not robust to f*-estimation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a pre-trained, universal configuration agent that zero-shot generalizes to unseen tasks, including real black-box optimization problems. This claim requires the method to form its state and reward without oracle knowledge of the problem. The paper violates that requirement: Eq. (8) defines the reward as δ × (f*_{p,t−1} − f*_{p,t})/(f*_{p,0} − f*_p), and Appendix C state features sopt_i,1, sopt_i,2, sopt_i,3, and sopt_i,7 all normalize objective values by f*_p, while features sopt_i,6 and sopt_i,8 require the distance to the true optimum x* and are therefore also f*-dependent. In real black-box optimization, f*_p and x* are unknown by definition; no estimation procedure is provided. Consequently, ConfigX cannot even compute its observation vector for a genuinely unseen problem, and the training reward is undefined outside benchmarks with known optima. The paper never discloses this as a limitation, and it directly undercuts the 'universal' and 'zero-shot' claims. Independent support exists for within-family DE generalization and for the value of the transformer architecture (Table 1, Figure 4), but those results are all obtained in settings where f* is supplied. The missing-prior-MetaBBO-baselines and weak PSO/GA transfer are secondary; the f* dependency is the decisive correctness risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConfigX, a MetaBBO framework that modularizes evolutionary algorithms into interchangeable sub-modules and trains a Transformer-based policy with multitask reinforcement learning to dynamically configure any algorithm structure sampled from this module space. The policy is pretrained on DE variants over BBOB instances and then evaluated zero-shot on in-distribution DE tasks, on out-of-distribution real-world problem benchmarks (protein docking and HPO-B), and on out-of-distribution PSO/GA algorithm structures. The paper also studies fine-tuning for lifelong learning and reports ablations of the Transformer components. The central claim is that a single pretrained configuration agent can generalize zero-shot to unseen tasks and outperform state-of-the-art baselines.","tokens_in":29997,"tokens_out":5464,"duration_ms":53372,"significance":"If the central claim were correct, ConfigX would be a substantive step toward a universal automatic configuration agent for evolutionary algorithms. The Modular-BBO system is a genuinely useful engineering contribution: the module catalog is large, the topology rules are practical, and the within-family zero-shot results on DE (Table 1, Figure 4, left) are empirically convincing. The paper also provides detailed appendices and additional experiments on training-set size and sub-module-set size. However, the central claim is undermined by an unacknowledged oracle dependency: the reward and the state features require the global optimum f* (and in two features the optimum location x*) of every problem instance. In genuine black-box optimization these quantities are unknown, so ConfigX cannot even form its observation vector on a truly unseen real problem, and the 'universal' and 'zero-shot' claims in the abstract are not supported. The comparison with SMAC3 is also not apples-to-apples because SMAC3 is used as a static configurator while ConfigX reconfigures at every step.","major_comments":[{"comment":"The reward defined in Eq. (8), rt = δ × (f*_{p,t−1} − f*_{p,t}) / (f*_{p,0} − f*_p), and the state features in Appendix C (Eqs. (9)–(11) and (15), which normalize objective values by f^{0,*} − f^*, and Eqs. (14) and (16), which require the distance to the optimum x*) all depend on the global optimum f* (and x*) of each problem instance. In black-box optimization f* and x* are unknown by definition, and the paper provides no estimation procedure. Consequently, ConfigX cannot compute its observation vector on any genuinely unseen problem, and the training reward is undefined outside benchmarks with known optima. This dependency is not disclosed as a limitation anywhere in the manuscript, and it directly undercuts the 'universal' and 'zero-shot generalization' claims in the abstract and in §3.2.","section":"§3.3, Eq. (8); Appendix C, Eqs. (9)–(16)"},{"comment":"The paper's own results on T(2)test,out, the out-of-distribution algorithm structures (PSO/GA), show zero-shot performance that the authors describe as 'not as expected' and attribute to the structural difference between DE and PSO/GA. Yet the abstract claims 'robust zero-shot generalization to unseen tasks.' The evidence supports only within-family zero-shot transfer on DE, not cross-algorithm generalization. The claim should be restricted accordingly, and the discrepancy between the abstract and the experimental finding should be addressed.","section":"§4.2, observation 4; Figure 4, right"},{"comment":"The SMAC3 baseline is used as a static configurator: it searches for a single configuration per algorithm for the entire optimization process, whereas ConfigX emits a new configuration at every optimization step. Because dynamic per-step configuration has strictly more expressive power, the comparison does not fairly establish that ConfigX 'outperforms state-of-the-art baselines.' The evaluation also omits any competing MetaBBO methods for dynamic algorithm configuration (e.g., GLEET or RL-DAS), which would be the most relevant comparators for the paper's stated contribution.","section":"Appendix E.2; §4.2"}],"minor_comments":[{"comment":"The experimental setup in §4.1 states that training lasts for 50 epochs, but Algorithm 2 in Appendix D iterates 'for epoch ← 1 to 100'. These numbers are inconsistent and should be aligned.","section":"§4.1 vs. Appendix D, Algorithm 2"},{"comment":"The policy outputs a Gaussian distribution N(µ, Σ) over the configuration vector, but many configuration dimensions are discrete or categorical (e.g., operator selections in MULTISTRATEGY modules in Table 3). The paper does not specify how discrete choices are sampled or projected from the continuous Gaussian output, which leaves the action semantics underspecified.","section":"§3.3, Eq. (7); Table 3"},{"comment":"The normalization expressions such as 'min{ f_i / f^{0,∗} − f^∗ }' should be written with parentheses, i.e., 'min{ f_i / (f^{0,∗} − f^∗) }', to avoid ambiguity about which quantity is in the denominator.","section":"Appendix C, Eqs. (9)–(11), (15)"},{"comment":"The y-axis of Figure 4 is not labeled; the reader must infer from Appendix E.2 that it shows the min-max normalized performance metric. Adding a y-axis label would improve readability.","section":"Figure 4"},{"comment":"There is a typo in the ablation discussion: 'embdeddings' should be 'embeddings'.","section":"§4.4"},{"comment":"The text claims that SMAC3 'leads to a larger overall training time on all four task sets,' but Table 4 reports per-task-set training time, not a cumulative sum across task sets. The claim should be clarified or supported by the appropriate aggregate numbers.","section":"Table 4"}],"recommendation":"reject","confidential_remarks":"The oracle dependency on f* is not a minor omission; it is structural. The reward and four of the nine state features cannot be computed without the global optimum, and two additional features require the optimizer's location. This makes the method inapplicable to genuine black-box problems, which is the paper's stated use case. The comparison to a static SMAC3 baseline further weakens the headline claim. In my view, the remaining contribution—modularization plus within-family DE zero-shot transfer—does not justify acceptance in its current form, and the core limitation cannot be fixed without substantially redesigning the state and reward definitions and re-running all experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ConfigX is a real attempt at a universal EA configurator, and the modularization is the most interesting piece. What's actually new: Modular-BBO breaks EAs into tokenized sub-modules and a shared Transformer policy is meta-trained over a joint algorithm–problem task space with multitask PPO. Prior MetaBBO work is tailored to one algorithm (RL-DAS, GLEET, MADAC), so this is a step in a new direction. The within-DE experiments are the paper's strength: a policy pre-trained on 32 DE algorithms over BBOB zero-shots to unseen DE structures, beats SMAC3, and transfers to Protein-Docking and HPO-B. The ablations show the attention and positional encoding contribute. Fine-tuning beats retraining by 2–3x. Those results look real.\n\nThe soft spot is load-bearing. Reward (Eq. 8) and state features (Appendix C, features 1–3, 6–8) use the global optimum f* and, for two features, the true optimum x*. The paper never flags this. In genuine black-box optimization f* and x* are unknown by definition, so ConfigX cannot even compute its state or reward on a truly unseen problem. 'Zero-shot generalization to unseen tasks' is therefore restricted to benchmarks where the answer is already known. The paper's own PSO/GA zero-shot is weak, so the universal agent claim is further confined to DE. Also, the 'outperforms state-of-the-art baselines' line is premature: SMAC3 is a general BO tool, but no prior MetaBBO methods (RL-DAS, GLEET, MADAC) are compared. No code is released.\n\nNone of this undermines the framework's value for benchmark-driven algorithm configuration. With the f* dependence disclosed, the claim narrowed to DE-family configuration with fine-tuning for new structures, and a few MetaBBO baselines added, this would be a solid paper. I agree with the stress-test: this is a correctness risk in the presentation, not a fabrication. I part ways with the reader's REJECT: this is a major-revision case, not a desk reject.\n\nThis paper deserves a serious referee. The framework is novel, the empirical core is careful, and the flaw is correctable in revision. I would not cite it as-is, but I'd read a revised version. Send it to review, with the expectation of major revision.","headline":"ConfigX's modular EA configuration is genuinely new, but the universal zero-shot claim falls because the reward and state need the global optimum f* (and true optimum x*), which real black-box problems don't provide.","tokens_in":30511,"tokens_out":4294,"would_cite":false,"duration_ms":41562,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConfigX claims that one pretrained policy can configure many evolutionary algorithms and zero-shot to unseen tasks.","keywords":["meta-learning for black-box optimization","evolutionary algorithm configuration","multitask reinforcement learning","Transformer policy","modular algorithm design","zero-shot generalization","lifelong learning","automatic configuration"],"falsifier":"Run the exact ConfigX pipeline on a problem instance whose global optimum $f^*$ is withheld at runtime; the state features in Appendix C and the reward in Eq. (8) both normalize by $f^{0,*} - f^*$, so the state and reward are undefined unless $f^*$ is known. The claim of zero-shot generalization to unseen tasks would fail if the method cannot form a state or reward on any benchmark where $f^*$ is not supplied.","tokens_in":29445,"feed_emoji":"🧬","tokens_out":9214,"duration_ms":84749,"temperature":0.7,"pith_summary":"ConfigX aims to answer whether a single pretrained agent can configure diverse evolutionary algorithms, rather than requiring one specialized controller per algorithm. The paper decomposes EAs into modular sub-modules, samples legal algorithm structures from the resulting space, and meta-learns a Transformer-based configuration policy through multitask reinforcement learning over a joint space of algorithms and problems. If the claim holds, automatic configuration becomes reusable: a user could adopt a new EA or problem without training a fresh policy. Experiments report zero-shot gains over a Bayesian-optimization configurator on both synthetic and real problem sets, plus faster fine-tuning on unseen algorithm families.","feed_headline":"One pretrained model configures whole families of EAs","feed_subtitle":"Trained mostly on DE variants, ConfigX zero-shots to PSO/GA structures and real problems, and fine-tunes faster than retraining.","key_machinery":"The load-bearing machinery is the joint optimization task space $\\mathcal{T} = \\mathcal{A} \\times \\mathcal{I}$, where $\\mathcal{A}$ is the space of legal algorithm structures generated by Modular-BBO, a hierarchical module system with controllable and uncontrollable sub-modules, topology rules, and a 16-bit module id, and $\\mathcal{I}$ is the problem instance space. The policy is a Transformer that fuses each module id with nine landscape and status features, adds positional encodings, applies three blocks of module-aware multi-head self-attention, and decodes a per-module configuration. PPO optimizes the multitask objective $\\frac{1}{KN}\\sum_{i=1}^{KN}\\sum_{t=1}^{H}\\gamma^{t-1}R(s_t^i, c_t^i)$.","core_discovery":"ConfigX's central claim is that universal configuration is learnable: one policy, pretrained on a moderate set of optimization tasks, can dictate configurations for evolutionary algorithms it was never trained on and for problems outside its training distribution. The paper represents an EA as a sequence of typed sub-modules, each carrying a 16-bit module id and nine normalized optimization-status features, and lets a Transformer with module-aware self-attention read that sequence and emit per-module configuration distributions. Training maximizes the accumulated normalized improvement across many tasks with PPO. The authors report that the pretrained model zero-shots well within its algorithm family and to unseen real-world problems, outperforms the SMAC3 automatic configurator, and recovers strong performance on PSO/GA structures via fine-tuning when zero-shot transfer is weaker.","pith_inferences":["Beyond the paper's tests, the reward and state features require the global optimum $f^*$; on a truly black-box problem with unknown $f^*$, ConfigX as specified cannot compute its own state or reward, so the method's demonstrated scope is benchmarks with known optima.","A natural extension not explored in the paper is training with a learned surrogate of the optimum, which would let the same architecture run in genuine black-box settings; that is the key experiment I would want next.","The module-aware attention suggests a diagnostic: checking which module tokens the policy attends to for DE versus PSO/GA could reveal whether weak zero-shot transfer comes from a genuinely shared representation or from family-specific shortcuts.","Because the algorithm space is hand-curated from existing EA literature, the universal claim is bounded by the module catalog; adding a genuinely new operator type still requires extending Modular-BBO, though fine-tuning may help."],"forward_implications":["A user who wants to configure a new DE variant could plug it into Modular-BBO and use the pretrained model directly instead of running a fresh configuration search.","Pretraining on synthetic problems transfers to real problem suites such as protein docking and HPO-B, suggesting that industrial tasks can be configured without per-task tuning.","When a genuinely new algorithm family appears, the pretrained weights provide a warm start that reaches the same performance in roughly one-third to one-half the learning steps of retraining from scratch.","Dynamic per-step configuration can outperform a single static configuration found by Bayesian optimization, because the policy adapts parameters as optimization progresses."],"supporting_citations":[{"why":"Supplies the PPO policy-gradient algorithm used to optimize the multi-task reinforcement learning objective.","marker":"Schulman et al. 2017"},{"why":"Supplies the Transformer architecture, multi-head self-attention, and positional encoding used in the configuration policy.","marker":"Vaswani et al. 2017"},{"why":"Provides the BBOB synthetic problem suite used to construct training and in-distribution task sets.","marker":"Hansen et al. 2010"},{"why":"Provides protein-docking problem instances for the out-of-distribution real-problem test set.","marker":"Hwang et al. 2010"},{"why":"Provides HPO-B hyperparameter-optimization tasks for the out-of-distribution real-problem test set.","marker":"Arango et al. 2021"},{"why":"SMAC3 is the automatic-configuration baseline the paper compares against in zero-shot experiments.","marker":"Lindauer et al. 2022"},{"why":"Source of the nine optimization-status state features the policy conditions on.","marker":"Guo et al. 2024"}],"fun_headline_variants":["One pretrained RL model configures any evolutionary algorithm","ConfigX: one RL policy configures diverse EAs in zero-shot","Pretrain once, configure any EA: ConfigX zero-shots","Multitask RL yields a universal configuration agent for EAs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"ConfigX's reward and state features assume every problem's global optimum is known in advance, which is precisely what a black-box optimization problem hides.","fun_headline_variants_meta":{"raw":{"variants":["One pretrained RL model configures any evolutionary algorithm","ConfigX: one RL policy configures diverse EAs in zero-shot","Pretrain once, configure any EA: ConfigX zero-shots","Multitask RL yields a universal configuration agent for EAs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001222,"raw_usage":{"total_tokens":5017,"prompt_tokens":929,"completion_tokens":4088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":4016}},"tokens_in":545,"tokens_out":4088,"duration_ms":26061,"temperature":1.0,"reasoning_tokens":4016,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:46:37.248697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact ConfigX pipeline on a problem instance whose global optimum $f^*$ is withheld at runtime; the state features in Appendix C and the reward in Eq. (8) both normalize by $f^{0,*} - f^*$, so the state and reward are undefined unless $f^*$ is known. The claim of zero-shot generalization to unseen tasks would fail if the method cannot form a state or reward on any benchmark where $f^*$ is not supplied.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BBOB synthetic problem suite used to construct training and in-distribution task sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides protein-docking problem instances for the out-of-distribution real-problem test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SMAC3 is the automatic-configuration baseline the paper compares against in zero-shot experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the nine optimization-status state features the policy conditions on."}],"review_version":1}