REVIEW 3 major objections 7 minor 1 cited by
Splitting an LLM's optimization program into generate_config, improve, and perturb, then combining them via basin-hopping, sets new hexagon-packing records for n=11, 12, 15, and 16 and, with small human edits, lifts an autocorrelation lower
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 01:11 UTC pith:SO7I7LWV
load-bearing objection A genuinely useful modular reformulation of LLM-guided evolution with promising hexagon results, but the missing feasibility proof and the missing spherical-code section keep the current draft from being trusted at face value. the 3 major comments →
ImprovEvolve: Basin-Hopping Meets LLM-Guided Evolutionary Search
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
ImprovEvolve claims that the right unit for LLM-guided evolution is a triple of operators — a feasible initializer, a local improver, and an intensity-controlled perturbation — rather than a monolithic end-to-end optimizer; the paper evolves a class implementing these three methods and composes them in a two-stage monotonic basin-hopping loop with a geometrically decaying intensity schedule, and shows on identical budgets that this reformulation dominates the monolithic baseline. The same hexagon program, evolved only for n=11, sets new records for n=11 (L=3.9245), 12 (3.9416), 15 (4.4473), and 16 (4.5275), and the autocorrelation program reaches C(f)=0.9512 unedited and 0.96258 after human
What carries the argument
The load-bearing object is the three-method interface — generate_config(seed), improve(x), perturb(x, sigma) — together with the two-stage schedule of Algorithm 1: Stage A refines K random seeds and keeps the best; Stage B repeats R rounds of M steps in which sigma decays geometrically from sigma_max to sigma_min and each step perturbs then improves the current best, accepting only strict improvements (T=0 basin-hopping). The interface does the work by converting the LLM's task from designing a complete global search to designing two local move generators, and it is what gives the resulting program its transferability to unseen problem sizes and its ability to accept an external starting sol
Load-bearing premise
The paper's record values are taken as certified by its own validation code, and for the headline autocorrelation improvement the load-bearing premise is that the human edits (much finer resolution and a hot start from the prior record) are not the real cause of the gain — the modular interface is.
What would settle it
Independently re-run the published HEX 11 class under Algorithm 1 and verify it reaches L=3.9245 with non-overlapping, contained hexagons; and apply the same human edits (progressive resolution up to 1.6M steps, relaxed clipping, more L-BFGS iterations) directly to the monolithic program that produced the 0.96102 autocorrelation bound, without the generate/improve/perturb interface, to see whether it alone reaches 0.96258. The modularity claim fails if the packings are invalid or the monolithic program with identical edits matches the bound.
If this is right
- The same program evolved for n=11 generalizes to n=12–30, producing valid structured packings at sizes with no previously published baseline.
- Because improve accepts any configuration, an evolved class can be hot-started from a human- or machine-built solution; the ACI2 gain from 0.96102 to 0.96258 came from refining the prior record.
- The ablation (A only 3.9282, B only 3.9269, A+B 3.9245) indicates both stages are necessary; the modular interface is what makes them composable.
- The monolithic baseline's evolved hexagon program produced overlap errors when validated on n=12, while the modular program did not; the abstract also reports that the approach lowers the best-known maximum cosine on a majority of 90 spherical-code instances by up to 2.4%.
- When a strong LLM was asked to make the same style of small edits that helped the human-expert variant, the edited programs performed worse than the unedited version, indicating that current models do not reliably choose safe hyperparameter changes.
Where Pith is reading between the lines
- Extension: the paper's diagnosis suggests a general recipe — factor a hard global optimization into a local solver and an exploration operator, embed them in a classical metaheuristic, and let the LLM fill in only the two operators. This is testable on combinatorial problems or black-box benchmarks where monolithic program search is known to stagnate.
- Extension: the headline autocorrelation gain is credited to the modular interface, but the paper does not run the decisive control — applying the same human edits (progressive resolution to 1.6M steps, relaxed clipping) directly to the monolithic program. Until that comparison is made, 0.96258 is evidence for the interface plus human scaling, not for either alone.
- Extension: results are single-run best values with no confidence intervals; a natural follow-up is repeated evolution to estimate variance, or a swap of the outer loop (e.g., simulated annealing with T>0) to test whether the modular advantage is independent of the basin-hopping wrapper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ImprovEvolve, an LLM-guided evolutionary framework that replaces the usual monolithic program (which directly outputs a candidate solution) with a modular class exposing generate_config, improve, and perturb, and then combines these subroutines through monotonic basin-hopping with a scheduled perturbation intensity. The method is tested on two AlphaEvolve benchmark problems: hexagon packing in a hexagon and the second autocorrelation inequality. The authors report new state-of-the-art hexagon packings for n=11,12,15,16 (and, after human edits, n=14,17,23), and an improved ACI2 lower bound of 0.96258 obtained by a human-edited variant seeded with AlphaEvolve's solution. The paper also contains an ablation of the two validation stages and a comparison with the GigaEvo baseline. The abstract further claims improvements on spherical codes, but no spherical-code experiment appears anywhere in the body.
Significance. If the hexagon-packing results are certified, the paper makes a useful and timely point: for LLM-guided evolutionary search, decomposing the target program into initialization, local improvement, and perturbation operators can be more effective than asking the LLM to produce an end-to-end optimizer. The modular interface also enables transfer across problem sizes (HEX n) and easy seeding from expert constructions, which are concrete strengths. The authors are transparent about the human edits in the +E variants, and the two-stage ablation on HEX 11 supports the view that both multi-seed initialization and basin-hopping matter. However, the current manuscript lacks independent feasibility certificates for the claimed packings, the headline ACI2 number is not attributable to the evolved program alone, and the abstract contains an unsupported spherical-codes claim. These issues are fixable but are load-bearing for the published version.
major comments (3)
- [§4.1, Appendix A.2.1, Tables 1 and 3] The reported hexagon-packing side lengths are not certified. The appendix code for improve() minimizes L + w_cont*containment + w_ov*overlap with L-BFGS-B/SLSQP and returns the raw final parameters; no independent feasibility check is performed, and Section 4 explicitly relaxes the invalid-configuration requirement during final validation. Because the penalty tolerances (margin=1e-7, ftol=1e-10/1e-11) do not guarantee exact constraint satisfaction, the L-values in Tables 1 and 3 could correspond to configurations with small overlaps or containment violations. Please release the exact coordinates for every claimed SOTA packing and provide a post-hoc certificate of geometric feasibility (e.g., maximum violation computed in high precision or an exact SAT/containment check). Without this, the central hexagon SOTA claims are unverified.
- [§4.2, Appendix B.2.1, Table 1] The unedited ImprovEvolve program reaches C(f)=0.9512, below AlphaEvolve's 0.96102. The headline 0.96258 is produced by ImprovEvolve+E, which (i) starts from AlphaEvolve's 50,000-step solution and (ii) applies human edits: a progressive resolution schedule up to 1.6M steps, removal of the 1e-10 lower clip, and increased L-BFGS iterations. These are domain-specific human choices, not outputs of evolution. Therefore the contribution of the modular interface to this improvement is not established. I request either a clear re-framing of 0.96258 as a human-LLM collaboration that does not measure the evolved architecture alone, or a control experiment applying the same starting point and human edits to a monolithic baseline (e.g., AlphaEvolve's own program). Without such a control, the abstract's wording 'the evolved and human-scaled program pushes the lower bound' overstates the evidence.
- [Abstract] The abstract claims: 'For spherical codes, the ImprovEvolve program lowers the best-known maximum cosine for the majority of 90 randomly chosen diverse state-of-the-art spherical codes, achieving relative improvements of up to 2.4%.' The body of the paper contains no spherical-codes section, table, figure, or discussion. This is an unsupported empirical claim in the abstract. Please either add the full experimental details (setup, random selection protocol, baselines, results) or remove the claim. If retained, it must be reproducible.
minor comments (7)
- [§2] The sentence 'Recently, Novikov et al. [11], Romera-Paredes et al. [12] proposed FunSearch and AlphaEvolve frameworks, respectively' reverses the mapping: [11] is AlphaEvolve and [12] is FunSearch.
- [§4.1] The text says the evolved program was validated 'for all 11 ≤ n ≤ 24' but then reports results for n=25–30 as well. State the validation range consistently.
- [Table 4] The column headings 'A only B only A + B A + B + LLM 1 Lattice + B2' are difficult to parse. Expand the abbreviations in the caption or table header.
- [Appendix A.2.1] The code listings contain visible '↩→' line-wrap artifacts; please clean these up in the camera-ready version.
- [Author affiliations] The affiliations contain unresolved 'AXXX' placeholders that should be completed.
- [§5.1] The ablation reports single best L-values without runs or variance. Given the stochastic nature of evolution and validation acknowledged in the Limitations, please state the number of runs and whether the ordering in Table 4 is stable.
- [Abstract and §4.1] The phrase 'lightly human-edited' is subjective; the paper lists exactly three concrete edits per problem. Consider replacing 'lightly' with a more precise description.
Circularity Check
No significant circularity: the central results are tested against external baselines and the paper is transparent about human edits and starting points; only a minor in-house framework reuse is present.
full rationale
I examined the paper's derivation chain for each claimed result. The hexagon-packing SOTA claims (HEX 11, 12, 15, 16) are empirical outputs of the ImprovEvolve validation scheme and are compared against external prior results (Friedman's human packings and AlphaEvolve's published values), not against quantities derived from the method itself. The ablation study (Table 4) varies Stage A, Stage B, and LLM-editing independently, so the contribution of the basin-hopping decomposition is not established by construction. The ACI2 +E result is explicitly labeled as human-edited and as starting from AlphaEvolve's published solution, with the human changes itemized in Appendix B.2.1; it is therefore a transparently reported refinement of an external input, not a hidden renaming of the input as a fresh prediction. The GigaEvo framework is cited as the backbone and baseline, and two of the three co-authors are GigaEvo contributors; however, GigaEvo is open-source and the baseline comparison is executed in the paper rather than imported by citation, so this self-citation is not load-bearing. The main non-circularity caveat is a correctness risk rather than a circularity: the paper relaxes the validity requirement during final validation (Section 4, 'this requirement is relaxed during the final validation'), and no independent feasibility certificate is given for the reported hexagon L-values. That concern affects trust in the numerical claims, but it does not make the derivation circular. Overall, no step reduces by definition or by self-citation to its own input, so the circularity score is low.
Axiom & Free-Parameter Ledger
free parameters (4)
- Basin-hopping validation schedule (K, M, R, sigma) =
HEX: K=10, M=11, R=15, sigma in [1e-3, 1e2]; ACI2: K=3, M=6, R=5, sigma in [1e-3, 1e2]
- MAP-Elites archive behavior bins =
150
- ACI2 +E human-edit constants =
resolutions 100k to 1.6M, beta=20000, iters=10000, no lower clipping
- HEX +E human-edit constants =
SLSQP optimizer, softened penalties, unbounded bounds, maxiter=10000
axioms (5)
- domain assumption Repeated perturb()+improve() from a small number of seeds with T=0 acceptance finds near-global optima
- domain assumption The fitness function and the evolved penalty-based optimizers certify feasible packings and faithful ACI2 discretizations
- domain assumption LLM stochasticity and the single best run are representative
- ad hoc to paper The human edits in +E are 'minor' and do not replace the algorithmic contribution
- domain assumption Cited baselines (AlphaEvolve records, Friedman packing table) are correct
Cite this review
Pith. "Pith review of ImprovEvolve: Basin-Hopping Meets LLM-Guided Evolutionary Search." pith.science (2026). https://pith.science/paper/SO7I7LWV
@misc{pith2026260210233,
author = {Pith},
title = {Pith review of: ImprovEvolve: Basin-Hopping Meets LLM-Guided Evolutionary Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/SO7I7LWV}},
note = {Machine review of arXiv:2602.10233}
}
read the original abstract
LLM-guided evolutionary computation, most notably AlphaEvolve, has been remarkably successful in discovering novel mathematical constructions by solving challenging optimization problems. The standard approach is to evolve a monolithic program that directly outputs a candidate solution. We present ImprovEvolve, an algorithmic alternative that drastically reduces cognitive load on the LLM. Instead of prompting the model for an end-to-end optimizer, we evolve a program with three specialized operators of initialization, local improvement, and perturbation. We then approach the optimum by iteratively applying local improvements and intensity-scheduled perturbations, effectively driving a basin-hopping search with LLM-evolved subroutines. For hexagon in hexagon packing, ImprovEvolve discovers new state-of-the-art packings of 11, 12, 15, and 16 hexagons, and additionally for 14, 17, and 23 hexagons after minimal expert tuning of the generated code. For the second autocorrelation inequality, the evolved and human-scaled program pushes the lower bound from 0.96102 to 0.96258. For spherical codes, the ImprovEvolve program lowers the best-known maximum cosine for the majority of 90 randomly chosen diverse state-of-the-art spherical codes, achieving relative improvements of up to 2.4%.
Figures
Forward citations
Cited by 1 Pith paper
-
From I/O to Code with Discovery Agent
DIO-Agent frames IO2Code as LLM-driven evolutionary search over programs with a Transformation Priority Premise to favor simple hypotheses, outperforming baselines on a new IO2CodeBench.
Reference graph
Works this paper leans on
-
[1]
Henrique Assumpção, Diego Ferreira, Leandro Campos, and Fabricio Murai. 2026. CodeEvolve: an open source evolutionary coding agent for algorithm discovery and optimization. arXiv:2510.14150 [cs.AI] https://arxiv.org/abs/2510.14150
Pith/arXiv arXiv 2026
-
[2]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)
Pith/arXiv arXiv 2025
-
[3]
Erich Friedman. 2015. Erich’s Packing Center. https://erich-friedman.github.io/ packing/. Accessed: 2026-06-02
2015
-
[4]
Mikhail Ganzhinov. 2022. Highly symmetric lines. arXiv:2207.08266 [math.FA] https://arxiv.org/abs/2207.08266
Pith/arXiv arXiv 2022
-
[5]
Bogdan Georgiev, Javier Gómez-Serrano, Terence Tao, and Adam Zsolt Wag- ner. 2025. Mathematical exploration and discovery at scale.arXiv preprint arXiv:2511.02864(2025)
Pith/arXiv arXiv 2025
-
[6]
Yonatan Gideoni, Yujin Tang, Sebastian Risi, and Yarin Gal. 2025. Random Baselines for Simple Code Problems are Competitive with Code Evolution. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code. https://openreview. net/forum?id=rbVIpbmbTc
2025
-
[7]
Aaron Jaech and Alan Joseph. 2025. Further Improvements to the Lower Bound for an Autoconvolution Inequality. arXiv:2508.02803 [math.CA] https://arxiv. org/abs/2508.02803
Pith/arXiv arXiv 2025
-
[8]
Valentin Khrulkov, Andrey Galichin, Denis Bashkirov, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Andrey Kuznetsov, and Ivan Oseledets. 2025. GigaEvo: An Open Source Optimization Framework Powered By LLMs And Evolution Algorithms. arXiv:2511.17592 [cs.NE] https://arxiv.org/abs/2511.17592
arXiv 2025
-
[9]
Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. 2025. ShinkaE- volve: Towards Open-Ended And Sample-Efficient Program Evolution. arXiv:2509.19349 [cs.CL] https://arxiv.org/abs/2509.19349
Pith/arXiv arXiv 2025
-
[10]
Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites. arXiv:1504.04909 [cs.AI] https://arxiv.org/abs/1504.04909
Pith/arXiv arXiv 2015
-
[11]
Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131(2025)
Pith/arXiv arXiv 2025
-
[12]
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models.Nature625, 7995 (2024), 468–475
2024
-
[13]
2025.OpenEvolve: an open-source evolutionary coding agent
Asankhaya Sharma. 2025.OpenEvolve: an open-source evolutionary coding agent. https://github.com/algorithmicsuperintelligence/openevolve
2025
-
[14]
David J. Wales and Jonathan P. K. Doye. 1997. Global Optimization by Basin- Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Contain- ing up to 110 Atoms.The Journal of Physical Chemistry A101, 28 (July 1997), 5111–5116. doi:10.1021/jp970984n
-
[15]
Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, Hao Cheng, Pengcheng He, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. 2025. ThetaEvolve: Test-time Learning on Open Problems. arXiv:2511.23473 [cs.LG] https://arxiv.org/abs/2511.23473
Pith/arXiv arXiv 2025
-
[16]
Chi, Wang-Cheng Kang, Derek Zhiyuan Cheng, and Beidou Wang
Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, Chi Wang, Ed H. Chi, Wang-Cheng Kang, Derek Zhiyuan Cheng, and Beidou Wang. 2026. PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution. arXiv:2601.10657 [cs.NE] https://arxiv.org/abs/2601.10657
arXiv 2026
-
[17]
Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, et al. 2026. Learning to Discover at Test Time.arXiv preprint arXiv:2601.16175(2026). KDD ’26, Under review, Jeju, Korea Kravatskiy et al. A Hexagon Packing Problem: Task Description and Evolved Solution A.1 Prompts ...
Pith/arXiv arXiv 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.