REVIEW 3 major objections 4 minor 12 references
FunL2O: LLM-Guided Feature Function Design for Learning to Optimize
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Feature functions in learning-to-optimize can be discovered by LLM-guided evolution, and the evolved representations consistently outperform hand-crafted ones.
desk verdict A carefully executed L2O feature-search framework with strong controls, but the 'consistently outperform across four LLMs' claim overshoots what the per-model tables show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the 'semantic feature contract' plus a measured-outcome evolutionary loop. The contract defines, for each pipeline, the allowed input fields, required output shape and width, invariants (determinism, finiteness, required hand-crafted channels), and an executable validator; it turns the open-ended space of possible feature programs into an admissible set that a constrained LLM can propose within. Every valid candidate is then evaluated the expensive way: the original learning-to-optimize model is retrained with that feature function and scored by the pipeline's native metric, ordered lexicographically (feasibility first, then violation, then objective or solver work)
What would settle it
An independent replication that retrains every pipeline with a matched-width random-feature function (same seeds, budget, and evaluation protocol) across all eight hosts; if the random-feature baseline matches or beats the evolved function on the held-out test split, the claim that evolved feature content drives the gains is falsified. The paper already runs this control on six of the LP and solver-facing settings; extending it to the remaining tasks would settle the question.
Extended reading notes
Core claim
The paper's central claim is that the feature function is the overlooked variable in learning-to-optimize, and that an LLM-guided evolutionary loop can find better feature functions than the ones experts write by hand. In FunL2O, a large language model proposes candidate Python feature functions subject to a 'semantic contract' that fixes what information is available and what tensor shape must be returned; valid candidates replace the hand-crafted features only, the original model is retrained with the same data, loss, and schedule, and the resulting optimization outcome—feasibility, gap, iterations, pivots, or wall time—is the fitness signal. The search keeps the best programs as elites an
Load-bearing premise
The load-bearing premise is that resizing the fixed model's input layer to accommodate a different feature width is an inert 'mechanical' change; if that resize changes learning dynamics in any meaningful way, the claim that only the feature function changed—and therefore that measured gains are due to feature content—loses force.
Editorial extensions
If this is right
- Feature-function design becomes a separable, automatable step in learned optimization, so future pipelines can search over representations instead of inheriting fixed ones.
- Because the returned feature functions are ordinary executable preprocessing code, they can be audited and reused; they also double as interpretable hypotheses about which problem structure (cost relative to coverage, directed mass asymmetry, normalized scales) matters.
- The matched-width random-feature controls suggest that at least part of the gain comes from semantic content rather than simply feeding more numbers into a larger input layer.
- The evolutionary-feedback comparison (independent sampling versus elite-guided search) shows that measured feedback itself contributes beyond the best-of-many effect.
- Stronger LLMs produce successful feature functions more reliably, but the gain is not tied to a single provider since several different LLMs yield improvements.
Reading between the lines
- The paper treats resizing the model's input layer to match a new feature width as a 'mechanical interface change'; a testable inference is that some of the measured gains could come from the wider input layer altering optimization dynamics, which the random-width controls only partially rule out.
- A natural extension the authors do not explore: the same contract-and-evolve loop could be applied to other pipeline components (loss functions, training schedules, solver interfaces) or to generating entirely new contracts, though feature functions are the most portable component.
- Because the search cost is offline and the deployed feature function is LLM-free, the economic case strengthens as the number of future instances grows; in repeated-optimization settings, paying 48 retraining runs may be worthwhile even for modest per-instance gains.
- The representative programs suggest transferable design patterns—normalizing by coverage, exposing directed mass, adding scale-invariant views—that could seed hand-designed features in problem classes beyond those tested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FunL2O proposes an LLM-driven program-evolution framework that searches over executable feature functions for learning-to-optimize (L2O) pipelines. The feature function is replaced while the model architecture, training procedure, data, solver, and downstream evaluation metric remain fixed; each candidate is validated by a semantic contract, the host pipeline is retrained, and the downstream optimization outcome is used as the fitness signal. The method is evaluated on eight L2O pipelines spanning LP solution prediction, constrained QP/QCQP/SOCP prediction, primal-dual warm-starting, simplex basis and pivot decisions, and MILP Predict-and-Search and learned-backdoor branching, using up to four LLM proposers. The paper claims that evolved features consistently outperform handcrafted representations across all tasks and LLMs, and that the gains are attributable to feature content rather than width expansion or best-of-many selection, supported by equal-budget and matched-width control experiments.
Significance. If the central claim is properly supported, the paper makes a valuable contribution: it introduces a general and automatable approach to a previously fixed component of L2O pipelines, with deployment requiring no LLM. The strengths are the clean separation between LLM proposal and external validation (no circularity), seed-preserving selection that prevents forced replacement, the equal-budget control on PDHG that isolates evolutionary feedback from best-of-48 sampling, and the matched-width random controls for six settings. The paper provides a reproducible protocol and a source artifact. However, the headline claim of 'consistent' improvement across four LLMs is contradicted by the paper's own per-model tables, and the isolation of feature content from width changes is only partially controlled. These issues are fixable by narrowing the claims and strengthening the statistical presentation.
major comments (3)
- [Proposal-model dependence; Tables 7-11] The abstract states: 'Across continuous and discrete optimization tasks and four LLMs, the evolved features consistently outperform hand-crafted representations.' The paper's own per-model results contradict this. The 'Proposal-model dependence' paragraph reports improvements in 15, 17, 17, and 6 cells for Claude, GPT-5.5, Gemini, and DeepSeek, respectively. Appendix Tables 7-11 show that DeepSeek improves none of the 12 FSNet tasks and none of the 3 DC3 tasks, and ties on 14 of 20 shared continuous cells. Even for the stronger proposers, there are many equality cases (e.g., Table 8, Claude on nonconvex QP, nonsmooth QP, and nonsmooth QCQP). The conclusion 'evolved features improve solution prediction ... without changing the underlying models or solvers' is therefore too broad. The claim should be narrowed to 'FunL2O with capable LLM proposers reliably improves L2O features, while weake
- [Methodology, Eq. (6); Controlled Analyses 'Matched-Width Features'] The claim that measured gains are due to feature content rather than the 'mechanical interface change' of resizing the input layer (Eq. 6) is load-bearing. The matched-width random controls in Table 14 cover only six settings (five IPM-MPNN distributions and PDHG-Net). For FSNet, DC3, Smart Initial Basis, Learning to Pivot, and both MILP hosts, no width-matched control is provided. If resizing alters learning dynamics (capacity, gradient flow, effective regularization), then 'without changing the underlying models' is not established for those hosts. The paper should either extend the matched-width controls to all hosts where the interface permits, or explicitly state that the isolation claim is supported only in the controlled settings.
- [Metrics and Aggregation; Tables 2, 3, 7-11] All reported comparisons are means of three seed-paired repetitions, but no variance, confidence intervals, or significance tests are provided. Several improvements are tiny (DC3: 2.0-4.3% gap reduction; Smart Initial Basis: +1.62 percentage points iteration saving) and could be within run-to-run noise. Additionally, the MILP summary in Table 3 and the text state that 'the best FunL2O variant outperforms the handcrafted features on both metrics across all three domains.' This is an oracle selection across the two proposal-model runs; for example, FunL2O-GPT on Predict-and-Search CA has a final primal gap of 1.023% versus 0.423% for handcrafted, i.e., worse. The 'evolved features improve MILP search' conclusion is supported only by cherry-picking the better variant per row, not by either variant individually. Please report per-variant results without best-of-selection and add uncertainty
minor comments (4)
- [Table 2, Feasibility column] The feasibility column reports values such as '0.957→0.942' (set cover) and '0.988→0.941' (facility location), which are decreases in a higher-is-better metric, yet the table is otherwise framed as improvement-oriented. Clarify the direction or mark these as decreases; the current presentation is ambiguous.
- [Abstract] The phrase 'the first unified framework' is a strong novelty claim. Consider softening to 'a unified framework' unless a systematic prior-art search has been performed; this would not weaken the contribution.
- [Prompt and Proposal-Model Configuration] The paper notes that temperature and top-p are not overridden and follow provider defaults. This is a reproducibility concern; please record the actual sampling parameters or state clearly that default values are part of the configuration.
- [Controlled Analyses] The equal-budget control is described only for PDHG-Net. The text in the appendix says 'The available controls test the two principal alternative explanations' and 'These controls provide convergent evidence,' but the scope of the controls is limited to one host for feedback and six for width. Consider adding a sentence in the main text that these mechanism-level controls are representative, not universal.
Circularity Check
No circular derivation; search is externally evaluated and self-citations are not load-bearing.
full rationale
FunL2O's loop is an empirical program search, not a derivation. Candidate feature functions are proposed by an LLM and filtered by a semantic contract; fitness is assigned by retraining the unchanged host L2O pipeline and measuring native validation metrics (Eqs. 6–8, 15–18). The proposal model does not see validation instances or test outcomes and does not assign fitness. Because selection and reporting are separated by disjoint train/validation/test splits, the reported test improvements are not tautological consequences of the ranking procedure. The seed-preservation rule (handcrafted function remains eligible and is returned if no candidate improves) does mean that on the validation ranking any returned non-seed program is better by construction, but the displayed results are test-set measurements, so the central performance claim is empirical rather than definitional. The only candidate circularity-adjacent issue is the 'mechanical interface change' when input width changes; that is an untested architectural-confounding assumption, not a reduction of the result to its inputs. Matched-width random controls partially address it. Self-citations (e.g., Cai, Huang, and Dilkina 2024) appear as host baselines and related work, not as load-bearing uniqueness theorems or forbidden-alternative arguments. The abstract's 'consistently outperform' wording overstates the paper's own per-LLM tables (DeepSeek improves only 6 of 20 shared cells), but that is a correctness/evidence concern, not circularity.
Assumptions & free parameters
free parameters (6)
- search_generations =
8
- proposal_slots_per_generation =
6
- elite_count =
2
- max_repairs =
3
- feature_width_caps =
K<=64 for DC3, K<=32 for SIB/LTP; others pipeline-specific
- LLM_sampling_parameters =
provider defaults (temperature/top-p not overridden)
assumptions (4)
- domain assumption The semantic contract and validator prevent the LLM and generated code from accessing reference solutions or labels during search.
- domain assumption Validation performance is a reliable proxy for test performance when selecting feature functions.
- domain assumption The host L2O pipelines are fixed and the input-layer resize is the only architectural change.
- domain assumption The LLM proposer behaves as intended, generating reasonably diverse and syntactically valid feature programs from the prompt.
Cite this review
Pith. "Pith review of FunL2O: LLM-Guided Feature Function Design for Learning to Optimize." pith.science (2026). https://pith.science/paper/YOWAHYO4
@misc{pith2026260727389,
author = {Pith},
title = {Pith review of: FunL2O: LLM-Guided Feature Function Design for Learning to Optimize},
year = {2026},
howpublished = {\url{https://pith.science/paper/YOWAHYO4}},
note = {Machine review of arXiv:2607.27389}
}
read the original abstract
Learning-to-optimize (L2O) methods accelerate repeated optimization by training models to predict solutions, warm starts, branching decisions, or other forms of solver guidance. A critical yet largely overlooked component of these pipelines is the feature function that maps problem instances to inputs for machine learning models. Existing L2O methods typically rely on hand-crafted features, making representation design manual and largely fixed across domains. We introduce FunL2O, the first unified framework for automating feature design through LLM-driven program evolution for L2O. In a FunSearch-style loop, an LLM proposes executable feature functions, while a fixed evaluation process retrains the original L2O model and measures downstream optimization performance. We evaluate FunL2O on linear and quadratic programming tasks involving solution prediction and warm-starting, as well as on mixed-integer optimization tasks using GNN-guided backdoor branching and Predict-and-Search. Across continuous and discrete optimization tasks and four LLMs, the evolved features consistently outperform hand-crafted representations. These results establish LLM-driven feature evolution as a general and effective approach to automating representation design in L2O.
Figures
Reference graph
Works this paper leans on
-
[1]
Table 11: Simplex-oriented host results, averaged over three independent, seed-paired repetitions
/ (col_deg + eps)) cons = torch.cat([cons_seed, torch.log1p(row_deg), torch.log1p(row_norm), cover_ratio], dim=1) vals = torch.cat([vals_seed, torch.log1p(col_deg), torch.log1p(col_norm), obj_eff], dim=1) The program exposes column coverage, row load, coefficient scale, and objective cost relative to coverage while preserving the handcrafted mean and stan...
arXiv 1947
-
[4]
Multi-task Repre- sentation Learning for Mixed Integer Linear Programming. InInternationalConferenceontheIntegrationofConstraint Programming, Artificial Intelligence, and Operations Re- search, 134–151. Cai, J.; Huang, W.; Long, B.; Cleaveland, M.; Deshmukh, J.V.;Lindemann,L.;andDilkina,B.2025a.Neuro-symbolic accelerationofmilpmotionplanningwithtemporallo...
-
[7]
InForty-first International Conference on Machine Learning
Contrastive predict-and-search for mixed integer linear programs. InForty-first International Conference on Machine Learning. Li,B.;Yang,L.;Chen,Y.;Wang,S.;Mao,H.;Chen,Q.;Ma, Y.;Wang,A.;Ding,T.;Tang,J.;etal.2024. PDHG-unrolled learning-to-optimize method for large-scale linear program- ming. InProceedings of the 41st International Conference on Machine Le...
2024
-
[9]
Alphaevolve: A coding agentforscientificandalgorithmicdiscovery.arXivpreprint arXiv:2506.13131. OpenAI
-
[10]
Sun,J.;Zhang,L.;Chen,G.;Xu,P.;Zhang,K.;andYang,Y
Deepseekmath-v2: To- wardsself-verifiablemathematicalreasoning.arXivpreprint arXiv:2511.22570. Sun,J.;Zhang,L.;Chen,G.;Xu,P.;Zhang,K.;andYang,Y
-
[29]
anthropic.com/news/claude-opus-4-8
Anthropic.2026.IntroducingClaudeOpus4.8.https://www. anthropic.com/news/claude-opus-4-8. Cai, J.; Huang, T.; and Dilkina, B
2026
-
[32]
Gemini3.1Pro
GoogleDeepMind.2026. Gemini3.1Pro. https://deepmind. google/models/model-cards/gemini-3-1-pro/. Han, Q.; Yang, L.; Chen, Q.; Zhou, X.; Zhang, D.; Wang, A.;Sun,R.;andLuo,X.2023. AGNN-GuidedPredict-and- Search Framework for Mixed-Integer Linear Programming. InThe Eleventh International Conference on Learning Rep- resentations. Harrison,J.;Metz,L.;andSohl-Di...
2026
-
[1987]
Version, 12(1987-2018):
Ibm ilog cplex optimization studio. Version, 12(1987-2018):
1987
Show all 12 references
-
[2022]
Chen,T.;Zhang,W.;Jingyang,Z.;Chang,S.;Liu,S.;Amini, L.;andWang,Z.2020
Learning to optimize: A primer and a benchmark.Journal of Machine Learning Research, 23(189): 1–59. Chen,T.;Zhang,W.;Jingyang,Z.;Chang,S.;Liu,S.;Amini, L.;andWang,Z.2020. Trainingstrongerbaselinesforlearn- ingtooptimize.AdvancesinNeuralInformationProcessing Systems, 33: 7332–7...
2020
-
[2023]
Ye, H.; Wang, J.; Cao, Z.; Berto, F.; Hua, C.; Kim, H.; Park, J.; and Song, G
M-l2o: Towards generalizable learning-to-optimize by test- time fast self-adaptation.arXiv preprint arXiv:2303.00039. Ye, H.; Wang, J.; Cao, Z.; Berto, F.; Hua, C.; Kim, H.; Park, J.; and Song, G
-
[2024]
InECAI 2024, 2418–2425
Learning Back- doors for Mixed Integer Linear Programs with Contrastive Learning. InECAI 2024, 2418–2425. IOS Press. Cai, J.; Huang, T.; and Dilkina, B
2024
-
[2025]
Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M
Llm-fe: Automated feature engineering for tabular data with llms as evolutionary optimizers.arXiv preprint arXiv:2503.14434. Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M. W.; Pfau,D.;Schaul,T.;Shillingford,B.;andDeFreitas,N.2016. Learning to learn by gradient descent by ...
2016 arXiv
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.