REVIEW 3 major objections 3 minor 40 references
An LLM-powered evolutionary loop writes new table features and improves classification in most tested cases.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 09:43 UTC pith:A2MWAFDF
load-bearing objection Honest, clearly-written funsearch-for-features paper with a real idea but empirical claims weakened by best-of-three selection over three runs without error bars. the 3 major comments →
Feature Generation Using LLMs: An Evolutionary Algorithm Approach
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a relatively small local language model can serve as the variation operator in an evolutionary feature-construction loop. Given only a short prompt listing column names, dtypes, and value ranges, the model returns a pandas-compatible Python expression that creates one new attribute. The new attribute is appended, the dataset's classification F1 is recomputed, and a fitness-proportional sampler retains the best-performing feature sets while still exploring lower-scoring ones. The paper reports that this loop improves F1 over the raw-feature baseline in 13 of 16 algorithm-dataset combinations, with notable gains on Monk-2 (+19.29 for decision trees) and Car (+9.13 for
What carries the argument
The engine of the approach is an evolutionary loop in which the LLM acts as the mutation operator: a prompt listing the current attributes' dtypes and possible value ranges asks the model to emit exactly one new attribute as a pandas-compatible Python expression; the expression is applied, F1 is recomputed, and a fitness-proportional sampler keeps the best-performing feature sets while retaining some lower-scoring ones to avoid local optima. Three prompt settings are tested: keeping generated features in the prompt (Agg), ignoring them (Skip), and anonymizing feature names and values (Anon).
Load-bearing premise
The load-bearing premise is that the reported F1 gains are real signal rather than an artifact of picking the best of three prompt settings across just three stochastic runs, with no confidence intervals or significance tests reported.
What would settle it
Run the pipeline many times under a pre-registered single prompt setting and compare the F1 improvement over raw features against run-to-run variance; if the average gain disappears or falls within the noise, the broad improvement claim fails. Separately, record the share of LLM outputs that are unparseable or inapplicable pandas expressions — if most queries return malformed code, the selection loop is mostly evaluating noise.
If this is right
- Feature construction can be automated without a predefined operator vocabulary: the LLM invents expressions over any number of base features, including ternary relations like the one in Monk-2.
- Even with fully anonymized feature names, generated features improve classification on several datasets, so the method does not depend on the model's semantic knowledge of the columns.
- A model small enough to run on a single workstation (7B parameters) suffices to drive the search, lowering the hardware barrier for automated feature engineering.
- Early stopping after 20 non-improving queries keeps runtime bounded; without it, generated expressions become longer and more likely to fail Python parsing.
- The largest gains occur on datasets where the target depends on multi-attribute logical relations, suggesting the approach is most valuable precisely where hand-coded feature combinations are impractical.
Where Pith is reading between the lines
- A testable extension: because the reported change uses the best of three prompt settings across only three stochastic runs, the true per-setting gain is likely smaller than the headline numbers; a pre-registered single-setting replication would separate signal from selection noise.
- The same loop should transfer to regression or other tabular tasks, since the mutation operator only needs an evaluator score; using R-squared as fitness is a natural follow-up.
- The operator-frequency analysis suggests the model's code-generation prior is biased toward arithmetic and numpy idioms; explicitly constraining or diversifying the allowed operator set could test whether that bias helps or hurts the search.
- The anonymized-setting gains hint that the approach may work as a dataset-agnostic feature constructor, potentially complementing conventional automated feature-engineering algorithms rather than replacing them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an evolutionary feature-generation pipeline that prompts LLaMA-3.1 7B to write pandas expressions, evaluates candidate feature additions by F1 on a downstream classifier, and maintains a population using fitness-proportional selection. Three prompt settings are compared: aggregating generated features into the prompt (Agg), skipping generated features (Skip), and anonymizing features (Anon). On eight datasets and two classifiers, Table 2 reports F1 improvements over raw features in 13 of 16 dataset-algorithm combinations, with particularly large gains on Monk-2 and Car. The paper interprets these results as evidence that LLM-generated mathematical/logical features can improve classification performance, including in settings where the LLM is given no semantic feature information.
Significance. If the reported improvements are robust, the paper makes a useful contribution: it shows that a modest-sized local LLM, combined with an evolutionary selection loop, can construct transparent tabular features without a predefined operator library. The anonymized setting is a genuinely interesting control for LLM prior knowledge, and the released code supports reproducibility of the framework. However, the empirical basis for the central claim is currently not strong enough. The headline '13 of 16' result relies on selecting the best of three pipeline settings per cell, with only three stochastic runs and no confidence intervals or significance tests. Several reported gains are within plausible run-to-run noise, and the parse-failure rate of generated code is acknowledged but never quantified. The idea is valuable, but the evidence needs substantial strengthening before the claim is established.
major comments (3)
- [§4.2, Table 2] The headline result ('13 of 16 experiments') and every 'change' entry are computed as the best-performing setting among Agg/Skip/Anon compared with Raw. Taking the maximum over three pipeline variants for each dataset/classifier biases the comparison: under a null in which every variant equals Raw, the maximum of three independent noisy F1 estimates is positive with probability about 0.875. The paper reports no standard deviations or confidence intervals, and each entry is an average of only three stochastic runs. Several reported changes (Car DT +0.52, Stu DT +0.47, Wine DT +0.39, Ion MLP −0.02) are of the size of plausible run-to-run variation. The Table 2 caption ('best-performing setting compared to the baseline') is also ambiguous about whether the setting was selected on the development set or after inspecting test results. Please report each setting separately with confidence inte
- [§5.2, footnote 4] The manuscript acknowledges that generated expressions frequently fail during Python parsing, especially in the Agg setting, but it never quantifies the parse-success rate. If a large fraction of mutations produce invalid code, the evolutionary loop is frequently evaluating unchanged or discarded samples, and the algorithm may effectively be selecting among a small number of parseable expressions. This is load-bearing for the claim that the LLM can generate useful features. Please report per-setting and per-dataset parse-success rates, describe how failed features are handled in Algorithm 1, and show that the final reported improvements are not driven by a few lucky valid expressions.
- [§4.2 (comparison baseline)] The abstract and conclusion attribute the improvements to the LLM's generation ability, but no non-LLM feature-generation baseline is provided. A simple baseline that generates random arithmetic/logical expressions over the same columns with the same operator set would clarify whether the gains come from the language model's prior or from the evolutionary search procedure itself. Without such a comparison, the current results only show that the pipeline can improve over Raw features; they do not yet demonstrate the specific contribution of the LLM.
minor comments (3)
- [§3.1, Algorithm 1] The pseudocode calls getPrompt and askLLM but these functions are only informally described later. A forward reference or a one-line definition near Algorithm 1 would improve readability.
- [Table 2] For the three anonymous datasets at the bottom of the table, the Anon column is marked with '-'. A sentence explaining whether Anon coincides with Skip for these datasets, and why no separate entry is reported, would prevent confusion.
- [§5.2] The Mann-Whitney U tests reported there compare the number of operators per generated function between Agg and Skip; they do not speak to the primary claim about F1 differences. If these are intended as secondary analyses, say so explicitly; otherwise, add significance tests on the F1 values.
Circularity Check
No circularity found: the headline claim is an empirical test-set measurement, independent of the generation and selection inputs.
full rationale
The paper's derivation chain is: raw features -> prompt (Fig. 1) -> LLM-generated pandas expressions -> evolutionary selection of attribute sets using development-set F1 (Algorithm 1, Section 3.1) -> final test-set F1 (Section 4.2). The headline claim ('in most cases, the language model can produce new features ... and improve classification results') is an empirical statement supported by test-set measurements in Table 2. Nothing in the paper defines the output quantity in terms of an input quantity: no equation is fitted to the test labels and then renamed a prediction; the pipeline's selection uses only the development split ('we optimize the pipeline on the development set and report the results on the test set'). The 'change' column is a summary of the best of the three reported settings, not a fitted constant that forces the outcome. There are no load-bearing self-citations: the cited works (funsearch [30], LLM feature engineering [12, 40]) are external prior work, and no uniqueness theorem or ansatz is imported from the authors' own publications. The manuscript's own caveats — malformed functions failing during parsing (Section 5.2, footnote 4) and reproducibility difficulty (Section 7) — are robustness and validity limitations, not circularity. A skeptical statistical concern (best-of-three setting selection over three stochastic runs with no error bars) is a multiple-comparison/noise issue, not a definitional reduction, and should be handled as correctness risk rather than circularity. Hence score 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- Number of iterations n =
400
- Maximum sample count M =
not reported
- Early-stopping patience =
20 queries
- Classifier hyperparameters (MLP, DT) =
not reported, scikit-learn defaults assumed
axioms (4)
- ad hoc to paper Generated Python expressions are directly parseable and applicable to a pandas DataFrame.
- domain assumption F1 on the development split is an adequate signal for selecting features that improve test F1.
- domain assumption The best-performing pipeline setting (Agg, Skip, or Anon) per dataset is a legitimate representation of the method.
- standard math scikit-learn's MLP and DecisionTree implementations and the F1 measure are correct standard tools.
read the original abstract
A crucial step in machine learning pipelines is to present each entity with features or attributes that are representative of the characteristics of the processed entities. Feature engineering is an important step in finding a relation among attributes that otherwise may not be processed by the ML algorithms. Meanwhile, Large Language Models have shown promising abilities in coding, mathematical reasoning, and processing world knowledge. In this work, we utilize an LLM for the problem of feature generation from tabular data based on the previously given features. We have created a pipeline that takes a set of attributes and a prompt to generate new features. Then, our selection algorithm selects the best-performing sets of attributes. We apply our method to eight datasets from different domains and data types. Our results show that, in most cases, the language model can produce new features based on mathematical and logical operators that are useful for the given tasks and can improve classification results.
Figures
Reference graph
Works this paper leans on
-
[1]
UCI Machine Learning Repository (1992), DOI: https://doi.org/10.24432/C5PC7J
Aeberhard, S., Forina, M.: Wine. UCI Machine Learning Repository (1992), DOI: https://doi.org/10.24432/C5PC7J
doi:10.24432/c5pc7j 1992
-
[2]
arXiv preprint arXiv:2402.00157 (2024)
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., Yin, W.: Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157 (2024)
Pith/arXiv arXiv 2024
-
[3]
Princeton University Press, Princeton (1957)
Bellman, R.: Dynamic programming. Princeton University Press, Princeton (1957)
1957
-
[4]
Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(8), 1798–1828 (2013).https://doi.org/10.1109/TPAMI.2013.50
-
[5]
UCI Machine Learning Repository (1997), DOI: https://doi.org/10.24432/C5JP48
Bohanec, M.: Car Evaluation. UCI Machine Learning Repository (1997), DOI: https://doi.org/10.24432/C5JP48
doi:10.24432/c5jp48 1997
-
[6]
ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)
2024
-
[7]
In: Proceedings of the fourteenth international conference on ar- tificial intelligence and statistics
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference on ar- tificial intelligence and statistics. pp. 215–223. JMLR Workshop and Conference Proceedings (2011)
2011
-
[8]
Digital4(1), 114–125 (2024)
Coello, C.E.A., Alimam, M.N., Kouatly, R.: Effectiveness of chatgpt in coding: a comparative analysis of popular large language models. Digital4(1), 114–125 (2024)
2024
-
[9]
Intelligent data analysis1(1- 4), 131–156 (1997)
Dash, M., Liu, H.: Feature selection for classification. Intelligent data analysis1(1- 4), 131–156 (1997)
1997
-
[10]
Infor- mation Sciences189, 176–190 (2012)
Dor, O., Reich, Y.: Strengthening learning algorithms by feature discovery. Infor- mation Sciences189, 176–190 (2012)
2012
-
[11]
Dubey, A., et al.: The llama 3 herd of models (2024),https://arxiv.org/abs/ 2407.21783
Pith/arXiv arXiv 2024
-
[12]
In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F
Han, S., Yoon, J., Arik, S.O., Pfister, T.: Large language models can automatically engineer features for few-shot tabular learning. In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F. (eds.) Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. ...
2024
-
[13]
In: SoutheastCon 2016
Heaton, J.: An empirical analysis of feature engineering for predictive modeling. In: SoutheastCon 2016. pp. 1–6. IEEE (2016)
2016
-
[14]
arXiv preprint arXiv:2407.02694 (2024)
Jeong, D.P., Lipton, Z.C., Ravikumar, P.: Llm-select: Feature selection with large language models. arXiv preprint arXiv:2407.02694 (2024)
Pith/arXiv arXiv 2024
-
[15]
Cureus15(5) (2023)
Karabacak, M., Margetis, K.: Embracing large language models for medical appli- cations: opportunities and challenges. Cureus15(5) (2023)
2023
-
[16]
In: 2016 IEEE 16th international conference on data mining (ICDM)
Katz, G., Shin, E.C.R., Song, D.: Explorekit: Automatic feature generation and selection. In: 2016 IEEE 16th international conference on data mining (ICDM). pp. 979–984. IEEE (2016)
2016
-
[17]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Khurana, U., Samulowitz, H., Turaga, D.: Feature engineering for predictive mod- eling using reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
2018
-
[18]
In: 2016 IEEE 16th international con- ference on data mining workshops (ICDMW)
Khurana, U., Turaga, D., Samulowitz, H., Parthasrathy, S.: Cognito: Automated feature engineering for supervised learning. In: 2016 IEEE 16th international con- ference on data mining workshops (ICDMW). pp. 1304–1307. IEEE (2016)
2016
-
[19]
arXiv preprint arXiv:2408.12025 (2024)
Li, D., Tan, Z., Liu, H.: Exploring large language models for feature selection: A data-centric perspective. arXiv preprint arXiv:2408.12025 (2024)
Pith/arXiv arXiv 2024
-
[20]
In: Proceedings of the fourth ACM international conference on AI in finance
Li, Y., Wang, S., Ding, H., Chen, H.: Large language models in finance: A survey. In: Proceedings of the fourth ACM international conference on AI in finance. pp. 374–382 (2023)
2023
-
[21]
Advances in Neural Information Processing Systems36(2024)
Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems36(2024)
2024
-
[22]
UCI Machine Learning Repository (2011), DOI: https://doi.org/10.24432/C55C88
Liu, Z.: Amazon Commerce Reviews. UCI Machine Learning Repository (2011), DOI: https://doi.org/10.24432/C55C88
-
[23]
Machine Learning49, 59–98 (2002)
Markovitch, S., Rosenstein, D.: Feature generation using general constructor func- tions. Machine Learning49, 59–98 (2002)
2002
-
[24]
arXiv preprint arXiv:2302.12170 (2023)
Meyerson, E., Nelson, M.J., Bradley, H., Gaier, A., Moradi, A., Hoover, A.K., Lehman, J.: Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170 (2023)
arXiv 2023
-
[25]
Communica- tion of IICM (Institute of Information and Computing Machinery, Taiwan)5(67- 72), 2 (2002)
Motoda, H., Liu, H.: Feature selection, extraction and construction. Communica- tion of IICM (Institute of Information and Computing Machinery, Taiwan)5(67- 72), 2 (2002)
2002
-
[26]
UCI Machine Learning Repository (1995), DOI: https://doi.org/10.24432/C55C7W
Nash, Warwick, Sellers, Tracy, Talbot, Simon, Cawthorn, Andrew, Ford, Wes: Abalone. UCI Machine Learning Repository (1995), DOI: https://doi.org/10.24432/C55C7W
doi:10.24432/c55c7w 1995
-
[27]
ACM Journal of Data and Information Quality15(2), 1–21 (2023)
Navigli, R., Conia, S., Ross, B.: Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality15(2), 1–21 (2023)
2023
-
[28]
Journal of Machine Learning Research12, 2825–2830 (2011)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research12, 2825–2830 (2011)
2011
-
[29]
UCI Machine Learning Repository (2021), DOI: https://doi.org/10.24432/C5MC89
Realinho, Valentim, Martins, V., Mónica, Machado, Jorge, Baptista, Luís: Pre- dict Students’ Dropout and Academic Success. UCI Machine Learning Repository (2021), DOI: https://doi.org/10.24432/C5MC89
doi:10.24432/c5mc89 2021
-
[30]
Nature625(7995), 468–475 (2024) Feature Generation Using LLMs: An Evolutionary Algorithm Approach 17
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M.P., Dupont, E., Ruiz, F.J., Ellenberg, J.S., Wang, P., Fawzi, O., et al.: Mathematical discoveries from program search with large language models. Nature625(7995), 468–475 (2024) Feature Generation Using LLMs: An Evolutionary Algorithm Approach 17
2024
-
[31]
In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation
Shafti, L.S., Pérez, E.P.: Constructive induction and genetic algorithms for learning concepts with complex interaction. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation. p. 1811–1818. GECCO ’05, Associa- tion for Computing Machinery, New York, NY, USA (2005).https://doi.org/ 10.1145/1068009.1068317,https://doi-org.prox...
arXiv 2005
-
[32]
In: Machine Learning Proceedings 1991, pp
Sutton, R.S., Matheus, C.J.: Learning polynomial functions by feature construc- tion. In: Machine Learning Proceedings 1991, pp. 208–212. Elsevier (1991)
1991
-
[33]
Data classification: Algorithms and applications p
Tang, J., Alelyani, S., Liu, H.: Feature selection for classification: A review. Data classification: Algorithms and applications p. 37 (2014)
2014
-
[34]
pandas development team, T.: pandas-dev/pandas: Pandas (feb 2020).https:// doi.org/10.5281/zenodo.3509134,https://doi.org/10.5281/zenodo.3509134
-
[35]
Teboul, A.: Diabetes health indicators dataset (2022),https://www.kaggle.com/ datasets/alexteboul/diabetes-health-indicators-dataset
2022
-
[36]
UCI Machine Learning Repository (1989), DOI: https://doi.org/10.24432/C5W01B
V, S., S., W., L., H., K., B.: Ionosphere. UCI Machine Learning Repository (1989), DOI: https://doi.org/10.24432/C5W01B
doi:10.24432/c5w01b 1989
-
[37]
In: Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10)
Wang, Z.: Causalbench: A comprehensive benchmark for evaluating causal reason- ing capabilities of large language models. In: Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10). pp. 143–151 (2024)
2024
-
[38]
UCI Machine Learning Repository (1992), DOI: https://doi.org/10.24432/C5R30R
Wnek, J.: MONK’s Problems. UCI Machine Learning Repository (1992), DOI: https://doi.org/10.24432/C5R30R
doi:10.24432/c5r30r 1992
-
[39]
arXiv preprint arXiv:2401.10034 (2024)
Wu, X., Wu, S.h., Wu, J., Feng, L., Tan, K.C.: Evolutionary computation in the era of large language model: Survey and roadmap. arXiv preprint arXiv:2401.10034 (2024)
Pith/arXiv arXiv 2024
-
[40]
arXiv preprint arXiv:2406.03505 (2024)
Zhang, X., Zhang, J., Rekabdar, B., Zhou, Y., Wang, P., Liu, K.: Dynamic and adaptive feature generation with llm. arXiv preprint arXiv:2406.03505 (2024)
arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.