REVIEW 4 major objections 6 minor 1 cited by
Tabular Feature Discovery With Reasoning Type Exploration
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read REFEAT guides an LLM through six reasoning styles and adaptively picks the best one, improving tabular feature engineering across 59 datasets.
desk verdict Sensible new combination of reasoning prompts and bandit selection, but the headline gains rest on a single split with no error bars and the paper's own counts don't add up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a set of six reasoning-type meta-prompts together with an epsilon-greedy multi-armed bandit. Each meta-prompt asks the LLM to adopt one reasoning mode and to output executable pandas transformations; the bandit treats the six modes as arms, with reward equal to the improvement in validation accuracy (classification) or RMSE reduction (regression) when that feature is added to the original data. Over 20 iterations the exploration rate decays from 1 to 0, so the controller moves from trying all modes to exploiting the best one, and the top-K features by validation gain are then concatenated to the dataset for final model training.
What would settle it
Re-run the full pipeline on the 59 datasets with multiple random seeds and report confidence intervals for the mean gain, including with the Bike regression dataset excluded; if the margin over the no-reasoning baseline shrinks to noise, or if per-feature validation gains fail to predict test gains when the top-K features are combined, the central claim would fail.
Extended reading notes
Core claim
In the paper's own framing, REFEAT's central claim is that diversity of reasoning guidance, not more search budget or a stronger LLM, is what unlocks better LLM-engineered features. The authors argue that six reasoning types each bias the LLM toward different kinds of transformations—inductive prompts toward empirical aggregates, causal prompts toward mechanism-like mediators, counterfactual prompts toward difference-based features—so cycling and adaptively selecting among them widens the search space without brute-force enumeration. They support this with an ablation showing that removing reasoning guidance cuts the average gain by more than half, that no single reasoning type matches the full set, and that bandit-based selection beats uniform cycling. They also report that the resulting features are structurally deeper (more operations, more nesting) and carry higher mutual information with the target, which they read as evidence that the method discovers genuinely new signal rather than just recombining obvious operations.
Load-bearing premise
The load-bearing premise is that a single-split validation gain from adding one feature at a time reliably indicates which features will help once combined with others; if that signal is noisy or top-K features interact negatively, the reported gains would not replicate.
Editorial extensions
If this is right
- On 59 tabular datasets, REFEAT beats each baseline in more than 67.04% of pairwise comparisons and records the largest mean gain with both linear/logistic (+5.66%) and XGBoost (+5.42%) learners.
- Ablation evidence indicates that reasoning guidance itself, not just LLM use, drives the improvement: removing the reasoning prompts cuts the mean gain by more than half.
- The bandit controller adds value beyond uniform cycling, implying that datasets favor different reasoning modes and that the mode can be learned online.
- Features produced under reasoning guidance have higher operation count and depth and higher mutual information with the target, suggesting they are semantically richer rather than merely larger in number.
- The approach transfers across LLM backbones and is reported to be particularly effective on datasets the LLM knows less well, so it may help in specialized domains.
Reading between the lines
- A testable extension the paper only gestures at: collect the bandit's per-dataset arm distribution across many datasets and use it to fine-tune an LLM that predicts a good reasoning-type prior from a task description, removing the 20-iteration warm-up.
- The same reasoning-type bandit recipe should transfer to other LLM generation loops with cheap reward signals, such as code repair, SQL generation, or molecule suggestion, where the model's default mode collapses to common patterns.
- The negative correlation between LLM familiarity and extra gain suggests the method's practical value will grow as LLMs are pushed toward niche domains; a user with a specialized tabular set should expect larger benefits than on common benchmarks.
- The single-split validation reward is the fragile link in the pipeline; replacing it with a cross-validated or multi-seed reward could make the top-K selection substantially more reliable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes REFEAT, an LLM-based feature discovery method for tabular data. REFEAT builds six reasoning-type meta-prompts (deductive, inductive, abductive, analogical, counterfactual, causal), selects among them with an epsilon-greedy multi-armed bandit, evaluates each candidate feature by its marginal validation gain, and augments the original data with the top-K features. The method is evaluated on 59 OpenML datasets with linear/logistic regression and XGBoost against ORIGINAL, AutoFeat, OpenFE, CAAFE, FeatLLM, and OCTree. The paper reports higher mean and median gains than all baselines, plus ablations and analyses of feature complexity, mutual information, and reasoning-type usage.
Significance. If the reported gains are stable, the paper is a useful contribution to LLM-based feature engineering: the reasoning-type decomposition is a clean idea, the bandit formulation is appropriate, and the benchmark is substantially broader than in several prior works. The ablations separating reasoning guidance from adaptive selection are informative, and the feature-complexity and mutual-information analyses give useful qualitative evidence for the mechanism. The main value, however, depends on the reliability of the headline performance claims, which currently rest on a single split with no error bars. The contribution is therefore promising but not yet established at the level claimed.
major comments (4)
- [Section 4.1, Table 1] The central claim of consistent outperformance rests on a single train/validation/test split with default-parameter models and no seeds, confidence intervals, or significance tests. The median gains are modest (+1.23% linear, +0.85% XGBoost), while the mean gains are inflated by a few large outliers: Bike RMSE drops from 1.938E+04 to 8.438E+03 and Grid-Stability from 5E-04 to 2E-04. Since only 8 regression datasets are used, these two outliers can dominate the pooled mean. This is not a stylistic issue but a load-bearing one: the headline 'wins against every baseline in more than 67.04% of pair-wise comparisons' is computed from one run per dataset. Please repeat the evaluation over multiple seeds and data splits, report distributions or confidence intervals, and include paired significance tests across datasets.
- [Section 4.1, Appendix E] The aggregate results are internally inconsistent. The text says a mean of +4.65% and a median of +1.17% 'across the 59 classification tasks,' but the corpus is 51 classification plus 8 regression tasks; Table 1 reports 5.66/1.23 for linear and 5.42/0.85 for XGBoost; Section 4.2 repeats 5.42/0.85 for 'Ours'; and Appendix E refers to 57 datasets. These discrepancies make it unclear which numbers are the definitive result and undermine the reported gains. Please reconcile the dataset count, the task split, and all aggregate statistics in the text, tables, and appendices.
- [Section 4.1, Tables 10-11] FeatLLM is applicable only to classification and is marked N/A on all 8 regression datasets, while other baselines and REFEAT report results on all 59 datasets. The paper does not state whether Table 1's mean/median and the pair-wise win ratios are computed on a common subset of datasets for each comparison. If FeatLLM's aggregates use 51 datasets while other methods use 59, the 'wins against every baseline' claim is not a like-for-like comparison. Please report all methods on the same dataset subset or clearly state the per-comparison denominators and verify that the conclusions are unchanged.
- [Sections 3.2 and 4.1] The feature selection procedure ranks each candidate by its marginal validation gain when added individually to the original features, but the final test evaluation adds all top-K features simultaneously. The paper does not assess interactions among the selected top-K features or the stability of the validation ranking across splits. A feature that appears beneficial in isolation on one split can reduce performance when combined with other selected features. Please report the agreement between validation-selected feature sets and test performance, or rerank/select features jointly, to support the claim that the selected features generalize.
minor comments (6)
- [Section 4.2, Table 2] The ablation table does not state which downstream learner is used; the reported 5.42/0.85 for 'Ours' matches the XGBoost row in Table 1, but the text does not say so. Please specify the learner and report results for both linear and XGBoost settings.
- [Section 5.4, Figure 5] The reasoning-type labels for the baseline are obtained by feeding the baseline's chain-of-thought back to the same LLM family and asking it to classify the trace. Without human annotation or inter-annotator agreement, the distribution in Figure 5 may reflect the labeling prompt's biases rather than the baseline's actual reasoning mixture. Please add an independent validation of the labeling procedure.
- [Appendix E, Table 6] The timing analysis is confusing: the text says 'our model's peak performance occurs on the iteration 7,' while the table shows 'Peak Performance (REFEAT) 7 7.41%' and 'Peak Performance (Baseline) 13 2.51%'; the sentence 'ours reached to the peak earlier than the baseline on the iteration 13' appears to contain an error. Please clarify the iteration numbers and the table formatting.
- [Appendix C] The implementation details do not report random seeds, API temperature, model version, or response sampling parameters for the LLM calls. Since the generated features are stochastic, these details are needed for reproducibility.
- [Section 5.1, Figure 3] The familiarity correlations are reported as point values without confidence intervals or significance tests; given the small number of datasets and the presence of outliers, the claimed positive and negative correlations should be quantified more carefully.
- [Section 3.2] The bandit hyperparameters (exploration schedule, learning rate, iteration budget, and feature budget K) are fixed without sensitivity analysis. A short paragraph or appendix figure showing robustness to these choices would strengthen the claim that the adaptive selection mechanism, rather than a particular schedule, drives the results.
Circularity Check
No significant circularity: the headline test-set gains are held out and selected by validation rewards, not by construction.
full rationale
The paper's central claim—held-out test accuracy/RMSE gains over ORIGINAL and six baselines—is not circular. Features are proposed by an LLM and ranked by validation-set gain Δt (Section 3.2), and final performance is measured on a separate test set (Section 4.1); no parameter is fitted to test labels, and the reported test gains are not equal to the validation rewards by construction. The baselines are also subject to the same validation-selection protocol, so the comparison is not rigged by the selection rule. The self-referential elements are real but peripheral: meta-prompts are bootstrapped by GPT-4.1-mini itself (Section 3.1), and Section 5.4 uses the same LLM family to label baseline chain-of-thought traces. These could bias interpretive analyses (reasoning-type distributions, familiarity correlations), but they do not determine the headline accuracy numbers. The paper's internal inconsistencies (e.g., '+4.65% across the 59 classification tasks' vs. Table 1's 5.66/1.23 and 5.42/0.85; Appendix E's 57 datasets) are correctness and reporting risks, not circularity. Under the proportionality rule, the central derivation is self-contained, so no circular step is identified.
Assumptions & free parameters
free parameters (4)
- epsilon exploration schedule =
linear decay from 1 to 0
- learning rate alpha =
1/n_rt (harmonic step)
- iteration budget =
20
- feature budget K =
10
assumptions (3)
- domain assumption Validation-set performance gain of a single added feature is a reliable reward signal for the bandit and for ranking features.
- domain assumption The six reasoning-type meta-prompts elicit genuinely distinct reasoning modes from the LLM.
- domain assumption Generated pandas expressions execute correctly on all datasets.
Cite this review
Pith. "Pith review of Tabular Feature Discovery With Reasoning Type Exploration." pith.science (2026). https://pith.science/paper/TWSRZPJC
@misc{pith2026250620357,
author = {Pith},
title = {Pith review of: Tabular Feature Discovery With Reasoning Type Exploration},
year = {2026},
howpublished = {\url{https://pith.science/paper/TWSRZPJC}},
note = {Machine review of arXiv:2506.20357}
}
read the original abstract
Feature engineering for tabular data remains a critical yet challenging step in machine learning. Recently, large language models (LLMs) have been used to automatically generate new features by leveraging their vast knowledge. However, existing LLM-based approaches often produce overly simple or repetitive features, partly due to inherent biases in the transformations the LLM chooses and the lack of structured reasoning guidance during generation. In this paper, we propose a novel method REFeat, which guides an LLM to discover diverse and informative features by leveraging multiple types of reasoning to steer the feature generation process. Experiments on 59 benchmark datasets demonstrate that our approach not only achieves higher predictive accuracy on average, but also discovers more diverse and meaningful features. These results highlight the promise of incorporating rich reasoning paradigms and adaptive strategy selection into LLM-driven feature discovery for tabular data.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 1 Pith paper
-
NSR-Boost: A Neuro-Symbolic Residual Boosting Framework for Industrial Legacy Models
NSR-Boost patches a frozen GBDT by adding LLM-generated, Bayesian-tuned symbolic experts in error regions, beating baselines on six public datasets and improving a deployed financial model.
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Nikhil Abhyankar, Parshin Shojaee, and Chandan K Reddy. 2025. Llm-fe: Automated feature engineering for tabular data with llms as evolutionary optimizers. CoRR
work page 2025
-
[4]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[5]
Sebastian Bordt, Harsha Nori, Vanessa Cristiny Rodrigues Vasconcelos, Besmira Nushi, and Rich Caruana. 2024. Elephants never forget: Memorization and learning of tabular data in large language models. In First Conference on Language Modeling
work page 2024
-
[6]
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785--794
2016
-
[7]
Wei Fan, Erheng Zhong, Jing Peng, Olivier Verscheure, Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. 2010. Generalized and heuristic-free feature construction for improved accuracy. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 629--640. SIAM
work page 2010
-
[8]
Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):155--170
1983
Show all 36 references
-
[9]
Robert C Gray, Jichen Zhu, and Santiago Onta \ n \'o n. 2020. Regression oracles and exploration strategies for short-horizon multi-armed bandits. In 2020 IEEE Conference on Games (CoG), pages 312--319. IEEE
2020
-
[10]
Sungwon Han, Seungeon Lee, Meeyoung Cha, Sercan O Arik, and Jinsung Yoon. 2025. Llm-guided self-supervised tabular learning with task-specific pre-text tasks. Transactions on Machine Learning Research (TMLR)
2025
-
[11]
Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. 2024. Large language models can automatically engineer features for few-shot tabular learning. In International Conference on Machine Learning, pages 17454--17479. PMLR
2024
-
[12]
Noah Hollmann, Samuel M \"u ller, and Frank Hutter. 2023. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. Advances in Neural Information Processing Systems, 36:44753--44775
2023
-
[13]
Franziska Horn, Robert Pack, and Michael Rieger. 2019. The autofeat python library for automated feature engineering and selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 111--120. Springer
2019
-
[14]
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. 2021. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928--23941
2021
-
[15]
James Max Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pages 1--10. IEEE
2015
-
[16]
Udayan Khurana, Horst Samulowitz, and Deepak Turaga. 2018. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32
2018
-
[17]
Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasrathy. 2016. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th international conference on data mining workshops (ICDMW), pages 1304--1307. IEEE
2016
-
[18]
Jaris K \"u ken, Lennart Purucker, and Frank Hutter. 2024. Large language models engineer too many simple features for tabular data. In NeurIPS 2024 Third Table Representation Learning Workshop
2024
-
[19]
Volodymyr Kuleshov and Doina Precup. 2014. Algorithms for multi-armed bandit problems. arXiv preprint arXiv:1402.6028
2014 arXiv
-
[20]
David Lewis. 1973. Counterfactuals and comparative possibility. In IFS: Conditionals, Belief, Decision, Chance and Time, pages 57--85. Springer
1973
-
[21]
DD Lucas, R Klein, J Tannahill, D Ivanova, S Brandon, D Domyancic, and Y Zhang. 2013. Failure analysis of parameter-induced simulation crashes in climate models. Geoscientific Model Development, 6(4):1157--1171
2013
-
[22]
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. 2019. Autocross: Automatic feature crossing for tabular data in real-world applications. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Disco...
2019
-
[23]
Renata CB Madeo, Clodoaldo AM Lima, and Sarajane M Peres. 2013. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pages 46--52
2013
-
[24]
S \'e rgio Moro, Paulo Cortez, and Paulo Rita. 2014. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22--31
2014
-
[25]
Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. 2024. Optimized feature generation for tabular data via llms with decision tree reasoning. Advances in Neural Information Processing Systems, 37:92352--92380
2024
-
[26]
Leland Gerson Neuberg. 2003. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4):675--685
2003
-
[27]
Tom Overman, Diego Klabjan, and Jean Utke. 2024. Iife: Interaction information based automated feature engineering. arXiv preprint arXiv:2409.04665
2024 arXiv
-
[28]
Charles S Peirce. 1903. Harvard lectures on pragmatism. Collected Papers, 5:188--189
1903
-
[29]
D Qi, J Peng, Y He, and J Wang. 2023. Auto-fp: An experimental study of automated feature preprocessing for tabular data. Advances in Database Technology-EDBT, 27(1):129--142
2023
-
[30]
a fer, Carsten Grabow, Sabine Auer, J \
Benjamin Sch \"a fer, Carsten Grabow, Sabine Auer, J \"u rgen Kurths, Dirk Witthaut, and Marc Timme. 2016. Taming instabilities in power grid networks by decentralized control. The European Physical Journal Special Topics, 225:569--582
2016
-
[31]
Qitao Shi, Ya-Lin Zhang, Longfei Li, Xinxing Yang, Meng Li, and Jun Zhou. 2020. Safe: Scalable automatic feature engineering framework for industrial tasks. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pages 1645--1656. IEEE
2020
-
[32]
Aleksandrs Slivkins and 1 others. 2019. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning , 12(1-2):1--286
2019
-
[33]
Jack W Smith, James E Everhart, WC Dickson, William C Knowler, and Robert Scott Johannes. 1988. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261. American Medi...
1988
-
[34]
Richard S Sutton, Andrew G Barto, and 1 others. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge
1998
-
[35]
Amy E Taylor, Rebecca C Richmond, Teemu Palviainen, Anu Loukola, Robyn E Wootton, Jaakko Kaprio, Caroline L Relton, George Davey Smith, and Marcus R Munaf \`o . 2019. The effect of body mass index on smoking behaviour and nicotine metabolism: a mendelian randomization study. H...
2019
-
[36]
Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. 2023. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning, pages 41880--41901. PMLR
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.