REVIEW 4 major objections 5 minor 14 references
MedFeat claims that large language models can engineer clinical features far more effectively when the generation loop is told which features the model already relies on and what it cannot learn alone.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
MedFeat iteratively prompts an LLM to create clinical features guided by SHAP importance and the downstream model's inductive bias, improving tabular prediction AUC/F1 on five EHR tasks.
T0 review reviewed 2026-08-03 challenge →
load-bearing objection A clever and ambitious LLM feature-engineering framework whose abstract overclaims; the design deserves peer review but the empirical evidence as reported does not support the headline numbers. the 4 major comments →
MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Clinical Tabular Prediction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that feature engineering should be conditioned on the downstream model's actual behavior. Starting from a baseline model, MedFeat computes Shapley additive explanations (SHAP) on the validation set, ranks features by importance, samples small islands of influential features, and prompts an LLM to write executable transformations that specifically target patterns the learner cannot represent—nonlinear interactions for linear models, temporal and aggregate statistics for tree ensembles. Candidates are evaluated by retraining the model and accepting the best only if validation AUC improves by a tolerance of 0.01, then the importance scores are recomputed and the loop repeat
What carries the argument
The load-bearing mechanism is the MedFeat iteration: (1) train a baseline learner on the current feature set; (2) compute SHAP (Shapley additive explanations) importance on validation data; (3) sample K feature islands, which are small subsets drawn proportional to normalized importance, with repeated temporal measurements grouped to handle collinearity; (4) prompt the LLM with the island, a model-awareness paragraph that tells it what the learner can and cannot represent, and a memory bank of accepted and rejected features; (5) execute candidates locally, retrain, and accept the best island only if validation AUC clears the current baseline by tolerance beta = 0.01; (6) update the memory an
Load-bearing premise
The loop accepts features based on a single validation-set AUC comparison with a fixed 0.01 tolerance, assuming that this comparison is a stable, representative signal of held-out generalization despite severe class imbalance and split-to-split variance.
What would settle it
Re-run MedFeat on the same splits but with the acceptance rule changed to require the validation AUC gain to exceed the bootstrap standard error of the validation metric. If accepted features no longer improve held-out AUC or F1, the reported gains were validation noise. Also shuffle the SHAP importance rankings before island sampling; if held-out performance still improves, the importance signal is not the active ingredient.
If this is right
- In the untuned or default regime, MedFeat reports best or second-best AUC on all tasks for both learner families, with the largest lift on 24-hour inpatient mortality (AUC 0.686 to 0.740 for trees) and consistent F1 gains across tasks.
- After 400-trial hyperparameter optimization, AUC gains mostly shrink but F1 gains persist, especially on the most imbalanced tasks, so the practical claim is that feature engineering helps most when tuning budgets are limited.
- The acceptance criterion and importance guidance produce features that tend to appear in the top-10 SHAP rankings of the final model, supporting the interpretability goal.
- MedFeat features transfer across settings: features generated on the ICU cohort improve AUC and reduce run-to-run variability when applied to a general inpatient cohort, and a model with MedFeat features keeps higher, more stable AUROC under temporal shift than a model retrained annually without new features.
- The framework avoids sending patient-level records to the LLM, which is a prerequisite for many clinical privacy constraints.
Where Pith is reading between the lines
- A testable extension is to replace the fixed 0.01 validation-AUC tolerance with a confidence-interval or bootstrap rule; the paper's own tables show split-to-split standard deviations that often exceed reported gains, so a noise-calibrated acceptance rule could separate signal from luck.
- If SHAP-guided islands are truly the active ingredient, then feeding the LLM shuffled importance rankings should destroy most of the gain; this null experiment would directly test the explainability-driven design.
- The stronger gains for logistic regression than for trees after HPO suggest the framework is most valuable when the downstream model is deliberately kept simple for interpretability or regulatory reasons, which could make simple linear models competitive with heavily tuned ensembles.
- The temporal-shift results raise the possibility that the engineered features encode more invariant clinical mechanisms than raw measurements, which could be exploited by using them as priors for models deployed in new sites with different coding systems.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. MedFeat is an iterative, LLM-driven feature engineering framework for clinical tabular prediction. It uses SHAP importance scores to sample small 'islands' of features, conditions the LLM prompt on the downstream model's inductive bias (logistic regression vs. XGBoost), and accepts/rejects proposed transformations using a validation-AUC tolerance. The authors evaluate on five clinical tasks from IORD, MIMIC-IV, and HRS, comparing against raw baselines, classical AFE (OpenFE), and LLM-based methods (CAAFE, FeatLLM, OCTree), in both untuned and HPO-tuned settings. They also report ablations, cross-cohort feature transfer, and temporal-shift robustness. The abstract claims that MedFeat 'statistically significantly outperforms state-of-the-art baselines, with an average improvement of more than 10%'.
Significance. The core idea—using model-aware, explainability-guided feedback to direct LLM feature generation—is timely and relevant to clinical tabular learning. If the empirical claims held, the framework would be a practical advance over both operator-based AFE and existing LLM-based feature engineering, with the additional privacy benefit of not sending raw patient records. The paper includes useful ablations and a thoughtful discussion of model-dependent feature value. However, the current evidence is not sufficient to support the headline statistical claims: no significance tests are reported, the average improvement figure is not reproducible from the tables, and the acceptance rule is specified inconsistently between the text and Algorithm 1. These issues are load-bearing for the paper's main claims and require substantive revision.
major comments (4)
- [Abstract; Tables 2, 3, 11] The abstract claims 'statistically significantly outperforms' and 'an average improvement of more than 10%'. Neither claim is supported by the reported evidence. No significance tests, confidence intervals, or paired comparisons are provided anywhere; with only three random splits, the standard errors are large. Averaging the AUC percentage improvements across Tables 2, 3, and 11 gives roughly 1.5% (not >10%); the >10% figure appears to arise from relative F1 gains on near-zero F1 baselines (e.g., F1 0.014→0.018 is reported as +28.57%). Many AUC gains are within one standard deviation of the baseline (e.g., Table 2, 24h ICU mortality: 0.760±0.008 vs 0.776±0.020; heart failure: 0.685±0.002 vs 0.687±0.009). The authors should either provide statistically valid evidence (e.g., paired bootstrap or permutation tests across more seeds) or substantially temper the claims.
- [§3.5 vs Appendix A, Algorithm 1] There is a direct contradiction in the acceptance rule. §3.5 states that a candidate is accepted if L_{t,k*} ≥ L_base + β, whereas Algorithm 1 accepts if L_{t,k*} ≥ L_base − β. These are different thresholds and will select different feature sets. Additionally, §3.5 defines L as 'validation loss' and selects k* = arg min_k L_{t,k}, while Algorithm 1 uses k* = arg max_k L_{t,k}; if L is a loss, higher is worse, and accepting when L_{t,k*} ≥ L_base + β would accept a worse model. This ambiguity makes the reported results non-reproducible and must be fixed.
- [§4.3, Tables 2 and 3] The iterative acceptance rule assumes that validation AUC is a stable signal under severe class imbalance, but the paper's own numbers show baseline SDs often larger than the gains and larger than the default tolerance β=0.01. For example, in Table 2 the 24h inpatient mortality AUC gain is 0.054 while the baseline SD is 0.033; in Table 3 several XGBoost AUC gains are 0.00–0.01 with SDs of 0.005–0.038. With only three splits, the validation signal used for feature selection is likely dominated by noise. The paper should report the number of accepted features per split, the validation AUC trajectory, and ideally a nested or repeated-validation estimate of selection stability. Appendix I's acknowledgment that 'importance ordering might vary across splits' further reinforces this concern.
- [Table 3 and §4.4] The HPO results do not support the abstract's unqualified 'statistically significantly outperforms'. In Table 3, XGBoost AUC gains after HPO are 0.83%, 0.00%, 0.56%, 0.14%, and 0.65%—all within one standard deviation. The text appropriately says 'remains competitive', but the abstract and conclusion should be qualified to distinguish the untuned regime, where gains are larger, from the tuned regime, where they largely vanish. This is a central claim of the paper and needs to be represented accurately.
minor comments (5)
- [Figures 2 and 3] Figure 2 reports 'five different runs' but shows no error bars or significance tests. Figure 3 shows a single temporal trajectory without confidence intervals. For claims of reduced variability and drift robustness, error bars across repeated runs or resampling are needed.
- [Table 4] Ablation results are reported only as percentage drops relative to the full model, without the underlying absolute AUC/F1 values and without error bars. On very small F1 values (e.g., 0.013–0.018), a 35.9% drop is less meaningful; absolute values and standard deviations should be included.
- [§3.5] The symbol L is called 'validation loss' but is used as a higher-is-better metric (AUC). This is confusing and should be renamed (e.g., 'score' or 'metric') to avoid the appearance of accepting worse loss.
- [Appendix D, Table 1] AutoFeat is listed in Table 1 and in the methods list, but excluded from experiments due to runtime. This is understandable, but the table should note that it was not evaluated. Also, FeatLLM is marked 'Privacy Preserving: ×' in Table 1, yet Appendix D says few-shot samples were removed to ensure privacy; this inconsistency should be clarified.
- [Throughout] Minor typos and inconsistent capitalization: 'Medfeat' vs 'MedFeat', 'statisitics' in the prompt example, and several run-on sentences. These do not affect the technical content.
Circularity Check
No circularity found: MedFeat's validation-guided search is conventional model selection with independent held-out evaluation; self-citations are not load-bearing.
full rationale
The paper's central derivation is a wrapper-style search: §3 defines the objective as selecting transformations that maximize L on D_val, §3.5 accepts candidates only when validation performance clears a tolerance, and §4.3 evaluates once on a held-out test set. This is standard model selection, not a circular reduction: the validation signal is an input to the search, and the held-out test performance is not used to define or fit anything. SHAP importance (§3.2) conditions island sampling and prompting, but acceptance is governed by validation AUC, and the final test evaluation is independent of the SHAP feedback loop. The ablations in Table 4 are empirical comparisons, not definitional identities. Self-citations (e.g., Wei et al. 2024 for IORD preprocessing, Xu et al. 2025 for LLM domain knowledge) are background/data-preprocessing references and do not supply the core claim; no uniqueness theorem or imported ansatz is load-bearing. The limitation in Appendix I that SHAP importance ordering may vary across splits is a stability/statistical concern, not circularity. Likewise, the apparent mismatch between §3.5's acceptance condition (L >= L_base + beta) and Algorithm 1's (L >= L_base - beta), and the unsupported 'statistically significant >10%' abstract claim, are correctness/reproducibility issues rather than circular steps. Overall the derivation chain is self-contained and externally evaluated.
Axiom & Free-Parameter Ledger
free parameters (5)
- tolerance β =
0.01
- number of iterations T =
3
- number of islands K =
2
- island size m =
3
- LLM temperature =
0.7
axioms (3)
- domain assumption SHAP importance computed on the validation set is a stable, informative signal for guiding feature generation under class imbalance and collinearity.
- domain assumption An LLM can propose clinically meaningful and generalizable features from feature metadata and SHAP rankings alone, without seeing patient-level data.
- domain assumption Validation-set AUC (with tolerance β=0.01) is a reliable criterion for accepting features that will transfer to the held-out test set.
Cite this review
Pith. "Pith review of MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Clinical Tabular Prediction." pith.science (2026). https://pith.science/paper/7472XIYQ
@misc{pith2026260302221,
author = {Pith},
title = {Pith review of: MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Clinical Tabular Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/7472XIYQ}},
note = {Machine review of arXiv:2603.02221}
}
read the original abstract
In clinical tabular prediction, classical machine learning models with feature engineering often outperform neural methods. LLMs are increasingly used to automate this process, acting as domain experts that propose diverse feature transformations to boost downstream performance. However, existing LLM-based methods decouple feature generation from the downstream model: the LLM receives no signal about which features currently drive predictions or where the model's representational capacity falls short, so proposals are neither targeted to promising regions of the feature space nor tailored to the learner's inductive bias. This shortcoming is amplified in healthcare data, which simultaneously exhibits class imbalance, heterogeneous feature spaces, and strict interpretability requirements. In this paper, we propose MedFeat, the first feature engineering framework inspired by the workflow of machine learning practitioners, leveraging model-awareness and feature importance signals to iteratively guide feature discovery for clinical tabular learning. We evaluate MedFeat on a broad range of challenging real-world clinical tasks and show that it statistically significantly outperforms state-of-the-art baselines, with an average improvement of more than 10% over the baseline across models with distinct inductive biases.
Figures
Reference graph
Works this paper leans on
-
[1]
Produce features that is difficult to learn for XGBoost such as row-wise grouping and aggregation, global summary statisitcs over multiple features, complex temporal trending
-
[2]
Do not produce features that is easy to learn for XGBoost such as simple thresholding, trivial interaction, naive scaling, simple temporal patterns for a single feature
-
[3]
Prioritise features that are likely to improve performance
Use given SHAP-based feature importance to guide the feature engineering process. Prioritise features that are likely to improve performance
-
[4]
Use evidence-based clinical reasoning to ground the feature
-
[5]
Prioritise features similar to previous accepted features and avoid features similar to failed features. ... Output requirements: 11 - Your answer must be a Python function snippet wrapped in ‘‘‘python ... ‘‘‘ and nothing else. ... Dataset description: ... Available features and Feature importance (SHAP values): ... Previously successful features: ... Pre...
-
[8]
Normalize the "age" column (min-max scaling)
-
[9]
index_of_multiple_deprivation_score
Normalize the "index_of_multiple_deprivation_score" column (min-max scaling)
-
[10]
Compute the interaction term by multiplying the normalized values
-
[11]
age" or
Handle missing data: if either "age" or "IMD score" is missing, set the interaction feature to NaN. Relevance to XGBoost Model: Tree-based models benefit from multiplicative interactions, especially for features with high SHAP importance. Combining age and deprivation captures a clinically relevant pattern that may indicate high risk in specific subgroups...
-
[12]
age: SHAP importance=0.0974, cumulative_importance=0.0974
-
[13]
hours_since_admission: SHAP importance=0.0731, cumulative_importance=0.1706
-
[14]
age_imd_interaction: SHAP importance=0.0646, cumulative_importance=0.2352 ... D. Additional Implementation details Island Generation.To account for strong collinearities, we treat repeated measurements of the same underlying variable as a temporal group when computing and averaging explainability signals for feature selection. For instance, if the feature...
2017
-
[2019]
URL https://api.semanticscholar. org/CorpusID:196194314. Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., and McDermott, M. Publicly available clinical bert embeddings. InProceedings of the 2nd clinical natural language processing workshop, pp. 72– 78, 2019. Baxter, J. A model of inductive bias learning.Journal of artificial intel...
Pith/arXiv arXiv 2019
-
[2026]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A
Produced and distributed by the University of Michigan with funding from the National Institute on Aging (grant number NIA U01AG009740), Ann Arbor, MI. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. At- tention is all you need.Advances in neural information processing systems, 30, 2017. Wang, Z...
Pith/arXiv arXiv 2017
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.