{"id":"0d44beec-fb0f-4588-9f14-30976e7ca24a","arxiv_id":"2505.15076","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A router-selector-generator LLM agent team with offline PPO and dual memories unifies feature selection and generation, reporting improved downstream performance on six tabular datasets.","lead":"The paper introduces MAGS, a multi-agent system in which a router decides whether to add or remove features, a generator creates new ones, and a selector deletes redundant ones. It reports consistent accuracy gains over existing feature engineering methods on six tabular datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline PPO training data and reward scores may be computed on the same evaluation split, letting the router overfit to the reported test metric; the split protocol is unspecified.","rationale":"The Reader's weakest assumption is exactly the concern I identify: that the 400 offline PPO training samples are drawn from the same distribution as the final evaluation and that the learned router may fit the evaluation metric. My reading of Section 3.3.2 and Appendix B.4 confirms this gap: the paper describes collecting (prompt, answer, score) triplets where the score is downstream performance, but never states whether that performance is measured on training, validation, or test data. Since the router's only learning signal is this score, any overlap with the evaluation data would directly contaminate the central comparison in Table 1. The lack of error bars compounds this: with LLM-based agents there is inherent stochasticity, and single-run numbers cannot establish 'consistent' superiority. I do not see a separate internal inconsistency in the method's formalization; the ablation study (Figure 3) shows each component contributes, and the robustness check (Table 3) is supportive if the split is clean. The paper also self-reports high computational cost and prompt-length limits in Appendix C, which do not undermine the central claim. Therefore the appropriate verdict remains CONDITIONAL, pending the split clarification and reproducibility artifacts. I agree with the Reader's identification; my concrete test pins down exactly what would settle the concern: a clear statement of the data split used for reward computation, and a held-out re-run with variance reporting.","tokens_in":65,"tokens_out":2578,"duration_ms":33077,"concrete_test":"Ask the authors to provide the exact train/validation/test split for each of the six datasets and to confirm in writing that the 400 offline PPO samples, including the performance scores used as rewards, were generated only from the training and validation folds and never from the final test fold. Then re-run the full MAGS pipeline on the held-out test sets with at least five random seeds and report mean and standard deviation. If the test-set performance drops to within noise of the best baseline, or if any reward computation touched the test split, the reported superiority is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MAGS 'consistently outperforms' six generation and six selection baselines rests on the router policy learned by offline PPO (Section 3.3.2). The policy is trained to maximize a reward that is the downstream task performance of each routing decision. Appendix B.4 states that '400 samples' are collected to fine-tune the router, but the paper never specifies on which data split those samples and their performance scores are obtained. If the 400 trajectories are generated from the full dataset, or from the same test split used for the numbers in Table 1, then the router is effectively trained on the evaluation metric, and the reported gains (e.g., openml_607: MAGS 0.947 vs. best baseline 0.890) may reflect test-set overfitting rather than a genuinely better routing strategy. The paper also reports no error bars or multiple-seed variance, so with stochastic LLM agents (GPT-3.5-Turbo for generator/selector, LLaMA-3.2-3B for router) it is impossible to tell whether the margins are significant. This is the load-bearing weak spot: the claim of superiority requires that the router's training data and reward computation be strictly separated from the final evaluation, and that separation is neither described nor demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAGS, a multi-agent system that unifies feature selection and feature generation for tabular data. A router agent decides at each step whether to generate or select features; a generator agent proposes new features via a predefined operator set; a selector agent masks redundant features. The authors add short-term memory (within-iteration feedback) and long-term memory (top-performing demonstrations) to support in-context learning of the generator and selector, and they fine-tune the router with offline PPO on 400 collected samples. Experiments on six datasets compare MAGS with six feature generation and six feature selection baselines, and report main results, ablations, robustness across downstream classifiers, and a case study. The central claim is that this jointly scheduled, memory-augmented, PPO-trained agent team consistently outperforms the baselines on the reported metrics.","tokens_in":13075,"tokens_out":7053,"duration_ms":60702,"significance":"If the reported gains hold under a clean evaluation protocol, MAGS would be a useful contribution to automated feature engineering. The framing of feature augmentation as router-selector-generator teaming is reasonable, the memory mechanism is clearly motivated, and the paper includes ablations, a robustness check, a traceability case study, prompt templates, and an explicit limitations section. However, the central comparison rests on a router policy trained by offline PPO whose data-split protocol is never specified, and all headline numbers are reported without variance estimates or statistical tests. As submitted, the evidence therefore does not yet establish the claimed consistent superiority over baselines.","major_comments":[{"comment":"The paper never specifies the data split used to collect the 400 (prompt, answer, score) samples for offline PPO training of the router, nor the split used to compute the downstream performance scores reported in Table 1. Because the router policy is optimized to maximize downstream task performance, the same quantity that appears in the evaluation objective in Eq. (1), any overlap between the PPO training rewards and the final evaluation data could make the reported gains in Table 1 an artifact of fitting the router to the test metric. The authors must state exactly which rows are used for PPO data collection, how each downstream performance score is computed, and confirm that no evaluation data is used in router training, reward computation, or long-term memory demonstrations.","section":"Section 3.3.2; Appendix B.4"},{"comment":"The claim that 'MAGS consistently outperforms all competing methods on most datasets and evaluation metrics' is stronger than Table 1 supports. On svmguide3, ELLM-FT achieves F1 = 0.845 while MAGS achieves F1 = 0.840, and several other margins are at the third decimal (for example, german_credit accuracy 0.776 vs. 0.774). The paper reports no error bars, no multiple-seed results, and no statistical tests, even though GPT-3.5-Turbo, LLaMA-3.2-3B, and PPO training are all stochastic. The phrase 'consistently outperforms' should be restricted to the metrics where it is actually true, and variance information is needed to make the comparison meaningful.","section":"Section 4.2; Table 1"},{"comment":"The ablation results are interpreted as showing 'consistent performance degradation' when each component is removed, but on svmguide3 all four variants are within 0.001 to 0.006 of MAGS (0.849 vs. 0.855), and no variance is reported. Table 2's column labels and arrows are also ambiguous: '36%↓38%' can be read as either percentage-point changes or relative changes, and the table does not state which data split produced the router decision statistics. Repeated runs with confidence intervals would be needed to support the component-wise conclusions.","section":"Section 4.3; Figure 3; Table 2"},{"comment":"The experimental protocol is underspecified: there is no statement of the train/test split, cross-validation scheme, random seeds, or number of independent runs for the six datasets. The downstream Random Forest model may be deterministic, but the LLM agents and the PPO fine-tuning are not, so the point estimates in Tables 1-3 are not reproducible without these details. This is load-bearing because the entire empirical contribution depends on the reported numbers being stable across runs.","section":"Section 4.1; Appendix B"}],"minor_comments":[{"comment":"The sentence 'Short-Term Memory is the actions in the trajectory of one agentic exploration iteration without restart' is incomplete; it should define the memory as the sequence of action records, feature states, and performance scores within an iteration.","section":"Section 2"},{"comment":"The dataset name is misspelled as 'svmuide3'; it should be 'svmguide3'.","section":"Appendix B.1; Table 4"},{"comment":"The Limitations section refers to 'Transformer and Selector agents'; the intended terms appear to be 'Generator and Selector agents'.","section":"Appendix C"},{"comment":"The reference to 'The Secret Sauce to Winning' is informal and not cited; a proper citation or URL should be provided.","section":"Section 1"},{"comment":"The numbers in the case-study diagram (54, 20, 17, 25, 18, 7) are not explicitly defined in the caption; the axes and counts should be labeled so that 'generates 4 new features while removing 20 redundant ones' can be read directly from the figure.","section":"Figure 4"},{"comment":"The limitation that generated features may not generalize across tasks is appropriately stated, but it is in tension with the abstract's claim of consistent superiority; the abstract and Section 4.2 should be qualified to task-specific performance.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-split issue is the main gate for this paper. If the authors cannot demonstrate a clean separation between the offline PPO training data and the Table 1 evaluation data, the central empirical claim fails. The meaning of 'offline PPO' also needs clarification, since the 400 samples appear to be generated by the system's own exploration; the data collection process must be described precisely. I do not see a novelty problem with the teaming and memory design, but the paper's current reporting makes it impossible to judge how much of the reported gain is genuine routing improvement rather than test-set fitting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the MAGS paper. Bottom line: it's a coherent, reasonably original system paper, and the router-selector-generator unification is a genuine architectural contribution, but the empirical case for \"consistently outperforms\" is under-built: no code, no error bars, and, more importantly, no explicit split between the offline PPO training data and the final evaluation. Given that the router reward is downstream task performance on the same datasets, the split question is load-bearing.\n\nWhat's genuinely new: framing feature selection and generation as a teaming problem, with a router agent deciding which to invoke, plus short-term and long-term memories to guide the generator/selector via in-context learning, and offline PPO to fine-tune the router. The ablation is the best part—four variants (w/o RL, w/o Router, w/o Long, w/o Short) and Table 2 showing the router shifts its behavior after RL fine-tuning (e.g., more generation on low-dimensional, more selection on openml_589). That supports the internal story. The limitations section is honest about cost and token limits, which I appreciate.\n\nSoft spots, in order of severity. First, the data protocol. Section 3.3.2 says 400 samples are collected for offline PPO, and Appendix B.4 gives no detail on which split they come from. If the 400 trajectories and their reward scores are drawn from the full dataset or the same test split used for Table 1, the gains are partly fitted to the evaluation metric. The paper needs a clear statement of train/validation/test separation for the router training data, ideally with the router validated on a held-out task. This is fixable, but it's the difference between a solid claim and a speculative one. Second, no error bars or seeds anywhere. With stochastic LLM agents, the margins in Table 1 (e.g., openml_607: 0.947 vs 0.890) might be real, but we can't tell. Third, only six small datasets, all <=1000 samples and <=50 features, so the \"consistently outperforms\" claim is narrow. Fourth, robustness check in Table 3 is on one dataset only.\n\nThe math is straightforward (no deep theory), the citations are fine. The paper is honestly written and internally consistent. Who is it for? People building agentic systems for tabular data or feature engineering; it's a useful system paper to know about, not a definitive result.\n\nRecommendation: send it to peer review, but as conditional—require code release, error bars or seeds, and a description of the offline-data split. The idea deserves referee time; the evidence doesn't yet support the headline claim as stated.","headline":"Plausible agentic feature-engineering system with a genuine router-selector-generator contribution, but the missing train/test split protocol and error bars leave the headline claim under-supported.","tokens_in":13567,"tokens_out":2658,"would_cite":false,"duration_ms":21990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that unifying feature selection and generation through a three-agent LLM team consistently outperforms separate feature engineering baselines on six tabular datasets.","keywords":["feature engineering","multi-agent LLM system","feature selection","feature generation","tabular data","offline reinforcement learning","in-context learning","agentic AI"],"falsifier":"If a held-out evaluation, in which the router's offline PPO training data are collected on separate datasets from those used in final evaluation, showed the performance advantage shrinking or disappearing, the central claim would be weakened. Concretely: retrain the router on 400 samples from three of the six datasets, then evaluate on the other three, and compare against the reported same-dataset results; a large drop would indicate the reported gains are partly memorization of the evaluation distribution.","tokens_in":12633,"feed_emoji":"🤖","tokens_out":5614,"duration_ms":44444,"temperature":0.7,"pith_summary":"This paper argues that feature engineering for tabular data should be a single coordinated process that both creates new features and removes redundant ones, rather than two separate stages. It proposes MAGS, a multi-agent LLM system with a router agent that decides at each step whether to generate or select, a generator agent that crosses features with mathematical operators, and a selector agent that masks out low-value dimensions. The router is fine-tuned with offline PPO, and both short-term and long-term memories provide in-context learning demonstrations. On six tabular datasets, the paper reports that MAGS consistently outperforms six feature-generation and six feature-selection baselines across classification and regression metrics, and that ablation variants confirm the router and both memories each contribute to the gains.","feed_headline":"Three LLM agents unify feature selection and generation","feed_subtitle":"Router, selector, and generator agents coordinate additive and reductive steps, beating separate pipelines on six tabular datasets.","key_machinery":"The mechanism is the three-agent collaboration loop. The router agent is a small LLM (LLaMA-3.2-3B) that outputs a binary choice—generation or selection—with an explanation; the generator and selector agents (GPT-3.5-Turbo) then produce new feature tokens or a feature mask, respectively. Feature sets are encoded as postfix token sequences so that both agents operate on the same representation. Short-term memory feeds the current iteration's action-performance records back into the prompts; long-term memory randomly samples K high-performing demonstrations from the memory pool. The router is fine-tuned via offline PPO on 400 collected (prompt, answer, score) triples, where the score is the downstream performance of the resulting feature set.","core_discovery":"The central claim is that unification improves performance: scheduling feature generation and selection jointly, through LLM agents, produces better downstream task performance than doing either alone or in sequence. Treating a feature set as a postfix token sequence lets LLM agents manipulate it directly: the selector outputs a binary mask over feature tokens, the generator outputs new tokens built with a fixed operator set, and the router chooses between them based on the current feature state and task history. The paper shows empirically that this router-selector-generator loop, augmented with short-term (recent trajectory) and long-term (high-quality historical demonstrations) memories and an offline-PPO-fine-tuned routing policy, yields the best scores on the evaluated datasets, with the router learning dataset-specific scheduling tendencies (e.g., more generation on low-dimensional data, more selection on noisy high-dimensional data).","pith_inferences":["A natural next test is cross-dataset generalization: fine-tune the router on some datasets and evaluate on unseen ones; the paper does not report this, so the current results do not establish that the routing policy transfers beyond the six datasets.","The postfix token representation and fixed operator set could be extended to other tabular transformations (e.g., aggregations, imputations) by expanding the operator set, which the agent prompts already support in principle.","The offline-PPO router is a general scheduling device: the same fine-tuning scheme could coordinate generation and selection in other AutoML loops, such as hyperparameter optimization or data cleaning, where the binary choice is between two agent actions.","The paper's stated limitations—high LLM overhead and token limits—suggest MAGS is most practical for small-to-medium tabular datasets; a cheaper routing model or a more compact feature encoding would be needed for wide tables."],"forward_implications":["Feature engineering for tabular data can be reframed as a sequential agentic decision process, making LLM-based teaming a viable alternative to classic AutoML feature pipelining.","The router learns dataset-specific scheduling: it favors generation on low-dimensional datasets and selection on noisy or high-dimensional ones, suggesting the policy adapts to data characteristics.","Both long-term and short-term memories are necessary: ablations show removing either degrades performance, so memory-based in-context learning is a core contributor, not decoration.","The feature sets produced are traceable and interpretable: on openml_586, MAGS removed 20 redundant features and added 4 new ones, whereas the generation-only baseline added 17 and the selection-only baseline removed 18 without adding.","Performance gains persist across six downstream model families (GB, KNN, LR, MLP, SVM, RF) on messidor, indicating the generated features are not overfitted to one classifier."],"supporting_citations":[{"why":"The strongest feature-generation baseline (ELLM-FT) whose performance MAGS must exceed; the paper explicitly highlights it as the most competitive generation method.","marker":"[2]"},{"why":"TTG, an RL-based feature transformation baseline, represents the sequential decision-making approach that MAGS extends with selection and routing.","marker":"[12]"},{"why":"GRFG, a group-wise RL feature generation baseline, one of the six generation methods MAGS is compared against.","marker":"[20]"},{"why":"MARLFS, a multi-agent RL feature selection baseline, the main selection-side competitor.","marker":"[15]"},{"why":"FSNS, a generative feature selection method, another selection baseline that MAGS outperforms.","marker":"[4]"},{"why":"NFS, an early neural feature search method, contextualizes the generation baselines.","marker":"[1]"}],"fun_headline_variants":["LLM agents unify feature selection and generation","Router, selector, generator: agentic feature teamwork","Multi-agent memory guides feature pruning and growth","Offline RL tunes router for smarter feature edits","Agentic system outperforms separate feature tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the 400 offline samples used to fine-tune the router being representative of the feature states and datasets encountered during evaluation; the paper does not state that these samples are disjoint from the test data, so the router could in principle be fitting the evaluation metric rather than learning a general scheduling policy.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents unify feature selection and generation","Router, selector, generator: agentic feature teamwork","Multi-agent memory guides feature pruning and growth","Offline RL tunes router for smarter feature edits","Agentic system outperforms separate feature tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3736,"prompt_tokens":895,"completion_tokens":2841,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2771}},"tokens_in":511,"tokens_out":2841,"duration_ms":18334,"temperature":1.0,"reasoning_tokens":2771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:25:29.013437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a held-out evaluation, in which the router's offline PPO training data are collected on separate datasets from those used in final evaluation, showed the performance advantage shrinking or disappearing, the central claim would be weakened. Concretely: retrain the router on 400 samples from three of the six datasets, then evaluate on the other three, and compare against the reported same-dataset results; a large drop would indicate the reported gains are partly memorization of the evaluation distribution.","supporting_citations":[{"cited_title":"Evolutionary large language model for automated feature transformation","cited_arxiv_id":null,"evidence_quote":"The strongest feature-generation baseline (ELLM-FT) whose performance MAGS must exceed; the paper explicitly highlights it as the most competitive generation method."},{"cited_title":"Feature engineering for predictive modeling using reinforcement learning","cited_arxiv_id":null,"evidence_quote":"TTG, an RL-based feature transformation baseline, represents the sequential decision-making approach that MAGS extends with selection and routing."},{"cited_title":"Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction","cited_arxiv_id":null,"evidence_quote":"GRFG, a group-wise RL feature generation baseline, one of the six generation methods MAGS is compared against."},{"cited_title":"Automated feature selection: A reinforcement learning perspective.IEEE Transactions on Knowledge and Data Engineering, 35 (3):2272–2284, 2021","cited_arxiv_id":null,"evidence_quote":"MARLFS, a multi-agent RL feature selection baseline, the main selection-side competitor."},{"cited_title":"Neuro-symbolic embedding for short and effective feature selection via autoregressive generation.ACM Transactions on Intelligent Systems and Technology, 16(2):1–21, 2025","cited_arxiv_id":null,"evidence_quote":"FSNS, a generative feature selection method, another selection baseline that MAGS outperforms."},{"cited_title":"Neural feature search: A neural architecture for automated feature engineering","cited_arxiv_id":null,"evidence_quote":"NFS, an early neural feature search method, contextualizes the generation baselines."}],"review_version":1}