{"id":"c30c6421-5d99-4afc-b3ef-d135543f4b31","arxiv_id":"2505.21963","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-agent-driven search over fine-tuning and model-merging steps discovers post-training pipelines that beat fixed baselines by up to 9 points on tool use and 3.7 points on math, with caveats about held-out evaluation.","lead":"LaMDAgent uses an LLM agent to automatically pick which fine-tuning or model-merging steps to run, evaluate the resulting model, and remember what worked. In two experiments it reports improved math and tool-use scores while keeping general instruction-following behavior, though the tool-use gain is measured on the same benchmark the agent optimizes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 2's headline +9.0 AceBench gain is the max of 100 search scores on the same benchmark used as reward, with no held-out split or random-policy control in the paper; this is the load-bearing weakness.","rationale":"The paper has real strengths: Experiment 1 uses a validation split for selection, reports held-out test and OOD scores, and includes a random-policy ablation, which supports the claim that LLM-guided search is better than random search at equal iteration count. My concern is confined to Experiment 2 and to the paper's abstract, which leads with the +9.0 tool-use improvement. In Section 4.1 the evaluation protocol for AceBench is described only as 'we report the accuracy'; no split is mentioned, in contrast to the explicit validation/test split in Section 3.1. Section 4.2 reports the best model generated over 100 iterations, and Figure 5 tracks the maximum score, so the reported number is the maximum of an in-sample search trajectory. Best-of-N selection on the reward benchmark is expected to inflate scores even with fixed temperature, because the search trajectory is stochastic and the benchmark has limited size; without a held-out tool-use benchmark or repeated seeds, the +9.0 point gain cannot be distinguished from selection noise. The correct remedy is a held-out evaluation of the final selected pipeline, ideally on a disjoint tool-use benchmark, with multiple seeds and a random-policy comparison. Since this is exactly the condition the reader attached, the verdict remains CONDITIONAL/UNCHANGED.","tokens_in":17520,"tokens_out":6946,"duration_ms":82897,"concrete_test":"Hold out 20% of AceBench Normal tasks (or use a disjoint tool-use benchmark such as BFCL) for final evaluation. Run the Section 4.1 search with reward computed only on the remaining 80%, then evaluate the selected Top-1 pipeline on the held-out split; repeat with three seeds. If the held-out AceBench/BFCL gain over Gemma2-2B-Instruct is within noise of zero, the reported +9.0 improvement is best-of-search selection bias; if it remains large (e.g., at least 5 points), the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LaMDAgent autonomously discovers post-training pipelines that improve tool use by 9.0 points—requires the scores fed back to the agent to be trustworthy estimates of the target capability. In Experiment 1 this is handled: Section 3.1 describes validation and held-out test splits, and Section 3.2 reports test-set results plus a random-policy ablation. Experiment 2 is not handled the same way. Section 4.1 states only that AceBench is used to evaluate tool usage; no split is described for selecting the final model, and Section 4.2 reports the 'best model generated by LaMDAgent' as 0.500 versus 0.410 baseline. Figure 5 confirms this is the maximum along the search trajectory, and the score evolution in Section 4.2 shows every step of the Top-1 pipeline was scored on AceBench. Selecting the maximum of 100 evaluations on the reward benchmark is an in-sample selection procedure: even apart from stochasticity, the reported +9.0 is an optimistic estimate of performance on AceBench-like tool-use tasks, not an unbiased estimate of a deployable gain. Without a held-out tool-use benchmark or repeated seeds, the headline number cannot be distinguished from best-of-N selection artifacts. Experiment 1's controlled design is real evidence for the framework, but the paper's abstract headline relies on the uncontrolled Experiment 2 result. The Limitations section does not flag this missing held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LaMDAgent, an LLM-agent-driven framework that automates post-training pipeline construction by iteratively enumerating candidate actions (SFT, TIES-Merging), selecting actions via an LLM informed by memory, evaluating the resulting models on target-task scores, and updating memory from that feedback. Two experimental settings are presented: Experiment 1 teaches arithmetic, commonsense, and reading-comprehension skills to Gemma2 2B and reports a 3.7-point average improvement on math-related tasks over a fully fine-tuned baseline, with a held-out test split and a random-policy ablation; Experiment 2 enhances tool-use capability of Gemma2 2B Instruct and reports a 9.0-point AceBench accuracy improvement while preserving MT-Bench score. The paper also studies data-size and model-size scaling as cost-reduction strategies, concluding that data-size scaling helps while model-size scaling introduces challenges.","tokens_in":17837,"tokens_out":4316,"duration_ms":46559,"significance":"If the reported results hold, LaMDAgent would be a practically useful step toward automating post-training pipeline design, reducing the manual expertise currently required. The framework is clearly described, and Experiment 1 has several strengths: a held-out test set, a random-policy ablation, and comparison against grid-searched TIES and full fine-tuning. The data-size scaling evidence is also a useful preliminary result. However, the headline claim of a 9.0-point tool-use improvement rests on an uncontrolled in-sample selection procedure in Experiment 2, where the best score of a 100-step search on the same AceBench metric used as reward is reported without a held-out split or repeated runs. This needs to be addressed before the central claim is fully supported.","major_comments":[{"comment":"The reported +9.0 AceBench improvement in Experiment 2 is an in-sample selection result, not an out-of-sample estimate. Section 4.1 states only that AceBench is used to evaluate tool usage and does not describe a held-out split for model selection or final reporting; Section 4.2 reports the best model generated by LaMDAgent, and Figure 5 confirms this is the maximum score along the 100-iteration search trajectory. Since the same AceBench metric is used both as the reward fed back to the agent and as the reported evaluation, selecting the maximum of 100 evaluations is susceptible to best-of-N selection bias, and the +9.0 figure cannot currently be distinguished from an optimization artifact. The authors should report performance on a held-out AceBench split (or a separate tool-use benchmark) for the final model, or provide repeated-seed runs with selection-corrected estimates, and the Limitations section should acknowledge this missing held-out evaluation.","section":"Section 4.1, Section 4.2, Figure 5"},{"comment":"Experiment 1's central quantitative claim is based on a single search trajectory without repeated runs, confidence intervals, or significance testing. Although temperature is set to 0 and a seed is listed in the example config, the stochasticity of LLM-based action selection and SFT training is not otherwise controlled, and the selection of Top-1, Top-2, and Top-3 models by validation score means the reported test-set margins (e.g., 1.9 points average over Fully Fine-Tuned) could reflect selection noise. The authors should run the framework multiple times (or at least provide evidence that the temperature-0 setting produces deterministic trajectories) and report the variance of the reported averages.","section":"Section 3.1, Table 1"},{"comment":"The conclusion that model-size scaling 'introduces new challenges' is supported by only a single transfer from Gemma2 2B to Gemma2 9B, with no repeated trials or error bars. Table 3 reports single numbers for top-1, top-50, top-80, top-90, and top-100 pipelines, and the reversal between Top-1 and Top-50 may be within noise. To make the scaling claim load-bearing, the authors should either add repeated runs with confidence intervals or temper the conclusion to reflect the preliminary nature of the observation.","section":"Section 5, Table 3"}],"minor_comments":[{"comment":"Equation (3) defines a weighted multi-task score with hand-set scaling factors alpha_k, but the example config in Figure 11 lists \"score_aggregation\": \"mean\". Please clarify which aggregation was actually used in each experiment and how alpha_k were chosen in Experiment 1, where all metrics are accuracies in a similar range.","section":"Section 2.4 and Figure 11"},{"comment":"The caption contains the typo 'Action Eunumeration'; it should read 'Action Enumeration'.","section":"Figure 1 caption"},{"comment":"The phrase 'trainining' appears in the first paragraph of Section 5; please correct it. Similarly, 'target tasks' is misspelled as 'taget tasks' in Section 6 (Related Work).","section":"Section 5"},{"comment":"The claim 'to our knowledge, this is the first work to automate post-training using LLM agents' is difficult to verify and should be softened or supported by a more systematic search of concurrent work on LLM-driven pipeline optimization.","section":"Section 6"},{"comment":"In Experiment 2, only Gemma2 2B Instruct is listed as an initial model, yet TIES-Merging requires at least two models; please explain how models for merging were initialized (e.g., through the SFT actions themselves) so that the action space is fully specified.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The framework is timely and the Experiment 1 design is a good step, but the currently reported headline result for tool use is not yet convincing without a held-out evaluation or repeated runs. I would encourage the editor to request those additions rather than reject, as the core idea is defensible and the missing control appears feasible within the manuscript's scope. Also, the authors may want to include code or configuration files for reproducibility; the appendix already provides useful templates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: the framework is a sensible agentic search over post-training actions, and Experiment 1 is honestly controlled. The problem is Experiment 2. The +9.0 AceBench gain is the maximum score along a 100-step search on the same benchmark used as the reward, with no held-out split and no variance estimate. The stress-test note lands on reading the paper; the concern is real.\n\nWhat's actually new: LaMDAgent unifies SFT and TIES-Merging as actions in an LLM-agent loop and updates a textual memory from task scores. That is a natural extension of the authors' earlier self-improvement work, and it makes a concrete claim: an agent can compose fine-tuning and merging into pipelines that beat simple alternatives. The paper is also careful in places, fixing hyperparameters, using temperature 0, and including a random-policy ablation in Experiment 1. That ablation is the strongest evidence in the paper—the agent's selection beats random selection on held-out test tasks. The data-size scaling result is a useful practical finding, and the discovered pipelines are interpretable.\n\nThe soft spots are proportionate to the claims. Experiment 2 is not a controlled evaluation. AceBench is the reward and the reported metric; the best model is selected from 100 scored iterations. Picking the maximum of 100 evaluations will inflate the gain even without deliberate overfitting. The paper does not describe a held-out AceBench split, does not report repeated runs, and does not include a random-policy baseline there. The Limitations section does not flag this. That is not a minor omission, because the abstract's headline claim rests on it. The novelty claim also needs sharper separation from the same authors' NAACL 2025 paper, though the concrete framework and results here are new.\n\nWho gets value: practitioners and researchers working on automated post-training, agentic model development, and benchmark-driven search. The paper is worth a serious referee, but the revision should be held to a clear standard: a held-out tool-use evaluation (or another tool-use benchmark), repeated seeds or confidence intervals, and a random-policy control in Experiment 2. If the held-out result holds, this is a solid contribution. If not, it is still a useful framework with a weaker empirical claim.","headline":"LaMDAgent is a sensible agentic post-training search framework with a well-controlled first experiment, but the headline +9.0 tool-use gain is the best-of-100 on the reward benchmark, so the empirical claim is optimistic until held-out evaluation is added.","tokens_in":18358,"tokens_out":2693,"would_cite":true,"duration_ms":28749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent that sequences fine-tuning and model-merging steps can beat hand-built post-training pipelines, reporting gains of 3.7 math points and 9.0 tool-use points in the paper's experiments.","keywords":["LLM agents","post-training pipeline","supervised fine-tuning","TIES-Merging","model merging","tool-use ability","instruction following","automatic pipeline search"],"falsifier":"Hold out an AceBench split that the agent never observes, rerun the 100-iteration search with the same action pool, and compare the best model's held-out score with the base model's score; if the 9.0-point gap does not reproduce on the held-out split, the central tool-use claim is a selection artifact rather than a general gain. A second check is to rerun Experiment 1 with freshly drawn validation folds and report the mean test score of the top-ranked pipelines instead of the single best one.","tokens_in":17318,"feed_emoji":"🤖","tokens_out":10216,"duration_ms":86677,"temperature":0.7,"pith_summary":"The paper tries to establish that the whole post-training pipeline for an LLM — which actions to run, on which data, in which order — can be discovered autonomously by an LLM agent rather than designed by human experts. The agent enumerates actions from a pool of fine-tuning and model-merging operations, runs one chosen action, scores the resulting model on target benchmarks, and updates a textual memory that guides future choices. Two experiments report that this loop outperforms hand-built baselines: a 3.7-point average gain on math tasks with other skills intact, and a 9.0-point gain in tool-use accuracy on an already instruction-tuned model while instruction-following is preserved. If the claim is right, adapting a model to a new task reduces to specifying an action pool and a benchmark, lowering the specialized knowledge needed for post-training.","feed_headline":"Agent-built pipelines lift tool-use accuracy by 9 points","feed_subtitle":"An LLM agent explores fine-tuning and model-merging steps and learns from scores to keep instruction-following intact.","key_machinery":"The carrying mechanism is a four-step loop: (1) enumerate all concrete actions from action types (SFT, TIES-Merging) and an object pool of models, datasets, and hyperparameters; (2) have an LLM choose an action type and then the objects, guided by a textual memory and by explicit exploration directives; (3) run the action to produce a new model and score it on target tasks, aggregating metrics of different scales through a weighted sum $s_{\\mathrm{multi}}=\\sum_k \\alpha_k s_k^{\\mathrm{single}}$; (4) use a second LLM call to rewrite the memory from the latest action, its score, and all previous trials. The memory is the component that accumulates experience, and the exploration directives are what prevent the agent from collapsing onto one repeated action.","core_discovery":"On its own terms, the paper's central claim is that an LLM-driven search over SFT and TIES-Merging actions, scored by downstream tasks and remembered as text, can discover post-training pipelines that outperform strong baselines. In Experiment 1 the best pipeline improves average math accuracy by 3.7 points over a fully fine-tuned model trained on all data, while keeping other task scores comparable. In Experiment 2 the best pipeline raises AceBench accuracy from 0.410 to 0.500 while the MT-Bench score stays essentially flat (0.810 versus 0.804). The paper also claims the search is not brute force: ablations show that random action selection underperforms agent selection, and removing SFT from the action space costs 12.9 test-set points in Experiment 1. A further claim is that pipelines ranked best at small data size keep their ranking when the data is scaled up, whereas small score gaps between pipelines can reverse when the base model is scaled up.","pith_inferences":["The 9.0-point tool-use gain is the maximum AceBench score along the search trajectory, and the paper does not describe a held-out AceBench split; a conservative reading treats the gain as measured on the evaluation set used, and a definite claim of generalization would require re-evaluating the discovered pipeline on a disjoint benchmark split.","The same memory-update loop should extend to any model-improvement operation expressible as typed actions, such as preference learning or synthetic-data generation; the paper lists these directions as future work.","The data-scaling result suggests a practical budgeting recipe—explore on roughly 1,000 examples per dataset, then scale up the winning pipelines—while the model-size caveat implies that transferring pipelines across base-model sizes should rely on large score gaps rather than small ones."],"forward_implications":["Post-training becomes a delegation problem: a practitioner supplies an action pool and a benchmark, and the loop proposes, executes, and scores candidate pipelines without hand-crafting curricula.","Discovered pipelines can include strategies human experts tend to overlook, such as merging specialist models before fine-tuning on all data, widening the set of viable recipes.","Because data-size scaling preserves the ranking of explored pipelines, cheap small-data exploration can select pipelines for expensive larger-data runs.","For instruction-tuned models, the loop can add a new skill such as tool use without the catastrophic forgetting seen under naive full fine-tuning, provided the action space permits enough data-ordering freedom."],"supporting_citations":[{"why":"Supplies TIES-Merging, the model-merging action type used in both experiments.","marker":"Yadav et al., 2023"},{"why":"Provides Gemma 2, the base model for both experiments.","marker":"Rivière et al., 2024"},{"why":"Supplies GSM8k, a target math task and one of the specialist training datasets in Experiment 1.","marker":"Cobbe et al., 2021"},{"why":"Supplies MT-Bench, the instruction-following metric that Experiment 2 aims to preserve.","marker":"Zheng et al., 2023"},{"why":"Supplies AceBench, the tool-use benchmark behind the reported 9.0-point improvement.","marker":"Chen et al., 2025"},{"why":"Supplies Agent-FLAN, the main tool-use training data used as SFT objects in Experiment 2.","marker":"Chen et al., 2024"},{"why":"Documents the math-first-then-general curriculum that the paper cites to interpret its Top-1 discovered pipeline.","marker":"Dong et al., 2024"}],"fun_headline_variants":["Agent-crafted pipelines beat manual fine-tuning on math tasks","LLM agent finds post-training steps that lift accuracy 9 points","Autonomous agent optimizes SFT and merging for better LLMs","Agent search discovers fine-tuning recipes humans miss","Data scaling makes agent pipeline search cost-effective"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark scores fed back to the agent are unbiased estimates of the capability a practitioner cares about, so that picking the best of roughly 100 explored pipelines does not overfit the evaluation metric.","fun_headline_variants_meta":{"raw":{"variants":["Agent-crafted pipelines beat manual fine-tuning on math tasks","LLM agent finds post-training steps that lift accuracy 9 points","Autonomous agent optimizes SFT and merging for better LLMs","Agent search discovers fine-tuning recipes humans miss","Data scaling makes agent pipeline search cost-effective"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001011,"raw_usage":{"total_tokens":4292,"prompt_tokens":986,"completion_tokens":3306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":3227}},"tokens_in":602,"tokens_out":3306,"duration_ms":25457,"temperature":1.0,"reasoning_tokens":3227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:18:45.112953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out an AceBench split that the agent never observes, rerun the 100-iteration search with the same action pool, and compare the best model's held-out score with the base model's score; if the 9.0-point gap does not reproduce on the held-out split, the central tool-use claim is a selection artifact rather than a general gain. A second check is to rerun Experiment 1 with freshly drawn validation folds and report the mean test score of the top-ranked pipelines instead of the single best one.","supporting_citations":[],"review_version":1}