{"id":"ef95ec56-95ab-4cea-9885-eaae178e98d1","arxiv_id":"2608.10471","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An agent-driven prompt optimizer that controls its own search beats a leading fixed optimizer on four benchmarks, at fewer evaluations and with smaller prompts.","lead":"This paper introduces RLMOpt, a prompt optimizer whose search strategy is run by a language-model agent while a fixed program enforces the scoring and safety rules. In matched tests across four benchmarks it beat a strong standard optimizer on average with fewer evaluations and never scored below its starting point.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution is the load-bearing risk: RLMOpt's gains over GEPA could stem from harness components (composite scoring, Pareto selection, no-regression floor, significance gate, polish, BFCL demos) rather than the adaptive RLM policy, and Section 7 concedes no isolation.","rationale":"The reader's weakest-assumption analysis already identifies attribution as the key risk, and I agree. The paper is honest: Section 7 states that experiments establish the complete system without isolating components, and the BFCL wrapper asymmetry is acknowledged. The central contribution, however, is framed as 'making the search policy itself language-model-driven' (Section 1), so the system-level results only support that claim if the adaptive policy, not the harness, drives the gains. The harness additions are substantial: a per-field composite with fixed weights, Pareto selection, a no-regression floor, a 1.65-SE gate, a deterministic polish stage that can append 3/8/15 gold examples and a structural rewrite, and on BFCL a demonstrations component. Any of these could plausibly account for the reported +0.021 mean and the BFCL +0.033 margin. The 'never underperformed its seed' robustness point is similarly weakened because Algorithm 1 always includes the seed in the final candidate set and selects the best composite on the Pareto frontier, making validation non-regression a construction rather than a discovered property; the test-score claim is stronger but grouped with the same statistic. A single ablation—replacing the RLM proposer with a fixed reflective-mutation proposer inside the same harness—would settle whether the adaptive policy is load-bearing. Until that ablation runs, CONDITIONAL is the right verdict, and my read does not change it.","tokens_in":22903,"tokens_out":5749,"duration_ms":50827,"concrete_test":"Run the attribution ablation proposed in Section 7 on all four benchmarks: hold the RLMOpt harness, budget (B=500), task LM, optimizer LM, scoring, selection, floor, significance gate, and polish stage fixed, but replace the RLM agent's candidate-proposal mechanism with GEPA's reflective-mutation loop (a fixed procedure), and compare held-out four-task mean and per-benchmark margins to GEPA-light and full RLMOpt. If the fixed-proposer-plus-harness variant matches RLMOpt's margins within one paired SE, the adaptive policy is not the source of the gains. A supplementary check: on BFCL, disable the demonstrations component (single-component prompt) and see whether the +0.033 margin persists; if it collapses, the BFCL result is an artifact of the unexposed component.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that an LM-driven search policy outperforms a fixed search procedure—requires that the adaptive RLM agent be the source of the observed gains. In the reported head-to-head, RLMOpt differs from GEPA not only in the search policy but also in the deterministic harness: per-field composite scoring with fixed weights (Eq. 2), Pareto-based selection (Eq. 3), a no-regression floor of 0.05 (Eq. 4), a 1.65-SE significance gate, a final polish stage that adds up to five validation-selected variants (Appendix E), and on BFCL an optimizable demonstrations component that the GEPA wrapper cannot expose. Any or all of these could produce the +0.021 four-task mean advantage; the paper explicitly states 'Our experiments therefore establish the performance of the complete system, but do not isolate the contribution of each component' (Section 7). The BFCL column is the clearest case: the largest margin (+0.033) coincides with the only benchmark where the compared systems edit different object spaces, so that margin is not even a system-level comparison. The 'never underperformed its seed' statistic is likewise an artifact of final selection (Algorithm 1, steps 9–12), which always includes the seed in the candidate set and selects the best composite on the Pareto frontier, guaranteeing no validation regression; presenting test runs as evidence of reliability overstates what the floor guarantees.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RLMOpt, a prompt optimizer in which a recursive language model (RLM) agent controls the outer search procedure—deciding which tools to call, which candidates to evaluate, how to allocate budget, and when to stop—while a deterministic harness owns objective scoring, Pareto-based selection, and regression constraints. The method is evaluated on four benchmarks (Chia, HotpotQA, IFBench-2025, and BFCL multi-turn) against GEPA. The authors report that at a single matched seed RLMOpt obtains the best held-out score on all four benchmarks and improves the four-task mean from 0.589 to 0.610; across 11 matched benchmark-seed comparisons it wins 9, never falls below its seed on test while GEPA does so twice, and uses fewer downstream rollouts. They also argue that optimization gains are governed by seed-prompt headroom rather than by search budget. The paper is careful to disclose that the experiments establish the complete-system performance and do not isolate the contribution of the adaptive search policy.","tokens_in":23039,"tokens_out":6774,"duration_ms":66361,"significance":"If the attribution concern were resolved, this would be a useful contribution to prompt optimization: it demonstrates a concrete instance of an LM-driven search controller with a deterministic safety harness, and it reports unusually transparent matched comparisons with paired standard errors, per-seed tables, full budget accounting, verbatim optimized prompts in Appendix B, and a reproducible record schema in Appendix F. The paper also makes a falsifiable headroom-based prediction about when optimization helps. However, the central design claim—that the adaptive LM-driven search policy, rather than the accompanying harness components, is responsible for the observed gains—is not established by the current experiments, and the paper explicitly concedes this in Section 7. The 'never underperforms its seed' statistic is also partly guaranteed by construction. These issues are load-bearing for the paper's main conclusion, so the manuscript needs additional work before its central claim can be accepted.","major_comments":[{"comment":"The central claim—that an LM-driven adaptive search policy is responsible for the reported gains over GEPA—is not supported by the experiments as designed. Section 7 states that \"Our experiments therefore establish the performance of the complete system, but do not isolate the contribution of each component,\" and this concession applies exactly to the headline comparison. RLMOpt differs from GEPA simultaneously in the adaptive policy, composite per-field scoring (Eq. 2), Pareto-based selection (Eq. 3), the no-regression floor (Eq. 4), the 1.65-SE significance gate, the final polish stage (Appendix E), and, on BFCL multi-turn, an optimizable demonstrations component that the GEPA wrapper cannot expose. Any combination of these components could produce the +0.021 four-task mean advantage, which is small relative to the reported per-run standard errors. The BFCL column is the clearest case: the largest single margin (+0.033) occurs on the only benchmark where the two systems edit different object spaces, so even the system-level comparison is not apples-to-apples there. To support the abstract's and conclusion's attribution of the gains to the adaptive policy, the authors should either provide an ablation that holds the harness, budget, task LM, and optimizer LM fixed while replacing the adaptive policy with a fixed search procedure, or explicitly rescope the paper's central claim to the complete RLMOpt system.","section":"Section 5.2 and Algorithm 1 (steps 8–12)"},{"comment":"The statistic \"never produced a prompt that underperformed its seed\" is partly guaranteed by construction and should not be presented as independent empirical evidence of search reliability. Algorithm 1 always includes the seed prompt in the final candidate set, computes validation composite scores for all candidates, forms the Pareto frontier, and selects the candidate with the highest composite score. Consequently, the returned prompt cannot have a validation composite below the seed's validation composite. The test-level observation that no RLMOpt run fell below its seed is a meaningful empirical fact, but the presentation in Section 5.2 (\"The final row counts runs in which the optimized prompt scored below the seed prompt it started from\") and in the Conclusion conflates a design guarantee on validation with an observed property on test. The paper should explicitly state that the validation-level no-regression property is by construction, and should separate the test-level no-regression result from the harness guarantee when using this statistic as evidence of reliability.","section":"Conclusion versus Table 3"},{"comment":"The conclusion states without qualification that \"RLMOpt achieves the best held-out score on every benchmark,\" but this is only true for the single-seed matched comparison in Table 2. The multi-seed means in Table 3 show that GEPA has the higher Chia mean (0.630 vs 0.622) and that the IFBench-25 means overlap within one standard deviation. The abstract correctly qualifies the claim as \"In a matched comparison at a single seed,\" but the conclusion drops this qualification. The conclusion should either be restricted to the single-seed comparison or replaced by the multi-seed summary: RLMOpt leads on three of four benchmark means and on the overall mean, but not on all four benchmarks.","section":"Conclusion"}],"minor_comments":[{"comment":"The sentence justifying the single-seed protocol says the no-regression floor makes a single seed \"particularly appropriate for RLMOpt\"; this is circular, since the floor is a property of the method being evaluated rather than a reason to avoid averaging over seeds. The multi-seed analysis in Section 5.2 is the right response, and the single-seed justification should be removed or rephrased.","section":"Section 4"},{"comment":"The negative-control results for the ceiling regime are reported only as approximate scores (0.91 and 0.77) without naming the exact benchmark split, model, or seed; this makes the negative-control claim hard to verify. Please specify the setup or move the numbers to the appendix with full configuration details.","section":"Section 6.2"},{"comment":"The BFCL-mt row for RLMOpt(light) reports wall-clock time but marks tokens as \"—\"; please explain why token counts are unavailable for that run, since the other rows report them.","section":"Table 5"},{"comment":"The prompt-size comparison in Table 6 is only reported at seed 7, but the abstract states the 27–79% range without this qualification; please add the seed restriction to the abstract or provide size ranges across the multi-seed runs.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent and within scope for the journal. The core problem is attribution: the central claim about the adaptive policy is untested, and the BFCL comparison is not even a clean system-level comparison. This is fixable with an ablation or a rescoped claim, so I recommend major revision rather than rejection. I have no concerns about citation practice or novelty disclosure beyond what is stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about RLMOpt is that it is a real system paper, not a wrapper on an existing method. The design point—an LM agent controlling the search policy while a deterministic harness owns scoring and selection—is genuinely new relative to GEPA, OPRO, and MIPROv2, and the paper is admirably transparent about what it did and did not do. The matched head-to-head against GEPA is reported with paired standard errors, per-seed tables, and full budget accounting, and the central system-level claim (RLMOpt beats GEPA on four benchmarks at fewer rollouts) is defensible from the reported numbers. The appendices are unusually complete: optimized prompts are reproduced verbatim, the tool interface and selection rules are specified precisely enough to reimplement, and the limitations section is honest rather than defensive.\n\nThe soft spots are real but not fatal, and the paper itself concedes the biggest one. The headline \"never underperformed its seed\" is guaranteed by construction: Algorithm 1 always includes the seed in the final candidate set, and the no-regression floor plus Pareto selection means the final choice cannot be below seed on validation. Counting that as empirical evidence of reliability overstates what the runs show. Second, the paper explicitly says (Section 7) that the experiments establish the performance of the complete system but do not isolate the contribution of the adaptive RLM policy. Since RLMOpt differs from GEPA in multiple harness components at once—composite per-field scoring, Pareto selection, significance gate, polish stage, and on BFCL an optimizable demonstrations component the GEPA wrapper cannot touch—the central design claim (adaptive policy is what helps) is unsubstantiated. The BFCL margin in particular is not even a clean system-level comparison. The headroom thesis is inferred from benchmarks selected to have headroom, two unlisted near-ceiling negative controls, and a two-run synthetic diagnostic; it is plausible and appropriately hedged, but it is a thin basis for a general mechanism claim.\n\nWho is this for? Researchers working on prompt optimization or LM-driven search control will find it a useful system paper with a clear separation of concerns and honest reporting. It deserves a serious referee, but the referee should ask for an attribution ablation (fixed harness with a fixed search procedure versus the adaptive policy) and for code and run configurations to be released. As it stands, the paper's contribution is the system and the matched evaluation, not the demonstrated superiority of adaptive search control.","headline":"A well-documented system paper whose headline robustness statistic is partly guaranteed by construction and whose central attribution claim is explicitly conceded; still deserves serious peer review, but the authors should be pushed to isolate the adaptive policy from the harness.","tokens_in":23776,"tokens_out":1159,"would_cite":true,"duration_ms":12555,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RLMOpt claims that letting a language-model agent control the prompt-search policy, with a deterministic harness enforcing scoring and selection, beats a fixed search procedure on all four benchmarks and never falls below its seed prompt.","keywords":["prompt optimization","recursive language models","language-model agents","adaptive search policy","Pareto-based selection","headroom analysis","multi-turn tool calling","prompt efficiency"],"falsifier":"Keep the harness, budget, task LM, optimizer LM, and seed fixed, and replace the agent's adaptive controller with a fixed search procedure that proposes the same kinds of edits from the same failure-analysis tools; if the fixed procedure matches or beats RLMOpt's held-out scores, the adaptive policy is not the source of the gains. A second check: on a task already at the model's prompting ceiling, verify that neither optimizer improves over the seed—the headroom claim predicts this directly.","tokens_in":22533,"feed_emoji":"🤖","tokens_out":6385,"duration_ms":55086,"temperature":0.7,"pith_summary":"Prompt optimizers today decide in advance how to search: the algorithm chooses candidates, evaluations, and stopping rules while the language model only writes prompts. RLMOpt tests a different arrangement, in which the search policy itself is a language-model agent that inspects the task, reads failure traces, proposes and commits candidates, spends an evaluation budget, and decides when to stop, while a deterministic harness owns scoring and selection. The paper claims this adaptive division of labor beats a fixed search procedure on all four benchmarks it tries—clinical extraction, multi-hop question answering, verifiable instruction following, and multi-turn tool calling—with a higher four-task mean (0.610 against 0.589), fewer rollouts, and smaller prompts. The deeper claim is that what limits prompt optimization is not search budget but the headroom left in the seed prompt: once a task is near the model's prompting ceiling, more search does not help. A reader should care because this reframes prompt optimization from 'search longer' to 'reach the available gain reliably and stop.'","feed_headline":"Adaptive prompt search beats fixed optimizer on 9 of 11 runs","feed_subtitle":"A language-model agent controls the search; gains come from seed headroom, not budget.","key_machinery":"The load-bearing object is the recursive language model (RLM) agent acting as the search policy in a tool-based environment, coupled to a deterministic harness. An RLM is a language model that operates over a programmatic environment and can recursively invoke sub-models; here it writes short programs that call tools for introspection, failure analysis, sub-LM synthesis, candidate evaluation, and stopping. The harness executes all scoring, computes per-field composite scores, enforces Pareto-frontier selection, a per-field regression floor, and a 1.65-standard-error significance gate, and runs a final polish stage, so no language-model decision can trade one output field against another or commit a noisily worse candidate. This separation is what lets the agent explore adaptively while preserving reproducibility and the no-regression property.","core_discovery":"On its own terms, RLMOpt's central discovery is that replacing a hand-coded outer search loop with a recursive language model agent—one that writes code in a REPL to call inspection, diagnosis, synthesis, and evaluation tools—produces a more efficient and more reliable prompt optimizer than a fixed evolutionary search procedure. In matched single-seed comparisons the agent-led system obtains the best held-out score on all four benchmarks and leads the four-task mean (0.610 against 0.589); across 11 benchmark–seed repetitions it wins 9 and never returns a prompt that scores below its seed, while the fixed baseline falls below its seed twice. The paper also reports that optimization gains are governed by seed headroom rather than budget: near-ceiling tasks show no improvement from either optimizer, and a synthetic diagnostic shows that a larger budget can even hurt held-out accuracy by overfitting a small validation set.","pith_inferences":["Because RLMOpt and the fixed baseline differ in several components at once (composite scoring, Pareto selection, no-regression floor, significance gate, polish stage, and an optimizable demonstrations component), the paper's evidence supports the complete system; whether the adaptive policy itself carries the gain remains an open question that a policy-only ablation would settle.","The never-below-seed record is partly by construction on validation—the harness cannot commit a regressing candidate—so the interesting empirical claim is that the same floor transfers to held-out test scores; a reader should look at per-seed test comparisons rather than the validation guarantee.","If headroom is the binding constraint, one practical extension is a cheap pre-search headroom estimate (for example, a few contrasting prompt variants on a small sample) to decide whether optimization is worth running at all.","The uncalibrated stopping policy suggests that a learned stop rule based on remaining per-field headroom, rather than a fixed budget fraction, could reduce run-to-run variance and the observed large-budget overfitting."],"forward_implications":["If RLMOpt is right, the object a language model controls in prompt optimization should include the search trajectory itself, not just candidate text.","Optimization effort should be allocated by measured seed headroom: tasks near the model's prompting ceiling should be left alone, and budgets on high-headroom tasks should be spent on reliable convergence rather than exhaustive search.","Prompt size can drop sharply without losing performance—the optimized prompts are 27–79% of the baseline's length—suggesting concise, rule-based prompts are often enough.","Multi-turn agent systems should be optimized as component maps (system prompt, tool descriptions, demonstrations) rather than as single strings.","A no-regression floor plus statistical gating can make single-run optimization dependable on small validation sets, though the paper notes validation overfitting remains a risk when evaluation data is tiny."],"supporting_citations":[{"why":"Supplies the fixed evolutionary search baseline that RLMOpt is matched against on all four benchmarks.","marker":"[1]"},{"why":"Provides the recursive language model formulation on which the agent search policy is built.","marker":"[12]"},{"why":"Provides the multi-turn function-calling benchmark and gold trajectories used for the agentic tool-use evaluation.","marker":"[20]"},{"why":"Underlies the judge-rationale feedback the agent reads when a predicted field mismatches the gold.","marker":"[21]"},{"why":"Supplies the multi-hop question answering benchmark and reference gold spans used in the HotpotQA runs.","marker":"[22]"},{"why":"Supplies the clinical-trial eligibility extraction corpus and the six scored entity fields for the Chia benchmark.","marker":"[24]"},{"why":"Supplies the verifiable instruction-following benchmark and its 58-constraint verifier registry for the IFBench-2025 runs.","marker":"[25]"}],"fun_headline_variants":["Recursive LM agent optimizes prompts, beats GEPA 9/11","LM-controlled search wins 9/11, never below seed, smaller prompts","Prompt optimization gains come from seed headroom, not budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured advantage is attributed to the adaptive language-model-driven search policy, but the compared systems also differ in scoring, selection, regression floor, significance gating, polish, and an extra optimizable component, so if those harness parts, not the policy, drive the gains, the central design claim is unsubstantiated.","fun_headline_variants_meta":{"raw":{"variants":["Recursive LM agent optimizes prompts, beats GEPA 9/11","LM-controlled search wins 9/11, never below seed, smaller prompts","Prompt optimization gains come from seed headroom, not budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001196,"raw_usage":{"total_tokens":4979,"prompt_tokens":1041,"completion_tokens":3938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":3877}},"tokens_in":657,"tokens_out":3938,"duration_ms":25318,"temperature":1.0,"reasoning_tokens":3877,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:21:18.932187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Keep the harness, budget, task LM, optimizer LM, and seed fixed, and replace the agent's adaptive controller with a fixed search procedure that proposes the same kinds of edits from the same failure-analysis tools; if the fixed procedure matches or beats RLMOpt's held-out scores, the adaptive policy is not the source of the gains. A second check: on a task already at the model's prompting ceiling, verify that neither optimizer improves over the seed—the headroom claim predicts this directly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-turn function-calling benchmark and gold trajectories used for the agentic tool-use evaluation."},{"cited_title":"Zheng, W.-L","cited_arxiv_id":null,"evidence_quote":"Underlies the judge-rationale feedback the agent reads when a predicted field mismatches the gold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-hop question answering benchmark and reference gold spans used in the HotpotQA runs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the clinical-trial eligibility extraction corpus and the six scored entity fields for the Chia benchmark."},{"cited_title":"Benchmark suite (58-verifier registry), 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the verifiable instruction-following benchmark and its 58-constraint verifier registry for the IFBench-2025 runs."}],"review_version":1}