{"id":"9df5c48e-fd79-43a7-bd51-1716a5deffbd","arxiv_id":"2603.23994","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hidden design choices—starting artifact, credit horizon, and batching of trial evidence—can determine whether iterative LLM generative optimization works, and there is no simple universal setup across domains.","lead":"This paper argues that LLM-based generative optimization stays brittle because engineers must make hidden design choices about what to edit and what feedback to feed back. Case studies suggest starting artifacts, credit horizons, and batching can decide success or failure, which matters for anyone trying to ship self-improving agents.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Generalization from three case studies plus a 9% survey figure to a productionization hurdle for most applications is the load-bearing soft spot.","rationale":"The reader correctly isolated the representativeness/generalization premise as the weakest assumption linking the case-study findings to the productionization claim; that is the single most load-bearing concern. The abstract’s language is carefully hedged on the empirical side (“can determine,” “we find”) but then asserts a major cross-domain hurdle; the gap is exactly the missing evidence that the three factors are primary and that the survey non-use is explained by them. No stronger internal inconsistency appears in the available text. Because the CACHEABLE full-text block is empty, methods, variance, baselines, and survey methodology remain uninspectable, so the appropriate stance remains UNVERDICTED with no change to the reader’s verdict. A full-text pass that supplies the survey instrument, domain-selection rationale, and effect-size comparisons against alternative factors could raise or lower the claim, but on present evidence the concern stands and does not require a different verdict label.","tokens_in":2473,"tokens_out":612,"duration_ms":20282,"concrete_test":"Locate the source of the 9% agent-survey statistic and extract the free-text or coded reasons respondents gave for not using automated optimization. If setup/design-choice difficulty is not among the top cited barriers (vs. compute cost, eval harness absence, model quality, etc.), the causal attribution from the three factors to the adoption hurdle weakens and the broad conclusion should be narrowed to the three studied domains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that hidden setup choices (starting artifact, credit horizon, batching of trials/errors) can make or break LLM generative optimization and that the lack of a simple universal loop is a major productionization/adoption hurdle. The abstract supports the first half with three domain-specific findings (reachable solutions depend on start in MLAgentBench; truncated traces still help Atari; larger minibatches do not monotonically help BBEH) and the second half with a 9% survey statistic. The load-bearing step is the move from those three empirical sensitivities plus the adoption rate to the claim that these factors affect most applications and are primary drivers of brittleness. Nothing in the abstract establishes (a) that the three domains are representative of the broader space of code/workflow/prompt optimization, (b) that the three factors dominate alternatives (base-model capability, reward/eval design, search budget, tool reliability), or (c) that non-use in the survey is caused by setup opacity rather than cost, latency, missing harnesses, or model limits. If any of (a)–(c) fails, the productionization-hurdle conclusion does not follow from the reported findings.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that iterative generative optimization with LLMs—using execution feedback to improve code, workflows, or prompts—is promising for self-improving agents but remains brittle in practice, citing that only 9% of surveyed agents used automated optimization. Brittleness is attributed to “hidden” design choices required to set up a learning loop: what the optimizer may edit and what learning evidence to supply at each update. The authors study three factors claimed to affect most applications—the starting artifact, the credit horizon over execution traces, and batching of trials/errors into learning evidence—via case studies on MLAgentBench, Atari, and BigBench Extra Hard (BBEH). Reported findings are that starting artifacts determine reachable solutions on MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. They conclude that the lack of a simple universal loop setup is a major productionization/adoption hurdle and offer practical guidance for these choices.","tokens_in":2736,"tokens_out":1231,"duration_ms":22770,"significance":"If the empirical sensitivities hold and the productionization claim is well supported, the paper would be a useful contribution to the LLM-agent and automated-optimization literature: it names under-specified setup decisions that practitioners routinely face, and it frames generative optimization as a learning-loop design problem rather than only a model-capability problem. The three concrete factors (starting artifact, credit horizon, evidence batching) and domain case studies could help standardize reporting and reduce brittle re-implementations. Strengths claimed in the abstract—explicit case studies across code/agent/prompt-style tasks and practical guidance—are the right kind of deliverable for an empirical systems paper. The significance of the broad “major hurdle for most applications” conclusion, however, depends on representativeness of the domains, survey evidence, and comparison against alternative drivers of non-adoption (cost, latency, eval design, base-model limits).","major_comments":[{"comment":"Abstract / conclusion: The load-bearing move from three domain findings plus a 9% survey figure to “these factors affect most applications” and “a major hurdle for productionization and adoption” is not yet justified by the abstract’s evidence. The manuscript must either (i) argue representativeness of MLAgentBench, Atari, and BBEH for the broader space of code/workflow/prompt optimization, or (ii) narrow the claim to “in these settings, setup choices can determine success.” Without that, the productionization conclusion does not follow from the reported sensitivities alone.","section":null},{"comment":"Abstract (survey claim): The 9% adoption statistic is used both to define brittleness and to motivate the causal story that hidden setup choices drive non-use. The paper needs survey methodology (sample definition, what counts as “automated optimization,” response rate) and, more importantly, evidence that non-use is caused by setup opacity rather than cost, latency, missing harnesses, reward design, or model limits. If causation is not established, the survey should be framed as motivation, not as support for the mechanism.","section":null},{"comment":"Case studies (MLAgentBench / Atari / BBEH): The three directional findings are central, but the abstract does not report controls, sample sizes, variance, baselines, or ablations against alternative drivers (base model, search budget, eval reliability, editable scope beyond the named factors). For the claim that design decisions “can determine whether generative optimization succeeds,” each study needs a clear success criterion, comparison conditions (e.g., full vs truncated traces; multiple starts; minibatch sizes with held-out generalization), and enough runs to show the effect is not noise. Truncated-trace Atari gains and non-monotonic BBEH minibatches are especially easy to over-interpret without those details.","section":null},{"comment":"Scope of the three factors: The paper asserts that starting artifact, credit horizon, and batching “affect most applications,” yet does not show they dominate or even rank above other free parameters (reward/eval design, tool reliability, proposal temperature, memory of past trials, multi-objective tradeoffs). A major revision should either empirically compare against at least one alternative driver per domain or explicitly limit the thesis to “three under-documented factors that matter,” not “primary drivers of brittleness.”","section":null}],"minor_comments":[{"comment":"Define “credit horizon,” “learning evidence,” and “generative optimization” early and consistently; they are paper-specific terms and should not rely only on the abstract’s brief gloss.","section":null},{"comment":"Clarify what “truncated traces can still improve” means operationally (which prefix length, relative to full-trace baseline, absolute vs relative reward).","section":null},{"comment":"State whether practical guidance is checklist-style, decision-tree, or domain-conditional; readers will look for actionable defaults after the negative “no universal setup” claim.","section":null},{"comment":"Cite and position against related LLM optimization / self-refine / evolutionary prompt-optimization lines so the “rarely made explicit” claim is checkable.","section":null}],"recommendation":"major_revision","confidential_remarks":"The CACHEABLE full-manuscript block provided for review is empty beyond the abstract, so this report is necessarily grounded in the abstract, the stated claims, and the load-bearing generalization step. If the complete PDF has full methods, variance, survey details, and ablations, several major comments may reduce to presentation or minor revision; if not, the productionization claim remains overreached. Fit for a serious cs.LG venue is reasonable as an empirical systems/analysis paper, but only if the causal and representativeness gaps are closed or the claims are narrowed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: generative optimization with LLMs is brittle less because of the optimizer itself and more because engineers must make hidden setup choices—what can be edited, how long a credit horizon to use on traces, and how to batch trials into learning evidence—and those choices can decide success or failure. The paper makes that explicit and shows it with three case studies (MLAgentBench, Atari, BBEH).\n\nWhat is actually new is the framing and the empirical demonstration that these classical learning-system knobs matter for LLM-driven artifact improvement, plus the claim that they are rarely reported. The abstract’s directional findings are useful: starting artifacts change reachable solutions, truncated traces can still help Atari agents, and larger minibatches are non-monotonic on BBEH. Naming “credit horizon” and “learning evidence” as first-class design objects is practical and overdue for the agent-optimization literature. The 9% survey figure is a concrete adoption signal, not just rhetoric.\n\nThe soft spot is the generalization step, not the case studies themselves. Moving from three domains plus low survey adoption to “these factors affect most applications” and “major hurdle for productionization” is load-bearing and under-supported in the abstract. Nothing yet shows the three domains are representative, that these three factors dominate alternatives (eval design, budget, tool reliability, base model), or that non-use is caused by setup opacity rather than cost or missing harnesses. That is a real but proportionate concern: the local findings can still be right while the broad hurdle claim overreaches.\n\nThis is for people building or evaluating self-improving agents and automated prompt/code/workflow optimizers. Readers who care about reporting standards and reproducible learning loops get value; pure theory people less so. Math and formal results are not the point here—this is empirical diagnosis. Citation pattern and full methods are not inspectable from what we have, so soundness stays provisional until variance, baselines, and artifacts are visible.\n\nI would send it to peer review. It is important enough in the subfield and honest enough about a real engineering failure mode to deserve referee time, even if the productionization conclusion needs tightening. Engage with the work; treat the three factors as a useful checklist and the universal-setup claim as a hypothesis still under test.","headline":"Clear diagnosis that setup choices (start artifact, credit horizon, batching) can make or break LLM generative optimization; the productionization leap from three case studies plus a 9% survey is the soft spot.","tokens_in":3433,"tokens_out":581,"would_cite":true,"duration_ms":10765,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Hidden design choices in LLM generative-optimization loops decide whether iterative improvement works, and the lack of a universal setup blocks production use.","keywords":["generative optimization","large language models","self-improving agents","learning loops","credit assignment","execution feedback","design choices","agent brittleness"],"falsifier":"A larger survey of production agents that records starting artifact, credit horizon, and batching policy for each system, then tests whether those three variables still predict success outside the paper’s three domains; if they do not, the general-hurdle claim fails.","tokens_in":3309,"feed_emoji":"🔄","tokens_out":911,"duration_ms":18477,"temperature":0.7,"pith_summary":"Generative optimization uses large language models to repeatedly improve artifacts such as code, workflows, or prompts from execution feedback. The approach is promising for self-improving agents, yet almost unused in practice: only 9 percent of surveyed agents apply any automated optimization. The authors argue the brittleness comes from design choices that engineers must make but almost never write down—what the optimizer is allowed to edit, and what execution evidence counts as the right learning signal at each step. They examine three factors that appear in most applications: the starting artifact, how far back credit is assigned in execution traces, and how trials and errors are batched into learning evidence. Case studies on machine-learning agents, Atari controllers, and hard reasoning benchmarks show these choices can determine success or failure, even though prior work rarely states them. The paper concludes that the absence of a simple, universal way to configure learning loops across domains is a central obstacle to adoption and supplies practical guidance for making the choices explicit.","feed_headline":"Hidden loop choices make LLM optimizers succeed or fail","feed_subtitle":"Three rarely stated design decisions decide whether iterative improvement works across domains","key_machinery":"The generative-optimization learning loop: the configuration that fixes what an LLM may edit and what execution feedback (credit horizon and batched trial/error evidence) is supplied at each update. This loop is the mechanism that either enables or blocks iterative artifact improvement.","core_discovery":"Brittleness in iterative generative optimization with LLMs arises because setting up a learning loop forces hidden design decisions about editable scope and learning evidence. Three factors that affect most applications—the starting artifact, the credit horizon over execution traces, and the batching of trials into evidence—can decide whether optimization succeeds at all. Different starting artifacts change which solutions are reachable, truncated traces can still improve agents, and larger minibatches do not monotonically improve generalization. Because no simple universal setup exists across domains, productionization and adoption remain limited.","pith_inferences":["Tooling that forces teams to declare editable scope, credit horizon, and batching policy up front could cut silent failures when generative optimization is adopted.","The same hidden-choice problem likely appears in non-LLM iterative search that relies on black-box feedback, pointing to a broader design-pattern gap.","A meta-optimizer that searches over starting artifacts and credit horizons themselves may eventually automate the currently manual setup step.","Adoption rates may rise faster from shared defaults and documentation than from new algorithmic variants alone."],"forward_implications":["Engineers must treat starting artifact, credit horizon, and batch size as first-class, documented decisions rather than afterthoughts.","Self-improving agents will stay rare until domain-specific defaults or checklists for these choices become standard practice.","Optimization papers that omit these setup details cannot be reliably reproduced or transferred to new domains.","Practical guidance that makes the three factors explicit can raise the fraction of agents that successfully automate improvement.","Success on one task family does not transfer without re-examining the same three design choices."],"fun_headline_variants":["Hidden design choices decide if LLM generative optimizers work","Starting artifacts set reachable solutions in LLM optimization","Credit horizons and minibatches make or break iterative LLM loops","Truncated traces can still improve Atari agents with LLMs","No universal learning loop setup limits LLM optimizer adoption"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The broad claim that these three factors and the missing universal setup are the main reasons generative optimization stays rare rests on three case studies and one survey being representative of most real applications.","fun_headline_variants_meta":{"raw":{"variants":["Hidden design choices decide if LLM generative optimizers work","Starting artifacts set reachable solutions in LLM optimization","Credit horizons and minibatches make or break iterative LLM loops","Truncated traces can still improve Atari agents with LLMs","No universal learning loop setup limits LLM optimizer adoption"]},"model":"grok-4.5","effort":"low","cost_usd":0.0051,"raw_usage":{"total_tokens":1382,"prompt_tokens":798,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":51000000,"prompt_tokens_details":{"text_tokens":798,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":524,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":798,"tokens_out":60,"duration_ms":4903,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T19:08:01.929855+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A larger survey of production agents that records starting artifact, credit horizon, and batching policy for each system, then tests whether those three variables still predict success outside the paper’s three domains; if they do not, the general-hurdle claim fails.","supporting_citations":[],"review_version":1}