{"id":"7650b043-b3db-42c2-a6c2-02447676df91","arxiv_id":"2611.23068","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The Resolution Hypothesis states that agent evolution is bottlenecked by identifying improvements, not proposing them, and the paper provides large-scale evidence that evaluation noise prevents reliable selection at both the candidate and method levels.","lead":"This paper proposes the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), the authors find that selection is no better than random at identifying improvements, and that published rankings often rest on luck.","discovery_kind":"new_method","skeptic_critique":{"audit":{"quote":"The candidate-level evidence is evaluated on Terminal-Bench v2.1 only: the Proposal Pool in Section~\\ref{sec:proposals} evaluates $n{=}100$ skills proposed from 10 Terminal-Bench training tasks across 10 disjoint test tasks, the Selection Pool in Section~\\ref{sec:selection} evaluates $n{=}100$ skills proposed from Terminal-Bench and DeepSWE trajectories across 11 tasks, and the trajectory re-evaluation in Section~\\ref{sec:accumulation} covers two methods and two models. Whether the conclusions hold for larger pools, different generation strategies, or other benchmark domains remains an open question.","status":"already_addressed","question":null,"consequence":null,"objection_id":"O1","published_as":"withdrawn"},"model":"deepseek/deepseek-v4-flash","headline":"Withdrawn: the paper already says this.","rationale":"","tokens_in":52595,"tokens_out":596,"duration_ms":8107,"concrete_test":"","verdict_should_be":null,"load_bearing_attack":"The paper's words: “The candidate-level evidence is evaluated on Terminal-Bench v2.1 only: the Proposal Pool in Section~\\ref{sec:proposals} evaluates $n{=}100$ skills proposed from 10 Terminal-Bench training tasks across 10 disjoint test tasks, the Selection Pool in Section~\\ref{sec:selection} evaluates $n{=}100$ skills proposed from Terminal-Bench and DeepSWE trajectories across 11 tasks, and the trajectory re-evaluation in Section~\\ref{sec:accumulation} covers two methods and two models. Whether the conclusions hold for larger pools, different generation strategies, or other benchmark domains remains an open question.”","agreement_with_reader":null},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes the Resolution Hypothesis, which states that agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at two levels: (1) at the method level, typical margins between methods are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper includes a large-scale empirical study with seven evolution methods, six models, over 10M LLM calls, and 1,030 independent evolution runs. The authors also audit large reported gains and find they often come from evaluation shortcuts that vanish after repair.","tokens_in":52881,"tokens_out":2281,"duration_ms":17769,"significance":"Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.","major_comments":[],"minor_comments":[{"quote":"In total, the experiments consumed over 10M LLM calls and 400K CPU core hours.","comment":"The paper states 'over 10M LLM calls and 400K CPU core hours' but does not provide a breakdown of how these resources were distributed across experiments. A table showing the cost per experiment group would be helpful for reproducibility and for readers assessing the feasibility of similar studies.","section":"Section 3.2 (Experiments)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":null,"comment":"Table 1 (tab:variance-text) is very dense and difficult to read. Consider splitting it by benchmark or method family, or using a visual representation (e.g., a heatmap) to make the patterns more accessible.","section":"Table 1 (Variance measurement)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":null,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Execution (1 rollout) & 49.7\\,$\\pm$\\,1.9 & 62.1\\,$\\pm$\\,42.9 \\\\","comment":"Table 2 reports pairwise accuracy and false acceptance rates with credible intervals. The false acceptance rate for execution-based selection is 62.1% with a very wide interval (±42.9%). The paper should comment on why this interval is so wide and whether this affects the conclusions.","section":"Section 5.2 (Selection) and Table 2","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Across the 177 questions of the released splits, this option appears in 85 and is the correct answer in all 85; it appears in 46 of the 106 test questions.","comment":"The discussion of LiveMathematicianBench is important but somewhat buried in the appendix. The finding that the 'stronger result' option is always correct when it appears is a significant evaluation shortcut that undermines the benchmark. This should be highlighted more prominently, perhaps in the main text.","section":"Appendix D.2 (Per-Benchmark Results)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Unlike Appendix~\\ref{app:estimators}, which reports $\\hat{\\tau}$ itself and therefore sets negative estimates to zero, we retain negative corrected estimates here, because we compare $\\hat{\\tau}_t^2$ across rounds and clipping would bias the small early-round estimates upward.","comment":"Figure 7 (tau-growth) shows run-to-run variance growing over rounds. The paper should clarify whether the noise-corrected estimate can become negative (as noted in Appendix F) and how this affects the interpretation of early rounds.","section":"Section 5.3 (Noise Accumulates Across Rounds)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Across 171 benchmark--model results in the main tables of nine agent-evolution papers, the proposed method leads its strongest competitor by a median of 3.4 pp, with an interquartile range of 1.2 to 6.6 pp.","comment":"The paper lists several limitations but does not discuss the potential for publication bias in the margin survey (Appendix C.3). Papers that find small or negative margins may be less likely to be published, so the reported margins may overestimate typical effect sizes. This should be acknowledged.","section":"Section 7 (Limitations)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"A competitor is another optimization method (prompt, skill, memory, or harness optimization); zero-shot prompting, human-written prompts or harnesses, and fixed retrieval baselines are not competitors.","comment":"The survey of reported margins is useful, but the paper should clarify whether the 'strongest competitor' is always another evolution method or could be a non-evolution baseline. The definition says 'another optimization method (prompt, skill, memory, or harness optimization); zero-shot prompting, human-written prompts or harnesses, and fixed retrieval baselines are not competitors.' This is clear but should be stated in the main text as well.","section":"Appendix C.3 (Margins Reported in Agent-Evolution Papers)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"We define the \\emph{resolution} of an evaluation pipeline as the smallest true improvement it can reliably distinguish from noise.","comment":"The paper uses the term 'resolution' informally. It might be helpful to define it more precisely, perhaps as the minimum detectable effect size given the variance components and a desired power level. The current definition ('the smallest true improvement it can reliably distinguish from noise') is a good start but could be formalized.","section":"Section 4.1 (Single Reports Cannot Rank Methods)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null}],"open_questions":null,"recommendation":"uncertain"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper makes a clear, well-supported argument that the bottleneck in agent evolution is evaluation resolution, not proposal quality. The authors run over 1,000 independent evolution runs and 2.1 million test trajectories across seven methods, six models, and four benchmarks—by a wide margin the largest reproducibility study of agent evolution I have seen. The central claim, the Resolution Hypothesis, is stated plainly and tested at both the method level (where typical margins are smaller than single-run noise) and the candidate level (where selection is near chance). The gain audit is particularly effective: showing that the largest reported gains come from evaluation shortcuts that a single sentence can reproduce and that vanish after repair is a clean demonstration of the hypothesis.\n\nThe paper is honest about its limitations. The candidate-level evidence comes from Terminal-Bench v2.1 only, the trajectory re-evaluation covers two methods and two models, and the authors acknowledge that the conclusions may not generalize to larger pools, different generation strategies, or program evolution. The code and data are promised upon acceptance but not yet released, which limits reproducibility today.\n\nA few soft spots worth noting. The candidate pools use n=100 skills; whether the Quality@K curves would continue to climb at larger K is untested. The empirical-Bayes estimator has a free parameter (kappa_max) that affects the separation between observed and null curves, though the authors test sensitivity across a range. The trajectory judges are evaluated on a single model (DeepSeek-V4-Flash) without an ACE training loop, so the null result for LLM judges may not be the last word.\n\nThis paper is for researchers working on self-improving agents, evaluation methodology, or reproducibility in AI. It deserves a serious referee and would benefit from code release and replication on additional benchmarks. I recommend peer review with a request for the code and data.","headline":"A large-scale empirical study argues that agent evolution stalls not because proposers fail to generate better artifacts, but because noisy evaluations cannot reliably identify them. The evidence is substantial and the argument is coherent, but the candidate-level results are limited to one benchmark and the code is not yet released.","tokens_in":53153,"tokens_out":463,"duration_ms":6528},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper argues that agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them.","keywords":["agent evolution","evaluation noise","resolution hypothesis","self-improving agents","LLM agents","benchmark reliability","selection noise","variance decomposition"],"falsifier":"If a future study on a different benchmark or with a different set of methods finds that single-run rankings reliably identify the best method (e.g., the top-ranked method in a single run agrees with the true ranking more than 80% of the time), that would contradict the paper's claim that typical method margins are below single-run resolution.","tokens_in":54352,"feed_emoji":"🔬","tokens_out":1315,"duration_ms":10029,"temperature":0.2,"pith_summary":"The paper proposes the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.","feed_headline":"Agent evolution stalls due to noisy evaluation, not poor proposals","feed_subtitle":"Paper argues that identifying improvements, not proposing them, is the bottleneck in self-improving agents","key_machinery":"The Resolution Hypothesis: the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them. The paper defines evaluation resolution as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. The paper uses a variance decomposition separating evaluation noise (σ²) from run-to-run variance (τ²) to measure resolution.","core_discovery":"The paper argues that agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. The authors provide evidence at both levels where evaluation scores are used: (1) at the method level, typical margins are smaller than the difference between two independent runs of the same method, so single-run rankings are largely determined by noise; (2) at the candidate level, selection is no better than random at identifying helpful artifacts, and errors accumulate across rounds. The paper argues that the bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.","headline_check":{"rule":"report_a_claim_v1","sent_back":false,"unattributed":{}},"pith_inferences":["The paper's findings suggest that many published rankings of agent evolution methods may be unreliable, as typical margins are smaller than single-run variance.","The paper's analysis implies that evaluation shortcuts (e.g., adapting to a grader that credits only one answer format) can produce large reported gains that vanish when the evaluation is repaired, calling into question the validity of such gains.","The paper's framework could be applied to other domains where noisy evaluation is used for selection, such as hyperparameter optimization or neural architecture search.","The paper's recommendation of component-wise evaluation (Quality@K for proposers, selected-artifact quality for selectors) could be adopted as standard practice in the field."],"forward_implications":["If the Resolution Hypothesis is correct, then improving agent evolution requires either enlarging the improvements being proposed or making smaller improvements reliably detectable.","The paper suggests that developing higher-resolution feedback may be as important as developing more powerful evolution algorithms.","The paper recommends publishing both evaluation noise (σ̂) and run-to-run variance (τ̂) alongside point scores so readers can judge whether a claimed margin exceeds the evaluation's resolution.","The paper proposes component-wise evaluation: Quality@K curves for proposers and selected-artifact quality curves for selectors, to split one comparison that falls below the resolution into two that exceed it."],"supporting_citations":[{"why":"Introduces SkillOpt, a method for evolving agent skills that the paper builds on and compares against.","marker":"skillopt2026"},{"why":"Introduces ReasoningBank, a memory-playbook evolution method included in the paper's experiments.","marker":"ouyang2026reasoningbank"},{"why":"Introduces Trace2Skill, a skill-distillation method included in the paper's experiments.","marker":"trace2skill2026"},{"why":"Introduces GEPA, a prompt-evolution method included in the paper's experiments.","marker":"agrawal2025gepareflectivepromptevolution"},{"why":"Introduces SkillOpt-Lite, a lightweight skill-evolution method included in the paper's experiments.","marker":"skilloptlite"},{"why":"Introduces AHE, a harness-evolution method included in the paper's experiments.","marker":"lin2026ahe"},{"why":"Introduces Meta-Harness, a harness-evolution method included in the paper's experiments.","marker":"lee2026metaharnessendtoendoptimizationmodel"},{"why":"Introduces HarnessLens, a harness-evolution method included in the paper's experiments.","marker":"xu2026verifysmarterevolvefurther"},{"why":"Provides the Terminal-Bench benchmark used extensively in the paper's experiments.","marker":"terminalbench"},{"why":"Provides the false-commit analysis that the paper connects to its findings on noisy selection.","marker":"shawn2026pace"}],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the evaluation noise and run-to-run variance measured on the four benchmarks (ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1) generalize to other benchmarks and settings.","cache_creation_input_tokens":0},"created_at":"2026-11-18T06:48:55.975049+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.0ef1a750a3e3","input_kind":"tex","input_chars":175395,"input_total_chars":175395,"passes_missing":["circularity"],"published_standing":"established_narrower","published_confidence":"high","objection_audit":{"version":"pith-objection-audit@1","counts":{"raised":5,"confirmed":0,"questions":0,"withdrawn":5,"moved_to_minor":0,"math_error_not_certified":0,"withdrawn_reviewer_error":0,"typesetting_minor_dropped":0,"withdrawn_already_addressed":5,"referee_assessment_unpublished":1},"open_questions":[],"standing":{"basis":["first_pass_verdict"],"standing":"established_narrower","confidence":"high"},"standing_rule":"1. No standing is published (not_assessed) when the review read only the abstract, when the referee pass or the objection check did not run, or when the first pass gave no verdict. 2. The first pass's verdict sets the start: ACCEPT is established, CONDITIONAL is established_narrower, REJECT is not_supported. 3. Each objection that survived the check against the paper's text caps the standing: one that breaks the central claim caps it at not_supported, one that narrows it at established_narrower, one that weakens it at supported_with_gaps. The standing is the lowest of the start and the caps. 4. A first-pass REJECT stands only if a surviving objection breaks the central claim. Without one, the standing is the lowest cap of the surviving objections, or not_assessed when none survived. 5. The confidence is the lowest of the first pass's confidence and the confidence of each objection that set the standing, and is no higher than moderate while a question is open or part of the paper's text was not supplied. 6. If any objection was withdrawn because Pith's reviewer misread the paper, the confidence is no higher than moderate, whatever the first pass said: a review that misread the paper once may have misread it elsewhere.","input_kind":"tex","nothing_to_audit":false,"bar":{"enabled":false}},"run_record":{"models":["deepseek/deepseek-v4-flash"],"passes":{"reader":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.005337,"tokens_in":57398,"tokens_out":2364,"duration_ms":17328,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56","message_sha256":"b7d0b09980ebfae3112275b8aa62306643675a44135b535530398348a89adbd3","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"referee":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.005227,"tokens_in":52881,"tokens_out":2281,"duration_ms":17769,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"a852d99aea0ddfd900e1483f76e20777aae3afde9c016d5b8d6e5db01064c728","message_sha256":"527f01e8563fd1e6bb7330247b63210af4b368473f65e2fcb798b72f0c86411a","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"skeptic":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.004895,"tokens_in":52595,"tokens_out":596,"duration_ms":8107,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"8f88c0f61488b1e764c317998cddf35fffc33a1f20a3b1be54330d7843386fc7","message_sha256":"7c4b3add1384e620706046bf616caa746038228b59ed6f1886f7acc3b11256ef","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"desk_editor":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.004921,"tokens_in":53153,"tokens_out":463,"duration_ms":6528,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"e7211f6a3d2734c1025ce7e7e6d48f78b0944d0a32d5021f81be63b27721420a","message_sha256":"27d9b5e06ed71d93740add5dd39d2daf2a09a5b970569311d79ab07d7a273fe9","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"objection_audit":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.005215,"tokens_in":53876,"tokens_out":1715,"duration_ms":10716,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"6c7a6883d5517cf9e4945d83947200a36dcd152b8b56c32baf7500d8b3841d0a","message_sha256":"92e31647110c663394e7c4cde94d9bebc7862eadceb08c1d19eb44327e1661dd","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"pith_extraction":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.005185,"tokens_in":54352,"tokens_out":1315,"duration_ms":10029,"temperature":0.2,"input_sha256":"a01447a48368bf6637718f84146cb30887aad0d538642bbf7588cd10a4163ad8","prompt_sha256":"e4f56bf568e7854d0a1d2efb6ec96f8697d5132737cdb31cb7fc64acbd923974","message_sha256":"5cfc218b02922772d5ac116dab85568cacaeb6a68c76866bb36a2782d2dec3d5","cache_read_input_tokens":0,"cache_creation_input_tokens":0}},"version":"pith-run-record/v1","prompt_set_sha256":"0ef1a750a3e3214336d2646653d579e6e33dc3898345c6db3c93489867ffb5e6","configured_pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium"},"basis":{"kind":"full_text","source":"recorded","chars_given":175395,"chars_total":175395},"receipt":{"review_sha256":"a44cd811e9219038a8cbf922644aa00ae001a7ce26b47f0dfbb92ebcb3ef66e4","rule":"pith-review-sha256/v1","hashed_at":"2026-11-18T06:50:14.769385+00:00","fields":["id","arxiv_id","paper_version","created_at","pipeline_version","model_set","run_record","input_kind","input_chars","input_total_chars","passes_missing","verdict","confidence","novelty_score","correctness_risk","formal_verification","parameter_count","discovery_kind","one_line_summary","lead","skeptic_critique","referee_report","author_rebuttal","desk_editor","pith_extraction","objection_audit","published_standing","published_confidence"],"matches_current":true},"falsifier":"If a future study on a different benchmark or with a different set of methods finds that single-run rankings reliably identify the best method (e.g., the top-ranked method in a single run agrees with the true ranking more than 80% of the time), that would contradict the paper's claim that typical method margins are below single-run resolution.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":"tex"},"url":"https://pith.science/accuracy"},"review_version":1}