{"id":"6da88dae-408d-4f60-a4e5-6f07e51ecd49","arxiv_id":"2608.18066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Self-improving web agents show high run-to-run variance and degrade under shuffled task order; underspecification explains only part of the fragility.","lead":"This paper re-runs two memory-based self-improving web agents under repeated runs and shuffled task orders, and finds their reported gains are fragile: performance varies widely and often degrades under random order. It traces part of the problem to underspecified task and environment descriptions, and shows that adding rubrics and environment feedback only partially fixes it.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reversal claim (+1.5% to -4.5%) is underpowered: three runs per condition and two shuffled orders cannot distinguish it from noise, and Table 1's 17/24 count treats non-independent domain rows as independent evidence.","rationale":"I agree with the reader's CONDITIONAL verdict. The qualitative direction of the findings is plausible and the qualitative case studies (API memories, Haversine fallback, ambiguous task 118) are concrete. The paper also makes a sound methodological recommendation: report multiple runs and shuffled orders. However, the quantitative headline is not yet statistically grounded. The +1.5% default gain is explicitly non-significant (p=0.23), and the -4.5% shuffled deficit is based on two shuffle seeds with no significance test; the two shuffle means are close, but two draws from the space of orders are insufficient to establish a systematic order effect. The 17/24 variance-increase count is pseudoreplicated because domain rows within a benchmark come from the same three runs; an F-test on the largest variance ratio shows the effect is not significant. The 31% closure figure is not reproducible from the paper's own Fig. 5, and the underspecification interventions are tested on the same shuffled orders used for diagnosis, leaving open the possibility of overfitting. None of these issues are fatal to the paper's message, but they are exactly the kind of fragility the paper warns about—applied to its own evidence. With additional runs and shuffle seeds, the claims could be substantiated; until then, CONDITIONAL is the right verdict.","tokens_in":23147,"tokens_out":15540,"duration_ms":134631,"concrete_test":"Re-run RBank on WebArena under the default order and at least 10 randomly sampled task orders, with at least 3 runs each, then compute a bootstrap 95% CI for the mean pass-rate difference between shuffled and default conditions; if the CI overlaps zero, the task-order degradation claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim—that memory-based self-improvement turns an expected gain into a degradation—rests on three runs per condition and only two shuffled orders. On WebArena, the +1.5% RBank gain already carries p=0.23 over three runs, so the default-order 'gain' is not established; the -4.5% shuffled-order deficit is the average of two shuffle means (49.8 and 50.7 vs baseline 54.8) with no reported significance test. With three runs per condition, the standard error of each mean is large enough that the reversal could reflect sampling noise. Table 1's '17 of 24' variance-increase count compounds the problem: all domains within a benchmark share the same three runs, so the 24 comparisons are pseudoreplicated; the effective sample for the amplification claim is about 3 runs per method per benchmark, not 24. For the most striking variance ratio (GitLab RBank SD 3.34 vs baseline 1.98), an F-test with 2 and 2 degrees of freedom gives a one-sided p≈0.26 and a 95% CI spanning roughly (0.07, 111), so the variance amplification is not significant even in the strongest case. Finally, the 31% closure figure in Section 6 is not reproducible from Fig. 5: Shuffle-1 recovers 2.9/5.0=58%, Shuffle-2 recovers 1.1/4.1=27%, and no aggregation of these values yields 31%. The underspecification case studies are suggestive, but the quantitative fragility claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper re-evaluates two memory-based self-improving agents (Agent Workflow Memory and ReasoningBank) on three web-browsing benchmarks (WebArena, VisualWebArena, SCUBA) under a broader evaluation protocol that includes three repeated runs and two shuffled task orders. The authors report three main findings: (i) run-to-run variance is large for the no-memory baseline and is amplified by self-improvement in 17 of 24 domain-method cases; (ii) the default task order imposes an implicit easy-to-hard curriculum, and under shuffled orders RBank's WebArena gain turns from +1.5% into a -4.5% degradation; and (iii) manual inspection of memories identifies task and environment underspecification as a driver, and adding rubrics, environment feedback, and prompt modifications closes about 31% of the shuffled-order degradation. The paper argues for more rigorous multi-run, shuffled-order evaluation protocols and for human oversight interfaces.","tokens_in":23474,"tokens_out":4848,"duration_ms":41844,"significance":"If the quantitative findings hold, the paper makes a valuable contribution by showing that single-run, default-order evaluations of self-improving web agents can be misleading, and by documenting concrete memory failure modes (API suggestions in a browser-only environment, requests for user confirmation, Haversine fallback strategies) that are informative for future work. The qualitative case studies are descriptive and vivid, the code is released, and the recommendation to report multi-run statistics is sensible. However, the strength of the quantitative claims currently exceeds the statistical support: the headline reversal, the 17/24 amplification count, and the 31% closure figure all need more careful statistical treatment or re-framing as exploratory observations. The paper's own limitation statement in Appendix A acknowledges the qualitative, subset-based nature of the memory inspection, which is appropriate, but it does not address the statistical power issues in the main experiments.","major_comments":[{"comment":"The central claim that shuffled task order turns a +1.5% RBank gain on WebArena into a -4.5% degradation is not statistically supported as presented. The default-order gain is already not significant (p = 0.23, as the authors report in §3.2), and the shuffled-order deficit is an average of two shuffle means (49.8% and 50.7% vs. baseline 54.8%) with no reported significance test, confidence interval, or per-shuffle replication. With three runs per condition and only two fixed shuffled orders, the reversal could easily be sampling noise. Please report per-condition confidence intervals, significance tests (or bootstrap intervals), and ideally more runs and a larger sample of random task orders before claiming a significant degradation.","section":"§3.3 and Figure 3"},{"comment":"The '17 of 24 cases' count of variance amplification does not carry the evidentiary weight the text implies, because domain rows within a benchmark share the same three runs and are therefore pseudoreplicated; the effective sample for the amplification claim is on the order of three runs per method per benchmark, not 24 independent comparisons. Moreover, with n = 3 per condition, the sample standard deviations have large sampling error: even the most striking case (GitLab RBank SD 3.34 vs. baseline SD 1.98) yields an F-test with 2 and 2 degrees of freedom and a one-sided p of roughly 0.26. The qualitative trend is plausible, but the quantitative 'amplifies variance' claim should be framed as exploratory, with appropriate error bars on the variance estimates themselves.","section":"§3.2 and Table 1"},{"comment":"The 31% closure figure is not reproducible from the numbers reported in Figure 5. For Shuffle-1, the gap between RBank (49.8%) and baseline (54.8%) is 5.0 percentage points, and +All (52.7%) closes 2.9 of that gap (58%); for Shuffle-2, the gap is 4.1 points and +All (51.8%) closes 1.1 points (27%). No weighted or unweighted aggregation of these values yields 31%. The authors should show the exact calculation used for the 31% claim, or revise the claim to match the reported data; as written, this figure is a load-bearing part of the abstract and conclusion but is unverifiable.","section":"§4.2, Figure 5, and §6"},{"comment":"The validation of the underspecification hypothesis is partly circular. The failure modes that motivate +PMod (API suggestions, user-confirmation requests, external-site strategies) were identified by manually inspecting memories from the same shuffled runs on which +PMod is then tested, and the prompt modification explicitly forbids exactly those cited behaviors. The +All improvement on those same runs is therefore an in-sample fit rather than a fair out-of-sample test of the underspecification account. To support the causal claim, the authors should validate on a held-out task order (e.g., Shuffle-3 or a separate benchmark) or pre-register the rubric/environment/prompt interventions before inspecting the memories.","section":"§4.1–4.2 and Table 4"}],"minor_comments":[{"comment":"The section heading 'Indentifying Underspecification' contains a typo; it should read 'Identifying Underspecification'.","section":"§4 title"},{"comment":"The layout of Table 2 is confusing: the column headers 'AWM (Claude-3.5-Sonnet)' and 'RBank (Gemini-2.5-Pro)' with task counts '812 Tasks' and '684 Tasks' make it unclear whether the baseline rows are computed on different subsets; please clarify the comparison protocol in the caption.","section":"Table 2"},{"comment":"Figure 5 reports per-condition pass rates without showing the underlying run-level values or error bars. If each bar is an average over three runs (as in §3.1), please overlay the individual run points or add standard deviations so the reader can judge the overlap between conditions.","section":"Figure 5"},{"comment":"The notation 'Pass@3' and 'Passˆ3' appears in Tables 5 and 6 but is only briefly referenced in §3.1; please define both metrics in the table captions or in the surrounding text.","section":"Appendix D, Tables 5 and 6"},{"comment":"The sentence 'With very few exceptions [17, 18], prior work on web agent evaluation reports pass rate over one single run' cites [17] and [18] as exceptions; please make explicit what those exceptions report (e.g., multiple runs or variance-aware evaluation) so the reader can verify the claim.","section":"§3.1"},{"comment":"The limitation statement in Appendix A is honest but could be more direct: it should also acknowledge that the 31% closure figure and the 17/24 amplification count are based on a small number of runs and orders, since those are the quantitative pillars of the paper's conclusions.","section":"Appendix A"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading and worth sending to serious reviewers, but the headline numbers are less solid than the abstract implies. The qualitative direction — memory-based self-improvement amplifies run-to-run variance and is sensitive to task order — is plausible and consistent across three benchmarks. The specific reversal claim (+1.5% to -4.5%) rests on three runs per condition and two shuffled orders. On WebArena the default-order gain already has p=0.23, so the \"gain\" is not established; the shuffled-order deficit is an average of two shuffle means with no significance test. With n=3, the standard error of each mean is big enough that the reversal could be sampling noise. The 17-of-24 variance count in Table 1 is pseudoreplicated: all domains within a benchmark share the same three runs, so the effective sample is about three runs per method per benchmark, not 24. Even the largest variance ratio (GitLab, SD 3.34 vs 1.98) is not significant under an F-test with two and two degrees of freedom. So the paper's claim that it has demonstrated fragility quantitatively is not supported yet.\n\nWhat is genuinely new and valuable: applying known reliability concerns to memory-based self-improving agents, with new measurements and a nice diagnosis. The default-order implicit curriculum visualizations are convincing. The memory case studies — API suggestions in a browser-only environment, \"user confirmation\" leading to wait loops, the Haversine contagion — are concrete and useful. Using a stronger baseline is the right call and makes the diminishing-returns result meaningful. The paper is honest about its qualitative inspection and limitations.\n\nSoft spots beyond power: the 31% closure figure does not match the paper's own numbers. From Figures 5a and 5b, +All closes roughly 58% of the Shuffle-1 gap and 27% of the Shuffle-2 gap; no aggregation gives 31%. The underspecification validation is partly circular: failure modes were identified by inspecting memories from the same shuffled runs, then +PMod was written to forbid those behaviors and tested on the same orders. The claimed code release has a malformed URL in this version, so I couldn't check reproducibility. The citation pattern looks fine, with appropriate prior work on self-improvement and agent reliability.\n\nWho it is for: anyone evaluating self-improving agents or working on web agent benchmarks; it is a good discussion piece for a reading group. My verdict: it deserves a serious referee, not a desk reject, but revision needs more runs, more shuffled orders, a fixed closure figure, and a reanalysis that treats domains as pseudoreplicates. The qualitative message is probably right; the quantitative precision is not.","headline":"A useful and honest fragility study whose headline numbers rest on a very small sample; the qualitative message is probably right, but the quantitative claims need more runs and more shuffles.","tokens_in":24007,"tokens_out":2988,"duration_ms":27171},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Memory-based self-improving web agents are fragile: repeated runs amplify variance and shuffled orders turn gains into losses.","keywords":["self-improving agents","memory bank","evaluation variance","task order","underspecification","web agents","reliability","benchmark stress-testing"],"falsifier":"Run the same AWM and RBank protocols on WebArena with at least ten additional runs and six to ten random task-order shuffles. If the best-worst gaps and variance ratios do not systematically increase when memory is added, or if shuffled-order performance does not fall below the no-memory baseline in most seeds, the fragility claim would not survive. A second decisive check: if adding rubrics and environment feedback to memory construction fully closes the shuffled-order gap over many seeds, then underspecification would be the whole story rather than a partial cause.","tokens_in":22939,"feed_emoji":"📉","tokens_out":9022,"duration_ms":72874,"temperature":0.7,"pith_summary":"This paper re-evaluates two memory-based self-improving web agents—Agent Workflow Memory and ReasoningBank—under conditions broader than the originals used: three identical runs instead of one, and two shuffled task orders alongside the default order. It finds that the self-improving loop amplifies run-to-run variance in about 71% of the settings examined, with best-versus-worst run gaps reaching up to 10.4 percentage points, and that a method which gains +1.5% under the default order loses 4.5% on average under shuffled orders on WebArena. The paper traces part of this fragility to underspecification: memory-construction prompts do not tell the agent that the environment cannot execute APIs or request human confirmation, so the agent saves plausible-but-useless memories that later distract it. Adding rubrics, environment feedback, and clearer prompt constraints closes about 31% of the shuffled-order degradation, but a gap remains, which the paper takes as evidence that other factors contribute. The practical upshot is an argument for evaluating self-improving agents across multiple runs and randomized task orders before trusting any single-run result.","feed_headline":"Shuffled task order makes self-improving agents worsen","feed_subtitle":"Repeated runs and shuffled orders show memory-based agents amplify noise and lean on a hidden curriculum","key_machinery":"The load-bearing object is the memory bank maintained by the self-improvement loop: after each task, a memory construction module writes one to three textual items about the trajectory, and later tasks retrieve those items and condition their actions on them. Because memory construction is conditioned on stochastic outcomes and on earlier memories, small run-to-run or order-to-order differences compound, which is the mechanism behind the variance amplification and the order sensitivity. The second piece of machinery is the evaluation protocol itself—three runs and two shuffled orders against the default—which is what exposes the fragility. The third is the underspecification diagnosis: the generic memory prompts omit environment constraints such as no APIs and no human confirmation, and omit task rubrics, so the agent learns superficially plausible but ineffective strategies; adding that information is the intervention that recovers part of the loss.","core_discovery":"On its own terms, the paper's claim is that memory-based self-improvement, as currently built, is not a reliable improvement mechanism: it can increase variance rather than reduce it, and its gains depend on the accident of task order. Concretely, when the same AWM and RBank protocols are run three times, the variance metrics rise relative to a no-memory baseline in 17 of 24 cases; on the WebArena GitLab subset the best-worst gap grows from 4.4% for the baseline to 7.8% with ReasoningBank. Under the default order, RBank appears to improve WebArena by 1.5%, but across shuffled orders its average is 4.5% below the baseline, and the paper shows that the default order itself encodes an easy-to-hard curriculum. The proposed explanation, supported by manual inspection of stored memories, is that task and environment underspecification makes memory construction generate plausible but inapplicable lessons—API suggestions in a browser-only environment, medical advice for a shopping task, a Haversine-distance fallback that occasionally satisfies a string-matching rubric—and these lessons then spread to later tasks. Providing rubrics, environment feedback, and explicit constraints in the memory prompt closes 31% of the observed degradation, which the paper reads as partial confirmation of the underspecification mechanism while acknowledging that the remaining gap is unexplained.","pith_inferences":["If the variance-amplification count were recomputed with many more than three runs, the exact 17-of-24 figure would likely shift; the paper's own p-value of 0.23 for the headline +1.5% gain suggests that a larger re-run could shrink or even reverse some of the measured differences.","The Haversine and rubric-matching examples suggest a direct testable extension: a memory-validation filter that blocks items proposing actions unsupported by the environment, such as APIs or human confirmation, could be compared against the paper's prompt-modification intervention.","If task order acts as an implicit curriculum, then the default order of a benchmark should be reported alongside results; otherwise two agents that differ only in memory method cannot be fairly compared across papers that happen to use different orderings.","The paper's observations connect naturally to reward hacking: the agent that computed Haversine distances satisfied a string-matching rubric without performing the intended task, which suggests that memory-based self-improvement can amplify rubric exploits—something the paper observes but does not develop into a method."],"forward_implications":["Single-run pass rates on web-agent benchmarks should be read as high-variance estimates rather than stable scores, especially when a self-improving memory loop is involved.","Methods that appear to improve under a benchmark's default task order may owe part of their gain to an implicit easy-to-hard curriculum, not to the memory mechanism itself.","Evaluating self-improving agents under at least two randomized task orders is a cheap stress test that can reveal degradation invisible in the default order.","Memory prompts should include environment constraints and task rubrics; doing so recovered about a third of the shuffled-order loss in this study.","The remaining gap after adding specifications implies that other, unidentified factors also drive fragility, so reporting uncertainty is not enough—memory content itself needs validation."],"supporting_citations":[{"why":"This is the Agent Workflow Memory method being re-evaluated; it supplies the AWM memory-construction protocol whose variance is tested.","marker":"[11]"},{"why":"This is the ReasoningBank method being re-evaluated; it supplies the RBank memory construction and retrieval protocol and the prior reported gain the paper re-examines.","marker":"[12]"},{"why":"This is the primary benchmark; it provides the WebArena tasks, the default order with its implicit curriculum, and the rubric evaluators used in the underspecification analysis.","marker":"[13]"},{"why":"This is the second benchmark; it provides the VisualWebArena tasks used to test multi-run variance and shuffled task order.","marker":"[14]"},{"why":"This is the enterprise-centric benchmark; it provides the SCUBA tasks and one of the baseline harnesses used in the variance tables.","marker":"[15]"},{"why":"This is cited for evidence that WebArena task queries are often ambiguous and evaluators can be buggy, which grounds the underspecification hypothesis.","marker":"[18]"},{"why":"This is the baseline agent harness for WebArena and VisualWebArena; it serves as the no-memory starting point that AWM and RBank are compared against.","marker":"[20]"},{"why":"This work argues that agent reliability should be a primary evaluation axis; the paper extends that perspective from single-session to multi-session self-improving agents.","marker":"[10]"}],"fun_headline_variants":["Self-improving agents amplify noise and rely on a hidden curriculum","Shuffling tasks exposes the fragility of self-improving agents","Underspecification makes self-improving agents store useless lessons","Default task order hides a curriculum self-improving agents depend on"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that three runs and two shuffled orders are enough to characterize variance and order sensitivity; with only three runs, the sample standard deviation is itself noisy, and the 17-of-24 amplification count and the 4.5% degradation could be substantially sampling artifact rather than stable properties.","fun_headline_variants_meta":{"raw":{"variants":["Self-improving agents amplify noise and rely on a hidden curriculum","Shuffling tasks exposes the fragility of self-improving agents","Underspecification makes self-improving agents store useless lessons","Default task order hides a curriculum self-improving agents depend on"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00163,"raw_usage":{"total_tokens":6574,"prompt_tokens":1128,"completion_tokens":5446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":5375}},"tokens_in":744,"tokens_out":5446,"duration_ms":33163,"temperature":1.0,"reasoning_tokens":5375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T20:04:23.004570+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same AWM and RBank protocols on WebArena with at least ten additional runs and six to ten random task-order shuffles. If the best-worst gaps and variance ratios do not systematically increase when memory is added, or if shuffled-order performance does not fall below the no-memory baseline in most seeds, the fragility claim would not survive. A second decisive check: if adding rubrics and environment feedback to memory construction fully closes the shuffled-order gap over many seeds, then underspecification would be the whole story rather than a partial cause.","supporting_citations":[{"cited_title":"Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig","cited_arxiv_id":null,"evidence_quote":"This is the primary benchmark; it provides the WebArena tasks, the default order with its implicit curriculum, and the rubric evaluators used in the underspecification analysis."},{"cited_title":"VisualWebArena: Evaluating multimodal agents on realistic visual web tasks","cited_arxiv_id":null,"evidence_quote":"This is the second benchmark; it provides the VisualWebArena tasks used to test multi-run variance and shuffled task order."},{"cited_title":"Webarena verified, 2025","cited_arxiv_id":null,"evidence_quote":"This is cited for evidence that WebArena task queries are often ambiguous and evaluators can be buggy, which grounds the underspecification hypothesis."},{"cited_title":"W ALT: Web agents that learn tools","cited_arxiv_id":null,"evidence_quote":"This is the baseline agent harness for WebArena and VisualWebArena; it serves as the no-memory starting point that AWM and RBank are compared against."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}