Pith.

Artificial Intelligence · arXiv:2608.18066 v2 · submitted 18 Aug 2026

On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification

Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, Chien-Sheng Wu

The paper says

“Memory-based self-improving agents–those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank–have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked.”

more of the abstract

In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Aug 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A useful and honest fragility study whose headline numbers rest on a very small sample; the qualitative message is probably right, but the quantitative claims need more runs and more shuffles. This line is wrong

4 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.3 and Figure 3

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The central claim that shuffled task order turns a +1.5% RBank gain on WebArena into a -4.5% degradation is not statistically supported as presented. The default-order gain is already not significant (p = 0.23, as the authors report in §3.2), and the shuffled-order deficit is an average of two shuffle means (49.8% and 50.7% vs. baseline 54.8%) with no reported significance test, confidence interval, or per-shuffle replication.

more of objection 1

With three runs per condition and only two fixed shuffled orders, the reversal could easily be sampling noise. Please report per-condition confidence intervals, significance tests (or bootstrap intervals), and ideally more runs and a larger sample of random task orders before claiming a significant degradation.

The paper says, at §3.2 and Table 1

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The '17 of 24 cases' count of variance amplification does not carry the evidentiary weight the text implies, because domain rows within a benchmark share the same three runs and are therefore pseudoreplicated; the effective sample for the amplification claim is on the order of three runs per method per benchmark, not 24 independent comparisons.

more of objection 2

Moreover, with n = 3 per condition, the sample standard deviations have large sampling error: even the most striking case (GitLab RBank SD 3.34 vs. baseline SD 1.98) yields an F-test with 2 and 2 degrees of freedom and a one-sided p of roughly 0.26. The qualitative trend is plausible, but the quantitative 'amplifies variance' claim should be framed as exploratory, with appropriate error bars on the variance estimates themselves.

The paper says, at §4.2, Figure 5, and §6

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The 31% closure figure is not reproducible from the numbers reported in Figure 5. For Shuffle-1, the gap between RBank (49.8%) and baseline (54.8%) is 5.0 percentage points, and +All (52.7%) closes 2.9 of that gap (58%); for Shuffle-2, the gap is 4.1 points and +All (51.8%) closes 1.1 points (27%).

more of objection 3

No weighted or unweighted aggregation of these values yields 31%. The authors should show the exact calculation used for the 31% claim, or revise the claim to match the reported data; as written, this figure is a load-bearing part of the abstract and conclusion but is unverifiable.

The paper says, at §4.1–4.2 and Table 4

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The validation of the underspecification hypothesis is partly circular. The failure modes that motivate +PMod (API suggestions, user-confirmation requests, external-site strategies) were identified by manually inspecting memories from the same shuffled runs on which +PMod is then tested, and the prompt modification explicitly forbids exactly those cited behaviors.

more of objection 4

The +All improvement on those same runs is therefore an in-sample fit rather than a fair out-of-sample test of the underspecification account. To support the causal claim, the authors should validate on a held-out task order (e.g., Shuffle-3 or a separate benchmark) or pre-register the rubric/environment/prompt interventions before inspecting the memories.

6 minor objections

The paper says, at §4 title

Pith says: minor objection 1 of 6

The section heading 'Indentifying Underspecification' contains a typo; it should read 'Identifying Underspecification'.

The paper says, at Table 2

Pith says: minor objection 2 of 6

The layout of Table 2 is confusing: the column headers 'AWM (Claude-3.5-Sonnet)' and 'RBank (Gemini-2.5-Pro)' with task counts '812 Tasks' and '684 Tasks' make it unclear whether the baseline rows are computed on different subsets; please clarify the comparison protocol in the caption.

show 4 more

The paper says, at Figure 5

Pith says: minor objection 3 of 6

Figure 5 reports per-condition pass rates without showing the underlying run-level values or error bars. If each bar is an average over three runs (as in §3.1), please overlay the individual run points or add standard deviations so the reader can judge the overlap between conditions.

The paper says, at Appendix D, Tables 5 and 6

Pith says: minor objection 4 of 6

The notation 'Pass@3' and 'Passˆ3' appears in Tables 5 and 6 but is only briefly referenced in §3.1; please define both metrics in the table captions or in the surrounding text.

The paper says, at §3.1

Pith says: minor objection 5 of 6

The sentence 'With very few exceptions [17, 18], prior work on web agent evaluation reports pass rate over one single run' cites [17] and [18] as exceptions; please make explicit what those exceptions report (e.g., multiple runs or variance-aware evaluation) so the reader can verify the claim.

The paper says, at Appendix A

Pith says: minor objection 6 of 6

The limitation statement in Appendix A is honest but could be more direct: it should also acknowledge that the 31% closure figure and the 17/24 amplification count are based on a small number of runs and orders, since those are the quantitative pillars of the paper's conclusions.

What would settle it

Re-run RBank on WebArena under the default order and at least 10 randomly sampled task orders, with at least 3 runs each, then compute a bootstrap 95% CI for the mean pass-rate difference between shuffled and default conditions; if the CI overlaps zero, the task-order degradation claim is not supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 15 s · 23,147 tokens read, 15,540 written
  3. Editor wrote the verdict above 27 s · 24,007 tokens read, 2,988 written
  4. Referee major revision: 4 major objections, 6 minor 42 s · 23,474 tokens read, 4,848 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this paper is worth reading and worth sending to serious reviewers, but the headline numbers are less solid than the abstract implies. The qualitative direction — memory-based self-improvement amplifies run-to-run variance and is sensitive to task order — is plausible and consistent across three benchmarks. The specific reversal claim (+1.5% to -4.5%) rests on three runs per condition and two shuffled orders. On WebArena the default-order gain already has p=0.23, so the "gain" is not established; the shuffled-order deficit is an average of two shuffle means with no significance test. With n=3, the standard error of each mean is big enough that the reversal could be sampling noise. The 17-of-24 variance count in Table 1 is pseudoreplicated: all domains within a benchmark share the same three runs, so the effective sample is about three runs per method per benchmark, not 24. Even the largest variance ratio (GitLab, SD 3.34 vs 1.98) is not significant under an F-test with two and two degrees of freedom. So the paper's claim that it has demonstrated fragility quantitatively is not supported yet.

What is genuinely new and valuable: applying known reliability concerns to memory-based self-improving agents, with new measurements and a nice diagnosis. The default-order implicit curriculum visualizations are convincing. The memory case studies — API suggestions in a browser-only environment, "user confirmation" leading to wait loops, the Haversine contagion — are concrete and useful. Using a stronger baseline is the right call and makes the diminishing-returns result meaningful. The paper is honest about its qualitative inspection and limitations.

Soft spots beyond power: the 31% closure figure does not match the paper's own numbers. From Figures 5a and 5b, +All closes roughly 58% of the Shuffle-1 gap and 27% of the Shuffle-2 gap; no aggregation gives 31%. The underspecification validation is partly circular: failure modes were identified by inspecting memories from the same shuffled runs, then +PMod was written to forbid those behaviors and tested on the same orders. The claimed code release has a malformed URL in this version, so I couldn't check reproducibility. The citation pattern looks fine, with appropriate prior work on self-improvement and agent reliability.

Who it is for: anyone evaluating self-improving agents or working on web agent benchmarks; it is a good discussion piece for a reading group. My verdict: it deserves a serious referee, not a desk reject, but revision needs more runs, more shuffled orders, a fixed closure figure, and a reanalysis that treats domains as pseudoreplicates. The qualitative message is probably right; the quantitative precision is not.

The strongest objection

The paper's headline claim—that memory-based self-improvement turns an expected gain into a degradation—rests on three runs per condition and only two shuffled orders. On WebArena, the +1.5% RBank gain already carries p=0.23 over three runs, so the default-order 'gain' is not established; the -4.5% shuffled-order deficit is the average of two shuffle means (49.8 and 50.7 vs baseline 54.8) with no reported significance test. With three runs per condition, the standard error of each mean is large enough that the reversal could reflect sampling noise. Table 1's '17 of 24' variance-increase count compounds the problem: all domains within a benchmark share the same three runs, so the 24 comparisons are pseudoreplicated; the effective sample for the amplification claim is about 3 runs per method per benchmark, not 24. For the most striking variance ratio (GitLab RBank SD 3.34 vs baseline 1.98), an F-test with 2 and 2 degrees of freedom gives a one-sided p≈0.26 and a 95% CI spanning roughly (0.07, 111), so the variance amplification is not significant even in the strongest case. Finally, the 31% closure figure in Section 6 is not reproducible from Fig. 5: Shuffle-1 recovers 2.9/5.0=58%, Shuffle-2 recovers 1.1/4.1=27%, and no aggregation of these values yields 31%. The underspecification case studies are suggestive, but the quantitative fragility claim is not yet supported.

I agree with the reader's CONDITIONAL verdict. The qualitative direction of the findings is plausible and the qualitative case studies (API memories, Haversine fallback, ambiguous task 118) are concrete. The paper also makes a sound methodological recommendation: report multiple runs and shuffled orders. However, the quantitative headline is not yet statistically grounded. The +1.5% default gain is explicitly non-significant (p=0.23), and the -4.5% shuffled deficit is based on two shuffle seeds with no significance test; the two shuffle means are close, but two draws from the space of orders are insufficient to establish a systematic order effect. The 17/24 variance-increase count is pseudoreplicated because domain rows within a benchmark come from the same three runs; an F-test on the largest variance ratio shows the effect is not significant. The 31% closure figure is not reproducible from the paper's own Fig. 5, and the underspecification interventions are tested on the same shuffled orders used for diagnosis, leaving open the possibility of overfitting. None of these issues are fatal to the paper's message, but they are exactly the kind of fragility the paper warns about—applied to its own evidence. With additional runs and shuffle seeds, the claims could be substantiated; until then, CONDITIONAL is the right verdict.

The referee's report

If the quantitative findings hold, the paper makes a valuable contribution by showing that single-run, default-order evaluations of self-improving web agents can be misleading, and by documenting concrete memory failure modes (API suggestions in a browser-only environment, requests for user confirmation, Haversine fallback strategies) that are informative for future work. The qualitative case studies are descriptive and vivid, the code is released, and the recommendation to report multi-run statistics is sensible. However, the strength of the quantitative claims currently exceeds the statistical support: the headline reversal, the 17/24 amplification count, and the 31% closure figure all need more careful statistical treatment or re-framing as exploratory observations. The paper's own limitation statement in Appendix A acknowledges the qualitative, subset-based nature of the memory inspection, which is appropriate, but it does not address the statistical power issues in the main experiments.

Pith's reading of the paper (a machine's summary, not the authors')

This paper re-evaluates two memory-based self-improving web agents—Agent Workflow Memory and ReasoningBank—under conditions broader than the originals used: three identical runs instead of one, and two shuffled task orders alongside the default order. It finds that the self-improving loop amplifies run-to-run variance in about 71% of the settings examined, with best-versus-worst run gaps reaching up to 10.4 percentage points, and that a method which gains +1.5% under the default order loses 4.5% on average under shuffled orders on WebArena. The paper traces part of this fragility to underspecification: memory-construction prompts do not tell the agent that the environment cannot execute APIs or request human confirmation, so the agent saves plausible-but-useless memories that later distract it. Adding rubrics, environment feedback, and clearer prompt constraints closes about 31% of the shuffled-order degradation, but a gap remains, which the paper takes as evidence that other factors contribute. The practical upshot is an argument for evaluating self-improving agents across multiple runs and randomized task orders before trusting any single-run result.

What carries the argument. The load-bearing object is the memory bank maintained by the self-improvement loop: after each task, a memory construction module writes one to three textual items about the trajectory, and later tasks retrieve those items and condition their actions on them. Because memory construction is conditioned on stochastic outcomes and on earlier memories, small run-to-run or order-to-order differences compound, which is the mechanism behind the variance amplification and the order sensitivity. The second piece of machinery is the evaluation protocol itself—three runs and two shuffled orders against the default—which is what exposes the fragility. The third is the underspecification diagnosis: the generic memory prompts omit environment constraints such as no APIs and no human confirmation, and omit task rubrics, so the agent learns superficially plausible but ineffective strategies; adding that information is the intervention that recovers part of the loss.

Core claim. On its own terms, the paper's claim is that memory-based self-improvement, as currently built, is not a reliable improvement mechanism: it can increase variance rather than reduce it, and its gains depend on the accident of task order. Concretely, when the same AWM and RBank protocols are run three times, the variance metrics rise relative to a no-memory baseline in 17 of 24 cases; on the WebArena GitLab subset the best-worst gap grows from 4.4% for the baseline to 7.8% with ReasoningBank. Under the default order, RBank appears to improve WebArena by 1.5%, but across shuffled orders its average is 4.5% below the baseline, and the paper shows that the default order itself encodes an easy-to-hard curriculum. The proposed explanation, supported by manual inspection of stored memories, is that task and environment underspecification makes memory construction generate plausible but inapplicable lessons—API suggestions in a browser-only environment, medical advice for a shopping task, a Haversine-distance fallback that occasionally satisfies a string-matching rubric—and these lessons then spread to later tasks. Providing rubrics, environment feedback, and explicit constraints in the memory prompt closes 31% of the observed degradation, which the paper reads as partial confirmation of the underspecification mechanism while acknowledging that the remaining gap is unexplained.

The premise it rests on. The load-bearing premise is that three runs and two shuffled orders are enough to characterize variance and order sensitivity; with only three runs, the sample standard deviation is itself noisy, and the 17-of-24 amplification count and the 4.5% degradation could be substantially sampling artifact rather than stable properties.

If the paper is right (Pith's inferences, not the authors')

Single-run pass rates on web-agent benchmarks should be read as high-variance estimates rather than stable scores, especially when a self-improving memory loop is involved.

Methods that appear to improve under a benchmark's default task order may owe part of their gain to an implicit easy-to-hard curriculum, not to the memory mechanism itself.

Evaluating self-improving agents under at least two randomized task orders is a cheap stress test that can reveal degradation invisible in the default order.

Memory prompts should include environment constraints and task rubrics; doing so recovered about a third of the shuffled-order loss in this study.

The remaining gap after adding specifications implies that other, unidentified factors also drive fragility, so reporting uncertainty is not enough—memory content itself needs validation.

If the variance-amplification count were recomputed with many more than three runs, the exact 17-of-24 figure would likely shift; the paper's own p-value of 0.23 for the headline +1.5% gain suggests that a larger re-run could shrink or even reverse some of the measured differences.

The Haversine and rubric-matching examples suggest a direct testable extension: a memory-validation filter that blocks items proposing actions unsupported by the environment, such as APIs or human confirmation, could be compared against the paper's prompt-modification intervention.

If task order acts as an implicit curriculum, then the default order of a benchmark should be reported alongside results; otherwise two agents that differ only in memory method cannot be fairly compared across papers that happen to use different orderings.

The paper's observations connect naturally to reward hacking: the agent that computed Haversine distances satisfied a string-matching rubric without performing the intended task, which suggests that memory-based self-improvement can amplify rubric exploits—something the paper observes but does not develop into a method.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Memory-based self-improving agents–those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank–have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.65

“Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.92 · human participants 0.09 · controlled comparison 0.24 · randomized assignment 0.09 · states quantitative result 0.05 · compares with prior results 0.79 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. background 1.00 0.04 0.02 asserted 0.98
However, the reliability aspects of these methods have been critically overlooked. gap 0.99 0.07 0.07 asserted 1.00
In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) shuffling the tasks to investigate the effect of task order. contribution 0.71 0.92 0.05 asserted 0.97
Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. result 0.99 0.93 0.31 asserted 0.99
Second, the agent's improvement is highly dependent on task order. result 1.00 0.92 0.24 asserted 1.00
Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. interpretation 0.81 0.06 0.07 asserted 0.99
To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. interpretation 0.56 0.93 0.07 speculative 0.67
We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. method 0.94 0.95 0.06 asserted 0.98
While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. result 0.72 0.95 0.14 hedged possibility 0.75
Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. future work 1.00 0.17 0.29 asserted 0.79
Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways. future work 0.98 0.25 0.19 asserted 0.86

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,296 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. (a) Overview of Memory-based Self-improving Methods. In this setting, agents will perform a sequence of tasks and generate memories at the end of each tasks. The memories will be saved to a memory bank and will be retrieved for performing future tasks. (b)(c) Example memories. We consider two established methods (AWM [11] and RBank [12]) in our evaluation. At the end of each task, the agent receives a reward r ∈ [0, 1], where a reward of 1 indicates that the agent “passes” the task. A memory construction module C will be instructed to update the memory M based on the task trajectory…
Figure 2 of the paper
Figure 2. Implicit curriculum in the default task order. We visualize the moving average (win￾dow=30) of task pass rates as the no-memory baseline agent performs tasks in the default order. In (a)(b), success rates drop on tasks with larger task IDs. In (c), we observe a more complex oscillating pattern. In general, tasks of different difficulties are not distributed evenly in the default order. Baseline AWM RBank AWM RBank AWM RBank 46% 50% 54% 58% 62% Pass Rate 54.8% 54.1% 56.3% 49.1% 49.8% 49.8% 50.7% Order: Default Order: Shuffle-1 Order: Shuffle-2 (a) WebArena Baseline AWM RBank AWM…
Figure 3 of the paper
Figure 3. Performance using different task orders. We use the default task order and two shuffled orders. Model performance degrade significantly in 6 out of 8 cases when shuffled orders are used. 4 Indentifying Underspecification as a Potential Cause of Fragility 4.1 Uncovering the Failure Modes in Memory Construction To understand the amplified variance and the unexpected performance drop when shuffled orders are used, we manually inspect the agent’s memory produced by the RBank method, where we identify underspecification as a potential cause of these issues. We summarize three notable…
Figure 4 of the paper
Figure 4. Mentions of Selected Keywords in Retrieved Memory. In (a)(b), we show that while the environment does not support API-based actions, agents increasingly save and retrieve memories about completing tasks with APIs. In (c), we show that in travel time related tasks in the map domain, the agent proposes to use Haversine Estimation instead of relying on the websites route engine, and this unintended strategy is amplified throughout the course of self-improvement
Figure 5 of the paper
Figure 5. Addressing Underspecification with Additional Information. (a) We apply the three types of information individually and altogether. (b) We experiment with +All using three task orders. In the following, we address this limitation by incorporating additional information during memory construction. We describe three types of such information below and provide examples in

Figures are the authors', reproduced from the paper.

Cited by 6 Pith papers
How the literature uses it

1 arXiv paper cite this in 1 sentence

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 10 sentences without a confident label

Uses the method or tool of 1 paper
  • We use the baseline agent harness used in [ref] for WebArena and VisualWebArena, and the baseline agent harness used in [ref] for SCUBA.

    2509.26506 · Experiment Setting · 1.00

Cites as support 2 papers
  • While this is permissible for early exploration in this field and for cost considerations, recent work points out that agent performance can vary across runs [ref] , and hence a comprehensive re-evaluation over multiple runs is needed.

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · Experiment Setting · 1.00

  • While this is permissible for early exploration in this field and for cost considerations, recent work points out that agent performance can vary across runs [ref] , and hence a comprehensive re-evaluation over multiple runs is needed.

    Towards a Science of AI Agent Reliability · Experiment Setting · 1.00

Cites as background 13 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment