Pith

open record

sign in
Browse

arxiv: 2607.07702 · v1 · pith:NGOF4PL2 · submitted 2026-07-08 · cs.CL

From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization

Ying Chang , Jiahang Xu , Xuan Feng , Chenyuan Yang , Peng Cheng , Yuqing Yang This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-09 01:30 UTCglm-5.2pith:NGOF4PL2record.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: Comparison of context construction strate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] reproduced from arXiv: 2607.07702
classification cs.CL
keywords agent optimizationcausal localizationexecution dependency graphbackward slicingroot cause analysistrajectory analysisLLM agentsprompt optimization
0
0 comments X

The pith

Causal Slicing Finds Root Causes in Agent Traces, Boosting Success 1.4x

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When LLM-based agents fail at long-horizon tasks, the visible error often appears far downstream from its actual origin — a crash at step 50 might stem from a bad parameter generated at step 5. This paper proposes STRACE, a framework that treats execution traces not as flat logs but as dependency graphs, enabling backward causal slicing to isolate the true root-cause module for optimization. The core mechanism is a textual Execution Dependency Graph (EDG) — a lightweight map of data and control flow between agent components, inferred from source code by an LLM. Using the EDG, STRACE traces failures backward from their manifestation point, pruning irrelevant steps and identifying the upstream module whose logical defect triggered the downstream symptom. It then synthesizes generalized heuristics and injects them into the root-cause module's prompt. On a formal verification benchmark (VeruSAGE-Bench), this approach lifted success rates from 42.5% to 58.5%, a 1.4x improvement over human-expert-designed agents, while keeping optimization costs bounded even as training traces scaled to 453.

Core claim

The central discovery is that the context-noise trade-off in agent optimization — the tension between feeding full traces (too noisy) and truncating them (loses causal evidence) — can be broken by constructing a dependency graph of the agent's modules and performing backward causal slicing from the failure manifestation point. This remaps optimization targets from symptom nodes to root-cause nodes: in the case study, 12 of 25 selected traces had their attributed failure module corrected to an upstream component. The EDG itself need not be perfectly accurate; perturbation experiments show that adding or removing 25% of edges causes only marginal performance change (56% to 54% at worst), while

What carries the argument

Textual Execution Dependency Graph (EDG): a text-based node-edge map of data and control dependencies among agent modules, inferred from source code. Used as a structural prior for backward slicing and root-cause isolation.

If this is right

  • Agent optimization frameworks can shift from symptom-level patching to structural root-cause repair, potentially reducing the number of brittle, overfit prompt updates in production agent systems.
  • The dependency-graph approach generalizes beyond prompt optimization to harness engineering — localized diagnoses could guide updates to tool-use policies, skills, and other editable agent components.
  • If the EDG construction step can be automated reliably across arbitrary agent codebases, this method becomes a plug-in diagnostic layer for any agent system with sufficient implementation visibility.
  • The robustness to graph perturbation (25% edge noise causing minimal degradation) suggests that approximate structural priors suffice, lowering the barrier to adoption for systems with partial or noisy code visibility.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The backward-slicing approach implicitly assumes that agent failures are predominantly single-root-cause events. If failures are frequently caused by the interaction of two or more modules jointly (a combinatorial defect), tracing backward from the manifestation node along individual dependency paths may still miss the true cause.
  • The framework's requirement for codebase-level visibility means it is best suited for open-source or internally-developed agent systems. Fully black-box API-based agents (e.g., commercial computer-use agents) would need an alternative structural prior source, such as inferring dependencies from trace patterns alone.
  • The stability under graph perturbation raises a question the paper does not fully address: if adding 10% of random edges improves performance (56% to 62%), the added edges may occasionally capture real dependencies the LLM missed, suggesting that the base graph construction is conservative and could benefit from complementary static analysis tools.
  • The method's effectiveness on formal verification (VeruSAGE-Bench) versus its more modest gains on web navigation (WebArena) suggests that causal localization provides the most value when failures propagate through deep, structured reasoning chains rather than through exploratory, stateful web interactions.

Load-bearing premise

The entire causal localization mechanism depends on an LLM reliably constructing a faithful dependency graph from the agent's source code. If the LLM misses a critical data or control dependency, the backward slicing will not find the true root cause, and the optimizer will patch the wrong module.

What would settle it

If the dependency graph constructed by the LLM systematically misses the dependency path connecting the true root-cause module to the failure manifestation node, then backward slicing will produce an incomplete or incorrect causal slice, and the resulting prompt optimization will target the wrong module.

Figures

Figures reproduced from arXiv: 2607.07702 by Chenyuan Yang, Jiahang Xu, Peng Cheng, Xuan Feng, Ying Chang, Yuqing Yang.

Figure 2
Figure 2. Figure 2: Illustration of the STRACE framework. The system optimizes long-horizon agents via four phases: (1) Structural Modeling, which constructs execution topology; (2) Failure Pattern Mining and Trace Filtering, which compresses failures into representative traces; (3) Causal Localization, which performs causal context distillation and root cause isolation; and (4) Inductive Policy Optimization, which injects he… view at source ↗
Figure 3
Figure 3. Figure 3: Success rate on 50 independent tasks and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distinguishing Symptoms from Root Causes. Top: Causal localization remaps 25 traces from 5 manifest nodes to 6 targets, exposing an up￾stream source. Bottom: A compute_repair loop is traced to assertion_reasoning_pipeline, enabling a targeted policy optimization to bypass the recursive path. 4.4 Case Study: Tracing Symptoms to Root Causes To empirically validate the distinction between symptom (vm) and cau… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of a Failure Summarization Map generated during the optimization of the IronKV project. STRACE first parses raw traces into structured execution records containing the global task outcome, local node status, and module invocation sequence. The map then summarizes the trace corpus along two analytical dimensions: Statistical Severity, which estimates the conditional probability of global failu… view at source ↗
Figure 6
Figure 6. Figure 6: Instruction for Phase 1–Structural Modeling. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Instruction for Phase 2 – Failure Pattern Mining and Trace Filtering. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Instruction for Phase 3 – Causal Localization. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Instruction for Phase 4 – Inductive Policy Optimization. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
read the original abstract

The optimization of long-horizon agents increasingly relies on reflection-based mechanisms, where a large language model (LLM) acts as an optimizer to diagnose agent failures and improve agent policies. However, real execution traces are difficult to use directly for optimization: large trace collections are often redundant and heterogeneous, making optimization inefficient and prone to overfitting to low-value failures; meanwhile, each individual trajectory also contains many irrelevant steps, while naive context reduction methods such as truncation or sliding windows can discard causally important evidence and produce misleading optimization signals. To resolve this dilemma, we introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework that constructs high signal-noise optimization contexts for more precise and effective optimization. At the batch level, STRACE mines failure patterns to filter redundant traces and retain representative failures; within each selected trace, it performs causal localization over a textual dependency graph to remove non-causal steps and identify the true root-cause module for optimization. Empirical results demonstrate that STRACE significantly outperforms standard context-filtering baselines. Notably, on a challenging formal verification task (VeruSAGE-Bench), it successfully optimizes human-expert designed agents, delivering $1.4\times$ success-rate improvement (42.5% to 58.5%). The code is available at https://github.com/moomight/STRACE .

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 7 minor

Summary. The paper introduces STRACE, a four-phase framework for optimizing long-horizon LLM agents. The method (1) constructs a textual Execution Dependency Graph (EDG) from the agent's source code, (2) filters redundant failure traces via statistical and structural pattern mining, (3) performs backward slicing over the EDG to localize root-cause modules, and (4) synthesizes generalized prompt heuristics for the identified modules. STRACE is evaluated on HotpotQA, WebArena, and VeruSAGE-Bench, with the headline result being a 16-point absolute SR improvement (42.5% to 58.5%) on VeruSAGE-Bench. Code is publicly available. The framework is well-motivated and the end-to-end pipeline is clearly described.

Significance. The paper addresses a real and practical problem: context-noise trade-offs in reflexive agent optimization. The VeruSAGE-Bench result (o4-mini with STRACE reaching 58.5% SR, competitive with Claude Sonnet 4 hands-off at 59.4%) is a concrete, falsifiable demonstration that structured trace analysis can close the gap between smaller and frontier models on domain-intensive tasks. The reproducible code release and the perturbation study (Table 4) on EDG robustness are commendable. The cost-performance analysis (Figure 3, Table 5) is a useful contribution for practitioners. However, the significance of the paper's central novel contribution—causal localization—is called into question by the ablation results, as detailed below.

major comments (4)
  1. Table 2 (ablation): The paper's central novel contribution is Phase 3 (Causal Localization), which distinguishes STRACE from prior work by identifying 'the true root-cause module' rather than 'optimizing the wrong target.' However, Table 2 shows that removing causal localization entirely (both 'Current Node' and 'Full Trace' variants) only drops SR from 56% to 54%—a 2-point gap. By contrast, removing trace filtering costs 10 points (56%→46%) and removing structural modeling costs 8 points (56%→48%). The §4.4 case study references a 200-case setting, meaning a 2-point gap corresponds to approximately 4 tasks. The paper does not report confidence intervals or significance tests for these ablation results. Without such analysis, it is unclear whether the 2-point difference is statistically meaningful. This is a load-bearing concern: if causal localization is statistically indistinguishable从
  2. Table 1 vs. Table 2 discrepancy: Table 1 reports STRACE at 58.5% SR on VeruSAGE-Bench, while Table 2 reports 56% SR for the full STRACE ablation. The paper does not explain this discrepancy. If the ablation uses a different subset or configuration, this should be stated explicitly. If the difference reflects variance, it further underscores the need for significance testing. This discrepancy also affects the interpretation of the ablation: if the full STRACE SR varies by 2.5 points between experimental configurations, the 2-point ablation gap for causal localization is within the noise floor.
  3. §3.4, Phase 3: The term 'causal localization' implies a formal causal inference method, but the actual mechanism is an LLM performing backward slicing over a heuristically constructed dependency graph and then doing semantic reasoning to identify the root cause. The paper does not formalize what constitutes a 'causal' relationship versus a mere data or control dependency. This matters because the backward slicing (Step 1) is purely structural—it retains any node with a dependency path to the manifestation node—while the root cause isolation (Step 2) is an LLM judgment call. The paper should clarify that 'causal' here refers to the LLM's semantic assessment of where the logic first deviated, not to a formal causal identification procedure. This would help readers calibrate expectations and facilitate comparison with formal causal inference approaches.
  4. Table 1, WebArena: STRACE underperforms on Reddit (36.4% vs. GEPA's 27.3% is a gain, but on GitLab STRACE scores 17.1% vs. GEPA's 9.8%, while on CMS STRACE ties TextGrad at 16.2%). More importantly, on Shopping, STRACE ties TextGrad at 30.8% but does not exceed it. The paper claims STRACE 'achieves the strongest overall performance' on WebArena (23.7% ALL vs. TextGrad's 17.3%), but this aggregate is dominated by the Reddit and GitLab subdomains. The paper should acknowledge that STRACE's advantage is domain-dependent and discuss why the structural approach helps more in some web tasks than others. Additionally, the WebArena sample sizes per subdomain appear small (the paper does not specify exact counts), which raises the same significance concerns as the ablation.
minor comments (7)
  1. §4.1: The bottleneck threshold k=5 and exemplar count s=5 are listed as hyperparameters but no sensitivity analysis is provided. A brief ablation or justification for these values would strengthen the paper.
  2. Table 3: The graph quality assessment is 'expert-reviewed' but the methodology is not described. How many experts? What was the review protocol? Were they given the ground-truth graph? This limits the reproducibility of the EDG quality claim.
  3. Figure 3: The x-axis label 'Number of Training Cases' goes up to 453, but it is unclear whether each point represents an independent run or a cumulative evaluation. Clarification needed.
  4. §3.2: The EDG construction relies on an LLM reading the agent's source code. The paper notes this requires 'sufficient visibility into the target agent system' (Limitations), but does not discuss failure modes when the codebase is large or when the LLM misidentifies module boundaries. A brief discussion of scaling behavior would help.
  5. Appendix B.2: The distinction between 'data dependencies' and 'control dependencies' is conceptually clear but the paper does not provide examples of edges that were missed or incorrectly inferred in practice. Including 1-2 such examples from the perturbation study would make the robustness claim more concrete.
  6. Table 7: STRACE-enhanced o4-mini (58.5%) is compared against GPT-5 (50.0%) and Claude Sonnet 4 (59.4%). The comparison with Claude Sonnet 4 is nearly a tie. The paper should note that STRACE's advantage over GPT-5 may partly reflect the base agent architecture (VeruSAGE) rather than STRACE alone, since GPT-5 is used in a hands-off mode without the VeruSAGE harness.
  7. The paper uses the term 'causal' throughout (causal localization, causal slice, causal context) in a sense that differs from the standard causal inference literature. A footnote or brief clarification on first use would improve precision.

Simulated Author's Rebuttal

4 responses · 1 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points: (1) the ablation gap for causal localization is small and lacks significance testing; (2) an unexplained discrepancy between Table 1 (58.5%) and Table 2 (56%) for full STRACE; (3) the term 'causal localization' may overstate the methodological contribution relative to what is formally a heuristic backward-slicing plus LLM semantic reasoning procedure; and (4) STRACE's WebArena advantage is domain-dependent and per-subdomain sample sizes are unspecified. We address each below.

read point-by-point responses

  1. Referee: Table 2 ablation: causal localization only contributes a 2-point gap (56% to 54%), corresponding to ~4 tasks out of 200, with no confidence intervals or significance tests. If this gap is within noise, the paper's central novel contribution is not empirically supported.

    Authors: The referee is correct that the 2-point ablation gap for causal localization is small relative to the gaps for trace filtering (10 points) and structural modeling (8 points), and that without significance testing we cannot formally rule out noise. We will address this in revision. However, we respectfully disagree that the ablation gap is the primary evidence for causal localization's value. The ablation measures end-to-end success rate, which conflates two distinct effects of Phase 3: (a) cost reduction via context compression and (b) correctness of root-cause attribution. Table 2 and Table 5 (Appendix C.1) show that replacing causal localization with full-trace context nearly doubles Phase 4 cost ($2.96 → $5.93) while matching SR, confirming the cost benefit. More importantly, Figure 4 provides direct evidence that causal localization changes the optimization target: in the 200-case setting, 12 of 25 traces are re-mapped from manifestation nodes to upstream root-cause nodes, expanding optimization targets from 5 to 6 distinct modules. Without this re-mapping, the optimizer would update the wrong module for nearly half the traces. The end-to-end SR gap is small precisely because the VeruSAGE agent has 16 optimizable modules and many failures happen to surface near their root cause; in agents with deeper dependency chains, the re-mapping effect would be larger. That said, we agree the paper should not claim causal localization is the most impactful component based on SR alone. We will: (1) add bootstrap confidence intervals for all ablation variants, (2) add a per-trace analysis showing how often causal localization changes the target module and how those re-mapped traces perform, and (3) revise the framing to position causal localization's contribution as improving the revision: no

  2. Referee: Table 1 vs. Table 2 discrepancy: Table 1 reports STRACE at 58.5% SR on VeruSAGE-Bench, while Table 2 reports 56% for the full STRACE ablation. This is unexplained and, if it reflects variance, undermines the 2-point ablation gap for causal localization.

    Authors: This is a fair point and we apologize for the lack of clarity. The discrepancy arises because Table 1 and Table 2 use different training data sizes. Table 1 reports the main result using the full training set (453 trajectories), while Table 2 reports the ablation using a reduced training set (200 trajectories) to keep ablation costs manageable across five variants. This difference in training data explains the 2.5-point gap. We will state this explicitly in the revised Table 2 caption and in the ablation section. We acknowledge that this configuration difference, combined with the absence of confidence intervals, makes the ablation gap for causal localization difficult to interpret. Adding significance testing (as committed above) will help clarify whether the 2-point gap is meaningful at the 200-case scale. If the gap is not statistically significant, we will say so honestly and reframe the contribution of causal localization around the cost savings and target re-mapping evidence rather than the SR delta. revision: partial

  3. Referee: §3.4 Phase 3: The term 'causal localization' implies formal causal inference, but the actual mechanism is LLM-based backward slicing over a heuristically constructed dependency graph plus LLM semantic reasoning. The paper should clarify that 'causal' refers to the LLM's semantic assessment, not formal causal identification.

    Authors: We agree with this characterization and will revise the paper to clarify it. The term 'causal' in STRACE refers to the goal of identifying the causally responsible module for a failure, not to a formal causal identification procedure (e.g., do-calculus or structural causal models). The backward slicing step (Step 1) is purely structural: it traverses data and control dependency edges backward from the manifestation node, retaining only nodes in the dependency closure. The root cause isolation step (Step 2) is an LLM judgment call that performs semantic reasoning over the sliced context to identify where the execution first deviated from intended logic. We will add a paragraph in §3.4 explicitly stating that STRACE does not perform formal causal inference, that the dependency graph is a heuristic structural prior (not a causal DAG), and that the root cause designation relies on LLM semantic assessment. We will also add a brief comparison to formal causal inference approaches in §2.3 to help readers calibrate expectations. revision: yes

  4. Referee: Table 1 WebArena: STRACE's advantage is domain-dependent (strong on Reddit and GitLab, ties or does not exceed baselines on Shopping and CMS). The paper should acknowledge this and discuss why. Also, per-subdomain sample sizes are unspecified, raising significance concerns.

    Authors: The referee is correct that STRACE's WebArena advantage is domain-dependent. We will add a discussion of this. The pattern is: Shopping and CMS tasks are relatively short-horizon with procedural workflows where the failure point is usually near the manifestation point, so causal localization provides less additional signal and STRACE merely ties the best baseline (TextGrad at 30.8% on Shopping). Reddit and GitLab tasks involve longer interaction chains with more complex control flow, where failures propagate across multiple steps—exactly the setting where structural backward slicing and root-cause re-mapping provide the most value. We will add this analysis to §4.2. Regarding sample sizes: WebArena uses 165 test tasks total across the four subdomains (Shopping: 50, CMS: 30, Reddit: 45, GitLab: 40). We will add these counts to Table 1. We agree that with these sample sizes, per-subdomain differences of a few percentage points are not individually significant, and we will note this. The aggregate WebArena result (23.7% vs. 17.3% for the best baseline) is more reliable, but we will avoid claiming per-subdomain superiority where the gap is small. revision: yes

standing simulated objections not resolved
  • The ablation gap for causal localization (2 points on 200 cases) may not reach statistical significance even after adding bootstrap confidence intervals. If it does not, we cannot honestly claim that causal localization improves end-to-end SR based on the ablation alone; we would need to reframe its contribution around cost savings and target re-mapping evidence. We will report the significance test results transparently regardless of outcome.

Circularity Check

0 steps flagged

No significant circularity found; the derivation chain is a forward pipeline with external benchmarks and ablation validation.

full rationale

The paper's derivation chain is a forward pipeline: Phase 1 constructs the EDG from source code (an independent input), Phase 2 uses the EDG plus execution traces to filter representative failures, Phase 3 uses the EDG to perform backward slicing and root-cause isolation on filtered traces, and Phase 4 converts localized evidence into prompt updates. At no point does a claimed output reduce by construction to its own inputs. The EDG is constructed from the agent's codebase (Section 3.2), not from the traces it later analyzes; the causal slice is derived by traversing EDG edges backward from a manifestation node (Section 3.4), which is a graph algorithm applied to an independently constructed structure, not a definition that presupposes its conclusion. The evaluation uses external benchmarks (HotpotQA, WebArena, VeruSAGE-Bench) and external baselines (TextGrad, GEPA, etc.). The ablation study (Table 2) independently removes each component and measures the effect. The perturbation study (Table 4) tests robustness to EDG noise by externally injecting edge perturbations. The expert-reviewed EDG quality assessment (Table 3) is a validation step, not a circular definition. The reader's concern that the same LLM constructs the EDG and performs causal deduction is a validity/correctness concern (can the LLM faithfully extract dependencies?), not a circularity concern (the output is not defined in terms of the input by construction). The skeptic's observation that causal localization adds only 2 SR points in ablation is about the empirical load-bearingness of the contribution, not about circularity. I assign a score of 1 rather than 0 only because the paper relies on the same LLM (Claude Sonnet 4.5) for both EDG construction and causal reasoning, creating a mild methodological coupling, but this does not constitute a formal circularity where a prediction reduces to its inputs by definition or self-citation chain.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The framework introduces two hand-tuned parameters (k, s) and relies on three domain assumptions about LLM capabilities for graph construction, slicing, and abstraction. The EDG is a new entity with supporting evidence.

free parameters (2)
  • bottleneck threshold k = 5
    Used in the trace filtering phase to select top-k high-impact manifest modules. Chosen as a fixed value in experiments.
  • exemplar count s = 5
    Used to select the number of representative traces per cluster. Fixed for all experiments.
axioms (3)
  • domain assumption An LLM can reliably construct a faithful textual dependency graph from source code.
    Section 3.2 relies on this for the structural prior. The entire causal localization mechanism depends on this graph being accurate.
  • domain assumption Backward slicing over a dependency graph preserves causally relevant information for optimization.
    Section 3.4 Step 1 assumes that the dependency closure captures the necessary context for root cause isolation.
  • domain assumption LLM-based inductive abstraction can generate generalized, reusable heuristics from episodic failures.
    Section 3.5 assumes the optimizer can synthesize robust rules rather than overfitting to instance-specific failures.
invented entities (1)
  • Textual Execution Dependency Graph (EDG) independent evidence
    purpose: Serves as the topological foundation for trace filtering and causal localization.
    The paper provides a perturbation study (Table 4) showing STRACE's effectiveness is sensitive to the graph's presence and robust to moderate noise, providing falsifiable evidence for its utility.

pith-pipeline@v1.1.0-glm · 22455 in / 1753 out tokens · 378082 ms · 2026-07-09T01:30:47.748347+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages · 2 internal anchors

  1. [1]

    Evaluating Large Language Models Trained on Code

    Evaluating large language models trained on code.Preprint, arXiv:2107.03374. Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei-Yue Wang

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    gradient descent

    Stop summation: Min-form credit assign- ment is all process reward model needs for reasoning. Preprint, arXiv:2504.15275. 9 DeepSeek-AI. 2025. DeepSeek-V3.1 release. https: //api-docs.deepseek.com/news/news250821. Accessed: 2026-05-20. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yu- jiu Yang. 2025. EvoPro...

    work page arXiv 2025

  3. [3]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Reflexion: Language agents with verbal rein- forcement learning.Preprint, arXiv:2303.11366. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. AdaPlanner: Adaptive plan- ning from feedback with language models.Preprint, arXiv:2305.16653. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, X...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    The workflow proceeds as follows:

    1, our HotpotQA agent consists of four se- quential sub-agents designed to handle multi-hop reasoning. The workflow proceeds as follows:

    work page

  5. [5]

    Its goal is to extract relevant information and form a preliminary context

    Summarize1:This module processes the ini- tial question and the paragraphs retrieved in the first hop. Its goal is to extract relevant information and form a preliminary context

    work page

  6. [6]

    Create Query Hop2:Based on the initial summary and the original question, this mod- ule generates a refined search query to retrieve missing evidence required for the second hop

    work page

  7. [7]

    Summarize2:This module integrates the newly retrieved passages from the second hop with the prior context, synthesizing a compre- hensive summary that consolidates evidence from both retrieval rounds

    work page

  8. [8]

    During optimization, STRACE optimizes the system prompts for all four modules simultane- ously

    Final Answer:Finally, this module reasons over the aggregated summaries to produce the final answer string (and supporting facts) for the multi-hop question. During optimization, STRACE optimizes the system prompts for all four modules simultane- ously. A.2 WebArena Agent Workflow For WebArena, we followed the original We- bArena Chain-of-Thought agent de...

    work page 2024

  9. [9]

    Observe:The browser environment converts the current webpage into an accessibility-tree 1https://github.com/gepa-ai/gepa-artifact 2https://github.com/web-arena-x/webarena representation, exposing interactive elements such as buttons, links, text boxes, and menus

    work page

  10. [10]

    Reason and Act:Given the task objective and the current observation, the Chain-of-Thought prompt guides the language model to reason about the page state and output one concrete action, such as clicking an element, typing into a field, scrolling, navigating to a URL, or stopping with a final answer

    work page

  11. [11]

    Execute:The predicted action is executed in the browser environment, producing a new webpage state

    work page

  12. [12]

    During optimization, STRACE targets the prompt of this single WebArena agent

    Repeat:The agent repeats this observe– reason–act cycle until it issues a stop action or reaches the maximum interaction budget. During optimization, STRACE targets the prompt of this single WebArena agent. Since the baseline does not contain explicit sub-agents, plan- ners, or external memory modules, the optimized component is the agent’s instruction pr...

    work page 2025

  13. [13]

    The current code is submitted to the Verus verifier

    work page

  14. [14]

    If verification fails, theError Analyzerparses the error message

    work page

  15. [15]

    ARouter(or High-level Planner) selects the most appropriate repair action from a pool of 35 available actions based on the error type and code context

    work page

  16. [16]

    The selectedSub-agent(Executor) generates a specific patch (e.g., adding a lemma func- tion, enabling a specific prover mode, apply- ing divide-and-conquer for a specific property to be proved)

    work page

  17. [17]

    nonlinear arithmetic

    The patch is applied, and the loop repeats. The process terminates when the code verifies suc- cessfully or when the budget (20 attempts or 20 minutes) is exhausted. 3https://github.com/microsoft/ verus-proof-synthesis 12 Sub-agent Structure.The framework com- prises 15 specialized sub-agents covering various aspects of formal verification, including but ...

    work page 2025

  18. [18]

    • Track repeated failures:Maintain awareness of how many times each ac- tion type has been attempted and rejected under the current proof context

    Failure-aware stopping condition and ac- tion switching. • Track repeated failures:Maintain awareness of how many times each ac- tion type has been attempted and rejected under the current proof context. • Stopping condition:If an action has been attempted 2–3 times and is consis- tently rejected (especially due to syntax errors or does not fix the target...

    work page

  19. [19]

    Sub-agent-selection boundaries and action- able guidance. • General requirement:The guidance field must be specific and actionable (avoid generic suggestions); include con- crete anchors from the visible code and the current goal structure. • USELEMMA:Identify specific helper lemmas/proof functions available in the file and state what each lemma estab- li...

    work page