From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization

Chenyuan Yang; Jiahang Xu; Peng Cheng; Xuan Feng; Ying Chang; Yuqing Yang

REVIEW

From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization

Ying Chang, Jiahang Xu, Xuan Feng, Chenyuan Yang, Peng Cheng, Yuqing Yang

Computation and Language · Submitted 8 Jul 2026 · 19 references

Pith's verdict: STRACE applies program-analysis concepts to LLM agent optimization with strong empirical results, but the ablation undercuts the headline causal-localization mechanism.

T0 · AI review

How this review was made: An AI referee (glm-5.2) read the paper on July 9, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 4 major objections · 7 minor · Ask about this paper · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2607.07702 (version 1)

Pith IDNGOF4PL2, Pith's permanent record of this paper and its review

CitationChenyuan Yang, Jiahang Xu, Peng Cheng, Xuan Feng, Ying Chang, Yuqing Yang. "From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization." arXiv:2607.07702 (2026). https://pith.science/paper/NGOF4PL2

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

The headline: STRACE is a solid systems paper that applies trace filtering and dependency-graph-guided context construction to agent prompt optimization. The empirical results on VeruSAGE-Bench are strong — 42.5% to 58.5% SR is a real improvement over GEPA and other baselines. Code is public. But the ablation raises a question the paper doesn't address: the component the paper headlines (causal localization via backward slicing) may not be doing much work, while the less-glamorous trace filtering and structural modeling phases carry the gains. The stress-test note on Table 2 is largely correct and is the main thing you should know about this paper. Removing causal localization drops SR from 56% to 54% — a 2-point gap that is within noise for the sample sizes here. Removing trace filtering costs 10 points; removing structural modeling costs 8 points. The paper's narrative centers on Phase 3 (causal localization, root-cause isolation, backward slicing) as the distinguishing contribution, but the data says Phase 2 (trace filtering) and Phase 1 (structural modeling as a prior) are the load-bearing components. The paper does not acknowledge this mismatch. That said, the paper does several things well. The framework is well-engineered: the deterministic trace parser, the diversity-based trace selection, and the cost analysis (Table 5) are all solid. The perturbation study (Table 4) addresses the reader's concern about EDG construction fidelity — STRACE is robust to ±25% edge perturbation, which is reassuring. The case study in Figure 4 showing re-mapping of 12/25 traces to upstream root causes is illustrative, though it would be more convincing with a quantitative link to outcome improvement. The WebArena results are mixed (STRACE underperforms on some subtasks) and HotpotQA gains are modest, but the VeruSAGE-Bench results are the real story and they hold up. The two free parameters (bottleneck threshold k, exemplar count s) are standard and not a concern. The 'Textual Execution Dependency Graph' is not really an invented entity — it's a text edge list, which the paper is transparent about in Appendix B.2. Who is this for? Researchers and practitioners working on automated agent optimization, especially in compound AI systems with long-horizon traces. The cost-performance trade-off analysis and the scalability results (Figure 3) are useful for this audience. Recommendation: This deserves a serious referee. The empirical contribution is real and the system is well-built. But the referee should push hard on the ablation mismatch — if causal localization is the novel contribution, the paper needs to either show it matters more than Table 2 suggests or reframe the contribution around what actually drives the gains. The paper should also report confidence intervals or significance tests, especially given the small sample sizes on VeruSAGE-Bench subtasks.

Referee report

Major objections (4)

Table 2 (ablation): The paper's central novel contribution is Phase 3 (Causal Localization), which distinguishes STRACE from prior work by identifying 'the true root-cause module' rather than 'optimizing the wrong target.' However, Table 2 shows that removing causal localization entirely (both 'Current Node' and 'Full Trace' variants) only drops SR from 56% to 54%—a 2-point gap. By contrast, removing trace filtering costs 10 points (56%→46%) and removing structural modeling costs 8 points (56%→48%). The §4.4 case study references a 200-case setting, meaning a 2-point gap corresponds to approximately 4 tasks. The paper does not report confidence intervals or significance tests for these ablation results. Without such analysis, it is unclear whether the 2-point difference is statistically meaningful. This is a load-bearing concern: if causal localization is statistically indistinguishable从
Table 1 vs. Table 2 discrepancy: Table 1 reports STRACE at 58.5% SR on VeruSAGE-Bench, while Table 2 reports 56% SR for the full STRACE ablation. The paper does not explain this discrepancy. If the ablation uses a different subset or configuration, this should be stated explicitly. If the difference reflects variance, it further underscores the need for significance testing. This discrepancy also affects the interpretation of the ablation: if the full STRACE SR varies by 2.5 points between experimental configurations, the 2-point ablation gap for causal localization is within the noise floor.
§3.4, Phase 3: The term 'causal localization' implies a formal causal inference method, but the actual mechanism is an LLM performing backward slicing over a heuristically constructed dependency graph and then doing semantic reasoning to identify the root cause. The paper does not formalize what constitutes a 'causal' relationship versus a mere data or control dependency. This matters because the backward slicing (Step 1) is purely structural—it retains any node with a dependency path to the manifestation node—while the root cause isolation (Step 2) is an LLM judgment call. The paper should clarify that 'causal' here refers to the LLM's semantic assessment of where the logic first deviated, not to a formal causal identification procedure. This would help readers calibrate expectations and facilitate comparison with formal causal inference approaches.
Table 1, WebArena: STRACE underperforms on Reddit (36.4% vs. GEPA's 27.3% is a gain, but on GitLab STRACE scores 17.1% vs. GEPA's 9.8%, while on CMS STRACE ties TextGrad at 16.2%). More importantly, on Shopping, STRACE ties TextGrad at 30.8% but does not exceed it. The paper claims STRACE 'achieves the strongest overall performance' on WebArena (23.7% ALL vs. TextGrad's 17.3%), but this aggregate is dominated by the Reddit and GitLab subdomains. The paper should acknowledge that STRACE's advantage is domain-dependent and discuss why the structural approach helps more in some web tasks than others. Additionally, the WebArena sample sizes per subdomain appear small (the paper does not specify exact counts), which raises the same significance concerns as the ablation.

Minor objections (7)

§4.1: The bottleneck threshold k=5 and exemplar count s=5 are listed as hyperparameters but no sensitivity analysis is provided. A brief ablation or justification for these values would strengthen the paper.
Table 3: The graph quality assessment is 'expert-reviewed' but the methodology is not described. How many experts? What was the review protocol? Were they given the ground-truth graph? This limits the reproducibility of the EDG quality claim.
Figure 3: The x-axis label 'Number of Training Cases' goes up to 453, but it is unclear whether each point represents an independent run or a cumulative evaluation. Clarification needed.
§3.2: The EDG construction relies on an LLM reading the agent's source code. The paper notes this requires 'sufficient visibility into the target agent system' (Limitations), but does not discuss failure modes when the codebase is large or when the LLM misidentifies module boundaries. A brief discussion of scaling behavior would help.
Appendix B.2: The distinction between 'data dependencies' and 'control dependencies' is conceptually clear but the paper does not provide examples of edges that were missed or incorrectly inferred in practice. Including 1-2 such examples from the perturbation study would make the robustness claim more concrete.
Table 7: STRACE-enhanced o4-mini (58.5%) is compared against GPT-5 (50.0%) and Claude Sonnet 4 (59.4%). The comparison with Claude Sonnet 4 is nearly a tie. The paper should note that STRACE's advantage over GPT-5 may partly reflect the base agent architecture (VeruSAGE) rather than STRACE alone, since GPT-5 is used in a hands-off mode without the VeruSAGE harness.
The paper uses the term 'causal' throughout (causal localization, causal slice, causal context) in a sense that differs from the standard causal inference literature. A footnote or brief clarification on first use would improve precision.

Simulated authors' rebuttal

read the rebuttal and 4 point-by-point responses

We thank the referee for a careful and constructive review. The referee raises four major points: (1) the ablation gap for causal localization is small and lacks significance testing; (2) an unexplained discrepancy between Table 1 (58.5%) and Table 2 (56%) for full STRACE; (3) the term 'causal localization' may overstate the methodological contribution relative to what is formally a heuristic backward-slicing plus LLM semantic reasoning procedure; and (4) STRACE's WebArena advantage is domain-dependent and per-subdomain sample sizes are unspecified. We address each below.

Referee: Table 2 ablation: causal localization only contributes a 2-point gap (56% to 54%), corresponding to ~4 tasks out of 200, with no confidence intervals or significance tests. If this gap is within noise, the paper's central novel contribution is not empirically supported.

Authors: The referee is correct that the 2-point ablation gap for causal localization is small relative to the gaps for trace filtering (10 points) and structural modeling (8 points), and that without significance testing we cannot formally rule out noise. We will address this in revision. However, we respectfully disagree that the ablation gap is the primary evidence for causal localization's value. The ablation measures end-to-end success rate, which conflates two distinct effects of Phase 3: (a) cost reduction via context compression and (b) correctness of root-cause attribution. Table 2 and Table 5 (Appendix C.1) show that replacing causal localization with full-trace context nearly doubles Phase 4 cost ($2.96 → $5.93) while matching SR, confirming the cost benefit. More importantly, Figure 4 provides direct evidence that causal localization changes the optimization target: in the 200-case setting, 12 of 25 traces are re-mapped from manifestation nodes to upstream root-cause nodes, expanding optimization targets from 5 to 6 distinct modules. Without this re-mapping, the optimizer would update the wrong module for nearly half the traces. The end-to-end SR gap is small precisely because the VeruSAGE agent has 16 optimizable modules and many failures happen to surface near their root cause; in agents with deeper dependency chains, the re-mapping effect would be larger. That said, we agree the paper should not claim causal localization is the most impactful component based on SR alone. We will: (1) add bootstrap confidence intervals for all ablation variants, (2) add a per-trace analysis showing how often causal localization changes the target module and how those re-mapped traces perform, and (3) revise the framing to position causal localization's contribution as improving the revision: no
Referee: Table 1 vs. Table 2 discrepancy: Table 1 reports STRACE at 58.5% SR on VeruSAGE-Bench, while Table 2 reports 56% for the full STRACE ablation. This is unexplained and, if it reflects variance, undermines the 2-point ablation gap for causal localization.

Authors: This is a fair point and we apologize for the lack of clarity. The discrepancy arises because Table 1 and Table 2 use different training data sizes. Table 1 reports the main result using the full training set (453 trajectories), while Table 2 reports the ablation using a reduced training set (200 trajectories) to keep ablation costs manageable across five variants. This difference in training data explains the 2.5-point gap. We will state this explicitly in the revised Table 2 caption and in the ablation section. We acknowledge that this configuration difference, combined with the absence of confidence intervals, makes the ablation gap for causal localization difficult to interpret. Adding significance testing (as committed above) will help clarify whether the 2-point gap is meaningful at the 200-case scale. If the gap is not statistically significant, we will say so honestly and reframe the contribution of causal localization around the cost savings and target re-mapping evidence rather than the SR delta. revision: partial
Referee: §3.4 Phase 3: The term 'causal localization' implies formal causal inference, but the actual mechanism is LLM-based backward slicing over a heuristically constructed dependency graph plus LLM semantic reasoning. The paper should clarify that 'causal' refers to the LLM's semantic assessment, not formal causal identification.

Authors: We agree with this characterization and will revise the paper to clarify it. The term 'causal' in STRACE refers to the goal of identifying the causally responsible module for a failure, not to a formal causal identification procedure (e.g., do-calculus or structural causal models). The backward slicing step (Step 1) is purely structural: it traverses data and control dependency edges backward from the manifestation node, retaining only nodes in the dependency closure. The root cause isolation step (Step 2) is an LLM judgment call that performs semantic reasoning over the sliced context to identify where the execution first deviated from intended logic. We will add a paragraph in §3.4 explicitly stating that STRACE does not perform formal causal inference, that the dependency graph is a heuristic structural prior (not a causal DAG), and that the root cause designation relies on LLM semantic assessment. We will also add a brief comparison to formal causal inference approaches in §2.3 to help readers calibrate expectations. revision: yes
Referee: Table 1 WebArena: STRACE's advantage is domain-dependent (strong on Reddit and GitLab, ties or does not exceed baselines on Shopping and CMS). The paper should acknowledge this and discuss why. Also, per-subdomain sample sizes are unspecified, raising significance concerns.

Authors: The referee is correct that STRACE's WebArena advantage is domain-dependent. We will add a discussion of this. The pattern is: Shopping and CMS tasks are relatively short-horizon with procedural workflows where the failure point is usually near the manifestation point, so causal localization provides less additional signal and STRACE merely ties the best baseline (TextGrad at 30.8% on Shopping). Reddit and GitLab tasks involve longer interaction chains with more complex control flow, where failures propagate across multiple steps—exactly the setting where structural backward slicing and root-cause re-mapping provide the most value. We will add this analysis to §4.2. Regarding sample sizes: WebArena uses 165 test tasks total across the four subdomains (Shopping: 50, CMS: 30, Reddit: 45, GitLab: 40). We will add these counts to Table 1. We agree that with these sample sizes, per-subdomain differences of a few percentage points are not individually significant, and we will note this. The aggregate WebArena result (23.7% vs. 17.3% for the best baseline) is more reliable, but we will avoid claiming per-subdomain superiority where the gap is small. revision: yes

Objections the simulated rebuttal did not resolve

The ablation gap for causal localization (2 points on 200 cases) may not reach statistical significance even after adding bootstrap confidence intervals. If it does not, we cannot honestly claim that causal localization improves end-to-end SR based on the ablation alone; we would need to reframe its contribution around cost savings and target re-mapping evidence. We will report the significance test results transparently regardless of outcome.

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Causal Slicing Finds Root Causes in Agent Traces, Boosting Success 1.4x

The reading

When LLM-based agents fail at long-horizon tasks, the visible error often appears far downstream from its actual origin — a crash at step 50 might stem from a bad parameter generated at step 5. This paper proposes STRACE, a framework that treats execution traces not as flat logs but as dependency graphs, enabling backward causal slicing to isolate the true root-cause module for optimization. The core mechanism is a textual Execution Dependency Graph (EDG) — a lightweight map of data and control flow between agent components, inferred from source code by an LLM. Using the EDG, STRACE traces failures backward from their manifestation point, pruning irrelevant steps and identifying the upstream module whose logical defect triggered the downstream symptom. It then synthesizes generalized heuristics and injects them into the root-cause module's prompt. On a formal verification benchmark (VeruSAGE-Bench), this approach lifted success rates from 42.5% to 58.5%, a 1.4x improvement over human-expert-designed agents, while keeping optimization costs bounded even as training traces scaled to 453.

What carries the argument

Textual Execution Dependency Graph (EDG): a text-based node-edge map of data and control dependencies among agent modules, inferred from source code. Used as a structural prior for backward slicing and root-cause isolation.

Extended reading notes

Core claim

The central discovery is that the context-noise trade-off in agent optimization — the tension between feeding full traces (too noisy) and truncating them (loses causal evidence) — can be broken by constructing a dependency graph of the agent's modules and performing backward causal slicing from the failure manifestation point. This remaps optimization targets from symptom nodes to root-cause nodes: in the case study, 12 of 25 selected traces had their attributed failure module corrected to an upstream component. The EDG itself need not be perfectly accurate; perturbation experiments show that adding or removing 25% of edges causes only marginal performance change (56% to 54% at worst), while

Load-bearing premise

The entire causal localization mechanism depends on an LLM reliably constructing a faithful dependency graph from the agent's source code. If the LLM misses a critical data or control dependency, the backward slicing will not find the true root cause, and the optimizer will patch the wrong module.

Editorial extensions

If this is right

Agent optimization frameworks can shift from symptom-level patching to structural root-cause repair, potentially reducing the number of brittle, overfit prompt updates in production agent systems.
The dependency-graph approach generalizes beyond prompt optimization to harness engineering — localized diagnoses could guide updates to tool-use policies, skills, and other editable agent components.
If the EDG construction step can be automated reliably across arbitrary agent codebases, this method becomes a plug-in diagnostic layer for any agent system with sufficient implementation visibility.
The robustness to graph perturbation (25% edge noise causing minimal degradation) suggests that approximate structural priors suffice, lowering the barrier to adoption for systems with partial or noisy code visibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The backward-slicing approach implicitly assumes that agent failures are predominantly single-root-cause events. If failures are frequently caused by the interaction of two or more modules jointly (a combinatorial defect), tracing backward from the manifestation node along individual dependency paths may still miss the true cause.
The framework's requirement for codebase-level visibility means it is best suited for open-source or internally-developed agent systems. Fully black-box API-based agents (e.g., commercial computer-use agents) would need an alternative structural prior source, such as inferring dependencies from trace patterns alone.
The stability under graph perturbation raises a question the paper does not fully address: if adding 10% of random edges improves performance (56% to 62%), the added edges may occasionally capture real dependencies the LLM missed, suggesting that the base graph construction is conservative and could benefit from complementary static analysis tools.
The method's effectiveness on formal verification (VeruSAGE-Bench) versus its more modest gains on web navigation (WebArena) suggests that causal localization provides the most value when failures propagate through deep, structured reasoning chains rather than through exploratory, stateful web interactions.

Original abstract

The optimization of long-horizon agents increasingly relies on reflection-based mechanisms, where a large language model (LLM) acts as an optimizer to diagnose agent failures and improve agent policies. However, real execution traces are difficult to use directly for optimization: large trace collections are often redundant and heterogeneous, making optimization inefficient and prone to overfitting to low-value failures; meanwhile, each individual trajectory also contains many irrelevant steps, while naive context reduction methods such as truncation or sliding windows can discard causally important evidence and produce misleading optimization signals. To resolve this dilemma, we introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework that constructs high signal-noise optimization contexts for more precise and effective optimization. At the batch level, STRACE mines failure patterns to filter redundant traces and retain representative failures; within each selected trace, it performs causal localization over a textual dependency graph to remove non-causal steps and identify the true root-cause module for optimization. Empirical results demonstrate that STRACE significantly outperforms standard context-filtering baselines. Notably, on a challenging formal verification task (VeruSAGE-Bench), it successfully optimizes human-expert designed agents, delivering $1.4\times$ success-rate improvement (42.5% to 58.5%). The code is available at https://github.com/moomight/STRACE .

Figures

**Figure 1.** Comparison of context construction strate

**Figure 2.** Illustration of the STRACE framework. The system optimizes long-horizon agents via four phases: (1) Structural Modeling, which constructs execution topology; (2) Failure Pattern Mining and Trace Filtering, which compresses failures into representative traces; (3) Causal Localization, which performs causal context distillation and root cause isolation; and (4) Inductive Policy Optimization, which injects heuristics into identified root cause modules. tify the atomic functional modules (the vertex set V). Simultaneously, it infers the textual edges E by analyzing the artifacts these…

**Figure 3.** Success rate on 50 independent tasks and

**Figure 4.** Distinguishing Symptoms from Root Causes. Top: Causal localization remaps 25 traces from 5 manifest nodes to 6 targets, exposing an upstream source. Bottom: A compute_repair loop is traced to assertion_reasoning_pipeline, enabling a targeted policy optimization to bypass the recursive path. 4.4 Case Study: Tracing Symptoms to Root Causes To empirically validate the distinction between symptom (vm) and cause (vr), we present two analyses from the VeruSAGE-Bench optimization process in

**Figure 5.** Visualization of a Failure Summarization Map generated during the optimization of the IronKV project. STRACE first parses raw traces into structured execution records containing the global task outcome, local node status, and module invocation sequence. The map then summarizes the trace corpus along two analytical dimensions: Statistical Severity, which estimates the conditional probability of global failure given each local node error, and Structural Path Patterns, which identifies recurring pathological trajectories such as self-loops, oscillations, and dead ends. These aggregated…

**Figure 6.** Instruction for Phase 1–Structural Modeling.

**Figure 7.** Instruction for Phase 2 – Failure Pattern Mining and Trace Filtering.

**Figure 8.** Instruction for Phase 3 – Causal Localization.

**Figure 9.** Instruction for Phase 4 – Inductive Policy Optimization.

Figures reproduced from arXiv:2607.07702 by the authors.

References

Works cited (19)

[1]

Evaluating large language models trained on code.Preprint, arXiv:2107.03374. Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei-Yue Wang

View on Pith Pith review arXiv
[2]

Stop summation: Min-form credit assign- ment is all process reward model needs for reasoning. Preprint, arXiv:2504.15275. 9 DeepSeek-AI. 2025. DeepSeek-V3.1 release. https: //api-docs.deepseek.com/news/news250821. Accessed: 2026-05-20. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yu- jiu Yang. 2025. EvoPro...

View on Pith arXiv 2025
[3]

Reflexion: Language agents with verbal rein- forcement learning.Preprint, arXiv:2303.11366. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. AdaPlanner: Adaptive plan- ning from feedback with language models.Preprint, arXiv:2305.16653. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, X...

View on Pith Pith review arXiv 2023
[4]

1, our HotpotQA agent consists of four se- quential sub-agents designed to handle multi-hop reasoning. The workflow proceeds as follows:

View on Pith
[5]

Summarize1:This module processes the ini- tial question and the paragraphs retrieved in the first hop. Its goal is to extract relevant information and form a preliminary context

View on Pith
[6]

Create Query Hop2:Based on the initial summary and the original question, this mod- ule generates a refined search query to retrieve missing evidence required for the second hop

View on Pith
[7]

Summarize2:This module integrates the newly retrieved passages from the second hop with the prior context, synthesizing a compre- hensive summary that consolidates evidence from both retrieval rounds

View on Pith
[8]

Final Answer:Finally, this module reasons over the aggregated summaries to produce the final answer string (and supporting facts) for the multi-hop question. During optimization, STRACE optimizes the system prompts for all four modules simultane- ously. A.2 WebArena Agent Workflow For WebArena, we followed the original We- bArena Chain-of-Thought agent de...

View on Pith 2024
[9]

Observe:The browser environment converts the current webpage into an accessibility-tree 1https://github.com/gepa-ai/gepa-artifact 2https://github.com/web-arena-x/webarena representation, exposing interactive elements such as buttons, links, text boxes, and menus

View on Pith
[10]

Reason and Act:Given the task objective and the current observation, the Chain-of-Thought prompt guides the language model to reason about the page state and output one concrete action, such as clicking an element, typing into a field, scrolling, navigating to a URL, or stopping with a final answer

View on Pith
[11]

Execute:The predicted action is executed in the browser environment, producing a new webpage state

View on Pith
[12]

Repeat:The agent repeats this observe– reason–act cycle until it issues a stop action or reaches the maximum interaction budget. During optimization, STRACE targets the prompt of this single WebArena agent. Since the baseline does not contain explicit sub-agents, plan- ners, or external memory modules, the optimized component is the agent’s instruction pr...

View on Pith 2025
[13]

The current code is submitted to the Verus verifier

View on Pith
[14]

If verification fails, theError Analyzerparses the error message

View on Pith
[15]

ARouter(or High-level Planner) selects the most appropriate repair action from a pool of 35 available actions based on the error type and code context

View on Pith
[16]

The selectedSub-agent(Executor) generates a specific patch (e.g., adding a lemma func- tion, enabling a specific prover mode, apply- ing divide-and-conquer for a specific property to be proved)

View on Pith
[17]

The patch is applied, and the loop repeats. The process terminates when the code verifies suc- cessfully or when the budget (20 attempts or 20 minutes) is exhausted. 3https://github.com/microsoft/ verus-proof-synthesis 12 Sub-agent Structure.The framework com- prises 15 specialized sub-agents covering various aspects of formal verification, including but ...

View on Pith 2025
[18]

Failure-aware stopping condition and ac- tion switching. • Track repeated failures:Maintain awareness of how many times each ac- tion type has been attempted and rejected under the current proof context. • Stopping condition:If an action has been attempted 2–3 times and is consis- tently rejected (especially due to syntax errors or does not fix the target...

View on Pith
[19]

Sub-agent-selection boundaries and action- able guidance. • General requirement:The guidance field must be specific and actionable (avoid generic suggestions); include con- crete anchors from the visible code and the current goal structure. • USELEMMA:Identify specific helper lemmas/proof functions available in the file and state what each lemma estab- li...

View on Pith

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 7 sentences without a confident label

Uses the method or tool of 4 papers

For VeruSAGE-Bench, we use the hierarchical router-executor multi-agent framework [ref] with 16 optimizable modules and [ref] as the backbone.

VeruSAGE: A Study of Agent-Based Verification for Rust Systems · Experimental Setup · 0.97
For HotpotQA, we use a DSPy-based [ref] four-stage multi-hop reasoning workflow with 4 optimizable modules and [ref] .

GPT-4o System Card · Experimental Setup · 1.00
For HotpotQA, we use a DSPy-based [ref] four-stage multi-hop reasoning workflow with 4 optimizable modules and [ref] .

GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning · Experimental Setup · 1.00

Cites as support 3 papers

Feeding the full trajectory (a) overwhelms the optimizer with irrelevant successful steps and low-SNR context, inviting spurious correlations and hallucinated prompt patches [ref] .

TextGrad: Automatic "Differentiation" via Text · Introduction · 0.97
In realistic settings, batched execution trajectories contain heterogeneous failures with different frequencies, severities, and causes [ref] .

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories · Introduction · 0.96
This dilemma exposes a fundamental reality of agentic workflows: the node where an error manifests (the downstream symptom) is often not the node where it originates (the upstream root cause) [ref] .

AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? · Introduction · 0.99

Cites as background 19 papers

To handle complex workflows, recent frameworks leverage reflexive feedback [ref] or modular textual gradients [ref] to optimize multi-stage pipelines.

Automatic Prompt Optimization with "Gradient Descent" and Beam Search · Automated Prompt Optimization · 0.82
To handle complex workflows, recent frameworks leverage reflexive feedback [ref] or modular textual gradients [ref] to optimize multi-stage pipelines.

Reflexion: Language Agents with Verbal Reinforcement Learning · Automated Prompt Optimization · 0.82
A predominant paradigm for optimizing such agent systems is reflexive optimization [ref] , which uses an LLM to diagnose agent-system failures and evolve agent policies based on historical feedback derived from agent trajectories.

Large Language Models as Optimizers · Introduction · 0.89

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.

[1] [1]

Evaluating large language models trained on code.Preprint, arXiv:2107.03374. Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei-Yue Wang

View on Pith Pith review arXiv

[2] [2]

Stop summation: Min-form credit assign- ment is all process reward model needs for reasoning. Preprint, arXiv:2504.15275. 9 DeepSeek-AI. 2025. DeepSeek-V3.1 release. https: //api-docs.deepseek.com/news/news250821. Accessed: 2026-05-20. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yu- jiu Yang. 2025. EvoPro...

View on Pith arXiv 2025

[3] [3]

Reflexion: Language agents with verbal rein- forcement learning.Preprint, arXiv:2303.11366. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. AdaPlanner: Adaptive plan- ning from feedback with language models.Preprint, arXiv:2305.16653. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, X...

View on Pith Pith review arXiv 2023

[4] [4]

1, our HotpotQA agent consists of four se- quential sub-agents designed to handle multi-hop reasoning. The workflow proceeds as follows:

View on Pith

[5] [5]

Summarize1:This module processes the ini- tial question and the paragraphs retrieved in the first hop. Its goal is to extract relevant information and form a preliminary context

View on Pith

[6] [6]

Create Query Hop2:Based on the initial summary and the original question, this mod- ule generates a refined search query to retrieve missing evidence required for the second hop

View on Pith

[7] [7]

Summarize2:This module integrates the newly retrieved passages from the second hop with the prior context, synthesizing a compre- hensive summary that consolidates evidence from both retrieval rounds

View on Pith

[8] [8]

Final Answer:Finally, this module reasons over the aggregated summaries to produce the final answer string (and supporting facts) for the multi-hop question. During optimization, STRACE optimizes the system prompts for all four modules simultane- ously. A.2 WebArena Agent Workflow For WebArena, we followed the original We- bArena Chain-of-Thought agent de...

View on Pith 2024

[9] [9]

Observe:The browser environment converts the current webpage into an accessibility-tree 1https://github.com/gepa-ai/gepa-artifact 2https://github.com/web-arena-x/webarena representation, exposing interactive elements such as buttons, links, text boxes, and menus

View on Pith

[10] [10]

Reason and Act:Given the task objective and the current observation, the Chain-of-Thought prompt guides the language model to reason about the page state and output one concrete action, such as clicking an element, typing into a field, scrolling, navigating to a URL, or stopping with a final answer

View on Pith

[11] [11]

Execute:The predicted action is executed in the browser environment, producing a new webpage state

View on Pith

[12] [12]

Repeat:The agent repeats this observe– reason–act cycle until it issues a stop action or reaches the maximum interaction budget. During optimization, STRACE targets the prompt of this single WebArena agent. Since the baseline does not contain explicit sub-agents, plan- ners, or external memory modules, the optimized component is the agent’s instruction pr...

View on Pith 2025

[13] [13]

The current code is submitted to the Verus verifier

View on Pith

[14] [14]

If verification fails, theError Analyzerparses the error message

View on Pith

[15] [15]

ARouter(or High-level Planner) selects the most appropriate repair action from a pool of 35 available actions based on the error type and code context

View on Pith

[16] [16]

The selectedSub-agent(Executor) generates a specific patch (e.g., adding a lemma func- tion, enabling a specific prover mode, apply- ing divide-and-conquer for a specific property to be proved)

View on Pith

[17] [17]

The patch is applied, and the loop repeats. The process terminates when the code verifies suc- cessfully or when the budget (20 attempts or 20 minutes) is exhausted. 3https://github.com/microsoft/ verus-proof-synthesis 12 Sub-agent Structure.The framework com- prises 15 specialized sub-agents covering various aspects of formal verification, including but ...

View on Pith 2025

[18] [18]

Failure-aware stopping condition and ac- tion switching. • Track repeated failures:Maintain awareness of how many times each ac- tion type has been attempted and rejected under the current proof context. • Stopping condition:If an action has been attempted 2–3 times and is consis- tently rejected (especially due to syntax errors or does not fix the target...

View on Pith

[19] [19]

Sub-agent-selection boundaries and action- able guidance. • General requirement:The guidance field must be specific and actionable (avoid generic suggestions); include con- crete anchors from the visible code and the current goal structure. • USELEMMA:Identify specific helper lemmas/proof functions available in the file and state what each lemma estab- li...

View on Pith