Pith. sign in

REVIEW 4 major objections 4 minor 21 references

Frozen agents can follow useful text policies, but cannot reliably learn them from their own failures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:40 UTC pith:XLL44TIX

load-bearing objection A careful, honestly-scoped negative result on agent-level TextGrad; the capacity/induction split is genuinely useful, but the bottleneck claim outruns the (all-CI-span-zero) learning evidence. the 4 major comments →

arxiv 2607.20668 v1 pith:XLL44TIX submitted 2026-07-22 cs.CL cs.AI

From Agent Failures to Text Policies: What Works and What Breaks

Jaideep Ray , Ankit Goyal This is my paper
classification cs.CL cs.AI
keywords text optimizationlanguage agentstextual policiespolicy inductioncredit assignmentfrozen modelsrule generationselection gate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TextGrad-style optimization treats natural-language feedback as a gradient for improving a system without retraining its weights. This paper separates that idea into two abilities for frozen agents: can the agent follow a useful text policy, and can the agent learn such a policy from its own trajectories? Using human-written rules with correct task-family routing, two frozen 7B agents gain 5.0 success points on a text-game benchmark, showing useful policy text exists and can be executed. But policies the learners produce from summaries, step-by-step traces, counterfactual branches, or an iterative search all fail to reliably beat fixed prompting. The paper concludes that agent-level text optimization is limited by rule generation and selection, not by executing text updates.

Core claim

On the paper's own terms, the central discovery is a clean dissociation: policy execution is easy for frozen language agents, while policy induction is not. With privileged human-written rules and family routing, pooled success on TextWorldExpress rises from 15.63% to 20.63% (a gain of 5.0 points, CI +1.88 to +9.38), and average interaction shortens by 2.64 turns. Yet learned rules—whether from compact summaries, context-bounded step traces, same-prefix counterfactual branches, or an official reflective search—do not reliably beat the fixed prompt; their point estimates fall at or below baseline and the intervals span zero. A manual audit of 32 generated rules finds only 7 grounded, executab

What carries the argument

The paper's central machinery is the textual policy-update loop, formalized as τ ∼ π_θ(·|x), g_text = D(τ, R(τ)), x' = U(x, g_text), where a frozen actor generates a trajectory, a diagnosis function converts trajectory and reward into textual feedback, and an update function proposes a new policy. The prototype called RulePI instantiates this with a learner that produces one typed rule of at most 80 words per model and task family, and a selection gate that scores candidates (e.g., r + 0.5s − 0.4i − 0.2p − 0.1q − 0.004T) and accepts only if the score exceeds the fixed policy by at least 0.001. Four diagnostic tests—capacity, evidence, counterfactual, and search—are used to localize failures,

Load-bearing premise

The claim that generation and selection are the bottleneck assumes the null results are real rather than artifacts of low statistical power, the 80-word single-rule format, or using the same 7B model as both proposer and actor.

What would settle it

Take the same frozen actors and propose rules by a stronger or differently prompted generator on the same text-game families, then evaluate with family-aware gates and matched budgets. If such rules reliably beat fixed prompting, the bottleneck is proposer capability, not selection. Alternatively, if human-written rules with correct family routing fail to improve on a different benchmark, the capacity claim would collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Richer trajectory evidence (step-aligned traces over compact summaries, same-state branches over full retries) improves intermediate credit signals but not held-out policy quality, so adding more feedback alone will not fix agent-level TextGrad.
  • Because a correct family-scoped rule executes well, future optimizers can focus their search on generating a diverse set of typed rules and on validation gates that check support, consistency, and regressions by family rather than a single pooled mean.
  • The 5.0-point capacity gain is an upper bound for human-guided policy text; autonomous systems must recover this gain via generation and selection before they can match it.
  • Search-based methods (one official reflective search) produced inconclusive pooled results with conflicting model-level effects; the claim is not that search cannot work, only that at the tested budget it does not reliably close the gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the 15–38-word length of the human rules suggests the 80-word format is not the binding constraint; a follow-up could test whether asking the proposer for several independent candidate rules per family and then selecting with a calibrated gate closes the gap.
  • The paper itself notes its counterfactual study produced only seven valid rules, so the update-operator conclusions rest on a small sample; a dedicated experiment with many paired branches per family could confirm whether the proposer's abstraction failures persist at scale.
  • The capacity test bundles family routing with rule content; a reader might infer that automatic family detection is a separate, untested skill that could be as hard as rule generation, and worth measuring explicitly.
  • If the diagnosis transfers, then weight-update-free agent improvement will depend less on designing better feedback signals and more on building 'policy compilers' that convert local comparisons into scoped, reusable rules.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies whether frozen LLM agents can improve by editing their text instructions, framing it as agent-level TextGrad. It separates two abilities: (i) execution—can a frozen agent follow a useful textual policy—from (ii) learning—can a system generate and select such a policy from the agent's own experience. In a capacity test, human-written policies supplied with the correct benchmark-family label improve two frozen 7B agents on TextWorldExpress by +5.0 success points (95% CI [1.88, 9.38]), showing that useful policy text exists and actors can use it. The paper then reports three end-to-end learning experiments: policies generated from step-by-step traces, from counterfactual same-prefix branches, and from official GEPA search. All three fail to show a reliable held-out improvement over a fixed prompt; the pooled differences are -1.88 [−5.62, 1.88], -1.25 [−3.75, 0.00], and +1.25 [−3.13, 5.63] respectively. A manual audit of 32 generated rules and a replication of the selection gate are used to attribute this gap to failures in rule generation and rule selection rather than to policy execution. The paper concludes that the main challenge for agent-level TextGrad is reliably generating and selecting text policies, not executing them.

Significance. If the central claim holds, the paper makes a useful empirical contribution: it cleanly separates capacity from induction in text-policy optimization for frozen agents, and it provides a concrete capacity upper bound (human-written rules, +5.0 points) that future learning methods can be measured against. The experimental design is careful in several respects: policies are frozen before held-out evaluation, comparisons are paired on the same seeds and families, and the paper openly reports per-family heterogeneity and exploratory intervals. The capacity result is statistically solid with a pooled CI that excludes zero. The paper also gives reproducible details for the trace and counterfactual protocols and the selection gate. The main weakness is that the bottleneck attribution—generation and selection are the limiting factors—rests on null end-to-end results with wide CIs, a non-blind single-coder audit, and an uncalibrated hand-chosen gate. The data support a gap between capacity and observed learning performance, but not uniquely a diagnosis that generation/selection is the cause; low power, proposer capability, and the 80-word single-rule format are plausible alternative explana

major comments (4)
  1. [§5.2, Table 2] The central claim that trajectory-based learning 'does not recover the gain' rests on three end-to-end comparisons whose 95% confidence intervals all span zero (trace vs fixed −1.88 [−5.62, 1.88]; branch vs fixed −1.25 [−3.75, 0.00]; GEPA vs fixed +1.25 [−3.13, 5.63]). These data are compatible with a true zero effect, but they are also compatible with meaningful positive or negative effects. The paper correctly says these are 'not reliably' better, but the abstract and conclusion go further, attributing the gap specifically to generation and selection bottlenecks. That causal diagnosis is not uniquely identified from null results. To support the bottleneck claim, the paper should either (a) provide a sensitivity/power analysis showing that the design could detect a 5-point improvement if one existed, or (b) explicitly frame the conclusion as 'we found no evidence of a benefit' and treat
  2. [Appendix A, Table 7] The manual taxonomy of 32 generated rules is the primary qualitative evidence for the generation bottleneck. The paper concedes in Appendix A that the audit is 'descriptive, single-coder, not blind, and not independently double-coded.' A single-coder non-blind audit risks confirmation bias, especially because the auditor knows which rules came from which condition and likely knows the held-out outcomes. This weakens the specific breakdown (only 7/32 grounded, 16/32 shortcut/semantic/malformed) and the conclusion that generation quality is the issue. The paper should either obtain a second coder and report inter-rater agreement, or relegate this taxonomy to a hypothesis-generating observation with a much weaker claim. As it stands, this audit is a load-bearing part of the bottleneck attribution.
  3. [Appendix A, Selection protocol and audit] The selection gate is a hand-chosen, uncalibrated formula (score = r + 0.5s − 0.4i − 0.2p − 0.1q − 0.004T; acceptance ≥ 0.001 over fixed). The replication audit shows it accepts a harmful rule (Mistral coin, −20 points) and rejects a helpful rule (Mistral arithmetic, +5 points). With only eight development instances per model-family, these two errors could easily be noise rather than evidence that selection is structurally flawed. The gate's behavior is not compared to any alternative (e.g., a per-family paired test, uncertainty-aware selection, or a higher threshold). Since the paper uses this gate in all three end-to-end studies, the null results could stem from a poorly calibrated gate, not from an inherent selection bottleneck. The paper should either calibrate the gate against a validation set with known helpful/harmful rules, or show that the gate's errors are not due to small-samp
  4. [§5.3, Appendix A (Counterfactual protocol)] The counterfactual study produces only seven valid rules, and the paper acknowledges that 'only seven valid rules support claims about the update process itself.' Moreover, the same 7B model serves as both the acting agent and the rule proposer/reflector (stated in the GEPA section and the limitations). This means that the generation failures observed—vague advice, semantic errors, instance shortcuts—could reflect the limited capability of the 7B proposer rather than a fundamental difficulty of 'turning trajectory evidence into a reusable policy.' The paper's diagnosis would be stronger if it varied the proposer (e.g., a larger model) or showed that the same failures occur with a stronger proposer. Otherwise, the claim in §5.3 that 'two transitions limit end-to-end learning' overstates what a single-actor design can show.
minor comments (4)
  1. [§3, first paragraph] Typo: 'acapacity test' should be 'a capacity test'.
  2. [Table 2] The row labels 'Evidence', 'End to end', 'Credit', and 'Search' are slightly confusing because some rows are intermediate comparisons (trace vs summary, branch vs retry) while others are end-to-end. Consider separating the table into two panels or re-labeling rows to make clear which are end-to-end and which are diagnostic.
  3. [Abstract and Conclusion] The phrase 'clear gap' in the abstract implies the null results establish a true absence of effect. Consider rewording to 'we find little or no evidence that generated policies improve held-out success' to match the statistical strength of the results.
  4. [Appendix B] The paper reports that GEPA used '642 construction rollouts and 62 reflection calls.' It would help to clarify whether these counts are per model or pooled; the text says 'Together, the two searches use 642 environment rollouts and 62 reflection calls,' which seems to be pooled. A per-model breakdown would aid reproducibility.

Circularity Check

0 steps flagged

No significant circularity; the claims are held-out empirical comparisons with no parameter fitted to the predicted outcome and no load-bearing self-citation chain.

full rationale

The paper's central claim is an empirical gap between policy capacity and learned-policy performance, not a derivation from model equations. The capacity test uses human-written rules with supplied family routing and is explicitly labeled privileged information ('The rules and family routing are privileged'), so it is not presented as an autonomous prediction. Learned-policy comparisons freeze policies on held-out seeds, and the selection gate formula (r + 0.5s − 0.4i − 0.2p − 0.1q − 0.004T, acceptance ≥ 0.001) is stated to be fixed before test and uncalibrated; no parameter is fitted to the outcome being predicted. The generated-rule taxonomy is a descriptive qualitative audit and explicitly assigns labels 'without using held-out outcome,' so it does not reduce to the results it explains. Although the audit is non-blind, single-coder, and not independently double-coded, that is a measurement limitation, not circularity. There is also no load-bearing self-citation: references to TextGrad, GEPA, and other systems are external baselines or related work, not author self-citations invoked to force the conclusion. No step can be exhibited in which a claimed result is equivalent to its input by construction, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim rests on three supporting planks: (1) representativeness of the two 7B models and TextWorldExpress, (2) trustworthiness of a single-coder non-blind rule taxonomy, and (3) appropriateness of a hand-chosen selection gate. The paper explicitly acknowledges all three as limitations, making the central claim reasonable but conditional.

free parameters (3)
  • text-game gate weights (r, s, i, p, q, T) = score = r + 0.5s − 0.4i − 0.2p − 0.1q − 0.004T
    Hand-chosen coefficients in the development gate that decides whether a learned rule is deployed; the paper's end-to-end negative results pass through this gate. The paper states the formula was fixed before test but not calibrated or preregistered.
  • TextArena gate weights (r, s, i, u, T) = score = r + 0.5s − 0.5i − 0.25u − 0.005T
    Same as above, for the TextArena benchmark. These weights affect which rules are selected and thus the measured end-to-end gap.
  • acceptance threshold = 0.001
    A learned rule is kept only if its development score beats fixed policy by at least 0.001. The paper's sensitivity test shows stricter thresholds change acceptance (10/10 to 6/10) and test success, so the threshold is consequential for the negative result.
axioms (3)
  • domain assumption TextWorldExpress findings generalize to other agent domains
    The abstract frames 'the main challenge for agent-level TextGrad' generally, but the central measurements are on text games with two 7B models. The Limitations section concedes the findings may not transfer to web, software, robotics, or safety-critical agents.
  • domain assumption The single-coder manual taxonomy is a reliable measure of rule quality
    Counts from the 32-rule audit (7 grounded, 9 generic, 16 shortcuts/errors) are used to attribute failure to generation. The appendix states the audit is descriptive, single-coder, not blind, and not independently double-coded.
  • ad hoc to paper The hand-chosen gate formula and 0.001 threshold are representative of selection procedures
    Selection failure is diagnosed through a specific gate that was not calibrated. A better-calibrated gate could reduce or remove the end-to-end gap; the paper itself calls its post-hoc gate sensitivity analysis non-validating.

pith-pipeline@v1.3.0-alltime-deepseek · 10546 in / 13399 out tokens · 110298 ms · 2026-08-01T09:40:57.407735+00:00 · methodology

0 comments
read the original abstract

TextGrad improves language-model systems by revising text from feedback. Its core thesis is that natural-language feedback can act as a gradient for optimizing text components without changing model weights. Applying it to agents is harder because feedback arrives only after a sequence of actions, making it difficult to identify which decision caused failure. We study this problem by separating the ability to follow a useful policy from the ability to learn that policy from experience. Our main finding is a clear gap between these two abilities. Human-written policies improve two frozen 7B agents on TextWorldExpress by 5.0 success points, showing that useful policy text exists. However, policies generated from agent trajectories do not reliably outperform fixed prompting, even with richer traces, counterfactual evidence, or iterative GEPA search. The main challenge for agent-level TextGrad is therefore not executing textual policy updates, but reliably generating and selecting them from experience.

Figures

Figures reproduced from arXiv: 2607.20668 by Ankit Goyal, Jaideep Ray.

Figure 1
Figure 1. Figure 1: Agent-level TextGrad lifecycle. Actor weights remain frozen while training evidence proposes text-policy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 2 linked inside Pith

  1. [1]

    Optimizing Generative

    Yuksekgonul, Mert and Bianchi, Federico and Boen, Joseph and Liu, Sheng and Lu, Pan and Huang, Zhi and Guestrin, Carlos and Zou, James , journal =. Optimizing Generative. 2025 , doi =

    2025

  2. [2]

    Advances in Neural Information Processing Systems , volume =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =. 2023 , url =

    2023

  3. [3]

    2023 , url =

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =

    2023

  4. [4]

    2023 , url =

    Zhao, Andrew and Huang, Daniel and Xu, Quentin and Lin, Matthieu and Liu, Yong-Jin and Huang, Gao , journal =. 2023 , url =

    2023

  5. [5]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , doi =

    2023

  6. [6]

    2018 , url =

    C\^ot\'e, Marc-Alexandre and K\'ad\'ar, \`Akos and Yuan, Xingdi and Kybartas, Ben and Barnes, Tavian and Fine, Emery and Moore, James and Tao, Ruo Yu and Hausknecht, Matthew and El Asri, Layla and Adada, Mahmoud and Tay, Wendy and Trischler, Adam , journal =. 2018 , url =

    2018

  7. [7]

    2025 , url =

    Guertler, Leon and Cheng, Bobby and Yu, Simon and Liu, Bo and Choshen, Leshem and Tan, Cheston , journal =. 2025 , url =

    2025

  8. [8]

    and C\^ot\'e, Marc-Alexandre , journal =

    Jansen, Peter A. and C\^ot\'e, Marc-Alexandre , journal =. 2022 , url =

    2022

  9. [9]

    2024 , url =

    Dong, Yihong and Luo, Kangcheng and Jiang, Xue and Jin, Zhi and Li, Ge , booktitle =. 2024 , url =

    2024

  10. [10]

    2026 , url =

    Ling, Chen and Chen, Pei and Guan, Albert and Qu, Jiaming and Akbar, Shayan Ali and Gopinathan, Madhu and Cornejo, Erwin , journal =. 2026 , url =

    2026

  11. [11]

    Computing Research Repository , volume =

    Kintsugi: Learning Policies by Repairing Executable Knowledge Bases , author =. Computing Research Repository , volume =. 2026 , url =

    2026

  12. [12]

    Computing Research Repository , volume =

    Experiential Reinforcement Learning , author =. Computing Research Repository , volume =. 2026 , url =

    2026

  13. [13]

    and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J

    Agrawal, Lakshya A. and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J. and Jiang, Meng and Potts, Christopher and Sen, Koushik and Dimakis, Alexandros G. and Stoica, Ion and Klein, Dan and Zaharia, Matei and Khattab, Omar , journal =. 2026 , url =

    2026

  14. [14]

    Environment-Grounded Automated Prompt Optimization for

    Fernandes, Rean Clive and Fehring, Lukas and Eimer, Theresa and Lindauer, Marius and Feurer, Matthias , journal =. Environment-Grounded Automated Prompt Optimization for. 2026 , url =

    2026

  15. [15]

    arXiv preprint arXiv:2511.07919 , year =

    Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison , author =. arXiv preprint arXiv:2511.07919 , year =

    arXiv

  16. [16]

    arXiv preprint arXiv:2606.27136 , year =

    Joint Learning of Experiential Rules and Policies for Large Language Model Agents , author =. arXiv preprint arXiv:2606.27136 , year =

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2508.04660 , year =

    Composing Policy Gradients and Prompt Optimization for Language Model Programs , author =. arXiv preprint arXiv:2508.04660 , year =

    Pith/arXiv arXiv

  18. [18]

    2025 , url =

    Wang, Zihan and Wang, Kangrui and Wang, Qineng and Zhang, Pingyue and Li, Linjie and Yang, Zhengyuan and Jin, Xing and Yu, Kefan and Nguyen, Minh Nhat and Liu, Licheng and Gottlieb, Eli and Lu, Yiping and Cho, Kyunghyun and Wu, Jiajun and Fei-Fei, Li and Wang, Lijuan and Choi, Yejin and Li, Manling , journal =. 2025 , url =

    2025

  19. [19]

    2026 , url =

    Peng, Jiangweizhi and Liu, Yuanxin and Zhou, Ruida and Fleming, Charles and Wang, Zhaoran and Garcia, Alfredo and Hong, Mingyi , journal =. 2026 , url =

    2026

  20. [20]

    Hindsight Credit Assignment for Long-Horizon

    Tan, Hui-Ze and Yang, Xiao-Wen and Chen, Hao and Shao, Jie-Jing and Wen, Yi and Shen, Yuteng and Luo, Weihong and Du, Xiku and Guo, Lan-Zhe and Li, Yu-Feng , journal =. Hindsight Credit Assignment for Long-Horizon. 2026 , url =

    2026

  21. [21]

    2026 , url =

    Yao, Huaiyuan and Da, Longchao and Liu, Xiaoou and Fleming, Charles and Chen, Tianlong and Wei, Hua , journal =. 2026 , url =

    2026