arxiv: 2601.06794 · v2 · submitted 2026-01-11 · 💻 cs.AI

Recognition: no theorem link

No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning

Zhicong Li , Lingjie Jiang , Yulan Hu , Xingchen Zeng , Yixia Li , Xiangwen Zhang , Guanhua Chen , Zheng Pan

show 2 more authors

Xin Li Yong Liu

Authors on Pith no claims yet

Pith reviewed 2026-05-16 15:58 UTC · model grok-4.3

classification 💻 cs.AI

keywords co-evolving criticscritique-guided reinforcement learningopen-world agentsECHO frameworkcascaded rolloutsaturation-aware gain shapingdual-track GRPOhindsight-guided optimization

0 comments

The pith

Co-evolving the critic with the policy prevents stale feedback and raises long-horizon success in open-world agent training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that static critics in critique-guided reinforcement learning quickly become outdated because an agent's error patterns change during training. ECHO solves this by running a synchronized co-evolutionary loop in which the critic and policy are updated together. A cascaded rollout lets the critic produce multiple diagnoses per trajectory, group-structured advantage estimation is performed, and a saturation-aware gain shaping objective pushes the critic to keep inducing small gains even on strong trajectories. Dual-track GRPO updates keep the two components aligned. The result is more stable training curves and higher success rates on long-horizon tasks in open-world environments.

Core claim

ECHO jointly optimizes the policy and critic through a synchronized co-evolutionary loop that uses cascaded rollouts to generate multiple diagnoses per trajectory, saturation-aware gain shaping to reward incremental improvements on high-performing trajectories, and dual-track GRPO updates to keep feedback synchronized with the evolving policy, producing more stable training and higher long-horizon task success.

What carries the argument

The co-evolutionary loop consisting of cascaded rollout for group-structured advantage estimation, saturation-aware gain shaping objective, and dual-track GRPO updates that keep the critic's diagnoses aligned with the current policy.

If this is right

Critic feedback remains useful across the entire training run instead of losing value as the policy improves.
Group-structured advantage estimates from multiple diagnoses per trajectory reduce variance in policy updates.
Saturation-aware shaping prevents the critic from stopping to provide useful signals once trajectories reach high performance.
Dual-track GRPO updates maintain alignment between policy and critic without requiring separate offline critic retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same synchronization loop could be applied to other model-based feedback sources such as reward models or value heads that are trained alongside the policy.
In settings where human feedback is expensive, the co-evolution may reduce the frequency of needed human overrides.
The approach may scale to multi-agent or multi-task open-world settings where error patterns shift differently across tasks.

Load-bearing premise

The cascaded rollout and saturation-aware objective will keep the critic synchronized with the policy without creating new instabilities or biased advantage estimates.

What would settle it

Training curves in which success rate plateaus or drops after the first 20 percent of episodes while the critic's average diagnostic quality, measured by its correlation with actual outcome improvement, declines.

Figures

Figures reproduced from arXiv: 2601.06794 by Guanhua Chen, Lingjie Jiang, Xiangwen Zhang, Xingchen Zeng, Xin Li, Yixia Li, Yong Liu, Yulan Hu, Zheng Pan, Zhicong Li.

**Figure 1.** Figure 1: Comparison of critic paradigms. (a) Conventional Static Paradigms: Use decoupled, frozen critic modules initialized from off-the-shelf templates or fine-tuned separate models, resulting in static evaluation and inflexible feedback. (b) Our ECHO Paradigm: Policy and critic co-evolve organically. The policy first generates an initial rollout τo, refined to τr using the critic’s diagnostic guidance c. Both mo… view at source ↗

**Figure 2.** Figure 2: Overview of ECHO training with saturation-aware (SA) critic rewards. At step [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Failure-pattern drift across training phases. We visualize failed trajectories from [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Effect of saturation-aware gain shaping on last-mile refinement. We plot density scatter maps of pre [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Training reward curves across four environ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

read the original abstract

Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ECHO gives a concrete algorithmic fix for stale critics in LLM agent RL via co-evolution and cascaded rollouts, but the abstract's claims rest on unshown experiments.

read the letter

The main thing here is that the paper introduces ECHO to keep the critic synchronized with an evolving policy in critique-guided RL for open-world LLM agents. It uses cascaded rollouts to produce multiple diagnoses per trajectory for group advantage estimates, adds saturation-aware gain shaping to reward incremental gains on strong trajectories, and applies dual-track GRPO updates to maintain alignment. This directly targets the real issue that static critics lose utility as the agent's error patterns shift during on-policy training. The framing of the problem and the specific loop are clear and practical for this subfield. The saturation-aware term and the cascaded mechanism look like reasonable engineering choices to avoid plateaus and generate structured advantages. The paper does a solid job spelling out why offline critics fail and sketching how the co-evolutionary setup could keep feedback useful over long horizons. That said, the abstract states that ECHO produces more stable training and higher long-horizon success, yet supplies no numbers, baselines, ablations, or error bars. Without those details it is hard to tell whether the gains are real or whether the cascaded rollouts introduce the kind of unaccounted variance and bias in advantage estimates that the stress-test note raises. If the full paper contains careful comparisons and checks on estimation stability, the contribution strengthens; otherwise the central claim stays unverified. This work is aimed at researchers already working on RL for LLM agents in complex environments. Someone in that niche would get value from the algorithmic details even if the results require more scrutiny. It deserves a serious referee because the problem is concrete and the proposed mechanisms are specific enough to evaluate. I would send it out for review rather than desk reject, with the expectation that revisions will need to show the empirical backing and address potential bias in the group advantages.

Referee Report

2 major / 1 minor

Summary. The paper introduces ECHO, a framework for co-evolving a policy and critic in critique-guided RL for LLM agents. It proposes a cascaded rollout mechanism to generate multiple diagnoses per trajectory for group-structured advantage estimation, a saturation-aware gain shaping objective to escape learning plateaus, and dual-track GRPO updates to maintain critic-policy synchronization, claiming more stable training and higher long-horizon task success in open-world environments.

Significance. If the empirical claims hold with proper controls, the co-evolutionary approach could meaningfully advance on-policy RL for agents by mitigating critic staleness, a known issue in evolving error distributions. The cascaded rollout and gain-shaping ideas are technically interesting and could generalize beyond the specific setting if shown to bound bias in advantage estimates.

major comments (2)

[Abstract] Abstract: The central empirical claim ('ECHO yields more stable training and higher long-horizon task success') is stated without any quantitative results, baselines, ablation details, or error bars. This absence prevents verification of the magnitude of improvement or confirmation that gains arise from sustained co-evolution rather than transient effects.
[Methods] Methods (cascaded rollout description): The group-structured advantage estimation from multiple diagnoses per trajectory lacks an explicit derivation or bound showing that the saturation-aware gain shaping term prevents systematic bias as the policy distribution shifts beyond the initial rollout support. Without re-weighting by diagnosis uncertainty or normalization to the evolving policy, advantage estimates risk the bias identified in the stress-test note.

minor comments (1)

[Abstract] Abstract: The acronym expansion 'Evolving Critic for Hindsight-Guided Optimization' appears after the first use of ECHO; move the definition to first mention for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our ECHO framework. The comments highlight opportunities to strengthen the abstract's empirical presentation and the theoretical grounding of our advantage estimation. We respond to each major comment below and commit to revisions that address the concerns without overstating our current results.

read point-by-point responses

Referee: [Abstract] Abstract: The central empirical claim ('ECHO yields more stable training and higher long-horizon task success') is stated without any quantitative results, baselines, ablation details, or error bars. This absence prevents verification of the magnitude of improvement or confirmation that gains arise from sustained co-evolution rather than transient effects.

Authors: We agree that the abstract would benefit from quantitative highlights to allow immediate assessment of the claimed improvements. The body of the manuscript (Section 4) already contains the supporting experiments, including comparisons to baselines, ablation studies, training curves with error bars, and metrics for both stability and long-horizon success. In the revised version we will update the abstract to include specific quantitative findings, such as relative success-rate gains and stability improvements drawn directly from our tables and figures, while preserving the overall length constraints. revision: yes
Referee: [Methods] Methods (cascaded rollout description): The group-structured advantage estimation from multiple diagnoses per trajectory lacks an explicit derivation or bound showing that the saturation-aware gain shaping term prevents systematic bias as the policy distribution shifts beyond the initial rollout support. Without re-weighting by diagnosis uncertainty or normalization to the evolving policy, advantage estimates risk the bias identified in the stress-test note.

Authors: We appreciate the referee's emphasis on rigorously bounding bias in the advantage estimator. The cascaded rollout and saturation-aware shaping are motivated by the need to adapt to shifting error distributions, and the dual-track GRPO updates are intended to keep the critic synchronized. To address the gap, we will add a dedicated paragraph in the Methods section that derives the group-structured advantage under the co-evolutionary loop and shows how the shaping term, combined with multiple diagnoses per trajectory, limits bias relative to the initial rollout support. We will also incorporate a short discussion of the stress-test results to illustrate empirical robustness; if a tighter theoretical bound requires additional assumptions or re-weighting, we will note this as a limitation and outline it for future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity: ECHO framework is an independent algorithmic proposal

full rationale

The provided abstract and description introduce ECHO as a co-evolutionary RL framework using cascaded rollout for group-structured advantages, saturation-aware gain shaping, and dual-track GRPO updates. No equations, fitted parameters, or self-citations are shown that reduce any claimed prediction or result to its own inputs by construction. The central claims rest on the described synchronization mechanism rather than self-definitional loops or renamed empirical patterns. This qualifies as a normal non-finding of circularity for an algorithmic contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The method implicitly assumes that co-evolution can be maintained through the described update rules without additional regularization.

pith-pipeline@v0.9.0 · 5515 in / 1083 out tokens · 28052 ms · 2026-05-16T15:58:48.711409+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 12 internal anchors

[1]

GPT-4 Technical Report

Gpt-4 techni- cal report.arXiv preprint arXiv:2303.08774. AI Anthropic

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Training Verifiers to Solve Math Word Problems

Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others

work page internal anchor Pith review Pith/arXiv arXiv
[3]

Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston

work page internal anchor Pith review Pith/arXiv arXiv
[4]

InFindings of the association for computational linguistics: ACL 2024, pages 3563–3578

Chain-of-verification reduces hallucination in large language models. InFindings of the association for computational linguistics: ACL 2024, pages 3563–3578. Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, and 1 others

work page 2024
[5]

InFindings of the Association for Computational Linguistics: ACL 2025, pages 14588–14604

Llm crit- ics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. InFindings of the Association for Computational Linguistics: ACL 2025, pages 14588–14604. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others

work page 2025
[6]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song

work page internal anchor Pith review Pith/arXiv arXiv
[7]

Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, and 1 others

Boosting mllm reason- ing with text-debiased hint-grpo.arXiv preprint arXiv:2503.23905. Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, and 1 others

work page arXiv
[8]

ArXiv:2507.10628 [cs]

GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforce- ment Learning.arXiv preprint. ArXiv:2507.10628 [cs]. Laurens van der Maaten and Geoffrey Hinton

work page arXiv
[9]

Andrew Y Ng, Daishi Harada, and Stuart Russell

Llm critics help catch llm bugs.arXiv preprint arXiv:2407.00215. Andrew Y Ng, Daishi Harada, and Stuart Russell

work page arXiv
[10]

Self-critiquing models for assisting human evaluators

Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others

work page internal anchor Pith review arXiv
[11]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew 9 Hausknecht

work page internal anchor Pith review Pith/arXiv arXiv
[12]

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

Alfworld: Aligning text and em- bodied environments for interactive learning.arXiv preprint arXiv:2010.03768. Richard S Sutton, Andrew G Barto, and 1 others

work page internal anchor Pith review Pith/arXiv arXiv 2010
[13]

Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

Do not step into the same river twice: Learning to reason from trial and error.arXiv preprint arXiv:2510.26109. Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, and 1 others. Self-evolving critique abilities in large language models. InSecond Conference on Language Modeling. Kimi Te...

work page internal anchor Pith review Pith/arXiv arXiv
[14]

Kimi K2: Open Agentic Intelligence

Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534. Qwen Team and 1 others

work page internal anchor Pith review Pith/arXiv arXiv
[15]

Qwen2 Technical Report

Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022a. Scienceworld: Is your agent smarter than a 5th grader?arXiv preprint arXiv:2203.07540. Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Ji- aqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, and Yanghua Xiao

work page internal anchor Pith review Pith/arXiv arXiv
[16]

arXiv preprint arXiv:2510.09388

Hint: Helping ineffective rollouts navigate towards effectiveness. arXiv preprint arXiv:2510.09388. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171. Xumeng Wen, Zihan Liu, Shun Zheng, ...

work page arXiv
[17]

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs

Rein- forcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245. Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, and 1 others

work page internal anchor Pith review Pith/arXiv arXiv
[18]

Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others

Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755. Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others

work page arXiv
[19]

Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong

En- hancing llm reasoning via critique models with test- time and training-time supervision.arXiv preprint arXiv:2411.16579. Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong

work page arXiv
[20]

Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang

Teaching language models to critique via reinforcement learn- ing.arXiv preprint arXiv:2502.03492. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang

work page arXiv
[21]

Learning to Reason under Off-Policy Guidance

Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report.arXiv preprint arXiv:2505.09388. Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li,...

work page internal anchor Pith review Pith/arXiv arXiv
[22]

arXiv preprint arXiv:2506.22157

Train- ing language model to critique for better refinement. arXiv preprint arXiv:2506.22157. Feng Zhang, Zezhong Tan, Xinhong Ma, Ziqiang Dong, Xi Leng, Jianfei Zhao, Xin Sun, and Yang Yang. 2025a. Adhint: Adaptive hints with difficulty priors for reinforcement learning.arXiv preprint arXiv:2512.13095. Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng W...

work page arXiv