Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Outcome-only RL collapses on multi-turn evidence reading; per-turn information-gain rewards prevent the collapse and lift F1 from 0.252 to 0.518.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:50 UTC pith:ZJANNJBF

load-bearing objection A useful diagnosis of GRPO's reward-variance collapse in multi-turn agent training, but the paper's namesake mechanism—information gain—is not isolated from generic variance injection. the 4 major comments →

arxiv 2607.16244 v1 pith:ZJANNJBF submitted 2026-06-26 cs.LG cs.AIcs.CL

CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents

Hao Dou This is my paper
classification cs.LG cs.AIcs.CL
keywords multi-turn reinforcement learningGRPOcredit assignmentinformation gainevidence readingHotpotQAreward variance collapse
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when a multi-turn evidence-reading agent is trained with outcome-only GRPO, the reward distribution can collapse: all sampled trajectories receive the same format penalty, group-relative advantages vanish, and the policy-gradient loss becomes zero, leaving the model stuck emitting garbled output. The paper identifies this zero-advantage lock-in as the root cause of GRPO's instability on HotpotQA at the 3B scale. To fix it, CIGPO injects variance at intermediate evidence-reading turns by rewarding the marginal increase in the frozen reference model's log-likelihood of the gold answer, using separate normalization for the information-gain and final-F1 rewards plus a curriculum that gradually raises the IG weight. With this mechanism, CIGPO reaches a standard F1 of 0.518 from a 0.252 base, while the best GRPO checkpoint reaches 0.430 and the final GRPO checkpoint collapses to 0.000. A sympathetic reader would care because it isolates a concrete, diagnosable failure mode in outcome-only RL and shows that a lightweight per-turn signal can keep training alive without a learned reward model.

Core claim

The paper's central claim is that GRPO's group-relative advantage computation requires non-zero reward variance within each group, and in multi-turn evidence reading the reward often collapses to a single value at the format-penalty floor, producing zero advantages and a zero policy-gradient loss. CIGPO prevents this collapse by assigning a per-turn reward to every intermediate evidence-reading turn: the contextual information gain, defined as the change in the frozen reference model's log-likelihood of the ground-truth answer after reading a new evidence block. On HotpotQA with Qwen2.5-3B-Instruct, CIGPO maintains meaningful reward variance across 200 training steps and improves standard F1

What carries the argument

The central object is the per-turn information-gain reward r_IG(t) = log p_ref(y* | q, e_<=t) - log p_ref(y* | q, e_<t), computed with a frozen reference model and placed on the last token of each intermediate turn. This signal is combined with the final-turn F1 reward, and the two are normalized separately within each GRPO group before advantage computation. An IG-weight curriculum (linearly from 0.1 to 0.3 over 200 steps) introduces the auxiliary reward gradually, and a wide safety clip (±50.0) preserves natural variance while preventing numerical instability. The mechanism that carries the argument is variance injection: the per-turn IG reward ensures that even when final-answer rewards a

Load-bearing premise

The method rests on the assumption that the frozen reference model's log-likelihood of the gold answer, conditioned only on the question and the evidence blocks read so far, is a meaningful and sufficiently stable measure of evidence-reading quality; the paper itself flags that this signal can be noisy and that format-violating trajectories still carry non-trivial IG, and the small group size of 2 makes the advantage computation inherently brittle.

What would settle it

Run GRPO on the same HotpotQA setup with a larger group size (e.g., 8): if the collapse disappears without any per-turn IG rewards, then the zero-advantage deadlock is an artifact of the N=2 binary advantage structure rather than a general failure of outcome-only GRPO. Alternatively, replace the IG reward with random per-turn noise matched to the same variance and retrain CIGPO: if training remains stable, the information content of the IG signal is irrelevant and the effect is purely variance injection.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the diagnosis is right, any outcome-only GRPO variant applied to multi-turn agent tasks should exhibit the same collapse whenever all group members receive identical terminal rewards, and training logs can detect it by watching score/reward means, zero-advantage group ratio, and pg_loss.
  • The success of CIGPO suggests that a frozen reference model can provide useful turn-level credit assignment without a separately trained process reward model, simplifying multi-turn agent training.
  • The F1-EM divergence seen late in CIGPO training indicates that the combined IG+F1 reward improves partial token recall while reducing exact-match precision, implying that reward design choices trade off these two metrics in a predictable way.
  • CIGPO's stability at group size 2 is notable because at N=2 advantages are binary; the variance-injection effect appears strong enough to overcome the most brittle advantage regime.
  • The paper's stated restriction to the Qwen2.5-3B HotpotQA setting means the immediate practical consequence is limited to that configuration, but the variance-injection principle is a candidate mechanism for other multi-turn settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper leaves open is whether the information content of the IG signal matters at all: replacing the reference-model log-likelihood with random per-turn noise matched to the same variance would test whether any non-constant intermediate reward can prevent zero-advantage lock-in, not just IG.
  • Since format-violating trajectories still show non-trivial cumulative IG (2.97), excluding malformed turns from IG normalization through validity gating could further improve signal-to-noise; this is a concrete, testable refinement suggested by the paper's own data.
  • The small group size (N=2) may be the main amplifier of the collapse; if so, the practical fix for well-resourced settings might be simply increasing group size, and CIGPO's benefit would concentrate in memory-constrained regimes where larger groups are infeasible.
  • The F1-EM trade-off implies that an F1-only or EM-only reward might produce a different optimum than the IG+F1 combination; investigating whether a precision-aware final reward resolves the divergence would clarify how much of F1's late-stage gain is attributable to the IG signal rather than the F1 reward.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the instability of outcome-only GRPO when training a Qwen2.5-3B-Instruct agent for multi-turn evidence reading on HotpotQA. It reports that GRPO initially improves (standard F1 0.430 at step 50) but collapses by step 150 to 100% format-violating outputs, and diagnoses a zero-advantage lock-in: all trajectories receive the same -2.0 format penalty, group-relative advantages vanish, and the policy-gradient loss becomes zero. The paper proposes CIGPO, which adds per-turn rewards computed as the marginal increase in a frozen reference model's log-likelihood of the ground-truth answer after reading each evidence block, along with separate normalization of IG and F1 rewards and a curriculum on the IG weight. CIGPO is reported to reach standard F1 0.518 at step 200, with format violations decreasing to 12.3%, and to maintain non-zero reward variance throughout training. The authors explicitly restrict their claims to the Qwen2.5-3B HotpotQA setting with group size N=2.

Significance. If the result holds, the paper identifies a concrete and plausible failure mode for small-group GRPO in multi-turn agent training, and proposes a lightweight mitigation that avoids training a separate process reward model. The diagnostic tables (especially Table 6) are concrete and checkable, and the paper is unusually honest about its limitations, including the lack of statistical testing, the small group size, and the failure to isolate components. However, the central attribution of the improvement to information gain, rather than to generic reward variance, is not currently supported by the experiments. The paper's own statements in Sections 7.1, 7.2, and 7.4 acknowledge that any varying turn-level signal might suffice, that the components are not ablated, and that all results come from a single run. These issues are load-bearing for the title and abstract claims, but they are addressable with additional experiments.

major comments (4)
  1. [§7.2, Tables 3–4] The central attribution to contextual information gain is not supported by the experiments. CIGPO changes three components relative to GRPO—per-turn IG rewards, separate per-metric normalization, and the IG-weight curriculum—and §7.2 explicitly states these are not isolated. §7.1 goes further and says that whether the injected signal is IG, a learned process reward, or a simpler heuristic may be secondary. Under the paper's own variance-injection hypothesis, any turn-level reward with sufficient group variance could prevent zero-advantage collapse. The title and abstract nonetheless credit IG specifically. A control run that replaces the IG reward with a variance-matched non-semantic signal (e.g., random noise or turn index), while keeping separate normalization and the curriculum, is essential; without it, the reported F1 improvement cannot be attributed to information gain.
  2. [§7.4, Tables 3–4] All empirical conclusions rest on a single training run per configuration, with no confidence intervals or significance tests. The paper acknowledges this. Given that GRPO itself is unstable (collapsing at different speeds in different runs), a single run does not establish that CIGPO's 0.518 F1 is reliably better than GRPO's best 0.430, nor that CIGPO avoids collapse generally. At minimum, three seeds per configuration with mean±std (or paired bootstrap on the 1,000-example evaluation set) are needed for the headline comparison.
  3. [§3.1, Table 2, §7.4] The GRPO collapse diagnosis is confounded by the choice N=2. With group size 2, advantages are binary (0 or ±1); any pair of identical rewards—including two format violations—gives zero advantage. The paper itself notes this may be partially an artifact of small group size. The claim that 'outcome-only GRPO collapses' in multi-turn settings is therefore too strong as stated; it should be restricted to the small-group regime unless experiments with N≥4 show the same collapse. This does not invalidate the observation, but it is load-bearing for the paper's general framing.
  4. [§6.5, Eq. (4), Table 7] The diagnostic correlation between cumulative IG and correctness is partly definitional and does not establish that IG rewards cause better evidence use. Both IG and F1 are computed against the same ground-truth y*, and the reference model's log-likelihood of y* will be higher for easier questions or for evidence blocks that happen to contain the answer string; the paper acknowledges this confound. Table 7 also shows format-violating trajectories have non-trivial cumulative IG (2.97), and the paper notes the lack of validity gating. This weakens the claim that IG is a clean signal of successful evidence use; an analysis of per-turn advantage directions or an IG-only ablation is needed.
minor comments (5)
  1. [§4.1] Eq. (4) uses pπref while Eq. (1) uses pref; unify the notation. Also make the conditioning explicit: e_1,...,e_t vs. e≤t is inconsistent.
  2. [§5] State whether the 1,000 training and 1,000 test examples are disjoint and how they were sampled from HotpotQA. Also specify whether evaluation uses greedy decoding or sampling; this matters for comparing checkpoints.
  3. [§6.4] The training-log metrics include 'critic/score/mean' although GRPO does not use a critic. Clarify the logger naming to avoid confusion.
  4. [Figures 1–2] The figure captions are present, but the figures themselves appear to be missing from the submitted text. Ensure they are included in the final version.
  5. [§7.4] The sentence reporting that preliminary 1.5B runs were unstable for both GRPO and CIGPO is an important negative result. Consider reporting it with concrete numbers or a small table rather than a single sentence.

Circularity Check

1 steps flagged

Headline F1 result is empirical and not circular; one side-claim (IG correlates with correct evidence use) is partly self-referential because cumulative IG and correctness both use the same ground-truth y*.

specific steps
  1. self definitional [Section 4.1 Eq. (4); Section 6.5 Table 7]
    "rIG_t = log p_ref(y*|q,e_1,...,e_t) - log p_ref(y*|q,e_1,...,e_{t-1}) ... Trajectories that produce correct answers have the highest cumulative IG (4.06), while valid wrong trajectories have the lowest IG (2.10)."

    Cumulative IG is, by construction, the total increase in the frozen reference model's log-likelihood of the gold answer y* over the episode. The outcome groups in Table 7 are defined by F1 against the same y* (F1>0.5 = Correct; F1=0 = Wrong). The reported correlation is therefore partly a check of the reward against a label derived from the same ground-truth token sequence, not an independent measure of evidence-reading quality. The paper itself cautions 'higher IG may reflect easier questions rather than better evidence-reading strategy' (Sec. 6.5). This does not make the headline F1 result circular, since F1 is measured independently of the IG training signal.

full rationale

The paper's central result is an empirical comparison: CIGPO reaches standard F1 0.518 versus 0.430 for the best GRPO checkpoint and 0.000 for final GRPO, with training-log evidence of zero-advantage collapse. These numbers are measured on a held-out test set and do not reduce to the reward definition by construction. The GRPO collapse diagnosis (Table 6) is an observed training-log fact. The method's reliance on IGPO [14] is not a self-citation and is not load-bearing: CIGPO adapts the idea to a closed evidence pool, and the main empirical claim would stand even if the IG signal were replaced by another turn-level reward, as the paper itself concedes in Sec. 7.1 ('Whether the injected signal is IG, a learned process reward, or a simpler heuristic may be secondary'). The only partial circularity is the side-claim in Sec. 6.5 that cumulative IG is correlated with successful evidence use: both the IG signal and the correctness label are functions of the same ground-truth answer y*, so the correlation is partly definitional. The paper explicitly acknowledges this caveat and does not use the correlation as the primary evidence for the F1 improvement. No self-citation chain, uniqueness theorem, or ansatz-smuggling is present. Score 2 reflects one minor self-referential side-claim; the central derivation/experiment is self-contained.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claim rests on standard GRPO equations, a domain assumption about reference-model log-likelihood measuring evidence quality, and several hand-chosen hyperparameters (IG curriculum, safety clip, normalization mode, format penalty, group size). No new physical or model entities are introduced. The paper itself lists the most important caveats: single run, no error bars, N=2 group size, and no ablations.

free parameters (5)
  • IG curriculum range (λIG) = 0.1 → 0.3 linear over 200 steps
    Hand-chosen schedule introduced in Eq. 7; no sensitivity analysis is reported.
  • IG safety clip = ±50.0
    Chosen after preliminary experiments showed that aggressive clipping (±0.5) collapses IG variance (Section 7.4).
  • Separate normalization mode = IG and F1 normalized independently
    Selected over joint normalization; the paper states joint normalization collapsed IG variance in preliminary experiments (Sections 7.2 and 7.4).
  • Format penalty = -2.0
    Hand-set minimum reward for format violations (Table 2); the exact value shapes the collapse dynamics described in Section 6.4.
  • GRPO group size N = 2
    Hardware constraint (2×24GB GPUs), but the paper admits the zero-advantage deadlock may be partially an artifact of N=2 (Section 7.4).
axioms (5)
  • standard math GRPO group-relative advantage formula A_i = (r_i − μ_group)/σ_group and the clipped surrogate objective correctly describe the training dynamics.
    Background from DeepSeekMath [4]; the paper reads training-log metrics through this framework in Section 6.4.
  • domain assumption The frozen reference model's log-likelihood log p_ref(y*|q, e_≤t) is a meaningful measure of how much the evidence read so far supports the correct answer.
    Entered via Eq. 1/4; the paper flags reference-model dependence and calibration concerns in Section 7.4.
  • domain assumption The HotpotQA evidence blocks are pre-indexed and the 1000-example train/test split is representative for this study.
    Section 5; chunking/indexing details are not released, so the setup must be taken on faith.
  • ad hoc to paper Preserving non-zero reward variance among GRPO group members is sufficient to prevent zero-advantage collapse.
    This is the paper's 'variance-injection' thesis (Sections 4 and 7.1), but no controlled ablation rules out alternative explanations such as normalization or curriculum effects.
  • domain assumption Training-log metrics (pg_loss=0, advantages mean 0, entropy ≈4809) indicate an unrecoverable optimization deadlock.
    Section 6.4 diagnoses this from one run at step 200; the paper acknowledges that small group size may make the artifact specific to this setup.

pith-pipeline@v1.3.0-alltime-deepseek · 10143 in / 12341 out tokens · 104747 ms · 2026-08-02T09:50:10.574842+00:00 · methodology

0 comments
read the original abstract

Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit. In HotpotQA experiments with Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 0.430) but subsequently collapses to 100% format-violating outputs. Training-log diagnosis reveals a zero-advantage lock-in mechanism: all sampled trajectories receive the minimum format penalty (-2.0), group-relative advantages vanish, and the policy-gradient loss becomes zero--an optimization deadlock. We propose a variance-injection strategy: by assigning per-turn rewards to intermediate evidence-reading turns, we prevent the group reward distribution from collapsing to a single value--preserving the variation that GRPO's group-relative advantage requires. Contextual Information-Gain Policy Optimization (CIGPO) implements this strategy using the marginal increase in the frozen reference model's log-likelihood of the ground-truth answer as the per-turn signal. With separate normalization of IG and F1 rewards and an IG-weight curriculum, CIGPO reaches a standard F1 of 0.518 on HotpotQA at the 3B scale (from 0.252 base; +105%), compared with 0.430 for the best GRPO checkpoint and 0.000 for the final GRPO checkpoint. CIGPO maintains meaningful reward variance and avoids zero-advantage lock-in throughout training. These results identify reward-variance collapse as a concrete failure mode of outcome-only GRPO and show that turn-level IG rewards can prevent it in this HotpotQA setting.

Figures

Figures reproduced from arXiv: 2607.16244 by Hao Dou.

Figure 1
Figure 1. Figure 1: Overview of the CIGPO training framework. Given a question and a pool of pre-indexed evidence documents, the policy model (trained via GRPO with IG-augmented rewards) selects and reads evidence blocks across multiple turns. At each intermediate turn t, the contextual information gain ∆t = log pref(y ∗ | q, e≤t) − log pref(y ∗ | q, e<t) is computed using the frozen reference model and assigned as a per-turn… view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics. Left: Standard F1 (format violations scored as 0). CIGPO improves from 0.429 to 0.518; GRPO initially reaches 0.430 at step 50 but collapses to 0.000. Right: Format violation rate. GRPO violations rise from 21% to 100%; CIGPO violations decrease from 32% to 12%. 6.3 Full Benchmark Results [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 9 linked inside Pith

  1. [1]

    S. Yao, J. Zhao, D. Yu, et al. ReAct: Synergizing reasoning and acting in language models. InICLR, 2023

    2023

  2. [2]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, et al. Toolformer: Language models can teach themselves to use tools. InNeurIPS, 2023

    2023

  3. [3]

    L. Wang, C. Ma, X. Feng, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), 2024

    2024

  4. [4]

    Z. Shao, P. Wang, Q. Zhu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Z. Yang, P. Qi, S. Zhang, et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InEMNLP, 2018

    2018

  6. [6]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, et al. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023

    2023

  7. [7]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  8. [8]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. InNeurIPS, 2022

    2022

  9. [9]

    Y. Bai, S. Kadavath, S. Kundu, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    DeepSeek-AI, D. Guo, D. Yang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Putta, E

    P. Putta, E. Mills, N. Garg, et al. Agent Q: Advanced reasoning and learning for autonomous AI agents. arXiv preprint arXiv:2408.07199, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    C.-K. Wu, Z. R. Tam, C.-Y. Lin, et al. StreamBench: Towards benchmarking continuous improvement of language agents.arXiv preprint arXiv:2406.08747, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Zhang, S

    D. Zhang, S. Zhoubian, Z. Hu, et al. ReST-MCTS*: LLM self-training via process reward guided tree search.arXiv preprint arXiv:2406.03816, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    G. Wang, S. Dai, G. Ye, et al. Information gain-based policy optimization: A simple and effective approach for multi-turn LLM agents.arXiv preprint arXiv:2510.14967, 2025

    arXiv 2025

  15. [15]

    B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison, 2009. 15

    2009

  16. [16]

    Z. Wu, Y. Hu, W. Shi, et al. Fine-grained human feedback gives better rewards for language model training. InNeurIPS, 2023

    2023

  17. [17]

    Lightman, V

    H. Lightman, V. Kosaraju, Y. Burda, et al. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    M. Minsky. Steps toward artificial intelligence.Proceedings of the IRE, 49(1):8–30, 1961

    1961

  19. [19]

    R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018

    2018

  20. [20]

    Qwen, A. Yang, B. Yang, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    W. Kwon, Z. Li, S. Zhuang, et al. Efficient memory management for large language model serving with PagedAttention. InSOSP, 2023. 16

    2023