Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Hindsight Policy Optimization claims that long-horizon language-agent RL instability comes from pointwise comparisons in a huge discrete action space, and that an intent-space Wasserstein comparison yields a bounded-variance, critic-free si

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:40 UTC pith:GMNSVW7G

load-bearing objection A genuinely new credit-assignment idea for LLM agents, with promising results, but the theory overreaches at Eq. (14) and the main experiments are inflated by best-checkpoint-on-test reporting. the 4 major comments →

arxiv 2607.16257 v1 pith:GMNSVW7G submitted 2026-06-28 cs.LG cs.AIcs.CL

From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training

Zishang Jiang , Tingyun Li , Jinyi Han , Xinyi Wang , Sihang Jiang , Yizhou Ying , Xiaojun Meng , Jiansheng Wei
show 2 more authors
Jiaqing Liang Yanghua Xiao
This is my paper
classification cs.LG cs.AIcs.CL
keywords hindsight policy optimizationlong-horizon reinforcement learninglanguage agentsWasserstein distanceintent spacevariance reductioncredit assignmentLLM training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the instability of reinforcement learning for long-horizon language agents comes from comparing the current policy and the hindsight (successful-trajectory) distribution pointwise in a vast discrete language action space. It introduces Hindsight Policy Optimization (HPO), which embeds state-action pairs into a semantic intent space and uses the Wasserstein distance between the two distributions as the learning signal. The paper argues that this shift turns an estimator whose variance can be arbitrarily large into one bounded by the embedding-space diameter, and that this stabilizes and accelerates training without a critic. Experiments on search QA and a text-based game environment report consistent gains and show that the step-level signal can drive learning even when outcome rewards are removed.

Core claim

This paper establishes that a policy gradient estimator can be built by treating 'what the agent would have done, knowing the final outcome' (the hindsight distribution) as the target, then measuring the Wasserstein distance in an intent space between the current occupancy and that hindsight occupancy. The dual Kantorovich potential serves as a per-step advantage. Because this potential is 1-Lipschitz, its variance under the current policy is bounded by D²/4, where D is the diameter of the embedded state-action support; the KL-based pointwise ratio signal has variance equal to a chi-squared divergence that can be arbitrarily large. HPO therefore replaces the unbounded pointwise ratio with a

What carries the argument

The central object is the hindsight occupancy measure ρhπ(s,a)=ρπ(s,a)Qπ(s,a)/Z, the distribution over state-action pairs that would be obtained by resampling actions after observing a successful outcome. HPO compares ρhπ with the current occupancy ρπ through the 1-Wasserstein distance W1(ρπ, ρhπ) on the intent embedding space; the optimal dual function f* — the Kantorovich potential — provides the step-level advantage. Its 1-Lipschitz property is what yields the bounded variance, allowing semantic neighbours to share statistical evidence instead of being treated as isolated atoms.

Load-bearing premise

The theoretical hindsight distribution is defined relative to the current policy, but offline HPO constructs it once from a fixed set of eight successful trajectories per query collected by the initial policy and never re-estimates it as training changes the policy, so the step-level advantage can be a biased estimate of the intended hindsight signal with no bound on that distribution-shift bias.

What would settle it

Train HPO on a task where per-step causal credit is known by counterfactual action replacement, and check whether the step-level advantage ranks actions in the same order as the ground-truth causal effect; disagreement would show that bounded variance was achieved at the price of unacceptably biased credit assignment. A second check: keep the embedding fixed and measure the empirical variance of the Kantorovich potential across batches as the horizon grows; if it grows without bound despite a fixed embedding diameter, the assumptions behind the variance bound are violated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-horizon language-agent training can be stabilized without training a critic, avoiding the slow warm-up of PPO while retaining step-level credit assignment.
  • The variance bound D²/4 means the learning signal degrades gracefully with the semantic diameter of the intent embedding, so better embedding spaces directly improve signal-to-noise ratio.
  • Because the step-level advantage comes from mass transport between distributions, statistical evidence is shared across semantically similar surface forms, improving sample efficiency in sparse-reward settings.
  • HPO's step-level advantage alone is sufficient to drive policy improvement, offering a route to learning from offline data that has already been evaluated without requiring new outcome rewards.
  • In the evaluated long-horizon tasks, HPO improves average performance by 4–7 points over PPO and GRPO while adding less than 1% per-iteration overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the hindsight distribution were periodically refreshed from the current policy's successful rollouts, the distribution-shift bias in offline HPO might shrink; the paper's online/offline comparison suggests a middle ground that combines stability with adaptivity, but the paper does not test it.
  • The same intent-space Wasserstein argument should transfer to dense or continuous rewards via the reweighted hindsight definition in Appendix B, and to multimodal agents if a suitable semantic embedding exists; these extensions are left untested.
  • HPO's practical guarantees depend on the quality of the semantic encoder; the reported encoder-ablation rank correlations show consistency across encoders, but rank alignment does not by itself establish that the resulting advantages are correct credit assignment.
  • A natural extension would use HPO's step-level advantage as a dense reward for a critic-based method, potentially combining the variance reduction of both families; this is not explored in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Hindsight Policy Optimization (HPO), a policy-gradient method for long-horizon language-agent RL. It defines a hindsight distribution of successful state-action pairs, projects these into a semantic embedding space, and uses the Kantorovich potential of the 1-Wasserstein distance between the current and hindsight distributions as a step-level advantage signal. This is combined with a GRPO-style episode-level advantage. The authors claim that aggregating semantically similar actions in intent space yields a bounded-variance estimator and more stable training. Experiments on SearchQA and TextCraft with Qwen2.5-3B/7B show consistent improvements over PPO, GRPO, Search-R1, and ZeroSearch, and ablations indicate that the step-level advantage alone can drive learning. The paper also reports negligible computational overhead for the added Wasserstein computation.

Significance. If the theoretical claims were valid, HPO would be a notable contribution: a critic-free, semantically aware estimator that reduces gradient variance in long-horizon LLM agent training while adding under 1% overhead. The empirical results are strong and well supported: multiple benchmarks, multiple seeds, ablations on the step-level weight, embedding-model robustness, and an interpretability case study. The paper also provides a public code link. However, the theoretical foundation has serious gaps: the key policy-gradient identity is asserted without proof, the variance bound is generic and not tied to the actual estimator, and the offline variant suffers from an unquantified distribution-shift bias. The central claim therefore currently rests on empirical evidence plus a heuristic bias-variance argument.

major comments (4)
  1. [§4.3, Eq. (14); §4.4, Lemma 4.4] Equation (14) asserts ∇θJ(θ) = E_{ρπ}[f* ∇logπ] with f* the Kantorovich potential of W1(ρπ, ρhπ), but Lemma 4.2 only derives this form for the KL divergence, where the signal is the pointwise ratio ρhπ/ρπ. W1 is a different functional; its derivative with respect to ρπ does not automatically yield the policy gradient of J. No proof or bias bound is given that E[f*∇logπ] is a valid ascent direction. Moreover, the sign conventions conflict: Eq. (14) defines δW1/δρπ = f*, while Lemma 4.4 defines f* = −δW1/δρπ, and Eq. (16) uses AS = (−f* − mean)/std. The case study in §E.2 assigns positive AS to the final correct answer and negative AS to an uninformative step, which is reversed if f* is the usual Kantorovich potential for W1(ρπ, ρhπ). This sign inconsistency makes the actual updated quantity ambiguous and needs resolution.
  2. [§4.4, Lemma 4.4] The bound Varρπ(f*) ≤ D²/4 holds for any 1-Lipschitz function on the support of ρπ, not specifically for the Kantorovich potential. It therefore does not validate f* as a useful credit-assignment signal. Furthermore, the bound is on the population variance of f* under ρπ, whereas Algorithm 1 uses an empirical distributions constructed from G rollouts (and 8 offline trajectories for HPOoff). The variance of the finite-sample estimator is not analyzed. The paper's claim of a 'bounded-variance estimator' is thus not actually established. The authors should state the estimator whose variance is bounded and provide a finite-sample bound, or weaken the claim.
  3. [§4.2, Eq. (12); Appendix B; Algorithm 1] The hindsight distribution is defined theoretically as ρhπ = ρπ Qπ / Z, which depends on the current policy π. In offline HPO, ρhπ is instead constructed once from a fixed set of 8 successful trajectories generated by the initial policy. As training changes the policy, this fixed distribution is not recomputed against the current Qπ, so the step-level advantage is a biased estimator of the intended hindsight signal. No bound on this distribution-shift bias is provided. Since the main results (HPOoff) rely on this offline construction, the central theoretical justification does not apply to the primary evaluation setting.
  4. [§2.2, Eq. (12); §4.2] The occupancy measures in Eq. (12) are defined as sums over time of Dirac measures without a normalizing factor, yet the Wasserstein distance in Eq. (13) requires probability measures. With γ = 1, the (1−γ) factor in §2.2 vanishes, so the measures are unnormalized. The paper needs to clarify how ρπ and ρhπ are normalized before computing W1, or the theoretical statements are not well-defined.
minor comments (5)
  1. [§4.3, Proposition 4.3] Typo in the proposition title: 'estimatior' should be 'estimator'.
  2. [Appendix A.1] Typo: 'search senario' should be 'search scenario'.
  3. [§E.2] The case study is repeated three times ('Example 1: the advantage of GRPO', 'Example 1: the advantage of PPO', 'Example 1: the advantage of HPO') with identical trajectories. This appears to be an editorial duplication; only the HPO version is relevant to the main text.
  4. [§D.1, Lemma 3.1 Proof] The proof of Lemma 3.1 invokes a 'standard simplifying assumption' that g(a) and Q̂(s,a) are conditionally independent given s, but this assumption is not stated in the lemma. It should be stated explicitly in the main text, since the variance decomposition does not hold in full generality.
  5. [§4.3, Eq. (16)] The normalization in Eq. (16) uses std(f*) in the denominator but subtracts mean(−f*) in the numerator; this is algebraically equivalent to normalizing −f*. Please clarify the intended sign convention and ensure consistency with Eq. (14).

Circularity Check

0 steps flagged

No circularity found: the variance bound is a genuine consequence of 1-Lipschitzness; the main theoretical weakness (Eq. 14) is an unproved substitution, not a circular reduction.

full rationale

I traced the derivation chain. Definition 4.1 / Appendix B sets ρhπ(s,a)=ρπ(s,a)Qπ(s,a)/Z, and Lemma 4.2 / D.3 rewrites the policy gradient theorem as Eρπ[(ρhπ/ρπ)∇logπ] up to a positive constant; this is an algebraic identity, not a fitted prediction, and it does not presuppose the W1 result. Lemma 4.4's bound Varρπ(f*)≤D²/4 follows from the Kantorovich potential being 1-Lipschitz plus Popoviciu's inequality; the KL variance formula is chi-squared divergence by direct calculation. Neither theorem is calibrated to the experimental data, so there is no fitted-input-called-prediction pattern. The hand-tuned ω and fixed 8-trajectory offline hindsight set are design choices, not parameters fitted to the predicted quantity. I found no self-citations or imported uniqueness theorems; the Wasserstein duality is cited to standard external references (Villani; Ambrosio et al.). The one load-bearing weakness is Eq. (14), which asserts that replacing the KL variational derivative in Lemma 4.2 by the W1 Kantorovich potential yields a policy gradient; the paper does not prove that −δW1/δρπ preserves an ascent direction for J(θ). That is an omitted/invalid inference and a correctness risk, but it is not circular—Eq. (14) is asserted, not reduced to the definitions. Likewise, the known limitation that offline HPO's fixed hindsight distribution drifts from the on-policy Qπ is an acknowledged bias, not a circular construction. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central claim rests mainly on a domain assumption about semantic embeddings and on the ad-hoc approximation of the on-policy hindsight distribution by a fixed offline set. Free parameters are limited to ω and the offline trajectory count; no fundamentally unexplained new entities are introduced.

free parameters (2)
  • ω (step-level advantage weight) = 0.5
    Eq (17) interpolates episode-level and step-level advantages; ω=0.5 is hand-chosen, and Figure 7 shows sensitivity to it.
  • Number of offline correct trajectories per query = 8
    Appendix A.2 uses 8 offline correct trajectories per query to construct the intent-space hindsight distribution; this is an arbitrary choice that directly affects the quality of the step-level signal.
axioms (6)
  • domain assumption The pretrained embedding encoder Φ maps semantically similar state-action pairs to nearby Euclidean points.
    Section 4.2 assumes Qwen3-Embedding-0.6B captures action semantics well enough for Wasserstein aggregation to improve value estimation.
  • ad hoc to paper The offline hindsight distribution built from initial-policy successes approximates the theoretical on-policy ρhπ = ρπ Qπ/Z.
    Section 4.2 introduces offline HPO with a fixed set of successful trajectories; no bias bound is given for the distribution shift as policy updates.
  • domain assumption The simplifying assumption that g(a) and Q(s,a) are conditionally uncorrelated given s.
    Appendix D.1 uses this to derive the variance decomposition in Lemma 3.1; without it the stated product-form variance is not exact.
  • standard math The support diameter D in the intent space is finite and the variance bound D²/4 is meaningful.
    Lemma 4.4 invokes bounded support and Popoviciu's inequality; in practice D can grow with the embedding scale and horizon, but the formal statement assumes finiteness.
  • domain assumption Binary episodic rewards with γ=1 and sparse terminal reward.
    Section 2.1 formulates the MDP with sparse rewards; the hindsight distribution definition and experiments rely on this.
  • standard math Standard policy gradient theorem and occupancy-measure one-to-one correspondence.
    Section 2.2 and Lemma 4.2 rely on these standard RL facts.

pith-pipeline@v1.3.0-alltime-deepseek · 20998 in / 11418 out tokens · 105043 ms · 2026-08-02T09:40:45.427615+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has become a widely adopted technique for improving large language models (LLMs) on complex tasks. Despite this progress, existing RL methods still face challenges in training agents with longer-horizon interactions. One major bottleneck is distinguishing the contribution of different actions in long-horizon interaction, leading to high optimization variance. To address this, we introduce a novel policy gradient method, Hindsight Policy Optimization (HPO), that projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them. We theoretically and empirically show that aggregating semantically similar states and actions in the intent space yields a bounded-variance estimator and improves policy performance stably. Our code is available online.

Figures

Figures reproduced from arXiv: 2607.16257 by Jiansheng Wei, Jiaqing Liang, Jinyi Han, Sihang Jiang, Tingyun Li, Xiaojun Meng, Xinyi Wang, Yanghua Xiao, Yizhou Ying, Zishang Jiang.

Figure 1
Figure 1. Figure 1: Existing RL methods struggle in long-horizon inter￾actions. GRPO becomes unstable with longer-horizon interac￾tions, while PPO remains stable but converges slowly due to critic warmup. Despite this progress, existing RL algorithms still face chal￾lenges in training agents for longer-horizon interactions. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: , as the interaction horizon increases, the mismatch (Vπ(st) − V¯ π(s0))2 can grow large, directly weakening the variance-reduction effect of the baseline by Theorem 3.3 and, in extreme cases, leading to higher gradient variance than using no baseline at all [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of HPO. HPO performs semantic projection and compares the policy distribution with the hindsight distribution in the intent space. By solving the Wasserstein distance between the two distributions, it obtains step-level advantages, which are combined with episode-level advantages to update the policy. the action space of LLMs is exponentially large, the space of intents they can express is sig… view at source ↗
Figure 4
Figure 4. Figure 4: The reward dynamics of HPO and other baseline methods with Qwen2.5-7B over SearchQA, where curves and uncertainty ranges are computed using the mean and standard deviation over three random seeds, respectively. GRPO exhibits unstable train￾ing, PPO converges more slowly, while HPO achieves fast and stable convergence. bias introduced by AS is limited and practically acceptable. Overall, these results highl… view at source ↗
Figure 5
Figure 5. Figure 5: Training reward dynamics of HPO using only step￾level advantages with Qwen2.5-3B. The dark blue curve is EMA￾smoothed from the raw data. Without episode-level rewards, training remains stable and effective, indicating that step-level advantages provide a reliable learning signal. Qwen2.5-7B-Instruct with Qwen3-0.6B-Embedding as the encoder, and record a per-iteration time breakdown. Rollout Old Prob Ref Pr… view at source ↗
Figure 6
Figure 6. Figure 6: Per-iteration training time breakdown of HPO with Qwen2.5-7B. Blue bars denote components shared with GRPO, while orange bars denote HPO-specific additions. The added overhead in HPO is negligible (< 0.8%). As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training reward dynamics of Qwen2.5-3B-Instruct with different ω settings. (As = +0.424) User: < information > DOC 1 ... The Life and Times of Hank Greenberg is a 1998 documentary film written, directed, and produced by Aviva Kempner about Hank Greenberg, first baseman of the Detroit Tigers, who was inducted into the Baseball Hall of Fame. ... DOC 2 ... DOC 3 ... < /information > Assistant: < think > From … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 17 linked inside Pith

  1. [3]

    K., Xu, H., and Shen, W

    Hu, J., Liu, J. K., Xu, H., and Shen, W. Reinforce++: Stabilizing critic-free policy optimization with global ad- vantage normalization.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv

  2. [4]

    R., Ilhan, F., Tekin, S

    Hu, S., Huang, T., Liu, G., Kompella, R. R., Ilhan, F., Tekin, S. F., Xu, Y ., Yahn, Z., and Liu, L. A survey on large language model-based game agents.arXiv preprint arXiv:2404.02039,

    Pith/arXiv arXiv

  3. [5]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  4. [6]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Pith/arXiv arXiv

  5. [7]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551,

    Pith/arXiv arXiv

  6. [9]

    Gui- r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458,

    Luo, R., Wang, L., He, W., Chen, L., Li, J., and Xia, X. Gui- r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458,

    Pith/arXiv arXiv

  7. [11]

    From reasoning to code: Grpo opti- mization for underrepresented languages.arXiv preprint arXiv:2506.11027,

    10 From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training Pennino, F., Raimondi, B., Rondelli, M., Gurioli, A., and Gabbrielli, M. From reasoning to code: Grpo opti- mization for underrepresented languages.arXiv preprint arXiv:2506.11027,

    Pith/arXiv arXiv

  8. [12]

    Adapt: As-needed decompo- sition and planning with language models

    Prasad, A., Koller, A., Hartmann, M., Clark, P., Sabharwal, A., Bansal, M., and Khot, T. Adapt: As-needed decompo- sition and planning with language models. InFindings of the Association for Computational Linguistics: NAACL 2024, pp. 4226–4252,

    2024

  9. [13]

    A., and Lewis, M

    Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the composi- tionality gap in language models. InFindings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 5687–5711,

    2023

  10. [14]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O

    URL https://arxiv.org/abs/2412.15115. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  11. [15]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  12. [16]

    Zerosearch: In- centivize the search capability of llms without searching

    Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y ., Jiang, Y ., Xie, P., Zhang, Y ., Huang, F., and Zhou, J. Zerosearch: In- centivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588,

    Pith/arXiv arXiv

  13. [19]

    Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning

    Xue, Z., Zheng, L., Liu, Q., Li, Y ., Zheng, X., MA, Z., and An, B. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. InNeurIPS 2025 Fourth Workshop on Deep Learning for Code,

    2025

  14. [20]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y ., Cohen, W., Salakhut- dinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380,

    2018

  15. [21]

    The landscape of agentic reinforcement learning for llms: A survey.arXiv preprint arXiv:2509.02547,

    Zhang, G., Geng, H., Yu, X., Yin, Z., Zhang, Z., Tan, Z., Zhou, H., Li, Z., Xue, X., Li, Y ., et al. The landscape of agentic reinforcement learning for llms: A survey.arXiv preprint arXiv:2509.02547,

    Pith/arXiv arXiv

  16. [22]

    and Zhang, A

    Zhang, Z. and Zhang, A. You only look at screens: Mul- timodal chain-of-action agents. In Ku, L.-W., Martins, A., and Srikumar, V . (eds.),Findings of the Association for Computational Linguistics: ACL 2024, pp. 3132– 3149, Bangkok, Thailand, August

    2024

  17. [23]

    doi: 10.18653/v1/2024

    Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.186. URL https://aclanthology. org/2024.findings-acl.186/. 11 From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training Appendix A. Experimental Details A.1. Task Description SearchQA.The deep search senario features a search engine–based environme...

    doi:10.18653/v1/2024 2024

  18. [24]

    making-of

    Therefore, Var( ˆGb |s) =E ∥g(a)∥2 |s E X 2 |s . Finally, if the baseline is chosen as the conditional mean b(s) =E[ ˆQ(s, a)|s], then E[X|s] = 0 and thus E[X 2 |s] = Var(X|s) =Var( ˆQ(s, a)−b(s)|s), yielding Var( ˆGb |s) =E ∥g(a)∥2 |s Var ˆQ(s, a)−b(s)|s , D.2. Proof of Theorem 3.3 Theorem D.2(Optimal Baseline and Excess Variance).Among all scalar baseli...

    1998

  19. [26]

    making-of

    Comparing the production years of the two documentaries, The Life and Times of Hank Greenberg (1998) and Burden of Dreams (1982), it is clear that Burden of Dreams was produced earlier. Therefore, the answer is 1982.< /think> <answer>1982< /answer> (As = +0.377) User:Congratulations! You have answered the question correctly!!! 17 From Outcomes to Actions:...

    1998

  20. [28]

    making-of

    Comparing the production years of the two documentaries, The Life and Times of Hank Greenberg (1998) and Burden of Dreams (1982), it is clear that Burden of Dreams was produced earlier. Therefore, the answer is 1982.< /think> <answer>1982< /answer> (As = +1.29) User:Congratulations! You have answered the question correctly!!! 19 From Outcomes to Actions: ...

    1998

  21. [30]

    making-of

    Comparing the production years of the two documentaries, The Life and Times of Hank Greenberg (1998) and Burden of Dreams (1982), it is clear that Burden of Dreams was produced earlier. Therefore, the answer is 1982.< /think> <answer>1982< /answer> (As = +0.707) User:Congratulations! You have answered the question correctly!!! 20 From Outcomes to Actions:...

    1998

  22. [31]

    Therefore, the answer is

    Comparing the production years of the two documentaries, The Life and Times of Hank Greenberg (1998) and Burden of Dreams (1982), it is clear that Burden of Dreams was produced earlier. Therefore, the answer is

    1998

  23. [32]

    Therefore, the answer is 1982.< /think> <answer>1982< /answer> (As = +0.377) User:Congratulations! You have answered the question correctly!!! 21

    Comparing the production years of the two documentaries, The Life and Times of Hank Greenberg (1998) and Burden of Dreams (1982), it is clear that Burden of Dreams was produced earlier. Therefore, the answer is 1982.< /think> <answer>1982< /answer> (As = +0.377) User:Congratulations! You have answered the question correctly!!! 21

    1998

  24. [2008]

    N., Liu, L., et al

    Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M. N., Liu, L., et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073,

    Pith/arXiv arXiv

  25. [2017]

    A., Kang, F., Wang, J

    Just, H. A., Kang, F., Wang, J. T., Zeng, Y ., Ko, M., Jin, M., and Jia, R. Lava: Data valuation without pre-specified learning algorithms.arXiv preprint arXiv:2305.00054,

    Pith/arXiv arXiv

  26. [2018]

    Agentgym- rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755,

    Xi, Z., Huang, J., Liao, C., Huang, B., Guo, H., Liu, J., Zheng, R., Ye, J., Zhang, J., Chen, W., et al. Agentgym- rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755,

    Pith/arXiv arXiv

  27. [2023]

    When not to trust language models: Investi- gating effectiveness and limitations of parametric and non- parametric memories.arXiv preprint arXiv:2212.10511, 7,

    Mallen, A., Asai, A., Zhong, V ., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investi- gating effectiveness and limitations of parametric and non- parametric memories.arXiv preprint arXiv:2212.10511, 7,

    Pith/arXiv arXiv

  28. [2024]

    D., Sugawara, S., and Aizawa, A

    Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing a multi-hop qa dataset for compre- hensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060,

    Pith/arXiv arXiv 2011

  29. [2025]

    Fireact: Toward language agent fine-tuning

    Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915,

    Pith/arXiv arXiv