{"id":"54c6cdf0-6f8c-409b-b4ea-87809ae938e4","arxiv_id":"2607.22724","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ProGPO adds a first-visit observation-coverage advantage only when an entire rollout group fails, improving group-based RL for long-horizon LLM agents on ALFWorld and WebShop.","lead":"This paper proposes ProGPO, a training tweak for LLM agents that, when every sampled attempt at a task fails, gives extra credit to attempts that visit new observations instead of repeating old actions. It reports consistent gains on ALFWorld and WebShop over three group-based RL baselines, with the largest improvements on the hardest tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Improvement claim rests on single training seeds; reported error bars are inference-seed only, so 'consistently improves' is not yet established.","rationale":"I considered the coverage-closeness assumption identified by the reader. It is a genuine limitation, but the paper explicitly labels it an empirical property (Remark 4.2), supports it with negative controls (Shuffle/Random/Neg) and a worked example, and acknowledges environment dependence in the limitations section. That assumption is therefore a stated boundary condition, not an internal inconsistency. The more decisive gap for the central empirical claim is statistical: the final-checkpoint numbers in Table 1 come from one training seed each, with error bars reflecting only inference-seed variation. Since ProGPO's non-interference guarantee is per-group and says nothing about how fallback updates interact with future rollouts, the claimed consistent improvements need multi-seed training evidence. Appendix 10.4 reports only two 30-epoch training seeds, and Table 11's five-seed lambda sweep uses a different evaluation protocol. This does not overturn the method or the reader's CONDITIONAL verdict; it sharpens the condition: before 'consistently improves' is taken at face value, the authors should report multi-seed final-checkpoint training variance. My recommendation is therefore UNCHANGED relative to the reader, with a more precise reason for the conditionality.","tokens_in":30151,"tokens_out":5825,"duration_ms":66792,"concrete_test":"Run 5 independent training-seed runs (same hyperparameters, K=8, 150 epochs) for GiGPO vs. GiGPO+ProGPO on Qwen2.5-1.5B ALFWorld and WebShop, evaluating each final checkpoint with ≥3 inference seeds. If ProGPO's per-seed delta is positive in ≥4 of 5 seeds on ALFWorld and the mean delta exceeds the inference-seed standard deviation, the headline 'consistently improves' is supported. If deltas straddle zero, the claim must be downgraded to a single-seed demonstration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's caption says 'Most results are averaged over 3 random seeds,' but Appendix 8.8 limits this to re-evaluating one final checkpoint with three independent inference seeds, and Appendix 10.4 states controlled comparisons fix the training seed. The only multi-training-seed evidence is a 30-epoch snapshot (Table 10), not the final 150-epoch checkpoints. This matters because ProGPO's non-interference guarantee (Proposition 4.3) only says the fallback does not alter updates on reward-informative groups; every claimed gain must come from all-fail groups. The size of that gain is a training-dynamics effect, not an inference-noise effect, so it cannot be quantified by re-running evaluation three times. With a single training seed, aggregate improvements (e.g., 1.5B ALFWorld GiGPO 86.7→91.4; WebShop 65.0→72.4) could be partly seed luck; near-ceiling cases such as 7B HGPO ALFWorld 95.4→95.3 already show the effect can vanish. The λaux sweep in Table 11 uses five seeds but reports a different evaluation protocol and only for the default row, so it does not substitute for multi-seed final-checkpoint comparisons of base vs. ProGPO.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Progress-conditioned Group Policy Optimization (ProGPO), a fallback advantage estimator for group-based RL of LLM agents. In rollout groups where all trajectories fail, the reward-based advantages of GRPO/GiGPO/HGPO are identically zero; ProGPO replaces them with a normalized first-visit observation-coverage score, and otherwise leaves the base advantage untouched. The authors provide propositions showing that the progress score is a relaxation of success, that the method does not modify updates on reward-informative groups, and that the fallback branch is group-normalized REINFORCE on the progress score. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B-Instruct report consistent improvements over GRPO, GiGPO, and HGPO under matched compute budgets, with negative controls (Shuffle, Random, Neg, Obs-Change), an ablation of conditional gating, and a hyperparameter sweep.","tokens_in":30382,"tokens_out":4282,"duration_ms":51113,"significance":"If the empirical claims hold, this is a valuable contribution: a simple, model-free repair for a genuinely degenerate regime of group-based policy optimization. The paper's strengths are explicit and welcome: no auxiliary critic or reward model, an exact non-interference statement, a transparent per-group characterization of the fallback gradient, matched compute budgets between base and augmented methods, and a battery of negative controls that test the mechanism rather than only reporting aggregate gains. The theoretical core (Propositions 4.1, 4.3, 4.4 and the finite-group bias calculation) is sound and carefully scoped; the authors also honestly flag that within-all-fail-group coverage closeness to success is an empirical property (Remark 4.2) and that observation-level coverage may not transfer to latent-state environments (Section 12). The main weakness is that the central comparative claim — \"consistently improves\" — rests on a single training seed at the final checkpoints reported in Table 1; this needs to be fixed before the empirical conclusions are supportable.","major_comments":[{"comment":"The claim that ProGPO \"consistently improves\" group-based baselines is not yet established by the reported evidence. Table 1's caption says results are \"averaged over 3 random seeds,\" but Appendix 8.8 clarifies that this is re-evaluation of one final checkpoint under three inference seeds, and Appendix 10.4 states that controlled comparisons fix the training seed. The only multi-training-seed evidence is a 30-epoch snapshot (Table 10), not the 150-epoch final checkpoints. This matters because Proposition 4.3 guarantees the fallback only modifies all-fail groups, so every claimed gain is a training-dynamics effect whose uncertainty cannot be quantified by inference-seed variance. The near-ceiling 7B HGPO ALFWorld result (95.4→95.3) already shows the effect can vanish, and aggregate gains such as GiGPO 86.7→91.4 or WebShop 65.0→72.4 could be partly seed luck. Please report final-checkpoint","section":"§5.2, Table 1; Appendix 8.8; Appendix 10.4"},{"comment":"The hyperparameter-sensitivity table is internally inconsistent. The text states that the default λ_aux=0.3 achieves 92.2% overall success, with 0.1 at 94.5% and 0.5/0.7 lower. However, the row labeled \"1.0\" in Table 11 has exactly the same per-category values as the main Table 1 GiGPO+ProGPO row (All 91.4±1.6), while the row labeled \"0.3 (default)\" reports a different protocol (five training seeds) and a different aggregate (92.2±2.0). Either the row labels are swapped or the evaluation protocols differ in a way that invalidates direct comparison. In addition, the claim that λ_aux is \"fixed at 0.3\" is qualified in Appendix 8.3 by the deployed λ_eff = λ_aux q_fail, so the sweep in Table 11 is not a sweep of the actually deployed scale unless this is reconciled. Please correct the table and clarify which quantity was swept under which evaluation protocol.","section":"§5.3, Appendix 10.6, Table 11"},{"comment":"There is a mismatch between the analyzed estimator and the deployed implementation. Eq. (7) and Propositions 4.3–4.4 treat λ as a fixed positive constant, while Algorithm 2 and Eq. (30) set λ_eff = λ_aux q_fail(B), where q_fail is the fraction of all-fail groups in the current update. The authors acknowledge this in the audit note and correctly state that a common positive multiplier preserves the groupwise direction of Proposition 4.4, so the non-interference claim is unaffected. Nevertheless, the theoretical claim that the fallback \"reverts exactly\" to the base estimator and that ProGPO has \"fixed λ_aux=0.3 across all environments\" is presented in the main text without this qualification. Please state the deployed scaling law in Section 4.2 or explicitly defer it to the implementation appendix, and clarify whether the reported λ_aux sweep controls the effective scale or only its ceilin","section":"§4.2, Eq. (7); §8.3, Algorithm 2"}],"minor_comments":[{"comment":"The notation σ_R(G_x), σ_P(G_x), ̅R, and ̅P is used in Eq. (7) but the population-variance convention is defined only in Appendix 7.2, Eq. (14). Define it in the main text to make the estimator self-contained.","section":"Eq. (5)–(7)"},{"comment":"The text says \"More details shown in Appendixx 10.3\" — typo \"Appendixx.\" Also, the Edit-D variant is named but not described; please state how edit distance is computed and normalized.","section":"§5.3, Appendix 10.3"},{"comment":"The footnote for Table 11 says entries with a standard deviation are five-seed averages, but the main Table 1 uses three inference seeds from one checkpoint, and Table 8 similarly mixes five-seed and single-seed rows. Please standardize the seed reporting labels so a reader can tell which rows support variance claims.","section":"Table 8, Appendix 10.6"},{"comment":"Reference [31] and [32] are the same paper (SPA-RL); the duplication should be removed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the theoretical framing is honest, but the central comparative claim currently rests on single training seeds at the final checkpoint. If the authors can supply multi-seed final-checkpoint numbers and reconcile Table 11, I would expect the paper to become acceptable; without that, the \"consistently improves\" claim is not supported. I would also ask the editor to ensure the appendix's disclosed implementation details (λ_eff, seed protocol) are reflected in the main text before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kate —\n\nYou should know two things about arXiv:2607.22724: it has a genuinely simple and useful idea, and its headline claim is weaker than the prose suggests.\n\nWhat's new: when a rollout group in GRPO/GiGPO/HGPO is all-fail, the reward advantage is identically zero. ProGPO detects that degeneracy and substitutes a normalized first-visit observation-coverage score as the advantage. If reward variance is present, it reverts exactly to the base estimator (Prop 4.3). That non-interference property is the paper's best design decision: it bounds the downside to the all-fail regime, and the fallback disengages as the policy improves. The paper is honest that coverage is a relaxation, not a proxy for task value (Remark 4.2, Prop 4.1), and the empirical controls are the right ones—Shuffle and Random break the trajectory–score binding, Neg reverses it, Obs-Change shows that naive novelty-gating gets gamed by cycling. The per-category breakdowns and worked example groups make the mechanism easy to see.\n\nThe soft spot is the seed story. Table 1's error bars are three inference seeds on a single trained checkpoint. The 'consistently improves' claim across two benchmarks, two scales, and three bases is all single training seed. Appendix 10.4 adds a second ProGPO training seed at 30 epochs (32.0 and 43.0 vs baseline 15.6), which is encouraging but not the same as reporting final-checkpoint variance over three training seeds. The λaux sweep's default row does report five training seeds, but under a different evaluation protocol, and it's only the default configuration. I don't think the gains are fake—the improvements are large and directionally consistent—but 'consistently' is not established until we see multi-training-seed variance on the final checkpoints. The fix is cheap: retrain a few runs, or at minimum rephrase the claim as 'on the seeds we ran.'\n\nThe other caveat is the load-bearing assumption: within an all-fail group, higher coverage puts you closer to success. The paper treats this as empirical, and the two environments support it, but noisy or detour-heavy observations could game the coverage signal. That's a generalization concern, not a flaw in the current evidence.\n\nBottom line: this is a solid, well-executed empirical paper with a real idea. It deserves a serious referee. I'd ask for multi-seed training runs before acceptance; if that's infeasible, the authors should dial the claim back. I'd bring it to reading group—the conditional-fallback pattern is one people will reuse.","headline":"ProGPO is a simple, honest, and potentially useful fix for the all-fail regime in group-based RL; the only real weakness is that the headline 'consistently improves' rests on single training seeds.","tokens_in":30934,"tokens_out":3406,"would_cite":true,"duration_ms":35096,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProGPO: extracting learning signal from all-fail rollout groups using first-visit observation coverage.","keywords":["group-based policy optimization","credit assignment","sparse reward","LLM agents","first-visit observation coverage","all-fail groups","long-horizon tasks","reinforcement learning"],"falsifier":"Run ProGPO on a variant of ALFWorld where a 'wander' action changes the observation text (or any environment where observations become more diverse without approaching the goal) and compare success rate against the base estimator. If adding the fallback lowers success relative to the baseline, the coverage-progress link fails. A more direct measurement: on collected all-fail groups, compute the correlation between first-visit coverage and ground-truth distance to success; a non-positive correlation across groups would falsify the core assumption.","tokens_in":29954,"feed_emoji":"🤖","tokens_out":6103,"duration_ms":60729,"temperature":0.7,"pith_summary":"Group-based policy optimization trains LLM agents by comparing trajectories within a rollout group, but on hard long-horizon tasks many groups contain no successful trajectory. When every reward is zero, the comparison collapses and the policy receives no gradient, a self-reinforcing credit trap that lets repeated low-effect actions persist. ProGPO shows that failed trajectories are not equally uninformative: within an all-fail group, some trajectories visit many distinct observations while others cycle or stall. It uses first-visit observation coverage as a fallback advantage, assigned only when reward variance is zero, and provably reverts exactly to the base estimator when outcome variation exists. On ALFWorld and WebShop with 1.5B and 7B models, this consistently improves over three group-based baselines, with the largest gains on the hardest, all-fail-prone subtasks.","feed_headline":"Coverage signal rescues RL from all-fail rollout groups","feed_subtitle":"When entire rollout groups fail, counting new states per trajectory restores a learning signal; hardest tasks gain most.","key_machinery":"The first-visit coverage score P(τ) = (D(τ)-1)/T, where D(τ) is the number of distinct observations in the trajectory and T its length, counts each transition to a new observation exactly once and gives zero credit to revisits or no-op actions. This score powers a three-branch conditional advantage estimator: the standard reward-normalized advantage when reward variance exceeds a threshold; a fallback advantage λ(P_i - mean P)/σ_P when all rewards are zero but coverage varies; and zero otherwise. The gating guarantees non-interference (the gradient differs from the base only where the base gradient is identically zero), and the fallback branch is exactly group-normalized REINFORCE with P as","core_discovery":"The paper's central claim is that the all-fail credit trap—a rollout group in which every trajectory fails, so outcome rewards give zero relative advantage—can be broken without auxiliary models or reward shaping by looking at the behavioral contrast among the failed trajectories. ProGPO scores each trajectory by its first-visit coverage, P = (D-1)/T, the fraction of steps that transition to a previously unseen observation, and uses it only when the reward-based advantage is identically zero. The estimator is a strict extension of the base group-relative estimator: on every group with non-degenerate reward variance, ProGPO reduces exactly to the base advantage, so the difference in gradients","pith_inferences":["The coverage signal is environment-dependent: in settings where observation strings change without genuine progress (noise, adversarial detours, hidden state), the fallback could reinforce wandering. A stress test with observation noise injected into otherwise solvable tasks would delimit the method's domain.","The conditional-fallback recipe could generalize beyond coverage: any cheap proxy that is a necessary condition for success (e.g., subgoal flags, information gain) could be plugged into the same gate, provided it is only used when reward variance is zero.","The exponential disengagement bound predicts that the method's advantage over baselines should be largest early in training and on hard task distributions; a longitudinal study across task difficulty would make this testable.","Because coverage is computed from the observation string, the method is naturally applicable to other text-rich agent environments; whether it transfers to visual or latent-state environments is an open question the paper explicitly defers."],"forward_implications":["Rollout groups that previously contributed zero gradient can now contribute weak, behavior-aligned signal, reducing wasted compute on hard long-horizon tasks.","Because ProGPO is a strict extension of the base estimator, any group-based policy optimization method can adopt it without changing behavior on informative groups.","The success-set containment result means pushing probability toward high-coverage trajectories keeps the policy inside a region that contains all successful behavior, so the fallback cannot contradict the task objective in principle.","As the policy improves, the all-fail trigger probability decays at most exponentially in the group size times success probability, so the fallback automatically hands control back to reward when it becomes informative.","The measured gains concentrate on the hardest subtasks, suggesting a route to training agents on tasks where outcome-only RL currently stalls."],"fun_headline_variants":["First-visit coverage busts RL's all-fail credit trap","When all rollouts fail, count new states to learn","Coverage-only advantage rescues failed rollout groups","ProGPO: Break the all-fail trap with state novelty","No rewards? Novelty signal still guides agentic RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Within an all-fail group, trajectories that visit more distinct observations are closer to eventual success than those that visit fewer, so reinforcing high-coverage failures (and demoting low-coverage ones) improves the policy; this is an empirical property, not a theorem.","fun_headline_variants_meta":{"raw":{"variants":["First-visit coverage busts RL's all-fail credit trap","When all rollouts fail, count new states to learn","Coverage-only advantage rescues failed rollout groups","ProGPO: Break the all-fail trap with state novelty","No rewards? Novelty signal still guides agentic RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1223,"prompt_tokens":746,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":490,"tokens_out":477,"duration_ms":5119,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:50:39.042027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ProGPO on a variant of ALFWorld where a 'wander' action changes the observation text (or any environment where observations become more diverse without approaching the goal) and compare success rate against the base estimator. If adding the fallback lowers success relative to the baseline, the coverage-progress link fails. A more direct measurement: on collected all-fail groups, compute the correlation between first-visit coverage and ground-truth distance to success; a non-positive correlation across groups would falsify the core assumption.","supporting_citations":[],"review_version":1}