{"id":"ad464b14-743c-4cbc-ac64-71adea4662b8","arxiv_id":"2608.08255","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"EFCA combines immediate environment feedback and recent state-history signals into a return reweighting that improves step-level credit assignment for LLM agents in ALFWorld and WebShop.","lead":"An RL method called EFCA reweights step-level rewards using short-term environmental feedback and recent state history, instead of relying only on final outcomes. It reports higher ALFWorld scores and WebShop task quality for Qwen2.5 agents, though the comparison baseline table contains inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Borrowed 7B HGPO baseline is identical to the 1.5B row, so EFCA's headline +4.04 ALFWorld / +3.50 WebShop gains over HGPO at 7B rest on an internally inconsistent comparison.","rationale":"The paper's central claim is empirical: EFCA consistently improves both task success and task quality over strong baselines. That claim requires fair baseline numbers. The 7B HGPO row in Table 1 being numerically identical to the 1.5B row is a concrete internal inconsistency, not a disagreement with consensus. Section 4.1.2 says baselines were directly reused from prior papers, so this row is a red flag that the borrowing process was not checked. If the 7B HGPO numbers are wrong, the +4.04 ALFWorld and +3.50 WebShop gains highlighted in Section 4.2.1 are unverified, directly weakening the 'consistently improves' claim. I considered whether the absent validation split and hyperparameter tuning on the test benchmarks is more load-bearing, but that practice is common and harder to settle; the duplicated row is specific and checkable. I also credit the method's simplicity, the honest limitations section that admits pattern-matching limits and possible wrong penalties, and the ablations showing both signals matter. The appropriate outcome remains conditional acceptance: require corrected or re-run baselines and release of the exact pattern sets and code before the central claim is accepted.","tokens_in":17670,"tokens_out":4419,"duration_ms":41440,"concrete_test":"Pull the original GraphGPO and HGPO papers and verify whether the Qwen2.5-7B HGPO numbers in Table 1 match those sources and whether the 1.5B and 7B rows are genuinely identical there. If the 7B row is copied or absent, re-run HGPO-7B with the paper's stated protocol (N=8, three seeds, same evaluation) and recompute the EFCA-vs-HGPO deltas at 7B. If the re-run values differ by more than the reported standard deviations, declare the 7B HGPO comparison unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 lists Qwen2.5-7B-Instruct HGPO with exactly the same values as the 1.5B row for every ALFWorld and WebShop metric (97.53/81.20/99.60/100.00/85.63/82.57/91.99 and 85.56/71.54). The caption states that HGPO results are reused from original papers, so either the 7B run was never executed and the 1.5B row was copied, or a transcription error occurred. Section 4.2.1 specifically highlights EFCA's +4.04 ALFWorld and +3.50 WebShop Task Score gains over HGPO at 7B, so the central empirical claim depends on this baseline being correct. Section 4.1.2 says 'we directly reuse part of the reported results', which means the paper does not establish that EFCA was trained with the same hyperparameters, rollout group size, seeds, and evaluation protocol as the borrowed baselines; the identical rows are direct evidence that the borrowing was not verified. If the 7B HGPO numbers are wrong, the claimed improvement over HGPO at the larger scale is unsupported, and the 'consistently improves' claim is weakened. Even setting HGPO aside, EFCA still edges GraphGPO, so the method may be salvageable, but the paper as written does not cleanly support its headline quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Environmental Feedback-based Credit Assignment (EFCA), a plug-in return-reweighting mechanism for critic-free stepwise policy optimization. EFCA defines a short-term feedback credit c_t^fb by matching normalized environment feedback against hand-curated positive and negative pattern sets P and N, and a medium-term state-history credit c_t^hist based on a K-step window of those feedback credits (Eqs. (3)-(6)). The two credits are combined as C_t = alpha c_t^fb + beta c_t^hist and converted into a step weight w_t = clip(1 + lambda C_t, w_min, w_max), which multiplies the original discounted step return (Eqs. (7)-(8)). The method is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B and 7B backbones, with the headline claim that EFCA consistently improves task success and task quality over GiGPO, GraphGPO, and HGPO baselines (Section 4.2.1).","tokens_in":18000,"tokens_out":9044,"duration_ms":73034,"significance":"If fully supported, EFCA would be a valuable lightweight alternative to learned critics and reward models for long-horizon agentic RL: it is simple, transparent, environment-grounded, and adds negligible wall-clock overhead, as the reported runtime comparison suggests. The paper includes clear ablations separating the two process signals and provides pseudo-code that makes the algorithm easy to reimplement. However, the central empirical support is weakened by three problems: a duplicated Qwen2.5-7B HGPO baseline row in Table 1; hyperparameter values selected on the same ALFWorld benchmark used for the headline results; and reused baseline numbers whose training configurations are not shown to match those of EFCA. Because the reported gains over HGPO at 7B rest on the duplicated row, the quantitative claims are only partially supported as written.","major_comments":[{"comment":"The Qwen2.5-7B-Instruct HGPO row is numerically identical to the Qwen2.5-1.5B-Instruct HGPO row for every reported metric, including the standard deviations (97.53/81.20/99.60/100.00/85.63/82.57/91.99 and 85.56/71.54). Since Section 4.2.1 uses this row to claim +4.04 ALFWorld and +3.50 WebShop Task Score improvements over HGPO at the 7B scale, those specific comparisons are unsupported. The row must be replaced by an actual 7B HGPO run or the corresponding claims must be removed. The caption's marker scheme (dagger vs. double dagger) also needs clarification for this row, because it is not clear which numbers were taken from the GraphGPO paper and which from the HGPO paper.","section":"Table 1 / Section 4.2.1"},{"comment":"The main ALFWorld results appear to use hyperparameters selected on the same benchmark that is reported as the headline result. The ablation sweeps (lambda, beta, alpha in Figure 2; K, eta, w_min, w_max in Table 2) are conducted on ALFWorld, and the best-performing values are then used for the main comparison. No held-out validation split or multiple-testing correction is described. This post-hoc selection inflates the reported improvements; the authors should either fix the hyperparameters before evaluation, use a validation split, or report results across the full sweep and show that the conclusion is robust.","section":"Section 4.3.2 / Figure 2 / Table 2"},{"comment":"The paper states that 'all RL-based methods share the same hyperparameter configuration' while also saying that part of the baseline results are directly reused from the GraphGPO and HGPO papers. These statements are incompatible: borrowed results were produced under the original papers' training budgets, rollout group sizes, memory settings, seeds, and evaluation protocols. Without evidence that those settings match EFCA's setup, the reported gaps between EFCA and borrowed baselines may be due to training-configuration differences rather than to credit assignment. The authors should either rerun all baselines in the same pipeline or list the exact configuration used for each borrowed result.","section":"Section 4.1.2 / Section 4.1.3"},{"comment":"The claim that EFCA 'consistently improves both task success and task quality over strong baselines' is stronger than Table 1 actually supports. On WebShop, EFCA's task success rate is lower than GraphGPO at both scales (75.91 vs. 78.65 at 1.5B; 78.91 vs. 80.31 at 7B), and on several ALFWorld subcategories EFCA underperforms GraphGPO (e.g., Look 81.32 vs. 100.0 at 1.5B; Pick 97.93 vs. 100.0 at 7B). The wording should be qualified to comparisons on overall ALFWorld score and WebShop Task Score, or the authors should show a statistical test supporting the stronger claim.","section":"Abstract / Section 4.2.1"}],"minor_comments":[{"comment":"The increased standard deviation of the reweighted returns is a mechanical consequence of multiplying all step returns by a non-constant weight, so by itself it does not demonstrate that credit assignment is more discriminative. The interpretive claim should be softened or supported by an analysis stratified by successful and failed trajectories.","section":"Section 4.3.1 / Figure 4"},{"comment":"The hyperparameter table gives 'typical / tested values' but does not state the exact values used in the main experiments. Please report the final K, eta, w_min, w_max, alpha, beta, and lambda for each benchmark and backbone, and also make the P and N pattern sets available for reproducibility.","section":"Appendix B.3 / Table 2"},{"comment":"The footnote markers are difficult to follow: the 1.5B HGPO row is marked with a dagger, which the caption attributes to the GraphGPO paper, while the 7B HGPO row is marked with a double dagger, attributed to the HGPO paper. The caption should clarify which original source supplied each row or block of entries.","section":"Table 1 caption"},{"comment":"In the ablation text, 'removing the medium-term state-history signal' and 'disabling the short-term feedback signal' correspond to beta = 0 and alpha = 0 in Figure 2, respectively; this correspondence should be stated explicitly in the text.","section":"Section 4.3.2"}],"recommendation":"major_revision","confidential_remarks":"The duplicated 7B HGPO row looks like a transcription error rather than a deliberate misrepresentation, but it is load-bearing for the paper's headline comparison and must be fixed before the manuscript can be considered further. The post-hoc hyperparameter selection on ALFWorld is the second issue I would insist on addressing. I would not reject the paper outright: the method is simple, well specified, and the ALFWorld gains over GraphGPO at both scales do not depend on the broken HGPO row, so the central idea may well survive a corrected comparison. However, the current version cannot be accepted as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and worth taking seriously: for critic-free stepwise RL, reweight step returns with two environment-grounded signals—short-term feedback pattern matching and a medium-term stagnation penalty. That specific combination is not in the GiGPO, HGPO, or GraphGPO baselines cited here, and the method is described cleanly. The ablations show both signals contribute, and the cost analysis suggests the overhead is negligible. Credit where it is due: the paper is easy to read, the plug-in design is sensible, and the limitations section is honest about pattern-based matching and the exploration-step problem.\n\nThe soft spots are real and load-bearing. Table 1 lists the Qwen2.5-7B HGPO row with values identical to the 1.5B row for every metric. The paper says it reused results from the original papers, but this copy-paste was not verified. That means the advertised +4.04 ALFWorld and +3.50 WebShop gains over HGPO at 7B rest on a baseline that cannot be trusted. The hyperparameters (alpha, beta, lambda, K, eta) were swept on the same ALFWorld and WebShop benchmarks used for the headline numbers, so part of the improvement likely comes from post-hoc selection rather than the method itself. The abstract's \"consistently improves over strong baselines\" overreaches given these problems.\n\nThat said, the method may still work. Even without HGPO, EFCA edges GraphGPO on several metrics, and the ablation direction is consistent. The paper just does not support its cleanest claims as written. A referee could fix this with a corrected baseline table, a proper validation split for hyperparameters, and a more measured abstract.\n\nWho is this for? Researchers working on credit assignment for LLM agents, especially critic-free stepwise RL, will find the idea useful even if this version is not fully convincing. I would send it to peer review rather than desk reject, because the core contribution is novel enough and the empirical story, once cleaned up, could be solid. But I would ask for major revision before acceptance.\n\nMy honest recommendation: engage with it, but treat the 7B comparison as unverified and the headline gains as provisional until the authors fix the table and release code and pattern sets.","headline":"A plausible plug-in credit assignment idea whose headline numbers are undermined by a duplicated 7B baseline row and benchmark-tuned hyperparameters.","tokens_in":18583,"tokens_out":1380,"would_cite":false,"duration_ms":15067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EFCA reweights per-step returns with environment feedback, improving ALFWorld and WebShop scores.","keywords":["credit assignment","agentic reinforcement learning","environment feedback","stepwise policy optimization","long-horizon tasks","return reweighting","LLM agents","ALFWorld"],"falsifier":"Re-run the HGPO baseline on the Qwen2.5-7B model using the same rollout group size, memory window, and random seeds as EFCA; if the reproduced baseline differs from the 1.5B numbers currently copied into the 7B row of Table 1, or if EFCA no longer beats it by the reported margins, then the claimed consistent improvement over strong baselines is not established.","tokens_in":17440,"feed_emoji":"🤖","tokens_out":13144,"duration_ms":111848,"temperature":0.7,"pith_summary":"Long-horizon agentic reinforcement learning suffers from sparse, delayed rewards, so step-level credit must be reassigned from final outcomes. The paper argues that environments already emit process feedback during interaction—what the action did, whether it changed state, whether it repeated a failure—and that this feedback is an underexploited source of credit. It proposes EFCA (Environmental Feedback-based Credit Assignment), which adds a short-term feedback credit and a medium-term state-history credit to the base step return, reweighting returns so locally effective actions get stronger supervision and ineffective loops get weaker supervision. On ALFWorld and WebShop, across 1.5B and 7B backbones, the paper reports best overall ALFWorld scores (95.31 and 96.03) and best WebShop Task Scores (89.81 and 89.06) among the compared stepwise RL methods. If these results hold, critic-free agentic RL can be improved simply by using the feedback the environment already provides.","feed_headline":"Environment feedback as step credit lifts agentic RL scores","feed_subtitle":"Short-term feedback and stagnation penalties reweight returns, beating strong baselines on ALFWorld and WebShop.","key_machinery":"The load-bearing object is the environment-grounded multi-timescale credit score $C_t = \\alpha c^{\\mathrm{fb}}_t + \\beta c^{\\mathrm{hist}}_t$, converted into a step-dependent return weight. The short-term feedback credit $c^{\\mathrm{fb}}_t$ is computed by normalizing the environment's textual feedback and testing it against a hand-constructed positive pattern set $P$ (for example, 'you pick up') and negative pattern set $N$ (for example, 'nothing happens'), returning $+1$, $-1$, or $0$. The medium-term state-history credit $c^{\\mathrm{hist}}_t$ looks back over a window of $K$ steps: a window of all negative credits returns $-1$, a full window with no positive credit returns a milder $-\\eta$, and otherwise it returns $0$. These credits are combined and mapped through a clipped affine weight $w_t = \\mathrm{clip}(1 + \\lambda C_t, w_{\\min}, w_{\\max})$ that multiplies the base discounted return. This lets EFCA add process-level supervision to existing stepwise optimizers without any value network or learned reward model.","core_discovery":"EFCA's central claim is that the base step-level return $R^{\\mathrm{base}}_t = \\sum_{t'=t}^T \\gamma^{t'-t} r_{t'}$, used by critic-free stepwise optimizers, is an incomplete credit signal because it only reflects long-term outcome. The paper's mechanism reweights this return with environment-grounded process evidence: normalized textual feedback is matched against pattern sets to give $c^{\\mathrm{fb}}_t \\in \\{-1,0,+1\\}$, and a $K$-step window of these values produces $c^{\\mathrm{hist}}_t$, which penalizes consecutive failures or a window with no positive progress. The combined score $C_t = \\alpha c^{\\mathrm{fb}}_t + \\beta c^{\\mathrm{hist}}_t$ is turned into a clipped weight $w_t = \\mathrm{clip}(1 + \\lambda C_t, w_{\\min}, w_{\\max})$, applied as $\\tilde{R}_t = w_t R^{\\mathrm{base}}_t$. On ALFWorld and WebShop with 1.5B and 7B backbones, the paper reports best overall ALFWorld scores (95.31 and 96.03) and best WebShop Task Scores (89.81 and 89.06) among the compared stepwise RL methods, with ablations showing the two signals are complementary.","pith_inferences":["A natural extension would replace hand-built pattern sets with semantic feedback understanding, so the credit signal transfers across environments with different feedback vocabularies; the paper lists this as future work rather than testing it.","Because EFCA only modifies return computation, the same reweighting scheme could probably be added to other critic-free policy optimization pipelines beyond the stepwise methods tested.","The state-history penalty could in principle suppress actions that are locally unproductive but globally necessary, such as exploration or enabling actions; testing this trade-off would require a causal or hierarchical credit model the paper does not provide.","The reported 6.36% wall-clock saving is suggestive but not a controlled causal comparison; a dedicated head-to-head with identical rollout scheduling would determine whether feedback matching itself is faster."],"forward_implications":["Stepwise RL optimizers can be improved by reweighting their step returns with environment feedback, without changing their grouping or optimization logic.","Both task success and task quality can improve together, as seen in the WebShop Task Score, which measures purchase quality rather than just completion.","Penalizing repeated ineffective patterns through the medium-term history signal contributes a separate, measurable gain; ablations show about a four-point drop when either signal is removed.","The reported gains hold at two model scales (1.5B and 7B), so the mechanism is not tied to one backbone capacity.","The added computation is small: pattern matching and reweighting add negligible cost, with lower reported wall-clock time per training step than the HGPO baseline."],"supporting_citations":[{"why":"Defines HGPO, the stepwise optimizer whose returns EFCA reweights and whose baseline results are reused for comparison.","marker":"He et al. (2026)"},{"why":"Defines GiGPO, a stepwise grouping optimizer and comparison baseline that EFCA's reweighting plugs into.","marker":"Feng et al. (2025)"},{"why":"Defines GraphGPO, the strongest stepwise baseline whose ALFWorld and WebShop numbers EFCA is compared against.","marker":"Cheng et al. (2026)"},{"why":"Supplies ALFWorld, the embodied environment whose textual feedback patterns and task suite are the first benchmark.","marker":"Shridhar et al. (2020)"},{"why":"Supplies WebShop, the web interaction environment and task-score metric used in the second benchmark.","marker":"Yao et al. (2022a)"},{"why":"Defines the ReAct protocol the agent follows and provides a prompting baseline.","marker":"Yao et al. (2022b)"},{"why":"Defines GRPO, the trajectory-level group baseline that motivates the stepwise credit assignment setting.","marker":"Shao et al. (2024)"}],"fun_headline_variants":["Feedback and history reweight rewards for agentic RL","Two environment signals fix credit assignment in agentic RL","Multi-timescale feedback improves long-horizon agentic RL","EFCA: Environment feedback as step credit for RL","Reward reweighting from feedback beats baselines in agentic RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes the baseline numbers were produced under the same model, rollout settings, and evaluation protocol as EFCA, and Table 1's identical HGPO rows across the two model sizes suggest that assumption was not verified.","fun_headline_variants_meta":{"raw":{"variants":["Feedback and history reweight rewards for agentic RL","Two environment signals fix credit assignment in agentic RL","Multi-timescale feedback improves long-horizon agentic RL","EFCA: Environment feedback as step credit for RL","Reward reweighting from feedback beats baselines in agentic RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3362,"prompt_tokens":1034,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":2246}},"tokens_in":650,"tokens_out":2328,"duration_ms":17984,"temperature":1.0,"reasoning_tokens":2246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:14:40.836744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HGPO baseline on the Qwen2.5-7B model using the same rollout group size, memory window, and random seeds as EFCA; if the reproduced baseline differs from the 1.5B numbers currently copied into the 7B row of Table 1, or if EFCA no longer beats it by the reported margins, then the claimed consistent improvement over strong baselines is not established.","supporting_citations":[],"review_version":1}