{"id":"13303a68-78e4-4154-b134-384eca774b39","arxiv_id":"2510.26270","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GEPO adds graph-centrality-based intrinsic rewards, dynamic discounts, and two-level advantage shaping to group-based RL, improving LLM agent success on ALFWorld, WebShop, and a private Workbench benchmark.","lead":"This paper trains AI agents for long, multi-step tasks by building a running map of the states they visit and using that map to give extra credit to actions that pass through important \"bottleneck\" states. It reports larger success-rate gains than several baselines on three interactive benchmarks, but the abstract and the main text describe different experiments and different gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical graph's betweenness centrality is never validated against ground-truth task-criticality; because C_v drives all three GEPO signals (Eqs. 4, 6, 8, 12), a sampling-artifact mismatch would misdirect the entire method.","rationale":"I agree with the reader's weakest_assumption; it is the hinge of the central claim. The novelty of GEPO is not group-based RL but the graph-derived structural signals, and all three are multiplied by C_v. If the empirical betweenness centrality is an artifact of the current policy's sampling distribution, the intrinsic reward, dynamic discount, and topology-aware advantage are not 'task-aware' at all; they are a biased self-reinforcing loop. This makes the concern load-bearing even though the paper is internally coherent.\n\nI considered two other candidate concerns. First, the abstract/full-text discrepancy in headline numbers is serious but is a reporting/consistency issue; it does not identify a mechanism-level failure. Second, the lack of significance tests matters because several headline gaps (ALFWorld 7B: 94.9±3.8 vs 90.8±1.3; WebShop 7B: 80.5±6.7 vs 75.2±3.8) are within ~1 standard error of the difference. That weakens the strength of the empirical claim but is secondary to whether the graph signal itself is meaningful. The Workbench +10.9% gain is large and the ablation pattern is consistent, but Workbench is proprietary and the graph-validation issue still applies.\n\nThe concrete test I propose is feasible precisely because ALFWorld is a simulator with a known state space; one can enumerate a ground-truth graph without invoking any oracle beyond the environment definition. The same style of test could be applied to WebShop by hand-labeling a set of critical pages/tasks, but ALFWorld is the cleanest. The paper's own Table 4 shows that performance initially rises with graph size (n from 2 to 16) then falls, which is consistent with my concern: larger n produces more complete coverage, but eventually noisy edges dilute centrality. This pattern is observable but not diagnostic by itself.\n\nIf the empirical-vs-true centrality correlation check fails, the appropriate remedy is not a rewrite of the math but a re-examination of the graph-construction step (δ, embedding choice, or using a more exploration-balanced rollout policy). Until that check is run, the correct verdict is CONDITIONAL: the mechanism is plausible, the code/data are not released, and the central assumption is unverified. I therefore recommend keeping the reader's CONDITIONAL verdict.","tokens_in":20431,"tokens_out":4644,"duration_ms":48250,"concrete_test":"In ALFWorld, extract the simulator's ground-truth state-transition graph (rooms, receptacles, valid actions) and compute exact betweenness for each state. During a full GEPO training run, record the empirical C_v values at the end of each iteration and compute the Spearman rank correlation between empirical and ground-truth centrality. If this correlation is poor (e.g., ρ < 0.6) or fails to increase as the rollout count n grows from 8 to 20, then the graph-derived signals are tracking policy artifacts, not task-critical bottlenecks. To make this a crisp pass/fail, also run GEPO once with empirical C_v replaced by ground-truth betweenness; if success rate does not match or exceed the empirical version, the online graph-construction pipeline is not capturing the structure the paper claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 constructs the state-transition graph online from the current policy's sampled trajectories, merging states by Sentence-BERT cosine similarity with threshold δ=0.9, and then computes betweenness centrality on this empirical graph. Betweenness is defined over shortest paths in the observed graph; in a partial, policy-dependent graph, shortest paths and 'bottlenecks' are as much artifacts of sampling coverage as of true environment structure. A state that is rarely visited because the policy has not yet discovered it—e.g., the cabinet containing the target in ALFWorld—will have low empirical centrality until late in training, while a frequently revisited but task-irrelevant state (a countertop) can acquire high centrality simply by appearing in many sampled paths. Because Eqs. 4, 6, 8, and 12 all scale rewards, discounts, or advantages by C_v, any systematic mismatch between empirical centrality and true task criticality does not merely add noise; it actively reinforces coverage artifacts. The paper offers only a qualitative ALFWorld case (Appendix B) and a comparison of centrality metrics (Table 3), neither of which validates the mapping against ground truth. The threshold δ is also unvalidated; no sensitivity analysis is reported. This is not an internal inconsistency—the method is coherent—but the central mechanism is asserted rather than demonstrated. If the mapping fails, the claimed synergy among intrinsic reward, dynamic discount, and state-aware advantage collapses into three ways of amplifying the same sampling bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Graph-Enhanced Policy Optimization (GEPO), an extension of group-based policy optimization (GRPO/GiGPO) for multi-step LLM agents. GEPO builds an online state-transition graph from sampled trajectories by embedding textual observations with Sentence-BERT and merging states via a cosine threshold δ. Node/edge betweenness centralities are then injected into three learning signals: a structured intrinsic reward (Eq. 4), a dynamic discount factor (Eq. 6), and a trajectory/state-level advantage (Eqs. 8–13). Experiments on ALFWorld, WebShop, and a proprietary Workbench benchmark with Qwen2.5-1.5B/7B and Qwen3 models report consistent success-rate gains, with full-text abstract claims of +4.1% (ALFWorld 7B), +5.3% (WebShop 7B), and +10.9% (Workbench 7B) over GiGPO baselines. Ablations show that each component contributes and that pairwise removals produce drops larger than the sum of individual removals, which the paper interprets as synergy.","tokens_in":20847,"tokens_out":7441,"duration_ms":70358,"significance":"If the centrality signals genuinely identify task-critical bottleneck states, GEPO is a practical and lightweight contribution: it avoids GNNs and pretrained value functions, and the three-signal integration is a plausible way to densify sparse reward in long-horizon LLM-agent tasks. The paper is transparent about computational overhead, includes ablations, and compares centrality choices. However, the current evidence is not sufficient to support the claims as stated. The empirical support rests on three seeds with no significance testing; a proprietary benchmark is used without enough detail for reproduction; and the abstract is inconsistent with the method and with the full-text results. The most load-bearing assumption — that betweenness centrality on the agent's own sampled trajectory graph corresponds to task-criticality — is asserted rather than validated. The idea is promising, but the manuscript needs substantial strengthening before it can be accepted.","major_comments":[{"comment":"The arXiv abstract reports improvements of +1.1%, +3.2%, and +3.8% and describes a 'Task-Conditioned Criticality score' that combines topological betweenness with semantic similarity to the task prompt. The full text reports +4.1%, +5.3%, +10.9% on ALFWorld, WebShop, and Workbench, and Section 3 defines no such score — only node and edge betweenness C_v, C_e. The 'semantic similarity to the task prompt' component never appears in Eqs. 4–13. The two abstracts and the methodology must be reconciled; if the task-conditioned score is part of the method, it must be defined and evaluated, otherwise it should be removed from the abstract.","section":"Abstract vs. §3.2–3.5"},{"comment":"The central claim that empirical betweenness centrality reflects task-critical bottlenecks is not validated. The graph is constructed solely from trajectories sampled by the current policy, with states merged by a cosine threshold δ=0.9. Betweenness on this partial, policy-dependent graph conflates genuine environmental bottlenecks with sampling-coverage artifacts: a rarely visited but task-relevant state will have low centrality, while a frequently revisited but irrelevant state can acquire high centrality. Since C_v scales the intrinsic reward, dynamic discount, and both advantage terms, a systematic mismatch would reinforce coverage artifacts rather than task structure. Provide a ground-truth validation in a known environment (e.g., ALFWorld's underlying room/object graph) and a sensitivity analysis over δ (and, if possible, over the graph-update frequency).","section":"§3.2, Eqs. 4, 6, 8, 12"},{"comment":"Several headline gains are within one standard deviation of the baseline, with only three seeds and no significance test. For example, 7B WebShop success is 80.5±6.7 for GEPO versus 75.2±3.8 for GiGPO w/o std, and 7B ALFWorld is 94.9±3.8 versus 90.8±1.3. With n=3, the statement that GEPO 'consistently improves' is not statistically supported. Report individual seed values, more seeds, or a paired significance test across tasks/seeds. The current Table 1 should be treated as exploratory, not as evidence of consistent gains.","section":"§4.3, Table 1"},{"comment":"The 'super-additive synergy' claim is not supported by the reported statistics. For Workbench 1.5B, the sum of the individual drops (−2.3 and −1.4) is −3.7, while the pairwise removal is −7.2; but this difference-of-differences is computed from means of three seeds with no interaction test and no error propagation. Several single-component ablations overlap with the full model's uncertainty (e.g., 7B WebShop: 77.9±5.2 versus 80.5±6.7). Either provide a formal interaction analysis with appropriate uncertainty estimates, or replace the synergy claim with a more modest statement that the components appear to reinforce each other.","section":"§4.4, Table 2"},{"comment":"Workbench is a proprietary benchmark described only informally as a 'simulated business dashboard', and it is the source of the largest reported gain (+10.9% at 7B). This is a serious reproducibility problem: readers cannot assess the task distribution, the evaluation protocol, or whether the gain is due to task-specific properties. In addition, Appendix E reports Qwen3 results without any baseline comparison, so the claim that GEPO generalizes across model families is not demonstrated. Please provide full task definitions and evaluation details for Workbench, and include GiGPO (or GRPO) baselines for the Qwen3 experiments.","section":"§4.1.3, Workbench; Appendix E"}],"minor_comments":[{"comment":"Eq. (6) uses ΔC_v(t) without normalizing C_v, so the scale of centrality changes depends on graph size and betweenness magnitudes; specify whether C_v is normalized before the tanh. Also, §3.2 says centralities are recomputed when the graph 'doubles' or every K iterations, but Algorithm 1 recomputes them every iteration. Clarify which is actually used.","section":"Eq. (6) vs. §3.2, Algorithm 1"},{"comment":"The '1+C_v(s_t)' scaling assumes C_v is bounded or normalized; with unnormalized betweenness, large centrality values can dominate the local advantage and distort credit assignment. Specify the normalization procedure.","section":"Eq. (12)"},{"comment":"The description of clustering states as those that 'share the identical state observation' is ambiguous. If exact string matching is used, it is inconsistent with the semantic vertex merging in §3.2; if graph-vertex identity is used, say so explicitly.","section":"§3.5"},{"comment":"The centrality comparison is limited to ALFWorld and does not consistently separate the metrics: the 7B betweenness result (94.9±3.8) overlaps with eigenvector (92.3±0.9) and degree (91.5±1.8). Support the claim that betweenness is 'most robust' with more seeds or additional environments.","section":"§4.5, Table 3"},{"comment":"The figure legend says 'GEPO (200 steps)' and 'GiGPO (150 steps)', but the text compares per-step wall-clock time; explain why the training lengths differ and whether the comparison is apples-to-apples.","section":"Figure 3"},{"comment":"Several references are incomplete or duplicated: [1] has no title/authors; [10]/[11], [26]/[27], and [45]/[46] are duplicated entries. Clean up the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is coherent and the reported gains are consistently positive, but the evidence is weaker than the text claims. I am particularly concerned about (i) the mismatch between the arXiv abstract and the full-text abstract/method, (ii) the absence of any validation that empirical betweenness centrality corresponds to task-criticality, and (iii) the reliance on a proprietary benchmark for the largest result. If the authors cannot provide Workbench details or a ground-truth centrality check, I would lean toward rejection despite the promising results. The three-seed statistics should be treated as a minimum bar, not an acceptable standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean integration of graph-based credit assignment into group-based RL for LLM agents, and it deserves a serious look. The method is coherent: build an online state-transition graph from sampled trajectories, compute betweenness centrality, then feed that into an intrinsic reward, a dynamic discount, and a state-aware advantage term inside a GRPO-style loop. The combination appears to be genuinely new relative to GRPO, GiGPO, and earlier graph-RL work for LLMs. The ablations are thorough, the convergence curves are helpful, and the authors are upfront about the 20–30% computational overhead. Credit where it's due: the paper is clearly written and the mechanism is well-specified in equations.\n\nThat said, the empirical support is weaker than the claims. The abstract reports gains of +1.1%, +3.2%, and +3.8% on ALFWorld, WebShop, and search-augmented QA tasks, while the full text claims +4.1%, +5.3%, and +10.9% on ALFWorld, WebShop, and a proprietary Workbench benchmark. The abstract also introduces a \"Task-Conditioned Criticality\" score that never appears in the method section. This discrepancy is not a minor cosmetic issue; a reader cannot tell which numbers are the real ones. On top of that, only three seeds are reported, no significance tests are given, and several headline comparisons fall within one standard deviation—for example, WebShop 7B: GEPO 80.5±6.7 vs. GiGPO w/o std 75.2±3.8. The largest claimed gain (+10.9%) comes from a benchmark that is not public.\n\nThe load-bearing assumption—that betweenness centrality computed on the agent's own trajectory graph corresponds to task-critical bottleneck states—is plausible but never validated. The paper offers only a qualitative case study and a comparison of alternative centrality metrics. The stress-test concern about circularity is real: the graph is built from the current policy's rollouts, so high-centrality states may simply reflect where the policy has already spent samples, not where the task requires going. This is not a fatal internal inconsistency, but it is a gap between assertion and evidence. The δ=0.9 semantic merging threshold is also unanalyzed.\n\nWho is this for? Researchers working on credit assignment and exploration for LLM agents will find the framework worth understanding. It deserves a serious referee, but I would not cite it as evidence in its current form. The authors need to release code and data, reconcile the abstract with the full text, and provide significance tests or at least more seeds. Without that, the headline numbers are not trustworthy.","headline":"GEPO integrates graph centrality into LLM agent RL plausibly, but the evidence is thin and the abstract and full text disagree.","tokens_in":21325,"tokens_out":1962,"would_cite":false,"duration_ms":21905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that injecting graph centrality into group-based RL — through intrinsic rewards, dynamic discounting, and topology-aware advantage — consistently improves LLM agent success on sparse-reward tasks.","keywords":["LLM agents","reinforcement learning","credit assignment","state-transition graph","betweenness centrality","sparse rewards","exploration","graph-enhanced policy optimization"],"falsifier":"Annotate the true bottleneck states in a text environment (for example, the hallway and doorway states that must be crossed to complete household tasks) and compare them with the highest-centrality nodes in GEPO's graph. If the overlap is no better than chance, or if replacing GEPO's graph-derived intrinsic reward with any dense reward on the same graph reproduces the gains, the central claim is false.","tokens_in":20339,"feed_emoji":"🧭","tokens_out":6778,"duration_ms":60293,"temperature":0.7,"pith_summary":"Group-based reinforcement learning for LLM agents compares whole trajectories and gives every step in a trajectory the same credit, ignoring the fact that some states act as bottlenecks. GEPO builds a state-transition graph from the agent's own sampled trajectories, attaches a betweenness-based criticality score to each state, and converts it into three learning signals: an intrinsic reward for reaching pivotal states, a discount factor that grows when the agent enters a more central state, and an advantage function that weights decisions at critical states more heavily. The paper claims these signals work together: removing any one hurts success rate by one to three percentage points, and removing pairs hurts roughly twice as much as the sum of individual drops. On three text-based agent benchmarks, GEPO reports consistent absolute success-rate gains over strong group-based baselines, from around +1 to +11 percentage points depending on benchmark and model scale, with lower across-seed variance. A sympathetic reader would care because the method offers a parameter-light way to inject structural priors into sparse-reward LLM agent training without handcrafted dense rewards.","feed_headline":"Graph-aware credit assignment lifts LLM agent success by up to 10.9%","feed_subtitle":"A dynamic state graph marks bottleneck states as pivotal, guiding exploration and credit in sparse-reward agent training.","key_machinery":"The load-bearing object is a dynamically constructed state-transition graph. Textual observations are embedded and merged into nodes when cosine similarity exceeds a threshold, and transitions between them become edges; node and edge betweenness centrality (how often each lies on shortest paths) is then treated as strategic importance. This graph produces a Task-Conditioned Criticality score, and through it three learning signals: an intrinsic reward that adds centrality to the sparse extrinsic reward, a dynamic discount factor that clips a baseline discount scaled by the tanh of the centrality change, and a graph-enhanced return used for both trajectory-level and state-level advantage. The","core_discovery":"The central claim is that structural blindness—treating the environment's state space as an undifferentiated set—is a major cause of poor exploration, imprecise credit, and myopic planning in group-based RL for LLM agents, and that it can be fixed by an online state-transition graph. The paper's framework constructs the graph from the agent's own experience, merging text observations by embedding similarity, computes betweenness centrality as a proxy for whether a state lies on many shortest paths, and defines a Task-Conditioned Criticality score that combines this centrality with semantic similarity to the task prompt. That score feeds three mechanisms: a dense intrinsic reward, a dynamic d","pith_inferences":["If the centrality-to-task-bottleneck mapping is correct, the same three signals could be grafted onto other policy-gradient objectives beyond group-based RL, turning any trajectory sampler into a structure-aware learner; this is an extension the paper does not itself claim.","A cheaper way to test the mechanism would be to compute centrality only on successful trajectories; if the gains persist, the method is mostly rewarding reachable critical states rather than discovering them through exploration, which would narrow the paper's explanation.","The semantic-similarity component of the criticality score suggests the graph could be pruned to task-relevant subgraphs, which might reduce the noise observed when the graph grows too large—an inference, not a paper claim.","Because the graph is built only from sampled experience, an agent that explores poorly early on may never visit true bottlenecks; combining GEPO's rewards with an exploration bonus for novel high-degree states could make the structural prior more robust—again, an editorial inference."],"forward_implications":["On all three benchmarks and both model scales, GEPO's absolute success rate exceeds the strongest group-based baseline it was compared with, with the largest gain on the strictly ordered procedural-workflow task.","Removing any one of the three signals—intrinsic reward, topology-aware advantage aggregation, or dynamic discount—costs 1–3 points; removing pairs costs more than the sum of individual removals, indicating the signals reinforce each other.","Betweenness centrality outperforms degree, closeness, and eigenvector centrality as the graph signal, suggesting bottleneck identification, not hub popularity, drives the gain.","Across-seed variance is lower for GEPO than for flat group-based methods, implying the graph signal stabilizes training rather than just improving the best run.","Increasing the number of rollouts per iteration expands the graph and improves performance up to 16 rollouts, after which gains reverse—so graph scale should be tuned rather than maximized."],"fun_headline_variants":["Graph-enhanced policy optimization lifts LLM agent success","State-transition graph sharpens credit assignment for LLM agents","Task-conditioned criticality boosts multi-step LLM agent training","Graph-aware credit assignment yields up to 3.8% gains for LLM agents","Structural graph improves credit assignment in LLM agent training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that betweenness centrality computed on a graph built from the agent's own trajectory samples marks the states that are genuinely task-critical, so rewarding and discounting toward those states improves learning rather than merely chasing sampling artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Graph-enhanced policy optimization lifts LLM agent success","State-transition graph sharpens credit assignment for LLM agents","Task-conditioned criticality boosts multi-step LLM agent training","Graph-aware credit assignment yields up to 3.8% gains for LLM agents","Structural graph improves credit assignment in LLM agent training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1592,"prompt_tokens":788,"completion_tokens":804,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":718}},"tokens_in":532,"tokens_out":804,"duration_ms":7982,"temperature":1.0,"reasoning_tokens":718,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:13:40.578597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate the true bottleneck states in a text environment (for example, the hallway and doorway states that must be crossed to complete household tasks) and compare them with the highest-centrality nodes in GEPO's graph. If the overlap is no better than chance, or if replacing GEPO's graph-derived intrinsic reward with any dense reward on the same graph reproduces the gains, the central claim is false.","supporting_citations":[],"review_version":1}