{"id":"93718103-121c-443a-b232-55aa7fe6c381","arxiv_id":"2506.00539","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Clustering language-agent actions into shared intentions and averaging their rewards reduces reward variance and improves policy performance in open-ended dialogue tasks.","lead":"This paper introduces ARIA, a training method that clusters the natural-language actions of LLM agents into semantic 'intentions' and shares rewards among similar actions before running reinforcement learning. The reported effect is lower reward variance and roughly 10% better average task performance across negotiation, bargaining, and question-asking games.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma G.2 is false as stated: its proof compares second moments rather than variances, so the central variance-reduction theorem is unsupported; aggregation can increase gradient variance even when rewards are replaced by their exact cluster means.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that overall disposition. My concern is more specific than the reader's stated weakest assumption. The reader identified the unvalidated epsilon-bisimulation assumption as the main risk; the paper's own Appendix A flags the same dependence on embedding quality. However, the more decisive problem is internal to the proof of Lemma G.2: even granting perfect clustering in the sense of A~=E[A|C], the variance-reduction claim can fail because the gradient factor X=grad_log_pi is correlated with A inside clusters. The proof's central identity conflates a second-moment comparison with a variance comparison and silently requires conditional uncorrelatedness of A and X^2 given C. The explicit counterexample shows that Var(XA~) can exceed Var(XA). This means the paper's theoretical core, which is used to explain faster convergence, larger step sizes, and sample efficiency, is not merely unproven but false as stated. The experimental evidence does not repair the gap: Section 6 measures reward variance and smoothed loss curves, not the variance of the policy-gradient estimator. That said, ARIA may still be a useful empirical method; corrected experiments or a reworked estimator could preserve the practical contribution. The right verdict remains CONDITIONAL, pending a corrected theoretical statement or a rewording that does not claim a proven variance-reduction guarantee. I mark agreement as partial because the reader's weakest_assumption was about bisimulation, whereas my load-bearing concern is the algebra of Lemma G.2, although the reader's rationale did separately note that Lemma 4.2 was unjustified.","tokens_in":20416,"tokens_out":13556,"duration_ms":129006,"concrete_test":"Run a two-part check. (1) Reproduce the counterexample above, or run a random search over small discrete distributions with A~=E[A|C], and compute Var(XA) versus Var(XA~); if the inequality fails, Lemma G.2 is false as stated. (2) On the actual (h,a) samples from one training iteration of Section 5, compute the empirical variances of grad_log_pi(a|h)*A(h,a) and grad_log_pi(a|h)*A~(h,a) using the paper's own clustering and embeddings. If the real-data variance is not reduced, the abstract's 'significantly reduces policy gradient variance' claim must be withdrawn, or the estimator must be redefined as E[grad_log_pi*A|C] and the method re-specified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ARIA's central claim is that intention-space aggregation lowers the variance of the policy-gradient estimator, and this mechanism explains the reported gains. Lemma G.2 is the load-bearing step. The proof writes Var(XA)=E[X^2 A^2]-(E[XA])^2 and then shows E[X^2 A^2]-E[X^2 A~^2]=E[E[X^2(A-A~)^2|C]]>=0. The displayed identity is false in general: it holds only if E[X^2 A|C]=A~E[X^2|C], i.e. conditional uncorrelatedness of A and X^2 given the cluster; nothing in the setup guarantees this. Even when A~ is the exact conditional mean E[A|C], the variance can increase. Minimal counterexample: two equally likely clusters, C1: (X,A)=(2,0),(-1,1); C2: (-2,0),(1,1). Then A~=E[A|C]=0.5, E[XA]=0, Var(XA)=0.5, but E[X A~]=0 and Var(X A~)=0.625. Thus Lemma 4.2, and with it the variance-improved convergence Theorem 4.1, is not established. The paper's own Appendix A concedes that embedding quality matters, but that is a separate, unvalidated assumption; the algebraic flaw is internal and independent of whether embeddings satisfy epsilon-bisimulation. The empirical plots in Section 6 measure reward variance, not the variance of XA, so they do not substitute for the missing proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARIA, a method for training language agents in open-ended language action tasks by embedding actions and observations, clustering them into a low-dimensional intention space, aggregating rewards within each intention cluster, and optimizing the policy with offline REINFORCE. The authors report consistent and substantial empirical gains over offline and online baselines across four tasks (Twenty Questions, Guess My City, Bargaining, Negotiation), including iterative improvements, ablations, generalization to other base models, and statistical significance tests. The paper also claims a theoretical guarantee that intention-space reward aggregation reduces the variance of the policy-gradient estimator and introduces only O(epsilon) bias, based on Lemmas 4.1 and 4.2 and Theorem 4.2.","tokens_in":20747,"tokens_out":3928,"duration_ms":37619,"significance":"If the variance-reduction claim were rigorously established, ARIA would be a valuable contribution to RL for language agents, since it addresses a concrete practical problem (sparse, high-variance rewards in open-ended language action spaces) and is supported by a broad empirical evaluation. The empirical work is a clear strength: the comparisons cover multiple task types, multiple opponent models, several baselines, ablations, and generalization experiments, and the reported gains are large and mostly statistically significant. However, the central theoretical mechanism -- the claimed reduction in policy-gradient variance -- rests on Lemma G.2, whose proof is algebraically invalid, and the bias bound in Theorem 4.2 is conditional on an epsilon-bisimulation assumption that is never measured or validated. The empirical plots in Section 6.1 measure reward variance, not the variance of the score-weighted advantage, so they do not directly support the theoretical claim. Overall, the empirical contribution is credible and useful, but the paper's central conceptual framing currently overstates what is proved.","major_comments":[{"comment":"The proof of Lemma G.2 is mathematically invalid. Writing X = ∇logπ, the proof asserts the identity E[X^2 A^2] - E[X^2 A~^2] = E[E[X^2(A-A~)^2|C]] >= 0, but this identity holds only if E[X^2 A|C] = A~ E[X^2|C], i.e., if A and X^2 are conditionally uncorrelated given the cluster. Nothing in the setup guarantees this. In fact, even when A~ is the exact conditional mean E[A|C], replacing A by A~ can increase the variance of XA. A concrete counterexample is: two equally likely clusters, C1 with (X,A) = (2,0), (-1,1), and C2 with (X,A) = (-2,0), (1,1); then A~ = 0.5 in both clusters, E[XA] = 0, Var(XA) = 0.5, but E[XA~] = 0 and Var(XA~) = 0.625. Consequently, the variance-reduction claim in Lemma 4.2, and with it the convergence statement in Theorem 4.1, is not established.","section":"Appendix G, Lemma G.2 (and Lemma 4.2)"},{"comment":"The bounded-bias result is logically conditional on the assumption that all actions within each intention cluster are epsilon-bisimilar, but the paper never measures whether the embedding-space clusters actually satisfy this condition. The authors' own Appendix A concedes that clustering quality depends on the sentence embeddings and that misaligned clusters can impair learning. As a result, Theorem 4.2 provides a conditional statement, not evidence that the specific clustering used in the experiments has small bias. The paper should either validate the epsilon-bisimulation property empirically or substantially soften the claim that ARIA maintains 'a small bound of bias.'","section":"Section 4.2, Definition 1, Theorem 4.2"},{"comment":"The empirical variance analysis reports the variance of the raw and aggregated reward signals, but the theoretical claim in Lemma 4.2 concerns the variance of the gradient estimator (∇logπ) * A. Reducing Var(A) does not imply reducing Var((∇logπ) * A), as the counterexample in the first major comment shows. Thus the plots in Figure 4 cannot substitute for the missing proof, and the statement in the Abstract and Section 1 that ARIA 'significantly reduces policy gradient variance' is only supported at the level of the reward signal, not the gradient estimator.","section":"Section 6.1, Figure 4"},{"comment":"The theoretical analysis treats A~ as the exact conditional expectation E[A|C], but the algorithm computes an empirical cluster average over a finite dataset of collected trajectories. The gap between the population object and the finite-sample estimator is not analyzed; in particular, clusters with few samples may produce noisy aggregated rewards, and the variance-reduction argument does not account for this estimation error. The paper should either prove a finite-sample version of Lemma 4.2 or explicitly describe the theory as applying to the idealized population version.","section":"Section 3.3 and Section 4.1"}],"minor_comments":[{"comment":"The table header uses γ=0.01 for what the text describes as the SplitScore threshold ϵ; this notational mismatch should be corrected throughout the appendix.","section":"Appendix J, Table 5"},{"comment":"The online baseline is referred to as 'RAGEN(GRPO)' in one sentence but as 'StarPO' in the tables and elsewhere; please unify the naming.","section":"Appendix I.1"},{"comment":"There are several typos and formatting errors, including 'langugae' in Section 5.1, 'accepdt' in Appendix M, and inconsistent use of 'Single-Agent Scenatio' in Appendix M headings; a careful proofreading pass is recommended.","section":"Throughout"},{"comment":"The claim that SplitScore is 'bounded above a monotonically decreasing function' relies on n_{k,max} being monotonically decreasing, but this is asserted rather than proved for average-linkage hierarchical clustering; the statement should either be proved or made conditional.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of this paper is solid and likely publishable after revision: the experimental design is reasonable, the gains are consistent, and the ablations are informative. The main blocker is the theoretical section: Lemma G.2 is false as stated, and the variance-reduction mechanism that motivates the method is not established. I believe this is fixable within the scope of the manuscript by either proving a corrected variance bound under explicit assumptions or by reframing the contribution as an empirical reward-smoothing technique without the theoretical guarantee. I would not reject the paper, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the ARIA paper has a genuinely simple and usable idea—cluster semantic embeddings of language actions and observations, average rewards within clusters, then run offline REINFORCE. The empirical section is the strong part: consistent double-digit gains over offline and online baselines across four tasks, including adversarial games with different opponent models, plus a generalization check to Qwen models. The presentation is clear and the ablation on the split threshold is honest about sensitivity. That is real value for the language-agent RL subfield.\n\nThe soft spot is the theory. The paper claims that intention-space aggregation reduces the variance of the policy gradient estimator. Lemma 4.2 is the load-bearing step, and the proof in Appendix G is wrong. It compares second moments of the gradient score with A and with \\tilde A, but variance also includes the squared mean difference. The displayed identity only holds under conditional uncorrelatedness of A and (∇log π)^2 within clusters, which is not assumed. A minimal counterexample exists with two clusters where variance increases even when \\tilde A is the exact conditional mean. So Lemma 4.2, and with it Theorem 4.1's variance-improved convergence, is not established.\n\nThe bias bound in Theorem 4.2 is conditional on epsilon-bisimulation, which is never measured or validated. That is a separate weakness; the paper's own limitations appendix acknowledges embedding quality matters, but the algebraic flaw is internal and independent of that.\n\nThe empirical plots in Section 6 show reward variance dropping after aggregation, not the variance of the policy gradient estimate, so they do not substitute for the missing proof. There is also no released code or data, and the SplitScore threshold is tuned on the same tasks used for evaluation—minor but worth fixing.\n\nNet: the method is plausible and the empirical story is strong enough to deserve a serious referee. The theory section needs to be rewritten or dropped; as it stands, the central claim that aggregation reduces policy gradient variance is unsupported by the proof. If the theory is fixed or removed and artifacts are released, I expect this to be a useful contribution. This paper is for people working on RL for language agents, especially those fighting sparse rewards in open-ended action spaces.","headline":"Empirically promising reward-aggregation method whose central variance-reduction theorem rests on a false proof step; the experiments are worth a serious look, the theory is not.","tokens_in":21265,"tokens_out":2776,"would_cite":true,"duration_ms":24079,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ARIA claims that replacing token-level rewards with intention-cluster-averaged rewards reduces policy-gradient variance and yields average 9.95% performance gains across four open-ended language-agent tasks.","keywords":["language agents","reinforcement learning","reward aggregation","intention space","semantic projection","variance reduction","hierarchical clustering","policy gradient"],"falsifier":"Measure, on real trajectories from the four tasks, the within-cluster dispersion E[Var(A|C)] and the total-variation distance between next-state distributions for actions assigned to the same intention cluster; if the clusters contain actions with materially different rewards or transition effects, the bias bound fails and the reported gains could come from reward smoothing rather than from unbiased variance reduction.","tokens_in":20228,"feed_emoji":"🎯","tokens_out":4910,"duration_ms":45669,"temperature":0.7,"pith_summary":"The paper tries to establish that the reward sparsity and variance that cripple reinforcement learning for open-ended language agents can be tamed by aggregating rewards in an intention space rather than in the raw token space. It argues that semantically similar utterances rarely repeat in the exponentially large action space, so each sampled action receives a near-binary reward; grouping utterances by clustered embeddings and assigning the cluster-average reward densifies the signal. If true, language agents could be trained with far fewer samples and larger stable update steps. The paper reports an average 9.95% improvement over offline and online RL baselines across negotiation, bargaining, and two question-asking games, along with reduced reward variance and smoother convergence.","feed_headline":"Same-intention actions share rewards; agents win ~10% more","feed_subtitle":"ARIA clusters semantically similar moves and uses cluster-averaged rewards to train stable language-agent policies.","key_machinery":"The central mechanism is semantic projection plus reward aggregation: actions and observations are embedded and hierarchically clustered into an intention space C_k, each history-action pair is replaced by its cluster labels, and the return for each intention pair becomes the average of all trajectory rewards that map to it. This aggregated advantage, A~ = E[A|C], is the engine of the argument: the law of total variance shows Var(A~) = Var(A) - E[Var(A|C)], and an epsilon-bisimulation condition (Definition 1) bounds the bias introduced by averaging. SplitScore selects the number of clusters k by stopping when reward changes from further splits fall below a threshold.","core_discovery":"ARIA's central discovery is that replacing each action's sparse, high-variance reward with the average reward of all actions sharing its intention cluster removes the intra-cluster variance E[Var(A|C)] from the policy gradient estimate, at a bias cost controlled by how epsilon-bisimilar actions within a cluster are. The authors construct the intention space by embedding actions and observations and running hierarchical agglomerative clustering; a SplitScore stopping rule selects the cluster count by measuring whether further splits change the aggregated rewards. Training is offline REINFORCE on the aggregated advantages. The paper reports consistent wins across four tasks and across Llama and Qwen base models, with reward variance dropping and iterative retraining yielding further gains.","pith_inferences":["The variance decomposition is algorithm-agnostic, so cluster-averaged advantages could likely be dropped into PPO, GRPO, or actor-critic pipelines, even though the paper only demonstrates it inside REINFORCE and its online reward-model variant.","A cheap diagnostic for practitioners is to compute E[Var(A|C)] on a held-out batch before training; if it is large, the bias bound in Theorem 4.2 is unlikely to hold and aggregation may hurt rather than help.","The assumption that intentions are discrete and well-separated suggests ARIA will help most where utterances naturally fall into a few strategic categories, such as negotiation, tutoring, or customer service, and least where every utterance is genuinely novel.","The online variant's periodic reward-model updates hint that cluster granularity should also be re-selected as the policy changes; a testable extension is to make k adaptive per iteration rather than fixed."],"forward_implications":["Replacing raw advantages with cluster-averaged advantages reduces the variance of the policy-gradient estimate by removing the intra-cluster component E[Var(A|C)].","Offline REINFORCE with aggregated rewards outperforms BC, trajectory-wise DPO, step-wise DPO, SPAG, ArCHer, and StarPO by an average of 9.95% across the four evaluated tasks.","Iterating ARIA yields additional gains: +3.27% after a second round and +1.85% after a third round of data collection and training.","An online version that learns a reward model from aggregated rewards reaches higher rewards faster than ArCHer and StarPO on single-agent games.","The gains transfer to Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct, not just the Llama-3-8B-Instruct policy model."],"supporting_citations":[{"why":"Supplies the REINFORCE policy-gradient algorithm that ARIA optimizes with aggregated rewards.","marker":"[23]"},{"why":"Supplies hierarchical agglomerative clustering, the method used to construct the intention space.","marker":"[24]"},{"why":"Supplies the Twenty Questions and Guess My City single-agent environments and their evaluation setup.","marker":"[8]"},{"why":"Supplies the Bargaining and Negotiation adversarial environments and the win-rate evaluation protocol.","marker":"[43]"},{"why":"Supplies the text-embedding model used for semantic projection of actions and observations.","marker":"[45]"},{"why":"Supplies ArCHer, a hierarchical online RL baseline that ARIA must beat on single-agent tasks.","marker":"[20]"},{"why":"Supplies PPO, the underlying optimizer for the SPAG baseline and a standard online RL comparison.","marker":"[22]"},{"why":"Supplies the trajectory-wise DPO offline baseline that ARIA compares against.","marker":"[12]"}],"fun_headline_variants":["Cluster actions by intent, share rewards, win ~10% more","ARIA: Reward aggregation by intention cuts variance, boosts wins","Intention-driven reward sharing lifts agent performance ~10%","Share rewards within intention clusters to train better agents","ARIA reduces reward variance, gains 9.95% across tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bias bound assumes that actions clustered together really are epsilon-bisimilar, meaning they have nearly identical rewards and next-state effects, but the paper never measures whether embedding-space clustering actually produces such clusters.","fun_headline_variants_meta":{"raw":{"variants":["Cluster actions by intent, share rewards, win ~10% more","ARIA: Reward aggregation by intention cuts variance, boosts wins","Intention-driven reward sharing lifts agent performance ~10%","Share rewards within intention clusters to train better agents","ARIA reduces reward variance, gains 9.95% across tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1163,"prompt_tokens":872,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":488,"tokens_out":291,"duration_ms":3421,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:02:54.200978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on real trajectories from the four tasks, the within-cluster dispersion E[Var(A|C)] and the total-variation distance between next-state distributions for actions assigned to the same intention cluster; if the clusters contain actions with materially different rewards or transition effects, the bias bound fails and the reported gains could come from reward smoothing rather than from unbiased variance reduction.","supporting_citations":[{"cited_title":"Glee: A unified framework and benchmark for language-based economic environments, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Bargaining and Negotiation adversarial environments and the win-rate evaluation protocol."}],"review_version":1}