{"id":"eda4e08d-a4ca-4d25-a427-d68420b5fcac","arxiv_id":"2506.11912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Subtracting the state value from action values scales gradient updates to downweight frequent state-action pairs, helping agents learn causal state representations and generalize out-of-trajectory.","lead":"This paper argues that the advantage function, a standard trick for reducing noise in reinforcement learning, also helps agents ignore misleading patterns and generalize to new situations. The author identifies an algebraic scaling property and shows in three simple gridworlds that advantage-trained agents outperform Q-value-trained agents when forced off their usual paths.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiments do not instantiate the Φ-asymmetric advantage of Theorem 1, so the central causal claim rests on an untested mechanism.","rationale":"The reader's weakest_assumption is exactly the point I would stress: Theorem 1's identity is algebraically correct, but it depends on Q being full-state while V is a coarse average over the equivalence class induced by Φ. In the experiments, both Q and V are neural networks on the same observations, so the learned value function is not V(Φ(s)) and Q is subject to the same representation. The Limitations section explicitly acknowledges this. The normalization and batch-size results are suggestive, but they vary implementation details of PPO's advantage estimator rather than instantiating the coarse baseline required by the theorem. I do not see a fatal internal contradiction, and the paper is honest about its limited scope and lack of causal guarantees, so a conditional verdict remains appropriate. The proposed check would empirically settle whether the published advantage results are driven by the claimed scaling effect or by other properties of the value baseline. Since this is the same concern the reader identified, my read does not change the verdict.","tokens_in":13911,"tokens_out":8845,"duration_ms":228920,"concrete_test":"Implement Key2Door with the exact Theorem 1 object: keep Q̂_full as a tabular full-state estimate (location × key × action, from value iteration or Monte Carlo), use the coarse baseline V̂(Φ(s)) = Σ_{s'∈{s}_Φ} d̂π(s') V(s') / Σ_{s'∈{s}_Φ} d̂π(s') with Φ(s)=location only, and plug this  into the same PPO loss with the paper's hyperparameters. Compare OOT evaluation return and the KL(L=6, key vs no-key) heatmap against the paper's shared-feature advantage curves. If this faithful implementation loses the OOT advantage over Q training, the published experiments were not testing Theorem 1; if it reproduces the published advantage curves, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 is proved for A_Φ(s,a) = Q(s,a) − V(Φ(s)), where Q is defined on the full state and V averages over the Φ-equivalence class. The paper's own Limitations section (Sec. 7) concedes that 'if Q were instead learned entirely via function approximation, then it too would be subject to Φ, and Theorem 1 would no longer hold in its current form.' In the reported PPO and REINFORCE experiments, both the policy and value functions are feedforward networks on the same observation or observation stack; no explicit coarse Φ is constructed, and Q is never a full-state estimate independent of the learned representation. Standard GAE advantages use a learned V(s), not V(Φ(s)). Thus the condition under which the factor 1 − P(s,a|Φ(s)) appears in the advantage is not satisfied in the experiments. The OOT generalization results and KL heatmaps are consistent with the paper's story, but they cannot distinguish the proposed scaling mechanism from ordinary consequences of using a value baseline (variance reduction, clipping, feature sharing). Since the strongest claim is that this scaling breaks policy confounding, the mismatch between theory and implementation is the load-bearing weak point; the algebraic identity alone does not establish that advantage training learns causal representations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the advantage function, defined as A^π_Φ(s,a) := Q^π(s,a) − V^π(Φ(s)) where V^π(Φ(s)) averages the value over the Φ-equivalence class of s, downweights state-action pairs that are likely under the current policy by a factor 1 − P^π(s,a|Φ(s)), and that this scaling mitigates policy confounding and improves out-of-trajectory generalization. The main theoretical result (Theorem 1) gives an exact algebraic identity for A^π_Φ. The paper then presents experiments in three gridworlds comparing training with advantages versus Q-values, ablations of advantage normalization and batch size, and an analysis of learned policies via KL-divergence heatmaps.","tokens_in":14110,"tokens_out":8652,"duration_ms":104138,"significance":"The algebraic identity in Theorem 1 is correct and is proved cleanly in Appendix A, and it is a straightforward consequence of Definition 6 (the definition of V(Φ) as a conditional expectation). The paper is honest about its scope and explicitly states in Section 7 that the result does not generally extend when Q is a learned function approximator. The experiments are described with enough detail to be reproducible, and the observation that advantage normalization removes the out-of-trajectory benefit is an interesting empirical finding. However, the theoretical mechanism relies on an asymmetry between Q (full-state) and V (coarse representation) that is not instantiated in the reported experiments, and the causal claim is only supported heuristically. If the authors can resolve this gap—either by instantiating the coarse Φ in experiments or by extending the theory—the paper could make a useful contribution to understanding representation learning in policy gradient methods.","major_comments":[{"comment":"The experiments do not implement the actor-critic asymmetry required by Theorem 1. The theorem defines A_Φ(s,a)=Q(s,a)−V(Φ(s)) with Q evaluated on the full state and V averaging over the Φ-equivalence class; the paper's own Limitations section (Section 7) concedes that if Q is learned entirely via function approximation then 'Theorem 1 would no longer hold in its current form.' In the reported PPO and REINFORCE experiments, both the policy and the value network take the same observation stack as input, no explicit coarse representation Φ is constructed, and the Q-values used for the Q-value condition are Monte Carlo returns, not a separate full-state Q-function. Consequently, the empirical difference between the advantage and Q-value conditions cannot be attributed to the 1 − P scaling of Theorem 1 rather than to generic variance reduction or clipping. The paper should either modify the experiments to explicitly use a coarse Φ (e.g., define Φ by a hand-selected subset of observation variables and train V(Φ(s)) as the average over the class), or extend the theoretical analysis to function approximation.","section":"Section 6.1 and Section 7"},{"comment":"The 'scaling effect' is a direct algebraic consequence of Definition 6: V^π(Φ(s)) is defined as the conditional expectation of Q over the equivalence class, so A = Q − E[Q] = (1−p)(Q − E[Q | excluded]) is tautological given the definition. The paper should acknowledge more explicitly that this is not an intrinsic property of the standard advantage function A(s,a)=Q(s,a)−V(s), but of the specific coarse-grained baseline V(Φ(s)). Furthermore, the causal claim in Section 5.3—that this scaling breaks spurious correlations and encourages focus on causal factors—is heuristic; no formal result links the algebraic scaling to the behavior of stochastic gradient ascent or to representation learning. Please either add a formal statement (under simplifying assumptions) or clearly label this connection as a conjecture supported by the experiments.","section":"Section 5.2 (Theorem 1)"},{"comment":"The comparison does not isolate the proposed mechanism. Standard advantage baselines are known to reduce variance and often improve performance relative to pure Q-value (return) training; the ablation with advantage normalization (Figure 3, left) is suggestive but only shows that normalization hurts out-of-trajectory performance, not that the 1−P scaling, rather than any value baseline, is the cause. To support the paper's central claim, the experiments should include a condition with the standard full-state baseline A(s,a)=Q(s,a)−V(s) in addition to A_Φ(s,a)=Q(s,a)−V(Φ(s)) for an explicit Φ, and show that the additional coarse-representation baseline changes the learned representation and out-of-trajectory generalization in the predicted direction.","section":"Section 6 (Figures 2–3)"}],"minor_comments":[{"comment":"There is a typo: 'represantion' should be 'representation' in the first paragraph of the Limitations section.","section":"Section 7"},{"comment":"The notation for the alternative-state-action baseline is inconsistent: Theorem 1 and the surrounding text write \\tilde{Q}^\\pi(\\neg\\langle s_t,a_t\\rangle) without a subscript \\Phi, while the proof and Definition 6 use \\tilde{Q}^\\pi_\\Phi. Please use one consistent notation.","section":"Section 5.2"},{"comment":"The notation R^\\pi(\\Phi(s_t), a_t) and P^\\pi(\\Phi(s_{t+1}) | \\Phi(s_t), a_t) is not formally defined; it would help to state explicitly that these denote expected reward and transition probabilities under π given the coarse state, to avoid confusion with the reward function R defined on full states.","section":"Definition 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the theorem is correct, but the central empirical claim rests on a theory–experiment gap that is acknowledged in the paper's own Limitations section. The experimental results are consistent with the story but cannot distinguish the proposed mechanism from generic baseline effects. I would support a major revision rather than rejection, because the gap is addressable: the authors could add an explicit-Φ condition or provide a theoretical extension to function approximation. The paper's scope is narrow (three simple gridworlds), and the causal language in the abstract is somewhat stronger than the formal results justify, but the paper itself is careful in Section 7 to disclaim strong causal guarantees."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper proves a clean algebraic identity (Theorem 1) that, under a specific definition of advantage with a coarse-grained baseline, weights policy-gradient updates by the complement of the state-action probability under the current policy. It also shows in three simple gridworlds that agents trained on advantages rather than raw Q-values generalize better out-of-trajectory, and that normalizing advantages removes that benefit. The math is correct, and the experiments support the empirical claim.\n\nWhat's actually new: the identity formalizes an intuition that advantages downweight frequent state-action pairs. That's a useful contribution, and the paper is honest about its limits: it repeatedly notes that advantages give no guarantee of causal representations and that the environments are simple. The KL-divergence analyses are a nice addition, showing that advantage-trained policies actually condition on the causal variables in these tasks.\n\nThe main soft spot: the theorem assumes Q is evaluated on the full state while the baseline V(Φ(s)) averages over the equivalence class of a fixed coarse representation Φ. In the experiments, both Q and V are neural networks with shared features, so the theorem does not apply as stated. The paper acknowledges this in the Limitations section but still interprets the experimental results as evidence for the scaling mechanism. The normalization result is consistent with the story, but it is also consistent with generic baseline effects such as variance reduction or update scale. Without an ablation that isolates the 1−P scaling, the causal explanation is not directly tested.\n\nThere is also no released code or data, which is a minor issue for reproducibility given the simple setups.\n\nBottom line: the identity is worth having, and the empirical difference between Q-value and advantage training is real in these environments. The paper doesn't oversell itself, and the load-bearing weakness is the gap between theory and implementation, not a flaw in either part. People working on policy-gradient baselines or RL generalization will get something from it.\n\nRecommendation: send it to peer review. The referees should ask for experiments that engage with the theorem's assumptions, or for a revised narrative that separates the formal result from the empirical demonstration.","headline":"A clean algebraic identity about advantages under coarse baselines, with an honest empirical story whose causal interpretation needs stronger tests.","tokens_in":14609,"tokens_out":3572,"would_cite":true,"duration_ms":42338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the advantage function rescales policy-gradient updates by the complement of a state-action pair's probability under the policy, and that this rescaling mitigates policy confounding and improves out-of-trajectory…","keywords":["policy confounding","advantage function","causal state representation","out-of-trajectory generalization","policy gradients","spurious correlations","state abstraction","reinforcement learning"],"falsifier":"Run the Key2Door evaluation variant with PPO using advantages and no normalization; the paper predicts strong out-of-trajectory performance. Then run the identical training with advantages normalized within each batch; the paper predicts a performance collapse. If normalization produces little or no drop, or if unnormalized advantages offer no benefit over Q-values, the scaling effect is not the operative mechanism.","tokens_in":13657,"feed_emoji":"🎯","tokens_out":16585,"duration_ms":171467,"temperature":0.7,"pith_summary":"The paper aims to establish that the advantage function, the usual Q-minus-baseline correction in policy-gradient methods, does more than reduce gradient variance. The author proves that when the action value $Q^\\pi(s_t,a_t)$ is evaluated on the full state and the baseline $V^\\pi(\\Phi(s_t))$ is averaged over the equivalence class induced by a coarse representation $\\Phi$, the advantage equals a factor $1-P^\\pi(s_t,a_t\\mid\\Phi(s_t))$ times a contrast term. That factor rescales stochastic policy-gradient updates, shrinking the influence of state-action pairs the policy already visits often and amplifying rare, informative ones, which is exactly what breaks the spurious correlations that the policy itself creates. In three gridworlds, agents trained with advantages generalize to trajectories they never followed, whereas agents trained on raw Q-values fail, and normalizing the advantages removes the benefit. If correct, this turns a standard variance-reduction trick into a mechanism for learning causal rather than habitual state representations.","feed_headline":"Rescaling updates by rarity breaks spurious agent habits","feed_subtitle":"The Q-minus-baseline trick downweights familiar state-action pairs so agents generalize to never-taken trajectories.","key_machinery":"The central object is the advantage function under a coarse state representation, $A^\\pi_\\Phi(s_t,a_t)=Q^\\pi(s_t,a_t)-V^\\pi(\\Phi(s_t))$, in which $Q^\\pi$ sees the full state while $V^\\pi$ is aggregated over the equivalence class $\\{s_t\\}_\\Phi$. This asymmetry is deliberate: the baseline marginalizes away exactly the variables that $\\Phi$ discards, so the subtraction measures how much a particular full state-action pair deviates from its represented class. Theorem 1 converts that deviation into a product of a probability-complement weight and a contrast term, and that product is what reweights stochastic gradient samples during training. Corollary 1 then shows that for causal representations the same mechanism reduces to reweighting by the complement of the action probability.","core_discovery":"On the paper's own terms, the discovery is a decomposition identity for the advantage under a state representation. For any $\\Phi$, defining $A^\\pi_\\Phi(s_t,a_t)=Q^\\pi(s_t,a_t)-V^\\pi(\\Phi(s_t))$ yields $A^\\pi_\\Phi(s_t,a_t)=(1-P^\\pi(s_t,a_t\\mid\\Phi(s_t)))(Q^\\pi(s_t,a_t)-\\tilde Q^\\pi_\\Phi(\\neg\\langle s_t,a_t\\rangle))$, where $\\tilde Q^\\pi_\\Phi(\\neg\\langle s_t,a_t\\rangle)$ is the probability-weighted average Q-value over all other state-action pairs in the same equivalence class. Since stochastic gradient estimates sample pairs with frequency $P^\\pi(s_t,a_t\\mid\\Phi(s_t))$, the complement factor systematically downweights over-sampled pairs and boosts under-sampled ones. The paper argues that this breaks the feedback loop in which a policy makes a pair frequent, the frequent pair dominates gradient updates, and the agent accordingly discards causal variables that merely correlate with the frequent route. When $\\Phi$ is a causal (Markov) representation, Corollary 1 simplifies the factor to $1-\\pi(a_t\\mid\\Phi(s_t))$, so only action probabilities need reweighting once the representation already captures the true causal factors.","pith_inferences":["Beyond the paper: the complement-probability factor from Theorem 1 could be applied explicitly as a per-sample weight in Q-based or off-policy training, which would test whether the scaling effect transfers to settings without a separate advantage estimator.","Beyond the paper: the same identity suggests a diagnostic for representation quality, comparing $V^\\pi(\\Phi(s))$ with the class-aggregated Q-values; a large gap would flag variables that $\\Phi$ has collapsed but that still carry causal information.","Beyond the paper: if the mechanism scales to high-dimensional domains, then RL libraries that normalize advantages by default may be trading causal reliability for numerical stability, and a simple reparameterization could preserve both."],"forward_implications":["Agents trained on raw Q-values can lock onto spurious correlations and fail when forced off their usual trajectories; advantage-trained agents generalize well in all three environments.","Standard advantage normalization, as used in many PPO implementations, removes the per-sample scaling and reproduces the Q-value failure mode, making implementation details central to whether causal representations are learned.","Increasing batch size partly mitigates Q-based training failures, but the paper finds no reason to prefer large batches over simply training on advantages.","If the state representation is already causal, the advantage reweighting reduces to an action-probability factor, so the method does not need to correct for state frequencies."],"supporting_citations":[{"why":"Defines policy confounding and supplies the three evaluation environments (Key2Door, Frozen T-Maze, Diversion) that the experiments reuse.","marker":"Suau et al. (2024)"},{"why":"Shows that baselines in policy gradients can cause overly aggressive, committal updates, a failure mode the paper attributes to Q-based training.","marker":"Chung et al. (2021)"},{"why":"Analyzes how state-value baselines moderate update aggressiveness and ensure convergence, supporting the paper's account of the advantage mechanism.","marker":"Mei et al. (2022)"},{"why":"Provides the do-operator and the causal vocabulary in which policy confounding and causal state representations are formalized.","marker":"Pearl et al. (2016)"},{"why":"States the policy gradient theorem whose stochastic sample updates are the target of the advantage reweighting.","marker":"Sutton et al. (1999)"},{"why":"Supplies PPO, the main experimental algorithm, and its default advantage normalization is identified as erasing the scaling effect.","marker":"Schulman et al. (2017)"},{"why":"Supplies REINFORCE, the second algorithm used to confirm the advantage-versus-Q pattern.","marker":"Williams (1992)"}],"fun_headline_variants":["Advantage function downweights policy-favored actions to boost generalization","Rarity-weighted updates break spurious state-action habits","Advantage reweights actions by rarity to break policy confounding","Downweight frequent actions, break spurious habits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands or falls on the premise that $Q^\\pi(s_t,a_t)$ is estimated from the full state while the baseline $V^\\pi(\\Phi(s_t))$ is computed from the coarse representation; the paper's limitations section concedes that if $Q$ is learned entirely through function approximation it too is subject to $\\Phi$, and the stated theorem no longer holds.","fun_headline_variants_meta":{"raw":{"variants":["Advantage function downweights policy-favored actions to boost generalization","Rarity-weighted updates break spurious state-action habits","Advantage reweights actions by rarity to break policy confounding","Downweight frequent actions, break spurious habits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001195,"raw_usage":{"total_tokens":4931,"prompt_tokens":952,"completion_tokens":3979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":3913}},"tokens_in":568,"tokens_out":3979,"duration_ms":28605,"temperature":1.0,"reasoning_tokens":3913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T01:00:25.704601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Key2Door evaluation variant with PPO using advantages and no normalization; the paper predicts strong out-of-trajectory performance. Then run the identical training with advantages normalized within each batch; the paper predicts a performance collapse. If normalization produces little or no drop, or if unnormalized advantages offer no benefit over Q-values, the scaling effect is not the operative mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines policy confounding and supplies the three evaluation environments (Key2Door, Frozen T-Maze, Diversion) that the experiments reuse."},{"cited_title":"C., and Le Roux, N","cited_arxiv_id":null,"evidence_quote":"Shows that baselines in policy gradients can cause overly aggressive, committal updates, a failure mode the paper attributes to Q-based training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Analyzes how state-value baselines moderate update aggressiveness and ensure convergence, supporting the paper's account of the advantage mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the do-operator and the causal vocabulary in which policy confounding and causal state representations are formalized."},{"cited_title":"S., McAllester, D., Singh, S., and Mansour, Y","cited_arxiv_id":null,"evidence_quote":"States the policy gradient theorem whose stochastic sample updates are the target of the advantage reweighting."}],"review_version":1}