{"id":"fc82d965-ec3c-4cef-aeda-1c2985199cfc","arxiv_id":"2607.27574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PGS builds a removable activation offset from return-weighted action-score gradients and steers frozen policies across gridworld, chess, and football.","lead":"Policy Gradient Steering turns a handful of scored trajectories into a small, removable activation vector that shifts a frozen policy's behavior at inference time. The paper tests it in a two-route gridworld, chess puzzles, and football, and shows compatible objectives can be composed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chess results lack isolated action-KL checks: PGS's advantage could be an artifact of unmatched policy-change budgets at α from Eq. 12.","rationale":"The reader's weakest assumption—the local quadratic (Fisher) approximation—is the same concern I identify, but I sharpen it to a concrete, checkable gap: the chess experiments never report isolated held-out action KL. This is the most load-bearing concern because the paper's headline contributions include composability in chess, supported by comparisons across methods at a stated KL budget. The gridworld results include a KL verification (Table 2) and thus partially de-risk the calibration, but the chess experiments rely entirely on the quadratic extrapolation without showing the achieved KL. If the approximation is inaccurate, the effective policy-change budgets are unmatched, and the reported gains (e.g., PGS's highest isolated gain and min retention) may be artifacts of larger perturbations rather than genuine steering quality. This concern is not a refutation; the method may still work, and the paper even suggests backtracking as a fix. However, the omission of isolated KL values in the main chess tables is a missing piece of evidence that is directly testable. The verdict remains CONDITIONAL: the paper should add these measurements (and likely re-calibrate if needed) before full acceptance. I agree with the reader's choice of weakest assumption, hence agreement='agree'.","tokens_in":21677,"tokens_out":8085,"duration_ms":81515,"concrete_test":"For each of the three chess motifs and each method (PGS, ReFT, LoRA, Fine-tuning), measure the held-out action KL of each independently fitted adaptation before composition, at the α computed as in Appendix B.2. If any method's isolated KL deviates from 0.1 by more than 20%, or if the methods' KLs differ by more than 20% relative, re-calibrate α via binary search to exactly hit 0.1 for every method and recompute Tables 4–6. If PGS's isolated gain, composed gain, or minimum retention change materially, the matched-budget claim is compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central compositional claim rests on comparing methods at the same action-KL budget (ε=0.1). In chess, α is set via a local quadratic approximation to the KL (Eq. 12, Appendix D), with the curvature estimated from the realized action KL at coefficients ±10⁻³ (Appendix B.2). The quadratic extrapolates from a tiny scale to ε=0.1, which is 100× larger. If higher-order KL terms matter at this scale, the actual held-out KL for each independently fitted adaptation may differ substantially from 0.1, and across methods. The paper never reports isolated held-out action KL for chess—only composed KL (e.g., Tables 4–6, 17–20). If PGS's true isolated KL is larger than the baselines', its higher isolated gain and composition retention (Eq. 17) could reflect a larger policy-change budget, not a better direction. The gridworld experiments do verify held-out KL (Table 2), but the chess experiments—the primary evidence for composition—do not. The paper itself acknowledges in Section 7 that \"When the local Fisher approximation is inaccurate at finite scale, empirical action-KL backtracking could refine the closed-form coefficient.\" This is therefore a missing verification, not a demonstrated failure, but it is load-bearing: the matched-budget premise is unverified for the composition results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Policy Gradient Steering (PGS), a method that constructs an additive activation-space steering vector by accumulating return-weighted policy gradients of the log-likelihood of observed actions, computed from a small set of scored trajectories. The intervention is applied as h ← h + αv at inference time, leaving the base policy frozen. The paper evaluates PGS in three settings: a two-route gridworld where it shows that contrastive activation steering (CAA, COAST, K-Steer) fails to control the route decision, while PGS succeeds; chess puzzles where independently fitted PGS vectors for forks, pins, and skewers are combined and reported to compose constructively; and Google Research Football where PGS interventions are claimed to alter team behavior and transfer across opponents. The paper positions PGS as a compact, composable, and removable behavioral intervention method.","tokens_in":22026,"tokens_out":8014,"duration_ms":80632,"significance":"If the empirical claims are substantiated, PGS is a conceptually clean and practically useful addition to the post-training adaptation toolbox: it assigns credit to the actions that produce outcomes rather than to post-outcome representations, and it does so with a single gradient step and a storage overhead far smaller than LoRA or fine-tuning. The gridworld failure analysis (Section 4.3, Eq. 15) is a nice controlled demonstration of why outcome-based contrasts can fail at a branching decision. The paper also ships code and per-seed diagnostics (Appendix E), which supports reproducibility. However, the significance is currently tempered by two load-bearing empirical gaps: the chess experiments do not verify that each method actually operates at the same isolated action-KL budget, and the football transfer result lacks per-opponent unsteered baselines and is statistically underpowered. These issues must be addressed before the central claims can be fully accepted.","major_comments":[{"comment":"The matched-budget premise is unverified in the chess experiments. All methods set α via the local quadratic (Fisher) approximation α = sqrt(2ε/(v^T F v)) with ε=0.1, using curvature estimated at coefficients ±1e-3 (Appendix B.2). The paper reports only composed KL for the three-way composition, not the held-out isolated action KL for each independently fitted intervention. If higher-order KL terms are non-negligible at the applied scale, the actual isolated KL may differ across methods and across seeds, undermining the comparison of isolated gains in Table 4 and the composition-retention metric R_min in Eq. (17). The paper's own Section 7 acknowledges that the local Fisher approximation may be inaccurate at finite scale. The gridworld verification (Table 2) does not transfer automatically to the deeper LCZero policy. Please report isolated held-out action KL for each method and seed in","section":"§5.2, Tables 4–5, Eq. (12), Appendix B.2, Appendix D"},{"comment":"The cross-play transfer claim is not supported by the data as presented. The text states that 'each intervention increases mean completed passes relative to the unmodified policy across evaluation opponents,' but Table 9 contains only the PGS intervention counts and does not show the unmodified group_pressure policy's completed passes against each of the four opponents. Without per-opponent baselines, the reported numbers could reflect opponent-specific baseline differences rather than a transferable intervention effect. Please add the unsteered group_pressure per-opponent counts (e.g., as a Base row or separate table) so the reader can directly verify the claimed increase.","section":"§6.2, Table 9"},{"comment":"The football results are statistically weak due to very large standard deviations and small sample sizes (n=5). For example, PGS yields 29.80±16.04 completed passes and LoRA 32.00±17.68; the difference between the two is far within noise, yet the text says PGS yields 'a completed-pass count comparable to the strongest learned adaptation' and a 'pronounced shift' relative to base. The mean differences versus the base are large, but the high variance means that the relative ordering of methods is not reliable. Please provide confidence intervals or effect sizes, and temper the comparative claims accordingly. At minimum, state that differences among the top methods are not statistically separable at this sample size.","section":"§6.2, Table 8"}],"minor_comments":[{"comment":"The paper uses 'sample standard deviation' throughout, but with n=5 the standard error is much smaller; consider reporting both or explicitly noting the small sample size in the main tables.","section":"General"},{"comment":"The maia-1500 rows in Table 5 repeat Table 4 but with small numeric differences (e.g., PGS composed gain 0.805 vs 0.804, LoRA 0.804 vs 0.803). Please clarify whether these are the same runs or different seeds, and ensure consistency.","section":"Table 5"},{"comment":"The algorithm says 'Estimate the baseline b(o_t)' while the text states an action-independent baseline; please align the notation.","section":"Algorithm 1 (line 2)"},{"comment":"The importance ratio ρ_t is introduced after the main formula; consider defining it before Eq. (8) or noting that ρ_t=1 for on-policy data to avoid initial confusion.","section":"Eq. (8) and Section 3.2"},{"comment":"The 'Goal diff.' column for ReFT reports 0.00±0.00, which is suspicious; please verify that no goals occurred in any of the five matches for that condition.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The reader's and my assessments align on the central concern: the chess composition results hinge on an unverified matched-KL assumption, and the football transfer evidence is missing a baseline. These are not irreparable; they are empirical verification gaps rather than internal contradictions. The paper's method is clearly stated and the gridworld analysis is solid, so I see a path to revision. If the authors can supply isolated chess action-KL numbers (or show via backtracking that the budgets are matched) and add per-opponent football baselines, the paper would be substantially stronger. The current recommendation reflects the need for those checks, not a rejection of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is worth a serious look. It identifies a genuine failure mode of contrastive activation steering — when trajectories split from the same observation, the contrast encodes downstream state visitation rather than the branching action — and proposes a simple fix: accumulate return-weighted policy gradients with respect to the activation to build a removable task vector. The gridworld experiments are clean, and the chess composition results are intriguing if the KL matching is real. The football transfer section is the weakest part and should be read with caution.\n\nThe core novelty is not the ingredients — PPLM and LatentSeek have used gradient-based activation perturbations, and the paper explicitly disclaims novelty for gradient-to-task-vector correspondence. The distinctive combination is accumulating likelihood-ratio gradients over a frozen policy's actions to get a single reusable activation offset, and the paper makes a fair case that this is absent from the cited literature. The gridworld 'why contrastive fails' analysis is the cleanest part: Equation (15) makes the zero-contrast point precise, and Table 2 shows the Fisher-based calibration actually produces the target held-out KL there. The chess experiments are well designed in most respects — disjoint fit/calibration/test, multiple seeds, comparison across three Maia checkpoints, and a layer-support ablation. The paper is also honest about its limits, which is more than many submissions bother with.\n\nThe main soft spot is the missing isolated-KL verification in chess. All chess comparisons are made under a claimed matched action-KL budget of 0.1, but the coefficient alpha is extrapolated from a local quadratic fit at ±1e-3, and the paper never reports the actual held-out isolated KL for each method. It only reports composed KL (e.g., Table 4), which is not a substitute. If PGS's isolated KL is systematically larger than the baselines, its higher isolated gain and retention ratio (Equation 17) could just reflect a larger policy-change budget. The paper itself flags in Section 7 that empirical backtracking could refine the coefficient, which makes the omission more conspicuous. This is not a demonstrated failure — the gridworld calibration works — but it is a load-bearing missing check for the paper's central composition claim.\n\nThe football results are noisy. Standard deviations are often as large as the effects, Table 9 lacks unsteered per-opponent baselines, so 'transfers across opponents' cannot be properly evaluated from the reported numbers. The off-policy correction is acknowledged to be approximate, which is fine, but it limits the strength of the football claims. Code is described in the appendix but no public link is given.\n\nWho is this for? Anyone working in activation steering or inference-time policy adaptation. It deserves a serious referee — the method is simple, the experiments span three domains, and the gridworld analysis alone is worth publishing. I would want the chess KL gap and football baselines addressed before accepting, but the paper is not fatally flawed.\n\nMy recommendation: send to peer review, and ask for isolated held-out KL in chess and per-opponent baselines in football. It's a conditional accept.","headline":"A clean and mostly honest paper that adds a real method to activation steering, but the chess composition claim is built on a KL-matching premise that is never verified at the scale used.","tokens_in":22442,"tokens_out":5247,"would_cite":true,"duration_ms":45241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Policy Gradient Steering turns a few scored trajectories into a removable activation vector that steers behavior where contrastive activation methods cannot.","keywords":["policy gradient steering","activation steering","inference-time intervention","task vectors","reinforcement learning","behavioral control","composable interventions","multi-agent behavior"],"falsifier":"After applying a PGS vector to a chess policy, remove it by setting alpha = 0 on the same held-out puzzles and check that the canonical-move likelihood returns to the unsteered value; if the effect persists after removal (for example because of stateful normalization in the forward pass), the intervention is not cleanly removable as claimed.","tokens_in":21611,"feed_emoji":"🎯","tokens_out":13396,"duration_ms":113436,"temperature":0.7,"pith_summary":"Existing activation-steering methods build interventions from contrasts between representations reached after outcomes diverge; the paper shows this construction fails to steer a simple two-route gridworld policy because the contrast encodes the consequence of a route choice rather than the action that caused it. Policy Gradient Steering (PGS) instead treats steering as a reinforcement-learning problem: it accumulates gradients of a temporary behavioral objective with respect to an activation layer, weighting each action's score gradient by its behavioral return, and stores the result as one additive offset vector. The vector is calibrated under an action-KL budget using the activation-space Fisher matrix, so it can be scaled, composed, and removed at inference time while the base policy stays frozen. Across the three settings - gridworld, chess puzzles, and competitive football - PGS shifts route preference where contrastive baselines do not, its tactical vectors compose constructively, and its team-behavior effects transfer across opponents. If correct, PGS gives a compact, reusable interface for post-training behavioral control in exactly the cases where outcome-based contrasts cannot assign credit.","feed_headline":"Policy gradients steer behavior where activation contrasts fail","feed_subtitle":"A few scored rollouts become a scalable and composable inference-time intervention.","key_machinery":"The central object is the PGS steering vector v_PGS = E_D[rho_t (G_t - b(o_t)) grad_h log pi(a_t|o_t)]: the likelihood-ratio (REINFORCE) gradient of a temporary behavioral return with respect to a chosen activation layer, averaged over a small batch of rollouts. This vector carries the argument by assigning credit to the actions that produced outcomes and by being state-independent - one offset added at every decision. The companion identity is the Fisher calibration alpha = sqrt(2 epsilon / (v_PGS^T F v_PGS)), which treats the intervention as a small offset in a locally quadratic basin of the action KL and sets its strength to a policy-change budget epsilon. Per-decision importance ratios r","core_discovery":"The paper's central claim is that a policy-gradient estimator evaluated at a hidden activation - not at the policy weights - turns scalar trajectory feedback into a removable behavioral intervention that contrastive activation methods cannot reliably produce. PGS computes v_PGS = E_D[rho_t (G_t - b(o_t)) grad_h log pi(a_t|o_t)] over a small scored-trajectory batch, then applies h <- h + alpha v_PGS at inference. Because grad_h log pi(a|o) differs across actions even when their pre-action representations are identical, PGS credits the branching action, where the decision-local contrast used by contrastive activation addition is exactly zero. Steering strength alpha = sqrt(2 epsilon / (v_PGS^T","pith_inferences":["If PGS vectors compose as cleanly as the chess results suggest, activation-space behavioral arithmetic could extend beyond plain summation - e.g., subtracting one vector to suppress a behavior, or averaging vectors fitted on different data sources - without retraining; the paper does not test these operations.","The football results indicate steering effectiveness depends on which trajectories the vector is fitted on; fitting on mixed-opponent data could reduce variance and improve transfer, which the paper leaves as future work.","Because the Fisher calibration is a local quadratic approximation, one testable extension is replacing the closed-form alpha with a small empirical KL backtracking at larger budgets; if the measured KL then departs from the requested epsilon, composition results at high strength would need re-calibration.","PGS's action-level credit could apply to language models at the token level, where each token choice precedes a change in hidden state; if token-level PGS vectors compose across attributes, it would give an alternative to contrastive activation editing for LLM behavior control."],"forward_implications":["PGS can steer a policy when the deciding action and its alternative share the same pre-action representation, a setting where contrastive activation addition produces zero decision-local gradient and leaves behavior nearly unchanged.","Independently fitted PGS vectors compose constructively: in chess, summing fork, pin, and skewer vectors gives composed gains close to fine-tuning while preserving each objective's isolated improvement (minimum retention above one).","PGS team-behavior interventions transfer across opponents in football: a passing intervention fitted against one opponent increases completed passes when played against each of the other evaluated opponents.","PGS achieves these changes with a small stored artifact (1,152 scalars versus 663,552 for fine-tuning in the chess all-block comparison), supporting lightweight post-training control.","Because the base policy parameters are untouched, the same vector can be applied at different strengths, negated, or removed at inference time without retraining."],"fun_headline_variants":["Policy gradients outperform activation contrasts for steering","Scored rollouts create a temporary steering vector","Policy gradient steering is composable and reversible","When activation contrasts fail policy gradients take over","Policy gradients enable composable temporary behavior tweaks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The local quadratic (Fisher) approximation of the action KL is accurate enough at the finite steering strengths used, so the closed-form coefficient alpha = sqrt(2 epsilon / (v_PGS^T F v_PGS)) keeps all compared methods at the same actual policy-change budget.","fun_headline_variants_meta":{"raw":{"variants":["Policy gradients outperform activation contrasts for steering","Scored rollouts create a temporary steering vector","Policy gradient steering is composable and reversible","When activation contrasts fail policy gradients take over","Policy gradients enable composable temporary behavior tweaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00177,"raw_usage":{"total_tokens":6791,"prompt_tokens":688,"completion_tokens":6103,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":6036}},"tokens_in":432,"tokens_out":6103,"duration_ms":39239,"temperature":1.0,"reasoning_tokens":6036,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:22:43.045643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"After applying a PGS vector to a chess policy, remove it by setting alpha = 0 on the same held-out puzzles and check that the canonical-move likelihood returns to the unsteered value; if the effect persists after removal (for example because of stateful normalization in the forward pass), the intervention is not cleanly removable as claimed.","supporting_citations":[],"review_version":1}