Logit Dynamics in Softmax Policy Gradient Methods

Yingru Li

REVIEW 3 major objections 4 minor 2 cited by

Logit Dynamics in Softmax Policy Gradient Methods

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper derives the exact L2 norm of a softmax policy-gradient logit update, $\eta |A|\sqrt{1 - 2P_c + C(P)}$, showing that update size is governed by the chosen action's probability and the policy's collision probability.

desk verdict Correct but elementary algebra, framed as a foundational result; the convergence claims should be removed and the parameterization caveat taken seriously. read the letter →

arxiv 2506.12912 v1 pith:TB3SK637 submitted 2025-06-15 cs.LG cs.AIstat.ML

Yingru Li This is my paper

classification cs.LGcs.AIstat.ML

keywords softmaxpolicygradientlogitdynamicscollisionprobabilityRényientropyexploration-exploitationzero-sumupdatesself-regulationconfidence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper derives an exact expression for how much a softmax policy-gradient update changes the policy logits. Working in the direct logit parameterization, the author defines the logit update as the score-function step and shows that its L2 norm is $\eta |A| \sqrt{1 - 2P_c + C(P)}$, where $P_c$ is the probability of the chosen action and $C(P) = \sum_a P_a^2$ is the policy's collision probability. The formula reveals a self-regulating dynamic: when the policy is already confident about an action, the updates shrink toward zero, while rare exploratory choices trigger large updates. Because the logit updates also sum to zero, each step is a pure redistribution of probability mass rather than a uniform shift. If the identity holds as a description of intended logit changes, it supplies a simple, entropy-linked diagnostic for when policy-gradient training is moving quickly or stalling.

What carries the argument

The central object is the logit update vector $\Delta \mathbf{z}$, defined as the effect of one softmax policy-gradient step on the raw logits, with $\Delta z_j = \eta A\,(\partial\log\pi(a_c|s)/\partial z_j)$. The load-bearing identity is the softmax score function $\partial\log\pi(a_i|s)/\partial z_j = \delta_{ij} - P_j$, which turns the update into the two simple rules $\Delta z_c = \eta(1-P_c)A$ and $\Delta z_o = -\eta P_o A$; from these, the zero-sum property and the norm formula follow by direct expansion. The distribution-wide quantity doing the work is the collision probability $C(P) = \sum_a P_a^2$, the probability that two independent draws from the policy agree; it ranges from $1/n$ (uniform policy) to $1$ (deterministic policy) and equals $e^{-H_2(P)}$, linking the update magnitude to Rényi entropy of order 2. The machinery supplies the sensitivity factor $\sqrt{1-2P_c+C(P)}$ that multiplies $\eta|A|$, and it is what allows the paper to state that update vigor is modulated by both the chosen action and the policy's overall concentration.

What would settle it

Run one policy-gradient update on a softmax policy with direct logit parameters and known action probabilities; the measured L2 change in the logit vector must equal $\eta|A|\sqrt{1-2P_c+C(P)}$ exactly, since this is an algebraic consequence of the score function. For a neural-network policy, the same measurement on the actual logits will generally differ by the parameterization Jacobian, and that discrepancy is the observation that would delimit the formula's scope.

Watch

Extended reading notes

Core claim

The central claim is that the magnitude of a single softmax logit update, under the direct-logit parameterization, is exactly $$\|\$\Delta$ \mathbf{z}\|_2 = \eta |A| \sqrt{1 - 2P_c + C(P)},$$ with $C(P)=\sum_a P_a^2$ the collision probability of the policy distribution. The derivation starts from the softmax score function $\partial\log\pi(a_c|s)/\partial z_j = \delta_{cj} - P_j$, which gives $\Delta z_c = \eta(1-P_c)A$ for the chosen action and $\Delta z_o = -\eta P_o A$ for every other action. Summing these changes gives zero, so a policy-gradient step is a redistribution of relative logit strength. The paper reads the formula as a self-regulation mechanism: confident exploitation of a nearly deterministic action ($P_c\approx1$, $C(P)\approx1$) drives the update norm to zero, while exploration of a low-probability action produces an update of size up to $\eta|A|\sqrt{2}$. The collision probability is identified with $e^{-H_2(P)}$, the exponential of the negative Rényi-2 entropy, which makes the concentration dependence of learning vigor explicit.

Load-bearing premise

The load-bearing premise is that the logit update equals the score-function step $\eta A\,(\partial\log\pi/\partial z)$, which is exactly true only when the policy is parameterized directly by logits; for neural-network policies the parameterization Jacobian enters and the formula may not match the observed logit changes.

Editorial extensions

If this is right

A nearly deterministic policy essentially stops changing its logits when it chooses its dominant action, because the update norm $\eta|A|\sqrt{1-2P_c+C(P)}$ approaches zero; this acts as a built-in stabilizer near convergence.
Choosing a low-probability action produces the largest single-step shifts, bounded by $\eta|A|\sqrt{2}$, so rare exploratory experiences drive the strongest policy movement.
Because the logit changes sum to zero, each policy-gradient step only moves probability mass between actions rather than shifting the whole logit baseline.
The update magnitude depends on the whole distribution through $C(P)$, not only on the chosen action, so policies with equal $P_c$ but different concentration experience different learning vigor.
Since $C(P)=e^{-H_2(P)}$, the formula ties update vigor to Rényi entropy: a high-entropy policy updates vigorously while a low-entropy policy suppresses its own updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The identity is derived for direct logit parameters; for neural-network policies the true logit change includes the parameter-to-logit Jacobian, so an empirical check of whether the formula still tracks actual logit movement is a natural next test.
Because $C(P)$ can be computed without logarithms, the formula suggests a cheap online diagnostic: monitoring $\sqrt{1-2P_c+C(P)}$ during training could flag premature convergence when update magnitude collapses.
The self-regulation story hints at a design principle the paper does not pursue: modifying the score function to keep a floor under update magnitude as $P_c\to1$ could counteract premature convergence.
The zero-sum conservation suggests that adaptive per-logit learning rates, which rescale each action's update separately, would break the redistribution structure; whether that helps or hurts convergence is a testable follow-up.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: this paper proves a correct single-step identity for the L2 norm of the softmax logit score, but it is an elementary expansion of the standard per-logit update rules, and the paper's stability/convergence language is not supported by anything in it.

What is actually there: for the direct logit parameterization, the update is Δz_c = η(1−P_c)A and Δz_o = −ηP_o A. Squaring and summing gives ||Δz||_2 = η|A| sqrt(1 − 2P_c + C(P)) with C(P) = Σ P_a². That is arithmetically correct. The paper presents it cleanly, with tables and a useful intuition: update magnitude is large when P_c is small, and vanishes when the policy is already confident. The collision-probability name is a nice way to package ΣP_a², and the connection to Rényi-2 entropy is standard but correctly stated. No fitted constants, no circularity; it is a parameter-free derivation.

Where the soft spots are. First, the identity is about the logit score, not about actual logit changes under policy gradient in any parameterized policy. As the paper itself notes in Sec. 2.2, logits change through the Jacobian ∂z/∂θ. For a linear policy z_j = φ_jᵀ θ_j, the true update is Δz_c = ηA||φ_c||²(1−P_c) and Δz_o = −ηA||φ_o||² P_o. Eq. (10) is false in general, and even the zero-sum property (Eq. 8) is a special case. The paper never bounds or accounts for the Jacobian, so the universal framing in the abstract and conclusion overreaches.

Second, even under the direct parameterization, Eq. (10) is a one-step magnitude. It says nothing about the iterative dynamics, stability, or convergence. The abstract calls this a foundational insight into stability and convergence, but there is no argument connecting a decreasing step size to global convergence. That language should go.

The entropy appendix is fine but standard. No citation problems; the two references are appropriate. The paper is short and honest in its derivation, just over-ambitious in its claims.

Who this is for: someone who wants a compact exposition of why softmax policy gradient updates shrink as the policy becomes confident. It is not a result that will change how we think about neural policy gradients because the Jacobian is typically not identity. I would not cite it in my own work as a substantive contribution, but it could be a useful lecture note.

Bottom line: I would send this to a referee, but tell them the central task is to push the authors to reframe. The math is fine; the claims need to match it. That is a fixable problem, not a fatal one.

Referee Report

3 major / 4 minor

Summary. The paper studies the change in softmax logits induced by a single policy-gradient step. It defines a logit update as the learning rate times the advantage times the logit-score function, derives the per-action update equations (6)-(7), the zero-sum property (8), and the exact L2 norm identity (10), where the update magnitude is expressed through the chosen action's probability P_c and the collision probability C(P)=\sum_a P_a^2. An appendix relates collision probability to Rényi entropy and proves the entropy ordering H_2(P) \le H(P). The conclusion interprets the identity as a self-regulating mechanism that promotes stability and a natural exploration-exploitation transition.

Significance. The algebraic derivation of Eq. (10) is correct and parameter-free: no constants are fitted, and the identity follows directly from the softmax score function. If properly scoped, this is a clean observation about the direct logit parameterization of softmax policies, and the collision-probability interpretation is elegant and potentially useful for teaching or for analyzing tabular/direct logit implementations. However, the paper's broader claims—that the formula characterizes the logit dynamics of softmax policy gradient methods generally and that it provides a foundational insight into stability and convergence—are not supported by the presented analysis, which considers only a single update in a hypothetical direct logit parameterization.

major comments (3)

[Sec. 2.2, Eq. (5)] The logit update is defined in Eq. (5) as \Delta z_j = \eta A (\partial \log \pi(a_c|s)/\partial z_j), which deliberately discards the parameter Jacobian \partial z/\partial \theta introduced in Eq. (3). For an actual policy gradient update with parameters \theta, the logit change is \Delta z_j = (\partial z_j/\partial \theta)^T \Delta \theta, which reduces to Eqs. (6)-(7) only when the Jacobian is the identity. For example, with a linear softmax policy z_j = \phi_j^T \theta_j, the true update is \Delta z_c = \eta A \|\phi_c\|^2 (1-P_c) and \Delta z_j = -\eta A \|\phi_j\|^2 P_j for j \ne c; this coincides with Eq. (6)-(7) only if all feature norms equal 1. Consequently, the zero-sum property (Eq. (8)) and the magnitude formula (Eq. (10)) are properties of the direct logit parameterization, not of softmax policy gradient methods generally. The abstract and conclusion should be revised to state this scope explicitly.
[Sec. 5, items 3 and 4] Eq. (10) is a single-step, single-experience norm identity. The conclusion's statements that the formula 'promotes convergence', 'facilitates a natural transition from exploration to exploitation', and 'leads to stability' are not consequences of this identity. No iterative analysis, expected progress bound, Lyapunov argument, or convergence theorem is provided, even for the direct logit parameterization. These claims should be removed or clearly marked as informal interpretation rather than mathematical implications.
[Sec. 1] The introduction claims the analysis is 'independent of the specific policy parameterization \theta' and offers 'a parameterization-agnostic view of the learning update'. This is misleading: the derivation in Sec. 2.3 treats the logits themselves as the directly updated variables, which is a specific parameterization choice. The phrase 'parameterization-agnostic' should be replaced with a statement that the paper analyzes the logit-score component of the gradient, acknowledging that the actual logit change depends on the architecture-dependent Jacobian.

minor comments (4)

[Abstract] The abstract contains a grammatical error: 'We analyzes' should be 'We analyze'. It also uses a proportionality sign for a quantity that Eq. (10) states as an equality; please use '=' or clarify the omitted constant.
[Sec. 3, Tables 1 and 2] The column header 'Update Scaler' appears to be a misspelling of 'Update Scalar'. Additionally, the tables illustrate qualitative monotonicity but do not add information beyond Eqs. (6)-(7); consider shortening them or moving them to an appendix.
[Sec. 4.3] The 'key dynamics' statements such as 'When P_c \approx 1, \|\Delta z\|_2 \approx 0' are asymptotic approximations. Since the expression under the square root is always nonnegative, it would be more precise to state these as limits or to give the exact ranges for finite action sets.
[Appendix A.2, Theorem 1] The proof of the entropy ordering relies on the monotonicity of Rényi entropy in its order parameter without proving or citing this property. A citation or one-sentence justification would make the appendix self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Eq. (10) is a direct algebraic expansion of the explicitly defined logit update, with no fitted parameters or self-citation.

full rationale

The paper's load-bearing result, Eq. (10), is obtained by substituting the softmax logit score function (Eq. 4) into the explicitly defined logit update (Eq. 5), then expanding (1-P_c)^2 and using the definition C(P)=sum_a P_a^2. This is a self-contained algebraic identity rather than a prediction fitted to data or imported from prior work. There are no fitted parameters, no self-citations, no invoked uniqueness theorems, and no ansatz smuggled in via citation: the analysis begins from the softmax definition and the policy gradient score function. The caveat that the quantity labeled Δz is not the actual change in logits under a general parameterization (Sec. 2.2) is a scope limitation and a correctness concern, not a circularity, because the paper openly states it is analyzing the effective logit-score component rather than pretending the Jacobian is absent. The stability/convergence language in the abstract and conclusion extrapolates from a single-step magnitude identity; that extrapolation is an interpretive overreach, but it does not make the derivation circular. Accordingly no circular step is identified and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters. The only quantities (η, A, P) are inputs. The paper invokes the policy gradient theorem as background and defines the logit update in a way that sidesteps the parameterization Jacobian.

assumptions (3)

standard math Policy gradient theorem (Eq. 1): ∇θ J = E[∇θ log π A].
Cited to Sutton et al. [2]; assumed as background.
ad hoc to paper The logit update is defined as Δz_j = η A (∂log π(a_c|s)/∂z_j), ignoring the parameterization Jacobian ∂z/∂θ (Eq. 5, Sec. 2.2).
This is the paper's own definition to make the analysis parameterization-agnostic; it limits the result to a direct logit parameterization.
domain assumption A single experience (s, a_c, A) is considered; no expectation over states or actions.
The norm is for one update step, not an averaged or trajectory-level quantity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Logit Dynamics in Softmax Policy Gradient Methods." pith.science (2026). https://pith.science/paper/TB3SK637

@misc{pith2026250612912,
  author       = {Pith},
  title        = {Pith review of: Logit Dynamics in Softmax Policy Gradient Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TB3SK637}},
  note         = {Machine review of arXiv:2506.12912}
}

abstract

We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: $$ \|\Delta \mathbf{z}\|_2 \propto \sqrt{1-2P_c + C(P)} $$ This equation demonstrates that update magnitudes are determined by the chosen action's probability ($P_c$) and the policy's collision probability ($C(P)$), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
cs.LG 2025-09 conditional novelty 6.0 of 10

Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
cs.LG 2025-09 conditional novelty 5.0 of 10

EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 2 Pith papers

[1]

On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

2021
[2]

Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999. A Collision Probability, R´ enyi Entropy, and Shannon Entropy A.1 Definitions and Relationships Understanding the exploration-exploitation charac...

work page 1999

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

2021

[2] [2]

Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999. A Collision Probability, R´ enyi Entropy, and Shannon Entropy A.1 Definitions and Relationships Understanding the exploration-exploitation charac...

work page 1999