Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.
Logit Dynamics in Softmax Policy Gradient Methods
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: $$ \|\Delta \mathbf{z}\|_2 \propto \sqrt{1-2P_c + C(P)} $$ This equation demonstrates that update magnitudes are determined by the chosen action's probability ($P_c$) and the policy's collision probability ($C(P)$), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.
citation-role summary
method 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
method 1polarities
use method 1representative citing papers
citing papers explorer
-
SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.