Pith. sign in

REVIEW 2 major objections 5 minor 21 references

Exploration Behavior of Untrained Policies

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Untrained neural policies already determine how an agent explores.

desk verdict A genuinely new framing of exploration as an architecture/initialization problem, but the central heavy-tailed steady-state claim rests on a Fokker-Planck solution that does not solve the equation and is not normalizable. read the letter →

arxiv 2506.22566 v3 pith:6NWEVKWM submitted 2025-06-27 cs.LG cs.AIstat.ML

Jacob Adamczyk This is my paper · ORCID
classification cs.LGcs.AIstat.ML MSC 68T0760J6060G15
keywords reinforcementlearningpolicyinitializationexplorationinfinite-widthlimitGaussianprocessFokker-Planckequationheavy-taileddistributionReLUnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that exploration in reinforcement learning is shaped before any training by the architecture and initialization of the policy network. A fixed, smooth (Lipschitz) policy produces nearly straight-line, ballistic trajectories on short timescales, while re-sampling the policy from its initialization distribution at every step produces diffusive trajectories with heavy-tailed state visiting. For infinitely wide ReLU networks, the paper derives a closed-form diffusion coefficient that grows with distance from the origin and predicts a quasi-Cauchy power-law state distribution. This matters because it suggests exploration can be designed through policy initialization rather than through reward bonuses or training, giving a new zero-shot handle on sparse-reward problems.

What carries the argument

The argument runs on two limits. First, the infinite-width Gaussian process limit converts a randomly initialized feedforward network into a Gaussian process with an architecture-dependent kernel, so action correlations across states are exactly $K(s,s')$. Second, the continuous-time Fokker-Planck limit for the per-step resampled random walk in state space turns the kernel into a diffusion tensor, yielding the PDE $\partial p/\partial t = -\nabla\cdot(\mu p) + \tfrac12\nabla^2(\Sigma p)$. For ReLU networks the kernel is non-stationary and gives $\Sigma(s)=\sigma_b^2+\sigma_w^2\|s\|^2/\pi$, which is what produces position-dependent diffusion and the claimed heavy-tailed distribution. A separate Lipschitz argument bounds the deviation of fixed-policy trajectories from straight lines, supplying the ballistic half of the picture.

What would settle it

Run the per-step resampled ReLU policy under linear dynamics $s_{t+1}=s_t+\pi(s_t)$ in two and three dimensions with $\sigma_b>0$, record a long trajectory, and compare the empirical radial density to $f(r)\propto(\sigma_b^2+\sigma_w^2 r^2/\pi)^{-d/2}$; if the distribution is not heavy-tailed with that exponent, or if the total mass inside a large ball keeps growing with time, the claimed steady state fails.

Watch

Extended reading notes

Core claim

The central claim is that the exploration behavior of an RL agent at the very start of training is controlled by the untrained policy's architecture and weight initialization. Concretely, a fixed deterministic policy that is Lipschitz continuous maps nearby states to nearby actions, and under local dynamics this yields ballistic motion: the trajectory stays within a bound of order $\delta L_a L_\pi t^2$ of the straight line $s_0 + c t$. Sampling a fresh policy from the same initialization each timestep instead decorrelates actions across steps, producing diffusive motion; in the infinite-width limit the policy is a Gaussian process with kernel $K(s,s')$, and for a one-hidden-layer ReLU network the diffusion coefficient is $\Sigma(s)=\sigma_b^2+\sigma_w^2\|s\|^2/\pi$, leading the paper to a heavy-tailed steady-state solution $f(r)\propto(\sigma_b^2+\sigma_w^2 r^2/\pi)^{-d/2}$. The paper proposes a hybrid strategy—fixed network first, then per-step resampling—to combine ballistic escape with diffusive coverage, and demonstrates it in a hallway barrier experiment.

Load-bearing premise

The heavy-tailed steady-state conclusion rests on the assumption that the per-step resampled policy process has a well-defined, normalizable stationary distribution over states, but the paper's own closed-form density does not satisfy that condition except in special cases.

Editorial extensions

If this is right

  • If the claim holds, a practitioner can choose exploration behavior by choosing activation, width, and initialization variance, with no training required.
  • Per-step policy resampling becomes a simple, reward-free exploration strategy that produces fat-tailed state visitation in free space.
  • Hybrid schedules (fixed policy for $n$ steps, then resampling) give a tunable trade-off between directional escape and diffusive coverage, with $n$ bounded by the inverse Lipschitz constant and environment scales.
  • The framework bridges network kernel properties—stationary versus non-stationary—and trajectory geometry: stationary kernels produce uniform diffusion, non-stationary ones produce distance-dependent exploration.
  • The same Gaussian-process analysis can be applied to other activations (tanh, periodic, and others) to predict their exploration priors before running any RL loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The closed-form stationary density as written, $f(r)\propto(\sigma_b^2+\sigma_w^2 r^2/\pi)^{-d/2}$, does not integrate to finite total probability in $d$ dimensions (the volume element leaves a logarithmically divergent tail), so a strict reading is that the heavy-tailed result is a transient or cutoff-dependent statement rather than a true stationary distribution; the $d=2$ and $\sigma_b=0$ speci
  • Because the diffusion coefficient grows as $\|s\|^2$ for ReLU networks, the process is increasingly erratic far from the origin; in bounded environments this means the heavy tail will hit boundaries, and the 'exploration' may overshoot—a testable consequence is the first-passage time through a narrow doorway as a function of $\sigma_w$.
  • The per-step resampling scheme is equivalent to a particular stochastic process on weights; connecting it to parameter-reset training procedures could turn 'primacy bias' mitigation into a quantitative exploration-design tool.
  • The argument suggests that an architecture search over activations could be performed analytically at initialization by comparing kernels, before any environment interaction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper studies exploration behavior of randomly initialized neural network policies. It argues that fixed Lipschitz policies induce short-time ballistic trajectories (Lemma 1), and that re-sampling the policy from its initialization distribution at every timestep yields, through the infinite-width Gaussian process limit and a Fokker-Planck continuous-time limit, diffusive trajectories with a heavy-tailed stationary state distribution f(r) ∝ (σ_b² + σ_w² r²/π)^{-d/2} (Eq. (3)). The paper then sketches an experiment with a barrier and narrow hallway to illustrate ballistic, diffusive, and hybrid exploration strategies, and frames these findings as evidence that architecture and initialization can be used as design tools for exploration before training.

Significance. The conceptual direction is appealing: linking the NNGP kernel of an untrained policy to the statistics of the trajectories it induces offers a principled way to reason about exploration without reward shaping. The paper is transparent about using standard tools (Neal's infinite-width limit and the Fokker-Planck equation) and does not introduce fitted parameters, which is a strength. However, both central theoretical pillars are mathematically invalid in their current form: Lemma 1 is not true for the stated general dynamics, and Eq. (3) is neither a solution of the stationary Fokker-Planck equation nor a normalizable density. The heavy-tailed steady-state claim, which is the main novelty, is therefore unsupported. The qualitative experiments do not compensate because they illustrate but do not validate the theory. If the mathematical claims could be repaired, the framework would be a useful conceptual contribution for the RL exploration community; in the present form the paper does not meet the bar for publication.

major comments (2)
  1. [§2.4, Eq. (3)] Equation (3) is not a solution of the stationary Fokker-Planck equation in the stated generality, and the density it defines is not normalizable for any d ≥ 1. Substituting f(r) = (a + c r²)^{-d/2} with a = σ_b² and c = σ_w²/π into ∇²(Σ p∞) = 0 gives Δ(Σ f) = 2 c d a (1 - d/2)(a + c r²)^{-(d+2)/2}, which vanishes only for d = 2 or a = 0. Furthermore, ∫ r^{d-1}(a + c r²)^{-d/2} dr diverges logarithmically at infinity for every d ≥ 1, including when σ_b > 0, contradicting the statement that the density "is normalizable if σ_b > 0". Correspondingly, the diffusion with Σ(s) ~ ‖s‖² is transient rather than stationary: the radial process has a positive drift of order (d-1)cR/2, so no finite stationary measure exists. This invalidates Contribution (2) and the abstract's claim of non-trivial or heavy-tailed steady-state state-visitation distributions.
  2. [§2.2, Lemma 1] Lemma 1 as stated is false for general dynamics satisfying Assumption 1 and the stated Lipschitz conditions. The proof compares s_{t+1} = f(s_t, πθ(s_t)) to the candidate linear trajectory s₀ + ct, but nothing in the assumptions ensures f(s, a) = s + a. Already at t = 0 the error |f(s₀, c) - (s₀ + c)| need not vanish; for example, dynamics of the form f(s, a) = s + a + ε with a small constant ε satisfy the Lipschitz and locality assumptions, yet the trajectory is s₀ + (c + ε)t, not s₀ + ct. The lemma becomes correct only under the additional assumption f(s, a) = s + a, or with c defined as the actual per-step displacement, which is the 'simplest linear dynamics' used later in §2.4. As written, the lemma is advertised for general transition dynamics and thus overclaims.
minor comments (5)
  1. [§2.4, Eq. (1) vs Eq. (3)] There is a factor-of-π inconsistency in the kernel at coincident points: evaluating Eq. (1) at s' = s gives K(s, s) = σ_b² + σ_w²‖s‖², whereas Eq. (3) uses K(s, s) = σ_b² + (σ_w²/π)‖s‖². The kernel normalization should be fixed so the two equations agree.
  2. [§2.4, Eq. (2)] The continuous-time limit leading to the Fokker-Planck equation is not made precise: no scaling of the action magnitude or the time step is specified. A reader cannot verify the diffusion coefficient in Eq. (2) without knowing how the per-step action variance is related to the time unit.
  3. [§2.5, Figures 1-3] The experiments are purely qualitative. The figures lack axis labels, environment details, hyperparameters, number of seeds, and quantitative exploration metrics, which makes the claimed empirical support difficult to assess or reproduce.
  4. [§2.3, Theorem 4] The convergence notation πθ → GP(0, K) should be made precise as weak convergence of finite-dimensional distributions as the hidden widths tend to infinity. Also, K(s, s') is written as taking values in R^{d×d}, while the subsequent analysis treats it as a scalar; the relation between the scalar kernel and the output dimension should be clarified.
  5. [§2.4, 'quasi-Cauchy'] The term 'quasi-Cauchy' is used for the distribution in Eq. (3), but since that density is not normalizable, calling it a distribution is misleading; the terminology should be revised along with the mathematical claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the derivation is parameter-free and rests on external standard results (Neal's GP limit, Fokker-Planck theory); the unsupported normalizability claim in Eq. (3) is a mathematical error, not circular reasoning.

full rationale

The paper's central derivation chain is not circular. Lemma 1 is proved from stated Lipschitz and locality assumptions using an explicit induction argument in the appendix; no fitted parameter or self-citation is involved. The Gaussian-process limit is attributed to Neal's theorem [11], an external standard result, and the ReLU kernel in Eq. (1) is presented as a known closed-form expression, not as a conclusion derived from the target claims. The Fokker-Planck equation (2) is the standard continuous-time description of the stated per-step resampling process, and Eq. (3) is offered as a radial solution of the stationary condition. There is no step in which a quantity is defined in terms of the outcome it is supposed to predict, no fitted parameter is renamed as a prediction, and no load-bearing result is imported from the author's own prior work. The only substantive issue I found is a mathematical-correctness problem, not a circularity problem: the text at Section 2.4 states that f(r) ∝ (σ_b^2 + σ_w^2 r^2/π)^(-d/2) 'describes a heavy-tailed stationary distribution ... and is normalizable if σ_b > 0.' As the skeptical analysis notes, substituting this f into ∇²(Σ p∞)=0 leaves a nonzero residual except for d=2 or σ_b=0, and the radial integral ∫ r^(d-1)(a + c r²)^(-d/2) dr diverges for all d, so the claimed normalizable stationary density is unsupported as stated. This affects the strength of Contribution (2) and the abstract's 'non-trivial state-visitation distributions,' but it is an error in the mathematical derivation, not a circular reduction of the conclusion to its inputs. I therefore score circularity 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No fitted constants appear in the derivations; sigma_w and sigma_b are standard initialization variances treated as model inputs. The principal burden is carried by standard theorems, Neal's GP limit and the Fokker-Planck equation, and by domain assumptions about locality, Lipschitz dynamics, and the existence of a stationary density. The stationarity premise is the most fragile burden and is not met by the paper's own solution.

free parameters (2)
  • sigma_w
    Standard deviation of Gaussian weight initialization in the ReLU kernel. It is a conventional hyperparameter, not fitted to data, but the derived diffusion coefficient depends on it.
  • sigma_b
    Standard deviation of bias initialization. The paper claims sigma_b > 0 makes the stationary distribution normalizable, which is the key error in Eq. (3). It is not a fitted value.
assumptions (5)
  • domain assumption Reinforcement learning can be reduced to a reward-free Markov process for studying untrained policies; no intrinsic or exploration-dependent rewards are needed.
    Section 2.1 states the paper focuses on the reward-free Markov process that governs data collection. This is reasonable for initialization behavior, but it deliberately ignores how reward signals alter later exploration.
  • domain assumption The environment dynamics are local and Lipschitz (Assumption 1), and the main lemma restricts to Ls = 1.
    Section 2.2 and Lemma 1. The bound delta on successor-state distance and Lipschitz constants La and Ls are non-trivial restrictions. The paper acknowledges that Ls > 1 leads to exponential separation, so the ballistic claim is only proven for a narrow dynamical regime.
  • standard math A feedforward network with i.i.d. zero-mean weights and biases converges to a Gaussian Process in the infinite-width limit.
    Section 2.3, Theorem 4, attributed to Neal. The theorem requires variance scaling and activation conditions that are not fully restated, but the use of the known ReLU kernel is standard.
  • domain assumption Per-step policy resampling with small action size admits a continuous-time Fokker-Planck diffusion with covariance Sigma(s) = sigma_b^2 + (sigma_w^2/pi) ||s||^2.
    Section 2.4, Eq. (2). Treating the discrete Markov chain as an Ito diffusion requires a small-action continuous-time limit and ignores higher-order corrections; the paper provides no quantitative justification for the limit.
  • ad hoc to paper A stationary, normalizable probability density exists for the Fokker-Planck equation.
    Section 2.4, Eq. (3). The paper assumes stationarity and claims normalizability for sigma_b > 0. This premise fails for the proposed solution in all dimensions, making it the most fragile assumption in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploration Behavior of Untrained Policies." pith.science (2026). https://pith.science/paper/6NWEVKWM

@misc{pith2026250622566,
  author       = {Pith},
  title        = {Pith review of: Exploration Behavior of Untrained Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NWEVKWM}},
  note         = {Machine review of arXiv:2506.22566}
}
read the original abstract

Exploration remains a fundamental challenge in reinforcement learning (RL), particularly in environments with sparse or adversarial reward structures. In this work, we study how the architecture of deep neural policies implicitly shapes exploration before training. We theoretically and empirically demonstrate strategies for generating ballistic or diffusive trajectories from untrained policies in a toy model. Using the theory of infinite-width networks and a continuous-time limit, we show that untrained policies return correlated actions and result in non-trivial state-visitation distributions. We discuss the distributions of the corresponding trajectories for a standard architecture, revealing insights into inductive biases for tackling exploration. Our results establish a theoretical and experimental framework for using policy initialization as a design tool to understand exploration behavior in early training.

Figures

Figures reproduced from arXiv: 2506.22566 by the authors.

Figure 1
Figure 1. Though trajectories can change direction (as observed in the plot on the right), on short timescales, the trajectories can be well-approximated with a dominant linear drift [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Exploration through a narrow hall￾way. Ballistic trajectories from a fixed MLP struggle to pass through the barrier; stepwise re￾initialization produces overly-diffusive motion, while a hybrid strategy leverages both behaviors for efficient exploration. fat-tailed distributions that will eventually (but slowly) explore across the barrier. A hybrid switching strategy, where a fixed MLP is used for the first n steps b… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

  1. [1]

    Lipbab: Computing exact lipschitz constant of relu networks

    Aritra Bhowmick, Meenakshi D’Souza, and G Srinivasa Raghavan. Lipbab: Computing exact lipschitz constant of relu networks. In Artificial Neural Networks and Machine Learning– ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slo- vakia, September 14–17, 2021, Proceedings, Part IV 30, pages 151–162. Springer, 2021

    work page 2021

  2. [2]

    Radial basis functions

    Martin Dietrich Buhmann. Radial basis functions. Acta numerica, 9:1–38, 2000

    work page 2000

  3. [3]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    arXiv 2018

  4. [4]

    Rainbow: Combining im- provements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 32, 2018

    work page 2018

  5. [5]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018. 6 EXPLORATION BEHAVIOR OF UNTRAINED POLICIES

    work page 2018

  6. [6]

    Exploration in deep reinforce- ment learning: A survey

    Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforce- ment learning: A survey. Information Fusion, 85:1–22, 2022

    work page 2022

  7. [7]

    Lipschitz constant estimation of Neural Networks via sparse polynomial optimization

    Fabian Latorre, Paul Rolland, and V olkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004.08688, 2020

    work page Pith review arXiv 2004

  8. [8]

    Flipping coins to estimate pseudocounts for exploration in reinforcement learning

    Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pages 22594–22613. PMLR, 2023

    work page 2023

Show all 21 references
  1. [9]

    Periodic activation functions induce stationar- ity

    Lassi Meronen, Martin Trapp, and Arno Solin. Periodic activation functions induce stationar- ity. Advances in Neural Information Processing Systems, 34:1673–1685, 2021

    2021

  2. [10]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  3. [11]

    Bayesian learning for neural networks , volume 118

    Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012

    2012

  4. [12]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. PMLR, 2022

    2022

  5. [13]

    Deep reinforcement learning with plasticity injection

    Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Infor- mation...

    2023

  6. [14]

    Deep exploration via bootstrapped dqn

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016

    2016

  7. [15]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889–

  8. [16]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  9. [17]

    On bonus-based exploration methods in the arcade learning environment.arXiv preprint arXiv:2109.11052, 2021

    Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Marc G Belle- mare. On bonus-based exploration methods in the arcade learning environment.arXiv preprint arXiv:2109.11052, 2021

    2021 arXiv

  10. [18]

    Lipschitz regularity of deep neural networks: analysis and efficient estimation

    Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018

    2018

  11. [19]

    Q-learning

    Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279–292, 1992. 7 EXPLORATION BEHAVIOR OF UNTRAINED POLICIES

    1992

  12. [20]

    Simple statistical gradient-following algorithms for connectionist rein- forcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229–256, 1992

    1992

  13. [21]

    Neural architecture search with reinforcement learning

    Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. 8 EXPLORATION BEHAVIOR OF UNTRAINED POLICIES Appendix We first provide a useful inequality for obtaining our main result, Lemma 1, first focusing on the ca...

    2016 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.