REVIEW 4 major objections 4 minor 10 references
In off-policy RL, negative-advantage updates drive exponentially exploding gradients, and only hard filtering of low-reward samples — not soft re-weighting — can stop the collapse.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 01:06 UTC pith:E47LAHBF
load-bearing objection The practical warning about negative off-policy updates is worth hearing, but the central theorem doesn't survive its own appendix and the manuscript is unfinished. the 4 major comments →
Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is the Divergence Theory of Repulsive Optimization: for a Gaussian policy the score is ((a−µ)/σ², (a−µ)²/σ² − 1), so repulsion acts through squared displacement. Negative-advantage updates expand while positive updates contract, and shrinking variance makes the gradient at any fixed negative sample unbounded; hence every soft-weighting scheme either clones noise or collapses variance. The constructive half shows the cure: the closed-form solution of an optimistic DRO problem is a hard-thresholding distribution Q* ∝ P_data·I(R ≥ R_{1−κ}) (top-κ truncation), realized by DRPO with a variance-guided curriculum that survives extreme-noise regimes.
What carries the argument
Two pieces carry the argument. The first is the Gaussian policy's joint score function ∇_θ log π = ((a−µ)/σ², (a−µ)²/σ² − 1) in parameter space θ = (µ, ξ), with a positive-definite Hessian; it turns each advantage sign into a linearized recurrence — contraction I − ηÂH for positive samples, expansion I + η|Â|H with spectral radius > 1 for negative ones — and the squared term in the ξ-coordinate is what yields gradient growth at twice the displacement rate. The second is the optimistic DRO program with density-ratio bound w ≤ 1/κ, whose Lagrangian solution is the top-κ hard-thresholding distribution, dualized as an optimistic CVaR with threshold at the (1−κ)-quantile of rewards.
Load-bearing premise
The load-bearing step is the proof's linearization of the negative-sample update near a fixed anchor, which turns a gradient that genuinely grows with the square of the displacement (the log-variance term (a−µ)²/σ² − 1) into a fixed linear pull; if that approximation is not locally faithful, the predicted exponential explosion rate is unsupported.
What would settle it
Run the exact, unlinearized gradient dynamics of a one-dimensional Gaussian policy (µ, ξ) on a single persistent negative-advantage sample and record the gradient norm over time: the theory predicts growth of order ρ^{2t} with ρ > 1, so polynomial or bounded growth would refute the rate claim. A second, independent test: a soft-weighting scheme whose negative weight decays with remoteness, such as w ∝ exp(−α∥a−µ∥²/σ²), run on a Zipfian dataset — if it stabilizes training, the claim that only hard filtering can prevent divergence is refuted.
If this is right
- Soft-weighting methods (AWR, AsymRe) cannot in principle prevent divergence on heavy-tailed data: the aggregate noise force survives any positive weight, and driving weights to zero collapses the effective sample size.
- Hard filtering is the exact maximizer of the optimistic DRO objective: retaining the top-κ fraction of samples by reward has a closed-form proof, giving top-κ truncation a mathematical foundation rather than heuristic status.
- The failure is structural and worsens with confidence: as positive samples shrink policy variance, the gradient at any fixed negative sample grows without bound, so the repulsion-to-attraction gradient ratio diverges asymptotically.
- On-policy methods escape the curse because their expected squared gradient norm depends only on the covariance, not on where the mean sits, so no location-dependent accumulation occurs.
- No fixed filtering ratio is universally optimal: the variance-guided schedule (κ scaled with reward variance) is needed because aggressive filtering helps under policy noise but backfires under reward noise.
Where Pith is reading between the lines
- If the divergence mechanism is as general as the analysis implies, the same quadratic-repulsion pathology should appear in any continuous-action off-policy method with a learnable variance or temperature — behavior cloning from logs, robotics, and continuous control are direct places to look; the signature would be exploding log-variance gradients exactly when the policy becomes confident.
- The hard-filtering optimality result, stated for scalar rewards, transfers in principle to any scalar quality signal: top-quantile selection of preference scores in alignment-style training, or of learned Q-values (the paper's DRPO-Q variant already sketches the latter), should inherit the same denoising guarantee.
- A corollary the paper leaves implicit: because the policy's variance sets both how much signal it can absorb and how violently the tail reacts, adaptive variance or temperature scheduling is not a nicety but a stability requirement in heavy-tailed regimes.
- The paper's own sensitivity sweep suggests a boundary on the cure: under high reward noise, the top-κ set is dominated by lucky outliers, so filtering on raw reward can overfit — a testable hybrid would filter on a temporally smoothed or variance-normalized reward instead.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript claims to establish a 'Divergence Theory of Repulsive Optimization,' asserting that negative-advantage off-policy updates in Gaussian policies inevitably cause exponential intensity explosion and that soft-weighting methods cannot prevent this. It then proposes DRPO, an Optimistic Distributionally Robust Optimization (DRO) method whose closed-form solution is said to be hard filtering, with a variance-guided adaptive Top-P mechanism. The paper also reports experiments on a custom synthetic recommendation simulator. However, the central proof relies on an invalid linearization of the Gaussian score, the main algorithm is inconsistent with the text, and the manuscript merges two different papers (one on remoteness-aware control, one on DRO hard filtering).
Significance. If the Divergence Theory were correct, it would provide a unified explanation for instability in off-policy RL and would elevate hard filtering from a heuristic to a mathematical necessity—a substantial contribution. The DRO derivation in Theorem 4.1 is standard and correct as a CVaR dual, but it does not by itself establish the necessity of hard filtering; that necessity claim depends entirely on the Divergence Theory. The empirical results, though suggestive, are based on a single synthetic environment and omit key baselines. Given the central proof is invalid and the manuscript is internally inconsistent, the claimed contribution is not currently established.
major comments (4)
- [Appendix B, Theorem 3.2, Corollary 3.3] The proof of the central exponential-divergence claim replaces the negative-sample update with the linearization θ_{t+1} = θ_t − η C_base H(θ)(θ* − θ_t). This assumes ∇_θ log π(θ) ≈ −H(θ)(θ − θ*) near an anchor θ*. For the Gaussian policy in Eq. (1), the ξ-component is ((a−μ)^2/σ^2) − 1, which is not linear in θ, and no anchor θ* makes the score vanish (setting μ = a leaves the ξ-component at −1). Thus the spectral-radius argument and the O(ρ^{2t}) growth are unsupported. Corollary 3.3, Theorem 3.4, and the conclusion that soft-weighting cannot prevent divergence all rest on this invalid linearization.
- [Section 4.3 vs. Algorithm 1] The variance-guided mechanism is described by Eq. (15) as κ_t ∝ Var(R_t), but Algorithm 1 implements a different mechanism: Top-P is updated multiplicatively (±2%) based on comparing σ_subset to λ·σ_batch. These are not equivalent, and the ablation study in Table 2 attributes gains to 'Adaptive' as implemented in Algorithm 1, not to Eq. (15). The actual variance-guided curriculum is therefore not reproducibly specified in the text.
- [Title, Abstract, and Full Text] The arXiv title and first abstract describe 'Remoteness-Aware Control of Negative Off-Policy Updates,' with a method that attenuates the remote tail while leaving near-field updates unchanged. The full-text title and second abstract describe 'Optimistic DRO / Hard Filtering.' These are two different methods. The manuscript appears to be an incoherent merge of two separate papers; as submitted, it does not present a single, well-defined contribution.
- [Section 5.2, Table 1] The text claims that standard APG suffers catastrophic failure (Reward ≈ 0.01), providing empirical validation of the Divergence Theory, but APG is not listed in Table 1. No standard deviations, seeds, or confidence intervals are reported anywhere. The empirical evidence is therefore weaker than claimed, and the 'SOTA' conclusion on a custom simulator cannot be independently assessed.
minor comments (4)
- [References] Placeholder citations appear ('Lastname et al., 2024'; 'arXiv:2400.xxxxx'), and the OneRec citation is marked 'Please verify.' These need to be fixed.
- [Section 5 headings] Section 5 contains duplicated headings: '5.4 RQ3' and '5.5 RQ3.' This is a formatting error that should be corrected.
- [Appendix A] The Hessian H(θ) in Eq. (19) is the expected Hessian over the policy distribution, but Appendix B applies it to a single fixed sample. The distinction should be made explicit, since the sample Hessian has different entries (e.g., 2(a−μ)^2/σ^2 for the ξξ term).
- [Section 4.1] The phrase 'valid sub-distribution of the behavior policy' is imprecise; Q is a probability distribution, not a sub-distribution. This should be clarified.
Circularity Check
No significant circularity: the hard-filtering result is a direct CVaR/LP derivation and the divergence theory follows from an explicit (if questionable) linearization, not from the target conclusion.
full rationale
The paper's central derivations do not reduce to their own inputs. Theorem 4.1 (Optimality of Hard Filtering) is proven in Appendix F.1 by reformulating the DRO problem as a linear program over density ratios w with 0≤w≤1/κ, forming the Lagrangian, and showing the optimum is w* = (1/κ)I(R≥λ) with λ the (1−κ)-quantile; this is a standard optimistic-CVaR/LP duality result and is independent of the empirical success of the method. The Divergence Theory (Theorem 3.2, Corollary 3.3) is derived in Appendix B from the Gaussian score (Proposition 3.1) plus an explicit local linearization θ_{t+1}=θ_t−η C_base H(θ)(θ*−θ_t). Whether that linearization is valid for the log-variance component is a substantive mathematical concern, but it is a correctness/assumption issue, not circularity: the exponential expansion is a consequence of the assumed recurrence, not an input renamed as a prediction, and no parameter is fitted to the target divergence. The paper cites external standard results (Amari 1998; Rockafellar & Uryasev 2000; Sutton & Barto) and does not rely on a self-citation chain; the placeholder 'Lastname, F. et al. Asymre' and the unverified OneRec citation are manuscript defects but are not load-bearing for the theory. Empirical claims are tested on an external RecSim simulator with noise-injection and ablations. Thus there is no circular step; the score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- Top-P initial value p0 =
0.5
- Target variance ratio λ =
0.5
- Top-P update rates γ_up/γ_down =
1.02 / 0.98
- Learning rate η =
3e-4
axioms (4)
- domain assumption The policy is Gaussian with scalar variance σ²; score function has the form in Eq. (3)
- domain assumption The reward distribution is Zipfian / heavily skewed, so the majority of samples have negative advantages
- ad hoc to paper The negative-sample update can be linearized as θ_{t+1} = θ_t - η C_base H(θ)(θ* - θ_t)
- domain assumption Positive training drives variance to zero (∥a_pos − µ∥ < σ eventually)
Cite this review
Pith. "Pith review of Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates." pith.science (2026). https://pith.science/paper/E47LAHBF
@misc{pith2026260210430,
author = {Pith},
title = {Pith review of: Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates},
year = {2026},
howpublished = {\url{https://pith.science/paper/E47LAHBF}},
note = {Machine review of arXiv:2602.10430}
}
read the original abstract
Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.
Figures
Reference graph
Works this paper leans on
-
[3]
Offline reinforce- ment learning with implicit q-learning.arXiv preprint arXiv:2110.06169,
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning.arXiv preprint arXiv:2110.06169,
-
[4]
Crafting papers on machine learning
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,
2000
-
[5]
distance
H.1. Embedding Topology and Retrieval High-Dimensional Manifold.Standard datasets often treat items as independent discrete tokens. In contrast, RecSim models the item pool I and users U in a continuous embedding space Rd. While the dimension d is highly customizable to mimic various system architectures, we adopt the common industrial paradigm of d= 128 ...
2000
-
[7]
Peng, X. B., Kumar, A., et al. Advantage-weighted re- gression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177,
Pith/arXiv arXiv 1910
-
[2000]
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2010]
Rahimian, H. and Mehrotra, S. Distributionally robust op- timization: A review.arXiv preprint arXiv:1908.05659,
Pith/arXiv arXiv 1908
-
[2015]
Off-policy deep reinforcement learning without exploration
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. InInter- national Conference on Machine Learning (ICML), pp. 2052–2062,
2052
-
[2019]
Deep reinforcement learning in large discrete action spaces
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Kelcey, T., and Trivedi, R. Deep reinforcement learning in large discrete action spaces. InarXiv preprint arXiv:1512.07679,
-
[2020]
Accelerating online reinforcement learning with offline datasets
Nair, A., Gupta, A., Dalal, M., and Levine, S. Accelerating online reinforcement learning with offline datasets. In arXiv preprint arXiv:2006.09359,
Pith/arXiv arXiv 2006
-
[2022]
Offline re- inforcement learning: Tutorial, review, and perspectives
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline re- inforcement learning: Tutorial, review, and perspectives. arXiv preprint arXiv:2005.01643,
Pith/arXiv arXiv 2005
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.