Differentially Private Policy Gradient

Alexandre Rio; Igor Colin; Merwan Barlier

REVIEW 3 major objections 5 minor 39 references

Differentially Private Policy Gradient

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Differentially private policy gradients can keep non-private performance by recasting DP as a trust-region choice.

desk verdict A promising first step toward practical DP policy gradients, but the central privacy claim for the deep RL experiments is unsupported as written. read the letter →

arxiv 2501.19080 v1 pith:MFOONFVY submitted 2025-01-31 cs.LG

Alexandre Rio , Merwan Barlier , Igor Colin This is my paper · ORCID

classification cs.LG

keywords differentialprivacypolicygradienttrustregionreinforcementlearningtrajectory-levelDP-SGDPPORLHF

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that adding differential privacy to policy gradient reinforcement learning does not have to sacrifice the methods' performance or theoretical properties. The proposed route is to treat privacy noise and gradient clipping as a choice of trust-region size: with a correctly chosen clipping norm, the noisy update stays inside the trust region with high probability, and the gap to the non-private update can be bounded. The authors then build a practical PPO-style algorithm that is differentially private at the trajectory level and test it on control tasks, a simulated medication-dosing task, and an RLHF sentiment task. On these benchmarks, the private algorithm keeps most of the non-private performance for realistic privacy budgets, which would make it the first DP policy gradient method that works on deep RL and RLHF-style problems.

What carries the argument

The object that carries the argument is the noisy gradient update rule $\Delta\theta = \eta(\bar{S}\cdot g + \xi)$, with $\xi \sim \mathcal{N}(0, z^2 S^2 I_d)$. The paper observes that the clipping norm $S$ is exactly the knob that trust-region methods control, so the DP requirement becomes a constraint on the size of the update. The trust-region size $\Delta\theta^\top\Delta\theta/2$ is shown to follow a scaled non-central chi-square distribution, and its KL/Fisher counterpart $\tfrac{1}{2}\Delta\theta^\top F\Delta\theta$ a generalized non-central chi-square; the quantiles of these distributions yield closed-form upper bounds for $S$. The practical algorithm then uses per-user clipped PPO updates aggregated with Gaussian noise, which gives a formal $(\epsilon,\delta)$-trajectory-level DP guarantee via the Gaussian mechanism.

What would settle it

One concrete test is to measure, on a deep RL task, the actual fraction of DPPG updates that leave the trust region at the $S$ values used in the paper; if that fraction is much larger than the claimed $1-\beta$, the reduction of DP to trust-region computation fails for the practical algorithm. Alternatively, a trajectory-level membership inference attack that succeeds against DPPG-trained policies at the claimed $(\epsilon,\delta)$ budgets would directly contradict the privacy guarantee.

Watch

Extended reading notes

Core claim

The central claim is that the introduction of differential privacy into policy gradient methods can be reduced to computing appropriate trust regions, i.e., setting the clipping norm $S$ so that the noisy update $\eta(\bar{g} + \xi)$ stays inside a region where a non-private update would also be safe. For the standard policy gradient with an L2 trust region, the squared update norm follows a scaled non-central chi-square distribution; for TRPO's KL trust region, the approximate KL is a generalized non-central chi-square. From these distributions the paper derives upper bounds on $S$ that guarantee, with probability at least $1-\beta$, that the update stays within the trust region, and a separate bound that controls the objective gap between the private and non-private updates. The practical algorithm, DPPG, aggregates per-user clipped PPO updates and applies the Gaussian mechanism at the trajectory level, with a formal $(\epsilon,\delta)$-trajectory-level DP guarantee; empirically it retains most of PPO's performance at $\epsilon$ roughly between 1 and 10 on tasks that previous DP RL methods could not address.

Load-bearing premise

The theoretical guarantees are proven for a single update using the true gradient and, for the KL bound, a Fisher matrix estimated on public data, while the deployed algorithm uses estimated advantages, several local epochs, and a hand-tuned clipping norm, so the transfer of the trust-region and objective-gap guarantees to the full algorithm is not established.

Editorial extensions

If this is right

If DPPG works as claimed, differential privacy becomes applicable to deep RL and RLHF problems with formal trajectory-level privacy, whereas prior DP RL methods were limited to tabular and linear MDPs.
The trust-region equivalence gives a principled way to set the clipping norm $S$ from privacy parameters (noise multiplier, dimension, desired confidence) instead of treating it as a free hyperparameter.
Because user data is used in a single on-policy update and then discarded, the privacy budget does not degrade with the number of iterations, as formalized by Theorem 4.3.
The experiments indicate that privacy is nearly free for $\epsilon$ around 1 to 10 on control and dosing tasks, with the cost growing sharply for $\epsilon \ll 1$, especially in high-dimensional continuous control.
The RLHF sentiment experiment suggests that private fine-tuning of a language model by policy gradients is plausible at strong noise scales, although at some final-reward cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We note an implicit diagnostic: if the trust-region reduction transfers, the empirical fraction of DPPG updates that leave the trust region should be close to the claimed $1-\beta$; tracking $\|\Delta\theta\|$ or the consecutive-policy KL during training would expose where the single-step analysis stops applying.
The trajectory-level DP definition the paper introduces for the released policy could serve as a natural privacy target for RLHF systems, where the released model (not just the recommended actions) is what an attacker can query, but the paper does not validate this against real membership-inference attacks.
The paper leaves off-policy reuse and multi-pass training out of scope; extending DPPG to experience replay or multiple local epochs shared across users would require a new composition argument, since the current guarantee relies on parallel composition across disjoint user data.
A testable extension is to combine the per-user aggregation with privacy amplification by subsampling: for large $K$, the effective $\epsilon$ might be lower than the paper's direct Gaussian-mechanism accounting, which would strengthen the practical privacy claims.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something genuinely new: it connects DP gradient clipping to trust-region size and shows empirically that a clipped, noisy policy gradient can still learn on deep RL and RLHF-style tasks. The Riverswim experiments, where S is set from the theory, are a real validation of the trust-region idea. The writing is clear and the related work is fair. That is the credit it earns.

The soft spots are serious, and the second-pass stress-test note lands. Appendix C.2.1 confirms that the value function used for advantage estimation is trained on private data and not privatized unless it shares parameters with the policy. In the practical algorithm, the advantage estimates for all K users in an iteration depend on that shared value function. Add or remove one user's trajectory and the value function changes, which shifts every local update. The sensitivity of the averaged update is then not S/K; it can be on the order of S or worse. The Gaussian noise scale zS/K is therefore miscalibrated, and the claimed trajectory-level DP guarantee for Algorithm 2 does not follow. This is not a minor technicality: the title and abstract promise formal DP for the deep RL results, and that promise is unsupported.

Two other weaknesses, in proportion. First, the privacy budgets for the hard benchmarks are weak. MuJoCo uses z=0.05 and z=0.1, and RLHF uses z=0.01, which the paper does not convert to epsilon. Using their own formulas, those correspond to epsilons in the tens to thousands — not "realistic privacy budgets" in any meaningful sense. Second, the theory in Section 4 analyzes a single update with the true gradient, while the deployed algorithm uses estimated advantages, multiple local epochs, and a hand-tuned S. The claim that "DP can be reduced to trust regions without sacrificing theoretical properties" is only loosely connected to what is actually run.

Who is this for? Researchers working on private RL. They should read the trust-region analysis and the Riverswim results, but they should not cite the deep RL results as evidence of private learning until the value-function coupling is addressed. The paper deserves a serious referee because the direction is important and the core idea is sound — but it needs a major revision, with the privacy accounting fixed and the empirical claims recalibrated.

Referee Report

3 major / 5 minor

Summary. The paper introduces DPPG, a differentially private policy gradient method that clips per-user local updates, aggregates them over K users, and adds Gaussian noise calibrated to a claimed S/K sensitivity. The authors derive theoretical bounds (Propositions 4.4-4.7) on the clipping norm S that keep the noisy update inside an L2 or KL trust region with high probability and control the objective gap relative to the non-private update. They then present a practical PPO-based algorithm (Algorithms 2-3) claimed to satisfy trajectory-level DP (Theorem 4.3 and Section 5) and evaluate it on Riverswim, CartPole, Acrobot, HalfCheetah, Hopper, a diabetes dosing simulator, and an RLHF sentiment task. The central claim is that for realistic privacy budgets the performance loss is limited, and that the introduction of differential privacy can be reduced to the computation of appropriate trust regions.

Significance. If the privacy analysis were correct, DPPG would be a significant contribution: it is among the first attempts to apply formal trajectory-level DP to deep RL and RLHF, and it correctly identifies a connection between DP clipping and trust regions, backing it with a clean chi-square analysis. The Riverswim experiments validate the theory-computed S against a DP RL baseline, which is a strength. The paper also correctly avoids the common mistake of applying the epsilon<1 Gaussian mechanism formula for epsilon>=1 by using the improved Gaussian mechanism of Zhao et al. (2019). However, the validity of the formal guarantee for the practical algorithm is undermined by the coupling of all users' updates through the non-private value function, and the deep RL/RLHF experiments are run at privacy budgets that are far from the 'realistic' range claimed. The theoretical trust-region results apply only to an idealized true-gradient single-step update, so the paper's main conceptual claim is not fully established for the deployed algorithm.

major comments (3)

[§5 and App. C.2.1] The privacy accounting in Section 5 assumes that each clipped local update \bar{g}_u is a function of user u's trajectory alone, so the mean of K updates has sensitivity S/K and noise zS/K suffices for (epsilon,delta)-TDP. In the PPO implementation, however, the advantage estimates \hat{A}^u fed to Algorithm 3 are computed with a value function V_psi that is trained on the same private trajectories; Appendix C.2.1 states that the value function is trained on private data and is not privatized unless it shares parameters with the policy. Replacing or removing one trajectory changes V_psi, which changes the advantage estimates and therefore the local updates of all K users in that iteration. The sensitivity of the aggregated update is then at least ||g_u - g_u'||/K plus contributions from the other K-1 users, each of which can change by up to 2S; in the worst case the sensitivity is on the order of S, not S/K, and it is unbounded unless a Lipschitz property of the value-function training is established. Consequently, the noise scale zS/K does not provide the claimed (epsilon,delta)-TDP guarantee for Algorithm 2, and the application of Theorem 4.3 to the practical algorithm is not justified. The value-function updates must be privatized, or a new sensitivity analysis covering the value function must be supplied.
[§6, Table 1] The central claim in the abstract that 'for realistic privacy budgets, our approach shows limited impact' is not supported for the deep RL and RLHF experiments because the corresponding privacy budgets are not reported. Table 1 gives z=0.05 for HalfCheetah and Hopper, and the RLHF experiment uses z=0.01, but no epsilon values are provided. Using the authors' own conversion in Appendix B.1 for, say, delta=10^{-5}, z=0.05 yields epsilon approx 290 and z=0.01 yields epsilon approx 5400, which are orders of magnitude weaker than the epsilon in [1,10] range the paper itself cites as realistic. The paper should report epsilon for every experiment and either run the deep RL and RLHF tasks in the realistic epsilon range or substantially temper the central claim.
[§4.2 vs. §5] Propositions 4.4-4.7 establish trust-region containment and objective-gap bounds for a single update of the form theta += eta(\bar{g}+xi) with the true policy gradient g. The practical DPPG (Algorithm 2 with Algorithm 3) instead uses advantage estimates from a learned value function, performs multiple local epochs, and sets S as a tunable hyperparameter; Section 5 explicitly says 'We treat S as a tunable hyperparameter, hence avoiding the computation of the upper bounds derived in Section 4'. Therefore the paper's statement that 'the introduction of Differential Privacy can be reduced to the computation of appropriate trust regions' is not demonstrated for the algorithm that is actually evaluated. The authors should either extend the analysis to the estimated-advantage, multi-epoch setting or explicitly restrict the theoretical claim to the idealized update and clearly delineate the scope of the guarantees.

minor comments (5)

[Definition 4.2] The definition of trajectory-level DP uses the notation M_{-u}(D_M) in the probability inequality but then refers to M(D_M) as the released policy; this should be M(D_M) and M(D'_M) to be consistent with the definition of standard DP on a single trajectory.
[Algorithm 3] The pseudocode for the PPO local update clips with 'Clip(θ − θ0; C)' while the input specification names the clipping norm S; the symbol should be consistent (presumably S).
[Theorem 4.3 proof] In the proof of Theorem 4.3, 'cannot colide' should read 'cannot collude'.
[Section 4.2.1] The sentence 'setting S small enough that (5) and either (3 or 4) hold' is ambiguous because Eq. (3) is a distributional characterization, not a bound; the intended meaning is 'either (4) or the quantile bound from Proposition 4.4'.
[Appendix A.1.1] Equation (13) appears to have a typo: the second variance term is written as '(z^2S^2)^2 · 1^T F \bar{g}', but the derivation in the preceding lines indicates it should be the sum of squares of the components of F\bar{g}; the correct expression appears in Eq. (14).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical bounds are derived from standard chi-square/Cantelli inequalities and the privacy accounting is a direct Gaussian-mechanism application; no self-citations are load-bearing.

full rationale

The paper's claimed derivation chain is self-contained. Section 4.2.1 and 4.2.2 start from the noisy update Delta_theta = eta(bar{S}g + xi) and derive, via a scaled non-central chi-square distribution (Prop. 4.4), Markov/Cantelli bounds (Props. 4.5-4.6), and a generalized chi-square bound with the Fisher matrix (Prop. 4.7), explicit sufficient conditions on the clipping norm S for the update to remain in a trust region and for the one-step objective gap to be controlled. These are conditional probabilistic bounds, not equivalences to the conclusion by construction, and no parameter is fitted to data and then reported as a prediction. The practical Algorithm 2 explicitly treats S as a tunable hyperparameter rather than using the theory as a fitted predictor, and its per-step privacy guarantee is a direct application of the Gaussian mechanism with sensitivity S/K, with epsilon computed from z using the external mechanism of Zhao et al. (2019). The reference list contains no self-citations. The 'reduction of DP to trust regions' is an interpretive framing, not a circular input: the trust-region containment and objective-gap statements are proved, not assumed. The correctness caveat raised about the value function (Appendix C.2.1, where V_psi is trained on private data and not privatized unless it shares parameters with the policy, and footnote 1, where estimating F on private data is left to future work) is a gap in the sensitivity accounting for the deployed PPO variant, not a circular reduction of a result to its own input. Consequently no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The algorithm's practical performance depends on task-specific clipping norms and noise multipliers that are tuned rather than derived; the theory-derived S appears only in the Riverswim experiments. The remaining assumptions are standard RL and DP background. No new entities are introduced.

free parameters (3)

Clipping norm S = 0.05 (CartPole/Acrobot), 1.80 (MuJoCo), 0.08 (Dosing); theory-derived in Riverswim
The deep RL experiments tune S per task (Table 3), and Section 5 states S is a tunable hyperparameter. The central claim that S encodes a trust region is not exercised on these tasks.
Noise multiplier z = 1.0/3.0 (CartPole/Acrobot), 0.05/0.1 (MuJoCo), 0.1/0.5 (Dosing), 0.01 (RLHF)
Chosen per environment to control the privacy-utility tradeoff; epsilon is reported only for some tasks, and for RLHF z=0.01 corresponds to a very large epsilon.
Riverswim trust region and confidence hyperparameters = alpha=3.5, 1-beta=0.6
Chosen in Table 2 to set the theory-derived S for the tabular experiments; these are hand-picked values.

assumptions (5)

domain assumption Discounted infinite-horizon MDP with policy gradient theorem
Section 3.1; the objective and gradient estimates rely on standard RL formalism.
domain assumption KL divergence approximated by Fisher information quadratic form
Section 4.2.2, Eq. (6); the TRPO bound (Prop 4.7) inherits this standard approximation.
domain assumption Each user's trajectory is a record used once (on-policy), enabling parallel composition
Section 4.1 and Theorem 4.3; the privacy guarantee depends on this design.
standard math Gaussian mechanism with Zhao et al. (2019) extension for epsilon >= 1
Section 3.3 and Appendix B.1; converts noise multiplier z into the reported epsilon.
standard math Billboard lemma (Hsu et al. 2016; Vietri et al. 2020) for the JDP claim
Theorem 4.3 proof; standard result used as a black box.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Differentially Private Policy Gradient." pith.science (2026). https://pith.science/paper/MFOONFVY

@misc{pith2026250119080,
  author       = {Pith},
  title        = {Pith review of: Differentially Private Policy Gradient},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFOONFVY}},
  note         = {Machine review of arXiv:2501.19080}
}

read the original abstract

Motivated by the increasing deployment of reinforcement learning in the real world, involving a large consumption of personal data, we introduce a differentially private (DP) policy gradient algorithm. We show that, in this setting, the introduction of Differential Privacy can be reduced to the computation of appropriate trust regions, thus avoiding the sacrifice of theoretical properties of the DP-less methods. Therefore, we show that it is possible to find the right trade-off between privacy noise and trust-region size to obtain a performant differentially private policy gradient algorithm. We then outline its performance empirically on various benchmarks. Our results and the complexity of the tasks addressed represent a significant improvement over existing DP algorithms in online RL.

Figures

Figures reproduced from arXiv: 2501.19080 by the authors.

**Figure 2.** Asymptotic performance vs. privacy budget ϵ CARTPOLE (left) and ACROBOT (right) in log scale. ENV. PPO A2C DP POLICY GRADIENT CARTPOLE 493.2 ± 9.4 - z = 1.0 496.4 ± 6.6 z = 3.0 375.5 ± 86.7 ACROBOT −82.1 ± 5.1 - z = 1.0 −83.0 ± 4.4 z = 3.0 −89.8 ± 7.6 HALFCHEETAH 2105.5 ± 627.9 1003.3 ± 27.2 z = 0.05 1548.9 ± 399.1 z = 0.1 1328.4 ± 306.5 HOPPER 2208.2 ± 477.6 625.0 ± 106.8 z = 0.05 645.0 ± 176.4 z = 0.1 808.9 ± 321.… view at source ↗

**Figure 3.** plots the relationship between z and the total privacy budget ϵ spent by Algorithm 2. We can a break in the curves at ϵ = 1, corresponding to the switch between mechanisms M1 (right portion of the curves) and M2 (left portion of the curves). We can see that M2 provides less optimistic privacy budgets (larger ϵ) that M1 would if we (incorrectly) applied the latter for smaller values of z, although the difference is r… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: The Riverswim environment (taken from (Chowdhury & Zhou, 2021)). patients, and the RL agent controls the insulin uptake. Each trajectory corresponds to a full day of patient measures, and the state contains various health measures such as meal disturbance amounts, insu…

**Figure 5.** Figure 5: Cumulative regret on RIVERSWIM for ϵ = 1.0 (dashed line) and ϵ = 5.0 (solid line). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

**Figure 6.** Figure 6: Asymptotic performance vs. privacy budget ϵ CARTPOLE (left) and ACROBOT (right) for noise multipliers z ∈ {0.25, 0.5, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0}. 20 40 60 80 100 120 140 160 180 Batch 0.0 0.5 1.0 1.5 2.0 2.5 Average Batch Reward GPT-2 RLHF Fine-Tuning PPO DPPG z = 0…

**Figure 7.** Figure 7: Learning curves on the RLHF sentiment tuning task. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

**Figure 8.** Figure 8: Learning curves on CONTROL (top), MUJOCO (middle) and DOSING (bottom) environments. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

J., McMahan, H

Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , pp.\ 308--318. ACM , 2016. URL https://doi.org/10.1145/2976749.2978318

work page arXiv 2016
[3]

M., Crump, T., and Far, B

Afsar, M. M., Crump, T., and Far, B. H. Reinforcement learning based recommender systems: A survey. ACM Comput. Surv. , 55 0 (7): 0 145:1--145:38, 2023. doi:10.1145/3543846. URL https://doi.org/10.1145/3543846

doi:10.1145/3543846 2023
[4]

OpenAI Gym

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI Gym . CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540

arXiv 2016
[5]

B., Song, D., Erlingsson, \' U ., Oprea, A., and Raffel, C

Carlini, N., Tram \` e r, F., Wallace, E., Jagielski, M., Herbert - Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, \' U ., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021 , pp.\ 2633--2650. USENIX Association, 2021. URL https://w...

work page 2021
[6]

Chowdhury, S. R. and Zhou, X. Differentially Private Regret Minimization in Episodic Markov Decision Processes , December 2021. URL http://arxiv.org/abs/2112.10599. arXiv:2112.10599 [cs, math]

arXiv 2021
[7]

Privacy-constrained policies via mutual information regularized policy gradients

Cundy, C., Desai, R., and Ermon, S. Privacy-constrained policies via mutual information regularized policy gradients. In International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp.\ 2809--2817. PMLR , 2024. URL https://proceedings.mlr.press/v238/j-cundy24a.html

work page 2024
[8]

and Geisler, W

Das, A. and Geisler, W. S. Methods to integrate multinormals and compute classification measures, 2024. URL https://arxiv.org/abs/2012.14331

arXiv 2024

Show all 39 references

[9]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

2025 arXiv
[10]

Differential Privacy

Dwork, C. Differential Privacy . In Proceedings of ICALP, 2006. URL https://www.microsoft.com/en-us/research/publication/differential-privacy/

2006
[11]

Practical membership inference attacks against fine-tuned large language models via self-prompt calibration

Fu, W., Wang, H., Gao, C., Liu, G., Li, Y., and Jiang, T. Practical membership inference attacks against fine-tuned large language models via self-prompt calibration. CoRR, abs/2311.06062, 2023. URL https://doi.org/10.48550/arXiv.2311.06062

2023 doi arXiv
[12]

Membership inference attacks against temporally correlated data in deep reinforcement learning

Gomrokchi, M., Amin, S., Aboutalebi, H., Wong, A., and Precup, D. Membership inference attacks against temporally correlated data in deep reinforcement learning. IEEE Access , 11: 0 42796--42808, 2023. URL https://doi.org/10.1109/ACCESS.2023.3270860

2023
[13]

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, J...

2018
[14]

M., Xu, R., and Yang, H

Hambly, B. M., Xu, R., and Yang, H. Recent advances in reinforcement learning in finance. CoRR, abs/2112.04553, 2021. URL https://arxiv.org/abs/2112.04553

2021 arXiv
[15]

Hsu, J., Huang, Z., Roth, A., Roughgarden, T., and Wu, Z. S. Private matchings and allocations. SIAM J. Comput. , 45 0 (6): 0 1953--1984, 2016. URL https://doi.org/10.1137/15100271X

1953 doi
[16]

Kakade, S. M. A natural policy gradient. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada] , pp.\ 1531--1538. MIT Press, 2001

2001
[17]

A survey of reinforcement learning from human feedback

Kaufmann, T., Weng, P., Bengs, V., and H \" u llermeier, E. A survey of reinforcement learning from human feedback. CoRR, abs/2312.14925, 2023. URL https://doi.org/10.48550/arXiv.2312.14925

2023 doi
[18]

P., Hunt, J

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conferenc...

2016 arXiv
[19]

Deep reinforcement learning for personalized treatment recommendation

Liu, M., Shen, X., and Pan, W. Deep reinforcement learning for personalized treatment recommendation. Statistics in Medicine, 41, 06 2022

2022
[20]

B., Ramage, D., Talwar, K., and Zhang, L

McMahan, H. B., Ramage, D., Talwar, K., and Zhang, L. Learning differentially private recurrent language models. In 6th International Conference on Learning Representations, ICLR , 2018. URL https://openreview.net/forum?id=BJ0hF1Z0b

2018
[21]

P., Mirza, M., Graves, A., Lillicrap, T

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 1...

2016
[22]

Osband, I., Russo, D., and Roy, B. V. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013...

2013
[23]

How You Act Tells a Lot : Privacy - Leaking Attack on Deep Reinforcement Learning

Pan, X., Wang, W., Zhang, X., Li, B., Yi, J., and Song, D. How You Act Tells a Lot : Privacy - Leaking Attack on Deep Reinforcement Learning . Reinforcement Learning, 2019

2019
[24]

Paolella, M. S. Linear Models and Time-Series Analysis: Regression, ANOVA, ARMA and GARCH. Wiley Series in Probability and Statistics. John Wiley & Sons, Ltd., 2019. ISBN 9781119432036. URL https://onlinelibrary.wiley.com/doi/book/10.1002/9781119432036

2019 doi
[25]

B., Vassilvitskii, S., Chien, S., and Thakurta, A

Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H. B., Vassilvitskii, S., Chien, S., and Thakurta, A. How to DP -fy ML : A Practical Guide to Machine Learning with Differential Privacy , March 2023. URL http://arxiv.org/abs/2303.00654

2023 arXiv
[26]

How Private Is Your RL Policy ? An Inverse RL Based Analysis Framework

Prakash, K., Husain, F., Paruchuri, P., and Gujar, S. How Private Is Your RL Policy ? An Inverse RL Based Analysis Framework . Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (7): 0 8009--8016, June 2022. ISSN 2374-3468, 2159-5399. URL https://ojs.aaai.org/...

2022
[27]

and Wang, Y

Qiao, D. and Wang, Y. Near-optimal differentially private reinforcement learning. In Ruiz, F. J. R., Dy, J. G., and van de Meent, J. (eds.), International Conference on Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valencia, Spain, volume 206 o...

2023
[28]

Reinforcement learning for personalized medication dosing, 2019

Salas, N., Ferguson, B., and Zweig, J. Reinforcement learning for personalized medication dosing, 2019

2019
[29]

I., and Moritz, P

Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37, pp.\ 1889--1897. JMLR.org, 2015. URL http://proc...

2015
[30]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

2017 arXiv
[31]

Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. A. Deterministic policy gradient algorithms. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conf...

2014
[32]

S., McAllester, D

Sutton, R. S., McAllester, D. A., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999] , pp.\...

1999
[33]

Deep reinforcement learning for robotics: A survey of real-world successes

Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Mart \' n - Mart \' n, R., and Stone, P. Deep reinforcement learning for robotics: A survey of real-world successes. CoRR, abs/2408.03539, 2024. URL https://doi.org/10.48550/arXiv.2408.03539

2024 doi arXiv
[34]

Mujoco: A physics engine for model-based control

Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012

2012
[35]

Vietri, G., Balle, B., Krishnamurthy, A., and Wu, Z. S. Private reinforcement learning with PAC and regret guarantees. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Lear...

2020
[36]

and Hegde, N

Wang, B. and Hegde, N. Privacy-preserving q-learning with functional noise in continuous spaces. In Advances in Neural Information Processing Systems, volume 32, 2019

2019
[37]

Wang, L., Liu, J., Shao, H., Wang, W., Chen, R., Liu, Y., and Waslander, S. L. Efficient reinforcement learning for autonomous driving with parameterized skills and priors. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. URL https://doi....

2023 doi
[38]

Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8: 0 229--256, 1992. URL https://doi.org/10.1007/BF00992696

1992 doi
[39]

Reviewing and improving the gaussian mechanism for differential privacy

Zhao, J., Wang, T., Bai, T., Lam, K., Ren, X., Yang, X., Shi, S., Liu, Y., and Yu, H. Reviewing and improving the gaussian mechanism for differential privacy. CoRR, abs/1911.12060, 2019. URL http://arxiv.org/abs/1911.12060

1911 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

J., McMahan, H

Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , pp.\ 308--318. ACM , 2016. URL https://doi.org/10.1145/2976749.2978318

work page arXiv 2016

[3] [3]

M., Crump, T., and Far, B

Afsar, M. M., Crump, T., and Far, B. H. Reinforcement learning based recommender systems: A survey. ACM Comput. Surv. , 55 0 (7): 0 145:1--145:38, 2023. doi:10.1145/3543846. URL https://doi.org/10.1145/3543846

doi:10.1145/3543846 2023

[4] [4]

OpenAI Gym

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI Gym . CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540

arXiv 2016

[5] [5]

B., Song, D., Erlingsson, \' U ., Oprea, A., and Raffel, C

Carlini, N., Tram \` e r, F., Wallace, E., Jagielski, M., Herbert - Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, \' U ., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021 , pp.\ 2633--2650. USENIX Association, 2021. URL https://w...

work page 2021

[6] [6]

Chowdhury, S. R. and Zhou, X. Differentially Private Regret Minimization in Episodic Markov Decision Processes , December 2021. URL http://arxiv.org/abs/2112.10599. arXiv:2112.10599 [cs, math]

arXiv 2021

[7] [7]

Privacy-constrained policies via mutual information regularized policy gradients

Cundy, C., Desai, R., and Ermon, S. Privacy-constrained policies via mutual information regularized policy gradients. In International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp.\ 2809--2817. PMLR , 2024. URL https://proceedings.mlr.press/v238/j-cundy24a.html

work page 2024

[8] [8]

and Geisler, W

Das, A. and Geisler, W. S. Methods to integrate multinormals and compute classification measures, 2024. URL https://arxiv.org/abs/2012.14331

arXiv 2024