Pith. sign in

REVIEW 5 major objections 5 minor 41 references

What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims sparse-reward RL becomes tractable when reward functions have approximate low-rank plus sparse structure, and that a policy-aware matrix-completion estimator with confidence-gated abstention is how to exploit it.

desk verdict The paper's conceptual framing is new and worth a conversation, but the current manuscript's theoretical and empirical claims are not verifiable as written. read the letter →

arxiv 2509.03790 v2 pith:G56WN5Q6 submitted 2025-09-04 cs.LG cs.AI

Ibne Farabi Shihab , Sanjeda Akter , Anuj Sharma This is my paper · ORCID
classification cs.LGcs.AI
keywords sparse-rewardreinforcementlearningmatrixcompletionlow-rankstructuremissingnotatrandominversepropensityweightingconfidence-gatedabstentionsamplecomplexityreward
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the sample complexity of sparse-reward reinforcement learning is governed by structure in the reward function, not merely by the sparsity of feedback. It proposes Policy-Aware Matrix Completion (PAMC), which treats the unknown reward function as a matrix, assumes an approximate low-rank plus sparse decomposition, and recovers it from biased, policy-induced observations using inverse-propensity weighting. The paper proves this reduces the sample complexity from the general lower bound to polynomial, and ties completion quality directly to policy regret through a visitation-weighted bound. It also argues that a confidence-gated abstention mechanism lets the method degrade gracefully when assumptions fail. If these claims hold, sparse-reward RL shifts from a uniform exploration problem to a structured learning problem.

What carries the argument

The central object is the reward matrix R ∈ R^{|S|×|A|}, decomposed as R = L* + S* + E. The mechanism is policy-aware weighted matrix completion: inverse-propensity weights W_{sa} = 1/max(p_{sa}, ε_p) correct for missing-not-at-random sampling; a weighted nuclear norm plus weighted ℓ1 objective jointly recovers the low-rank component and the sparse component; and a calibrated confidence map C(s,a) gates which completed rewards the policy is allowed to trust, triggering abstention and an exploration bonus when intervals are wide.

What would settle it

Run PAMC on a small tabular MDP whose reward matrix is exactly low-rank, but use a behavior policy that assigns zero probability to one state-action pair in the optimal policy's support (so κ = 0). The theorem predicts the inverse-propensity weights diverge and recovery error is uncontrolled; measuring whether the estimated rewards stay accurate or the confidence gate abstains would directly test the positivity assumption.

Watch

Extended reading notes

Core claim

The paper establishes that the sparse-reward problem is not uniformly hard: under a general reward model, any learner needs Ω(|S||A|/p) samples, but if the true reward matrix can be written as a low-rank component plus a sparse component plus noise, recovery becomes polynomial. Concretely, Theorem 2 claims that a weighted robust principal-component pursuit estimator recovers the low-rank reward with error scaling as σ sqrt(r(|S|+|A|)/m_eff) plus a sparse-noise term, where m_eff is an inverse-propensity-weighted effective sample size. Theorem 3 then converts completion error into a policy regret bound under a visitation-weighted norm. The paper further claims that when assumptions weaken—high

Load-bearing premise

The guarantees require that the behavior policy has positive probability of visiting every state-action pair that the optimal policy would use; if exploration is deterministic or near-deterministic, the inverse-propensity weights become unstable and the recovery bounds collapse.

Editorial extensions

If this is right

  • If rewards are approximately low-rank plus sparse, reward recovery needs only polynomial samples in the effective sample size m_eff rather than Ω(|S||A|/p).
  • Policy regret is bounded by a visitation-weighted completion error, so improving completion under the optimal policy's reachable state-action distribution directly improves control performance.
  • When assumptions are violated, PAMC widens confidence intervals and abstains, degrading gracefully to exploration instead of catastrophically imposing wrong rewards.
  • The framework extends beyond rewards: low-rank successor features can be completed in the same way, and non-stationary rewards with bounded drift are handled by the same adaptive abstention mechanism.
  • On Atari-26, DM Control, MetaWorld, D4RL, and preference-based benchmarks, adding PAMC to strong baselines improves sample efficiency under compute-normalized comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim is right, environment designers could deliberately shape rewards to keep the reward matrix low-rank, making sparse feedback dramatically easier to learn from.
  • The confidence-gated abstention principle is not specific to matrix completion; attaching it to other learned predictors in the RL loop—dynamics models, value estimators, preference models—would be a natural test of the paper's general safety mechanism.
  • The theory suggests a practical pre-deployment check: estimate the singular-value spectrum of the reward matrix from a small uniform sample; if the effective rank is high, PAMC's guarantees do not apply and exploration baselines should remain in charge.
  • Because inverse-propensity weights become unstable when propensities are tiny, swapping them for a doubly robust estimator (which the paper mentions only in passing) could relax the positivity assumption and deserves a direct empirical comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes Policy-Aware Matrix Completion (PAMC) for sparse-reward RL. The reward matrix is modeled as L*+S*+E (low-rank + sparse + noise), and a weighted robust principal component analysis estimator is used with inverse-propensity weights to correct for MNAR sampling. The paper claims an impossibility lower bound for general rewards (Theorem 1), a recovery guarantee (Theorem 2), a visitation-weighted error-to-regret bound (Theorem 3), a sample-complexity bound (Theorem 4), and an IPW consistency lemma (Lemma 1). It also reports empirical gains on Atari-26, DM Control, MetaWorld MT50, D4RL, and preference-based RL, with confidence-based abstention. The main theoretical results are stated with proofs deferred to an appendix that is not present; the algorithm's propensity estimates are not covered by the oracle recovery theorem; and the empirical tables do not support the abstract's breadth.

Significance. If the recovery and regret guarantees held as stated, the paper would provide a useful bridge between matrix completion theory and sparse-reward RL, and the idea of confidence-gated abstention is a sensible safety mechanism. The paper also includes a candid Limitations section and stress-test diagnostics. However, the results are not verifiable as submitted: core proofs are missing, the main theorem assumes known sampling propensities while the algorithm estimates them, Theorem 1's proof sketch appears inconsistent, and the empirical support is narrower than claimed. The contribution is therefore not yet established.

major comments (5)
  1. [§3.2, Theorems 2 and 4] The main recovery and sample-complexity bounds are stated without proof. Appendix A contains only implementation details and Algorithm 1, not the promised derivations. The statement of Theorem 2 also has unclear dependencies: m_eff is defined as a sum of inverse propensities, so the displayed error appears to shrink as sampling probabilities become small, which is opposite to the usual missing-data intuition; a precise derivation is needed. Without the missing proofs, the central polynomial-sample-complexity claim cannot be evaluated.
  2. [§3.2, Theorem 1] The proof sketch does not support the stated lower bound Ω(|S||A|/(pε²)). The coupon-collector argument gives a 1/p dependence for identifying the nonzero reward pair, with no ε² mechanism. Moreover, the reward family R^{(i,j)} sets the nonzero reward to ε/(1−γ), while the theorem refers to functions differing by ε in expected return; under discounting these differ by a factor that is not specified. The impossibility result, which motivates the structural assumption, is therefore not established.
  3. [§3.2 and Algorithm 1 (lines 6–8)] Theorem 2 assumes policy-aware sampling probabilities p_sa are known constants in [p,_p_], truncated below by ε_p. In the actual PAMC loop, Algorithm 1 line 7 estimates propensities from a replay buffer filled by an evolving behavior policy. No error bound relates these estimated IPW weights to the oracle weights, and IPW is known to be sensitive to propensity misspecification. Consequently, the recovery guarantee does not apply to the implemented algorithm, and the regret bound in Theorem 3 inherits this gap.
  4. [§3.4, Theorem 3] As stated, the bound J(π*)−J(π_PAMC) ≤ C∥R̂−R∥_W + Õ(√log(1/δ)/n) is an error-transport inequality: if the completion error under the optimal policy's visitation is small, the policy is good. The weighted norm ∥·∥_W is not defined, n is not specified, and no account is taken of the gating/abstention mechanism or of optimization/exploration error in the policy update. This does not constitute the 'novel sample complexity' result claimed in the introduction.
  5. [§4.2, Table 2 and abstract] The abstract claims improvements over DrQ-v2, DreamerV3, Agent57, T-REX/D-REX, and PrefPPO across five benchmark families, but Table 2 reports only DrQ-v2, DreamerV3, and PrefPPO. Agent57 and T-REX/D-REX are not tabulated, and no results are shown for MetaWorld MT50 or D4RL despite textual claims. The single Atari 'HNS' aggregate lacks per-game breakdowns or significance tests. The empirical evidence is not sufficient for the breadth asserted.
minor comments (5)
  1. [§3.2] The weighted objective writes ∥L∥,W with an odd comma; this should be ∥L∥_*,W or another explicitly defined weighted norm.
  2. [Figure 6] The left panel's x-axis label reads 'Overlap Inverse (1/)' but the text says the quantity is 1/√κ. Please make the label explicit and consistent.
  3. [Table 1] Several formatting issues: baseline rows use '—' for SVD time/overhead, and the 'A100 Hours' column is concatenated in places (e.g., '≈18PAMC'). Clarify the units and align the entries.
  4. [§4.1] The protocol says 'all methods are granted the same number of environment interactions,' but Table 2 reports 'compute-normalized' comparisons. Please clarify whether the comparison controls environment steps, optimizer steps, wall-clock time, or all of these.
  5. [Reproducibility] The paper says code will be released; if it is available, provide a link or anonymized artifact in the manuscript. Also the statement 'all experiments were run with 5 random seeds' is not reflected in the reported confidence intervals for all tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 2 is an oracle matrix-completion bound and Theorem 3 is a standard performance-difference inequality; neither reduces to its inputs. The main vulnerability is an unverified link between estimated and oracle propensities, which is a correctness gap, not a circular derivation.

full rationale

The paper's central derivation chain is not circular. Theorem 2 assumes known policy-aware sampling probabilities p_sa in [p,p], truncated by epsilon_p, and states a weighted robust PCP error bound in terms of effective sample size m_eff, incoherence, and noise; this is a standard matrix-completion-style guarantee and no parameter is fitted to the reward matrix being predicted. Theorem 3 converts completion error into regret via a visitation-weighted norm; it is an inequality of performance-difference type, not a definitional restatement: the norm ||Rhat-R||_W is a sufficient statistic for the value gap, and the bound requires a proof through Bellman/performance-difference arguments. Lemma 1's positivity assumption (kappa>0) is an explicit condition, acknowledged in the Limitations section ('If the behavior policy is deterministic, weights can be unstable'), and the paper does not hide it. The remaining weakness is that Algorithm 1 line 7 estimates propensities from a replay buffer filled by a non-stationary policy, while Theorem 2's guarantee is for known p_sa; no bound accounts for propensity-estimation error. This is a gap between theory and implementation and a correctness risk, but it is not circular: the theorem is not defined in terms of the algorithm's output, nor is a fitted parameter renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. Empirical plots (e.g., error scaling with 1/sqrt(kappa)) check the predicted functional form rather than constructing the prediction from the fit. Therefore the paper's theoretical claims are independent of their conclusions, and the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. The 'confidence function' C(s,a) and 'abstention' mechanism are algorithmic constructs, not new independent variables. They have no falsifiable handle outside the algorithm itself, but they are not entities in the sense of new particles or forces. The free parameters and domain assumptions listed here are the hidden costs of the central claim.

free parameters (6)
  • rank hint r = not reported (illustrated range 1-120 in Fig. 7)
    Algorithm 1 takes a rank hint r as input; the recovery guarantee of Theorem 2 depends on the true rank r, but the algorithm only receives a hint, and the paper does not specify how r is selected.
  • weight clipping epsilon_p = not reported
    Algorithm 1 takes epsilon_p as input to truncate inverse-propensity weights; the truncation is required in Theorem 2 but no selection procedure is given.
  • confidence threshold tau = not reported
    Algorithm 1 abstains when C(s,a) < tau; the paper does not specify how tau is chosen or tuned.
  • regularization weights lambda_L, lambda_S = not reported
    The weighted nuclear norm and l1 penalties in the objective have weights lambda_L and lambda_S; the text says they 'can be tuned using doubly robust off-policy evaluation' but no tuning protocol or values are given.
  • embedding dimension d = 32 (saturation point, Fig. 3a)
    In the latent factorization extension, the embedding dimension d is a hyperparameter; Fig. 3a shows performance saturates around d=32, but the choice is not derived from the theory.
  • completion frequency K = 5,000-10,000 steps
    The completion step is run every K environment steps; Table 1 reports K=5-10k, but the value is a design choice not derived from guarantees.
assumptions (5)
  • domain assumption Reward matrix R is |S|x|A| and admits the decomposition R = L* + S* + E with rank(L*) <= r, S* sparse, E sub-Gaussian.
    This is the central structural assumption stated in Section 3.2 before Theorem 2. It is not justified from first principles; it defines the regime where PAMC works.
  • domain assumption Standard incoherence assumptions on L*.
    Invoked in Theorem 2, but the exact incoherence parameters are never stated in the main text; the reader cannot check whether they hold in the empirical tasks.
  • domain assumption Positivity/overlap: p_sa > 0 for all (s,a) in supp(pi*).
    Lemma 1 requires this for IPW consistency; the paper acknowledges deterministic policies can violate it, so the guarantees do not apply in the tabular deterministic limit.
  • domain assumption Propensity scores p_sa can be estimated from the behavior policy.
    Algorithm 1 line 7 says 'estimate propensities', but no estimator or its error is analyzed; in practice, MNAR weights estimated from off-policy data can be biased.
  • domain assumption The policy pi used for the regret bound visits states approximately according to the stationary distribution of pi*.
    Theorem 3 measures error under the visitation-weighted norm of the optimal policy, yet the algorithm only collects data under its own behavior policy; the paper does not address this discrepancy explicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?." pith.science (2026). https://pith.science/paper/G56WN5Q6

@misc{pith2026250903790,
  author       = {Pith},
  title        = {Pith review of: What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G56WN5Q6}},
  note         = {Machine review of arXiv:2509.03790}
}
abstract

Sparse-reward reinforcement learning (RL) remains fundamentally hard: without structure, any agent needs $\Omega(|\mathcal{S}||\mathcal{A}|/p)$ samples to recover rewards. We introduce Policy-Aware Matrix Completion (PAMC) as a first concrete step toward a structural reward learning framework. Our key idea is to exploit approximate low-rank + sparse structure in the reward matrix, under policy-biased (MNAR) sampling. We prove recovery guarantees with inverse-propensity weighting, and establish a visitation-weighted error-to-regret bound linking completion error to control performance. Importantly, when assumptions weaken, PAMC degrades gracefully: confidence intervals widen and the algorithm abstains, ensuring safe fallback to exploration. Empirically, PAMC improves sample efficiency across Atari-26 (10M steps), DM Control, MetaWorld MT50, D4RL offline RL, and preference-based RL benchmarks, outperforming DrQ-v2, DreamerV3, Agent57, T-REX/D-REX, and PrefPPO under compute-normalized comparisons. Our results highlight PAMC as a practical and principled tool when structural rewards exist, and as a concrete first instantiation of a broader structural reward learning perspective.

Figures

Figures reproduced from arXiv: 2509.03790 by the authors.

Figure 1
Figure 1. Our proposed framework. A bi-level RL loop where policy updates use confidence [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic of our method. The outer loop updates the policy [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Embedding analysis. Left: Performance saturates around [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A conceptual plot illustrating Theorem 3. Policy regret scales with completion error. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: PAMC scales effectively on the 50-task MetaWorld benchmark, leveraging shared struc [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Empirical validation. Left: Completion error scales linearly with [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Computational overhead scaling. Left: Overhead increases linearly with the latent rank [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 31 canonical work pages

  1. [1]

    An optimistic perspective on offline reinforcement learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp.\ 104--114. PMLR, 2020

    work page 2020

  2. [2]

    Concrete problems in ai safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. 2016

    work page 2016

  3. [3]

    Minimax regret bounds for reinforcement learning

    Mohammad Gheshlaghi Azar, Ian Osband, and R \'e mi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pp.\ 263--272. PMLR, 2017

    work page 2017

  4. [4]

    Never give up: Learning directed exploration strategies

    Adri \`a Puigdom \`e nech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020

    work page 2020

  5. [5]

    Successor features for transfer learning in reinforcement learning

    Andr \'e Barreto, Will Dabney, R \'e mi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer learning in reinforcement learning. In Advances in neural information processing systems, pp.\ 4055--4065, 2017

    work page 2017

  6. [6]

    Recent advances in hierarchical reinforcement learning

    Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13 0 (1-2): 0 41--77, 2003

    work page 2003

  7. [7]

    Unifying count-based exploration and hashing: A case study of model-based rl

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and hashing: A case study of model-based rl. In Advances in neural information processing systems, pp.\ 2035--2043, 2016

    work page

  8. [8]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018

    work page 2018

Show all 41 references
  1. [9]

    Exact matrix completion via convex optimization

    Emmanuel J Cand \`e s and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9 0 (6): 0 717--772, 2009

    2009

  2. [10]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in neural information processing systems, pp.\ 4299--4307, 2017

    2017

  3. [11]

    Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning

    Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. In Advances in Neural Information Processing Systems, pp.\ 5713--5723, 2017

    2017

  4. [12]

    First return, then explore

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. volume 590, pp.\ 580--586. Nature Publishing Group, 2021

    2021

  5. [13]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.\ 1126--1135. PMLR, 2017

    2017

  6. [14]

    Noisy networks for exploration

    Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. In International Conference on Learning Representations, 2018

    2018

  7. [15]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning, pp.\ 1050--1059. PMLR, 2016

    2016

  8. [16]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems, pp.\ 4878--4887, 2017

    2017

  9. [17]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp.\ 1321--1330. PMLR, 2017

    2017

  10. [18]

    Is q-learning provably efficient? In Advances in neural information processing systems, pp.\ 4863--4873, 2018

    Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? In Advances in neural information processing systems, pp.\ 4863--4873, 2018

    2018

  11. [19]

    Matrix factorization techniques for recommender systems

    Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. In Computer, volume 42, pp.\ 30--37. IEEE, 2009

    2009

  12. [20]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, pp.\ 1179--1191, 2020

    2020

  13. [21]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp.\ 6402--6413, 2017

    2017

  14. [22]

    Curl: Contrastive unsupervised representations for reinforcement learning

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp.\ 5639--5650. PMLR, 2020

    2020

  15. [23]

    Scalable agent alignment via reward modeling: A research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.07871, 2018

    2018 arXiv

  16. [24]

    Neural matrix completion

    Yao Ma, Tong He, Xiaorui Wang, Chao Zhou, and Jure Leskovec. Neural matrix completion. In International Conference on Machine Learning, pp.\ 4321--4331. PMLR, 2019

    2019

  17. [25]

    Geometric deep learning on graphs and manifolds using mixture model cnns

    Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5115--5124, 2017

    2017

  18. [26]

    Deep exploration via bootstrapped dqn

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp.\ 4026--4034, 2016

    2016

  19. [27]

    Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in neural information processin...

    2019

  20. [28]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  21. [29]

    Parameter space noise for exploration

    Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018

    2018

  22. [30]

    Masked autoencoder for distribution estimation

    Sai Rajeswar, Brian Manela, Florian Step, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Simon Lacoste-Julien. Masked autoencoder for distribution estimation. In Advances in Neural Information Processing Systems, pp.\ 18114--18128, 2022

    2022

  23. [31]

    Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization

    Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52 0 (3): 0 471--501, 2010

    2010

  24. [32]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017

    2017

  25. [33]

    Data-efficient reinforcement learning with self-predictive representations

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021

    2021

  26. [34]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018

    2018

  27. [35]

    Exploration: A study of count-based exploration for deep reinforcement learning

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. Exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp.\ 2753--2762, 2017

    2017

  28. [36]

    Transfer learning for reinforcement learning domains: A survey

    Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10 0 (7), 2009

    2009

  29. [37]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.\ 1094--1100. PMLR, 2020

    2020

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  31. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  32. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  33. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.