Pith. sign in

REVIEW 3 cited by

Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2010.14498 v2 pith:DFYHX5GH submitted 2020-10-27 cs.LG stat.ML

Aviral Kumar , Rishabh Agarwal , Dibya Ghosh , Sergey Levine This is my paper
classification cs.LGstat.ML
keywords valuedeepimplicitnetworkphenomenonunder-parameterizationbootstrappingdrop
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We identify an implicit under-parameterization phenomenon in value-based deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network. We characterize this loss of expressivity via a drop in the rank of the learned value network features, and show that this typically corresponds to a performance drop. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings. We formally analyze this phenomenon and show that it results from a pathological interaction between bootstrapping and gradient-based optimization. We further show that mitigating implicit under-parameterization by controlling rank collapse can improve performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    TeLAPA preserves behaviorally diverse policy neighborhoods in a shared latent space, improving MiniGrid continual RL transfer, revisit recovery, and retention over single-model preservation.

  2. Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Frozen random CNN feature extractors in PPO yield sparse readouts whose active-neuron count is claimed to track task complexity, but the flagship deterministic-Pong numbers are contradicted by the paper's own appendices.

  3. Recovering Plasticity of Neural Networks via Soft Weight Rescaling

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Soft Weight Rescaling shrinks each layer's weights toward their initialization at every step, bounding weight norms and improving plasticity and test accuracy in continual, warm-start, and single-task learning.

Pith tools