REVIEW 2 major objections 5 minor 31 references
Learning sparse representations in reinforcement learning
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Adding a k-Winners-Take-All sparse-coding layer to a temporal-difference network lets it solve three control tasks where standard backpropagation TD learning fails.
desk verdict A competent but largely retrospective simulation study: the Puddle-world kWTA advantage is real and statistically supported, but the central mechanism claim is untested because no capacity-matched dense baseline appears. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the k-Winners-Take-All (kWTA) nonlinearity applied to the hidden layer of a one-hidden-layer backpropagation network. In the paper's Algorithm 2, the net input vector is sorted, the top $k+1$ values are located, and a bias $b = \eta'_k - q(\eta'_k - \eta'_{k+1})$ is subtracted so that only the top $k$ units receive positive adjusted net input; a sigmoid then makes roughly $k=0.1n$ hidden units strongly active. This replaces smooth hidden activations with a sparse conjunctive code of the state, balancing overlap that supports generalization against separation that prevents interference, at an extra cost of $O(n + k \log k)$ per forward pass.
What would settle it
Train a regular backpropagation SARSA network with hidden units reduced so its number of active units matches the kWTA network's k in each task, e.g., 22/260/840 total hidden units instead of 220/2604/8400, holding learning rate, exploration, and episode limits fixed. If the dense network's value-function error and goal attainment match the kWTA network's, the paper's attribution of success to sparse conjunctive coding rather than reduced capacity is falsified.
Extended reading notes
Core claim
On its own terms, the paper establishes that SARSA temporal-difference learning succeeds on Puddle-world, Mountain-car, and Acrobot when the value-function network uses a hidden layer governed by k-Winners-Take-All lateral inhibition, while otherwise identical backpropagation networks and linear networks fail on at least some of these tasks. In Puddle-world the kWTA network's accumulated rewards deviate from optimal by less than one reward point and reach the goal from 99.9 percent of starting locations, versus 99.0 percent for the regular network and 93.3 percent for the linear network; in Mountain-car and Acrobot only the kWTA network learned a stable optimal policy. The authors interpret this as evidence that sparse conjunctive state representations, not fixed engineered codes, are what make TD with neural function approximation work, and that cortical lateral inhibition may explain why the brain's TD-like dopamine system does not show the same failures.
Load-bearing premise
The paper assumes that the kWTA network's advantage is caused by sparse conjunctive coding rather than by the much smaller number of active hidden units, and it does not include a control network with comparable active-unit count to rule out reduced model capacity as the cause.
Editorial extensions
If this is right
- If correct, the three reported failures of TD with neural networks are representational failures, not limits of temporal-difference learning itself.
- If correct, hard-wired coarse codes are unnecessary: backpropagation plus kWTA can learn state representations that give the same benefit as engineered CMAC codes.
- If correct, the same mechanism should reduce catastrophic interference whenever nearby states demand very different values, not just in the three benchmark tasks.
- If correct, the biological hypothesis is supported: cortical lateral inhibition is present precisely where and when dopamine-based TD learning must work.
Reading between the lines
- The paper does not test whether any hard top-$k$ sparsity layer would produce the same benefit; if top-$k$ ReLU matched kWTA, the causal story would be sparsity, not the specific lateral-inhibition mechanism.
- Because kWTA is non-smooth, the usual gradient-descent interpretation of backpropagation is invalid at the threshold; a rigorous account would require treating the layer as a projection, so convergence cannot be inferred from standard TD theorems.
- The three solved tasks are all low-dimensional; the strongest untested extension is whether kWTA in fully connected layers of deep networks prevents catastrophic interference in continual-learning benchmarks, which would follow if the paper's mechanism is the active ingredient.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes that adding a k-Winners-Take-All (kWTA) mechanism to the hidden layer of a one-hidden-layer SARSA network induces sparse conjunctive representations of state, and that this avoids catastrophic interference and supports TD learning. The hypothesis is tested empirically on three control tasks: Puddle-world, Mountain-car, and Acrobot. The authors compare three architectures (linear, regular backpropagation, and kWTA) using the same SARSA update, and report that the kWTA network outperforms the regular and linear networks, especially on Acrobot where the baselines fail. The paper is presented as evidence that lateral-inhibition-like sparsity can compensate for known weaknesses of neural-network TD learning, with implications for cognitive neuroscience. The main contribution is empirical; no new theoretical analysis is offered.
Significance. If the central claim is upheld, the paper would provide a useful and biologically motivated architectural modification for TD learning with function approximation, and it would strengthen the link between cortical sparse coding and reward-based learning. The paper has concrete strengths: it reports multi-seed statistics for Puddle-world (20 simulations per condition, t-tests), it provides the simulation code at a public URL, and it gives detailed environment and network hyperparameters that facilitate replication. The Puddle-world advantage of kWTA over regular backpropagation is statistically significant, and the Acrobot result is qualitatively clear. However, the key mechanistic conclusion, that sparse conjunctive coding is the causal factor, is not yet supported because the kWTA network also has about 10-fold fewer active hidden units than the regular network, confounding sparsity with reduced model capacity.
major comments (2)
- [§4.1, §5.1, §5.3] The central claim that sparse conjunctive representations produced by lateral inhibition are beneficial is not separated from a simple capacity reduction. In §4.1 the kWTA network keeps only k=10% of the hidden units active (Section 3.2), so for the Puddle-world, Mountain-car, and Acrobot tasks the kWTA network uses 22, 260, and 840 active hidden units respectively, whereas the regular backpropagation network uses 220, 2604, and 8400 active units. The paper never reports a control condition with a regular network whose number of active hidden units matches that of the kWTA network, nor any other capacity-limiting baseline such as a smaller dense hidden layer or a dropout rate. Without such a control, the observed improvements in Figures 8 and 10 cannot be attributed to sparse structuring rather than to fewer active basis functions; a reduced-capacity dense network may show the same interference-relief effect. This is load-bearing because the abstract and §1.2 explicitly frame the benefit as coming from sparse conjunctive coding, not from reduced capacity.
- [§5.2 and §5.3] The quantitative evidence for the Mountain-car and Acrobot advantages is incomplete. For Mountain-car, Figure 9 shows learning curves and value-function plots but no statistical tests or multi-run averages are reported; the text in §5.2 says the kWTA value function is 'closest numerically to optimal Q-table results' and 'the most stable,' but no numerical comparisons or error bars are given. For Acrobot, §5.3 states that only the kWTA network could learn the optimal policy, but the figure presents representative runs without quantitative success rates, average steps to goal, or statistics over seeds. Because the paper's central claim spans all three tasks, the lack of statistical backing for two of the three tasks weakens the evidence. At minimum, the authors should report mean and variance of final performance and a test of significance for Mountain-car and Acrobot, or clearly label these as illustrative single-run demonstrations.
minor comments (5)
- [Title page] The affiliation line contains a typo: 'Electrical Engineering and Computer Scinence' should be 'Science'.
- [§3.2] The sentence 'k is relatively smaller than n. For example k = 0.1× n is considered for the simulations reported in this chapter' refers to 'this chapter', which is thesis language; it should read 'in this paper'.
- [§4.1 and §5.1] The paper says results in Figure 7 were 'initially reported at Rafati and Noelle (2015)' and similar text appears for other figures; the relationship to prior conference papers and the new contribution of this manuscript should be stated more explicitly in the introduction or a dedicated section, so that the novelty is clear to the reader.
- [§5.1] The significance values in the text and in Figure 8 are slightly inconsistent: the text reports t(38)=2.205; p<0.05 for the goal-rate comparison, while Figure 8 labels the same statistic as p<0.005. The authors should reconcile these values.
- [§3.3] The paper acknowledges that backpropagation through the kWTA threshold is not a true gradient descent and that this discrepancy is ignored. This is a reasonable simplification, but a sentence discussing why small learning rates suffice (e.g., the bias is locally constant) would help the reader judge the validity of the approximation.
Circularity Check
No circularity: the central claim is an empirical simulation hypothesis, not a derived prediction, and self-citations are provenance rather than load-bearing evidence.
full rationale
The paper makes no formal derivation: the central claim is a hypothesis supported by simulations (Sections 4 and 5). The kWTA architecture is taken from external work (O'Reilly and Munakata, 2001; O'Reilly, 2001) and applied to SARSA; no fitted parameter is later renamed as a prediction. The comparisons against regular and linear networks on Puddle-world, Mountain-car, and Acrobot are self-contained simulations with the code available at http://rafati.net/td-sparse/. Self-citations (Rafati and Noelle, 2015, 2017) are provenance statements, e.g., "These results were initially reported at Rafati and Noelle (2015)", rather than load-bearing evidence: the paper itself reports the simulation design and results. Section 3.3's admission that the kWTA bias is ignored when computing backpropagation gradients is a technical caveat, not circularity. The absence of a capacity-matched dense control (e.g., a dense network with only 22 active hidden units in Puddle-world) is a potential confound for the claim that sparse conjunctive coding, rather than reduced active-unit count, drives the improvement; but a confound is not a circular reduction. No equation in the paper defines the target result in terms of the input, and no fitted parameter is relabeled as a prediction. Therefore the paper is not circular.
Assumptions & free parameters
free parameters (5)
- kWTA sparsity fraction k =
10% of hidden units (22, 260, 840)
- Hidden layer size =
220 (Puddle-world), 2604 (Mountain-car), 8400 (Acrobot)
- Learning rate alpha =
0.005, 0.001, 0.0001
- Input encoding mesh and Gaussian width =
21, 61, 21 input units per variable; Gaussian sigma = 1/mesh
- Exploration and termination schedule =
epsilon 0.1 or 0.05 with decay; max episodes 44,100 or 200,000; max steps 80, 3000, 2000
assumptions (5)
- standard math Standard MDP and SARSA convergence properties for tabular value functions.
- domain assumption kWTA dynamics adequately model cortical lateral inhibition.
- ad hoc to paper Backpropagation through the kWTA threshold is a good approximation of gradient descent.
- domain assumption Poor Regular network performance reflects catastrophic interference, not poor hyperparameter choice.
- domain assumption The three benchmark tasks instantiate the claimed failure mode.
Cite this review
Pith. "Pith review of Learning sparse representations in reinforcement learning." pith.science (2026). https://pith.science/paper/7P72UKZV
@misc{pith2026190901575,
author = {Pith},
title = {Pith review of: Learning sparse representations in reinforcement learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7P72UKZV}},
note = {Machine review of arXiv:1909.01575}
}
read the original abstract
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learning. These algorithms typically learn a mapping from the agent's current sensed state to a selected action (known as a policy function) via learning a value function (expected future rewards). TD Learning methods have been very successful on a broad range of control tasks, but learning can become intractably slow as the state space of the environment grows. This has motivated methods that learn internal representations of the agent's state, effectively reducing the size of the state space and restructuring state representations in order to support generalization. However, TD Learning coupled with an artificial neural network, as a function approximator, has been shown to fail to learn some fairly simple control tasks, challenging this explanation of reward-based learning. We hypothesize that such failures do not arise in the brain because of the ubiquitous presence of lateral inhibition in the cortex, producing sparse distributed internal representations that support the learning of expected future reward. The sparse conjunctive representations can avoid catastrophic interference while still supporting generalization. We provide support for this conjecture through computational simulations, demonstrating the benefits of learned sparse representations for three problematic classic control tasks: Puddle-world, Mountain-car, and Acrobot.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Boyan, J. A., Moore, A. W., 1995. Generalization in reinforcement learning: Safely approximating the value function. In: Advances in Neural Information Processing Systems 7. MIT Press, Cambridge, MA, pp. 369--376
work page 1995
-
[3]
Reinforcement learning: The good, the bad and the ugly
Dayan, P., Niv, Y., 2008. Reinforcement learning: The good, the bad and the ugly. Current Opinion in Neurobiology 18, 185--196
work page 2008
- [4]
- [5]
-
[6]
The utility of sparse representations for control in reinforcement learning
Liu, V., Kumaraswamy, R., Le, L., White, M., 2018. The utility of sparse representations for control in reinforcement learning. arXiv e-prints (1811.06626)
arXiv 2018
-
[7]
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al., 2015. Human-level control through deep reinforcement learning. Nature 518 (7540), 529--533
work page 2015
-
[8]
Montague, P. R., Dayan, P., Sejnowski, T. J., 1996. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience 16, 1936--1947
work page 1996
Show all 31 references
-
[9]
C., 2008
Noelle, D. C., 2008. Function follows form: Biologically guided functional decomposition of memory systems. In: Biologically Inspired Cognitive Architectures — Papers from the 2008 AAAI Fall Symposium
2008
-
[10]
C., 2001
O'Reilly, R. C., 2001. Generaliztion in interactive networks: The benefits of inhibitory competition and Hebbian learning. Neural Computation 13, 1199--1242
2001
-
[11]
C., McClelland, J
O'Reilly, R. C., McClelland, J. L., 1994. Hippocampal conjunctive encoding, storage, and recall: Avoiding a trade-off. Hippocampus 4 (6), 661--682
1994
-
[12]
C., Munakata, Y., 2001
O'Reilly, R. C., Munakata, Y., 2001. Computational Explorations in Cognitive Neuroscience. MIT Press, Cambridge, Massachusetts
2001
-
[13]
F., 2019
Rafati, J., Marcia, R. F., 2019. Deep reinforcement learning via l-bfgs optimization. arXiv e-print (arXiv:1811.02693)
2019 arXiv
-
[14]
C., 2015
Rafati, J., Noelle, D. C., 2015. Lateral inhibition overcomes limits of temporal difference learning. In: 37th Annual Cognitive Science Society Meeting. Pasadena, CA, USA
2015
-
[15]
C., 2017
Rafati, J., Noelle, D. C., 2017. Sparse coding of learned state representations in reinforcement learning. In: Conference on Cognitive Computational Neuroscience. New York City, NY, USA
2017
-
[16]
C., 2019 a
Rafati, J., Noelle, D. C., 2019 a . Learning representations in model-free hierarchical reinforcement learning. arXiv e-print (arXiv:1810.10096)
2019 arXiv
-
[17]
C., 2019 b
Rafati, J., Noelle, D. C., 2019 b . Unsupervised methods for subgoal discovery during intrinsic motivation in model-free hierarchical reinforcement learning. In: 33rd AAAI Conference on Artificial Intelligence (AAAI-19), 2nd Workshop on Knowledge Extraction From Games. Honolul...
2019
-
[18]
C., 2019 c
Rafati, J., Noelle, D. C., 2019 c . Unsupervised subgoal discovery method for learning hierarchical representations. In: 7th International Conference on Learning Representations, ICLR 2019 Workshop on ``Structure & Priors in Reinforcement Learning'', New Orleans, LA, USA
2019
-
[19]
Learning representations in reinforcement learning
Rafati Heravi, J., 2019. Learning representations in reinforcement learning. Ph.D. thesis, University of California, Merced. ://escholarship.org/uc/item/3dx2f8kq
2019
-
[20]
E., Hinton, G
Rumelhart, D. E., Hinton, G. E., Williams, R. J., 1986. Learning representations by back-propagating errors. Nature 323, 533--536
1986
-
[21]
R., 1997
Schultz, W., Dayan, P., Montague, P. R., 1997. A neural substrate of prediction and reward. Science 275 (5306), 1593--1599
1997
-
[22]
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., H...
2016 doi
-
[23]
S., 1988
Sutton, R. S., 1988. Learning to predict by the methods of temporal differences. Machine Learning 3, 9--44
1988
-
[24]
S., 1996
Sutton, R. S., 1996. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA, pp. 1038--1044
1996
-
[25]
S., Barto, A
Sutton, R. S., Barto, A. G., 1998. Reinforcement Learning: An Introduction, 1st Edition. MIT Press, Cambridge, MA, USA
1998
-
[26]
S., Barto, A
Sutton, R. S., Barto, A. G., 2017. Reinforcement Learning: An Introduction, 2nd Edition. MIT Press, Cambridge, MA, USA
2017
-
[27]
Temporal difference learning and TD-Gammon
Tesauro, G., 1995. Temporal difference learning and TD-Gammon . Communications of the ACM 38 (3)
1995
-
[28]
S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., Kavukcuoglu, K., 2017
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., Kavukcuoglu, K., 2017. Feudal networks for hierarchical reinforcement learning. In: Proceedings of Thirty-fourth International Conference on Machine Learning (ICML-17)
2017
-
[29]
A survey of sparse representation: Algorithms and applications
Zhang, Z., Xu, Y., Yang, J., Li, X., Zhang, D., 2015. A survey of sparse representation: Algorithms and applications. IEEE Access 3, 490--530
2015
-
[30]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.senten...
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.