REVIEW 5 major objections 5 minor
Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Symmetric ternary spiking neurons fail in deep Q-learning because their expected gradient vanishes; an asymmetric variant with a trainable negative threshold restores nonzero gradients and lifts scores to 130 percent of the binary…
desk verdict Promising asymmetric ternary neuron for spiking RL, but the 30% improvement is an artifact of ratio averaging and the theoretical explanation is shaky. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is the asymmetric ternary LIF neuron of Equation (19), which spikes $s_l(t)=1$ when $m_l(t) \ge v_{th}^p$, $s_l(t)=-1$ when $m_l(t) \le -v_{th}^n$, and $0$ between the thresholds, with $v_{th}^p \neq v_{th}^n$. Around it, the paper builds three pieces of analysis: a Gaussian approximation of the membrane-potential distribution derived from the Fokker-Planck equation for the subthreshold LIF; an information-entropy comparison showing $H(\tilde{s}_{AT})>H(\tilde{s}_B)$ for the same firing rate; and an expected-gradient identity that makes the symmetric neuron's expected gradient vanish when $p_+=p_-$. The trainable negative threshold is the mechanism that lets the optimizer move the neuron away from the vanishing-gradient point while keeping the extra ternary bit of capacity. Training stability is then carried by a block dynamical isometry calculation, Lemma 4.6, which gives $\phi(s_m s_m^T)=1-r$ and $\varphi(s_m s_m^T)=r-r^2$ for the spiking block, matching or beating the ReLU values used as the stability benchmark.
What would settle it
Run DTSQN training on Breakout while recording the actual surrogate gradient values that backpropagation uses, for instance the arctan or straight-through estimator from Table 1, and compare their empirical average for symmetric and asymmetric ternary neurons. The paper's Equation 16 implies this average is positive for both, so if the symmetric neuron's measured surrogate gradient is not smaller by an amount that tracks the score gap, the vanishing-expected-gradient mechanism is falsified even if the performance difference remains.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the poor Q-learning performance of ternary spiking neurons is caused by the symmetry of their thresholds, not by ternary encoding itself. For a symmetric ternary LIF, the Gaussian-like distribution of the subthreshold membrane potential makes the probabilities of a $+1$ spike and a $-1$ spike equal, so $\mathbb{E}[\partial \tilde{s}_T/\partial m] = \partial(p_+ - p_-)/\partial m$ is zero; the paper identifies this vanishing expected gradient, through the chain rule, with a biased gradient estimator that corrupts training. The asymmetric ternary LIF of Equation (19), with $v_{th}^p \neq v_{th}^n$, preserves $p_+ \neq p_-$, keeps the expected gradient nonzero, and still has higher information entropy than a binary LIF (Theorem 4.3). Empirically, the symmetric ternary network (DTSQN) falls to about 50 percent of the binary DSQN score, whereas the asymmetric network (DATSQN) reaches about 130 percent, and the training-gradient norm shows no vanishing or exploding behavior.
Load-bearing premise
The explanation stands on identifying the training signal with the derivative of spike probabilities: if a symmetric ternary neuron produces positive and negative spikes equally often, its expected gradient is claimed to be zero, and that zero is what hurts learning.
Editorial extensions
If this is right
- DATSQN can be deployed as a fully spiking Q-network with a rate-coded first layer and a shorter simulation window, without paying the performance penalty that forced earlier DSQNs to use a non-spiking, floating-point first layer.
- Ternary representation capacity is usable in reinforcement learning, not just in classification or language tasks, once the threshold asymmetry removes the vanishing expected gradient.
- The neuron adds only a trainable threshold parameter and no multiplications, so the energy-efficiency advantage of spiking networks is preserved.
- Because block dynamical isometry holds at least as well as for ReLU, networks built from asymmetric ternary neurons should avoid gradient vanishing and exploding in other deep architectures, not only in Q-learning.
- The learned thresholds drift toward an excitatory-dominated regime, with positive spikes outnumbering negative spikes by a ratio of about 65 percent or higher, which the paper connects to the biological excitatory-inhibitory balance known to support efficient learning.
Reading between the lines
- A testable consequence the paper leaves implicit: if the vanishing-expected-gradient mechanism is right, symmetric ternary neurons should degrade most in settings where the training signal is bootstrapped from the network's own predictions, as in Q-learning, while supervised tasks should be comparatively robust; this predicts where ternary spiking will need asymmetric thresholds.
- The analysis compares a strictly positive surrogate-gradient expectation with a zero expected derivative of the stochastic spike probability; the paper's explanation lives in that gap, so measuring the empirical average of the actual surrogate gradient during DTSQN training would directly test whether the bias mechanism, rather than some other property of symmetric thresholds, causes the drop.
- A cheaper alternative to threshold asymmetry would be a trainable bias that shifts the membrane-potential distribution, achieving $p_+ \neq p_-$ with symmetric thresholds; comparing that variant with DATSQN would show whether asymmetry itself or simply the nonzero probability difference is the active ingredient.
- For neuromorphic deployment, the need to train $v_{th}^n$ per neuron adds a gradient path through a threshold parameter that inference-only chips may not support, so the practical benefit may depend on whether on-chip or offline training can update this parameter.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies ternary spiking neurons in deep Q-learning. It reports that the existing symmetric ternary LIF neuron (DTSQN) performs worse than a binary-neuron DSQN on seven Atari games, hypothesizes that this is due to a zero expected gradient that arises when positive and negative spikes are equally probable, and proposes an asymmetric ternary neuron (DATSQN) with unequal positive and negative firing thresholds. The authors claim DATSQN improves over DSQN by 30% on average and provide a dynamical-isometry argument for training stability. The evaluation uses a fully spiking Bernoulli input encoder and a simulation time window of 20, which are more neuromorphic-friendly than the baseline setup.
Significance. If the central empirical claim were established, the paper would make a useful contribution to energy-efficient spiking reinforcement learning: it introduces a simple, multiplication-free neuron variant, evaluates it under a fully spiking input encoding, and includes an ablation separating the effect of a trainable threshold from that of asymmetric thresholds. The paper also explicitly attempts a theoretical account of why ternary neurons fail in Q-learning, which is a question of genuine interest. However, the current evidence does not support the headline 30% improvement: the number is an average of per-game ratios rather than an aggregate or statistically grounded effect, the experiments use a single training run per condition, and the theoretical mechanism confuses the expected surrogate gradient with the derivative of the stochastic spike expectation. The idea is promising and the empirical trend is suggestive, but the quantitative and statistical claims need substantial rework before the paper can be accepted.
major comments (5)
- [Section 5, Tables 4 and 5] The headline "30% improvement" is the mean of seven per-game DATSQN/DSQN ratios, not an aggregate improvement. Summing the raw scores in Table 4 gives roughly 61,537 for DATSQN versus 58,070 for DSQN, an aggregate gain of about 6%, not 30%. The ratio average is dominated by Beam Rider (193%), which has the largest absolute score scale. Moreover, Table 4 shows that in Boxing DATSQN (79.1 ± 7.1) is not above DSQN (79.3 ± 9.5), so the statement that DATSQN "consistently achieves higher scores than DSQN" is not accurate. Please report aggregate scores, per-game differences with confidence intervals, and avoid averaging scale-dependent ratios.
- [Section 5, 'Testing and Metrics' and Table 4] The experiments use a single training run per condition, and the reported standard deviation is over 10 evaluation episodes of the best checkpoint selected during training. This standard deviation reflects evaluation noise, not training variability, and best-checkpoint selection can inflate scores. With one seed, the differences are not statistically established; for example, Breakout is 207.7 ± 64 (DSQN) versus 252.14 ± 70 (DATSQN) and Gopher is 1842 ± 465 versus 2111.1 ± 648, both well within one standard deviation. Multiple training seeds (at least 3–5) and a fixed evaluation protocol are needed to support the central improvement claim.
- [Section 5, Hyperparameters and Table 3] The decay factor β and learning rate were tuned on Breakout, and Breakout is also one of the seven reported test games. This makes the reported improvement on Breakout optimistic and means the other games are not fully held out with respect to these hyperparameters. The authors should either tune on a separate game or environment, or report sensitivity to these hyperparameters across games.
- [Section 4.1, Equations (16)–(18) and Lemma 4.1] The theoretical explanation conflates two distinct quantities. Equation (16) defines the expected surrogate gradient E[GE(m,t)] = ∫ GE(m)p(m,t)dm, which is strictly positive for the listed surrogate functions and is the quantity used in backpropagation. Equations (17)–(18) instead compute dE[˜s]/dm = d(p+ − p−)/dm, which is the derivative of the expected stochastic spike, not the surrogate gradient used in training. Lemma 4.1 then follows almost by construction: maximum entropy is defined as p+ = p−, and Equation (18) is defined so that p+ = p− gives zero. This does not establish an independent trade-off between representation capacity and gradient bias, and it does not directly connect to the surrogate-gradient training described in Algorithm 1. The mechanism for DTSQN's degradation is therefore not supported by the presented mathematics; at minimum, the relationship between d(p+ − p−)/dm and the actual surrogate gradient update must be made explicit and justified.
- [Section 4.3, Lemma 4.6 and Theorem 4.7] The dynamical-isometry analysis assumes that the Jacobian of the threshold function has entries equal to 1 when m(t) ∈ (−vthn, vthp) and 0 otherwise. This is a straight-through estimator with a boxcar gradient, but the surrogate gradients listed in Table 1 and used elsewhere in the paper (Atan, STE, Sigmoid) are smooth positive functions, not boxcar functions. The isometry result is therefore not connected to the actual training procedure. Please clarify which gradient estimator is used in the isometry analysis and in the experiments, or revise the analysis to match the surrogate used in training.
minor comments (5)
- [Section 4.1, Remark 4.2] Remark 4.2 states that "the expected values of the gradient estimators are nonzero" and then attributes high bias to ternary LIF; this appears to contradict the preceding bullet that says the expected gradient is zero for symmetric ternary thresholds. Please rephrase to distinguish the expected surrogate gradient (positive) from the expected stochastic-spike gradient (zero for symmetric ternary).
- [Algorithm 1, line 10] There is a missing closing parenthesis in "s_k(t) = Threshold(mk(t)"; please fix the typo.
- [Section 3, Equation (2)] The reset potential v_reset is set to 0 in Table 3, but Equation (2) and the surrounding text treat v_reset as a general parameter. Consider stating explicitly that the analysis and experiments use v_reset = 0, since this is used in the Gaussian membrane-potential approximation.
- [Section 5, Figure 3] The caption of Figure 3(b) mentions "DATSQN with one versus two trainable threshold parameters," but the text says DATSQN uses two independent trainable thresholds. Please clarify whether the comparison is between one and two trainable thresholds within the asymmetric model, and report the corresponding numeric scores.
- [Throughout] The manuscript uses "SSN" in a few places (e.g., "SSNs currently lag") where "SNN" is intended; please correct the terminology.
Circularity Check
Lemma 4.1 is definitional: 'maximum entropy' (p+=p−) and the Eq.-18 'expected gradient' d(p+−p−)/dm are the same condition, so the claimed explanation of DTSQN degradation is built into the definitions; the empirical DATSQN-vs-DSQN comparison remains an independent experiment.
-
self definitional
[Section 4.1, Equations (15), (18), and Lemma 4.1]
"the maximum information entropy is achieved when p− = p+, reflecting a balanced and symmetric distribution ... For ternary neurons with symmetric thresholds, where vthn = vthp, we have p+ = p− ... Thus ... the expected gradient becomes zero. Lemma 4.1. The network achieves maximum representation capacity when the expected value of the gradient through the network approaches zero."
Equation (18) defines the 'expected value of the spike generation gradient' as ∂(p+ − p−)/∂m(t), while Equation (15) defines maximum ternary entropy (at fixed firing rate) as p+ = p−. Lemma 4.1 then 'proves' that maximum capacity forces the expected gradient to zero; this is just restating the equivalence p+ = p− ⇔ p+ − p− = 0. No independent constraint or mechanism is introduced, so the claimed capacity-bias trade-off is built into the definitions. The step is also disconnected from the actual training signal: Equation (16) defines the surrogate-gradient expectation E[GE(m)] used in backpropagation, which the paper itself says is strictly positive for all neuron types, so the Eq.-18 quantity is not the training gradient.
full rationale
The paper's main empirical claim—DATSQN outperforms DSQN—rests on Table 4/5 experiments and is not circular: the asymmetric-threshold network is trained and evaluated independently of the Lemma, and the comparison to DSQN is a direct measurement. The circularity is confined to the theoretical framing in Section 4.1. The paper defines the 'expected gradient' in Eq. 18 as d(p+−p−)/dm, and separately defines maximum entropy as p+=p−; Lemma 4.1 then announces that maximum capacity makes this gradient zero. That is a tautology, not a derived trade-off, and it is further disconnected from the surrogate gradient GE of Eq. 16 actually used in training. Because the theoretical explanation of DTSQN's degradation and the design motivation for the asymmetric thresholds rest on this definitional step, the paper has one partial circular step. There is no load-bearing self-citation: the cited Gaussian membrane-potential result [54], ternary model [31], and dynamic-isometry results [33] come from external groups. The empirical effect size and single-seed evaluation are separate correctness concerns, not circularity.
Assumptions & free parameters
free parameters (4)
- v_thn (negative threshold) =
initialized to 2, trainable
- decay factor beta =
0.9
- learning rate =
0.00005
- v_thp (positive threshold) =
1
assumptions (5)
- domain assumption Membrane potential distribution is Gaussian-like and symmetric about Vreset = 0
- ad hoc to paper The gradient estimator used in backpropagation equals dE[s]/dm
- domain assumption Stochastic spike model of Guo et al. [31]
- domain assumption Firing rate r is the same for binary and ternary neurons in the comparison
- domain assumption Firing rate r is between 0.1 and 0.5 and the input distribution is zero-mean for the isometry comparison
invented entities (1)
-
Asymmetric ternary LIF neuron
Cite this review
Pith. "Pith review of Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons." pith.science (2026). https://pith.science/paper/HKNEHJXH
@misc{pith2026250603392,
author = {Pith},
title = {Pith review of: Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons},
year = {2026},
howpublished = {\url{https://pith.science/paper/HKNEHJXH}},
note = {Machine review of arXiv:2506.03392}
}
read the original abstract
We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.
Figures
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.