Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures

Alex Kearney; Johannes G\"unther; Michael R. Dawson; Nadia M. Ady; Patrick M. Pilarski

REVIEW 3 major objections 6 minor 29 references

Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TIDBD, a self-tuning temporal-difference learner, matches hand-tuned TD predictions on a real robotic arm and flags sensor failures through its adapted step sizes.

desk verdict Sensible first deployment of TIDBD on a real robot stream, but the sensor-failure detection claim is not supported by the experiments. read the letter →

arxiv 1908.05751 v2 pith:KH4NSFVS submitted 2019-08-15 cs.LG cs.AIcs.ROstat.ML

Johannes Günther , Nadia M. Ady , Alex Kearney , Michael R. Dawson , Patrick M. Pilarski This is my paper · ORCID

classification cs.LGcs.AIcs.ROstat.ML

keywords temporal-differencelearningTIDBDstep-sizeadaptationgeneralvaluefunctionspredictiveknowledgerobotsensorfaultdetectionKanervacoding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that TIDBD, a temporal-difference learner that keeps a separate step size for every feature, can stand in for classic TD learning on real robotic data without the usual hand tuning. On the Modular Prosthetic Limb, a sensor-rich robotic arm whose 108 sensor signals are predicted by 108 general value functions, TIDBD reaches root-mean-squared prediction errors comparable to classic TD with a carefully swept learning rate, while using default meta-parameters. The paper also claims that the step sizes TIDBD learns react in distinct ways to sensor faults: stuck-sensor signals drive some step sizes up to about twice the normal maximum, while broken-sensor noise drives them down and lets the Horde keep predicting untouched sensors with lower error. If these claims hold, long-lived robots and prostheses could skip learning-rate searches and obtain a sensor-health signal directly from their own learning dynamics.

What carries the argument

TIDBD, Temporal-Difference Incremental $\Delta$-Bar-$\Delta$, is the mechanism that carries the paper's argument. It maintains one step size $\alpha_i = e^{\beta_i}$ per feature, and each time-step TD error $\delta$ updates $\beta_i$ by stochastic meta-descent on the squared error, scaled by an AutoStep normalizer that damps overshooting and by a decaying trace of recent weight updates. Because a feature's $\beta_i$ grows only when that feature's recent updates correlate with reducing error, TIDBD effectively learns which features matter and assigns them larger step sizes while shrinking irrelevant or noisy ones. The state representation is selective Kanerva coding, which maps the normalized 108-dimensional sensor readings to a fixed number of active binary prototypes; the paper's Horde of 108 GVF predictors all share this representation while each has its own per-feature step-size vector.

What would settle it

Run the stuck- and broken-sensor experiments with multiple Kanerva prototype placements and noise seeds, then fit a threshold or classifier to the resulting per-feature step-size statistics to separate normal, stuck, and broken conditions; the detection claim collapses if the distributions overlap or if the signatures vanish when the prototype placement changes.

Watch

Extended reading notes

Core claim

The paper's central claim is that per-feature step-size adaptation can make TD prediction practical on a real, high-dimensional, non-stationary data stream. In a Horde of 108 general value functions, each predicting one sensor's discounted future from a selective Kanerva-coded state, TIDBD with an initial step size of $1/(n\eta)$ matches the RMSE of classic TD whose step sizes were individually chosen by a 264-run parameter sweep; TIDBD starts with somewhat higher error but adapts as it learns. The paper further reports that simulated stuck sensors (four elbow signals replaced by $N(1,0.5)$ noise) cause some TIDBD step sizes to grow to roughly twice the maximum seen during healthy operation, while simulated broken sensors ($N(0,10)$ noise) cause the affected step sizes to shrink and reduce the average step size across the Horde. For the broken-sensor case, TIDBD's RMSE on the 104 untouched sensors is about 2.5 times lower than classic TD's. Finally, TIDBD's accumulated error stays comparatively flat across a wide range of initial step sizes and meta step sizes, while classic TD's performance forms a narrow bowl where only a small window of rates works.

Load-bearing premise

The sensor-failure detection claim rests on the assumption that the changed step-size histograms are a reliable, distinctive signature of the failure itself; the experiments only compare distributions visually, using synthetic noise and one random placement of Kanerva prototypes, with no threshold, classifier, or held-out evaluation.

Editorial extensions

If this is right

A robot can run a Horde of 108 predictions without a learning-rate sweep: TIDBD's full update costs about 0.28 seconds and 0.72 MB of extra memory in the paper's Python implementation, still fast enough for a prosthetic arm.
Per-feature step sizes provide a continuously updated internal signal that separates healthy sensors (moderate step sizes), stuck sensors (inflated step sizes), and broken sensors (suppressed step sizes).
When some sensors fail with high-variance noise, TIDBD keeps predictions about the healthy sensors about 2.5 times closer to the true returns than classic TD with tuned rates, because it down-weights the corrupted features.
Across a wide range of initial step sizes and meta step sizes, TIDBD's accumulated error stays relatively flat, so practitioners can use default settings instead of optimizing the learning rate.
In a lifelong setting with drifting sensor behaviour, TIDBD should outperform fixed-rate TD over time because it continually re-adapts step sizes rather than committing to pre-deployment values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The sensor-fault claim could be turned into a practical alarm by fitting a threshold or one-class classifier to normal-operation step-size statistics; the paper only shows visual histogram differences, not a detection procedure.
The paper's own explanation for stuck-sensor step-size growth implicates churn in randomly placed Kanerva prototypes under small noise, which suggests the specific signature may not transfer to other state encodings such as tile coding or learned features.
Because TIDBD's step sizes encode feature relevance from prediction error alone, the same mechanism could serve as a general feature-selection or introspection signal in other long-horizon, high-dimensional learning systems.
A single 30-minute data set from one arm limits the generality of the no-tuning-needed claim; repeating the comparison on different robots, tasks, and longer deployments without changing TIDBD's meta-parameters would be a direct test.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The genuinely new thing is that TIDBD has been run on a real robotic arm with 108 parallel sensor streams, and the comparison to tuned classic TD is mostly fair and useful. That alone is worth reporting. But the second headline claim—that TIDBD automatically detects sensor failures—does not hold up as stated. The evidence is a visual comparison of final step-size histograms across separate runs, with no threshold, classifier, false-positive rate, or held-out evaluation. I agree with your stress-test note: a global shift in the step-size distribution cannot localize a failing sensor without a model of baseline behavior. The paper even admits that in the stuck-sensor case the step sizes rose for all 104 unaffected sensors, which undercuts any notion of targeted detection.

The main RMSE comparison is plausible, and I give the authors credit for noting that the parameter sweep gave classic TD an advantage because step sizes were chosen on the full dataset. They still claim TIDBD avoids parameter search, but that's overstated: they swept the Kanerva coder parameters (n and eta) on the same data and used the best setting for TIDBD. The sensitivity study in Section 4.4 is nice, but without error bars or significance tests, the small RMSE differences in Table 2 are hard to interpret.

There are two artifacts that need attention in revision. First, the broken-sensor experiment reports RMSE values in the millions (1,315,850 vs 509,220) while the rest of the paper reports values around 1,000. That is a huge discrepancy and needs clarification or correction. Second, the manuscript contains what looks like an inserted block from a different AAMAS paper and a duplicated Figure 4 caption; these are fixable but reduce confidence in the provenance of the text. No code or data are released, which makes the empirical claims harder to verify.

Who is this for? Researchers working on step-size adaptation, predictive knowledge, and robot learning. They will get a useful data point, especially the robustness and cost analysis. I would not cite it in my own work unless I needed a robotics application example, but it deserves a serious referee. My recommendation: accept for peer review with major revision. Ask for a real detection evaluation, error bars, an explanation of the RMSE anomaly, and a cleaned manuscript.

Referee Report

3 major / 6 minor

Summary. The paper studies Temporal-Difference Incremental Delta-Bar-Delta (TIDBD) on a 30-minute recording from the Modular Prosthetic Limb, where 108 sensors are used both as state features and as cumulants in a Horde of general value functions. It compares TIDBD against classic TD whose learning rates were selected by an extensive sweep, and reports comparable root mean squared error. It then corrupts four elbow sensors with low-variance Gaussian noise ('stuck') or high-variance Gaussian noise ('broken') and interprets the final per-feature step-size distributions as signatures that allow automatic detection and characterization of these failures. Finally, it sweeps initial step sizes and meta step sizes to argue that TIDBD is more robust to initialization than classic TD. The paper claims three contributions: TIDBD as a practical alternative to tuned TD, sensor-failure detection/characterization via adapted step sizes, and step-size/meta-parameter robustness.

Significance. If the central claims held, the paper would be a useful demonstration of per-feature step-size adaptation in a real, high-dimensional robotic setting with many parallel predictions, and it would strengthen the case for meta-learning in predictive knowledge architectures. The use of real MPL data, 30 independent repetitions, and a full factorial TD baseline are strengths, as is the explicit report of added computation and memory. However, the strongest claims are broader than the evidence: the 'no parameter search' claim does not account for tuning of the Kanerva coder parameters, and the sensor-failure contribution is supported only by qualitative histogram differences rather than a defined detection procedure. With these qualifications, the results remain a valuable empirical step, but the paper needs revision to align its claims with the evidence.

major comments (3)

[Section 3, Table 1; Section 4.1] The claim that TIDBD avoids 'time- and labour-intensive setup' is overstated because the Kanerva coder parameters n=30000 and eta=0.032 were also selected by minimizing RMSE on the same experimental data and were then used for both algorithms. The text states that 'the parameters that yielded the best performance in terms of RMSE for classic TD also performed best in the parameter sweep for TIDBD,' and the TIDBD arm included a sweep over 24 combinations of n and eta. Therefore the comparison is really 'TIDBD with tuned representation parameters versus TD with tuned step sizes,' not 'TIDBD without parameter search.' Please either narrow the contribution to 'no learning-rate search' and report the sensitivity of TIDBD to n and eta (including default or random choices), or re-run the comparison under a protocol in which TIDBD does not receive the advantage of parameters selected on the evaluation data.
[Section 4.2 and 4.3, Figures 5-6] The sensor-failure detection contribution is not established. The evidence consists of visual comparisons of final step-size histograms across separate runs: Figure 5 shows larger step sizes in the stuck-sensor run and Figure 6 shows smaller means in the broken-sensor run. No detector, threshold, false-positive rate, or held-out evaluation is defined, and the claim that the distributions are 'clearly distinguishable' is a visual judgment. For broken sensors, the only quantitative evidence is average step sizes of 0.00037 versus 0.00065, reported without variance or overlap. For stuck sensors, Section 4.2 reports that the remaining 104 sensors also increased their step sizes, so the signature does not localize to the affected sensors; the explanation in terms of randomly placed prototypes is speculative. Please define a concrete detection rule and evaluate its sensitivity and specificity (e.g., classification accuracy or ROC over runs), or weaken the contribution to an observation about step-size dynamics.
[Table 2; Section 4.4, Figure 7] The comparability and robustness claims lack statistical support. Table 2 reports averages over 30 independent runs without confidence intervals or a paired comparison between TIDBD and classic TD, so 'perform comparably' is not quantified. Similarly, Section 4.4 and Figure 7 report sweeps over initial step sizes and meta step sizes but do not state how many runs each curve averages or provide error bars; the conclusion that TIDBD is more robust than TD would be stronger with variance or interval estimates. Please add error bars/confidence intervals and, where possible, paired statistical tests for the period-wise RMSE comparisons.

minor comments (6)

[Section 1] The paragraph beginning 'There are several learning rate adaptation methods' is not tightly connected to the paper's contribution, contains informal phrasing and typos such as 'its predominantly useful', and introduces deep-learning optimizers without a clear experimental role; please revise or remove it.
[Section 3, Eq. (1)] The normalization in Eq. (1) is written ambiguously because the absolute-value bars around G_t^(i) appear to apply only to the denominator but are placed around G_t^(i), and the numerator's x(s_t)^T w_t^(i) is not shown normalized by |G_t^(i)|; please define the normalized prediction and return explicitly.
[Section 4.3] The RMSE values 1,315,850.16 and 509,220.75 for the 104 functioning sensors are orders of magnitude larger than the values in Table 2, suggesting that Eq. (1) normalization was not applied; please clarify the definition and, if these are unnormalized, explain why they are comparable to the earlier RMSE numbers.
[Section 4.2 and Figure 5] There is a typo in Section 4.2 ('senors' should be 'sensors'), and the caption of Figure 5 references 'Subsection 4.2' in a way that may be confusing after typesetting; please fix the typo and update the cross-reference.
[Section 4.1] The standard deviations sigma_TD264=43,734.46, sigma_TD24=313.42, and sigma_TIDBD=1,507.24 are reported without precisely defining what quantity they summarize; please state the aggregation (e.g., accumulated RMSE over time, per-run) and the units.
[Section 5] The sentence 'we expect TIDBD to perform better when applied in a lifelong learning setting' is speculative and not tested in this paper; either remove it or qualify it as a hypothesis for future work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TIDBD is evaluated against an independent robotic-data benchmark, and the sensor-failure observations are post hoc interpretations rather than fitted predictions.

full rationale

The paper's load-bearing claim is empirical: TIDBD with default meta-parameters performs comparably to classic TD whose step sizes were chosen by a full factorial sweep over the same data. The paper explicitly acknowledges that this sweep gives classic TD an information advantage ('our parameter sweep over step sizes provided an advantage to classic TD'), so the comparison is not rigged in TIDBD's favor. TIDBD's step sizes are algorithmic outputs, not parameters fitted to the RMSE outcome; no fitted quantity is renamed as a prediction. The sensor-failure contribution is supported by comparisons of final step-size histograms across separate runs, with no detector, threshold, or statistical test. This is a weakness in evidence quality, but it is not circular: the step-size distributions are direct algorithmic results, not quantities constructed from the failure labels, and the paper even reports an unexpected global increase in step sizes for stuck sensors and explains it through the random Kanerva prototypes rather than suppressing it. The robustness experiments are run directly on the robotic data, and the citations to Kearney et al. (2019) and Mahmood et al. (2012) supply the TIDBD algorithm and suggested defaults, but the current experiments independently test those suggestions against classic TD. No load-bearing claim reduces, by definition or by construction, to its own inputs, and no uniqueness theorem or ansatz is imported through self-citation. The paper is therefore not circular, though the sensor-detection claim is methodologically under-supported.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claims rest on an empirical pipeline: the sensor stream, the Kanerva representation, the TIDBD and TD updates, and the synthetic fault injection. The main free parameters are representation and step-size settings, most of them selected by RMSE sweeps over the same data, which weakens the 'no parameter search' claim. The sensor-failure interpretation adds an ad hoc assumption that the chosen Gaussian noise models real faults. No new physical or algorithmic entities are introduced.

free parameters (8)

Number of Kanerva prototypes n = 30000
Selected among {10000, 20000, 30000, 40000} by minimizing RMSE on the full dataset; used for both TD and TIDBD runs.
Active prototype ratio eta = 0.032
Selected among six candidates by the same RMSE sweep; controls representation granularity and is used for both algorithms.
Classic TD step size alpha (per GVF) = between 0.001/(n*eta) and 0.256/(n*eta)
Chosen independently for each GVF to minimize summed RMSE over all time steps, giving classic TD an oracle-like advantage on the same data.
TIDBD initial step size = 0.00104 = 1/(n*eta)
Initial per-feature step size set from the chosen representation parameters; sensitivity to this value was tested in Section 4.4.
TIDBD meta step size theta = 0.01
Not fitted in this paper; taken from Kearney et al. (2019). The robustness sweep covers theta values, but the default is still a hand-picked constant from prior work.
TIDBD decay parameter tau = 10^4
Taken from Mahmood et al. (2012) and not swept in this paper; listed as a suggested value.
Discount factor gamma = 0.9
Chosen by the experimenters to give a 10-step time scale; affects all GVF returns and prediction targets.
Stuck and broken sensor noise distributions = N(1,0.5) and N(0,10)
Hand-chosen synthetic replacements for stuck and broken elbow sensors; no real fault data are used.

assumptions (6)

standard math The TD(lambda) and TIDBD update equations as written in Algorithm 1 are correct implementations of their published algorithms.
Algorithm 1 is presented without proof; correctness rests on prior derivations by Sutton (1992) and Kearney et al. (2019).
domain assumption The sensor stream can be treated as a Markov state and the discounted return is well-defined for each cumulant.
Section 2 defines GVFs and uses TD with linear function approximation; no verification is provided that the robot stream satisfies the Markov or stationarity assumptions.
domain assumption Selective Kanerva coding with uniformly random prototypes provides a sufficient representation for the 108-dimensional state.
Section 3 relies on Travnik and Pilarski (2017) and parameter sweeps; representation quality is not derived from first principles.
domain assumption TIDBD update rules and meta-parameter settings from prior work transfer to the robotic domain.
Section 2.1 adopts TIDBD from Kearney et al. (2019) with theta=0.01 and tau=10^4; theta sensitivity is tested, tau is not.
domain assumption RMSE normalized by |G_t| is a meaningful way to compare prediction error across sensors on different scales.
Equation (1); the normalization can produce very large values when returns are near zero, as seen in Section 4.3.
ad hoc to paper Gaussian noise with N(1,0.5) and N(0,10) adequately represents real stuck and broken sensors.
Section 3; chosen by the authors to simulate faults without validation against real fault data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures." pith.science (2026). https://pith.science/paper/KH4NSFVS

@misc{pith2026190805751,
  author       = {Pith},
  title        = {Pith review of: Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KH4NSFVS}},
  note         = {Machine review of arXiv:1908.05751}
}

read the original abstract

Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well suited for robotics is that they can be learned online and incrementally through interaction with the environment. However, a remaining challenge for many prediction-learning approaches is an appropriate choice of prediction-learning parameters, especially parameters that control the magnitude of a learning machine's updates to its predictions (the learning rate or step size). To begin to address this challenge, we examine the use of online step-size adaptation using a sensor-rich robotic arm. Our method of choice, Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step sizes on a feature level; importantly, TIDBD allows step-size tuning and representation learning to occur at the same time. We show that TIDBD is a practical alternative for classic Temporal-Difference (TD) learning via an extensive parameter search. Both approaches perform comparably in terms of predicting future aspects of a robotic data stream. Furthermore, the use of a step-size adaptation method like TIDBD appears to allow a system to automatically detect and characterize common sensor failures in a robotic application. Together, these results promise to improve the ability of robotic devices to learn from interactions with their environments in a robust way, providing key capabilities for autonomous agents and robots.

Figures

Figures reproduced from arXiv: 1908.05751 by the authors.

**Figure 1.** The Modular Prosthetic Limb (MPL), a robot arm with many degrees of freedom and sensors used for the experiments in this work. When hand-tuning each GVF, a single appropriate step size α > 0 is typically chosen, resulting in the use of the same step size for every feature. TIDBD, however, adjusts a vector of many step sizes— one step size for each feature. For a binary feature vector x(s) ∈ R n , for example, there … view at source ↗

**Figure 2.** Decoded percept data from the robot over the 30 minutes of the experiment. The periods of the arm resting and the periods of the arm moving are clearly distinguishable for the position, velocity and load sensors. The values of the temperature sensors increase over the experiment, with additional increases during the periods of movement. τ . Kearney et al. (2019) suggest θ = 10−2 to be a good choice over a variety of… view at source ↗

**Figure 3.** RMSE and violin plots for the experiment in Section 4.1. The top pane shows the RMSE for both classic TD and TIDBD for each of the different time periods. The middle and bottom panes show violin plots for the RMSE, for TIDBD and classic TD respectively. All results are the average over 30 independent runs. Lastly, to investigate the robustness of TIDBD with respect to the initial step sizes α and its meta step size … view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Step-size development over the course of the experiment. Each subplot shows the step-size distribution for a snapshot Figure 4. Step-size development over the course of the experiment. As TIDBD adapts the step sizes, this [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Step sizes distribution for the four elbow sensors (a) and the remaining 104 sensors (b), when the four elbow sensors are stuck. As described in Subsection 4.2, the step sizes increase noticeably compared to the original experiment. The biggest step sizes are two times…

**Figure 6.** Figure 6: Step sizes distribution for the four elbow sensors (a) and the remaining 104 sensors (b), when the four elbow sensors are broken. The step sizes for the four broken sensors are noticeably reduced when compared to the experiment without broken sensors. In comparing [PI…

**Figure 7.** Figure 7: Accumulated RMSE over the experiment, depending on the initial step size. The first plot shows the overall accumulated error over the whole range of tested step sizes for TD and TIDBD with different meta step sizes θ. While the performance of TD dramatically worsens fo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 28 canonical work pages

[1]

M., Para, M

Bridges, M. M., Para, M. P., and Mashner, M. J. (2011). Control system architecture for the modular prosthetic limb. Johns Hopkins APL Technical Digest 30, 217–222

work page 2011
[2]

N., Roszko, D

Dalrymple, A. N., Roszko, D. A., Sutton, R. S., and Mushahwar, V . K. (2019). Pavlovian Control of Intraspinal Microstimulation to Produce Over-Ground Walking. In bioRxiv preprint bioRxiv:785741

work page 2019
[3]

Drescher, G. L. (1991). Made-Up Minds: A Constructivist Approach to Artiﬁcial Intelligence (MIT press)

work page 1991
[4]

L., Dawson, M

Edwards, A. L., Dawson, M. R., Hebert, J. S., Sherstan, C., Sutton, R. S., Chan, K. M., et al. (2016). Application of real-time machine learning to myoelectric prosthesis control: A case series in adaptive switching. Prosthetics and orthotics international 40, 573–581 This is an unpublished technical report undergoing peer review, not a ﬁnal typeset artic...

work page 2016
[5]

and White, A

Jacobsen, A., Schlegel, M., Linke, C., Degris, T. and White, A. and White, M. (2019). Meta-descent for online, continual prediction. In Proceedings of the thirty-third AAAI Conference on Artiﬁcial Intelligence, 3943–3950

work page 2019
[6]

Kahn, G., Villaﬂor, A., Ding, B., Abbeel, P., and Levine, S. (2018). Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In Proceedings of the International Conference on Robotics and Automation (IEEE), 1–8

work page 2018
[7]

Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning

Kearney, A., Veeriah, V ., Travnik, J., Pilarski, P. M., and Sutton, R. S. (2019). Learning feature relevance through step size adaptation in temporal-difference learning. arXiv preprint arXiv:1903.03252

work page Pith review arXiv 2019
[8]

and Yang, G.-H

Li, X.-J. and Yang, G.-H. (2012). Fault detection for linear stochastic systems with sensor stuck faults. Optimal control applications and methods 33, 61–80

work page 2012

Show all 29 references

[9]

R., Sutton, R

Mahmood, A. R., Sutton, R. S., Degris, T., and Pilarski, P. M. (2012). Tuning-free step-size adaptation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (IEEE), 2121–2124

2012
[10]

and Sutton, R

Modayil, J. and Sutton, R. S. (2014). Prediction driven behavior: Learning predictions that drive ﬁxed responses. In Workshops at the Twenty-Eighth AAAI Conference on Artiﬁcial Intelligence

2014
[11]

Modayil, J., White, A., and Sutton, R. S. (2014). Multi-timescale nexting in a reinforcement learning robot. Adaptive Behavior 22, 146–160

2014
[12]

Ni, K., Ramanathan, N., Chehade, M. N. H., Balzano, L., Nair, S., Zahedi, S., et al. (2009). Sensor network data fault types. ACM Transactions on Sensor Networks (TOSN) 5, 25

2009
[13]

M., Dawson, M

Pilarski, P. M., Dawson, M. R., Degris, T., Carey, J. P., Chan, K. M., Hebert, J. S., et al. (2013). Adaptive artiﬁcial limbs: A real-time approach to prediction and anticipation. IEEE Robotics & Automation Mag. 20, 53–64

2013
[14]

M., Sutton, R

Pilarski, P. M., Sutton, R. S., Mathewson, K. W., Sherstan, C., Parker, A. S., and Edwards, A. L. (2017). Communicative capital for prosthetic agents. arXiv preprint arXiv:1711.03676

2017 arXiv
[15]

Ring, M. B. (1994). Continual Learning in Reinforcement Environments. [PhD Thesis], [Austin (TX)]: University of Texas at Austin

1994
[16]

Schlegel, M., White, A., Patterson, A., and White, M. (2018). General value function networks. arXiv:1807.06763 [cs, stat]

2018 arXiv
[17]

and Dickinson, A

Schultz, W. and Dickinson, A. (2000). Neuronal coding of prediction errors.Annual Review of Neuroscience 23, 473–500

2000
[18]

C., and Pilarski, P

Sherstan, C., Machado, M. C., and Pilarski, P. M. (2018). Accelerating learning in constructive predictive frameworks with the successor representation. arXiv preprint arXiv:1803.09001

2018 arXiv
[19]

Sherstan, C., Modayil, J., and Pilarski, P. M. (2015). A collaborative approach to the simultaneous multi-joint control of a prosthetic arm. In Proceedings of the International Conference on Rehabilitation Robotics (IEEE), 13–18

2015
[20]

C., and Pilarski, P

Sherstan, C., White, A., Machado, M. C., and Pilarski, P. M. (2016). Introspective agents: Conﬁdence measures for general value functions. In Proceedings of the International Conference on Artiﬁcial General Intelligence (Springer), 258–261 This is an unpublished technical repo...

2016
[21]

Silver, D., Yang, Q., and Li, L. (2013). Lifelong machine learning systems: Beyond learning algorithms. In AAAI spring symposium

2013
[22]

Sinclair, A. H. and Barense, M. D. (2018). Surprise and destabilize: Prediction error inﬂuences episodic memory reconsolidation. Learning & Memory 25, 369–381

2018
[23]

Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine learning 3, 9–44

1988
[24]

Sutton, R. S. (1992). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI. 171–176

1992
[25]

Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (MIT press)

2018
[26]

S., Modayil, J., Delp, M., Degris, T., Pilarski, P

Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., et al. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proceedings of the International Conference on Autonomous Agents and Mult...

2011
[27]

Travnik, J. B. (2018). Reinforcement Learning on Resource Bounded Systems. Master’s thesis, University of Alberta

2018
[28]

Travnik, J. B. and Pilarski, P. M. (2017). Representing high-dimensional data to intelligent prostheses and other wearable assistive robots: A ﬁrst comparison of tile coding and selective Kanerva coding. In Proceedings of the International Conference on Rehabilitation Robotics...

2017
[29]

White, A. (2015). Developing a Predictive Approach to Knowledge . [PhD Thesis], [Edmonton (AB, Canada)]: University of Alberta This is an unpublished technical report undergoing peer review, not a ﬁnal typeset article. 19

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

M., Para, M

Bridges, M. M., Para, M. P., and Mashner, M. J. (2011). Control system architecture for the modular prosthetic limb. Johns Hopkins APL Technical Digest 30, 217–222

work page 2011

[2] [2]

N., Roszko, D

Dalrymple, A. N., Roszko, D. A., Sutton, R. S., and Mushahwar, V . K. (2019). Pavlovian Control of Intraspinal Microstimulation to Produce Over-Ground Walking. In bioRxiv preprint bioRxiv:785741

work page 2019

[3] [3]

Drescher, G. L. (1991). Made-Up Minds: A Constructivist Approach to Artiﬁcial Intelligence (MIT press)

work page 1991

[4] [4]

L., Dawson, M

Edwards, A. L., Dawson, M. R., Hebert, J. S., Sherstan, C., Sutton, R. S., Chan, K. M., et al. (2016). Application of real-time machine learning to myoelectric prosthesis control: A case series in adaptive switching. Prosthetics and orthotics international 40, 573–581 This is an unpublished technical report undergoing peer review, not a ﬁnal typeset artic...

work page 2016

[5] [5]

and White, A

Jacobsen, A., Schlegel, M., Linke, C., Degris, T. and White, A. and White, M. (2019). Meta-descent for online, continual prediction. In Proceedings of the thirty-third AAAI Conference on Artiﬁcial Intelligence, 3943–3950

work page 2019

[6] [6]

Kahn, G., Villaﬂor, A., Ding, B., Abbeel, P., and Levine, S. (2018). Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In Proceedings of the International Conference on Robotics and Automation (IEEE), 1–8

work page 2018

[7] [7]

Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning

Kearney, A., Veeriah, V ., Travnik, J., Pilarski, P. M., and Sutton, R. S. (2019). Learning feature relevance through step size adaptation in temporal-difference learning. arXiv preprint arXiv:1903.03252

work page Pith review arXiv 2019

[8] [8]

and Yang, G.-H

Li, X.-J. and Yang, G.-H. (2012). Fault detection for linear stochastic systems with sensor stuck faults. Optimal control applications and methods 33, 61–80

work page 2012