REVIEW 4 major objections 6 minor 14 references
Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Feeding a terrain classifier forecasts of future sensor signals, learned online by general value functions, improves its accuracy across walking surfaces, with the largest gains on the most error-prone terrains.
desk verdict A modest empirical comparison of where to inject GVF predictions into a terrain classifier, but the headline claim about predictive information is confounded by unmatched input width and capacity, and the title promise of walking control is untested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a two-level hierarchy built around general value functions (GVFs), which are value functions whose cumulant can be any sensor signal rather than a reward. A fast predictive state uses Selective Kanerva Coding to turn the 30 sensor channels into sparse binary prototype activations, and true online temporal-difference learning (TOTD) updates GVFs that predict the cumulative discounted value of each signal about 0.5 seconds ahead (17 timesteps at $\gamma = 0.94$). A slower policy network for terrain decisions is trained continually with a replay buffer, and receives these GVF predictions either at its input or in its latent space. The GVF outputs are the information-carrying component being tested; the policy network is what converts actual plus predicted signals into a terrain label.
What would settle it
Train the same policy network with the GVF output streams replaced one-for-one by non-predictive signals (fixed or shuffled values) while keeping identical input size and parameter count; if such a control matches the 73.4% and 72.0% accuracies, the specific contribution of predictive content would be disproved.
Extended reading notes
Core claim
The central claim is that learned predictions of future sensor values, not just current sensor values, improve a policy network's ability to classify walking terrain during simulated online learning. The paper shows this by adding general value function outputs to a continual deep learning policy network in two ways: concatenated with the actual sensor signals at the input, and injected after the encoding layers in the latent space. Both additions raised end-of-training accuracy significantly relative to the no-GVF control (Input GVF vs. Policy Net: $p = 0.001$; Latent GVF vs. Policy Net: $p = 0.01$), while the two integration sites were not statistically distinguishable ($p = 0.42$). The authors interpret the terrain-specific gains, especially the 13.5-point improvement on down ramps, as predictive information helping when the classifier is most uncertain.
Load-bearing premise
The load-bearing assumption is that the accuracy gains come from the predictive meaning of the GVF signals, not from giving the network more inputs or more parameters, since the control network had fewer inputs and the GVF networks had extra capacity without a matched control.
Editorial extensions
If this is right
- A GVF-augmented policy net can be paired with an exoskeleton controller to choose terrain-appropriate walking strategies without manual mode switching.
- Terrain-specific gains on ramps, uneven ground, even ground, and turns suggest predictive information is most valuable exactly where classifiers are most error-prone.
- Since input and latent integration performed similarly overall, the simpler input-concatenation design may be sufficient for exoskeleton control, pending further analysis.
- Continual replay-based training combined with online GVF learning forms a plausible architecture for controllers that adapt as a user encounters new environments.
Reading between the lines
- A testable extension not run in the paper would be a capacity-matched and input-count-matched baseline; without it, part of the gain may be attributable to the larger effective feature space rather than prediction content.
- The predictive-information benefit is probably strongest when the classifier's uncertainty is high; one could test this by correlating per-timestep prediction error with misclassification rate.
- The same GVF-plus-policy architecture could be applied to upper-limb prosthetics or gait-phase detection, where anticipating the next sensor state is similarly useful.
- Future adaptive controllers could use the GVF prediction error itself as a signal for when the terrain has changed, triggering a strategy switch before misclassification occurs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hierarchical reinforcement learning (HRL) pipeline for adaptive exoskeleton control, in which lower-limb sensor signals (EMG, goniometer, pressure insoles) are used to learn general value functions (GVFs) that predict signal values roughly 0.5 s into the future. These GVF predictions are then integrated into a terrain-classification policy network in one of two ways: appended to the raw input stream or injected into a latent layer. Using data from ten participants walking over seven terrains, the authors compare these two GVF-augmented networks against a control policy net that receives only raw sensor signals. They report that both GVF-augmented networks achieve significantly higher end-of-training classification accuracy (73.4% and 72.0%) than the control (66.9%), with particularly large gains on down-ramp classification, and they conclude that predictive GVF information improves decision-making during terrain uncertainty.
Significance. If the central claim is valid, the paper provides a useful demonstration that learned temporal abstractions can improve online terrain classification from wearable sensor data, which is a plausible step toward adaptive lower-limb exoskeleton control. The work has concrete strengths: it uses human gait data, implements continual learning with a replay buffer, and reports per-terrain statistical comparisons. However, the empirical comparison does not isolate the predictive content of GVFs from the effects of added input dimensions and network capacity, and the evaluation appears to be confined to the training data stream with no held-out protocol. These issues must be resolved before the mechanistic conclusion that GVF predictions themselves improve accuracy can be accepted.
major comments (4)
- [Section 2 (Policy Network for Terrain Control Decisions) and Section 3 (Table 1)] The central comparison is confounded by input dimensionality and model capacity. The control Policy Net uses 30 raw sensor inputs, the Input GVF Policy Net uses 60 inputs (30 raw plus 30 GVF predictions), and the Latent GVF Policy Net adds an additional pathway after the encoding layer, increasing the number of parameters in subsequent layers. No control condition matches the input count or capacity while withholding the predictive semantics of GVFs. The reported significant accuracy differences (p = 0.001 and p = 0.01) could therefore arise simply from providing more features or more model capacity. A control using 30 time-delayed copies of the raw signals, or a larger-capacity network without GVF inputs, would directly address this concern. This is load-bearing because the Abstract and Conclusion attribute the improvement specifically to 'the addition of predictions made from GVFs.'
- [Section 3 (Figure 2 and Table 1)] All reported accuracies appear to be end-of-training values computed on the same data stream used to train the policy networks. The paper does not describe any held-out evaluation, such as a per-subject holdout, a temporal train/test split, or cross-validation. Without such a protocol, the accuracy improvements may reflect the networks' ability to memorize the training stream rather than generalize to unseen walking data. The manuscript should specify the evaluation protocol explicitly and report at least one held-out metric, even if the authors intend the paper as a demonstration of online learning rather than generalization.
- [Section 2 (Statistics) and Section 3] The statistical analysis lacks a clear statement of the unit of analysis and the number of independent samples used in the Kruskal-Wallis and Dunn tests. It is unclear whether the reported accuracies are per-participant end-of-training values (n = 10), per-gait-cycle values, or pooled timestep-level results. The number of training runs per participant, random seeds, and whether the same data are reused across networks are also not stated. This matters because the validity of the p-values depends on the independence and sample size of the accuracies being compared.
- [Section 1, Section 2, and Figure 1] The paper frames the work as an HRL framework for adaptive exoskeleton control, but the experiments stop at terrain classification; no exoskeleton control task, actuator policy, or closed-loop user study is presented. The abstract's statement that the goal is 'adaptive control strategies for lower-limb exoskeletons' and Figure 1's inclusion of an exoskeleton block overstate the demonstrated scope. This is acceptable as a preliminary study if the claims are softened or if a limitations paragraph explicitly states that exoskeleton control is future work, but as written the title and framing promise more than the experiments deliver.
minor comments (6)
- [Title and Section 2 headings] There are typographical spacing errors in 'General V alue Functions' (title) and 'T rue Online Temporal-difference Learning' (Section 2 heading). These should be corrected to 'Value' and 'True'.
- [Algorithm 1] The comment 'Offest (m - 1) * K' should read 'Offset'; more importantly, the intended indexing of the active prototype features is unclear and should be clarified in the text or pseudocode.
- [Section 2 (GVFs)] The equation for the number of prediction timesteps is written as 'timesteps = 1 1−γ', which is missing an explicit division and is visually ambiguous. It should be typeset as 1/(1−γ).
- [Section 2 (data preprocessing)] The phrase 'filtered with a 2nd-order low-pass filter at 5 Hz' should specify the filter type (e.g., Butterworth) and, ideally, the reason for the 5 Hz cutoff, to aid reproducibility.
- [Figure 2] The convergence curves are shown without a caption describing whether they are averaged over participants, how many runs are included, and what the shaded regions represent. Adding this information would make the figure interpretable.
- [Section 2 (Policy Network)] The total number of training timesteps or epochs and the number of replay-buffer updates are not specified; adding these details would allow readers to understand the duration of the online learning process and the convergence behavior.
Circularity Check
No significant circularity: GVF predictions are learned from sensor signals, not from terrain labels, and no load-bearing self-citation chain reduces the result to its inputs.
full rationale
The paper's claimed derivation has no circular step. GVFs are trained to approximate discounted future sensor-signal cumulants via TOTD (Section 2, GVF and TOTD subsections); the policy networks are trained to classify terrain labels. The two learning targets are distinct, and no equation defines the terrain-classification accuracy in terms of the GVF cumulant or defines the GVF cumulant in terms of the terrain label. The reported gains (73.4% and 72.0% vs. 66.9%) are empirical comparisons, not algebraic consequences of the definitions. The author-affiliated citations used in the paper ([5], [6], [8], [9]) supply prior methods and motivation (GVF-as-input, SKC, coarse/fine coding); they are not invoked as a uniqueness theorem or as the proof of the central claim, and the paper's own data and statistical tests independently carry the result. The most serious limitation is a missing input-count/capacity-matched control: the control net has 30 inputs while the Input GVF net has 60, and the Latent net adds a parallel pathway. This confounds the causal interpretation that predictive content, rather than added features or parameters, drove accuracy, but this is an experimental-design confound, not a circular derivation. The paper itself acknowledges inconclusiveness in Section 4: 'it remains inconclusive whether adding these predictions to the input or latent layers is superior.' Therefore no circularity is established.
Assumptions & free parameters
free parameters (7)
- SKC prototype count K=5000 =
5000
- Active prototype sets {500, 100, 25} =
500, 100, 25
- GVF discount factor gamma=0.94 =
0.94
- TOTD eligibility trace lambda=0.5 =
0.5
- Policy net learning rate alpha=0.001 =
0.001
- Replay buffer size and batch composition =
buffer 1000, batch 32, 50/50 new-replay
- Network architecture (decreasing-size encoding layers) =
not specified
assumptions (5)
- standard math Euclidean distance and quickselect on prototype distances define a valid state representation (SKC).
- standard math True online TD learning converges to the GVF for a fixed policy and function approximation.
- domain assumption The recorded sensor signals and hand-marked terrain transitions are synchronized and accurate.
- domain assumption Gait sensor signals from 10 participants not wearing an exoskeleton are representative of signals an exoskeleton controller would face.
- standard math Normality testing and the Kruskal-Wallis and Dunn procedures correctly model the accuracy distributions.
Cite this review
Pith. "Pith review of Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals." pith.science (2026). https://pith.science/paper/B4V5C7VA
@misc{pith2026250716983,
author = {Pith},
title = {Pith review of: Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals},
year = {2026},
howpublished = {\url{https://pith.science/paper/B4V5C7VA}},
note = {Machine review of arXiv:2507.16983}
}
read the original abstract
Rehabilitation technology is a natural setting to study the shared learning and decision-making of human and machine agents. In this work, we explore the use of Hierarchical Reinforcement Learning (HRL) to develop adaptive control strategies for lower-limb exoskeletons, aiming to enhance mobility and autonomy for individuals with motor impairments. Inspired by prominent models of biological sensorimotor processing, our investigated HRL approach breaks down the complex task of exoskeleton control adaptation into a higher-level framework for terrain strategy adaptation and a lower-level framework for providing predictive information; this latter element is implemented via the continual learning of general value functions (GVFs). GVFs generated temporal abstractions of future signal values from multiple wearable lower-limb sensors, including electromyography, pressure insoles, and goniometers. We investigated two methods for incorporating actual and predicted sensor signals into a policy network with the intent to improve the decision-making capacity of the control system of a lower-limb exoskeleton during ambulation across varied terrains. As a key result, we found that the addition of predictions made from GVFs increased overall network accuracy. Terrain-specific performance increases were seen while walking on even ground, uneven ground, up and down ramps, and turns, terrains that are often misclassified without predictive information. This suggests that predictive information can aid decision-making during uncertainty, e.g., on terrains that have a high chance of being misclassified. This work, therefore, contributes new insights into the nuances of HRL and the future development of exoskeletons to facilitate safe transitioning and traversing across different walking environments.
Figures
Reference graph
Works this paper leans on
-
[1]
S. Luo et al., “Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning,” J. NeuroEngineering Rehabil., vol. 20, no. 1, p. 34, 2023
work page 2023
-
[2]
Walking adaptability after a stroke and its assessment in clinical settings,
C. K. Balasubramanian, D. J. Clark, and E. J. Fox, “Walking adaptability after a stroke and its assessment in clinical settings,” Stroke Research and Treatment, vol. 2014, pp. 1–21, 2014
work page 2014
-
[3]
Review of control strategies for lower-limb exoskeletons to assist gait,
Baud, Romain et al., “Review of control strategies for lower-limb exoskeletons to assist gait,” J. NeuroEngineering Rehabil., vol. 18,1 119. 27 Jul. 2021
work page 2021
-
[4]
Learning how to Interact with a Complex Interface using Hierarchical Reinforcement Learning
G. Comanici et al., “Learning how to interact with a complex interface using hierarchical reinforcement learning,” arXiv:2204.10374v1 [cs.LG], 21 Apr 2022
work page Pith review arXiv 2022
-
[5]
P . M. Pilarski, T. B. Dick, and R. S. Sutton, ”Real-time prediction learning for the simultaneous actuation of multiple prosthetic joints,” IEEE 13th Int. Conf. on Rehabilitation Robotics (ICORR), Seattle, WA, USA, June 24-26, 2013, pp. 1–8
work page 2013
-
[6]
Gamma-Nets: Generalizing value estimation over timescale,
C. Sherstan et al., “Gamma-Nets: Generalizing value estimation over timescale,” Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), February 7–12, New York, New York USA, 2020, 10 pages
work page 2020
-
[7]
Experience replay for continual learning,
D. Rolnick et al., “Experience replay for continual learning,” Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, Canada, Dec. 8–14, 2019
work page 2019
-
[8]
J. B. Travnik and P . M. Pilarski, “Representing high-dimensional data to intelligent prostheses and other wearable assistive robots: A first comparison of tile coding and selective Kanerva coding,” 15th IEEE-RAS-EMBS Int. Confer- ence on Rehabilitation Robotics (ICORR 2017), July 17–20, 2017, QEII Centre, London, UK, pp. 1443–1450
work page 2017
Show all 14 references
-
[9]
Pavlovian control of intraspinal microstimulation to produce over-ground walking,
A. N. Dalrymple et al., “Pavlovian control of intraspinal microstimulation to produce over-ground walking,” J. Neural Eng., vol. 17, no. 3, p. 036002, 2020
2020
-
[10]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd Edition. MIT Press, Cambridge, MA, 2018
2018
-
[11]
Developing a predictive approach to knowledge
Adam White. Developing a predictive approach to knowledge. PhD thesis, University of Alberta, 2015
2015
-
[12]
Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction,
R. S. Sutton et al., “Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction,” 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , Taipei, Taiwan, May 2-6, 2011, Volume 1-3, pages 761–768
2011
-
[13]
Learning to predict by the methods of temporal differences,
R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine Learning, vol. 3, no. 1, pp. 9–44, 1988
1988
-
[14]
True online temporal-difference learning,
H. van Seijen et al., “True online temporal-difference learning,” Journal of Machine Learning Research, vol. 17, no. 145, pp. 1-40, 2016 4
2016
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.