Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Investigating Adaptive Tuning of Assistive Exoskeletons Using Offline Reinforcement Learning: Challenges and Insights

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Offline reinforcement learning can adapt exoskeleton effort thresholds from data.

desk verdict Reward function sets threshold equal to muscle effort, which under the paper's own control law puts the device at the activation boundary and prevents movement; the central claim is also missing any policy evaluation. read the letter →

arxiv 2505.00201 v1 pith:LQQOCVEX submitted 2025-04-30 cs.RO

Yasin Findik , Christopher Coco , Reza Azadeh This is my paper · ORCID
classification cs.RO
keywords offlinereinforcementlearningmulti-agentMixedQ-FunctionalsassistiveexoskeletonMyoPro2sEMGeffortthresholdscontinuousactionspacesadaptiveparametertuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an offline multi-agent reinforcement learning framework can replace manual calibration of an upper-limb exoskeleton's effort thresholds. Using the MyoPro 2 device, the authors treat the biceps threshold and the triceps threshold as separate cooperative agents and train them on pre-collected sessions with Mixed Q-Functionals (MQF), a value-based method for continuous actions. Their experiments on horizontal and vertical arm tasks indicate that the learned agents adjust thresholds to task-dependent values, low biceps thresholds for the vertical task and a $(20, 25)$ pair for the horizontal task, which, if correct, means assistance can be personalized without risky online trial-and-error. The authors also stress that evaluating these learned thresholds is difficult because new threshold values can lead to states absent from the offline dataset.

What carries the argument

The load-bearing mechanism is the Mixed Q-Functional (MQF), a value-based representation in which a neural network maps each state $s$ to coefficients $C(s)$ of basis functions over the continuous action space, and the Q-value of action $a$ is computed as $Q_F(s,a)=C(s)^\top \Phi(a)$. This turns evaluation of many candidate threshold actions into matrix operations, letting two cooperative agents, one for the biceps threshold and one for the triceps threshold, choose continuous values without policy-gradient sampling. A mixer network, using a VDN-style sum, combines agent Q-values into $Q_{\mathrm{total}}$ for centralized temporal-difference training, while target networks are updated softly each step.

What would settle it

Withhold one threshold pair from training, let the trained agents pick thresholds for the same participant and task, and record whether the resulting motor angle and sEMG effort states fall inside the training distribution and yield the predicted rewards; if the agents consistently produce out-of-distribution states or rewards below static baselines, the central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that offline value-based multi-agent RL can learn a continuous policy over effort thresholds from a fixed dataset of exoskeleton interactions. Each agent's Q-function is a Mixed Q-Functional mapping states to polynomial coefficients over the action interval $[20, 50]$; a VDN-style mixer combines the agents' values, and training minimizes temporal-difference error over logged transitions. The resulting policies select lower biceps thresholds when biceps effort dominates (vertical task) and a biceps/triceps pair of $(20, 25)$ when triceps effort dominates (horizontal task). The paper's claim is that this demonstrates adaptive, task-specific threshold tuning from data, while openly noting that the absence of real-time or out-of-distribution evaluation leaves user-level benefit as a potential rather than a measured outcome.

Load-bearing premise

The load-bearing premise is that the pre-collected dataset, from one participant with thresholds in steps of five, covers enough of the state-action space that the learned continuous threshold policy can be trusted on values and resulting states it never saw; the paper itself flags this in Section IV.C.

Editorial extensions

If this is right

  • If the claim holds, exoskeleton hyperparameters such as effort thresholds can be tuned without online exploration, removing a safety barrier to personalized assistive control.
  • The observed task-dependent optima mean that a single static calibration is suboptimal, and per-task learned adjustment should improve smoothness and user effort.
  • The multi-agent formulation provides a template for tuning several continuous control parameters at once, not just biceps and triceps thresholds.
  • The reward $r=e^{-d/c}$, which penalizes deviation between delta effort and the threshold, is a reusable objective for "just enough assistance" in other sEMG-driven devices.
  • Reliable evaluation will require either a learned transition model or real-world user studies, which the paper explicitly identifies as the path forward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A concrete test the paper does not run: train on thresholds from $\{20,25,30,35,40,45,50\}$ but hold out one combination, then see whether MQF generalizes; this would isolate dataset-coverage effects from algorithm capacity.
  • The same MQF pipeline could tune the proportional gain $k_p$ or ramp parameters, since those are additional continuous action dimensions, offering a direct extension beyond thresholds.
  • Because the paper's evaluation cannot observe out-of-distribution outcomes, a model-based offline RL variant that learns a transition model of motor angle and sEMG effort would make learned thresholds checkable without real-time testing.
  • With multiple participants, per-user threshold policies could be compared against the single-participant policy; if muscle-activation patterns differ across users, the current dataset may not yet support the personalized-assistance conclusion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an offline multi-agent reinforcement learning framework, based on Mixed Q-Functionals (MQF), to automatically adjust the biceps and triceps effort thresholds of the MyoPro 2 upper-limb exoskeleton from pre-collected data. Two agents are assigned to the two thresholds, and the reward is defined as r = exp(-d/c), where d is the absolute difference between the current delta effort and the threshold of the dominant muscle. Data were collected from a single participant performing horizontal and vertical arm movement tasks with thresholds varied in increments of 5 from 20 to 50. The authors report that their approach can dynamically adjust thresholds based on learned patterns and may improve user interaction, while acknowledging that evaluating performance remains difficult because generated actions may lead to out-of-distribution states.

Significance. If the central claim were established, the paper would address a practically important problem: reducing the manual recalibration burden for assistive exoskeletons. The framing of threshold tuning as a multi-agent offline RL problem is reasonable, and using MQF for continuous action spaces is a sensible algorithmic choice. However, the significance is currently undercut by two factors. First, the paper contains no empirical evaluation of the learned policy: no rollouts, no comparison with static thresholds, no reward or learning curves, and no quantitative outcome metrics. Second, and more fundamentally, the reward function is misaligned with the device dynamics described earlier in the same paper; maximizing it would drive thresholds to track the instantaneous delta effort, which places the device at the activation boundary and suppresses movement. The paper is honest about the dataset limitations and the OOD problem, which is commendable, but the reported results do not support the abstract's claim that the approach can improve user interaction and control.

major comments (3)
  1. [Section IV.C, Reward function design] The reward r = exp(-d/c), with d = |ΔE − threshold|, is maximized when threshold = ΔE, regardless of the positive constant c. Section II.B states that movement is initiated only when ΔE exceeds the threshold and that S_joint = k_p·ΔE. At threshold = ΔE, the device sits exactly at the activation boundary, producing no movement (under a strict inequality) or zero proportional speed (under a non-strict inequality). Thus the learning objective actively prefers threshold settings that suppress assistive movement. This is not merely an evaluation or coverage issue: even with ideal offline coverage and a perfect function approximator, a policy maximizing this reward would output thresholds that track the instantaneous delta effort, which is the opposite of a usable activation threshold. The reward must be redefined in terms of task success, movement smoothness, or user effort, and the revised objective must be justified with respect to the device dynamics in Section II.B.
  2. [Section IV.C, Findings] The paper reports no results from the trained models. Figures 6 and 7 show raw sensor data for selected static threshold combinations, not the behavior of the learned policies. There are no learned threshold trajectories, no comparison of learned thresholds to static baselines, no training reward curves, and no quantitative outcome metrics such as task completion time, user effort, or number of successful movements. The statement that 'the models can dynamically set thresholds for the muscles' is therefore unsupported by any reported experiment. To substantiate the central claim, the authors need to evaluate the learned policies, for example by logging the thresholds selected by the agents on held-out episodes, comparing their resulting behavior with the static thresholds in the dataset, and reporting task-relevant performance measures.
  3. [Section IV.B and Section IV.C, Data coverage and OOD actions] The load-bearing premise of the approach is that the offline dataset adequately covers the state-action distribution so that a learned continuous-action policy can generalize to new thresholds. The dataset, however, was collected from a single participant with thresholds quantized in increments of 5, and only 10 episodes per combination. The paper itself concedes in Section IV.C that 'when the models generate new actions (i.e., threshold values), they may lead to states that are not present in the dataset, making their outcomes unknown.' This concession directly undermines the claim that the framework can learn to adjust thresholds effectively. The authors should either restrict the action space to the observed threshold values, incorporate a learned transition model to estimate outcomes of unseen actions, or conduct real-world user studies to evaluate the learned thresholds; without one of these, the central claim remains unverified.
minor comments (4)
  1. [Section IV.C, State representation] The sentence 'the triceps agent modifies thbiceps' should read 'the triceps agent modifies thtriceps'; as written, both agents are described as adjusting the same threshold.
  2. [Section IV.C and Figure 7 caption] The caption of Figure 7 references 'Fig. 5(b)' when describing the horizontal task, but the horizontal task is shown in Figure 5(a). Please correct the cross-reference.
  3. [Throughout] There are several typographical errors, e.g., 'excees' in Section II.B and 'a upper-limb' in the conclusion. A careful proofreading pass is recommended.
  4. [Section II.B] The terminology is inconsistent: the threshold is described as 'activation levels required to initiate movement' but the control law uses delta effort ΔE. Clarify whether the threshold applies to ΔE or to the individual muscle effort, since this affects the reward design and the interpretation of Figures 6 and 7.

Circularity Check

1 steps flagged · score 8.0 of 10

The reward function makes the central 'improvement' circular: maximizing r = exp(-|Delta E - th|/c) by definition sets th = Delta E, which by the paper's own activation rule suppresses movement.

  1. self definitional [Section IV.C, 'Reward function design'; compared with Section II.B, 'Control Modes' (S_joint = k_p * Delta E).]
    "To guide learning, we define a reward function for each state-action pair, formulated as: r = e−d/c, where d represents the difference between ∆E and the threshold value of the dominant muscle, and c is a constant. The objective is to minimize this difference, ensuring that the exoskeleton provides continuous and adaptive assistance throughout task execution."

    For any c>0, argmax_th exp(-|Delta E - th|/c) = th = Delta E. The paper's own rule (Section II.B) requires Delta E to 'exceed the predefined threshold' before S_joint = k_p * Delta E is produced, so the reward's maximizing action sits exactly at the activation boundary and produces no (or zero-proportional) movement. The learned 'improvement' is thus defined as copying the current delta effort, an input feature, rather than any externally measured assistive benefit. The claim that thresholds are adjusted 'based on learned patterns, potentially improving user interaction and control' restates the reward construction; the paper itself adds that outcomes for generated actions are unknown, so no external benchmark breaks the circle.

full rationale

The paper's central claim that the proposed offline RL approach can improve user interaction by dynamically tuning thresholds rests entirely on the reward r = exp(-d/c), where d is the difference between the current delta effort and the threshold action. By construction, the reward is maximized when the threshold equals the current delta effort. The paper's own device model in Section II.B states that movement is initiated only when delta effort exceeds the threshold, so the reward's optimum is the boundary condition that prevents movement. Thus any policy trained to maximize this reward is simply reproducing the current input, and the reported 'potential improvement' is a self-referential consequence of the reward definition rather than an empirically validated outcome. The paper also explicitly admits that evaluating the learned thresholds is difficult because generated actions may lead to states not present in the dataset and their outcomes are unknown, and it reports no learned threshold trajectories, reward curves, or comparisons against static baselines. The self-citation of Mixed Q-Functionals by one of the authors is not treated as circular here, because the algorithm choice does not by itself force the claimed result; the circularity is in the reward objective. Because the central positive claim reduces by definition to copying delta effort, the score is high, reflecting partial-to-strong circularity rather than a mere lack of evaluation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the dataset covering the state-action space, on a hand-defined reward with an unspecified constant, and on the validity of adapting MQF to offline learning. Two free parameters (c and basis order) are left unspecified. No new physical entities are introduced.

free parameters (2)
  • reward constant c
    Appears in r=exp(-d/c) in Section IV.C; its value is never given, and it controls the reward sharpness for threshold mismatches.
  • MQF basis polynomial order O
    Section III states the basis is assumed order 2 for illustration, but the actual order used in training is not reported; this determines the number of coefficients k.
assumptions (3)
  • domain assumption The offline dataset captures enough state-action diversity for a continuous-action policy to generalize.
    Section IV.B collects data with thresholds only at increments of 5 from one participant; Section IV.C admits generated actions may lead to states not in the dataset, so generalization is assumed rather than established.
  • ad hoc to paper The reward r=exp(-d/c) faithfully represents user experience and control quality.
    Section IV.C defines the reward without user studies or validation; the constant c is unspecified.
  • domain assumption MQF value-based offline learning with uniformly sampled max over actions is a valid offline RL objective.
    Section III adapts MQF to offline settings but provides no justification that max over uniform action samples yields a stable target for out-of-distribution actions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Adaptive Tuning of Assistive Exoskeletons Using Offline Reinforcement Learning: Challenges and Insights." pith.science (2026). https://pith.science/paper/LQQOCVEX

@misc{pith2026250500201,
  author       = {Pith},
  title        = {Pith review of: Investigating Adaptive Tuning of Assistive Exoskeletons Using Offline Reinforcement Learning: Challenges and Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LQQOCVEX}},
  note         = {Machine review of arXiv:2505.00201}
}
read the original abstract

Assistive exoskeletons have shown great potential in enhancing mobility for individuals with motor impairments, yet their effectiveness relies on precise parameter tuning for personalized assistance. In this study, we investigate the potential of offline reinforcement learning for optimizing effort thresholds in upper-limb assistive exoskeletons, aiming to reduce reliance on manual calibration. Specifically, we frame the problem as a multi-agent system where separate agents optimize biceps and triceps effort thresholds, enabling a more adaptive and data-driven approach to exoskeleton control. Mixed Q-Functionals (MQF) is employed to efficiently handle continuous action spaces while leveraging pre-collected data, thereby mitigating the risks associated with real-time exploration. Experiments were conducted using the MyoPro 2 exoskeleton across two distinct tasks involving horizontal and vertical arm movements. Our results indicate that the proposed approach can dynamically adjust threshold values based on learned patterns, potentially improving user interaction and control, though performance evaluation remains challenging due to dataset limitations.

Figures

Figures reproduced from arXiv: 2505.00201 by the authors.

Figure 1
Figure 1. Overview of the MyoPro 2 device, highlighting the place￾ment of sEMG sensors. data-driven techniques for more adaptive and responsive control [10]. Consequently, research has increasingly focused on developing automated hyper-parameter tuning methods to enhance the adaptability and usability of these systems [11]. Building on these advancements, we explore a reinforce￾ment learning-based approach to automate paramet… view at source ↗
Figure 2
Figure 2. Illustration of joint speed as a function of delta effort, highlighting the relationship between biceps and triceps activation. The graph is divided into three regions: flex, where triceps activation exceeds the threshold, extend, where biceps activation surpasses the threshold, and idle, where neither muscle reaches activation. The placement of triceps and biceps thresholds is marked, demonstrating their influence … view at source ↗
Figure 3
Figure 3. Visualization of muscle activation patterns, corresponding effort levels, and resulting actions. The left column categorizes different activation scenarios, including biceps contraction, triceps contraction, co-contraction, and imbalanced muscle activation. The middle column represents effort levels using effort bars, where biceps and triceps contributions are indicated. The right column shows the resulting action, … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of the proposed offline MARL architecture with MQF: (a) training prediction blocks, (b) updating target blocks. The red arrows indicate the direction of the backpropagation, while the blue arrows depict the target network updates for mixer and learners’ predic…
Figure 5
Figure 5. Figure 5: Demonstration of task movements for data collection using the MyoPro exoskeleton. (a) Depicts the horizontal task, where the user moves their arm laterally. (b) Shows the vertical task, involving an upward and downward arm motion. uniform distribution. The function Qto…
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Analysis of data collected from the horizontal task ( [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages

  1. [1]

    Powered exoskeletons for walking assistance in persons with central nervous system injuries: a narrative review,

    A. Esquenazi, M. Talaty, and A. Jayaraman, “Powered exoskeletons for walking assistance in persons with central nervous system injuries: a narrative review,” PM&R, vol. 9, no. 1, pp. 46–62, 2017

    work page 2017

  2. [2]

    Review of control strategies for lower-limb exoskeletons to assist gait,

    R. Baud, A. R. Manzoori, A. Ijspeert, and M. Bouri, “Review of control strategies for lower-limb exoskeletons to assist gait,” Journal of NeuroEngineering and Rehabilitation , vol. 18, pp. 1–34, 2021

    work page 2021

  3. [3]

    Wearable assistive robotics: A perspective on current challenges and future trends,

    U. Martinez-Hernandez, B. Metcalfe, T. Assaf, L. Jabban, J. Male, and D. Zhang, “Wearable assistive robotics: A perspective on current challenges and future trends,” Sensors, vol. 21, no. 20, p. 6751, 2021

    work page 2021

  4. [4]

    How adaptation, training, and customization contribute to benefits from exoskeleton assistance,

    K. L. Poggensee and S. H. Collins, “How adaptation, training, and customization contribute to benefits from exoskeleton assistance,” Science Robotics, vol. 6, no. 58, p. eabf1078, 2021

    work page 2021

  5. [5]

    Developments in hardware systems of active upper-limb exoskeleton robots: A review,

    R. Gopura, D. Bandara, K. Kiguchi, and G. K. Mann, “Developments in hardware systems of active upper-limb exoskeleton robots: A review,” Robotics and Autonomous Systems , vol. 75, pp. 203–220, 2016

    work page 2016

  6. [6]

    Control strategies used in lower limb ex- oskeletons for gait rehabilitation after brain injury: a systematic review and analysis of clinical effectiveness,

    J. de Miguel-Fern ´andez, J. Lobo-Prat, E. Prinsen, J. M. Font-Llagunes, and L. Marchal-Crespo, “Control strategies used in lower limb ex- oskeletons for gait rehabilitation after brain injury: a systematic review and analysis of clinical effectiveness,” Journal of neuroengineering and rehabilitation, vol. 20, no. 1, p. 23, 2023

    work page 2023

  7. [7]

    An adaptive assistance controller to optimize the exoskeleton contribution in rehabilitation,

    R. Nasiri, M. Shushtari, and A. Arami, “An adaptive assistance controller to optimize the exoskeleton contribution in rehabilitation,” Robotics, vol. 10, no. 3, p. 95, 2021

    work page 2021

  8. [8]

    Dynamic switching and real-time machine learning for improved human control of assistive biomedical robots,

    P. M. Pilarski, M. R. Dawson, T. Degris, J. P. Carey, and R. S. Sutton, “Dynamic switching and real-time machine learning for improved human control of assistive biomedical robots,” in 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob). IEEE, 2012, pp. 296–302

    work page 2012

Show all 21 references
  1. [9]

    Personal- izing exoskeleton assistance while walking in the real world,

    P. Slade, M. J. Kochenderfer, S. L. Delp, and S. H. Collins, “Personal- izing exoskeleton assistance while walking in the real world,” Nature, vol. 610, no. 7931, pp. 277–282, 2022

    2022

  2. [10]

    Human-in-the-loop optimiza- tion of shared autonomy in assistive robotics,

    D. Gopinath, S. Jain, and B. D. Argall, “Human-in-the-loop optimiza- tion of shared autonomy in assistive robotics,” IEEE robotics and automation letters, vol. 2, no. 1, pp. 247–254, 2016

    2016

  3. [11]

    Design of fuzzy logic parameter tuners for upper-limb assistive robots,

    C. Coco Jr, J. Spanos, H. Osooli, and R. Azadeh, “Design of fuzzy logic parameter tuners for upper-limb assistive robots,” WIP paper at 21st International Conference on Ubiquitous Robots (UR), New York, USA, pp. 386–389, June 24-27 , 2024

    2024

  4. [12]

    Myomo – The MyoPro Brace for Stroke Survivor’s Paralyzed Arm

    M. Myomo Inc., Cambridge, “Myomo – The MyoPro Brace for Stroke Survivor’s Paralyzed Arm.” [Online]. Available: https://myomo.com

  5. [13]

    Mixed q-functionals: Advancing value-based methods in cooperative marl with continuous action domains,

    Y . Findik and S. R. Ahmadzadeh, “Mixed q-functionals: Advancing value-based methods in cooperative marl with continuous action domains,” arXiv preprint arXiv:2402.07752 , 2024

    2024 arXiv

  6. [14]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  7. [15]

    Batch reinforcement learning,

    S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” in Reinforcement learning: State-of-the-art . Springer, 2012, pp. 45– 73

    2012

  8. [16]

    Value-decomposition networks for cooperative multi-agent learning,

    P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296 , 2017

    2017 arXiv

  9. [17]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in International Conference on Machine Learning. PMLR, 2018, pp. 4295–4304

    2018

  10. [18]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,

    K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y . Yi, “Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,” in International conference on machine learn- ing. PMLR, 2019, pp. 5887–5896

    2019

  11. [19]

    Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, G. Farquhar, B. Peng, and S. Whiteson, “Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning,” Advances in neural information processing systems, vol. 33, pp. 10 199–10 210, 2020

    2020

  12. [20]

    Continuous control with deep reinforce- ment learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforce- ment learning,” arXiv preprint arXiv:1509.02971 , 2015

    2015 arXiv

  13. [21]

    Investigating the generalizability of assistive robots models over various tasks,

    H. Osooli, C. Coco, J. Spanos, A. Majdi, and R. Azadeh, “Investigating the generalizability of assistive robots models over various tasks,” in 2024 21st International Conference on Ubiquitous Robots (UR) . IEEE, 2024, pp. 227–232

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.