{"id":"b7c76e7d-2bb8-4c8e-a325-a12cb6ff6fd5","arxiv_id":"2505.00201","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Offline multi-agent RL with Mixed Q-Functionals is applied to MyoPro 2 effort-threshold tuning, but the paper presents no validated evaluation of the learned policies, only raw data analyses.","lead":"This paper tests whether offline reinforcement learning can automatically tune the effort thresholds of a MyoPro 2 assistive exoskeleton, using two agents to adjust biceps and triceps sensitivity. It reports that the approach can adjust thresholds in principle, but provides no quantitative evidence that the learned settings improve or even match manual calibration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward function is degenerate: maximizing r = exp(-|delta E - th|/c) sets the threshold equal to the current delta effort, which under Section II.B leaves the exoskeleton at the activation boundary and yields no movement; this contradicts the central claim and is more fundamental than the…","rationale":"The reader's chosen weakest assumption is the out-of-distribution coverage of the single-participant, 5-increment dataset. That is a genuine and explicitly disclosed limitation. However, I see a more load-bearing internal inconsistency: the reward function defined in Section IV.C is maximized by setting the threshold equal to the current delta effort, whereas Section II.B defines movement as requiring delta effort to exceed the threshold. A policy that optimizes this reward would therefore drive the system to the activation boundary and provide no assistive movement. This is not a matter of dataset completeness or evaluation methodology; it is a mismatch between the learning objective and the device's control law. The reader's rationale does mention that the reward makes in-distribution success a consequence of the reward definition, but the weakest_assumption field identifies coverage instead. I consider my concern the same family of validity problems, hence 'partial' agreement. Since this concern reinforces rather than changes the reject verdict, the appropriate verdict_should_be is UNCHANGED. I am not raising this as a disagreement with community consensus; it is an internal inconsistency between two sections of the manuscript. No code, data, or policy outputs are provided to counter this reading, and the paper's own Section IV.C concedes that generated actions lead to unknown outcomes. The central claim therefore remains unsupported, and the reject verdict stands.","tokens_in":8014,"tokens_out":5593,"duration_ms":59583,"concrete_test":"Analytical check: for any state with delta E = 30 and any c > 0, compare r(th=30) = 1 with r(th=29) = exp(-1/c) < 1. The reward-maximizing threshold is 30, but Section II.B requires delta E to exceed the threshold for movement; at th = 30 the movement condition fails. This one-line calculation tests whether the reward's argmax is consistent with the device's activation rule. If the authors intend a different threshold interpretation, they should state the activation rule precisely and re-derive the reward; otherwise the learning objective cannot support the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires learned thresholds to improve user interaction. The learning signal in Section IV.C, r = exp(-d/c) with d the difference between delta E and the threshold of the dominant muscle, is maximized at th = delta E. For any state, the greedy action is to copy the current delta effort. Section II.B and Fig. 2 state that movement is initiated only when delta E exceeds the threshold and that speed then scales with delta E; at th = delta E the device sits exactly at the activation boundary, producing no movement if the condition is strict, or zero proportional speed if the condition is non-strict. Thus the reward actively prefers thresholds that suppress movement, the opposite of assistive intent. This is not an evaluation or coverage issue: even with perfect off-policy coverage and real-world testing, a policy optimizing this reward would set thresholds to track instantaneous delta effort, which is not a usable threshold for the MyoPro. The unspecified constant c does not change the argmax for any positive c. The paper reports no learned threshold trajectories, reward curves, or comparisons to static baselines, and Section IV.C itself admits OOD outcomes are unknown; but the reward degeneracy independently breaks the central claim because the optimization objective is misaligned with the device dynamics described earlier in the same paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an offline multi-agent reinforcement learning framework, based on Mixed Q-Functionals (MQF), to automatically adjust the biceps and triceps effort thresholds of the MyoPro 2 upper-limb exoskeleton from pre-collected data. Two agents are assigned to the two thresholds, and the reward is defined as r = exp(-d/c), where d is the absolute difference between the current delta effort and the threshold of the dominant muscle. Data were collected from a single participant performing horizontal and vertical arm movement tasks with thresholds varied in increments of 5 from 20 to 50. The authors report that their approach can dynamically adjust thresholds based on learned patterns and may improve user interaction, while acknowledging that evaluating performance remains difficult because generated actions may lead to out-of-distribution states.","tokens_in":8184,"tokens_out":2641,"duration_ms":27072,"significance":"If the central claim were established, the paper would address a practically important problem: reducing the manual recalibration burden for assistive exoskeletons. The framing of threshold tuning as a multi-agent offline RL problem is reasonable, and using MQF for continuous action spaces is a sensible algorithmic choice. However, the significance is currently undercut by two factors. First, the paper contains no empirical evaluation of the learned policy: no rollouts, no comparison with static thresholds, no reward or learning curves, and no quantitative outcome metrics. Second, and more fundamentally, the reward function is misaligned with the device dynamics described earlier in the same paper; maximizing it would drive thresholds to track the instantaneous delta effort, which places the device at the activation boundary and suppresses movement. The paper is honest about the dataset limitations and the OOD problem, which is commendable, but the reported results do not support the abstract's claim that the approach can improve user interaction and control.","major_comments":[{"comment":"The reward r = exp(-d/c), with d = |ΔE − threshold|, is maximized when threshold = ΔE, regardless of the positive constant c. Section II.B states that movement is initiated only when ΔE exceeds the threshold and that S_joint = k_p·ΔE. At threshold = ΔE, the device sits exactly at the activation boundary, producing no movement (under a strict inequality) or zero proportional speed (under a non-strict inequality). Thus the learning objective actively prefers threshold settings that suppress assistive movement. This is not merely an evaluation or coverage issue: even with ideal offline coverage and a perfect function approximator, a policy maximizing this reward would output thresholds that track the instantaneous delta effort, which is the opposite of a usable activation threshold. The reward must be redefined in terms of task success, movement smoothness, or user effort, and the revised objective must be justified with respect to the device dynamics in Section II.B.","section":"Section IV.C, Reward function design"},{"comment":"The paper reports no results from the trained models. Figures 6 and 7 show raw sensor data for selected static threshold combinations, not the behavior of the learned policies. There are no learned threshold trajectories, no comparison of learned thresholds to static baselines, no training reward curves, and no quantitative outcome metrics such as task completion time, user effort, or number of successful movements. The statement that 'the models can dynamically set thresholds for the muscles' is therefore unsupported by any reported experiment. To substantiate the central claim, the authors need to evaluate the learned policies, for example by logging the thresholds selected by the agents on held-out episodes, comparing their resulting behavior with the static thresholds in the dataset, and reporting task-relevant performance measures.","section":"Section IV.C, Findings"},{"comment":"The load-bearing premise of the approach is that the offline dataset adequately covers the state-action distribution so that a learned continuous-action policy can generalize to new thresholds. The dataset, however, was collected from a single participant with thresholds quantized in increments of 5, and only 10 episodes per combination. The paper itself concedes in Section IV.C that 'when the models generate new actions (i.e., threshold values), they may lead to states that are not present in the dataset, making their outcomes unknown.' This concession directly undermines the claim that the framework can learn to adjust thresholds effectively. The authors should either restrict the action space to the observed threshold values, incorporate a learned transition model to estimate outcomes of unseen actions, or conduct real-world user studies to evaluate the learned thresholds; without one of these, the central claim remains unverified.","section":"Section IV.B and Section IV.C, Data coverage and OOD actions"}],"minor_comments":[{"comment":"The sentence 'the triceps agent modifies thbiceps' should read 'the triceps agent modifies thtriceps'; as written, both agents are described as adjusting the same threshold.","section":"Section IV.C, State representation"},{"comment":"The caption of Figure 7 references 'Fig. 5(b)' when describing the horizontal task, but the horizontal task is shown in Figure 5(a). Please correct the cross-reference.","section":"Section IV.C and Figure 7 caption"},{"comment":"There are several typographical errors, e.g., 'excees' in Section II.B and 'a upper-limb' in the conclusion. A careful proofreading pass is recommended.","section":"Throughout"},{"comment":"The terminology is inconsistent: the threshold is described as 'activation levels required to initiate movement' but the control law uses delta effort ΔE. Clarify whether the threshold applies to ΔE or to the individual muscle effort, since this affects the reward design and the interpretation of Figures 6 and 7.","section":"Section II.B"}],"recommendation":"reject","confidential_remarks":"The paper reads as a preliminary workshop report rather than a completed journal contribution. The central claim is unsupported by any evaluation of the learned policy, and the reward function, as written, is degenerate with respect to the device dynamics. These are load-bearing issues that cannot be fixed by minor revision; they require a fundamental redesign of the objective and new experiments. I would also note that the manuscript does not provide code or data, which limits reproducibility. The paper's honest acknowledgment of its own limitations is a positive feature, but it does not compensate for the absence of evidence for the claimed contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's central claim—that offline MARL with MQF can adaptively tune MyoPro effort thresholds—is not backed by any experiment: the results section reports no rollouts, no learned threshold trajectories, no reward curves, and no comparison to static settings. The authors themselves say actions can lead to out-of-distribution states with unknown outcomes. Second, the reward is not just unevaluated; it is degenerate. r = exp(-|ΔE - th|/c) is maximized when th = ΔE, but Section II.B says movement begins only when ΔE exceeds the threshold. Setting th = ΔE puts the device exactly at the activation boundary, so no movement occurs. A policy optimizing this reward would suppress the assistive behavior the paper promises. That is a fundamental contradiction between the learning objective and the control law, not a quibble.\n\nWhat is creditable: the data collection is systematic—one participant, thresholds varied in steps of 5, tasks chosen to engage different muscles. The descriptive insights (lower biceps threshold helps vertical curls; biceps/triceps thresholds around 20/25 help horizontal reaches) are plausible and could inform a static tuner. The paper is also honest about dataset limitations and the need for real-world validation.\n\nBut the flaws are not minor. No code, no data, no hyperparameters (c, basis order O) are given, so nothing is reproducible. The novelty is thin: MQF is the authors' own prior method, and 'multi-agent' means one agent per scalar threshold. The abstract's claim about improving interaction and control is unsupported.\n\nWho is this for? People interested in the dataset or in a cautionary tale about reward design in human-in-the-loop RL might get something out of it, but as a research contribution it falls short.\n\nRecommendation: reject. I would not send this to peer review as is; the reward issue alone requires a redesign, and the missing evaluation would need to be extensive. If the authors fix the reward, add rollouts and baselines, and release the data, a future version could be worth examining.","headline":"Reward function sets threshold equal to muscle effort, which under the paper's own control law puts the device at the activation boundary and prevents movement; the central claim is also missing any policy evaluation.","tokens_in":8781,"tokens_out":4018,"would_cite":false,"duration_ms":40220,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline reinforcement learning can adapt exoskeleton effort thresholds from data.","keywords":["offline reinforcement learning","multi-agent reinforcement learning","Mixed Q-Functionals","assistive exoskeleton","MyoPro 2","sEMG effort thresholds","continuous action spaces","adaptive parameter tuning"],"falsifier":"Withhold one threshold pair from training, let the trained agents pick thresholds for the same participant and task, and record whether the resulting motor angle and sEMG effort states fall inside the training distribution and yield the predicted rewards; if the agents consistently produce out-of-distribution states or rewards below static baselines, the central claim is falsified.","tokens_in":7716,"feed_emoji":"🦾","tokens_out":4854,"duration_ms":49414,"temperature":0.7,"pith_summary":"The paper sets out to show that an offline multi-agent reinforcement learning framework can replace manual calibration of an upper-limb exoskeleton's effort thresholds. Using the MyoPro 2 device, the authors treat the biceps threshold and the triceps threshold as separate cooperative agents and train them on pre-collected sessions with Mixed Q-Functionals (MQF), a value-based method for continuous actions. Their experiments on horizontal and vertical arm tasks indicate that the learned agents adjust thresholds to task-dependent values, low biceps thresholds for the vertical task and a $(20, 25)$ pair for the horizontal task, which, if correct, means assistance can be personalized without risky online trial-and-error. The authors also stress that evaluating these learned thresholds is difficult because new threshold values can lead to states absent from the offline dataset.","feed_headline":"Offline RL tunes exoskeleton thresholds from logged data","feed_subtitle":"A multi-agent framework learns biceps and triceps settings for the MyoPro 2, reducing manual recalibration.","key_machinery":"The load-bearing mechanism is the Mixed Q-Functional (MQF), a value-based representation in which a neural network maps each state $s$ to coefficients $C(s)$ of basis functions over the continuous action space, and the Q-value of action $a$ is computed as $Q_F(s,a)=C(s)^\\top \\Phi(a)$. This turns evaluation of many candidate threshold actions into matrix operations, letting two cooperative agents, one for the biceps threshold and one for the triceps threshold, choose continuous values without policy-gradient sampling. A mixer network, using a VDN-style sum, combines agent Q-values into $Q_{\\mathrm{total}}$ for centralized temporal-difference training, while target networks are updated softly each step.","core_discovery":"On the paper's own terms, the central discovery is that offline value-based multi-agent RL can learn a continuous policy over effort thresholds from a fixed dataset of exoskeleton interactions. Each agent's Q-function is a Mixed Q-Functional mapping states to polynomial coefficients over the action interval $[20, 50]$; a VDN-style mixer combines the agents' values, and training minimizes temporal-difference error over logged transitions. The resulting policies select lower biceps thresholds when biceps effort dominates (vertical task) and a biceps/triceps pair of $(20, 25)$ when triceps effort dominates (horizontal task). The paper's claim is that this demonstrates adaptive, task-specific threshold tuning from data, while openly noting that the absence of real-time or out-of-distribution evaluation leaves user-level benefit as a potential rather than a measured outcome.","pith_inferences":["A concrete test the paper does not run: train on thresholds from $\\{20,25,30,35,40,45,50\\}$ but hold out one combination, then see whether MQF generalizes; this would isolate dataset-coverage effects from algorithm capacity.","The same MQF pipeline could tune the proportional gain $k_p$ or ramp parameters, since those are additional continuous action dimensions, offering a direct extension beyond thresholds.","Because the paper's evaluation cannot observe out-of-distribution outcomes, a model-based offline RL variant that learns a transition model of motor angle and sEMG effort would make learned thresholds checkable without real-time testing.","With multiple participants, per-user threshold policies could be compared against the single-participant policy; if muscle-activation patterns differ across users, the current dataset may not yet support the personalized-assistance conclusion."],"forward_implications":["If the claim holds, exoskeleton hyperparameters such as effort thresholds can be tuned without online exploration, removing a safety barrier to personalized assistive control.","The observed task-dependent optima mean that a single static calibration is suboptimal, and per-task learned adjustment should improve smoothness and user effort.","The multi-agent formulation provides a template for tuning several continuous control parameters at once, not just biceps and triceps thresholds.","The reward $r=e^{-d/c}$, which penalizes deviation between delta effort and the threshold, is a reusable objective for \"just enough assistance\" in other sEMG-driven devices.","Reliable evaluation will require either a learned transition model or real-world user studies, which the paper explicitly identifies as the path forward."],"supporting_citations":[{"why":"Supplies the physical platform: the MyoPro 2 exoskeleton with sEMG-driven elbow and wrist control.","marker":"[12]"},{"why":"Supplies the core algorithm, Mixed Q-Functionals, which represents continuous-action Q-values via basis-function coefficients.","marker":"[13]"},{"why":"Defines the offline RL paradigm the paper relies on to learn without online interaction.","marker":"[14]"},{"why":"Provides the batch learning foundation justifying training on a fixed dataset.","marker":"[15]"},{"why":"Supplies the VDN mixer used to combine the two agents' Q-values.","marker":"[16]"},{"why":"Provides evidence that adaptation and personalization drive exoskeleton benefit, motivating threshold tuning.","marker":"[4]"},{"why":"Demonstrates real-world personalization of exoskeleton assistance, the practical goal this work targets.","marker":"[9]"},{"why":"Suggests the transition-model approach the authors propose to handle out-of-distribution states.","marker":"[21]"}],"fun_headline_variants":["Offline RL learns exoskeleton effort thresholds","Multi-agent offline RL tunes biceps and triceps","Adaptive exoskeleton control from logged data","Data-driven effort thresholds for assistive exoskeletons","Reducing manual calibration via offline RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pre-collected dataset, from one participant with thresholds in steps of five, covers enough of the state-action space that the learned continuous threshold policy can be trusted on values and resulting states it never saw; the paper itself flags this in Section IV.C.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL learns exoskeleton effort thresholds","Multi-agent offline RL tunes biceps and triceps","Adaptive exoskeleton control from logged data","Data-driven effort thresholds for assistive exoskeletons","Reducing manual calibration via offline RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2847,"prompt_tokens":888,"completion_tokens":1959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1887}},"tokens_in":504,"tokens_out":1959,"duration_ms":15315,"temperature":1.0,"reasoning_tokens":1887,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:47:59.593730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Withhold one threshold pair from training, let the trained agents pick thresholds for the same participant and task, and record whether the resulting motor angle and sEMG effort states fall inside the training distribution and yield the predicted rewards; if the agents consistently produce out-of-distribution states or rewards below static baselines, the central claim is falsified.","supporting_citations":[{"cited_title":"Myomo – The MyoPro Brace for Stroke Survivor’s Paralyzed Arm","cited_arxiv_id":null,"evidence_quote":"Supplies the physical platform: the MyoPro 2 exoskeleton with sEMG-driven elbow and wrist control."},{"cited_title":"Batch reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the batch learning foundation justifying training on a fixed dataset."},{"cited_title":"How adaptation, training, and customization contribute to benefits from exoskeleton assistance,","cited_arxiv_id":null,"evidence_quote":"Provides evidence that adaptation and personalization drive exoskeleton benefit, motivating threshold tuning."},{"cited_title":"Personal- izing exoskeleton assistance while walking in the real world,","cited_arxiv_id":null,"evidence_quote":"Demonstrates real-world personalization of exoskeleton assistance, the practical goal this work targets."},{"cited_title":"Investigating the generalizability of assistive robots models over various tasks,","cited_arxiv_id":null,"evidence_quote":"Suggests the transition-model approach the authors propose to handle out-of-distribution states."}],"review_version":1}