{"id":"a316fafb-d081-491b-a9bb-1e7e352c4f54","arxiv_id":"2506.08344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A DRL policy selects among base, arm, and whole-body kinematic models for NMPC on a mobile manipulator, improving success rate and reducing per-call computation time in simulation.","lead":"Re4MPC uses a reinforcement learning agent to decide whether a mobile manipulator should plan with a simplified model of just the base, just the arm, or the whole body, and then feeds that choice into a nonlinear model predictive controller. In simulation, the learned policy picks lighter models and intermediate sub-goals, which reduces per-step computation time and raises success rates compared to a whole-body NMPC baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claim rests on per-call solve times only; end-to-end compute and total NMPC calls per episode are never reported, so 'more computationally efficient' is not established.","rationale":"The reader's conditional verdict is reasonable. I focus on the efficiency half of the central claim because it is stated unconditionally in the abstract ('more computationally efficient') and the reported data do not establish it. Table I gives per-model mean solve times and call counts, but the baseline's lower total call count is acknowledged, and the policy's inference overhead is absent; average per-call speedups from low-DOF models are compatible with the system being slower end-to-end. A timing experiment on the released code would settle this directly. This is a correctness/measurement issue, not a disagreement with consensus, and it is independent of the reward-shaping concern: even if reward design is acceptable, the efficiency statement needs an end-to-end accounting. I therefore keep the conditional verdict rather than accept or reject. If end-to-end timing does favor Re4MPC, the remaining reward/ablation concern would still need the ablations the reader requested.","tokens_in":12725,"tokens_out":8971,"duration_ms":107625,"concrete_test":"Instrument the released mobiman code to measure, for each of the 108 test configurations and across the five runs, total wall-clock time per episode (including RL policy inference, action decoding, NMPC solves, and simulation stepping) and total NMPC solve time per successful episode for Re4MPC-SAC-w/oRT and the ocs2wb baseline. If Re4MPC's end-to-end time or total NMPC time per success is not lower than baseline, the abstract's efficiency claim must be scoped to per-call solve latency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline claim of computational efficiency is not supported by the reported metrics. Table I reports mean per-call SLQ solve times separately for base, arm, and whole-body models, plus the number of NMPC calls per model. It never reports total computation per episode, end-to-end wall-clock time, or the cost of the learned policy (neural-network forward pass, action decoding, model switching, sub-goal computation) that runs online in Re4MPC. Since Re4MPC episodes tend to last longer (it avoids early failures and uses sub-goals), the total number of NMPC calls is higher for Re4MPC than for ocs2wb; the paper even notes ocs2wb 'exhibits fewer total computations.' Lower per-call times for simpler models can therefore be outweighed by more calls and by policy overhead. The claim 'more computationally efficient than the NMPC baseline' is consequently an interpretive leap, not a measured result. A separate confound: no ablation distinguishes the reactive model-selection mechanism from the learned sub-goal/target reward shaping, so the mechanism attribution in the conclusion is also underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Re4MPC, a framework that couples deep reinforcement learning (DRL) with nonlinear model predictive control (NMPC) for mobile manipulator motion planning. A DRL policy selects, at each action interval, one of three robot models (base, arm, or whole-body), the NMPC cost function and constraints, and a target (sub-goal or final goal). The framework is evaluated in a PyBullet-based simulation with a Jackal mobile base and a 6-DOF Kinova Jaco arm. The authors report that Re4MPC trained with PPO, SAC, or DQN achieves higher success rates than a whole-body NMPC baseline (ocs2wb), and that it computes trajectories more efficiently because it often selects lower-dimensional models.","tokens_in":13020,"tokens_out":5285,"duration_ms":57609,"significance":"If substantiated, the framework would be a useful contribution to integrated DRL/NMPC for mobile manipulation, and the open-source implementation (mobiman) is a clear strength. The mathematical formulation of integrating NMPC into a DRL loop is clearly presented. However, the current evidence does not establish the computational efficiency claim, and the attribution of gains to reactive model selection is confounded by the reward function's explicit DOF penalty. The success-rate advantage is plausible but lacks statistical support and has at least one internal contradiction.","major_comments":[{"comment":"The headline claim that Re4MPC is 'more computationally efficient' than the NMPC baseline is not supported by the reported metrics. Table I lists mean per-call SLQ solve times for each model and the number of NMPC calls per model, but it does not report total computation per episode, end-to-end wall-clock time, or the overhead of the learned policy (neural-network forward pass, action decoding, model switching, sub-goal computation). The paper itself notes that ocs2wb 'exhibits fewer total computations' because Re4MPC episodes tend to last longer. Without total computation or end-to-end time, lower per-call times for simpler models cannot establish the efficiency claim.","section":"Section IV.B, Table I"},{"comment":"The reduced whole-body usage is at least partly enforced by design. The reward term R_model penalizes the selected model with a constant whose value is proportional to the model's DOF, directly incentivizing low-DOF choices. Presenting the resulting reduction in whole-body usage as evidence of Re4MPC's efficiency is therefore circular to some degree. An ablation that removes the DOF penalty from R_model, or a comparison against a policy that selects models without this reward shaping, is needed to attribute the efficiency gain to reactive model selection rather than to the reward design.","section":"Section III.B.3, Table I"},{"comment":"Success rates are reported as point estimates without error bars, confidence intervals, or statistical significance tests, even though each configuration is averaged over five runs. This makes it difficult to assess whether the observed differences over the baseline are meaningful, especially for variants with small differences. The paper should report per-run results, standard deviations, or a statistical test to support the claim of higher success rates.","section":"Section IV.B, Fig. 3, Table I"},{"comment":"The baseline ocs2wb is not described in sufficient detail to ensure a fair comparison. It is stated to be a whole-body SLQ formulation with the goal as the MPC target, but the manuscript does not specify whether it uses the same cost function weights and constraints as Re4MPC when Re4MPC selects the whole-body model and goal target. Differences in cost tuning could confound the success-rate comparison.","section":"Section IV.A"}],"minor_comments":[{"comment":"Table I is hard to read: entries like '8684' and '01 2' appear to be two numbers concatenated without a separator. Please reformat the table for clarity.","section":"Table I"},{"comment":"The statement that 'all variants achieve higher success rates' than the baseline is contradicted by Table I, where re4mpc-DQN-w/oRT shows a 60% success rate versus 61% for ocs2wb.","section":"Section IV.B"},{"comment":"The numerical values of the reward constants (tau_success, tau_boundary, tau_collision, tau_roll, tau_max_step, tau_target) and thresholds (delta_g, delta_cd, delta_psi, delta_vartheta, delta_com) used in Eqs. (2)-(5) are not reported. Please provide these values in the paper or a clear pointer to the code.","section":"Section III.B.3"},{"comment":"The timing measurements in Table I lack a description of the compute platform. Please specify the hardware and software environment used for the timing experiments.","section":"Table I"},{"comment":"Some references appear incomplete; for example, [32] does not include publication venue or page numbers. Please ensure all references are complete.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's robotics scope, and the open-source release is a positive aspect. However, the experimental methodology is not yet at the level required for publication. The efficiency claim needs end-to-end timing, and the reward-shaping confound needs an ablation. The success-rate claims need statistical backing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new thing is the combination of a learned policy that selects among base, arm, and whole-body kinematic models inside an NMPC solver (SLQ in OCS2) for a mobile manipulator. Prior work either did DRL for sub-goal selection or model switching without learning. The paper lays out the POMDP formulation clearly, releases code, and reports a real success-rate gap: the best SAC variant gets 86% on 108 test configurations versus 61% for the whole-body baseline. They also state their limitations honestly, explicitly leaving robustness and generalization out of scope.\n\nThe soft spots are in the efficiency claim and the mechanism attribution. The abstract says \"more computationally efficient,\" but the evidence is per-call SLQ solve times only. Table I never gives total compute per episode, end-to-end wall-clock time, or the cost of the policy network and model-switching overhead. The paper itself notes that the whole-body baseline exhibits fewer total computations, and Re4MPC episodes run longer because it avoids early failures and uses sub-goals. So the per-call gains can be outweighed by more NMPC calls plus the policy and switching overhead. That is an interpretive leap, not a measured result.\n\nThe second issue is a reward confound. R_model penalizes whole-body usage proportional to DOF, so the policy is directly incentivized to favor simpler models. The resulting reduction in whole-body usage is therefore partly enforced by design, not emergent from the model-selection mechanism. The success-rate improvement is also entangled with the sub-goal generation; they do ablate the target reward, but not the model-selection mechanism itself. So the central claim that reactive model selection drives the gains is underdetermined.\n\nThese are addressable. The paper would be stronger with an end-to-end compute measurement, an ablation that removes the R_model penalty, error bars on success rates, and at least one out-of-distribution evaluation. Given the open-source release and the clear write-up, I would send this to review with major revisions. A desk reject would be too harsh; the framework is useful and the evaluation is honest about its scope.","headline":"A useful engineering contribution with a real confound: the efficiency and success gains are partly written into the reward, and end-to-end compute is never measured.","tokens_in":13565,"tokens_out":2075,"would_cite":false,"duration_ms":25040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reactive, learned switching among robot models for NMPC raises a mobile-manipulation task's success rate from 61% to 86% while cutting per-solve compute time.","keywords":["reactive motion planning","nonlinear model predictive control","multi-model planning","deep reinforcement learning","mobile manipulation","sequential linear quadratic","computational efficiency","whole-body control"],"falsifier":"Retrain Re4MPC with the model penalty set to a constant regardless of the selected model's degrees of freedom, keeping everything else identical. If whole-body model usage stays high and per-solve compute time does not drop, then the reported computational gains come from the reward, not from the multi-model selection mechanism itself. Alternatively, run the trained policy in an environment with obstacles and goal positions outside the training distribution; if its success rate falls to the baseline level, the advantage is overfitting to the training layout.","tokens_in":12557,"feed_emoji":"🤖","tokens_out":6005,"duration_ms":50505,"temperature":0.7,"pith_summary":"This paper claims that a motion planner's compute cost and success rate both improve when a learned policy is allowed to choose, at each decision step, which robot model the NMPC trajectory optimizer should use, and what sub-target to aim for. The target robot is a wheeled mobile manipulator, and the choice is among base-only, arm-only, and whole-body kinematic models. In a physics-based simulation with a stationary conveyor and a box, the best trained variant succeeds at 86% of 108 test configurations versus 61% for a whole-body-only NMPC baseline, while per-solve compute times drop when simpler models are selected. If this result holds beyond the tested task, it points to a template for scaling NMPC to high-DOF robots by spending solver effort only where it is needed.","feed_headline":"Learned model switching lifts robot planner to 86% success","feed_subtitle":"A DRL policy picks when to plan with the base, arm, or whole body, beating full-body NMPC in speed and success.","key_machinery":"The central mechanism is the parameterized action space $A_c$ (and its discrete counterpart $A_d$), which encodes three sub-actions: model selection $a_{model} \\in [0,1]$ mapped to one of three kinematic models via thresholds, constraint selection $a_{constraint} \\in [0,1]^{N_c}$ gated at 0.5, and target selection $a_{target} \\in \\mathbb{R}^7$ that picks either a sub-goal or the goal pose. These decoded settings define the NMPC problem in Eq. 1, which is solved by Sequential Linear Quadratic (SLQ), a DDP-based optimizer with relaxed barrier functions for constraints. The reward function includes the DOF-proportional model penalty, a goal-distance reward, and a target reward that encourages sub-goals for non-arm modes; together these terms drive the policy to prefer cheaper models when they suffice.","core_discovery":"Re4MPC formulates motion planning as a partially observable Markov decision process in which the agent's action is decoded into the NMPC problem definition: the robot model index (base, arm, or whole body), the constraint set, and the end-effector target (either a goal or a sub-goal). The NMPC solver runs for an action horizon with those settings, and the resulting state transitions and rewards shape the policy. The paper reports that after 100k training steps, the SAC-trained variant reaches 86% success versus 61% for the whole-body baseline, with fewer rollovers and collisions, and that the learned policy uses the whole-body model only about 30% of the time in its successful episodes. The computational gain is attributed to a reward term that penalizes the active model in proportion to its degrees of freedom, together with a target reward that steers the agent toward feasible sub-goals.","pith_inferences":["If the model-penalty reward is indeed the driver, the same pipeline could be retargeted to other high-DOF systems such as humanoids or multi-arm platforms by swapping the model set while keeping the POMDP action structure; this extension is left implicit in the paper.","The dependency on hand-tuned reward weights and thresholds (for example $\\tau_{target}$, $\\delta_{com}$, and $\\alpha_{sub}$) suggests a testable bound: performance should degrade gracefully as these parameters are perturbed, but no sensitivity analysis is provided.","A natural next experiment is to replace the learned policy with a hand-coded heuristic (e.g., use base-only when far from the goal and whole-body when close) and compare; if the heuristic matches Re4MPC, the contribution lies in the multi-model formulation rather than in the learning.","Because the training environment keeps the box fixed and the conveyor static, the reported gains may be specific to that layout; testing in randomized or dynamic scenes would clarify whether the policy learns true reactivity or a fixed switching schedule."],"forward_implications":["Re4MPC reduces whole-body model usage in successful episodes to roughly 30% for the SAC variant, implying the learned policy discovers when full-body optimization is unnecessary.","Per-solve computation time for base-model planning is roughly half that of whole-body planning, so a policy that switches models can cut online compute while maintaining or improving task success.","All Re4MPC variants surpass the baseline success rate by 100k training timesteps, with SAC variants converging fastest.","The discrete action space filters infeasible target poses, leading to different model and target selection frequencies than the continuous action space, a design choice the paper links to learning behavior."],"supporting_citations":[{"why":"Provides the Sequential Linear Quadratic algorithm that underlies the NMPC solver.","marker":"[15]"},{"why":"Extends SLQ with constraint handling for the continuous-time DDP formulation used in this paper.","marker":"[16]"},{"why":"Introduces the parameterized action space idea that Re4MPC adapts for model and target selection.","marker":"[21]"},{"why":"Shows DRL augmenting motion generation with parameterized actions, the closest prior hybrid baseline.","marker":"[22]"},{"why":"Supplies the open-source optimal-control library on which the multi-model NMPC is implemented.","marker":"[26]"},{"why":"Gives the relaxed barrier function method used to handle joint position and velocity limits in SLQ.","marker":"[30]"},{"why":"Provides the PPO algorithm used as one of the three RL trainers.","marker":"[35]"},{"why":"Provides the SAC algorithm, the variant that reports the highest success rate.","marker":"[36]"},{"why":"Provides the DQN algorithm used for the discrete action space variant.","marker":"[37]"}],"fun_headline_variants":["DRL selects robot model per task, lifting NMPC success to 86%","Reactive model choice in NMPC via DRL yields 86% success, less compute","Learned model switching improves robot planner success to 86%","Multi-model NMPC with DRL outperforms fixed model: 86% vs 61%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-designed reward function — specifically the DOF-proportional model penalty and the target-reward thresholds — is what teaches the policy to choose simpler models, so if those weights do not transfer to new tasks or environments, the reported efficiency and success gains will not transfer either.","fun_headline_variants_meta":{"raw":{"variants":["DRL selects robot model per task, lifting NMPC success to 86%","Reactive model choice in NMPC via DRL yields 86% success, less compute","Learned model switching improves robot planner success to 86%","Multi-model NMPC with DRL outperforms fixed model: 86% vs 61%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2956,"prompt_tokens":914,"completion_tokens":2042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1951}},"tokens_in":530,"tokens_out":2042,"duration_ms":15828,"temperature":1.0,"reasoning_tokens":1951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:11.806704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain Re4MPC with the model penalty set to a constant regardless of the selected model's degrees of freedom, keeping everything else identical. If whole-body model usage stays high and per-solve compute time does not drop, then the reported computational gains come from the reward, not from the multi-model selection mechanism itself. Alternatively, run the trained policy in an environment with obstacles and goal positions outside the training distribution; if its success rate falls to the baseline level, the advantage is overfitting to the training layout.","supporting_citations":[{"cited_title":"An efficient sequential linear quadratic algorithm for solving nonlinear optimal control problems,","cited_arxiv_id":null,"evidence_quote":"Provides the Sequential Linear Quadratic algorithm that underlies the NMPC solver."},{"cited_title":"Constraint handling in continuous-time ddp-based model predictive control,","cited_arxiv_id":null,"evidence_quote":"Extends SLQ with constraint handling for the continuous-time DDP formulation used in this paper."},{"cited_title":"Motion planner augmented reinforcement learning for robot manipulation in obstructed environments,","cited_arxiv_id":null,"evidence_quote":"Introduces the parameterized action space idea that Re4MPC adapts for model and target selection."},{"cited_title":"Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation,","cited_arxiv_id":null,"evidence_quote":"Shows DRL augmenting motion generation with parameterized actions, the closest prior hybrid baseline."},{"cited_title":"Relaxed logarithmic barrier function based model predictive control of linear systems,","cited_arxiv_id":null,"evidence_quote":"Gives the relaxed barrier function method used to handle joint position and velocity limits in SLQ."}],"review_version":1}