{"id":"a807680c-933c-45e7-adcc-242f027d6fb7","arxiv_id":"2502.02133","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that classifies MPC+RL hybrids by algorithmic role, with tables of about 50 works, a theory review, and a software overview.","lead":"This survey maps the rapidly growing body of work that combines model predictive control, a planning method that solves an optimization problem at every control step, with reinforcement learning, a method that learns policies from experience. It sorts the field by the role MPC plays inside the hybrid: teaching the policy as an expert, running inside the deployed controller, or evaluating states as a critic.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The role taxonomy is not a partition: MPC-as-expert-critic is admitted to imitate the expert actor, so assignment depends on loss function, not algorithmic role.","rationale":"The reader's weakest assumption is that the taxonomy requires clean role assignments; the cited appearances of [263] and [162] in multiple tables support this. I sharpen the concern: the overlap is not just a matter of individual ambiguous papers but is built into the definitions, because Section IX-A explicitly says expert-critic training imitates the MPC expert policy, and Section VII-A explicitly moves Q-function-based imitation into the critic category. Therefore the categories are mechanisms (action loss vs Q-value loss) rather than architectural roles, which weakens the claim that the scheme distinguishes 'in which algorithmic part of the RL framework the MPC is used' (Sect. VI). This is a genuine taxonomic weakness, but it does not invalidate the survey's descriptive value: the tables are informative, the theoretical section is mostly prior published results with references, and the manuscript is candid about gaps, including the question-marked convergence target in the terminal-value update in Sect. VIII-A. Separately, Theorem 10.1 is stated with 'Proof 10.1: See [262]', so the theory section relies on an external proof; this is secondary because the survey's load-bearing assertion is the classification, not the equivalence theorem. A conditional verdict with a required clarification of membership rules is the right outcome, which is what the reader already recommended, so I do not change the verdict.","tokens_in":52357,"tokens_out":5470,"duration_ms":57879,"concrete_test":"Build a minimal decision rule from Sections VI-IX: (1) Is MPC evaluated during deployment? If yes, role=policy. (2) If only during training, does the learner's loss use QMPC or the MPC action? QMPC-loss→critic; action-loss→expert actor. Apply this rule to the thirteen entries in Table VIII and to [263] and [162]. If any Table VIII entry (e.g., [219] or [259]) also satisfies the Section VII-A definition of an expert actor whose behavior is imitated, or if the two coders disagree on more than two of the fifteen rows, the categories are not mutually exclusive and the paper must either formalize multi-label assignments or revise the category definitions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Tables IV-IX organize the MPC+RL literature by the algorithmic role of MPC (expert actor, deployed policy, critic). The weakest point is that these roles are not mutually exclusive under the paper's own definitions. Section VII-A says of Q-function-based imitation: 'Since the MPC provides a Q-value function in addition to the expert policy... we categorize such methods in a class named MPC as a critic.' Section IX-A then states 'using an MPC as an expert critic leads to imitating the MPC expert policy.' Thus a fixed-MPC method that trains a policy from MPC data can be classified as expert actor (Sect. VII) or expert critic (Sect. IX) depending on whether the training loss uses actions or Q-values, not on where the MPC sits in the RL architecture. Consistent with this, [263] appears in Tables V and VII, and [162] in Tables III, IV, and VII; no decision rule is given for assigning a multi-role paper. If the taxonomy is explicitly multi-label, the tables should say so and define membership criteria; if it is a partition, the overlap in Sections VII-A and IX-A is a contradiction. Either way, the claimed shared vocabulary is underspecified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript surveys methods that combine model predictive control (MPC) and reinforcement learning (RL). It unifies the notation of the two fields, compares their practical and theoretical properties, and proposes a taxonomy based on the algorithmic role of MPC within an actor-critic RL framework: MPC as an expert actor (Sect. VII), MPC within the deployed policy (Sect. VIII), and MPC as a critic (Sect. IX), together with four parameterized-MPC architectures (integrated, hierarchical, parallel, parameterized; Sect. VI-A). The survey covers roughly fifty works in Tables IV–IX, includes a theory section (Sect. X) centered on an MPC–MDP equivalence theorem, and closes with software tools and implementation aspects. The paper is notably honest about its own limitations, explicitly flagging the sampling-notation issue in Remark 3.1 and marking the convergence target of the terminal-value-function update with a question mark in Sect. VIII-A2.","tokens_in":52482,"tokens_out":7633,"duration_ms":73917,"significance":"If the taxonomy is accepted, the paper provides a genuinely useful organizational contribution: a role-based vocabulary for the MPC+RL combination literature, an extensive tabulation of applications and algorithms, and a careful notation unification. The paper does not overclaim its theoretical grounding, and its software discussion is a practical asset. The central classification claim, however, is currently underspecified because the categories are not shown to be mutually exclusive or cleanly assignable, and the tables do not state a convention for works that realize multiple roles. Until this is resolved, the claimed 'shared vocabulary' is not yet a stable map of the field. The comparison tables and the honest treatment of open theoretical points are valuable independently of this issue.","major_comments":[{"comment":"The role taxonomy is presented as the paper's central organizational contribution, but the categories are not stated to be mutually exclusive, and the text gives conflicting guidance on assignment. Section VII-A explicitly moves Q-function-based imitation into the critic category ('we categorize such methods in a class named MPC as a critic'), while Section IX-A states that using an MPC as an expert critic 'leads to imitating the MPC expert policy.' A fixed-MPC method that trains a policy from MPC demonstrations can therefore be placed in either the expert-actor category (if the training loss uses actions) or the expert-critic category (if the loss uses Q-values), even though the MPC occupies the same position in the RL architecture. The tables reflect this ambiguity: reference [263] appears in Tables V and VII, and reference [162] appears in Tables III, IV, and VII, with no stated convention for multi-role entries. Please either declare the taxonomy explicitly multi-label and define membership criteria, or adjust the category definitions so that the classification is a partition of the surveyed works.","section":"§VII-A, §IX-A, Tables IV–VIII"},{"comment":"The passage defining closed-loop learning refers to 'the MPC policy µMPC according to Definition 4,' but no Definition 4 exists anywhere in the manuscript. Since the aligned-versus-closed-loop distinction is one of the two main axes of the deployed-policy category, the missing definition is load-bearing: the reader cannot verify what 'closed-loop optimal' means at the point where the taxonomy is introduced. Please add the missing definition at the point of use, or rephrase the passage to refer to the characterization later given in Sect. X (condition 1 of Theorem 10.1).","section":"§VIII-B"},{"comment":"The term 'MDP aligned' is used inconsistently. In §VIII-A, 'aligned learning' requires the MPC's model, costs, constraints, and terminal value function to 'individually approximate the MDP and the optimal value function'; in §X, 'the MDP aligned property ... demands that the MPC model exactly matches the state-transition kernel of the MDP' and is said to be 'rarely satisfied in practice.' These are different properties (component-wise approximation versus exact transition-kernel match), and the latter definition would make most of the Table V entries not 'MDP aligned' at all. Since the aligned-versus-closed-loop axis is central to the deployed-policy classification, the terminology should be reconciled, or two distinct names should be introduced.","section":"§VIII-A and §X"}],"minor_comments":[{"comment":"Figure 1 lists both 'Tab. IX: Literature using MPC as a critic' and 'Tab. IX: Theoretical results for combining MPC and RL'; the second of these should be Table X.","section":"Fig. 1"},{"comment":"Table VII's header reads 'MPC for Postrocessing' and Fig. 1 contains 'Literature wiht MPC as filter'; both are typos that should be corrected.","section":"Table VII and Fig. 1"},{"comment":"The statement that solver differentiation support 'is currently ongoing work in acados' and 'will be detailed in a future publication' is acceptable for a survey but should be time-stamped or softened to avoid ageing quickly.","section":"§XI-B1"},{"comment":"The sentence reporting that 'most authors report superiority of MPC' with cost reductions of 'less than 4%' is vague because the baseline (RL variant, task, and metric) is not stated in the surrounding text; please make the comparison explicit or point to the table entries that support it.","section":"§V-10"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a control-systems journal, and the proposed taxonomy could become a useful reference if the category membership problem is addressed. The main risk is that the taxonomy's terms are not yet precise enough to serve as the field-wide vocabulary the authors intend; a revision that turns the categories into an explicitly multi-label scheme with stated membership criteria would address this. The theory section is dominated by papers from the authors' own group (Gros, Zanon, Kordabad, Diehl and coauthors). This is not disqualifying for a survey, but the neutrality and coverage of the classification would be strengthened by a more substantial engagement with alternative framings, such as Bertsekas's ADP–MPC unification [3] and the ML-in-MPC survey [17]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious survey and the taxonomy is a real contribution. If you work on MPC-RL hybrids, this is now the reference entry point. The paper distinguishes three roles for MPC (expert actor, inside the deployed policy, critic) plus four parameterized-MPC architectures (integrated, hierarchical, parallel, parameterized), and it applies that scheme to roughly fifty papers in Tables IV-IX. That organizational core is new relative to the earlier surveys ([8], [16], [17]) and it will likely shape how people describe their own methods.\n\nWhat it does well: the background sections (RL, MPC, comparison) are competent and unified with one notation, which the field needs. The paper is candid about its own limits — Remark 3.1 on sampling notation, and the explicit question mark on the convergence target of the terminal-value-function update in Section VIII-A. That honesty is genuine. The software section is a practical plus. The theoretical section (Section X) concentrates on results from the authors' own group, but those are the substantive results in this area (MPC-MDP equivalence, dissipativity links), so the citation pattern is justified rather than self-promotional, and Theorem 10.1 points to a published proof.\n\nSoft spots, in proportion: first, the classification is not a partition. The paper says it categorizes 'in which algorithmic part... the MPC is used', and Fig. 2 even shows multiple MPC instances with different roles. Yet the tables assign each paper to one row, and works like [162] and [263] appear in several tables. This is fine as an indexing device, but the paper should state explicitly that categories are multi-label, and give a decision rule for the ambiguous cases. The most important ambiguity is the expert-actor/expert-critic boundary: Section VII-A says Q-function-based imitation gets categorized as 'MPC as a critic', while Section IX-A says an expert critic leads to imitating the MPC policy. The same method could land in either section depending on the loss function. The paper should acknowledge that the boundary is functional, not architectural. Second, there is no search/inclusion methodology, so a reader cannot assess coverage completeness. That is standard for surveys in this area, but a short paragraph on how the tables were compiled would improve reproducibility. Both are fixable in revision.\n\nBottom line: this deserves a serious referee. The taxonomy will be used, the tables provide a useful map, and the candor is refreshing. I'd accept with minor revisions, mainly the classification caveats. For a reading group, it's a good way to get everyone up to speed.","headline":"A solid, useful survey: the actor-critic role taxonomy is new and works as a map, but it is multi-label by design and should be described as such.","tokens_in":53130,"tokens_out":2055,"would_cite":true,"duration_ms":22003,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims the MPC+RL combination literature can be organized by where MPC sits inside an actor-critic framework: as expert actor, as part of the deployed policy, or as critic.","keywords":["model predictive control","reinforcement learning","actor-critic","survey","classification","imitation learning","closed-loop learning","control synthesis"],"falsifier":"Locate a published MPC+RL combination whose MPC component both provides training targets and remains in the deployed loop, forcing placement into two of the three roles; the paper's own placement of [263] in both Table V and Table VII indicates the categories are not exclusive.","tokens_in":52078,"feed_emoji":"🤖","tokens_out":4256,"duration_ms":38456,"temperature":0.7,"pith_summary":"This paper tries to establish a systematic way of dividing the growing literature that combines model predictive control (MPC) and reinforcement learning (RL). Instead of sorting papers by application or algorithm, it sorts them by the algorithmic role MPC plays inside an RL actor-critic framework. It claims three roles cover the field: MPC as an expert actor that guides or demonstrates for training, MPC inside the deployed policy (learned as an optimization layer or used for pre/postprocessing), and MPC as a critic that supplies value estimates. Alongside this, it defines four parameterized-MPC architectures (integrated, hierarchical, parallel, parameterized) and distinguishes aligned learning from closed-loop learning within the deployed-policy role. If the taxonomy is right, the field gains a common vocabulary and a structured map of roughly fifty works, plus a clear statement of the main methodological fork for training MPC-based policies.","feed_headline":"One taxonomy organizes MPC+RL methods by algorithmic role","feed_subtitle":"Survey places about fifty combination methods into expert-actor, deployed-policy, and critic roles, giving the field a shared vocabulary.","key_machinery":"The organizing device is the actor-critic decomposition of RL, used as a coordinate system. MPC is treated as a modular optimization layer that can be placed in the expert, actor, or critic position, and parameterized MPC is classified by how a learned function approximator enters the optimization problem: integrated, where the learned function depends on decision variables inside the optimization; hierarchical, where a learned function is evaluated beforehand and supplies parameters; parallel, where an NN and MPC are evaluated side by side and their outputs are combined; and parameterized, where learned parameters are constant and do not depend on state or decision variables. This machinery carries the classification: every surveyed combination is described by one role plus one architecture, and the aligned-versus-closed-loop distinction then splits the actor role.","core_discovery":"The central claim is that every way of combining MPC and RL can be understood as inserting the MPC optimization problem into one of three algorithmic parts of an actor-critic RL agent: as an expert actor whose fixed policy is mimicked or used to guide exploration (Section VII); as part of the deployed policy, either aligned with the MDP by learning model, terminal value function, or stage cost, trained for closed-loop optimality, or used for pre/postprocessing (Section VIII); or as part of the critic, where MPC computes or parameterizes value/action-value estimates (Section IX). The paper supports this with a modular diagram (Fig. 2), a survey of roughly 50 works in Tables IV through IX, and a theoretical section that links MPC-MDP equivalence results to these roles. It further argues that within the deployed-policy role the crucial methodological divide is between MDP-aligned learning, which makes each MPC component approximate the corresponding MDP component, and closed-loop learning, which changes model, cost, or constraints purely to improve closed-loop performance.","pith_inferences":["The taxonomy could function as a generative design tool: choosing a role and an architecture enumerates the combination space, exposing under-explored cells such as MPC as a learnable actor-critic.","As differentiation tools mature, closed-loop optimal learning may displace MDP-aligned learning for performance-critical applications because it optimizes the actual deployment objective rather than individual components.","The role-based map also unifies imitation learning, guided policy search, and safe RL: many safe-RL schemes are MPC-as-filter or MPC-as-expert, and labeling them by role could reveal redundant designs."],"forward_implications":["Researchers can place new MPC+RL algorithms into one of three roles and one of four architectures, making method comparisons and transfers easier.","The aligned-versus-closed-loop distinction clarifies the main design choice: either make each MPC component approximate the MDP (model, value, cost) or tune the whole optimization layer for closed-loop performance.","The theoretical results reviewed in Section X, especially the MPC-MDP equivalence theorems, justify using MPC as actor or critic by showing when a parameterized MPC can reproduce the optimal policy and value functions of a discounted MDP.","The survey's software section indicates that practical combination work is currently limited by solver support for differentiation and integration with learning frameworks; closing that gap would accelerate the field."],"supporting_citations":[{"why":"Establishes the conceptual relations and nearly orthogonal strengths of MPC and RL that motivate the whole combination survey.","marker":"[8]"},{"why":"Provides a unified mathematical framework comparing RL and MPC from the viewpoint of approximate dynamic programming.","marker":"[3]"},{"why":"Surveys machine learning inside MPC and aligns with the proposed categories, especially terminal value function learning versus closed-loop learning.","marker":"[17]"},{"why":"Proves the fundamental equivalence between economic MPC and MDP value functions that underpins the theoretical section.","marker":"[224]"},{"why":"Extends the equivalence to undiscounted MPC and gives Theorem 10.1 that parameterized MPC can capture the optimal policy, value function, and Q-function.","marker":"[262]"},{"why":"Introduces guided policy search, the basis for the MPC-as-expert-actor category.","marker":"[217]"},{"why":"Introduces the predictive safety filter, the basis for the MPC-as-postprocessing category.","marker":"[185]"},{"why":"Provides DAgger, the imitation-learning algorithm used across many surveyed MPC-as-expert works.","marker":"[193]"},{"why":"Uses a fixed MPC Q-function as an expert critic, grounding the MPC-as-critic category.","marker":"[219]"}],"fun_headline_variants":["MPC+RL combos sorted into three actor-critic roles","Taxonomy maps 50 MPC-RL hybrids to actor-critic slots","Survey: every MPC-RL blend fits an actor, critic, or policy role","Three roles unite MPC and RL: expert actor, policy, critic","How MPC bolsters RL: one taxonomy, three roles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The classification assumes that the three roles (expert actor, part of the deployed policy, part of the critic) are jointly exhaustive and that each surveyed work can be assigned cleanly to one role and one architecture; the paper itself places some works in multiple tables, so the map can overlap.","fun_headline_variants_meta":{"raw":{"variants":["MPC+RL combos sorted into three actor-critic roles","Taxonomy maps 50 MPC-RL hybrids to actor-critic slots","Survey: every MPC-RL blend fits an actor, critic, or policy role","Three roles unite MPC and RL: expert actor, policy, critic","How MPC bolsters RL: one taxonomy, three roles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1207,"prompt_tokens":914,"completion_tokens":293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":530,"tokens_out":293,"duration_ms":3147,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:12:53.090829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Locate a published MPC+RL combination whose MPC component both provides training targets and remains in the deployed loop, forcing placement into two of the three roles; the paper's own placement of [263] in both Table V and Table VII indicates the categories are not exclusive.","supporting_citations":[{"cited_title":"Equivalence of Optimality Criteria for Markov Decision Process and Model Predictive Control,","cited_arxiv_id":null,"evidence_quote":"Extends the equivalence to undiscounted MPC and gives Theorem 10.1 that parameterized MPC can capture the optimal policy, value function, and Q-function."}],"review_version":1}