{"id":"3ddb3f28-63c5-4b5c-b677-dba56952509c","arxiv_id":"2505.03238","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Combining supervised fine-tuning with closed-loop RL lets small Qwen LLMs tune an MPC controller, and the 3B model scores 63.3% versus 58.5% for GPT-4o on the paper's custom control adaptability metric.","lead":"This paper trains small language models, 1.5B and 3B, to adjust the controller of a scaled autonomous race car, using reinforcement learning with feedback from the driving simulator. It reports that the 3B model reaches a 63.3% control adaptability score, above the 58.5% of the much larger cloud-based GPT-4o.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Improve' aggregate is an unweighted mean of four per-behavior percentage improvements; alternative aggregations can reverse the Qwen2.5-3B-vs-GPT-4o ranking, and no variance is reported, so the headline superiority claim is not yet robust.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the 'Improve' metric is unspecified and lacks variance. My reading sharpens it: the numbers show the column is an unweighted mean of four percentage improvements, and this specific choice is what produces the 63.3% versus 58.5% ordering. Because the four underlying metrics have heterogeneous units and baselines, and because a natural alternative aggregation (sum of raw RMSEs) reverses the ranking, the headline claim is aggregation-dependent rather than a robust property of the models. The absence of variance makes the gap even more fragile. I therefore agree with the reader's CONDITIONAL verdict: the method is plausible, the code is available, and the training curves and real-robot deployment add credibility, but the central comparison cannot be accepted as stated until the aggregation is justified and error bars are supplied. The additional arithmetic inconsistency (20.2 vs 30.2 points) reinforces the need for careful reporting but is secondary to the undefined metric.","tokens_in":18219,"tokens_out":9350,"duration_ms":86013,"concrete_test":"Recompute the control adaptability score from the per-metric RMSEs in Table 2 under at least two alternative aggregation rules (e.g., sum of raw RMSEs relative to the default sum; geometric mean of per-behavior error ratios) and compare the Qwen2.5-3B and GPT-4o orderings. Additionally, obtain per-prompt standard deviations for each model across the five evaluation prompts (ideally with multiple seeds) and report whether the 63.3% versus 58.5% gap exceeds one standard error. If the ordering flips under a defensible aggregation or the gap is within noise, the headline superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a 3B model surpasses GPT-4o rests entirely on the 'Improve [%]' column in Table 2, yet no formula is given. The numbers imply it is the unweighted mean of the four per-metric percentage improvements (Qwen2.5-3B: (39.9+90.2+91.2+31.8)/4 = 63.3%; GPT-4o: 58.5%). This aggregation treats a 90% reduction in reversing error—which is trivially achieved by setting v_max negative—as equally valuable as a 40% reduction in centerline error, despite the metrics having different units and baselines (EC: 0.68 m, EV: 1.98 m/s, ER: 5.44 m/s, ES: 1.77 m/s^2). A simple alternative aggregation, summing raw RMSEs relative to the default sum, gives Qwen2.5-3B 76.8% improvement versus GPT-4o's 79.6%, reversing the reported ordering. Moreover, no per-prompt standard deviations or confidence intervals are reported; with only five evaluation prompts, the 4.8-point gap (63.3 vs 58.5) may be within noise. The paper also claims a 20.2-point improvement for Qwen2.5-1.5B (41.5% minus 11.3% is 30.2 points), an arithmetic inconsistency. Because the comparison to GPT-4o is the paper's headline result, the undefined and variance-free aggregation is the load-bearing weak point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the R1-Zero/RLVR recipe to embodied robotics: Qwen2.5 LLMs (1.5B and 3B) are SFT-tuned and then trained with GRPO in a closed loop with an F1TENTH-simulated 1:10 autonomous car, where the LLM must adapt MPC parameters to satisfy natural-language behavioral prompts (centerline tracking, target velocity, reversing, smooth driving). A second module, DecisionxR1, is trained on static binary adherence labels. Evaluation is conducted with rephrased prompts on an unseen, more complex track (The Grand Tour), and a 'control adaptability' improvement over the default MPC is reported. The headline results are that Qwen2.5-3B with SFT+RLVR reaches 63.3% versus GPT-4o's 58.5% on the aggregate 'Improve [%]' score, and that Qwen2.5-1.5B improves from 11.3% (SFT-only) to 41.5% (SFT+RLVR). The paper also reports quantization on a Jetson Orin AGX and a physical-robot demonstration of compound prompt following.","tokens_in":18521,"tokens_out":13570,"duration_ms":105076,"significance":"If the central comparison were robust, this would be a valuable contribution: it shows that small, edge-deployable LLMs can learn control-adaptation behavior through closed-loop RL on a single consumer GPU, without distilling from a large model. The experimental design has real strengths: training and evaluation on different tracks, randomized rephrased prompts, a controlled ablation across base/SFT/RLVR conditions, released code, training curves in Appendix A, and a physical deployment experiment. However, the headline '63.3% vs 58.5%' comparison is currently neither checkable nor robust: the aggregation rule behind the 'Improve [%]' column is undefined, no variance is reported, a defensible alternative aggregation reverses the ranking, and the evaluation metric is essentially the same objective as the training reward. These issues are fixable in revision, but the central claim should not be accepted as stated.","major_comments":[{"comment":"The 'Improve [%]' column that carries the paper's headline comparison is never defined. The reported values are consistent with an unweighted mean of the four per-metric percentage improvements (e.g., GPT-4o: (5.0+93.2+97.8+38.0)/4 = 58.5; Qwen2.5-3B SFT+RLVR: (39.9+90.2+91.2+31.8)/4 = 63.3), but no formula, weighting, or justification is given in the text, the table caption, or Appendix C. The choice of aggregation is load-bearing: on three of the four metrics (EV, ER, ES) GPT-4o is better than Qwen2.5-3B SFT+RLVR (0.14 vs 0.19 m/s, 0.12 vs 0.48 m/s, 1.10 vs 1.21 m/s^2), and the 3B model wins only on EC (0.41 vs 0.65 m). An equally simple aggregation, the sum of raw RMSEs relative to the default sum (0.68+1.98+5.44+1.77 = 9.87), yields 79.6% for GPT-4o (sum 2.01) versus 76.8% for Qwen2.5-3B (sum 2.29), reversing the reported ordering. Because the claim that a small LLM surpasses GPT-4o rests entirely on this column, the paper must state the aggregation rule explicitly, justify it (the metrics have different units and baselines; the ER improvement is nearly trivially achieved by setting negative velocity bounds, as shown in Appendix D), and demonstrate that the conclusion is stable across reasonable aggregation choices.","section":"Section 4.3, Table 2"},{"comment":"No variability information is reported for the control adaptability scores. The caption of Table 2 states that each entry is the mean over five independent runs using different prompts in B', but the per-prompt metric values, standard deviations, and confidence intervals are absent, as is any significance test. With only five evaluation prompts, the headline 4.8-point gap between 63.3% and 58.5% may be within run-to-run noise. The authors should provide the per-prompt breakdown (or at least the per-metric dispersion), add more evaluation prompts or temper the superiority claim, and apply a consistent policy for the extraction-failure rows: the Qwen2.5-3B SFT-only row (one extraction failure, Improve 50.4%†) is not directly comparable to rows with zero failures, so the reported 12.9-point gain over that baseline should be recomputed with a stated handling of failures.","section":"Section 4.3"},{"comment":"The evaluation metric substantially overlaps with the training reward. Rdrive in Eq. (3) is the relative RMSE improvement (EMPC_B - ELLM_B)/EMPC_B over the default MPC, and the 'Improve [%]' column in Table 2 is an aggregate of exactly this kind of relative improvement against the same default MPC, with baselines re-measured on the evaluation map. The SFT+RLVR models are therefore evaluated on the objective they were trained to maximize, whereas GPT-4o was never trained on this objective; the ranking thus partly reflects optimization of the authors' chosen metric rather than general control adaptability. The train/evaluation track separation and the rephrased prompts mitigate this concern but do not remove it. As a concrete test, I ask the authors to report at least one evaluation that is not derived from Rdrive (for example, a fixed-weight composite cost chosen before training, or judged adherence of the resulting driving trajectories), or to show that the Qwen-versus-GPT-4o ranking is stable under alternative metric definitions.","section":"Section 3.3, Eq. (3)"},{"comment":"The claimed 20.2%-point improvement for Qwen2.5-1.5B over its SFT-only baseline contradicts the numbers in Table 2. The text states 'SFT: 11.3% / SFT+RLVR: 41.5%' and attributes to this a 20.2%-point improvement, but 41.5 - 11.3 = 30.2 points. The same 20.2 value appears in the abstract, in contribution (iii), and in the conclusion, so this is not a localized typo; the arithmetic should be corrected and all affected claims revised.","section":"Abstract, Section 4.3, Section 5"}],"minor_comments":[{"comment":"Typos and wording issues should be fixed: 'succesfully' (Section 4.4), 'improvment' (Section 4.3), 'Adjsut' (Appendix C), 'trackign' (Appendix D), and 'qual contribution' (author footnote).","section":"Various"},{"comment":"The abstract and conclusion should state that the 63.3% score refers to the full-precision model: the deployed Q5-quantized 3B model achieves 47.9% (Table 5), which is below the GPT-4o reference of 58.5%. As written, the claim that on-board deployment can outperform larger models overstates what the deployed configuration actually demonstrates.","section":"Abstract and Section 5"},{"comment":"The caption should give the definition of the 'Improve [%]' aggregation in one sentence rather than requiring the reader to infer the rule from the numbers.","section":"Table 2 caption"},{"comment":"Clarify which model configuration requires approximately 11 GB of VRAM for GRPO training, and give the batch size and sequence settings used.","section":"Section 4.1"},{"comment":"The map figure would benefit from a scale bar and from explicit labels connecting the Circle map to the training rows and The Grand Tour map to Table 2.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The arXiv header of this manuscript states that it has been accepted for CoRL 2025; this does not affect my technical assessment, but it is worth verifying the provenance of the version under review. The principal risk to the published claim is the undefined 'Improve [%]' aggregation combined with the absence of variance reporting; I would ask the authors to release the per-prompt data for Table 2. If the ranking reverses under a defensible alternative aggregation, the contribution should be reframed as competitive with GPT-4o on a chosen metric rather than surpassing it. The topic fits the venue well, and the experimental design (train/test track separation, rephrased prompts, physical robot deployment, modest training compute) is a genuine strength that should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution is the loop: GRPO-trained Qwen models that, in closed loop with an MPC and simulator, output parameter edits to satisfy a natural-language driving command. The authors train on a circle map, evaluate on a more complex track with rephrased prompts, and ship code, training curves, and behavior sets. That is a legitimate step beyond the static-dataset R1-zero work and beyond the SFT-centric baselines they cite. I believe their novelty claim is fair.\n\nThe paper also does several things right. The appendices are unusually transparent: full behavioral sets, RAG contents, failure counts, training reward curves, and quantization numbers. They do not hide extraction failures. The real-robot demo is qualitative, but it is a real deployment with a quantized 3B model on a Jetson, and it shows the mechanism working.\n\nThe soft spots are real and concentrated in the headline. The 'Improve [%]' column in Table 2 is never defined. The numbers imply an unweighted mean of four per-metric percentage improvements; a different but equally plausible aggregation over raw RMSE totals reverses the Qwen2.5-3B vs GPT-4o ordering. Since the metrics are in different units, any aggregation is a choice, not a fact. Reporting no variance for five per-prompt runs makes the 4.8-point gap even harder to trust. There is also a plain arithmetic error: SFT+RLVR 41.5% versus SFT 11.3% is a 30.2-point gain, not the 20.2 claimed in the abstract, results, and conclusion. That typo matters because the 20.2 number is used as a headline result.\n\nThe evaluation score also mirrors the training reward (relative RMSE improvement over default MPC). That is not damning—the generalization evidence is the new track and rephrased prompts—but it means the 'adaptability' claim is limited to the four trained behavior categories, not a general measure of control adaptability. Statistical tests are absent, and the physical deployment is a single qualitative demonstration. Those are fixable reporting gaps, not evidence of fraud.\n\nWho should read this: anyone working on LLM-guided control, edge deployment, or RLVR in continuous control. I would not quote the GPT-4o comparison until the aggregate is specified and variance is reported. If this were submitted today, I would send it to referees rather than desk reject, with the instruction that the metric definition and per-prompt numbers are mandatory revisions.","headline":"Genuinely useful closed-loop RLVR/GRPO method for LLM-driven MPC adaptation, but the headline 3B-beats-GPT-4o claim rests on an undefined aggregate and an arithmetic slip.","tokens_in":19115,"tokens_out":3863,"would_cite":true,"duration_ms":37182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a 3-billion-parameter on-board LLM can beat cloud-based GPT-4o at adapting a race car's control behavior after closed-loop reinforcement learning.","keywords":["RobotxR1","closed-loop reinforcement learning","RLVR","GRPO","embodied AI","autonomous driving","model predictive control","small language models"],"falsifier":"Reproduce Table 2 by computing each model's control adaptability as the mean of the four per-metric improvement percentages over the default MPC on The Grand Tour map: for Qwen2.5-3B with SFT+RLVR this gives (39.9 + 90.2 + 91.2 + 31.8)/4 = 63.3%, and for GPT-4o (5.0 + 93.2 + 97.8 + 38.0)/4 = 58.5%, so the headline ranking is checkable from the table alone; a different aggregation rule would change the ranking. A further check is to run the five rephrased prompts per behavior on the same map with the released code and compare the per-metric RMSEs against the reported values.","tokens_in":17983,"feed_emoji":"🚗","tokens_out":4333,"duration_ms":42465,"temperature":0.7,"pith_summary":"The paper tries to establish that small, edge-deployable language models can learn robotic control skills by interacting with a simulated environment through closed-loop reinforcement learning, rather than only by imitating large models through supervised fine-tuning. It extends the R1-zero recipe, originally built for static math datasets, to a driving task where an LLM proposes changes to a model predictive controller's parameters and receives a reward based on actual driving error. The key claimed result is that a Qwen2.5-3B model trained with supervised fine-tuning followed by closed-loop reinforcement learning reaches a 63.3% control adaptability score, surpassing the 58.5% obtained by the much larger, cloud-bound GPT-4o. If correct, the result would mean that practical embodied intelligence does not require huge cloud models, since a small model trained through environmental feedback can outperform them on a real robot.","feed_headline":"A 3B on-board LLM beats GPT-4o at driving control","feed_subtitle":"After laps in simulation, it scores 63.3% control adaptability versus GPT-4o's 58.5% on a scaled car.","key_machinery":"The central mechanism is the MPCxR1 closed-loop training loop: an LLM outputs a candidate set of MPC parameters (new_mpc_params), the car drives a lap in the F1TENTH simulator under those parameters, and a driving reward Rdrive = max((EMPC_B − ELLM_B)/EMPC_B, −4) is computed from behavior-specific RMSE metrics, with GRPO updating the LLM. The trailing modules are DecisionxR1, which learns to judge whether the car adheres to a prompted behavior, and MPCxR1, which learns to adapt control parameters; both use RAG-augmented prompts and combine accuracy, formatting, and parameter-validity rewards.","core_discovery":"The paper claims that moving RLVR training from static datasets into a closed-loop simulator lets small LLMs learn to retune an MPC's cost weights and constraints so that a scaled autonomous car follows natural-language driving prompts. In the proposed two-stage procedure, a Qwen2.5-3B model first undergoes SFT and then GRPO-based RLVR, during which it proposes MPC parameter sets, the car completes laps in simulation, and the reward measures behavioral error reduction relative to the default MPC. The trained 3B model achieves a 63.3% control adaptability improvement over the default MPC on an unseen complex track with rephrased prompts, compared with GPT-4o's 58.5%; the 1.5B model gains 20.2 percentage points over its SFT-only baseline. The paper further claims that this works because the training signal is behavioral, derived from actual driving outcomes, not from text imitation.","pith_inferences":["The 'Improve' column in Table 2 matches the unweighted mean of the four per-metric improvement percentages: for Qwen2.5-3B with SFT+RLVR, (39.9 + 90.2 + 91.2 + 31.8)/4 = 63.3%, and for GPT-4o, (5.0 + 93.2 + 97.8 + 38.0)/4 = 58.5%, so the headline gap is reproducible from the table even though the paper never states the aggregation formula.","Because the reward is relative improvement over the default MPC, the metric's value depends on how well that default is tuned; a deliberately poor default would inflate all scores, so cross-paper or cross-system comparisons only make sense with an identical baseline controller.","A natural testable extension is to apply the same closed-loop GRPO recipe to another high-level controller interface, such as a manipulation planner with differentiable cost terms; the paper's own limitation section notes the framework is currently tailored to the autonomous car platform.","The appendix shows output token length decreases during RLVR despite no brevity reward, opposite to the long chain-of-thought behavior in DeepSeek R1, suggesting that embodied tasks favor short, decisive parameter edits, which could inform reward design in other embodied domains."],"forward_implications":["A 3B LLM trained with closed-loop RLVR can be quantized and deployed on an embedded Jetson Orin AGX, reaching about 38.78 tokens/s and 8.3 s average latency while handling compound prompts like 'Drive smoothly at 2 m/s'.","Behavior learned on a simple circular track transfers to a more complex, unseen evaluation track, indicating the closed-loop reward, not memorization of the training map, drives the adaptation.","SFT alone is insufficient: adding RLVR improves the 1.5B model from 11.3% to 41.5% control adaptability and the 3B model from 50.4% to 63.3%, while RLVR alone is worse than SFT, so the two-stage order matters.","Training is feasible on a single consumer GPU (about 24 hours and 11 GB VRAM for the 3B model), lowering the barrier to embodied RLVR experiments.","The comparison with GPT-4o suggests that for embodied control, closed-loop environmental feedback can compensate for a large gap in parameter count and cloud-scale pretraining."],"supporting_citations":[{"why":"Supplies the R1-zero/RLVR and GRPO training recipe that the paper extends from static math datasets to closed-loop robotics.","marker":"[18]"},{"why":"Provides the baseline DecisionxLLM/MPCxLLM architecture, RAG memories, LoRA-based SFT procedure, and the comparison setup used throughout the paper.","marker":"[32]"},{"why":"Defines the 1:10 scaled race car platform and autonomy stack used for training, evaluation, and physical deployment.","marker":"[39]"},{"why":"Provides the open-source F1TENTH simulator in which the closed-loop RL training and control adaptability evaluation run.","marker":"[42]"},{"why":"Provides the LoRA-based GRPO implementation that makes RLVR training feasible on a single consumer-grade GPU.","marker":"[23]"},{"why":"Provides the llama.cpp inference engine used for quantized onboard deployment on the Jetson Orin AGX.","marker":"[43]"}],"fun_headline_variants":["3B LLM trained via closed-loop RL outperforms GPT-4o in autonomous driving","Small on-board LLM beats cloud GPT-4o after simulator RL training","Closed-loop RL lets 3B LLM top GPT-4o in driving control","On-board 3B LLM scores 63.3% vs GPT-4o's 58.5% in driving","RL in simulator boosts small LLM beyond larger model for driving"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison rests on an unreported aggregation of four hand-chosen driving metrics into the 'Improve' column, and if those metrics or their weighting do not faithfully capture the prompted behaviors, the 63.3% versus 58.5% gap loses its meaning; no variance or formula is reported in the text.","fun_headline_variants_meta":{"raw":{"variants":["3B LLM trained via closed-loop RL outperforms GPT-4o in autonomous driving","Small on-board LLM beats cloud GPT-4o after simulator RL training","Closed-loop RL lets 3B LLM top GPT-4o in driving control","On-board 3B LLM scores 63.3% vs GPT-4o's 58.5% in driving","RL in simulator boosts small LLM beyond larger model for driving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3081,"prompt_tokens":1024,"completion_tokens":2057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1943}},"tokens_in":640,"tokens_out":2057,"duration_ms":13895,"temperature":1.0,"reasoning_tokens":1943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:56:10.148855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce Table 2 by computing each model's control adaptability as the mean of the four per-metric improvement percentages over the default MPC on The Grand Tour map: for Qwen2.5-3B with SFT+RLVR this gives (39.9 + 90.2 + 91.2 + 31.8)/4 = 63.3%, and for GPT-4o (5.0 + 93.2 + 97.8 + 38.0)/4 = 58.5%, so the headline ranking is checkable from the table alone; a different aggregation rule would change the ranking. A further check is to run the five rephrased prompts per behavior on the same map with the released code and compare the per-metric RMSEs against the reported values.","supporting_citations":[{"cited_title":"Baumann, E","cited_arxiv_id":null,"evidence_quote":"Defines the 1:10 scaled race car platform and autonomy stack used for training, evaluation, and physical deployment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the open-source F1TENTH simulator in which the closed-loop RL training and control adaptability evaluation run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LoRA-based GRPO implementation that makes RLVR training feasible on a single consumer-grade GPU."},{"cited_title":"Gerganov and O.-S","cited_arxiv_id":null,"evidence_quote":"Provides the llama.cpp inference engine used for quantized onboard deployment on the Jetson Orin AGX."}],"review_version":1}