{"id":"7296d9c5-c346-43e8-9cf5-08798e4c3617","arxiv_id":"2608.07004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new benchmark shows LLMs can design PID and sliding-mode controllers for nonlinear multi-DoF systems, and a 1.5B reasoning-distilled model matches larger models on the benchmark and in physical trials.","lead":"This paper introduces a benchmark of 132 feedback-controller design tasks for complex, multi-degrees-of-freedom dynamical systems and tests six large language models plus a small distilled model. It reports that larger models design working controllers more often, and that a 1.5B model trained on reasoning traces performs well enough for an edge-device robotic arm.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's pass/fail protocol is uncalibrated: without a non-LLM control baseline, absolute success rates and the edge-deployment claim are not established.","rationale":"The reader identified the hand-set criteria and missing classical baseline as the weakest point; my stress-test narrows this to a specific uncalibrated pass/fail condition and a decisive check. If a non-LLM controller passes the same 132 cases, then the reported success rates mostly measure the leniency of Section II-D rather than controller-design skill. This is load-bearing because the abstract's central claim is the feasibility of lightweight edge-deployable controller design, and absolute success on the benchmark is the evidence offered for it. The internal comparisons among models are less affected and appear consistent, so I am not moving the verdict to REJECT; the right outcome is to require the baseline, and ideally a threshold sensitivity analysis, before the absolute-capability claim can be accepted. This leaves the reader's CONDITIONAL verdict unchanged.","tokens_in":16345,"tokens_out":7463,"duration_ms":84939,"concrete_test":"Run all 132 CoDyControlBench cases with the exact Section II-D pass/fail test using a deterministic, hand-written saturated PID baseline with anti-windup, with gains set by a simple rule from the diagonal stiffness/mass bounds (e.g., Kp = 2*max k_ii, Kd = 2*sqrt(Kp), Ki = Kp/10, saturation at 10x the target). Compare its success rate and Eq. (7) score under Fixed-3 and Fixed-10 with Tables I and VIII. If the baseline achieves success within a few points of GPT's 94.8% or above the Think-Model's ~60%, the absolute success rates are uninformative and the edge-deployment claim must be weakened; if the baseline lands at or below Qwen's 50%, the benchmark has discriminative value and the current conditional verdict stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is in the evaluation protocol (Section II-D): the pass/fail test is feasibility screening, not controller-quality assessment. A trial only requires all final-1-s samples within a 2%/5% steady-state band and overshoot below 100%; Eq. (7) is scored only after passing, and no stability margin, settling time before the last second, control-effort bound, robustness check, or input-saturation penalty is applied. Since the 132 plants are fully actuated second-order systems and the code template permits saturation and integral anti-windup, a scripted high-gain saturated PID—no LLM and no per-task reasoning—could plausibly satisfy these criteria on most cases. The paper's own Section V shows GPT's edge over Qwen is mostly aggressive gains plus saturation (98% vs 21% use of output saturation), which is exactly what a lenient pass band rewards. Without a classical or scripted baseline anchoring the success rates, the absolute numbers (94.8%, 50%, Think-Model ~60%) do not yet establish controller-design capability, and the edge-deployment conclusion rests on an uncalibrated metric. The relative Think-vs-Answer comparison and the physical PAM trial are useful, but the core feasibility claim needs the baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoDyControlBench, a benchmark of 132 system configurations (spanning 1–6 DoFs, four system types, two coupling levels, three damping regimes, and two controller families) for evaluating LLM-based feedback controller design, and reports a three-run evaluation of six LLMs (GPT, Gemini, Claude, GLM, DeepSeek, Qwen). The authors find that GPT achieves the highest success rate (94.8%) and Qwen the lowest (50.0%), identify number of DoFs and controller type as the largest sources of variation, and compare GPT with Qwen to attribute the gap to gain selection and transient-limiting mechanisms. They then distill GPT-5.5 reasoning traces into a 1.5B-parameter model (Think-Model) using matched answer-distillation and base-model controls, report that the Think-Model maintains roughly stable success rates (54–69%) across 1–6 DoFs on a relaxed evaluation protocol, and validate the Think-Model on a one-DoF PAM-driven robotic arm where it achieves three-out-of-three successful tracking trials.","tokens_in":16623,"tokens_out":8519,"duration_ms":78100,"significance":"If the results hold, the paper provides a useful new benchmark for a relatively underexplored application of LLMs, and the matched answer-vs-reasoning distillation comparison is a valuable contribution: the Think-Model's edge over the Answer-Model and the base model is internally consistent, and the real-world PAM trials give a concrete demonstration of a 1.5B controller-design model on edge hardware. The paper is also transparent about several limitations, including the statement in Section II-D that the pass/fail criteria are for 'benchmark-level feasibility screening rather than formal safety verification' and the explicit disclosure of the relaxed protocol for the small models in Section III-C. However, the benchmark's validity as a measure of controller-design ability is not yet established, because no non-LLM control baseline is reported; the absolute success rates and the edge-deployment feasibility claim depend on the uncalibrated pass/fail threshold, and the small-model performance claims are framed in the abstract without the protocol caveats. These issues are addressable and do not undermine the relative comparisons within the paper.","major_comments":[{"comment":"The evaluation protocol is uncalibrated because no non-LLM control baseline is reported. The manuscript itself states that the pass/fail criteria are 'designed for benchmark-level feasibility screening rather than formal safety verification,' and the criteria (all final-1-s samples within a ±2% band and overshoot below 100%) are loose enough that a scripted high-gain saturated PID with anti-windup could plausibly pass most of the 132 fully actuated, constant-input-matrix plants. Section V provides supporting evidence: GPT's advantage over Qwen comes largely from aggressive gains plus output saturation (98% vs 21% use of saturation), which is exactly the behavior a lenient pass band rewards. Without a classical or scripted baseline (e.g., Ziegler-Nichols-tuned PID, a manually saturated high-gain PID, or an LQR) evaluated under the same protocol, the absolute success rates (94.8%, 50.0%, Think-Model ~60%) do not establish LLM controller-design capability, and the edge-deployment feasibility claim rests on an uncalibrated metric. Please add such a baseline and report its success rate and score.","section":"Section II-D, Eq. (7), pass/fail criteria"},{"comment":"The evaluation protocol for the 1.5B models and DeepSeek-V4-Pro differs from the main benchmark: the steady-state band is relaxed from 2% to 5%, fixed iterations increase from 3 to 10, and the text indicates only PID-family tasks were used in this comparison. The abstract and conclusion state that the Think-Model 'maintained stable performance across 1–6 DoFs' and 'outperformed the answer-distilled and base model on CoDyControlBench' without disclosing these relaxations. Because Table VIII's success rates are not directly comparable to Table I, the reader may overinterpret the distillation result. Please either evaluate the small models under the original 2%/Fixed-3 protocol as well, or explicitly state in the abstract and conclusion that the small-model claims are based on the relaxed protocol and the PID-only subset.","section":"Section III-C and Table VIII"},{"comment":"The composition of the 132 configurations is under-specified. The paper does not provide a table giving the number of systems per factor level (e.g., counts for each DoF by system type, damping regime, and coupling level), nor the total number of design tasks (each configuration appears to have both PID and SMC tasks, implying 264 tasks). The benchmark prompts and evaluation code are referenced as 'benchmark files' but no repository URL or supplementary file is included in the manuscript. Without this information, the benchmark cannot be reproduced, and the factor-level ranges in Table VII cannot be interpreted. Please include the full configuration grid and a release link.","section":"Sections II.A–II.B and Appendix A"}],"minor_comments":[{"comment":"The abstract contains a typo: 'traget tracking' should be 'target tracking.'","section":"Abstract"},{"comment":"The sentence 'For a fair comparison, DeepSeek-V4-Pro is additionally evaluated under the same configuration (PID, Fixed-10, 5% steady-state band)' is ambiguous if the 1.5B models were also evaluated on PID-only tasks; please state explicitly the controller-type scope of Table VIII.","section":"Section III.C"},{"comment":"The score weights (w_ess=2, w_mp=2, w_td=1) are hand-set; a brief sensitivity analysis (e.g., varying the weights and checking whether the score-based conclusions in Section IV change) would strengthen the control-quality comparisons.","section":"Section II.D"},{"comment":"The phrase '3DoF levels×24systems per level' should read '3 DoF levels (DoF=4,5,6) × 24 systems per level' for clarity.","section":"Section V"},{"comment":"The phrase 'Compare to it' should be 'Compared to it.'","section":"Section IV.B"},{"comment":"The PAM experiment section would benefit from stating the quantitative success criteria (5% steady-state band over the final time window) before reporting the 100% vs 33.3% success rates.","section":"Section III.C / Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the benchmark is potentially useful, but I cannot recommend acceptance until the evaluation protocol is anchored by a non-LLM baseline and the small-model protocol relaxations are disclosed in the abstract/conclusion. The missing configuration grid and repository link are also important for a benchmark paper; if the supplementary files contain these, the authors should state this explicitly in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things here are worth taking seriously. First, CoDyControlBench is a real extension of ControlBench: 132 configurations spanning 1-6 DoF, LTI/NLTI/LTV/NLTV, coupling, damping, and controller family. Second, the answer-vs-reasoning distillation comparison on a 1.5B student, with matched training episodes and a clear result, is a useful data point for edge deployment. The GPT-vs-Qwen analysis is also concrete: the gap comes down to gain scale and saturation, not code extraction.\n\nThe paper does its internal comparisons well: three runs per condition, standard deviations reported, and the GPT/Qwen failure-mode decomposition is informative. The physical PAM trials, three successes for Think-Model versus one for Answer-Model, are small but consistent with the benchmark story.\n\nNow the soft spots. The evaluation protocol in Section II-D is a feasibility screen, not a quality measure: final-second 2% steady-state band, overshoot below 100%, and a score computed only after passing. No stability margin, control effort bound, or robustness check. The paper itself notes GPT wins by using aggressive gains plus output saturation (98% vs 21%), which is exactly what a lenient band rewards. Without a classical baseline—say a scripted high-gain saturated PID—absolute success rates like GPT's 94.8% and the Think-Model's ~60% don't support the claim that LLMs are good controller designers, only that some LLMs are better at passing this particular test. The relative comparisons survive, but the edge-deployment conclusion leans heavily on an uncalibrated metric.\n\nTwo smaller issues. The thresholds were relaxed (2% to 5% band, 3 to 10 iterations) for the 1.5B models, so Table VIII is not directly comparable to Tables I-VI. That is acceptable for a floor-effect check, but it should be called out more plainly. And no code or benchmark files are released, so the central artifact is not reproducible. The paper includes a code template in the appendix, which helps, but not the 132 configurations or the evaluation harness.\n\nMy take: the paper deserves a serious referee. The benchmark is a genuine gap in the LLM-for-control literature, the distillation comparison is well-formed, and the physical validation is a nice addition. But the authors should be asked to release the benchmark files and to anchor the success rates with a non-LLM baseline before the absolute claims are published. I would not cite it in its current form, but I would read the revised version.","headline":"A genuine benchmark extension and a plausible distillation result, but the absolute claims need a classical baseline and artifact release before they become load-bearing.","tokens_in":17175,"tokens_out":2352,"would_cite":false,"duration_ms":23435,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 1.5B-parameter model trained on a stronger model's reasoning traces can design feedback controllers for complex dynamical systems well enough to run on edge hardware and track targets on a real…","keywords":["LLM-based controller design","feedback control","reasoning distillation","edge deployment","multi-DoF dynamical systems","PID tuning","sliding mode control","pneumatic artificial muscle"],"falsifier":"Run the same six models and the Think-Model on CoDyControlBench with the steady-state band tightened from ±2% to ±0.5% and the overshoot cap lowered to 20%, or inject a disturbance mid-simulation; if GPT's 94.8% success or the Think-Model's 54-69% stability collapses or reorders, the selected thresholds were carrying the result.","tokens_in":16128,"feed_emoji":"⚙️","tokens_out":6594,"duration_ms":64979,"temperature":0.7,"pith_summary":"The paper sets out to test whether large language models can design feedback controllers for complex dynamical systems—not just linear single-degree-of-freedom textbook plants—and whether a small model can do the job on edge hardware. To that end it introduces CoDyControlBench, 132 system configurations spanning 1-6 degrees of freedom, four system types, two coupling levels, three damping regimes, and two controller families. Six LLMs are scored on executable-code controller design; the strongest model reaches a 94.8% design success rate and the weakest reaches 50.0%. The central constructive claim is that reasoning distillation, training a 1.5B student on the teacher's diagnostic reasoning traces rather than only on final answers, produces a model that stays between roughly 54% and 69% success across all DoF levels and succeeds in all three physical trials on a pneumatic-artificial-muscle robotic arm. If true, lightweight, locally executed controller-design assistants are feasible without cloud-hosted frontier models.","feed_headline":"Distilled 1.5B LLM keeps controller design stable at 1-6 DoFs","feed_subtitle":"It beats answer-distilled and base models on a 132-system benchmark and tracks targets on a real pneumatic-muscle arm.","key_machinery":"The load-bearing artifact is CoDyControlBench itself: 132 parameterized mass-spring-damper systems in state-space form, each task requiring executable PID-family or sliding-mode-family controller code filled into a fixed template. Performance is first gated by pass/fail criteria—all outputs inside a ±2% steady-state band for the final second and overshoot below 100%—and then scored by Equation (7), a weighted penalty on steady-state error, overshoot, delay, and terminal oscillation. The second mechanism is reasoning distillation: the same 599 LTI tuning episodes train a 1.5B DeepSeek student either with teacher-generated diagnostic traces (Think-Model) or with concise rationales and code only (Answer-Model), with the small-model evaluation relaxed to a ±5% band and ten tuning iterations. The benchmark protocol plus the reasoning-trace supervision together carry the argument.","core_discovery":"The paper's central discovery is twofold. First, on CoDyControlBench a frontier commercial model (GPT) is the only evaluated model whose design success stays above 88% from 1 to 6 DoFs, while the largest aggregate performance swings come from number of DoFs and controller family rather than system type, coupling, or damping. Second, a 1.5B student distilled on GPT's reasoning traces reaches 54-69% success across all DoF levels, outperforms the answer-distilled variant, and achieves successful tracking in 3/3 physical trials on a PAM-driven arm. The paper attributes the best-versus-worst model gap to control-design knowledge—gain scaling and transient-limiting mechanisms such as output saturation and anti-windup—rather than to failures in generating executable code, since code-extraction failures are rare for both models.","pith_inferences":["Testable extension: tighten the benchmark gate from ±2% to ±0.5% steady-state band and lower the overshoot cap from 100% to 20%; the absolute success rates and possibly the model rankings would shift, testing whether the thresholds were carrying the result.","The distillation was trained only on LTI plants, so its success on nonlinear and time-varying benchmark cases suggests the student learned generic response-driven heuristics rather than memorized gains; a direct ablation would distill from a weaker teacher or shorter traces and measure how much of the Think-Model's margin survives.","Even the Think-Model fails in roughly 30-45% of benchmark trials, so a real edge deployment would likely need a fallback controller or human check rather than fully autonomous closed-loop authority.","The physical PAM validation used three trials at one target angle; scaling to more targets and adding disturbances would test whether the 3/3 success reflects generalizable tuning skill or favorable test conditions."],"forward_implications":["Edge deployment becomes practical: a 1.5B model running locally on a Jetson-class device can iteratively tune a PID controller to track a target on a nonlinear, hysteretic PAM arm.","DoF count and controller type are the benchmark dimensions that most separate models, so future controller-LLM work should emphasize high-dimensional dynamics and heuristic PID tuning rather than coupling or damping effects.","LLMs write executable sliding-mode controllers more reliably than they search PID gains, yet successful PID designs score better, revealing a trade-off between design feasibility and control quality.","The dominant failure mode of weaker models is mis-scaled gains, not non-executable code: the weakest model's controllers end in steady-state deviation above tolerance in 68.3% of attempts while diverging in only 0.2%.","Reasoning distillation transfers response-driven tuning strategies across system dimensions, while answer-only distillation does not generalize beyond low-DoF systems."],"supporting_citations":[{"why":"Defines ControlBench, the existing LLM controller-design benchmark that CoDyControlBench extends and compares against.","marker":"[24]"},{"why":"Supports the premise that 1.5B-scale models can run on hardware-accelerated single-board computers, motivating the student model size for edge deployment.","marker":"[33]"},{"why":"Provides the dynamic model and platform characteristics of the PAM-driven robotic arm used in the physical validation trials.","marker":"[34]"},{"why":"Supplies the multivariable-coupling complexity rationale behind the benchmark's coupling dimension.","marker":"[29]"},{"why":"Provides the sliding-mode-control design basis for the SMC-family tasks and their stability reasoning.","marker":"[32]"}],"fun_headline_variants":["Distilled 1.5B LLM beats answer-distilled on controller design","Edge-ready 1.5B model tracks targets on pneumatic-muscle arm","1.5B reasoning-distilled model keeps controller design stable at 1-6 DoFs","LLM control gap traced to knowledge, not code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that a trial passing the benchmark's hand-set thresholds—±2% steady-state band over the last second and overshoot under 100%, relaxed to ±5% and ten iterations for the small models—is a faithful proxy for successful controller design; if those thresholds are too loose or too task-specific, the absolute success rates and the edge-deployment conclusion do not transfer to real systems, even if relative model comparisons survive.","fun_headline_variants_meta":{"raw":{"variants":["Distilled 1.5B LLM beats answer-distilled on controller design","Edge-ready 1.5B model tracks targets on pneumatic-muscle arm","1.5B reasoning-distilled model keeps controller design stable at 1-6 DoFs","LLM control gap traced to knowledge, not code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001143,"raw_usage":{"total_tokens":4808,"prompt_tokens":1074,"completion_tokens":3734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":3650}},"tokens_in":690,"tokens_out":3734,"duration_ms":34102,"temperature":1.0,"reasoning_tokens":3650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:45:29.253495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six models and the Think-Model on CoDyControlBench with the steady-state band tightened from ±2% to ±0.5% and the overshoot cap lowered to 20%, or inject a disturbance mid-simulation; if GPT's 94.8% success or the Think-Model's 54-69% stability collapses or reorders, the selected thresholds were carrying the result.","supporting_citations":[{"cited_title":"Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers","cited_arxiv_id":"2604.24785","evidence_quote":"Supports the premise that 1.5B-scale models can run on hardware-accelerated single-board computers, motivating the student model size for edge deployment."},{"cited_title":"Optimization of spring constant of a pneumatic artificial muscle-spring driven antagonistic structure,","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic model and platform characteristics of the PAM-driven robotic arm used in the physical validation trials."},{"cited_title":"Skogestad and I","cited_arxiv_id":null,"evidence_quote":"Supplies the multivariable-coupling complexity rationale behind the benchmark's coupling dimension."},{"cited_title":"Experimental comparison of control strategies for a periph- eral nerve interface actuator,","cited_arxiv_id":null,"evidence_quote":"Provides the sliding-mode-control design basis for the SMC-family tasks and their stability reasoning."}],"review_version":1}