{"id":"434b9125-215a-480e-a40b-11e2c402d674","arxiv_id":"2507.12855","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DEMONSTRATE learns a zero-shot mapping from natural-language embeddings to MPC cost parameters from demonstrations, achieving tabletop manipulation success rates comparable to prior LLM-based pipelines.","lead":"This paper presents DEMONSTRATE, a pipeline that turns natural-language commands into optimal control problems using robot demonstrations instead of hand-crafted mathematical prompt examples. It reports success rates comparable to or better than prior LLM-to-control methods on tabletop manipulation tasks, and adds a validation step that blocks commands too different from the demonstrated examples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sub-task validation threshold in Eq. (3) is uncalibrated; embedding-space coverage is not shown to predict whether the learned map M yields a valid θ, so the hallucination-assessment claim is unsupported.","rationale":"The reader's weakest-assumption analysis focused on Assumption 1, that all tasks share the same m and ρ, and correctly noted that the experiments only cover a single task family of relative translations and rotations. I agree that the claimed broader applicability is unproven. However, the more immediately load-bearing internal gap is the sub-task validation gate in Eq. (3). Even within the tested family, the threshold t is a free parameter with no sensitivity analysis, and the paper does not establish that embedding-space coverage predicts whether the MLP-generated θ will produce a successful OCP. The three illustrative examples in Fig. 2 and the drawer experiment in Table II are anecdotal rather than a calibration study. Because the central novelty over NARRATE is precisely the ability to assess hallucinations before execution, this gap directly affects the main claim. I would keep the reader's conditional verdict: the paper is a plausible proof of concept, but the hallucination-assessment claim needs a systematic calibration experiment before being taken at face value.","tokens_in":12597,"tokens_out":5714,"duration_ms":69241,"concrete_test":"Construct a labeled OOD evaluation set of candidate sub-task commands by systematically varying distance magnitudes, object nouns, synonyms, and word order around the 90 demonstrated sub-tasks, plus clearly unrelated commands. For each command, run the full DEMONSTRATE pipeline and record (i) the coverage coefficient from Eq. (3) and (ii) whether execution succeeds/fails. Then plot the ROC curve for the coverage coefficient as a detector of execution failure and report false-accept and false-reject rates at the claimed threshold t=3.0. If the AUC is not clearly above chance or the t=3.0 operating point has high error rates, the hallucination-assessment claim is unsupported; if the gate is well-calibrated across these perturbations, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the uncalibrated coverage gate that carries the hallucination-assessment claim. Eq. (3) rejects a proposed sub-task when the residual of the least-squares fit of its embedding to the demonstration embeddings exceeds a user-defined threshold t, set to 3.0 in Section VI.D. But the quantity that actually determines whether the OCP is meaningful is the cost weight θ produced by the MLP M(q, e) at the output of the Optimization Designer (Section IV.B.1.c). The paper provides no Lipschitz or regularity guarantee for M, so embeddings that are close in the PCA-reduced space need not yield similar θ; a small coverage residual can still map to an invalid or unsafe OCP, and a large residual can map to a perfectly executable one. The evidence for the gate consists of three hand-picked examples in Fig. 2 and one drawer-distance extrapolation in Table II. No sensitivity analysis of t is reported, no ROC/calibration analysis, and no test of whether the gate's accept/reject decisions track downstream success rather than embedding geometry. Since avoiding LLM hallucination before execution is a core contribution, this missing calibration is load-bearing: if the threshold is arbitrary, the method has not demonstrated a principled hallucination assessment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DEMONSTRATE, a pipeline that maps natural-language task descriptions to optimal control problems (OCP) without requiring an LLM to generate cost or constraint expressions. The cost function is represented as a feature vector θ multiplied by shared feature functions φ(x,u,m), and constraints as shared templates g(x,u,ρ)≤0. The core idea is to learn, offline, a parametric mapping M from compressed sentence embeddings of sub-task descriptions to θ, using maximum-entropy inverse optimal control on demonstrations, under Assumption 1 that all tasks share m and ρ. Online, a task planner breaks a user command into sub-tasks, each embedding is checked against a coverage threshold (Eq. 3) to assess similarity to the demonstrations, and the learned M produces θ, which is used in an MPC. The paper reports simulation and hardware experiments on tabletop manipulation (stacking cubes, writing L, building a pyramid, wiping a pan) and compares against NARRATE, VoxPoser, and Code-as-Policies. The central claims are zero-shot generalization to new tasks and the ability to assess hallucinations before execution.","tokens_in":12796,"tokens_out":4180,"duration_ms":48515,"significance":"The work addresses a real bottleneck in LLM-based control: the need for expert-engineered in-context examples and the lack of pre-execution hallucination checks. The proposed combination of inverse optimal control with mult-task representation learning is a sensible and potentially practical direction. The paper is transparent about the experimental protocol, ships code and videos, and the optimization in Eq. (5) is a clean multi-task extension of maximum-entropy IOC. If the hallucination-gating mechanism were properly calibrated and the scope of the multitask assumption were validated, the method could be a useful contribution. However, as presented, the load-bearing claims about generalization and pre-execution hallucination assessment are not fully supported by the evidence.","major_comments":[{"comment":"Assumption 1 restricts all tasks to share the feature parameters m and ρ, with task specificity only through θ. This means the method can only represent tasks whose cost features and constraint templates are identical up to a cost-weight vector. The experiments exclusively consider relative pick-and-place sub-tasks (translations and rotations), which trivially satisfy this homogeneity. The conclusion (Section VII) nevertheless claims execution of 'a diverse array of tasks.' This overstates the validated scope. If a target task requires a different constraint type or cost feature structure, the method cannot represent it at all, regardless of embedding similarity. The paper should either demonstrate a broader class of task families or explicitly limit the claim to the tested pick-and-place setting.","section":"§IV.A.1.a, Assumption 1"},{"comment":"The sub-task validation gate is the primary mechanism for the paper's core claim of hallucination assessment before execution, but the threshold t is a hand-picked constant (3.0) with no sensitivity analysis, no calibration against downstream success, and no statistical characterization. More fundamentally, the coverage coefficient is computed in the embedding space, whereas the actual validity of the OCP depends on θ = M(q, e) produced by the MLP. The paper provides no Lipschitz or regularity guarantee for M, so two embeddings that are close in PCA-reduced space need not yield similar θ, and a small coverage residual does not imply a valid or safe OCP. The footnote in Section V.A/B implicitly acknowledges this gap by noting that the theoretical motivation from [35] does not directly apply because θ is a neural-network output. The evidence for the gate consists of three hand-picked examples (Fig. 2) and one extrapolation study (Table II). This is insufficient to support the pre-execution hallucination-assessment claim. The authors should provide either a theoretical bound relating embedding coverage to θ-validity or an empirical calibration study showing that gate accept/reject decisions track downstream task success.","section":"§IV.B.1.b, Eq. (3) and §VI.D"},{"comment":"The statement 'DEMONSTRATE shows comparable or higher success rates than the best performing base line model in all 4 tasks' is contradicted by the data in Table I for the Stack task: DEMONSTRATE achieves 88% success vs. CaP's 98%. The claim should be corrected to reflect the actual comparisons. Additionally, the baseline numbers are taken from the prior NARRATE paper rather than re-run in the same environment, and no confidence intervals or significance tests are reported. Given that the difference between 88% and 98% could be within statistical noise for 50 runs, the claim of 'comparable or higher' requires at least error bars or a significance test.","section":"§VI.C, Table I"}],"minor_comments":[{"comment":"In the line before Eq. (5), the optimization variables q and m are missing the estimated notation (hat) used elsewhere; similarly, the left-hand side of Eq. (5) omits the hat. Please make the notation consistent.","section":"Eq. (5) and surrounding text"},{"comment":"The dynamics constraint is written with indices i=1,...,N, but with x0 given, the natural range should be i=0,...,N-1 to cover all state transitions. The same index inconsistency appears in Eq. (6b) (k=1,...,N-1 vs. k=0,...,N-1 in the reformulated SMC problem).","section":"Eq. (1b)"},{"comment":"The sentence-transformer model 'all-mpnet-base-v2' is cited to [34] (the MPNet paper), but the sentence-transformers library is more directly associated with [33] (Sentence-BERT). It would be clearer to cite the actual model card or the SBERT framework.","section":"Section II and references"},{"comment":"The hardware experiments are described only qualitatively with a figure and video references; reporting success rates or per-trial outcomes for the hardware runs would strengthen the reproducibility claims.","section":"Section VI.F"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising, but the paper needs to address the calibration of the sub-task validation gate and the overclaim in Table I before I can support its publication. The Assumption 1 scope limitation could be acceptable for a proof of concept if the authors clearly delimit the claims and add a broader task family. The novelty is adequate for a robotics venue, but the current evidence is not yet at the level claimed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead DEMONSTRATE. The core idea is genuinely different from NARRATE: rather than having an LLM write out cost and constraint expressions, they learn a map from sentence embeddings to cost weights using multi-task IOC on demonstrations. That is a real shift and it removes the need for expert-crafted in-context examples. The multitask structure is honestly stated—Assumption 1 fixes shared m and rho, task-specificity is only in theta—and the experiments stick to families that satisfy it. The paper also ships code and videos, which helps.\n\nThe weak point is the sub-task validation gate, Eq. (3). The claim is that it detects hallucinations before execution, but the threshold t=3.0 is hand-picked, there is no sensitivity analysis, and the link between embedding distance and validity of the resulting OCP is not established. The MLP M has no Lipschitz regularity guarantee, so two nearby embeddings can still yield very different cost weights. The evidence is three examples in Fig. 2 and one drawer-distance extrapolation. That is not enough to support a hallucination-assessment claim.\n\nAlso, the baselines are not re-run; the numbers are taken from [11]. That is fine only if the environments are identical, which is not shown. Success rates have no confidence intervals, and the hardware results are qualitative. The demonstrations come from a scripted controller in simulation, not human teleoperation, so the main motivating use case is untested. These are not fatal, but they limit how strongly the claims can be stated.\n\nNone of this sinks the paper. The method is plausible, the experimental protocol is transparent, and the limitations are acknowledged. It deserves a serious referee. My recommendation: send it out, but expect a major revision. The authors should re-run or justify the baselines, add intervals, and either calibrate the gate or temper the hallucination claim.","headline":"A clever demonstration-driven alternative to LLM prompt engineering for MPC, with a genuinely useful validation idea, but the gate's calibration is unproven and the baseline comparison is inherited.","tokens_in":13404,"tokens_out":2926,"would_cite":false,"duration_ms":32828,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned map from task descriptions to control costs lets robots execute new commands from demonstrations, matching or beating LLM-based baselines.","keywords":["language-conditioned control","inverse optimal control","multi-task representation learning","model predictive control","hallucination detection","learning from demonstrations","zero-shot generalization","robotic manipulation"],"falsifier":"In a task family where the true cost weights are known, hold out several commands, run the offline learning on demonstrations of the rest, and compare the predicted $\\theta$ from the parametric map with the true $\\theta$: if the coverage test passes and the predicted weights yield trajectories that clearly fail the known objective, the central claim that the embedding-to-weight map is sufficient is wrong. Similarly, a command semantically inside the family whose embedding lands outside the demonstration span would show that the validation check is not a faithful hallucination test.","tokens_in":12344,"feed_emoji":"🤖","tokens_out":8465,"duration_ms":90430,"temperature":0.7,"pith_summary":"This paper tries to establish that a robotic controller can be specified by natural language without an LLM writing the underlying math: instead, demonstrations of sub-tasks teach a parametric map from compressed sentence embeddings to the cost weights of a model predictive control problem. The map is learned with inverse optimal control, and all tasks are constrained to share the same cost-feature and constraint templates, so a new spoken command within the same task family is executed zero-shot. A coverage-based validation step rejects commands whose embeddings are too far from the demonstrations, giving a pre-execution hallucination check that LLM-based controllers lack. In tabletop pick-and-place experiments, the authors report success rates comparable to or higher than the best baseline in all four tasks, with the acknowledged tradeoff that commands far from the demonstrations are refused or fail.","feed_headline":"Demonstrations replace LLM-written math for robot commands","feed_subtitle":"A few demonstrations teach the mapping from words to control costs, and a coverage check blocks out-of-family commands before execution.","key_machinery":"The load-bearing object is a shared, parameterized optimal-control template: cost features $\\varphi(x,u,m)$ and constraint functions $g(x,u,\\rho)$ are common across all tasks, while task identity lives entirely in the weight vector $\\theta$ predicted by the map $M$. Learning uses the maximum-entropy inverse optimal control likelihood from locally optimal demonstrations, approximated by gradients and Hessians of the cost, to fit $M$ and the shared parameter $m$ jointly; constraint parameter $\\rho$ is fit from safe and unsafe demonstrations with a satisfiability-modulo-convex program. The online sub-task validation uses the coverage coefficient of the target embedding against the span of demonstration embeddings to decide whether to re-plan or refuse execution.","core_discovery":"The central claim is that, for any task family whose costs and constraints share the parameterized form $c_t(x,u)=\\theta_t^\\top \\varphi(x,u,m)$ and $g(x,u,\\rho)\\le 0$ with shared $m,\\rho$, a map $M$ from PCA-compressed language embeddings $\\tilde{e}_t$ to task-specific cost weights $\\theta_t$ can be learned jointly with $m$ from suboptimal demonstrations via an entropy-maximization inverse optimal control objective. Constraints are estimated separately from demonstrations that avoid an unsafe region, using satisfiability-modulo-convex programming. Because $\\theta_t$ is produced by the map from the embedding rather than by LLM code generation, outputs are guaranteed to lie in the pre-defined task feature space, and the coverage test of Eq. (3) measures whether a planned sub-task is close enough to the demonstration set before execution. The paper reports comparable or higher success rates than the best baseline in all four simulated tasks and demonstrates hardware execution of stacking, L-shape, and pyramid tasks.","pith_inferences":["A testable consequence the authors do not pursue: the same shared template could support multiple task families if $m$ and $\\rho$ were learned once per family and composed, effectively making the pre-implemented feature library the true scope of the method.","Because validation operates on embeddings, commands that are semantically different but embedded nearby, or paraphrases with different numbers, might pass or fail the coverage test in ways unrelated to control feasibility; separating numeric from semantic content, as the conclusion suggests, could be evaluated directly.","If human demonstration datasets were used to train the map, the method could scale beyond the proof-of-concept simulation demonstrations; the paper lists this as future work, but it is a natural next experiment given the pipeline's reliance on demonstrations rather than on prompt engineering."],"forward_implications":["A user can issue new commands in the trained task family without an engineer writing symbolic costs or constraints; only demonstrations and language descriptions are needed.","The LLM's role is reduced to task decomposition and embedding, so failures shift from malformed code to the learned map, and the coverage check can catch out-of-family sub-tasks before execution.","Because the map is learned from a few demonstrations per sub-task (20 demonstrations for 90 sub-tasks in the experiments), the approach is practical for tabletop manipulation and likely scales with more demonstration data.","The reported success rates on the benchmark tasks are comparable to or higher than the best baseline in the paper's comparison, while far-from-demonstration commands are deliberately refused; the demonstrated design goal is reliability within a task family rather than open-ended generalization."],"supporting_citations":[{"why":"The NARRATE architecture that DEMONSTRATE modifies; supplies the LLM task planner, trajectory generator and tracker, and the baseline success-rate numbers.","marker":"[11]"},{"why":"Continuous inverse optimal control with locally optimal examples; provides the approximate maximum-likelihood objective used to fit the map and shared cost parameter.","marker":"[29]"},{"why":"Learning parametric constraints in high dimensions from demonstrations; provides the safe and unsafe demonstration procedure and SMC formulation used to estimate constraint parameters.","marker":"[31]"},{"why":"The masked-and-permuted sentence transformer used to embed sub-task descriptions into the vectors that feed the parametric map.","marker":"[34]"},{"why":"Nonlinear representation-learning guarantees; motivates the coverage coefficient used in the sub-task validation test of Eq. (3).","marker":"[35]"},{"why":"Maximum entropy inverse reinforcement learning; supplies the exponential likelihood model at the core of the cost-learning step.","marker":"[28]"},{"why":"The benefit of multitask representation learning; motivates sharing a common representation so example and target tasks are similar by construction.","marker":"[13]"},{"why":"Code as Policies, a baseline that has an LLM write executable robot code; provides the comparison that DEMONSTRATE must match.","marker":"[7]"},{"why":"VoxPoser, a baseline that composes 3D value maps from LLM affordances; provides another comparison point for success-rate benchmarking.","marker":"[40]"}],"fun_headline_variants":["No LLM math needed: demos teach robot control via embeddings","Robot control from few demos, no engineer-written equations","DEMONSTRATE: demos replace LLM math for safer robot commands","Language to control via demos, not LLM-generated formulas","Task demos, not code: zero-shot robot control from language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every task, demonstrated or commanded, fits the same cost and constraint template and differs only in a small set of task weights; command a task outside that family and the learned map has nothing valid to output.","fun_headline_variants_meta":{"raw":{"variants":["No LLM math needed: demos teach robot control via embeddings","Robot control from few demos, no engineer-written equations","DEMONSTRATE: demos replace LLM math for safer robot commands","Language to control via demos, not LLM-generated formulas","Task demos, not code: zero-shot robot control from language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1460,"prompt_tokens":1005,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":621,"tokens_out":455,"duration_ms":4724,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:37:04.189429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a task family where the true cost weights are known, hold out several commands, run the offline learning on demonstrations of the rest, and compare the predicted $\\theta$ from the parametric map with the true $\\theta$: if the coverage test passes and the predicted weights yield trajectories that clearly fail the known objective, the central claim that the embedding-to-weight map is sufficient is wrong. Similarly, a command semantically inside the family whose embedding lands outside the demonstration span would show that the validation check is not a faithful hallucination test.","supporting_citations":[{"cited_title":"Narrate: Versatile language architecture for optimal control in robotics,","cited_arxiv_id":null,"evidence_quote":"The NARRATE architecture that DEMONSTRATE modifies; supplies the LLM task planner, trajectory generator and tracker, and the baseline success-rate numbers."},{"cited_title":"Continuous inverse optimal control with locally optimal examples,","cited_arxiv_id":null,"evidence_quote":"Continuous inverse optimal control with locally optimal examples; provides the approximate maximum-likelihood objective used to fit the map and shared cost parameter."},{"cited_title":"Learning parametric constraints in high dimensions from demonstrations,","cited_arxiv_id":null,"evidence_quote":"Learning parametric constraints in high dimensions from demonstrations; provides the safe and unsafe demonstration procedure and SMC formulation used to estimate constraint parameters."},{"cited_title":"Mpnet: Masked and permuted pre-training for language understanding,","cited_arxiv_id":null,"evidence_quote":"The masked-and-permuted sentence transformer used to embed sub-task descriptions into the vectors that feed the parametric map."},{"cited_title":"Maximum entropy inverse rein- forcement learning,","cited_arxiv_id":null,"evidence_quote":"Maximum entropy inverse reinforcement learning; supplies the exponential likelihood model at the core of the cost-learning step."},{"cited_title":"The benefit of mul- titask representation learning,","cited_arxiv_id":null,"evidence_quote":"The benefit of multitask representation learning; motivates sharing a common representation so example and target tasks are similar by construction."},{"cited_title":"Code as policies: Language model programs for embodied control,","cited_arxiv_id":null,"evidence_quote":"Code as Policies, a baseline that has an LLM write executable robot code; provides the comparison that DEMONSTRATE must match."}],"review_version":1}