{"id":"abb7ce60-5b08-4107-9e10-0f4483681160","arxiv_id":"2502.02705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SGFT uses a simulation-trained value function to guide real-world exploration via potential-based reward shaping and short-horizon objectives, substantially improving fine-tuning sample efficiency.","lead":"This paper introduces Simulation-Guided Fine-Tuning (SGFT), a method that uses a value function trained in simulation to reshape rewards and shorten the planning horizon during real-world robot fine-tuning. On five real-world contact-rich manipulation tasks, SGFT reports reaching high success rates with far fewer samples than standard fine-tuning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical guarantee rests on the unverified 'improvable Vsim' assumption; without an empirical check of Definition 1 the theorem may not cover the contact-rich regime where SGFT claims novelty.","rationale":"I read the paper as making two intertwined claims: an empirical claim that SGFT accelerates real-world fine-tuning, and a theoretical claim that this acceleration is guaranteed even for short horizons under suitable conditions. The empirical evidence is suggestive but lacks statistical detail, and the theory is the paper's stated justification for why the method should generalize beyond the five test tasks. The reader identified the improvable assumption as the weakest point; I agree. Definition 1 is not a mild regularity condition like bounded rewards or Lipschitz dynamics. It asserts that the real dynamics permit one-step increases in Vsim at least as large as the simulation Bellman equation provides, everywhere. This is precisely the kind of condition that can fail in contact-rich manipulation, where the simulator may make motions feasible that are not feasible in reality. The paper's only evidence is a contrived pendulum example and an appeal to geometric intuition; no empirical check is provided even where it would be free, namely in the sim-to-sim benchmarks with known target dynamics. I also examined the proof of Theorem 1 and found that Lemma 2 as stated is internally inconsistent: with a perfect model (α=0) and H=1, the optimal H-step policy under the model is identical to the true optimal H-step policy, so the left side of Eq. (5) is zero while the right side is positive. This is a fixable proof bug, not a fatal flaw in the method, and it is secondary to the improvable-assumption issue. The single most load-bearing concern is therefore that the theoretical guarantee rests on an unverified structural condition that may not hold in the exact regime where the paper claims novelty. This does not invalidate the empirical results, but it means the paper should be accepted conditionally, with a direct test of Definition 1 and a corrected proof before the theoretical claim is taken as established.","tokens_in":21389,"tokens_out":13750,"duration_ms":132854,"concrete_test":"In the sim-to-sim benchmarks of Section 6.3, compute the improvement margin M(s) = max_a E_{s'~p_target(·|s,a)}[γVsim(s')] - Vsim(s) + r(s) over a dense sample of states visited during SGFT fine-tuning, using the exact target dynamics. Check whether M(s) ≥ 0 at essentially all sampled states (with a tolerance for function-approximation error). If M(s) is negative at a non-negligible fraction of states, the central premise of Theorem 1 fails in the very benchmarks used to validate SGFT, and the empirical success must be explained by a mechanism other than the stated theorem. If M(s) ≥ 0 throughout, the improvable assumption is supported in the sim-to-sim setting, partially resolving the concern; for the real-world tasks, repeat the same computation with the learned model p_hat on logged real states as a proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 applies only if Vsim is improvable with respect to Mreal (Definition 1): for every state, some action must satisfy E_{s'~preal}[γVsim(s')] - Vsim(s) ≥ -r(s). This is a per-state condition on the real dynamics, not a consequence of a large sim-to-real gap. The paper's support for it is a deterministic pendulum example where the sets of feasible next states coincide in simulation and reality, followed by an assertion that this geometric similarity is 'reasonable' for continuous control. But the five real tasks are selected precisely because contact dynamics are misspecified: in insertion the simulated policy wiggles the leg into the hole while real friction and observation noise impede that motion; in hammering the high-impact nail contact is not accurately modeled. In such settings it is entirely plausible that for some states the best real-world action increases Vsim more slowly than the simulation Bellman equation requires, making the margin negative. If the margin is negative anywhere, Lemma 3 (monotonicity of V*_H) can fail and Theorem 1 provides no bound; SGFT could in principle stall or be biased toward Vsim maxima that are not attainable under preal. The paper never measures this margin, even in the sim-to-sim benchmarks of Section 6.3 where the target dynamics are known exactly. Separately, Eq. (5) in Lemma 2 as printed is false: for α=0 and H=1, the left side is 0 but the right side is positive, so the proof of Theorem 1 needs correction. This reinforces that the theory is not presently a reliable guarantee for the claimed regime, even though the empirical method may still work.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Simulation-Guided Fine-Tuning (SGFT), a framework for sim-to-real policy adaptation that uses a value function Vsim learned in simulation as a potential-based reward shaper and shortens the fine-tuning horizon to an H-step objective. The authors implement two model-based instantiations (SGFT-SAC and SGFT-TDMPC-2) and evaluate them on five real-world contact-rich manipulation tasks plus sim-to-sim benchmarks, reporting substantial sample-efficiency gains over standard fine-tuning baselines. They also provide a theoretical analysis (Theorem 1) intended to show that, under an 'improvability' condition on Vsim with respect to the real dynamics, the H-step SGFT objective yields near-optimal real-world policies with suboptimality O(gamma/(1-gamma) alpha H + gamma^H epsilon).","tokens_in":21640,"tokens_out":11427,"duration_ms":94909,"significance":"If the experimental results hold, this is a strong empirical contribution: it demonstrates that a simulation-trained value function can serve as an effective exploration prior for real-world fine-tuning, succeeding on tasks where zero-shot transfer and standard fine-tuning fail, and it provides a structural justification for horizon shortening in model-based sim-to-real transfer. The core idea is simple, broadly applicable, and the paper includes real-hardware experiments across multiple contact-rich tasks. The theoretical analysis, once corrected, would be a useful addition to the literature on terminal-value MPC and potential-based reward shaping. The paper does not provide machine-checked proofs, and the theory contains several formal inconsistencies, but these appear fixable.","major_comments":[{"comment":"The definition of Q*_H is inconsistent with the H-step Bellman recursion and with the proof of Lemma 4. Section 4.1 defines Q*_H(s, pi) := E_{a~pi}[gamma V*_H(s') + r(s)], but the one-step lookahead for the H-step objective must use V*_{H-1} and the shaped reward bar{r}(s,s') = r(s) + gamma Vsim(s') - Vsim(s); indeed, Eq. (13) in the proof of Lemma 4 uses Q*_H(s, pi) = E[gamma V*_{H-1}(s') + bar{r}(s,s')]. As printed, the Section 4.1 display would not select the optimal first action for the H-step objective, and it is inconsistent with the rest of the theory. Please correct the display and align all subsequent uses of Q*_H.","section":"Section 4.1"},{"comment":"Lemma 2's statement omits the model-error factor alpha H that its own proof derives. The proof combines inequalities (6) and (7), each carrying a factor gamma(...) alpha H, so the combination yields a bound proportional to gamma((1-gamma^{H-1})/(1-gamma) Delta r + gamma^H Delta V) alpha H, not the alpha-free expression in Eq. (5). Without alpha H, the model-free case alpha=0 would leave a spurious delta = gamma Delta V when Lemma 4 is invoked, contradicting the O(gamma^H epsilon) model-free bound claimed in Theorem 1. The intended statement should include alpha H, and Theorem 1's proof should be checked against the corrected lemma.","section":"Lemma 2, Eq. (5)"},{"comment":"The theoretical guarantee is conditional on Vsim being improvable with respect to Mreal (Definition 1), but the paper provides no empirical evidence that this per-state condition holds for the tasks evaluated. The pedagogical pendulum example has identical sets of feasible next states in simulation and reality, which is precisely the regime where the condition is easiest to satisfy; the real tasks are selected because contact dynamics are misspecified, so the margin in (2) could be negative at some states. Since Lemma 3, and hence Theorem 1, fails if the margin is negative anywhere, the paper should measure or bound the margin at least in the sim-to-sim benchmarks of Section 6.3, where the target dynamics are known exactly, and ideally in the real-world experiments, before claiming the theory underpins the contact-rich results.","section":"Section 5, Definition 1 / Theorem 1"},{"comment":"Lemma 3 is stated with a strict inequality, sup_a E_{s'~preal}[gamma Vsim(s')] - Vsim(s) > -r(s), while Definition 1 and Theorem 1 use a non-strict inequality, >= -r(s). If equality holds at some state, the lemma's hypothesis as stated is not met, so the proof of Theorem 1 has a gap. The proof only needs >=, so the statement should be corrected to match Definition 1.","section":"Lemma 3"}],"minor_comments":[{"comment":"The text says 'We use H = 1 in all our experiments' but later states that SGFT-TDMPC-2 uses H = 4; please clarify which instantiation uses which horizon.","section":"Section 6.1"},{"comment":"The acronym 'PRBS' in the phrase 'Potential-Based Reward Shaping (PRBS)' is a typo for PBRS.","section":"Section 4.1"},{"comment":"In Lemma 1, the definition of Delta V uses 'Vs(s)' where it should be 'Vsim(s)'; in Lemma 3, the expectation is written as E_{s~preal} where it should be E_{s'~preal}.","section":"Lemma 1 and Lemma 3"},{"comment":"The statement says 'Assumes that' and uses the norm symbol for what should be an absolute value |Vsim(s) - V*_real(s)|; the phrase 'for H sufficiently small' is also vague and should specify the intended regime.","section":"Theorem 1"},{"comment":"In the displayed chain of inequalities around Eq. (14), there is an extra '+' sign that appears to be a typo.","section":"Lemma 4 proof"},{"comment":"The sentence 'We continue training SAC with a fixed temperature of alpha = 0.01 and with a UTD of 2d' appears to have a typo in '2d'; please state the correct update-to-data ratio.","section":"Appendix C"},{"comment":"The real-world success-rate curves do not report the number of trials or error bars; please state whether each curve is a single run or an average, and how many independent runs were performed.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The experimental contribution is significant and the core idea is compelling, but the theory section as written contains several load-bearing formal errors (Q*_H definition, Lemma 2's missing alpha H, Lemma 3's strict-inequality assumption) that must be corrected before the theoretical claims can be accepted. The unverified 'improvability' assumption is also a substantive concern that should be addressed with empirical evidence in the sim-to-sim benchmarks at minimum. I recommend major revision rather than rejection because the errors appear fixable and the empirical results strongly support the central sample-efficiency claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical core is the real contribution. SGFT — using a simulation-trained value function as a potential-based reward shaper and shortening the fine-tuning horizon — shows big sample-efficiency gains on five real contact-rich tasks, and the idea is described cleanly enough that a decent RL group could reproduce it. I'd send it to a serious referee.\n\nWhat's genuinely new is the specific combination, not the pieces. Potential-based shaping and horizon shortening are both known; so are sim-to-real fine-tuning frameworks. Putting Vsim into the shaping potential and then optimizing a short-horizon objective in the real world is a sensible, useful twist, and the two instantiations (SAC and TD-MPC2) show it plugs into existing methods. The sim-to-sim ablations on H are a nice touch, and the limitation section is honest.\n\nWhere it's soft: the theory leans hard on Definition 1, the 'improvable Vsim' condition. That is a per-state condition on the real dynamics, and the paper's support is a pendulum example plus an assertion about continuous control geometry. But the five tasks were chosen precisely because contact dynamics are badly misspecified. Nothing in the paper measures the margin in Definition 1, even in the sim-to-sim benchmarks where the target dynamics are known exactly. If it fails anywhere, Lemma 3 and the main theorem give no bound. That is not fatal to the empirical method, but it makes the advertised theoretical guarantee shaky in the regime that matters.\n\nThere are also concrete proof issues. Lemma 2's printed bound in Eq. (5) is false as stated: take alpha=0, H=1, the left side is zero and the right side is positive. Lemma 4's proof uses gamma V*_{H-1} in Q*_H, whereas Section 4.1 defines Q*_H with gamma V*_H. These feel like fixable typos, but as printed the proof does not work.\n\nFinally, the real-world plots have no error bars, trial counts, or seeds. Given how noisy contact-rich manipulation is, I'd want at least a few trials per task before taking the 'up to an order of magnitude' claim at face value.\n\nNet: the method is likely useful and the empirical demonstration is worth taking seriously. The theory needs either repair or a much more careful statement of assumptions. I'd recommend major revision with a real referee, not a desk rejection.","headline":"SGFT's empirical results are strong and the method is a useful sim-to-real fine-tuning addition, but the theory's load-bearing 'improvable Vsim' assumption is unverified and the proof has fixable gaps.","tokens_in":22282,"tokens_out":3337,"would_cite":true,"duration_ms":30258,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a simulation-learned value function, when used to reshape real-world rewards and shorten the learning horizon, makes robot fine-tuning an order of magnitude more sample-efficient, with a provable near-optimality…","keywords":["sim-to-real transfer","fine-tuning","value function guidance","potential-based reward shaping","model-based reinforcement learning","dexterous manipulation","sample efficiency","policy adaptation"],"falsifier":"Collect real-world transitions $(s,a,s')$ and evaluate the expression $\\max_a [\\gamma V_{\\mathrm{sim}}(s') - V_{\\mathrm{sim}}(s)] + r(s)$ on visited states; if any state yields a negative value, Definition 1 is violated and the paper's bound no longer applies, predicting that SGFT will stagnate or regress at that state.","tokens_in":21129,"feed_emoji":"🤖","tokens_out":10194,"duration_ms":81762,"temperature":0.7,"pith_summary":"Robot learning from real interaction is data-hungry, and physics simulators are cheap but systematically wrong in force-sensitive tasks. This paper's proposal, Simulation-Guided Fine-tuning (SGFT), attacks the mismatch from a different angle: instead of transferring the policy or the dynamics, it transfers the value function $V_{\\mathrm{sim}}$ learned in simulation, using it to reshape the real-world reward and to shorten the fine-tuning horizon to a few steps. The claim is that real-world data then only has to cover short, local action sequences while the simulator supplies the long-horizon structure, yielding up to an order of magnitude fewer real-world samples than standard fine-tuning on five dexterous manipulation tasks. A theorem makes the claim conditional on one structural property — that $V_{\\mathrm{sim}}$ is 'improvable' under real dynamics — under which the short-horizon SGFT objective provably converges to near-optimal real policies.","feed_headline":"Sim-trained value functions make real-robot fine-tuning 10x faster","feed_subtitle":"Five force-sensitive manipulation tasks solved in under two hours, where zero-shot sim-to-real transfer fails.","key_machinery":"The engine of SGFT is the potential-based reward shaping transformation $\\bar{r}(s,s') = r(s) + \\gamma V_{\\mathrm{sim}}(s') - V_{\\mathrm{sim}}(s)$, with $V_{\\mathrm{sim}}$ as the potential, combined with an $H$-step finite-horizon objective. The telescoping identity (Equation 1) shows that optimizing the shaped $H$-step return is equivalent, up to a policy-independent constant, to maximizing $\\gamma^H V_{\\mathrm{sim}}(s_H) + \\sum_{t=0}^{H-1} \\gamma^t r(s_t)$ — i.e., the simulator bootstraps the long-horizon tail while real data only needs to support $H$ steps of local search. The proof machinery includes a monotonicity property of the optimal $H$-step value under the improvable condition (Lemma 3) and a decomposition of suboptimality (Lemma 4), which together remove the $\\frac{1}{1-\\gamma}$ factor on the value-gap error that appears in prior MPC-style bounds. In implementations, Dyna-SGFT uses a learned model to hallucinate $H$-step branches from real states, and MPC-SGFT uses TDMPC-2 with the frozen $V_{\\mathrm{sim}}$ as terminal cost.","core_discovery":"The central claim, stated in the paper's own terms, is that optimizing a short-horizon objective with a simulation-learned value function as the terminal 'surplus' recovers near-optimal real-world behavior even when the simulator's dynamics are wrong in detail. The value function $V_{\\mathrm{sim}}$ defines an ordering over states that captures task-relevant structure — reaching, grasping, placing — and this ordering is assumed to survive the sim-to-real gap. The paper introduces the notion of improvability (Definition 1): for every state there exists an action satisfying $E_{s'\\sim p_{\\mathrm{real}}}[\\gamma V_{\\mathrm{sim}}(s')] - V_{\\mathrm{sim}}(s) \\ge -r(s)$. Under this condition, Theorem 1 bounds the suboptimality of the learned policy by $O\\!\\left(\\frac{\\gamma}{1-\\gamma}\\alpha H + \\gamma^H \\epsilon\\right)$, where $\\alpha$ is model error and $\\epsilon$ is the value gap — the crucial point being that the bound does not grow like $\\frac{1}{1-\\gamma}$ on $\\epsilon$, so small $H$ remains safe.","pith_inferences":["We infer that the same short-horizon, value-guided objective could work with other off-domain data sources (e.g., human demonstrations or offline datasets), because the load-bearing assumption is about the value function's ordering over states, not which domain produced the value.","We infer that improvableness can be monitored online: by estimating the quantity $\\max_a [\\gamma V_{\\mathrm{sim}}(s') - V_{\\mathrm{sim}}(s)] + r(s)$ from real transitions, a system could detect when the assumption fails and adaptively lengthen $H$ before relying on the guarantee.","We infer that $H$ is a bias-variance dial, and that an automatic rule for choosing $H$ from estimated model error would be a natural next step; the paper's peg-in-hole ablation shows tasks with higher precision needs benefit from larger $H$."],"forward_implications":["When the improvable condition holds, real-world fine-tuning reduces to a short-horizon (even $H=1$) policy search, so sample complexity depends on local dynamics rather than full task length.","Model-based RL becomes practical in the real world: learned models only need to be accurate for $H$ steps before the value function takes over, sidestepping compounding model error.","SGFT is a wrapper on top of standard RL algorithms; applying it to SAC and TDMPC-2 improves both, and prevents the catastrophic forgetting seen in standard fine-tuning.","The theory justifies using short horizons with very few real-world rollouts, making it possible to adapt policies in under an hour on contact-rich tasks like hammering and insertion."],"supporting_citations":[{"why":"Supplies the Potential-Based Reward Shaping formalism that guarantees policy invariance for infinite-horizon objectives, which SGFT repurposes with a finite horizon.","marker":"(Ng et al., 1999)"},{"why":"Provides the comparison MPC suboptimality bound and the model-return error lemma that the paper extends under the improvable condition.","marker":"(Bhardwaj et al., 2020)"},{"why":"Supplies TDMPC-2, the model-based backbone for the MPC-SGFT instantiation.","marker":"Hansen et al. (2024)"},{"why":"Provides the Dyna-style model-based policy optimization framework that Dyna-SGFT builds on.","marker":"(Janner et al., 2019)"},{"why":"Provides SAC, the base off-policy algorithm for the SGFT-SAC instantiation.","marker":"(Haarnoja et al., 2018)"},{"why":"The heuristic-guided RL result that the paper's improvable-condition argument is modeled on.","marker":"Cheng et al. (2021)"}],"fun_headline_variants":["Simulation value functions steer real-robot fine-tuning to 10x fewer samples","Sim-guided value functions cut real-robot fine-tuning data 10x","Simulation-learned value functions make real fine-tuning 10x more sample-efficient","Sim value funcs guide robot fine-tuning to 10x fewer real samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire guarantee sits on the assumption that the simulation-trained value function remains 'improvable' in the real world: at every state the robot can reach, some action must move toward states the simulator values highly without losing more than the immediate reward.","fun_headline_variants_meta":{"raw":{"variants":["Simulation value functions steer real-robot fine-tuning to 10x fewer samples","Sim-guided value functions cut real-robot fine-tuning data 10x","Simulation-learned value functions make real fine-tuning 10x more sample-efficient","Sim value funcs guide robot fine-tuning to 10x fewer real samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000899,"raw_usage":{"total_tokens":3931,"prompt_tokens":1061,"completion_tokens":2870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":2786}},"tokens_in":677,"tokens_out":2870,"duration_ms":18086,"temperature":1.0,"reasoning_tokens":2786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:23:25.355593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real-world transitions $(s,a,s')$ and evaluate the expression $\\max_a [\\gamma V_{\\mathrm{sim}}(s') - V_{\\mathrm{sim}}(s)] + r(s)$ on visited states; if any state yields a negative value, Definition 1 is violated and the paper's bound no longer applies, predicting that SGFT will stagnate or regress at that state.","supporting_citations":[{"cited_title":"Further suppose ∆r = max s r(s) − mins r(s) and ∆V = max s Vs(s) − mins Vs(s) are finite","cited_arxiv_id":null,"evidence_quote":"Provides the comparison MPC suboptimality bound and the model-return error lemma that the paper extends under the improvable condition."}],"review_version":1}