{"id":"09e58c05-e283-4dac-b8df-b37c4b4a693d","arxiv_id":"2602.08557","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Guiding goal-conditioned reinforcement learning with samples from a constrained feasible-state manifold lets a simulated double-sphere and a Panda-arm policy succeed far more often than RL with random resets.","lead":"This paper trains robot policies for complex contact-rich object manipulation by generating physics-valid start and goal states with a constraint solver and using them to guide reinforcement learning. The idea is that where an episode starts matters more than copying the optimizer's actions, and it beats a no-guidance baseline in two simulated domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'universal' success rates are evaluated on the same solver-generated D_s used for training resets, with no independent coverage check; D_s bias from Eq. (10) may make 83.1%/5.1% estimates of a sampler-specific distribution, not uniform over Sc.","rationale":"The reader's conditional verdict already identifies sample representativeness; I agree and sharpen it. The load-bearing premise is not merely that D_s covers Sc - it is that the evaluation distribution in Fig. 6 matches the formal p0 objective. Both fail together: the sampler has no coverage guarantee, and the finite D_s doubles as training and test distribution. This is an internal gap, not a disagreement with any external consensus: Eq. (2) promises U(Sc x Sc), while Sec. V and Fig. 6 use U(D_s x D_s). The baseline comparison (near-zero success without guidance) is convincing evidence of the method's practical value, and the paper's disclosure of feasibility rates and post-hoc goal exclusion is honest. But the strongest wording ('universal', 'any statically stable state', 'with (s,g) ~ p0') overreaches relative to the evidence. A single independent held-out test set generated by a different sampling mechanism would settle the matter; it is within reach for the simple double-sphere domain and feasible for the panda domain with a second solver. If the independent evaluation reproduces the reported rates, the conditional verdict should stand; if not, the paper should be revised to claim success on a solver-reachable subset. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":15172,"tokens_out":6804,"duration_ms":81097,"concrete_test":"Generate an independent test set D_test for the double-sphere domain without using the Eq. (10) sampler: e.g., enumerate/rejection-sample statically stable configurations from a fine grid over object pose and robot position using the analytic constraints (5)-(9), including all contact modes and balance goals. Randomly draw 1000 (s,g) pairs from D_test (ensuring D_test is disjoint from D_s by nearest-neighbor distance) and evaluate the final interpSched and trajSched policies from the 5 seeds used in Fig. 6. If success drops materially below 83.1% (or below 5.1% for a similarly constructed panda D_test), the reported 'uni' numbers are artifacts of sampler/training-set overlap; if success is comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Sample-Guided RL trains a universal goal-conditioned policy achieving high success under objective (3), i.e., with (s,g) drawn from p0 = U(Sc x Sc) (Eq. 2). The weakest link is the operationalization of p0. D_s (Eq. 11) is built by Augmented Lagrangian projection (Eq. 10) from box-uniform restarts and randomly chosen contact modes, retaining only feasible runs. Fig. 3(a) reports per-sample feasibility of only 58.7% (double sphere) and 17.9% (panda), and the paper gives no coverage, completeness, or ergodicity result for the retained set. Contact modes are restricted to 1-3 uniformly chosen support contacts, and rejected or locally stuck runs are discarded, so D_s may concentrate on solver-friendly regions of Sc. More concretely, the same D_s is used both to define the reset distribution during training (options (16)-(21); trajectory starts/goals are also sampled from D_s, Sec. IV-B) and to evaluate the final policies (Fig. 6): Sec. V-A fixes S = 10,000 state samples, and the 'uni' column samples start/goal pairs from this set. Thus the reported 83.1% and 5.1% are in-distribution, non-held-out success rates on a finite solver-generated set, not estimates of U(Sc x Sc). The post-hoc removal of balance goals (raising 83.1% to 99.0%) further narrows the evaluated distribution. The comparison against the baseline is still informative, but the 'universal' headline is not supported without an independent test set or coverage analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sample-Guided RL, a framework for goal-conditioned manipulation policies in which a model-based constrained sampler provides the start/goal distribution and guides RL through reset-state initialization. The authors define a Constrained Goal-conditioned MDP (CG-MDP) with p0 uniform over a feasible state set Sc, generate a constrained state dataset D_s via Augmented Lagrangian projection (Eq. 10), generate an open-loop trajectory dataset D_u via CMA-ES (Eq. 12), and then use options (16)-(21) to sample start/goal pairs during TD7 training, optionally adding behavior cloning (Eq. 14). Experiments on a double-sphere domain and a Panda-arm sphere domain report success rates of 83.1% (99.0% after excluding balance goals) on the former and 5.1% vs. a 0.1% baseline on the latter, with 61.2% on the trajectory-state distribution.","tokens_in":15695,"tokens_out":4068,"duration_ms":49063,"significance":"If the results hold, the paper makes a useful contribution to contact-rich manipulation RL: it gives a crisp formulation of how model-based constraint information can be injected into RL through reset-state distributions, without relying on expert demonstrations. The comparison of six guidance variants is systematic, and the reporting of per-sample feasibility, compute time, and variance over five runs is a strength. The main weakness is that the central 'universal policy' claim is evaluated on the same solver-generated D_s used for training, with no independent coverage analysis, so the reported numbers are best understood as in-distribution performance on a finite, possibly biased sample rather than estimates with respect to the stated p0 = U(Sc x Sc).","major_comments":[{"comment":"The sampler used to realize p0 is the Augmented Lagrangian projection from box-uniform restarts with randomly sampled contact modes, but the paper provides no coverage, completeness, or ergodicity guarantee. With per-sample feasibility rates of only 58.7% (double sphere) and 17.9% (panda), the retained D_s may be concentrated on solver-friendly regions of Sc. Since the same D_s is used both for training resets and for the final 'uni' evaluation in Fig. 6, the headline success rates (83.1% and 5.1%) are in-distribution estimates on a finite solver-generated set, not estimates over U(Sc x Sc). I request an independent coverage check or a held-out test set generated by a different mechanism than Eq. (10), e.g., a different initializer or contact-mode sampler, with a comparison of achieved success rates on that set.","section":"§V-B, Fig. 3(a), Eq. (10)"},{"comment":"The double-sphere 'w/o balance' column removes 'sphere-on-sphere balance goals' post hoc. These goals are part of Sc as defined in Eq. (1), and the objective (3) is over all of Sc x Sc. If such goals are unreachable or uncontrollable under the MDP, that should be established a priori and incorporated into the problem definition; otherwise the 99.0% figure is an evaluation on a modified distribution and cannot be presented as success on the stated objective. The paper should either redefine Sc to exclude such states with a principled reachability/controllability argument or report the 83.1% number as the primary result with the exclusion clearly labeled as a limitation.","section":"§V-D(iv), Fig. 6"},{"comment":"The 61.2% panda success rate is evaluated on start/goal pairs taken from trajectory states (consistent with the traj distribution), not from U(Sc x Sc). This number is therefore not evidence for a universal policy; it is an in-distribution measure on the same trajectory-state distribution used for training. The 'uni' rate of 5.1% is the only number directly relevant to the stated objective, and while it is much better than the 0.1% baseline, the absolute performance is low. The paper should clearly separate the trajectory-state results from universal results and, ideally, add an evaluation on a broader set of start/goal pairs that is independent of D_u.","section":"§V-D(iv), Fig. 6, panda row"}],"minor_comments":[{"comment":"The caption contains a typo ('panda shpere') and 'w/o balance' is used without defining the balance-goal criterion precisely in the figure caption.","section":"Fig. 6"},{"comment":"The constraint notation is inconsistent: h_c(x)=0 appears in Eq. (10) while the surrounding text uses h_c(s)=0. Please fix.","section":"Eq. (10)"},{"comment":"The symbol D is reused for both the BC batch in Eq. (14) and the datasets D_s/D_u. This is confusing; use a different symbol for the batch.","section":"§IV-C, Eq. (14)"},{"comment":"The abstract describes model-based constrained state sampling as 'highly efficient,' but Fig. 3(a) reports per-sample feasibility rates of 58.7%/17.9% and about one hour to generate 10,000 Panda samples. Consider softening the wording or framing efficiency relative to alternative resampling strategies.","section":"Abstract and §V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and the reset-guidance mechanism is worth publishing if the evaluation can be made to match the stated objective. The main blocker is the absence of an independent test for the universal-policy claim; this is fixable with additional experiments and a more careful interpretation of the existing numbers, so I recommend major revision rather than rejection. The post-hoc exclusion of balance goals should be treated as a limitation, not folded into the headline success rate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the conceptual move is real and the experiments are mostly honest, but the \"universal\" claim is tied to a solver-generated dataset that is both the training reset distribution and the evaluation distribution. That does not kill the technique, but the headline numbers are about a sampler-specific distribution, not the uniform p0 of Eq. (2).\n\nWhat is new: the CG-MDP formulation, using a constraint manifold to define the start/goal distribution; the practical scheme to sample feasible states with Augmented Lagrangian and project random interpolations onto the nearest feasible state; and the finding that state-initialization bias matters more than behavior cloning. That finding is well supported. The double-sphere domain is well designed and the results clearly beat the direct baseline. The Panda absolute numbers are low, but the contrast with a near-zero baseline is meaningful.\n\nThe soft spots are mostly about how evaluation relates to the stated objective. D_s is built from 10,000 feasible states, but with per-run feasibility of 58.7% (double sphere) and 17.9% (panda), no coverage or completeness check, random contact modes limited to 1-3, and infeasible runs discarded, D_s is best described as \"what the solver happened to find\" rather than a uniform sample of S_c. Figure 6's \"uni\" column samples the same D_s, so the 83.1% and 5.1% are in-distribution success rates on a finite, solver-biased set. The post-hoc \"w/o balance\" column is disclosed, but post-hoc exclusions need an a priori justification or a separate analysis. The \"traj\" column is explicitly on trajectory states, so it should not be part of the universal claim.\n\nWhat I would want before accepting: released code and data, a held-out independent test set or at least a coverage analysis of D_s, a comparison against one established goal-conditioned reset/curriculum method, and a pre-registered definition of which goal states are included. The citation pattern looks fair. The paper already admits several of these limitations, which is to its credit.\n\nThis is a paper worth engaging with. It deserves a proper peer review, but the revision needs to reframe the claims as \"solver-guided RL works on the solver's reachable distribution\" and add the missing checks.","headline":"The conceptual move is real and the experiments are mostly honest, but the 'universal' claim is tied to a solver-generated dataset that is both the training reset distribution and the evaluation distribution; the technique survives, but the headline numbers need to be reframed.","tokens_in":16086,"tokens_out":3165,"would_cite":true,"duration_ms":38416,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that constrained model-based state sampling, not behavior cloning, is the key to training universal contact-rich manipulation policies.","keywords":["goal-conditioned RL","constrained sampling","contact-rich manipulation","universal manipulation policy","state visitation bias","behavior cloning","augmented Lagrangian","non-prehensile manipulation"],"falsifier":"Retrain using a provably uniform sampler over S_c (e.g., rejection sampling or hit-and-run) with sufficient compute, then evaluate on the same start/goal distribution; if success rates drop to baseline levels, the reported gains are an artifact of solver bias. Alternatively, compute the success rate on states that are feasible but never visited by the solver, and check whether it falls sharply.","tokens_in":1389,"feed_emoji":"🤖","tokens_out":1674,"duration_ms":54107,"temperature":0.7,"pith_summary":"This paper tries to establish that injecting first-principles physics constraints into reinforcement learning, by shaping the start and goal state distribution, is what makes universal contact-rich manipulation policies learnable, not the usual behavior-cloning imitation of demonstrated actions. To do this, it formalizes a Constrained Goal-conditioned MDP in which feasible start and goal states are defined by differentiable collision, contact, and force equilibrium constraints, and sampled with a nonlinear constrained solver. The paper reports that guiding RL with these sampled states, especially through projected interpolation and a schedule that starts episodes near goals, achieves an 83.1% success rate on a double-sphere manipulation task and 5.1% (versus a 0.1% baseline) on a harder Panda-arm whole-body-contact task. A sympathetic reader would take the central insight to be that demonstrating relevant intermediate states matters more than demonstrating actions.","feed_headline":"Constraint-guided RL hits 83% success in contact-rich manipulation","feed_subtitle":"Model-based state resets, not action demonstrations, drive learning; the method beats a near-zero baseline on a harder Panda-arm task.","key_machinery":"The central object is the Constrained Goal-conditioned MDP (CG-MDP), which augments a goal-conditioned MDP with differentiable constraints g_c(s)≤0, h_c(s)=0 that define the set S_c of feasible states (collision-free, contact-consistent, static-equilibrium). The machinery that carries the argument is a proximal Augmented-Lagrangian sampler (Eq. 10) that projects uniform box samples onto S_c, plus the projected-interpolation sampler (Eq. 19) that maps random convex interpolations between two constrained states to the nearest feasible state in a feature embedding, and an optional behavior-cloning regularization (Eq. 14). These components together shape the state-visitation distribution of the","core_discovery":"The paper introduces Sample-Guided RL, which uses a model-based solver to sample configurations that satisfy collision, contact, and static-equilibrium constraints, then uses those samples to bias the initial state distribution of a goal-conditioned reinforcement learning agent. The central discovery is empirical: biasing state visitation with constraint samples — via trajectory-derived starts or via projected interpolation that snaps random convex blends of feasible states to the nearest feasible state in feature space — is what drives learning, while adding a behavior cloning loss has only minor impact. On the double-sphere domain, the best method reaches 83.1% success on random feasible s","pith_inferences":["The scheduled interpolation method can be viewed as an automatic curriculum that trains on easy near-goal starts first; this predicts that any goal-conditioned RL algorithm would benefit from the same schedule regardless of the base learner.","The near-perfect success after removing sphere-on-sphere balancing goals suggests the limiting factor is controller bandwidth for unstable equilibria, not state-space discovery; a higher-frequency control loop or a dedicated balancing sub-policy could close this gap.","The approach transfers to any black-box dynamical system for which a differentiable constraint manifold can be defined, such as in-hand manipulation, object pivoting, or legged locomotion — provided a sampler with better coverage than the reported 58.7% and 17.9% per-sample feasibility rates.","The paper's comparison of trajectory-based and interpolation-based guidance mirrors a bias-variance tradeoff; one testable consequence is that interpolation-based guidance should outperform trajectory-based guidance on simpler manifolds and underperform on manifolds where solver reachability is highly non-uniform."],"forward_implications":["State-visitation bias from model-based constraint samples can replace expert demonstrations as a practical way to guide goal-conditioned RL in contact-rich manipulation.","Behavior-cloning regularization adds little beyond the state bias, suggesting that costly trajectory optimization may be unnecessary in simpler domains.","The CG-MDP formulation provides a principled method to inject physics first principles — collision, contact, friction, static equilibrium — into an otherwise black-box RL problem.","The approach scales to whole-body contact manipulation, achieving 5.1% success versus a 0.1% baseline, indicating applicability to non-prehensile and dynamic tasks.","Scheduling starts near the goal and gradually moving them away acts as an effective curriculum, improving final policy performance and reducing training variance."],"fun_headline_variants":["Constraint-sampled resets boost RL to 83% on contact-rich tasks","Sample-Guided RL: 83% success via constraint-based resets","Model-based state resets drive 83% RL success on Panda arm","Constraint sampling resets beat baselines in contact-rich RL","Panda arm: constrained resets lift RL from near-zero to 83%"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The load-bearing premise is that the solver's feasible samples (58.7% and 17.9% per-sample success in the two domains) are representative of the full constrained manifold; without a coverage or ergodicity guarantee, the measured success rates may apply only to the solver-reachable subset.","fun_headline_variants_meta":{"raw":{"variants":["Constraint-sampled resets boost RL to 83% on contact-rich tasks","Sample-Guided RL: 83% success via constraint-based resets","Model-based state resets drive 83% RL success on Panda arm","Constraint sampling resets beat baselines in contact-rich RL","Panda arm: constrained resets lift RL from near-zero to 83%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3354,"prompt_tokens":718,"completion_tokens":2636,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":2549}},"tokens_in":462,"tokens_out":2636,"duration_ms":21115,"temperature":1.0,"reasoning_tokens":2549,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:13:23.693214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain using a provably uniform sampler over S_c (e.g., rejection sampling or hit-and-run) with sufficient compute, then evaluate on the same start/goal distribution; if success rates drop to baseline levels, the reported gains are an artifact of solver bias. Alternatively, compute the success rate on states that are feasible but never visited by the solver, and check whether it falls sharply.","supporting_citations":[],"review_version":1}