{"id":"4b84aa8f-3511-40b2-82af-fcc2942ac292","arxiv_id":"2505.08949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A video-guided multi-tree RRT planner that uses contact events and object poses from a human demonstration solves multi-step manipulation planning tasks that standard TAMP solvers fail on.","lead":"This paper shows a robot can use a video of a person moving objects as a guide for planning its own multi-step manipulations, like moving boxes onto a shelf or through a tunnel. The video supplies the key moments and object positions, which a search-based planner turns into collision-free robot motions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generalization evaluation silently discards scenarios where the demonstrated grasp sequence is infeasible, so the reported success rates are conditional and may overstate the method's ability to go beyond the demonstrated scene.","rationale":"The reader's weakest assumption is that the video supplies a complete and robot-feasible task-level plan. My stress-test identifies a concrete experimental consequence of that assumption: in Sec IV-C, the evaluation protocol resamples any scene where the demonstrated grasp sequence is not feasible, and the number of such resamples is not reported. This makes the generalization success rates conditional on exactly the assumption the reader flagged, and it is an addressable, quantifiable issue. The benchmark comparison in Fig 6 is not directly affected by this filter, so the core 'outperforms TAMP solvers' claim may survive. However, the generalization results in Fig 7 are a stated contribution, and without rejection counts the reported numbers cannot be interpreted. The numeric inconsistency in Table I (33% claimed versus 21% actual reduction) is a real but secondary reporting error. Because the concerns are specific and addressable, the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":14817,"tokens_out":6913,"duration_ms":72024,"concrete_test":"Instrument the generalization pipeline to log, for each scenario in Sec IV-C (start pose, goal pose, object, environment, combined), the total number of sampled parameter sets and the number rejected by the 'graspable by the same handle' condition. Recompute all metrics treating rejected samples as failures, or report rejection rates alongside the success rates. If the unfiltered success rates drop by more than a few percentage points or the rejection rate is substantial (e.g., >20%), the generalization claim must be weakened. Also specify which robot was used for the Fig 7 results, as the section currently does not state it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes generalization to novel poses, objects, and environments. In Sec IV-C, the evaluation protocol states: 'The parameters were resampled if all objects were not graspable by the same handle at the consecutive placement states.' This explicitly removes from the reported averages any scenario in which the video-demonstrated grasp sequence is not realizable. The number of discarded samples is never reported, and this filtering is not listed in the Limitations (Sec V). Consequently, the success rates in Fig 7 are conditional on the demonstration remaining robot-feasible, which is precisely the load-bearing assumption the reader identified. If many samples are rejected, the claim that 'the guidance from one video demonstration allows us to solve the given tasks for novel scenes' is stronger than the data support. This is not an inherent flaw of the planner but an experimental protocol that hides the impact of the assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a video-guided task-and-motion planning (TAMP) method. From an instructional video of a human manipulating known objects, the method extracts hand-object contact states and 6D object poses at contact changes (Sec. III-A). These poses define placement states in an admissible configuration space, and grasp states are connected to neighboring placement states through transitions (Sec. III-B). The planner grows multiple RRT trees rooted at these transitions, linking trees until the start and goal configurations are connected (Sec. III-C). The resulting path is simplified with random shortcuts and then refined by solving an optimal control problem to obtain smoother trajectories (Secs. III-E, III-F). The authors introduce a benchmark with three tasks (shelf rearrangement, tunnel transfer, waiter tray transfer) and compare against HPP RRT-connect and PDDLStream on the Franka Panda, UR5, and KMR iiwa robots (Sec. IV). They further evaluate generalization to new start/goal poses, objects, environments, and combined variations, and demonstrate real-robot execution (Sec. IV-C).","tokens_in":14932,"tokens_out":4748,"duration_ms":50356,"significance":"If the reported results hold, the paper makes a useful contribution by showing that a classical sampling-based planner can exploit a single video demonstration to solve multi-step manipulation tasks with sequential dependencies, without learning a policy. The method is geometrically transparent, re-plans when the scene changes, and is evaluated on a new benchmark with several robots. The paper also ships a project page with code and video, which supports reproducibility. The strongest point is that the planner must still discover collision-free configurations, select grasp handles, and connect the demonstrated states, so the output trajectories are not trivially determined by the input poses. The main weakness is that the generalization evaluation filters out cases where the video-demonstrated grasp sequence is infeasible, which narrows the scope of the generalization claim.","major_comments":[{"comment":"The generalization results are conditional on a resampling filter that is not accounted for in the reported success rates. The text states: 'The parameters were resampled if all objects were not graspable by the same handle at the consecutive placement states.' This removes exactly the scenarios in which the video-demonstrated placement sequence is not robot-feasible in the novel scene. The number or fraction of discarded samples is never reported, and the Limitations section (Sec. V) does not mention this filter. As a result, the claim that 'guidance from one video demonstration allows us to solve the given tasks for novel scenes' is stronger than the data support. Please report the rejection rate for each generalization scenario and, ideally, evaluate or discuss what happens when the demonstrated grasp sequence is infeasible, since this is a central limitation of the approach.","section":"Sec. IV-C, Fig. 7"},{"comment":"The success rates and average metrics are reported without any measure of variance, confidence interval, or number of successful runs underlying each mean. Since the planner is stochastic (random tree sampling, random configurations, random robot base poses) and some differences between methods are small, the reader cannot assess whether the reported improvements are statistically meaningful. Please report error bars, confidence intervals, or at least the per-run success counts for the 10 and 100 runs.","section":"Sec. IV-B, Sec. IV-C, Fig. 6, Fig. 7"}],"minor_comments":[{"comment":"The section heading 'Path optimization sing random shortcut' contains a typo; it should read 'Path optimization using random shortcut'.","section":"Sec. III-E"},{"comment":"The text states that trajectory refinement decreases path length by 33% on average, but the table shows a decrease from 32.6 rad to 25.7 rad, which is about 21%. Please correct the percentage or clarify the calculation.","section":"Table I and Sec. IV-C"},{"comment":"The baseline planner is referred to as both 'PDDLStreams' and 'PDDLStream'; please use one consistent name throughout.","section":"Sec. II and Sec. IV-B"},{"comment":"The names 'KUKA IIW A arm' and 'KMR IIW A robot' appear to contain typos; they should likely be 'KUKA iiwa' and 'KMR iiwa'.","section":"Sec. IV-B"},{"comment":"In the OCP formulation, the objective includes a sum over keyframe indices i of w_d d(q_i, F_i), but the optimization variables are the control torques u_t and the state x_t = (q_t, v_t). Please clarify how q_i is obtained from x_t for a keyframe index i and how the keyframe set K is defined in practice.","section":"Sec. III-F, Eq. (1)"},{"comment":"The OCP weights (w_d, w_x, w_u, w_c, w_b) are not specified. Please state whether these were fixed across all tasks or tuned per task, since this affects reproducibility.","section":"Sec. III-F"}],"recommendation":"major_revision","confidential_remarks":"The main load-bearing issue is the unreported resampling filter in the generalization evaluation; this should be fixed before publication. The comparison against baselines that do not receive the demonstration is fair for assessing the value of video guidance, but the abstract's wording 'outperforms state-of-the-art TAMP solvers' should be qualified as applying to the benchmark tasks and to planners without demonstration input. The paper is within the journal's scope and the algorithmic core is sound; the issues are fixable with additional reporting and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this arXiv paper is an extended version of the authors' ICRA 2023 paper, and the genuinely new parts are a generalization study over poses, objects, and environments, plus an OCP-based trajectory refinement. The core idea remains video-guided multi-tree RRT: extract contact states and object 6D poses from an instructional video, build an admissible configuration space from those placement/grasp states, and grow multiple trees to connect them. On the proposed three-task benchmark (shelf, tunnel, waiter) they show clear success-rate gains over HPP RRT-connect and PDDLStreams, especially on the tunnel task with its narrow passage. The real-robot execution with CosyPose-estimated start poses is a nice empirical step. That part is solid.\n\nThe soft spots are real and worth attention. First, the generalization protocol in Sec. IV-C resamples parameters if 'all objects were not graspable by the same handle at the consecutive placement states.' That filters out the exact cases where the demonstrated grasp sequence no longer works for the robot, which is precisely the load-bearing assumption for generalization. The rejection rate is never reported, so the success rates in Fig. 7 are conditional on the demonstration remaining robot-feasible. That overstates the claim that one video suffices for novel scenes. This should be quantified in revision.\n\nSecond, the reported metrics have no error bars or variance, despite up to 100 runs, making it hard to judge whether differences are meaningful. Third, there's a numeric inconsistency: the text claims a 33% average path-length reduction from trajectory refinement, but Table I shows 32.6 rad to 25.7 rad, about 21%. Minor but sloppy. Finally, the planner has no mechanism to recover if the video's intermediate placement is infeasible or occluded; the limitations section mentions this in passing but doesn't analyze how often it happens.\n\nThe central claim—that video-derived contact states and poses can guide a multi-tree RRT to solve sequential-dependency TAMP tasks—does hold up for the demonstrated scenes. The generalization claim is more conditional than the abstract suggests. Still, the paper is worth a serious referee; the issues are specific and addressable. I'd send it to peer review and ask for a revision that reports rejection rates and variance.\n\nWho's this for: TAMP and learning-from-demonstration people, especially those working on using video as a planning prior. The benchmark itself could be useful.","headline":"Video-guided multi-tree RRT works for the demonstrated scenes and the new generalization study is useful, but the success rates are conditioned on a resampling filter that hides how often the demonstration becomes infeasible.","tokens_in":15547,"tokens_out":2086,"would_cite":true,"duration_ms":19299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video guidance lets a multi-tree planner solve multi-step pick-and-place tasks with sequential dependencies, validated on three new benchmarks and a real robot.","keywords":["task and motion planning","video demonstration","multi-step manipulation","RRT planning","contact states","6D object pose estimation","trajectory optimization","manipulation benchmark"],"falsifier":"Take the tunnel task and shift the demonstrated intermediate placement pose just outside the robot's reachable workspace while a collision-free alternative placement exists elsewhere inside the tunnel; if the planner then fails or times out instead of finding the alternative, the claim that the video-derived placement states are sufficient guidance would be falsified.","tokens_in":14570,"feed_emoji":"🤖","tokens_out":10138,"duration_ms":90951,"temperature":0.7,"pith_summary":"The paper claims that a human instructional video can supply enough structure to make multi-step task-and-motion planning tractable: if the planner is told when objects are grasped and released and where objects are placed at those moments, it can solve tasks with sequential dependencies that baseline TAMP planners miss. The video is processed into a sequence of contact-state changes and a 6D pose for every object at each change; these define the small set of placement states the planner is allowed to use. The planner then grows multiple rapidly-exploring random trees rooted at transitions between neighboring placement and grasp states and links them into a start-to-goal path. The approach is tested on three new benchmark tasks (shelf rearrangement, transferring an object through a narrow tunnel, and tray-based waiter transfer) across several robots, and the planned paths are refined by an optimal-control step for execution on a real arm. If correct, it implies that one video demonstration can replace hand-coded task scripts for a class of sequential pick-and-place tasks.","feed_headline":"A guiding video solves multi-step robot pick-and-place tasks","feed_subtitle":"One human demo provides the grasp and release states a tree-based planner needs for sequential tasks.","key_machinery":"The machinery is the admissible configuration space—the set of configurations in which every object is either stably resting on a contact surface or rigidly held by the gripper—restricted to the placement states read off the demonstration video. Each contact-state change in the video fixes the object poses for one placement state, and the planner connects consecutive placement states through grasp states, seeding a new RRT-like tree at each sampled transition. The video thus provides the discrete task skeleton; the trees fill in collision-free robot motion without having to discover the contact sequence.","core_discovery":"On the paper's own terms, the central finding is that the hard combinatorial part of manipulation TAMP—discovering which pick-and-place sequence and which intermediate placements connect the start to the goal—does not need to be searched from scratch when a guiding video supplies the sequence. Contact-state recognizers and 6D pose estimators give the planner the times of grasp/release changes and the object poses at those times, which are turned directly into placement states. A multi-tree RRT that seeds trees at the transitions between these states then connects start and goal reliably. The paper reports that on the proposed benchmark this approach solves multi-object shelf tasks and the narrow-passage tunnel task within the time limit, where the comparison planners fail, and that the same demonstration remains usable when start poses, goal poses, object shapes, or furniture positions are varied.","pith_inferences":["Beyond the paper: the demonstration is effectively a task skeleton, so the same multi-tree planner could accept the skeleton from any source—language instructions, a programmatic goal specification, or a symbolic planner—as long as grasp/release states and intermediate object poses are provided.","Because the planner never searches over alternative contact sequences, solution quality is inherited from the demonstration; comparing plans generated from two different videos of the same task would reveal how much of the reported path length and grasp count comes from the video itself rather than from the planner.","Beyond the paper: the generalization experiments vary furniture translations but keep rotations fixed, so a natural stress test would rotate the shelf or table to see whether the frame-relative pose representation keeps the same demonstration usable."],"forward_implications":["Tasks that can be written as a sequence of pick-and-place actions on known objects become tractable for sampling-based planning from a single demonstration, including cases where an object must be parked at an intermediate location to be grasped later.","On the proposed benchmark, the video-guided planner solves the tunnel task for non-mobile robots and shelf tasks with up to three objects within the time limit, while the comparison planners fail on those cases; the waiter task is solved more reliably, though more slowly, than the symbolic TAMP baseline.","One demonstration can be reused after changing start object poses, goal object poses, object geometries, or the placement of furniture, because demonstration poses are stored relative to the contact-surface frames.","Trajectory refinement formulated as an optimal control problem shortens the final path (a 33% average reduction in the reported two-object shelf experiment) at the cost of about 60 seconds of extra computation, enabling smoother real-robot execution."],"supporting_citations":[{"why":"Defines the admissible configuration space as a set of placement/grasp states with transitions, the representation the paper builds on.","marker":"[5]"},{"why":"Supplies the bidirectional tree-growing RRT method used as a baseline and as the starting point for the multi-tree extension.","marker":"[3]"},{"why":"The symbolic TAMP baseline whose success rate and planning time are compared against in the benchmark.","marker":"[12]"},{"why":"The earlier conference version of this work, extended here with generalization experiments and trajectory refinement.","marker":"[6]"},{"why":"The hand-contact recognizer that yields the grasp/release contact states from the input video.","marker":"[19]"},{"why":"The 6D pose estimator that provides object poses at every contact-state change.","marker":"[20]"},{"why":"The numerical solver used to project random samples onto the constraints of a placement or grasp state.","marker":"[26]"},{"why":"The random-shortcut method reused for path optimization before the optimal-control refinement step.","marker":"[27]"}],"fun_headline_variants":["Video-guided RRT solves sequential manipulation tasks","One demo video supplies grasp and release states for TAMP","Multi-tree planner boots from video contact states","Robot learns pick sequence from human video","Video turns blind TAMP search into guided planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstration video must already encode a complete and robot-feasible plan: every intermediate object placement and every grasp shown must be one the robot can actually reach and execute, because the planner has no mechanism to invent a different contact sequence when a demonstrated step is infeasible.","fun_headline_variants_meta":{"raw":{"variants":["Video-guided RRT solves sequential manipulation tasks","One demo video supplies grasp and release states for TAMP","Multi-tree planner boots from video contact states","Robot learns pick sequence from human video","Video turns blind TAMP search into guided planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1593,"prompt_tokens":935,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":589}},"tokens_in":551,"tokens_out":658,"duration_ms":7176,"temperature":1.0,"reasoning_tokens":589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:44:36.482715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the tunnel task and shift the demonstrated intermediate placement pose just outside the robot's reachable workspace while a collision-free alternative placement exists elsewhere inside the tunnel; if the planner then fails or times out instead of finding the alternative, the claim that the video-derived placement states are sufficient guidance would be falsified.","supporting_citations":[{"cited_title":"Prehensile manipulation planning: Mod- eling, algorithms and implementation,","cited_arxiv_id":null,"evidence_quote":"Defines the admissible configuration space as a set of placement/grasp states with transitions, the representation the paper builds on."},{"cited_title":"Rrt-connect: An efficient approach to single- query path planning,","cited_arxiv_id":null,"evidence_quote":"Supplies the bidirectional tree-growing RRT method used as a baseline and as the starting point for the multi-tree extension."},{"cited_title":"Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning,","cited_arxiv_id":null,"evidence_quote":"The symbolic TAMP baseline whose success rate and planning time are compared against in the benchmark."},{"cited_title":"Multi-contact task and motion planning guided by video demonstration,","cited_arxiv_id":null,"evidence_quote":"The earlier conference version of this work, extended here with generalization experiments and trajectory refinement."},{"cited_title":"Understanding human hands in contact at internet scale,","cited_arxiv_id":null,"evidence_quote":"The hand-contact recognizer that yields the grasp/release contact states from the input video."},{"cited_title":"Cosypose: Consistent multi-view multi-object 6d pose estimation,","cited_arxiv_id":null,"evidence_quote":"The 6D pose estimator that provides object poses at every contact-state change."},{"cited_title":"On delaying collision checking in prm planning: Application to multi-robot coordination,","cited_arxiv_id":null,"evidence_quote":"The random-shortcut method reused for path optimization before the optimal-control refinement step."}],"review_version":1}