Pith.

Robotics · arXiv:2504.17784 v1 · submitted 24 Apr 2025 · 49 references

Gripper Keypose and Object Pointflow as Interfaces for Bimanual Robotic Manipulation

Yuyin Yang, Zetao Cai, Yang Tian, Jia Zeng, Jiangmiao Pang

The paper says

“Bimanual manipulation is a challenging yet crucial robotic capability, demanding precise spatial localization and versatile motion trajectories, which pose significant challenges to existing approaches.”

more of the abstract

Existing approaches fall into two categories: keyframe-based strategies, which predict gripper poses in keyframes and execute them via motion planners, and continuous control methods, which estimate actions sequentially at each timestep. The keyframe-based method lacks inter-frame supervision, struggling to perform consistently or execute curved motions, while the continuous method suffers from weaker spatial perception. To address these issues, this paper introduces an end-to-end framework PPI (keyPose and Pointflow Interface), which integrates the prediction of target gripper poses and object pointflow with the continuous actions estimation. These interfaces enable the model to effectively attend to the target manipulation area, while the overall framework guides diverse and collision-free trajectories. By combining interface predictions with continuous actions estimation, PPI demonstrates superior performance in diverse bimanual manipulation tasks, providing enhanced spatial localization and satisfying flexibility in handling movement restrictions. In extensive evaluations, PPI significantly outperforms prior methods in both simulated and real-world experiments, achieving state-of-the-art performance with a +16.1% improvement on the RLBench2 simulation benchmark and an average of +27.5% gain across four challenging real-world tasks. Notably, PPI exhibits strong stability, high precision, and remarkable generalization capabilities in real-world scenarios. Project page: https://yuyinyang3y.github.io/PPI/

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Aug 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The interface idea is sensible and the ablations line up, but the headline numbers have enough comparison and consistency problems that the quantitative claims need verification before they can be trusted. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 49 references say.

The paper says, at V-A and Appendix E

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The RLBench2 comparison is not shown to be apples-to-apples. In Section V-A the authors state that they regenerate the training data because the official dataset has robot shadows present during training but absent during evaluation, and because it lacks pointflow meta-information; PPI is trained on this regenerated set.

more of objection 1

However, Appendix E says that ACT and PerAct2 are reproduced with the official RLBench2 code and that DP3 and 3D Diffuser Actor scores are taken from their respective papers, without stating that any baseline was retrained on the regenerated dataset. If the baselines were trained on the original shadow-corrupted data while PPI used corrected data, the reported +16.1% average improvement in Table VI could be partly an artifact of dataset cleanup rather than of the proposed interfaces.

The paper says, at IV-A and Appendix G

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The real-world success metric is under-specified. The text says each method is evaluated across 10 settings with 3 repeated trials per setting, but Tables XI-XVII present exactly 10 binary rows per method, with SR values of 0.0 or 1.0 per row.

more of objection 2

It is unclear whether SR is a per-trial success rate (which would require 30 trials per method), a per-setting success rate with an aggregation rule such as 'any trial succeeded' or 'all trials succeeded', or a record of a single rollout per setting; the normalized score also needs a stated aggregation rule across the three trials. Because the abstract's average +27.5% real-world gain rests on these numbers, the authors should report exact trial-level counts and the aggregation rule for SR, Loc-SR, and normalized score.

The paper says, at Table I and Appendix G

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The Wipe the Plate Loc-SR for PPI is inconsistent between the main table and the detailed results. Table I reports 70.0 for Loc-SR, but Table XIII shows that in 8 of the 10 settings both the sponge and the plate are picked (cases 1, 2, 4, 5, 6, 8, 9, 10), which is 80.0.

more of objection 3

The reported overall average Loc-SR of 92.5 is also consistent with 80.0 for this task, since (100 + 100 + 80 + 90) / 4 = 92.5. Please correct the table and re-audit the other entries against the appendix.

4 minor objections

The paper says, at Throughout

Pith says: minor objection 1 of 4

There are several typographical errors, including 'Robik's Cubes' (Section IV-C), 'Muti-plate' (Table IV), 'performace' and 'excute' (Appendix C), and 'incoporates' (Section III-D).

The paper says, at Figure 2

Pith says: minor objection 2 of 4

The figure contains unexpanded placeholder labels such as 'Scene ??', 'Noisy [KP] ??', and '?,?' in the prediction module, which should be replaced with the actual token definitions or meaningful captions.

show 2 more

The paper says, at Appendix H, Table XVIII

Pith says: minor objection 3 of 4

The label 'Lighting' appears where 'Flickering' is intended in the 3D Diffuser Actor row of the Carry the Tray generalization table; this should be corrected to avoid ambiguity.

The paper says, at V-A

Pith says: minor objection 4 of 4

The sentence 'we regenerate the training data' should explicitly state whether the same regenerated data were used for all internal ablations in Table VII and for the final PPI evaluation, so readers can interpret the ablation relative to the baseline comparison.

What would settle it

Request the regenerated dataset and training configurations, then retrain ACT and PerAct2 (and ideally DP3 and 3D Diffuser Actor) on that exact corrected dataset with the same episode splits, training steps, and checkpoint-averaging protocol, and evaluate 100 rollouts per task as in Table VI. If averaged success of ACT and PerAct2 rises materially (e.g., PerAct2 from 40.0 toward 3D Diffuser Actor's 64.7), the claimed +16.1% margin is at least partly a data artifact. A simpler immediate check: ask the authors whether Appendix E's ACT/PerAct2 runs used the regenerated dataset; the text currently does not say so.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 58 s · 24,758 tokens read, 6,195 written
  3. Editor wrote the verdict above 41 s · 25,617 tokens read, 4,808 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 4 s · 25,084 tokens read, 7,118 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The core story is reasonable: add two auxiliary interfaces to a continuous diffusion policy—predicted gripper keyposes and object pointflow—and let action tokens attend to them. The ablations in Table VII are the most convincing part: adding either interface lifts the continuous baseline, and both together give the best result. That is a real, transferable recipe, and the unidirectional attention design is concrete. Credit is also due for the real-world generalization experiments (new objects, lighting, distractors) and the per-case appendix tables.

The soft spots are where the numbers meet the world. The simulation comparison is the big one. Section V-A says they regenerated the RLBench2 training data to remove robot shadows and add pointflow metadata, and PPI is trained on that set. Appendix E says the baseline numbers for ACT and PerAct2 come from reproducing official code 'in RLBench2', without saying they were retrained on the corrected data; DP3 and 3D Diffuser Actor are taken from their papers. If the baselines were trained on shadow-corrupted data and PPI on clean data, the +16.1% margin is not apples-to-apples. The paper needs to either retrain all baselines on the same regenerated set or show the shadow artifact is performance-neutral.

The real-world tables also have internal inconsistencies. Table I gives Ours Scan the Bottle Loc-SR as 80, but the appendix per-case table (Table XIV) yields 9/10 = 90 for that metric. Table VIII lists Simple PPI Carry SR as 50, while the per-case Table XI gives 4/10 = 40. Table VIII lists PPI Wipe Loc-SR as 70, while the per-case Table XIII gives 8/10 = 80. These are not rounding differences. A reader cannot tell which number is correct.

One more omission: the related work cites BiKC, a keypose-conditioned bimanual policy, but BiKC is not included in the RLBench2 comparison. If it has public results, it should be there.

My take: the architecture and interface idea are worth engaging, and the ablations suggest the mechanism is real. But the empirical claims as written are not yet solid enough to quote. This is exactly the kind of paper that should go to peer review with a specific request for controlled baselines, corrected tables, and ideally code release.

The strongest objection

The central claim is that PPI's architecture achieves state-of-the-art performance, with a +16.1% improvement on RLBench2 (80.8% vs. 64.7% for the best baseline). For this claim to hold, all methods must be trained and evaluated on the same data. Section V-A says the authors 'regenerate the training data' because the official RLBench2 set has robot shadows present in training but absent during evaluation, and because it lacks metadata for pointflow. PPI is trained on this regenerated set. However, Appendix E says: 'In the simulation benchmark, we report the scores for DP3, ACT, PerAct2, and 3D Diffuser Actor from their respective papers. For ACT and PerAct2, we reproduce the results using the official code in RLBench2...' It never states that these baselines were retrained on the corrected, shadow-free regenerated dataset. If ACT and PerAct2 were trained on the original shadow-corrupted data while PPI used the cleaned data, the +16.1% improvement could reflect dataset cleanup rather than the keypose/pointflow interfaces. The same ambiguity applies to DP3 and 3D Diffuser Actor, whose training data is not explicitly identified. The paper's internal ablations (Table VII) do support the value of both interfaces, so the concern is not that the method lacks merit; it is that the headline quantitative comparison is not demonstrably controlled.

The reader identified the same weakest assumption, and I agree. The simulation benchmark is the quantitative basis for the 'state-of-the-art' and '+16.1%' headline claims. The paper's own wording creates a direct ambiguity: Section V-A justifies regenerating the data, while Appendix E describes baseline results from official code without noting retraining on that regenerated data. Since shadows appear in training but not evaluation, point-cloud-based baselines are especially disadvantaged. The real-world results are less central: they show consistent qualitative gains but rely on small trial counts and contain minor arithmetic inconsistencies (e.g., Table I normalized-score averages off by 0.1 for two rows), which would not reverse the direction of the comparison. In contrast, the simulation margin could shrink or disappear if baselines are retrained on the corrected dataset. The ablations in Table VII are independent evidence that the interfaces contribute, so the concern targets the headline comparison rather than the method's internal validity. A conditional verdict with a request for baseline retraining or explicit clarification is appropriate; hence the reader's verdict should remain unchanged.

The referee's report

If the controlled comparisons are confirmed, the contribution is solid and timely: the paper integrates two auxiliary prediction targets (gripper keypose and object pointflow) into a continuous diffusion-policy framework, with internal ablations in Table VII showing that each interface separately improves success rate and that combining them yields the best performance. The method is clearly specified, including all losses, attention structure, hyperparameters, and training/inference procedures, and the appendix is unusually complete with per-case real-world results. The central caveat is that the headline quantitative claims are not yet demonstrably controlled: the RLBench2 comparison may mix training datasets between PPI and the baselines, and the real-world success-rate definition is ambiguous. The internal ablations and the attention visualizations are valuable evidence for the mechanism, but the abstract's quantitative claims need a controlled comparison to be fully credible.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that the two dominant strategies for bimanual manipulation—keyframe-based policies, which predict sparse gripper poses and execute them with motion planners, and continuous-control policies, which predict actions at every timestep—can be combined in one end-to-end diffusion policy rather than treated as alternatives. The proposed model, PPI, predicts two intermediate interfaces at keyframe timesteps: target gripper poses and the 3D pointflow of the manipulated object (where a set of points sampled on the object moves over the next few keyframes), and conditions continuous action generation on both. The claim is that these interfaces give a continuous policy the spatial localization of keyframe methods while preserving its ability to produce curved, collision-free trajectories. On the RLBench2 bimanual benchmark the method reports an average success rate of 80.8%, a +16.1% improvement over prior baselines, and on four real-world tasks an average +27.5% gain in success rate, with robustness to unseen objects, lighting changes, and distractor objects. If correct, the result shows that keyframe information is best used as conditioning for a continuous policy, not as an execution plan.

What carries the argument. The load-bearing object is the interface-conditioned diffusion transformer: a denoising network whose tokens carry explicit 3D positions and attend in a fixed causal order, with relative 3D offsets encoded through rotary embeddings. Continuous-action tokens attend to everything, keyframe-action tokens attend to pointflow and scene tokens, pointflow tokens attend to scene and language tokens, and all tokens attend to language. Three supporting mechanisms carry the argument. First, a 3D semantic neural field built from multi-view RGBD images, fusing self-supervised visual features into downsampled scene points via a hierarchical point encoder, gives every scene token spatial and semantic identity. Second, initial query points $F_0$ are sampled from the object using open-vocabulary detection and segmentation, anchoring the pointflow prediction to the object rather than to world coordinates; this converts the learning problem from estimating $p(F)$ to estimating $p(F \mid F_0)$. Third, a heuristic keyframe splitter identifies turning points in the demonstration and provides the supervision targets for the keypose interface. Together these components let the continuous policy see where the grippers should be and where the object is going, without ever committing to a motion-planned keyframe trajectory.

Core claim. The central discovery is that a single continuous-action diffusion transformer can localize as precisely as keyframe methods by predicting two auxiliary interfaces at keyframe timesteps: future gripper poses and the future positions of object query points (the pointflow). The model is trained end-to-end with three losses—denoising noise for continuous actions, denoising noise for keyframe actions, and direct regression of the pointflow—and uses a unidirectional attention pattern in which continuous-action tokens attend to language, scene, keypose, and pointflow tokens, while keypose tokens attend to pointflow and scene, and pointflow tokens attend to scene and language. The pointflow interface deliberately estimates the conditional distribution $p(F \mid F_0)$, the future positions of query points given their initial positions on the object, rather than absolute coordinates, which the paper argues is why the policy generalizes to unseen objects and distractors. Empirically, the paper reports 80.8% average success on seven RLBench2 tasks (best checkpoint 82.6%) versus 64.7% for the strongest prior method, and an average 62.5% success across four real-world tasks versus at most 35.0% for the baselines, with the largest margins appearing in localization success and stage-wise scores.

The premise it rests on. The reported simulation margin assumes the comparison is apples-to-apples: the paper regenerates the training data for the seven selected tasks to remove robot shadows and add pointflow metadata, but the baselines are reproduced with official code without explicitly stating they were retrained on that same corrected data, so part of the +16.1% could come from the data cleanup rather than from the proposed interfaces.

If the paper is right (Pith's inferences, not the authors')

Keyframe-based and continuous control need not be mutually exclusive: keyframe information can be injected as conditioning tokens in a continuous diffusion policy, preserving curved and collision-free trajectories.

The object pointflow is the larger source of gain: conditioning continuous actions on pointflow alone raises average success from 47.6% to 74.3%, while keypose alone raises it to 53.6%; combining both reaches 80.8% on RLBench2.

Because pointflow is predicted relative to initial object-anchored query points, the policy can be redirected to a novel object by changing the open-vocabulary prompt at the first timestep, enabling zero-shot manipulation of unseen rigid objects.

The method remains stable under lighting changes and object interference, which the paper attributes to interfaces that combine positional and semantic features rather than raw color.

The approach works with small expert datasets (20 demonstrations per real-world task) and can be sped up roughly four times with modest accuracy loss by reducing scene tokens and denoising steps, yielding a control policy running at about 10.5 Hz.

A natural extension is to replace the pose-tracking-based pointflow labels with point tracks or optical flow at data-collection time; if the conditional formulation is what drives generalization, weaker supervision should still yield most of the benefit.

The unidirectional attention design suggests the interfaces act as a learned compression of the scene for the action head; one testable consequence is that pruning scene tokens should degrade performance more gracefully when interfaces are present than when they are absent, and the reported 6x scene-token reduction in the fast variant is consistent with that.

If the reported margins survive a fair re-benchmark on identical training data, the two-interface recipe could become a standard add-on for behavior-cloning policies: predict a sparse goal and an object-motion field, then denoise dense actions conditioned on both.

The claim that pointflow is robust to distractors implies a concrete stress test: place a visually salient but task-irrelevant object between the camera and the target and measure whether attention weights and success rates remain target-focused, since the paper's visualizations suggest pointflow tokens concentrate on the manipulated object's edges.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Bimanual manipulation is a challenging yet crucial robotic capability, demanding precise spatial localization and versatile motion trajectories, which pose significant challenges to existing approaches. Existing approaches fall into two categories: keyframe-based strategies, which predict gripper poses in keyframes and execute them via motion planners, and continuous control methods, which estimate actions sequentially at each timestep. The keyframe-based method lacks inter-frame supervision, struggling to perform consistently or execute curved motions, while the continuous method suffers from weaker spatial perception. To address these issues, this paper introduces an end-to-end framework PPI (keyPose and Pointflow Interface), which integrates the prediction of target gripper poses and object pointflow with the continuous actions estimation. These interfaces enable the model to effectively attend to the target manipulation area, while the overall framework guides diverse and collision-free trajectories. By combining interface predictions with continuous actions estimation, PPI demonstrates superior performance in diverse bimanual manipulation tasks, providing enhanced spatial localization and satisfying flexibility in handling movement restrictions. In extensive evaluations, PPI significantly outperforms prior methods in both simulated and real-world experiments, achieving state-of-the-art performance with a +16.1% improvement on the RLBench2 simulation benchmark and an average of +27.5% gain across four challenging real-world tasks. Notably, PPI exhibits strong stability, high precision, and remarkable generalization capabilities in real-world scenarios. Project page: https://yuyinyang3y.github.io/PPI/

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.61

“In extensive evaluations, PPI significantly outperforms prior methods in both simulated and real-world experiments, achieving state-of-the-art performance with a +16.1% improvement on the RLBench2 simulation benchmark and an average of +27.5% gain across four challenging real-world tasks.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.07 · human participants 0.03 · controlled comparison 0.73 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.28

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Bimanual manipulation is a challenging yet crucial robotic capability, demanding precise spatial localization and versatile motion trajectories, which pose significant challenges to existing approaches. background 1.00 0.05 0.03 asserted 1.00
Existing approaches fall into two categories: keyframe-based strategies, which predict gripper poses in keyframes and execute them via motion planners, and continuous control methods, which estimate actions sequentially at each timestep. background 1.00 0.05 0.02 asserted 1.00
The keyframe-based method lacks inter-frame supervision, struggling to perform consistently or execute curved motions, while the continuous method suffers from weaker spatial perception. gap 0.58 0.08 0.02 asserted 1.00
To address these issues, this paper introduces an end-to-end framework PPI (keyPose and Pointflow Interface), which integrates the prediction of target gripper poses and object pointflow with the continuous actions estimation. contribution 0.89 0.89 0.08 asserted 1.00
These interfaces enable the model to effectively attend to the target manipulation area, while the overall framework guides diverse and collision-free trajectories. method 0.64 0.57 0.08 asserted 0.98
By combining interface predictions with continuous actions estimation, PPI demonstrates superior performance in diverse bimanual manipulation tasks, providing enhanced spatial localization and satisfying flexibility in handling movement restrictions. result 0.82 0.90 0.56 asserted 1.00
In extensive evaluations, PPI significantly outperforms prior methods in both simulated and real-world experiments, achieving state-of-the-art performance with a +16.1% improvement on the RLBench2 simulation benchmark and an average of +27.5% gain across four challenging real-world tasks. result 1.00 0.95 0.43 asserted 1.00
Notably, PPI exhibits strong stability, high precision, and remarkable generalization capabilities in real-world scenarios. result 0.68 0.87 0.13 asserted 1.00
Project page: https://yuyinyang3y.github.io/PPI/ future work 0.42 0.08 0.02 unclear 0.97

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,017 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (15)
Figure 1 of the paper
Figure 1. In contrast to (i) keyframe-based policies, which excel in spatial localization but struggle with movement restrictions
Figure 2 of the paper
Figure 2. Overview of PPI. (a) Perception. We first construct a 3D semantic neural field St and sample initial query points F0 for pointflow prediction. (b) Interface. Next, we define two intermediate interfaces: target gripper poses a k t and object pointflow F. (c) Prediction. Finally, a diffusion transformer incorporates robot proprio tokens ct, scene tokens St, language tokens l, pointflow query tokens F0 and action tokens a k t and a c t with gaussian noise. Using a carefully designed unidirectional attention, the model progressively denoises action predictions conditioned on the…
Figure 3 of the paper
Figure 3. Task process visualizations in four real-world tasks.
Figure 4 of the paper
Figure 4. Two real-world setups. 1) Carry the Tray. Both arms must collaboratively lift a tray stacked with cups or other objects from a lower platform and steadily transfer it to a small table. This task evaluates the ability to precisely locate the tray’s
Figure 5 of the paper
Figure 5. Visualization under object interference in Carry the
Figure 6 of the paper
Figure 6. Visualization under different lighting backgrounds in
Figure 9 of the paper
Figure 9. Heatmap of the attention weights of continuous action tokens. The y-axis represents continuous action tokens as queries. The x-axis sequentially displays continuous action, keypose, pointflow, and scene tokens as keys. (line 3 and 4 in Table VII). Conditioning on separate interfaces improves performance, likely due to the local spatial features they provide. Moreover, combining both interfaces yields further gains, highlighting the synergy between keypose and pointflow in enhancing performance on downstream tasks. D. Visualization Analysis In this section, we intuitively analyze how…
Figure 10 of the paper
Figure 10. Visualization of the attention weights of interface tokens. We use “Lift Tray” task on RLBench2 as an example. The red area corresponds to larger attention weight, while the blue area corresponds to smaller attention weight. Left: The attention weights of continuous tokens to the 1024 scene tokens in the vanilla continuous-action-based policy. Middle: The attention weights of point flow tokens to the scene tokens in PPI. Right: The attention weights of keypose tokens in PPI. query points are sampled from the tray rather than the cube atop it, the model learns to deprioritize the…
Figure 11 of the paper
Figure 11. Detailed architecture of 3D denoising model
Figure 12 of the paper
Figure 12. Detailed 10 settings of Carry the Tray. The tray remains relatively stationary while the platform is moved to 10 different locations, serving as 10 distinct settings. Train Train Test Test 1 2 3 4 5 1 2
Figure 13 of the paper
Figure 13. Detailed 10 settings of Handover and Insert the Plate. On the left, the plate varies across 5 slots on the rack, while on the right, the overall setup shifts between 2 designated positions, combining to form 10 different settings
Figure 14 of the paper
Figure 14. Detailed 10 settings of Wipe the Plate. On the left, the sponge changes positions between the 2 sides of the yellow training area, and on the right, the plate moves across the 5 red areas shown, creating a total of 10 settings. Train Test 9 7 5 3 1 10 8 6 4 2 random
Figure 15 of the paper
Figure 15. Detailed 10 settings of Scan the Bottle. Among the 10 settings, the bottle changes its position within the 10 red areas on the shelf, with variations in both position and rotation, while the scanner is placed randomly in minor adjustments on the platform. G. Detailed Real-world Results We define three metrics: Success Rate (SR), Localization Success Rate (Loc-SR), and Normalized Score. And each task has several intermediate stages. Specifically, Carry the Tray consists of four steps: (1) Pick the Tray, (2) Lift the Tray, (3) Place the Tray, (4) Return. The Loc SR is recorded as 100

Showing 13 of the paper's 15 figures. Figures are the authors', reproduced from the paper.

References (49)
  1. [1] Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. In Arxiv, 2024 on Pith
  2. [2] Wen Bowen, Yang Wei, Kautz Jan, and Birchfield Stan. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024 on Pith
  3. [3] Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. In Arxiv, 2022 on Pith
  4. [4] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023 on Pith
  5. [5] Shivin Devgon, Jeffrey Ichnowski, Ashwin Balakrishna, Harry Zhang, and Ken Goldberg. Orienting novel 3d objects using self-supervised learning of rotation trans- forms. In CASE, 2020 on Pith
  6. [6] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model. In Arxiv, 2023 on Pith
  7. [7] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. In RSS, 2022 on Pith
  8. [8] Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. In TRO, 2023 on Pith
  9. [9] Markus Grotz, Mohit Shridhar, Tamim Asfour, and Dieter Fox. Peract2: A perceiver actor framework for bimanual manipulation tasks. In Arxiv, 2024 on Pith
  10. [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020 on Pith
  11. [11] Stephen James and Andrew J. Davison. Q-attention: Enabling efficient learning for vision-based robotic ma- nipulation. In IEEE Robotics and Automation Letters , 2022 on Pith
  12. [12] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2024 on Pith
  13. [13] Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. In CoRL, 2024 on Pith
  14. [14] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. In CoRL, 2024 on Pith
  15. [15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. In Arxiv, 2023 on Pith
  16. [16] I Liu, Chun Arthur, Sicheng He, Daniel Seita, and Gaurav Sukhatme. V oxact-b: V oxel-based acting and stabilizing policy for bimanual manipulation. In CoRL, 2024 on Pith
  17. [17] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Arxiv, 2023 on Pith
  18. [18] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. In Arxiv, 2024 on Pith
  19. [19] Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicol... on Pith
  20. [20] Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In CoRL, 2023 on Pith
show 29 more
  1. [21] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence , 2017 on Pith
  2. [22] Carl Qi, Yilin Wu, Lifan Yu, Haoyue Liu, Bowen Jiang, Xingyu Lin, and David Held. Learning generalizable tool-use skills through trajectory generation. In IROS, 2024 on Pith
  3. [23] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Arxiv, 2017 on Pith
  4. [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Arxiv, 2021 on Pith
  5. [25] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Ro- man R ¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In Arxiv, 2024 on Pith
  6. [26] Daniel Seita, Yufei Wang, Sarthak J Shetty, Edward Yao Li, Zackory Erickson, and David Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In CoRL, 2023 on Pith
  7. [27] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic ma- nipulation. In CoRL, 2022 on Pith
  8. [28] Jiaming Song, Chenlin Meng, and Stefano Ermon. De- noising diffusion implicit models. In Arxiv, 2020 on Pith
  9. [29] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding. In Arxiv, 2023 on Pith
  10. [30] Ioan A Sucan, Mark Moll, and Lydia E Kavraki. The open motion planning library. In IEEE Robotics & Automation Magazine, 2012 on Pith
  11. [31] Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In ICRA, 2023 on Pith
  12. [32] Yang Tian, Jiyao Zhang, Zekai Yin, and Hao Dong. Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence. In CVPR, 2023 on Pith
  13. [33] Yang Tian, Jiyao Zhang, Guowei Huang, Bin Wang, Ping Wang, Jiangmiao Pang, and Hao Dong. Robokeygen: Robot pose and joint angles estimation via diffusion- based 3d keypoint generation. In ICRA, 2024 on Pith
  14. [34] Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic ma- nipulation. In ICLR, 2025 on Pith
  15. [35] Chenxi Wang, Hongjie Fang, Hao-Shu Fang, and Cewu Lu. Rise: 3d perception makes real-world robot imitation simple and effective. In IROS, 2024 on Pith
  16. [36] Hanqing Wang, Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang, et al. Grutopia: Dream general robots in a city at scale. In Arxiv, 2024 on Pith
  17. [37] Yixuan Wang, Guang Yin, Binghao Huang, Tarik Ke- lestemur, Jiuguang Wang, and Yunzhu Li. Gendp: 3d semantic fields for category-level generalizable diffusion policy. In CoRL, 2024 on Pith
  18. [38] Yixuan Wang, Mingtong Zhang, Zhuoran Li, Tarik Ke- lestemur, Katherine Driggs-Campbell, Jiajun Wu, Li Fei- Fei, and Yunzhu Li. D 3fields: Dynamic 3d descriptor fields for zero-shot generalizable rearrangement. In CoRL, 2024 on Pith
  19. [39] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte... on Pith
  20. [40] Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. In RSS, 2024 on Pith
  21. [41] Thomas Weng, Sujay Man Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet: Bi- manual cloth manipulation with a flow-based policy. In CoRL, 2022 on Pith
  22. [42] Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. In IROS, 2024 on Pith
  23. [43] Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gor- don Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In CoRL, 2024 on Pith
  24. [44] Dongjie Yu, Hang Xu, Yizhou Chen, Yi Ren, and Jia Pan. Bikc: Keypose-conditioned consistency policy for bimanual robotic manipulation. In Arxiv, 2024 on Pith
  25. [45] Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. In CoRL, 2024 on Pith
  26. [46] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. In RSS, 2024 on Pith
  27. [47] Harry Zhang, Ben Eisner, and David Held. Flowbot++: Learning generalized articulated objects manipulation via articulation projection. In CoRL, 2023 on Pith
  28. [48] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In RSS, 2023 on Pith
  29. [49] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper manipulation. In ICLR, 2023. APPENDIX A. Foundation Models PPI leverages foundation models in various aspects, either ... on Pith
Cited by 2 Pith papers
How the literature uses it

7 arXiv papers cite this in 9 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Cites it as background 3 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment