Pith Number

pith:AU4OE26S

pith:2024:AU4OE26SURC4YOGHCIKZG2RO6K

not attested not anchored not stored refs resolved

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

Chen Wang, Li Fei-Fei, Ruohan Zhang, Wenlong Huang, Yunzhu Li

Manipulation tasks are solved in real time by optimizing sequences of relational keypoint constraints generated automatically from language instructions and RGB-D observations.

arxiv:2409.01652 v2 · 2024-09-03 · cs.RO · cs.AI · cs.CV

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{AU4OE26SURC4YOGHCIKZG2RO6K}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations.

C2weakest assumption

The vision-language models will reliably generate correct, complete, and numerically stable Python constraint functions for arbitrary new tasks and scenes without introducing errors that break the optimizer or produce unsafe actions.

C3one line summary

ReKep encodes robotic tasks as optimizable Python functions over 3D keypoints that are generated automatically from language and RGB-D input, enabling real-time hierarchical planning on single- and dual-arm platforms without task-specific data.

References

158 extracted · 158 resolved · 19 Pith anchors

[1] L. P. Kaelbling and T. Lozano-P ´erez. Hierarchical planning in the now. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010 2010

[2] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Conference on robot learning, pages 245–255. PMLR, 2022 2022

[3] A. Simeonov, Y . Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V . Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 Inter 2022

[4] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research , pages 132–157. Springer, 20 2019

[5] DINOv2: Learning Robust Visual Features without Supervision 2023 · arXiv:2304.07193

Formal links

2 machine-checked theorem links

Cited by

27 papers in Pith

Point Tracking Improves World Action Models

A Survey on Vision-Language-Action Models for Embodied AI

Action with Visual Primitives

FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulation

COMPASS: Confined-space Manipulation Planning with Active Sensing Strategy

Receipt and verification

First computed	2026-05-17T23:38:48.511579Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

0538e26bd2a445cc38c71215936a2ef29a4cec6355412a7391ab2d9a77754a15

Aliases

arxiv: 2409.01652 · arxiv_version: 2409.01652v2 · doi: 10.48550/arxiv.2409.01652 · pith_short_12: AU4OE26SURC4 · pith_short_16: AU4OE26SURC4YOGH · pith_short_8: AU4OE26S

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/AU4OE26SURC4YOGHCIKZG2RO6K \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 0538e26bd2a445cc38c71215936a2ef29a4cec6355412a7391ab2d9a77754a15

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "66448a9fef71bb9aeed2a6f013fc0254372fead7269858ee59f22b2508bdd824",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.CV"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.RO",
    "submitted_at": "2024-09-03T06:45:22Z",
    "title_canon_sha256": "93a498b7c81204f6e0578dac255065e9366313125d4cb49b8e15fb629cfd4561"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2409.01652",
    "kind": "arxiv",
    "version": 2
  }
}