Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Robots learn to swing unfamiliar tools by predicting keypoint motion

2026-07-09 00:04 UTC pith:R4ZVWOXX

load-bearing objection Solid empirical recipe for cross-tool transfer, but the 'functional reasoning' claim is overstated — the hardest sub-problem (contact identification) is externally solved. the 3 major comments →

arxiv 2607.05780 v1 pith:R4ZVWOXX submitted 2026-07-07 cs.RO cs.AIcs.CV

FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

Chuhao Zhou , Liquan Wang , Shuxin Cao , Xiangyu Chen , Yuxuan Hu , Boyu Ma , Animesh Garg , Jianfei Yang This is my paper
classification cs.RO cs.AIcs.CV
keywords functional generalizationtool-use manipulationkeypoint trajectoriesintermediate representationrobot learningflow matchingaction-free learningaffordance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tackles a problem the authors call functional generalization: a robot trained to hit a target with a hammer should also succeed when handed a shoe, a book, or a scoop, even though each tool demands a different grip, contact point, and swing path. The core difficulty is that functionally equivalent tools look similar in image space but require entirely different actions, so a standard end-to-end policy that maps pixels directly to motor commands overfits to the tools it was trained on. The authors propose FORGE (Functional Reasoning and Grounded Execution), which splits the problem into two stages. First, a keypoint motion predictor trained on action-free video data forecasts how a handful of tracked 2D points on the tool and target should move over time, capturing the functional intent (where to contact, how to approach) without committing to any particular tool's appearance. Second, a separate execution policy trained on a small set of robot demonstrations translates those predicted keypoint trajectories into actual motor commands. The central claim is that 2D keypoint trajectories are the right intermediate representation for this task, striking a balance between affordance images (too static) and raw video prompts (too entangled with appearance). On a seven-tool hitting benchmark in simulation and real-world experiments, FORGE more than doubles the average success rate of end-to-end baselines on unseen tools, and also outperforms a method that uses generic keypoint tracking rather than function-aware prediction.

Core claim

The paper's central finding is that decoupling functional reasoning from action execution via 2D keypoint trajectories enables a robot to transfer a hitting function to unseen tools. The keypoint trajectory serves as a bridge representation: it is compact enough to be predicted from action-free observations (no robot data needed for the reasoning stage), structured enough to encode where contact should happen and how the tool should move toward the target, and groundable enough that a small execution policy can convert it into actions. The authors systematically compare this representation against affordance images and human video prompts, finding keypoint trajectories superior on both seen-

What carries the argument

FORGE's pipeline has two stages. Stage 1 trains a conditional flow-matching model (the keypoint motion predictor, called System-2) on action-free data: given a visual observation and current keypoint positions, it predicts future 2D keypoint trajectories. Keypoints are obtained by segmenting the tool mask with SAM2, sampling points via Farthest Point Sampling, and tracking them with CoTracker. Stage 2 freezes the predictor and trains a second flow-matching model (the execution policy, called System-1) on a smaller action-labeled dataset: it takes the predicted keypoint trajectories, adds pixel-level perturbations for robustness, and maps them to robot action chunks. The two-stage factor lets

Load-bearing premise

The entire pipeline depends on 2D keypoint trajectories being a sufficient intermediate representation to bridge from perceiving a novel tool to executing the correct action. If the 2D keypoints fail to capture depth or fine-grained spatial alignment needed for precise contact, the downstream execution policy cannot recover, and the authors' own failure analysis confirms this as a real limitation.

What would settle it

If a 3D keypoint or depth-aware representation replaced the 2D version and substantially improved success rates on the same benchmark, it would suggest that the 2D representation was a bottleneck rather than a sufficient bridge, weakening the claim that 2D keypoint trajectories are the right intermediate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the keypoint-trajectory approach generalizes beyond hitting to other functional tool-use tasks (cutting, scraping, prying), it could reduce the data cost of teaching robots to repurpose novel objects, since the reasoning stage needs no robot demonstrations.
  • The failure mode of imprecise hitting due to 2D representation limits suggests that extending to 3D keypoint trajectories or depth-aware representations could be a direct next step, which the authors themselves flag.
  • The monotonic improvement with action-labeled tool diversity (1-to-6: 0.21, 4-to-3: 0.36, 6-to-1: 0.52) quantifies a practical data trade-off: moderate robotic data suffices for grounding, but performance scales with tool coverage, giving practitioners a concrete budgeting curve.
  • The approach currently requires the hitting and target points to be specified externally; automating function-relevant keypoint identification would be necessary for fully autonomous deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The two-stage architecture implicitly claims that functional intent is a transferable abstraction separable from motor execution. If this holds, the same decomposition could apply to other perception-to-action gaps in robotics, such as grasping novel objects where visual similarity does not predict grasp kinematics.
  • The finding that generic keypoint tracking (ATM baseline) underperforms function-aware prediction suggests that not all intermediate representations are equal: the representation must be trained with functional supervision, not just geometric tracking. This raises the question of whether unsupervised discovery of functional keypoints could match the supervised approach.
  • The benchmark is limited to a single function (hitting) with seven tools. Whether keypoint trajectories remain the best intermediate representation for functions requiring sustained contact, multi-step tool use, or tool-on-tool interaction is untested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes FORGE, a two-stage policy for functional generalization in robotic tool use. The key idea is to decouple functional reasoning (predicting 2D keypoint trajectories from action-free data) from grounded execution (mapping keypoints to robot actions with limited demonstrations). The paper introduces a seven-tool hitting-function benchmark and shows that FORGE achieves ~2X improvement in average success rate over baselines (DP, FM, ATM) on unseen tools in both simulation and real-world experiments. Ablations compare intermediate representations (affordance images, human video prompts, keypoint trajectories) and test the effect of action-labeled tool diversity.

Significance. The paper addresses a meaningful problem—generalizing tool-use skills to novel tools that serve the same function. The two-stage decomposition that allows the keypoint motion predictor to train on action-free data while the execution policy trains on a smaller action-labeled set is a reasonable and practical design. The systematic comparison of intermediate representations (Table 2) and the function-aware vs. generic keypoint comparison (ATM baseline) are informative. Real-world validation on multiple unseen tools (Tables 6, Fig. 7) strengthens the empirical contribution.

major comments (3)
  1. Section 1 defines functional generalization as requiring three capabilities: 'locating its contact region, aligning it, and producing an appropriate motion.' However, Section 4.1 and Appendix A.1 confirm that the hitting point and target point are pre-specified as inputs to all methods. This means the first capability—identifying which part of an unseen tool serves the function—is externally solved rather than demonstrated by FORGE. The paper's central framing of 'functional generalization' and 'functional reasoning' therefore overstates what the experiments actually test. The 2X improvement (Table 1: 0.36 vs. 0.17 for DP) may be attributable to better motion conditioning via structured keypoint trajectories rather than the transferable functional reasoning claimed. The authors acknowledge this in Section 5 ('the functional-relevant hitting and target points are specified in this study')
  2. The evaluation is limited to a single function (hitting). While the paper frames functional generalization broadly ('repurposing a book, a stone, or a shoe to drive a nail'), the experimental validation covers only one functional category. This limits the strength of claims about 'functional generalization' as a general capability. The paper should either temper its claims to reflect that this is a single-function study or provide evidence that the approach transfers across functions.
  3. Table 1: The success metric depends on whether the robot 'strikes the target cube using the designated hitting point' (Section 4.1), but the tolerance for what counts as a successful hit is not specified in the main text. The reader's report flags this as a concern. The paper should state the tolerance explicitly and, ideally, report sensitivity to it, since the absolute success rates (0.36 overall) leave little margin before tolerance choices affect the qualitative conclusions.
minor comments (6)
  1. Section 3.2: The representation selection experiment (Fig. 2a) is described as a 'heuristic experiment' but the methodology—how many samples, what success criterion in pixel space, what tolerance—is not detailed in the main text. Table 2 provides results but the protocol should be clarified.
  2. Table 3: The 1-to-6 setting trains on only the hammer and tests on 6 unseen tools. The 0.21 average SR is reported, but it would be useful to know whether any individual unseen tool performs notably better or worse, as this could reveal tool-specific biases.
  3. Section 3.3, Eq. (2) and Eq. (5): The flow-matching formulation is standard, but the dimensionality of the keypoint space (N=5 keypoints × 2 coordinates × H horizon) and action space should be stated for reproducibility.
  4. Appendix A.1: The real-world benchmark is described as containing 6 tools (Table 4) but the main text (Section 4.1) mentions 3 tools for the primary real-world setting and 3 unseen tools in Appendix B.1. This discrepancy should be clarified.
  5. Figure 2: The notation π_sys1 and π_sys2 is used before being defined. The mapping between 'System-1'/'System-2' naming and 'execution policy'/'keypoint motion predictor' should be made explicit earlier.
  6. The paper would benefit from reporting standard deviations or confidence intervals on the success rates in Tables 1 and 6, given the moderate sample sizes (30 rollouts per setting in simulation, 10 per pattern in real-world).

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) that pre-specifying hitting and target points means FORGE does not fully demonstrate the 'locating' capability claimed in our framing of functional generalization; (2) that evaluation is limited to a single function (hitting), which limits the breadth of claims about 'functional generalization'; and (3) that the success tolerance for hitting is not specified in the main text and sensitivity analysis is lacking. We address each point below and commit to revisions in the revised manuscript.

read point-by-point responses

  1. Referee: Section 1 defines functional generalization as requiring three capabilities: locating the contact region, aligning it, and producing appropriate motion. But Section 4.1 and Appendix A.1 confirm hitting and target points are pre-specified, so the first capability is externally solved. The framing of 'functional reasoning' overstates what the experiments test, and the 2X improvement may be attributable to better motion conditioning rather than transferable functional reasoning.

    Authors: We agree with the referee that the current experiments do not test the full three-capability definition of functional generalization as stated in Section 1. Specifically, the hitting point and target point are provided as inputs to all methods (including baselines), so the 'locating' capability is not evaluated by our benchmark. We acknowledge that our framing overstates what the experiments demonstrate. In the revised manuscript, we will: (1) revise the definition in Section 1 to distinguish between the full functional generalization problem and the specific sub-problem we evaluate, which we will describe as 'functional motion generalization given specified contact regions'; (2) adjust the abstract and introduction to make clear that FORGE addresses the alignment and motion production components, while contact-region localization is left to future work (as already noted in Section 5); and (3) add a clarifying note that the 2X improvement should be interpreted as evidence that structured keypoint trajectories provide a better intermediate representation for motion conditioning across unseen tools, rather than as evidence that FORGE solves the full functional reasoning pipeline including contact identification. We believe the contribution remains meaningful: the keypoint trajectory representation and two-stage decomposition demonstrably improve cross-tool motion transfer even when contact points are given, and the systematic comparison of intermediate representations (Table 2) provides actionable insights for the broader problem. However, we accept that the claims must be scoped more precisely. revision: yes

  2. Referee: The evaluation is limited to a single function (hitting). While the paper frames functional generalization broadly, experimental validation covers only one functional category. The paper should either temper its claims or provide evidence that the approach transfers across functions.

    Authors: The referee is correct that our experimental validation is limited to the hitting function. We chose hitting because it cleanly exhibits the perception-to-action gap—tools with visually different geometries require different motions to achieve the same functional outcome—and because it admits a well-defined success metric. However, we agree that the broad framing of 'functional generalization' in the abstract and introduction is not fully supported by a single-function study. In the revised manuscript, we will: (1) explicitly state in the abstract and introduction that the current work instantiates functional generalization with the hitting function as a first case study; (2) add a sentence in Section 4 noting that extending to additional functions (e.g., cutting, scooping, scraping) is a natural next step and that the keypoint trajectory representation is in principle function-agnostic, but we do not claim empirical evidence of cross-function transfer; and (3) temper claims throughout to use 'functional generalization in the hitting function' or 'single-function functional generalization' where appropriate. We will not add new multi-function experiments in this revision cycle, as we cannot honestly claim results we have not obtained. We note this as a genuine limitation in the revised Discussion section. revision: yes

  3. Referee: Table 1: The success metric depends on whether the robot strikes the target cube using the designated hitting point, but the tolerance for what counts as a successful hit is not specified. The paper should state the tolerance explicitly and report sensitivity to it, since absolute success rates (0.36 overall) leave little margin before tolerance choices affect qualitative conclusions.

    Authors: We agree that the success tolerance should be specified in the main text. In our current implementation, a rollout is considered successful if the designated hitting point on the tool makes contact with the target cube, where contact is detected by the simulator's collision checker. The target cube has a side length of 2 cm, so the effective spatial tolerance is approximately 2 cm in the xy-plane. We will add this specification to Section 4.1 in the revised manuscript. Regarding sensitivity analysis: we agree this is a reasonable request and will include a sensitivity analysis varying the tolerance threshold (e.g., at 1 cm, 2 cm, and 3 cm) for the simulation results. We note that the qualitative ranking of methods is unlikely to change substantially, since FORGE's advantage stems from better alignment of the contact region with the target (as shown in the qualitative results), not from marginal tolerance boundary effects. However, we will report the sensitivity data honestly and let the reader assess. If the sensitivity analysis reveals that conclusions are fragile under tighter tolerances, we will report that transparently. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the two-stage pipeline is self-contained with genuinely separate data sources and independent evaluation.

full rationale

The paper's central derivation chain is not circular. Stage 1 (keypoint motion predictor) trains on action-free data where ground-truth keypoint trajectories are obtained externally via SAM2 segmentation + FPS sampling + CoTracker tracking (Appendix A.1). The flow-matching model (Eq. 2) learns to predict future keypoint trajectories from observations and current keypoints—these targets are externally generated, not defined by the model itself. Stage 2 (execution policy, Eq. 5) trains on a separate action-labeled dataset to map predicted keypoints to actions. The two stages use genuinely different data sources (action-free vs. action-labeled), and the predicted keypoint trajectories are not equivalent to the action output by construction. The '2X improvement' claim is evaluated against baselines (FM, DP, ATM) that receive identical inputs including the pre-specified hitting point, making the comparison fair. The skeptic's concern about pre-specified hitting points is a validity/external-validity issue (does the experiment test functional reasoning or motion conditioning?), not a circularity issue (is the result equivalent to the input by definition?). The paper honestly acknowledges this limitation in Section 5. Self-citations (e.g., Ma et al. 2026, An et al. 2026, Li et al. 2026, Turpin et al. 2021/GIFT with overlapping authors) appear only in related works and are not load-bearing for the method or its derivation. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The ledger captures the hand-tuned hyperparameters and core assumptions. The free parameters are standard for flow-matching and MPPI setups but are not derived from first principles. The key axiom is the sufficiency of 2D keypoints, which the paper itself partially questions in its limitations.

free parameters (5)
  • N (number of keypoints) = 5
    Chosen empirically for tool keypoint sampling via FPS.
  • H (prediction horizon) = 8
    Action chunk size and history horizon set to 8.
  • Perturbation magnitude = 1-5 pixels
    Random pixel perturbation range for robustness training.
  • Perturbation probability = 0.5
    Probability of applying keypoint perturbation.
  • MPPI reward weights = Various (e.g., 18.0, 100.0, 22.0)
    Hand-tuned weights for the four-stage MPPI planner to generate demonstrations.
axioms (3)
  • domain assumption 2D keypoint trajectories preserve sufficient functional intent to bridge the perception-to-action gap for tool use.
    The entire pipeline hinges on this representation being adequate, though the authors note 3D limitations in failure cases.
  • domain assumption The hitting and target points can be reliably specified or detected.
    The paper assumes these points are given; automatic identification is left to future work.
  • ad hoc to paper Functional generalization requires decoupling functional reasoning from action execution.
    This is the core architectural hypothesis (H1) motivating the two-stage design.

pith-pipeline@v1.1.0-glm · 16519 in / 2166 out tokens · 241106 ms · 2026-07-09T00:04:32.231900+00:00 · methodology

0 comments
read the original abstract

While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 9 internal anchors

  1. [1]

    Feedback World Model Enables Precise Guidance of Diffusion Policy

    Tuo An, Jindou Jia, Gen Li, Jingliang Li, Chuhao Zhou, Pengfei Liu, Bofan Lyu, Jiaqi Bai, Xinying Guo, Geng Li, et al. Feedback world model enables precise guidance of diffusion policy.arXiv preprint arXiv:2605.15705,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Object-Centric Representations Improve Policy Generalization in Robot Manipulation

    Alexandre Chapin, Bruno Machado, Emmanuel Dellandrea, and Liming Chen. Object-centric representations improve policy generalization in robot manipulation.arXiv preprint arXiv:2505.11563,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    World Model for Robot Learning: A Comprehensive Survey

    Bohan Hou, Gen Li, Jindou Jia, Tuo An, Xinying Guo, Sicong Leng, Haoran Geng, Yanjie Ze, Tatsuya Harada, Philip Torr, et al. World model for robot learning: A comprehensive survey.arXiv preprint arXiv:2605.00080,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects

    Jingliang Li, Jindou Jia, Tuo An, Chuhao Zhou, Xiangyu Chen, Shilin Shan, Boyu Ma, Bofan Lyu, Gen Li, and Jianfei Yang. Compassad: Intent-driven 3d affordance grounding in functionally competing objects.arXiv preprint arXiv:2604.02060,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Dit4dit: Jointly modeling video dynamics and actions for generalizable robot control.arXiv preprint arXiv:2603.10448,

    Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang, Andy Cui, Junwei Liang, and Shuo Yang. Dit4dit: Jointly modeling video dynamics and actions for generalizable robot control.arXiv preprint arXiv:2603.10448,

    work page arXiv

  8. [8]

    mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

    Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Octo: An Open-Source Generalist Robot Policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Any-point Trajectory Modeling for Policy Learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning.arXiv preprint arXiv:2401.00025,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Table 5 Main Reward Terms for MPPI-based Demonstration Generation.Here, dx, dy, and dxy denote the distances between the hitting point and the target along thex axis, the y axis, and thexy plane, respectively, while∆dx,∆ dy, and∆ dxy denote their step-wise reductions. For the vertical direction,zlift denotes the tool lifting height from its initial height...

    work page 2025

  12. [12]

    For baseline methods, we train each policy for 120,000 steps using the corresponding action-labeled data

    We build the simulation benchmark on Robomimic and conduct real-world experiments on a Franka robot platform. For baseline methods, we train each policy for 120,000 steps using the corresponding action-labeled data. For FORGE, we follow the two-stage training procedure described in Sec. 3.3. We first pretrain the keypoint motion predictor πsys2 on action-...

    work page 2023