{"id":"4ee483b6-1fa7-498c-8ea7-807681796b80","arxiv_id":"2506.21057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A semantic keypoint graph matched to novel objects lets imitation-learned manipulation policies generalize with a quarter of the demonstrations.","lead":"The authors propose a robot learning framework that turns a single image of an object into a graph of semantic keypoints, then uses template matching to transfer manipulation skills to new objects. The central claim is that this knowledge-driven representation gives higher success rates and fourfold sample efficiency over image-based diffusion policies on three real-world tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generalization claim rests on a single-template assumption that Table IV shows is only partially met, so the mechanism is not yet established.","rationale":"I read the paper as claiming that a single knowledge template per category, matched by structural-semantic alignment, is sufficient to abstract novel objects into a representation that enables data-efficient imitation learning. The load-bearing premise is exactly the one the reader identified: the fixed template and DINOv2 features must transfer reliably across category instances. The paper states this assumption in Section III-B, and the only direct evidence is Table IV, which shows only 59.2% of keypoints match within 3 cm even in rendered conditions. That is a partial failure of the assumption, not a clean confirmation. The real-robot generalization results in Table II involve only 10 trials per condition and no template-seed variation, so they cannot establish that the matching is reliable enough to drive the claimed generalization. The reader's conditional verdict is appropriate; my concern reinforces the same weakest assumption rather than identifying a different one. If the proposed concrete test shows high matched rates and stable success across template seeds, the concern would be resolved and the paper's central claim would be much better supported. For now, the verdict should remain conditional on those checks.","tokens_in":10717,"tokens_out":7097,"duration_ms":89788,"concrete_test":"Run the matching evaluation from Table IV on the actual unseen objects used in Table II, using three different training objects per category as template sources (e.g., three different mugs), and report matched rate and average keypoint error at 1/3/5 cm thresholds, plus policy success rates with at least 20 trials per template. If the matched rate at 3 cm is below about 60% for real novel objects, or if success varies by more than about 2/20 across template seeds, the single-template assumption is not met. Also release delta_f, delta_p, beta, and K for each task so the matching ablation can be reproduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism (Sec. III-B) assumes a fixed K-keypoint template, generated from one training object per category, generalizes within a category and that DINOv2 features provide reliable correspondences across instances. This is load-bearing because every policy observation is a matched keypoint vector; if matching is inaccurate or unstable, the policy trains and deploys on corrupted observations. The only quantitative test of this assumption, Table IV on rendered Google Scanned Objects, reports an average error of 3.66 cm and a matched rate (error < 3 cm) of only 59.2%. Thus about 40% of keypoints fail the 3 cm threshold even under favorable rendered conditions, before real-world occlusions, lighting changes, and unseen objects are added. The real generalization results (Table II) are 10-trial counts with no error bars, no template-seed variation, and no reporting of the matching thresholds delta_f, delta_p, beta, or K. Consequently, the observed success could reflect the diffusion policy tolerating noisy keypoints rather than the proposed knowledge template providing reliable generalization. If the single-template assumption fails for a category, the claimed knowledge-driven generalization and data-efficiency advantage are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a knowledge-driven imitation learning framework for robot manipulation. A category-level knowledge template is defined as a graph of K semantic keypoints sampled from a single training object, with each keypoint carrying a DINOv2 semantic feature and a 3D position. A coarse-to-fine template matching algorithm aligns this template to novel instances by jointly optimizing feature similarity and structural consistency, and the resulting matched keypoint coordinates are fed as observations into a Diffusion Policy. The method is evaluated on three real-world tasks (Mug, Tool, Drawer) against image-based Diffusion Policy and P3-PO, with additional experiments on unseen objects, novel backgrounds, and lighting conditions. The authors claim superior performance over image-based Diffusion Policy using roughly one-quarter of the demonstrations and a 54% increase in average success rate from introducing knowledge.","tokens_in":10961,"tokens_out":4624,"duration_ms":54019,"significance":"If substantiated, the proposed framework would be a meaningful step toward data-efficient and generalizable manipulation: it explicitly injects category-level structural-semantic knowledge into an imitation learning pipeline, uses a principled matching objective, and is evaluated on real hardware. The strengths of the paper include a clearly formulated coarse-to-fine matching scheme, the use of pre-trained DINOv2 features, and real-robot comparisons against reasonable baselines. However, the current evidence does not yet support the headline claims. The data-efficiency comparison in Fig. 4 rests on a one-success difference out of 20 trials, and the matching ablation in Table IV shows an average error of 3.66 cm with only a 59.2% matched rate under a 3 cm threshold, which is difficult to reconcile with the stated 1 cm precision requirement of the Mug task. The paper would need substantially stronger quantitative support to establish that the matching mechanism, rather than policy robustness, drives the observed results.","major_comments":[{"comment":"The central data-efficiency claim is not supported by the reported numbers. The comparison of \"our method with 24 demonstrations\" (35% success) against \"Diffusion Policy with 90 demonstrations\" (30% success) is a difference of one success in what appear to be 20 trials, which is well within binomial sampling noise. Please report the exact number of trials per configuration, provide confidence intervals or repeated-seed results, and perform a statistical test (e.g., Wilson intervals or a chi-square test) to justify the claim of superiority at one-quarter of the demonstrations.","section":"IV-C, Fig. 4"},{"comment":"The matching quality reported in Table IV does not establish the mechanism that the paper claims. The template matching achieves an average error of 3.66 cm and a matched rate of 59.2% under a 3 cm threshold, yet Section IV-A2 states that the Mug task requires 1 cm precision. If roughly 40% of keypoints are matched with error exceeding 3 cm under favorable rendered conditions, the observed policy success may be due to the diffusion policy tolerating noisy keypoints rather than to clean, reliable template matching. Please report the full error distribution per keypoint and per object category, and include an ablation that injects controlled keypoint noise to quantify the policy's sensitivity to matching errors.","section":"IV-E, Table IV"},{"comment":"The single-template assumption, stated in Section III-B as \"we assume that the semantic features and structure of the keypoints can be generalized among objects in the same category,\" is only weakly tested. In Table II, the proposed method is worse than P3-PO on the Mug task (6/10 vs 7/10) and ties on the Drawer task (5/10 vs 5/10), both on only 10 trials. These results do not demonstrate consistent generalization gains across categories. Please provide per-object results for all unseen objects, specify whether the same template (with the same K keypoints) was used for every object in a category, and report the template-seed variation to assess robustness of the generalization claim.","section":"III-B and IV-D"},{"comment":"The method as described is not reproducible because the key hyperparameters are never reported. The thresholds δf (feature distance), δp (position distance), the structural weight β, and the number of keypoints K appear in the matching formulation (Eqs. 3–5) and in the implementation (3–20 keypoints), but their values for each experiment are omitted. Please report the values used for all real-world and synthetic experiments, and include a sensitivity analysis for at least δf and β to demonstrate that the results are not sensitive to fine hand-tuning.","section":"III-C"}],"minor_comments":[{"comment":"The manuscript contains several typos and formatting inconsistencies, including \"pratice\" (Section III-B), \"Sematic Keypoint Sampling\" (Section III-B2), \"carse-to-fine\" (Section III-C), \"T ool\" (Section IV-A2 and Fig. 3), and \"Lightning\" instead of \"Lighting\" in Table III and Section IV-D.","section":"Throughout"},{"comment":"The percentage labels inside the bars in Fig. 4 are difficult to read and the sample sizes are not stated. Please annotate each bar with the number of trials and the success count.","section":"Fig. 4"},{"comment":"The policy implementation states that keypoint observations are obtained from the first frame and the policy runs open-loop. This design choice should be justified more explicitly, since it has important implications for the generalization results, especially under lighting changes that occur during an episode.","section":"IV-B3"},{"comment":"The statement that \"the introduction of knowledge leads to a 54% increase in the average success rate of Diffusion Policy\" is ambiguous: the computed difference is 54 percentage points in average success rate, not a 54% relative increase. Please rephrase to avoid misinterpretation.","section":"V"},{"comment":"The paper does not specify how point clouds from the two RealSense cameras are fused or how the semantic features are projected to the robot coordinate system beyond \"using camera calibration results.\" A brief description of the fusion and calibration procedure would improve reproducibility.","section":"IV-B1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the core idea of category-level semantic keypoint graphs is interesting. However, the current evidence does not support the main claims. The data-efficiency comparison is statistically indistinguishable, and the matching ablation suggests that the keypoint observations are substantially noisier than the task precision requires. I recommend requiring additional experiments that directly probe the matching mechanism, including error distributions, sensitivity analyses, and statistical tests, before the paper can be accepted. The claims in the abstract and conclusion should also be moderated to match the available evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2506.21057. The core contribution is a fixed semantic keypoint graph per category, used as a template, matched to new instances with a coarse-to-fine scheme that balances DINOv2 feature similarity with structural consistency, and then used as the observation space for a diffusion policy. That specific combination is new. The earlier graph-keypoint papers [40,41] do in-context learning, not policy training; P3-PO uses top-1 keypoint matching. The real-robot results are the best evidence: three tasks, consistent wins over Diffusion Policy and P3-PO on seen objects, and decent generalization to novel objects and environments. The synthetic matching ablation is also creditable: template matching cuts average error from 6.54 to 3.66 cm and lifts matched rate from 11% to 59%. That is a genuine improvement.\n\nThe soft spots are real but not fatal. The load-bearing assumption, that one hand-selected template per category transfers to all instances, is only partially supported. Table IV shows roughly 40% of keypoints still land beyond 3 cm even in clean rendered conditions, so the mechanism is not cleanly established. The real-robot success suggests the diffusion policy either tolerates that noise or the 3 cm threshold is conservative; the paper should say more about which. Reporting is the bigger issue: 20-25 trials per cell, no seeds or error bars, and the matching thresholds beta, delta_f, delta_p, and K are not reported. The data-efficiency plot is muddy, and the \"54% increase\" in the conclusion does not obviously follow from Table I. These are addressable.\n\nWho it's for: anyone working on keypoint or object-centric imitation learning, or on reducing demonstration counts for manipulation. It deserves a serious referee. My recommendation: conditional accept, with mandatory addition of seeds or more trials, reporting of the hyperparameters, a template-robustness check (vary the seed object and re-measure), and a corrected quantitative summary. The promised code and data release would help a lot.","headline":"A practical keypoint-graph representation for data-efficient imitation learning, with real-robot wins but evaluation reporting that needs tightening.","tokens_in":11476,"tokens_out":3897,"would_cite":true,"duration_ms":41970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed semantic keypoint graph, matched by coarse-to-fine structural and semantic alignment, lets imitation policies learn from a quarter of the demonstrations image-based diffusion needs.","keywords":["imitation learning","robot manipulation","semantic keypoint graph","template matching","generalization","sample efficiency","diffusion policy","DINOv2"],"falsifier":"Render a held-out set of object instances from the Google Scanned Objects dataset in unseen poses and measure the fraction of template keypoints matched within 3 cm. If coarse-to-fine template matching does not clearly beat the 11.2% top-1 matching rate on instances where DINOv2 features are not consistent, or if a category with high intra-class variation shows matched rates near chance, the claim that one fixed template spans a category would fail.","tokens_in":10534,"feed_emoji":"🤖","tokens_out":5337,"duration_ms":55127,"temperature":0.7,"pith_summary":"The paper tries to show that a robot imitation policy can learn manipulation skills from far fewer expert demonstrations when observations are replaced by a category-level semantic keypoint graph instead of raw images. It builds a fixed knowledge template for an object category from one image and matches that template to any new instance through a coarse-to-fine algorithm that balances semantic feature similarity with structural consistency. On three real-world tasks, the resulting knowledge-driven policy outperforms an image-based diffusion policy while using about one quarter of the demonstrations, and the authors report that injecting knowledge raises Diffusion Policy's average success rate by 54%. If correct, this would mean explicit category knowledge can substitute for large amounts of demonstration data in robot manipulation.","feed_headline":"Keypoint templates beat image diffusion policies with 1/4 of the demos","feed_subtitle":"Matching a fixed semantic keypoint graph to new objects yields generalization across novel objects, backgrounds, and lighting.","key_machinery":"The central object is the category-level semantic keypoint graph $T_C = \\{(\\hat{f}_k, \\hat{p}_k)\\}_{k=1}^K$: K semantic keypoints sampled from one object, where $\\hat{f}_k$ is a DINOv2 semantic feature and $\\hat{p}_k$ a 3D position. The load-bearing mechanism is the coarse-to-fine template matching algorithm, which solves a joint optimization of feature similarity and structural consistency. Coarse matching performs a rigid point-set registration with scaling, solved with the Umeyama algorithm under a RANSAC scheme, with mismatches penalized by a threshold on feature distance, and fine matching refines each keypoint's position within a neighborhood by balancing feature distance, structural position penalty, and a position-distance threshold. This matching output, rather than raw images, is the observation fed into a diffusion policy.","core_discovery":"The central claim is that a single fixed template of K semantic keypoints, generated from one training object and matched to every new observation, is a sufficient and beneficial observation space for manipulation imitation learning. The template is a graph: each keypoint carries a DINOv2 semantic feature and a 3D position, and matching minimizes a combined objective of feature distance and structural deformation after a coarse rigid registration with scaling, followed by a fine feature-aligned refinement. The matched keypoint coordinates are fed to a diffusion policy, which treats a novel object as a deformation of the template object. The paper reports that this scheme surpasses image-based Diffusion Policy on all three tasks and matches its success with roughly one quarter of the expert demonstrations, while also generalizing to unseen objects, backgrounds, and lighting conditions.","pith_inferences":["The same template-matching machinery could be run per frame to provide closed-loop keypoint observations, which would address the occlusion cases the paper sidesteps with open-loop control; the reported matching robustness suggests this is a natural extension.","Because the template is category-level and built from a single object, the framework is likely to transfer to new object categories if the chosen keypoints cover task-relevant parts; a stress test would be a category with high intra-class shape variation, where fine matching would need to absorb larger deformations.","The comparison between DINOv2 and DIFT suggests a testable design rule: feature extractors with more homogeneous feature fields are better suited for structural template matching, while locally discriminative features are better for per-point correspondence; this could be checked on other dense visual features.","If the reported gains hold across more tasks, knowledge-driven observation design becomes a complement to data scaling laws in imitation learning, letting practitioners trade demonstrations for a modest amount of per-category annotation."],"forward_implications":["A fixed per-category keypoint template can replace image observations, making manipulation policies learnable from about one quarter of the demonstrations required by image-based Diffusion Policy.","Template matching that enforces structure reduces keypoint ambiguity: the paper's matching improves the fraction of keypoints matched within 3 cm from 11.2% to 59.2% compared with top-1 keypoint matching.","The reported 54% average success-rate increase over Diffusion Policy implies that structured priors can provide some of the benefit that scaling demonstrations provides.","Open-loop execution using first-frame keypoint observations is sufficient for the tested tasks because the template anchors the current object to the training object's coordinate frame.","Generalization to unseen objects, backgrounds, and lighting follows from using a pre-trained semantic feature encoder whose correspondences are robust to environmental changes."],"supporting_citations":[{"why":"Supplies the DINOv2 semantic feature extractor whose per-pixel features define keypoint semantics and drive correspondence.","marker":"[28]"},{"why":"Diffusion Policy is both the policy backbone for the proposed method and the image-based baseline it must outperform with fewer demonstrations.","marker":"[5]"},{"why":"P3-PO is the keypoint-based baseline; its top-1 matching failure mode motivates the structural template matching design.","marker":"[21]"},{"why":"Umeyama's algorithm solves the rigid-with-scaling registration used in coarse template matching.","marker":"[38]"},{"why":"DIFT is the alternative semantic feature extractor compared in the ablation on correspondence quality.","marker":"[35]"},{"why":"SAM provides the object mask used to isolate the target object during keypoint sampling.","marker":"[20]"},{"why":"One of the most similar works that abstracts observations into a keypoint graph but uses geometry-based sampling without semantic features, against which the contribution is positioned.","marker":"[40]"},{"why":"The other most similar keypoint-graph work, also lacking semantic features, used as a point of contrast for the proposed semantic knowledge template.","marker":"[41]"}],"fun_headline_variants":["Semantic keypoint graph: learn manipulation with 25% of demos","Keypoint template matching generalizes imitation across conditions","One keypoint template, quarter demos, robust generalization","Data-efficient imitation: semantic keypoint templates beat images","Imitation learning with keypoint templates: fewer demos, robust"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that one fixed set of keypoints built from a single training object conveys enough semantic and structural information to describe every other object in the same category, and that DINOv2 features give trustworthy matches for those keypoints on novel instances.","fun_headline_variants_meta":{"raw":{"variants":["Semantic keypoint graph: learn manipulation with 25% of demos","Keypoint template matching generalizes imitation across conditions","One keypoint template, quarter demos, robust generalization","Data-efficient imitation: semantic keypoint templates beat images","Imitation learning with keypoint templates: fewer demos, robust"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001274,"raw_usage":{"total_tokens":5160,"prompt_tokens":842,"completion_tokens":4318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":4234}},"tokens_in":458,"tokens_out":4318,"duration_ms":35724,"temperature":1.0,"reasoning_tokens":4234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:16.509701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a held-out set of object instances from the Google Scanned Objects dataset in unseen poses and measure the fraction of template keypoints matched within 3 cm. If coarse-to-fine template matching does not clearly beat the 11.2% top-1 matching rate on instances where DINOv2 features are not consistent, or if a category with high intra-class variation shows matched rates near chance, the claim that one fixed template spans a category would fail.","supporting_citations":[{"cited_title":"DINOv2: Learning Robust Visual Fea- tures without Supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the DINOv2 semantic feature extractor whose per-pixel features define keypoint semantics and drive correspondence."},{"cited_title":"Diffusion Policy: Visuomotor Policy Learn- ing via Action Diffusion","cited_arxiv_id":null,"evidence_quote":"Diffusion Policy is both the policy backbone for the proposed method and the image-based baseline it must outperform with fewer demonstrations."},{"cited_title":"P3-PO: Prescriptive Point Priors for Visuo- Spatial Generalization of Robot Policies","cited_arxiv_id":null,"evidence_quote":"P3-PO is the keypoint-based baseline; its top-1 matching failure mode motivates the structural template matching design."},{"cited_title":"Least-squares estimation of transformation pa- rameters between two point patterns","cited_arxiv_id":null,"evidence_quote":"Umeyama's algorithm solves the rigid-with-scaling registration used in coarse template matching."},{"cited_title":"Emergent Correspondence from Image Diffusion","cited_arxiv_id":null,"evidence_quote":"DIFT is the alternative semantic feature extractor compared in the ablation on correspondence quality."},{"cited_title":"Few-Shot In-Context Imitation Learning via Implicit Graph Alignment","cited_arxiv_id":null,"evidence_quote":"One of the most similar works that abstracts observations into a keypoint graph but uses geometry-based sampling without semantic features, against which the contribution is positioned."}],"review_version":1}