{"id":"f8b85c4f-6775-4cfa-be7e-05ac4438c01d","arxiv_id":"2608.05215","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"From 204K egocentric human videos, the authors automatically extract visual, grasp, and trajectory affordances and train one vision-language model, VLAff, that predicts all three for robot manipulation.","lead":"This paper builds a large dataset of where, how, and how-to-move annotations for everyday object interactions, extracted automatically from people's first-person videos. It then trains a single vision-language model to predict all three at once and shows the outputs can drive a robot arm with no task-specific training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual-affordance SOTA claim may be circular: test labels come from the same automatic pipeline as training, with no human validation or stated disjoint split, and the real-robot gap is underpowered.","rationale":"The reader's weakest assumption — that EgoAffordance pseudo-labels are accurate enough to serve as ground truth — is closely related to my concern, but I sharpen it: the evaluation is not merely noisy-label training; the visual-affordance benchmark is generated by the same automatic pipeline and, apparently, from the same source datasets used for training, making the SOTA comparison circular. This is the single most load-bearing issue because it undermines one of the two headline empirical claims (state-of-the-art visual affordance prediction) without requiring any assumption about baseline unfairness or model architecture. The robot-manipulation numbers do provide some independent signal, but they are too small and methodologically confounded (GraspNet for baselines vs. hand retargeting for VLAff, no error bars, 50 real trials per method) to carry the full weight of the central claim alone. I am not saying the paper is wrong; I am saying the current evidence does not establish the claim as stated. The paper deserves credit for a large dataset and a plausible unified architecture, and the authors explicitly acknowledge trajectory plausibility limitations in the conclusion. My recommendation is unchanged from the reader: CONDITIONAL acceptance, contingent on releasing the dataset/model and validating labels and metrics independently. No ad hominem is intended; this is a verification gap, not an accusation of bad faith.","tokens_in":11908,"tokens_out":3317,"duration_ms":35319,"concrete_test":"Recompute Table II on a held-out set of episodes that are disjoint from EgoAffordance training, using labels independently annotated by humans (e.g., 500 frames, three annotators marking interaction regions), and report per-method IoU/NSS/SIM/KLD with confidence intervals. Also verify that the current 500-scene test set has no episode overlap with the training set. If the ranking collapses or VLAff's IoU drops below LISA/UAD on human labels, the visual-affordance SOTA claim fails; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the headline visual-affordance result (Table II) may be an artifact of the label-generation pipeline rather than a real advance. Section III.B derives ground-truth contact points from fingertip keypoint / object-mask intersection and trajectories from monocular depth, SfM, and inpainting on egocentric video. Section V.A.c then evaluates on '500 randomly sampled scenes from each egocentric video dataset' — i.e., labels produced by the same automatic pipeline, from the same source datasets used to build EgoAffordance (Section V.A.b), with no stated episode-level train/test split and no human validation. Under these conditions IoU/NSS/SIM/KLD measure agreement with the pseudo-labeler, so VLAff's 'state-of-the-art' visual-affordance claim is not established. The real-robot success rate is less circular, but it rests on 10 trials per task (50 per method); the 68% vs. 52% real-world gap is 8 successes out of 50, within plausible binomial noise, and baseline grasps come from GraspNet while VLAff uses hand retargeting, conflating grasp-prediction quality with affordance-prediction quality. The promised trajectory-generation evaluation in Section V is never reported. Together these make the central claim that joint affordance learning beats isolated baselines depend on unverified pseudo-labels and underpowered comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLAff, a vision-language-action affordance model that jointly predicts visual affordance heatmaps, MANO grasp parameters, and trajectory tokens from an RGB image and a language instruction. To train it, the authors build EgoAffordance, a large-scale dataset of 204K episodes with 5.6M visual affordances and 11.6M grasp/trajectory affordances, extracted from egocentric human videos through hand-object detection, hand mesh reconstruction, monocular depth, SfM, and inpainting (Section III). VLAff extends a pretrained VLM (Qwen2.5-VL) with <SEG>, <GRASP>, and trajectory tokens, and is trained with soft Dice, Smooth L1, and cross-entropy losses (Section IV). Experiments (Section V) claim state-of-the-art visual affordance prediction on a test set of 500 scenes per egocentric dataset, zero-shot manipulation in simulation (83.0% vs VidBot's 85.0%) and real-world (68.0% vs VidBot's 52.0%), and success of affordance-guided RL on an Open Fridge task.","tokens_in":12161,"tokens_out":3069,"duration_ms":33049,"significance":"If the empirical claims hold, the paper makes a substantial contribution: it demonstrates that automatically extracted, embodiment-agnostic actionable affordances from human videos can support a unified VLM-based model that transfers zero-shot to real robots, and it releases a large dataset of visual, grasp, and trajectory affordances. The architectural combination of a VLM with specialized decoders and the object-centric formulation are original and plausible. The paper also provides an extensive experimental suite, including real-robot trials and an RL-guidance ablation. The main value is in showing a concrete pathway from internet-scale human video to robot manipulation without robot-specific action labels.","major_comments":[{"comment":"The visual affordance test set is evaluated on 500 randomly sampled scenes from each egocentric video dataset, and the ground-truth labels are generated by the same automatic pipeline (Section III.B) used to produce the training data, with no stated episode-level train/test split and no human validation. Under these conditions, the reported IoU, NSS, SIM, and KLD values measure agreement with the pseudo-labeler, so the 'state-of-the-art' visual affordance claim in the abstract and Section V.B.1 is not established. The authors should provide a disjoint split (e.g., held-out episodes or objects) and, ideally, a human-annotated or otherwise independent validation set for the contact-point and trajectory labels.","section":"V.A.c and Table II"},{"comment":"The real-robot comparison is underpowered: 10 trials per task for 5 tasks gives 50 attempts per method, so the 68% versus 52% gap (34/50 vs 26/50) is within binomial sampling uncertainty (the standard error of a 68% rate over 50 trials is about 6.6 percentage points, and the difference is about 1.6 standard errors with no correction for multiple tasks). Additionally, baseline grasps come from GraspNet while VLAff uses hand retargeting, so the comparison conflates grasp-prediction quality with affordance-prediction quality. Please report per-task confidence intervals or significance tests, and use a shared grasp prediction strategy (or add an ablation that isolates the grasp source) to support the central claim that joint affordance learning beats isolated baselines.","section":"V.B.2 and Table III"},{"comment":"Section V.A.d promises an evaluation of 'trajectory generation effectiveness,' but no such quantitative evaluation is reported anywhere in Section V.B. The only trajectory-related results are indirect, through zero-shot manipulation success, and the model's trajectory tokens are never directly measured (e.g., waypoint error, percentage of physically plausible trajectories, or comparison against other trajectory predictors). Since trajectory prediction is one of the three core affordance outputs and is explicitly named in the abstract, this missing evaluation is load-bearing for the 'unified actionable affordances' claim. Add a dedicated trajectory evaluation or clearly state that trajectory quality is only assessed through the downstream robot tasks.","section":"V.A.d and V.B"},{"comment":"The description of the visual affordance test set does not specify whether the 500 randomly sampled scenes come from the same episodes that were used to build EgoAffordance (Section V.A.b) and whether the additional HANDAL and SceneFun3D data are included in the test sampling. If scenes are sampled at the frame level without an episode-level split, the test set may overlap with the training data, further weakening the generalization claim. Please specify the exact split procedure (episode-level disjoint sets, object categories, or source datasets) and report results separately per data source.","section":"V.A.b and V.A.c"}],"minor_comments":[{"comment":"The methods listed as 'segmentation-based' include LISA, which is a large-language-model-based reasoning segmentation model rather than a classic segmentation method; the normalization of its output to [0,1] should be described in more detail for reproducibility.","section":"V.A.c"},{"comment":"The trajectory sampling strategy in Eq. (6) samples the continuous coordinate p1 but does not specify how this continuous value is mapped to the nearest trajectory token bin used by the autoregressive model; please clarify the quantization mapping.","section":"IV.C, Eq. (6)"},{"comment":"Figure 5 shows learning curves for the Open Fridge RL task without error bars or standard deviations; it is unclear whether each configuration was run with multiple seeds, which is important for the claim that removing the visual heatmap leads to complete failure.","section":"V.B.2, Figure 5"},{"comment":"The 'Frame' column in Table I mixes raw frames for some datasets and interaction-sampled frames for EgoAffordance; this makes the comparison misleading and should be stated in the table caption.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of IROS and the dataset/model release could be of broad interest. The main technical concern is that the headline visual affordance evaluation is potentially self-consistent because the test labels come from the same automatic pipeline as the training data; this is fixable with a true disjoint split and human-validated labels. The statistical power issues in the real-robot comparison are also fixable with more trials and proper error reporting. I have no concerns about citation practice or novelty disclosure. If the authors address the evaluation protocol and the missing trajectory evaluation, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper builds something real — a large egocentric-video affordance dataset (EgoAffordance, 204K episodes with visual, grasp, and trajectory labels) and a VLM that predicts all three jointly. But the headline claims are not yet backed by the experiments as reported. The visual-affordance SOTA is measured against labels from the same automatic pipeline used to create the training data, the promised trajectory evaluation is missing, and the real-robot gap is based on 50 trials per method with a confounded baseline.\n\nWhat is genuinely new: no cited work jointly predicts visual, grasp, and trajectory affordances in one model, and EgoAffordance is much larger than existing affordance datasets with this combination. The object-centric representation — anchor at the 3D interaction point, grasp and trajectory expressed relative to it — is a sensible way to handle embodiment mismatch. The real-robot experiments, though small, at least show the pipeline can produce executable actions from video-derived labels, and the affordance-guided RL ablation cleanly shows the heatmap matters.\n\nWhere it wobbles. Section V.A.c evaluates visual affordance on 500 sampled scenes from the same egocentric datasets used to build EgoAffordance, with labels generated by the same Section III.B pipeline; no human validation and no stated episode-level train/test split. That makes IoU/NSS/SIM/KLD a measure of self-consistency, not generalization. Section V.A promises \"trajectory generation effectiveness\" but no trajectory metrics ever appear; the conclusion itself admits trajectories can be implausible. The zero-shot numbers: 68% vs 52% over 5 real tasks × 10 trials is 34/50 vs 26/50, within binomial noise, and the comparison is not apples-to-apples because VLAff gets hand-retargeted grasps while baselines get GraspNet. No error bars, no hyperparameters, no code or data release.\n\nThese are fixable rather than fatal. A proper revision should release the dataset and code, add human-validated or independently benchmarked test labels, run the trajectory evaluation that is promised, and give per-task confidence intervals. The core idea — unified actionable affordances from human video — is worth pursuing, and the dataset is a potentially valuable resource if made available.\n\nWho this is for: researchers working on affordance pretraining, learning from human video, and VLA models. It deserves a serious referee; I would send it out with a request for major revision rather than desk-reject.","headline":"Large new dataset and a sensible unified affordance model, but the headline results are measured against self-generated labels and underpowered robot trials.","tokens_in":12734,"tokens_out":2428,"would_cite":false,"duration_ms":24674,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single vision-language model that jointly predicts where to touch, how to grasp, and how to move can transfer human-video skills to robots without robot action data.","keywords":["affordance learning","vision-language-action models","zero-shot robotic manipulation","egocentric video","robot learning from human video","grasp pose estimation","trajectory generation","EgoAffordance dataset"],"falsifier":"Independently acquire 3D ground truth for a sample of EgoAffordance episodes using a calibrated multi-camera or RGB-D motion-capture setup, correct the contact points and trajectories accordingly, retrain VLAff, and also evaluate on a held-out domain not represented in the training videos. If performance drops to near the baselines, the reported gains reflect the label-generation pipeline and training-test overlap rather than transferable skill.","tokens_in":11668,"feed_emoji":"🤖","tokens_out":9357,"duration_ms":78461,"temperature":0.7,"pith_summary":"VLAff argues that a robot does not need robot-specific action data to learn manipulation: the where, how, and how-to-move of a task can be extracted from ordinary egocentric human video as object-centric affordances, and a single vision-language model can learn to emit all three at once. The paper builds EgoAffordance, a dataset of over 200K episodes with 5.6M visual affordance heatmaps and 11.6M grasp and trajectory affordances generated automatically from hand-object interactions in egocentric video. VLAff then predicts these affordances from an image and a natural-language instruction, and converts them into executable robot actions using 3D scene geometry. The result is the best reported visual affordance prediction and a 68 percent real-world zero-shot manipulation success rate, 16 points above prior work. If correct, this shows that human videos can substitute for expensive robot demonstration data across many manipulation tasks.","feed_headline":"Unified affordance model lifts real-robot zero-shot success to 68%","feed_subtitle":"Trains on egocentric human video only, yet beats previous zero-shot manipulation policies by 16 points in real kitchens.","key_machinery":"The load-bearing object is the unified affordance triple $f_{\\text{VLAff}}(I,L)=(A_v,A_g,A_t)$ produced by a vision-language model extended with three token types: a <SEG> token whose embedding feeds a segmentation decoder for the visual heatmap, a <GRASP> token decoded into 96-dimensional MANO hand parameters, and discretized trajectory tokens generated autoregressively through spatial binning of a 6D pose space. Two inference strategies convert these outputs into physically plausible 3D actions: in-context trajectory guidance, in which a high-level VLM supplies a coarse 3D direction vector used to sample the first trajectory token, and sampling-based selection, in which K candidate trajectories are scored against voxelized scene occupancy for collision avoidance. All grasp and trajectory predictions are expressed relative to the 3D interaction point derived from the heatmap, which is what makes the affordances object-centric rather than tied to a particular hand or robot.","core_discovery":"The paper's central claim is that jointly predicting three complementary actionable affordances—a visual affordance heatmap $A_v\\in\\mathbb{R}^{H\\times W}$ indicating where to interact, a grasp affordance $A_g\\in\\mathbb{R}^{96}$ (MANO hand parameters in a 6D rotation representation) encoding how the hand holds, and a trajectory $A_t\\in\\mathbb{R}^{T\\times 6}$ describing how the hand moves—within a large vision-language model produces better affordance grounding and policies that transfer directly to robots. The authors state that this joint training is why VLAff outperforms methods that learn affordance modalities in isolation, and that anchoring the grasp and trajectory at the heatmap's peak interaction point makes the representation object-centric and embodiment-agnostic. Empirically, VLAff reports the best visual affordance metrics in its comparison (IoU 0.121, NSS 1.542) and an average real-world zero-shot manipulation success rate of 68.0 percent, 16 points above the VidBot baseline.","pith_inferences":["If the pseudo-label pipeline is systematically biased (for instance, contact points biased by hand morphology or trajectories biased by egocentric depth errors), the visual affordance and zero-shot gains could partly reflect internal consistency rather than true physical correctness; an independent 3D ground-truth check would settle this.","The real-world superiority over VidBot might be driven less by joint affordance learning than by the added in-context trajectory guidance from an external high-level VLM, which injects a privileged semantic prior during inference; ablating the guidance would isolate this.","The same extraction pipeline could be applied to third-person or mixed-view human videos, removing the egocentric bias in depth and occlusion and potentially scaling the dataset further.","The object-centric trajectory anchoring should make VLAff usable with any fixed-base or mobile manipulator that has a parallel gripper; testing on a wider range of embodiments would clarify how embodiment-agnostic the representation really is."],"forward_implications":["Human egocentric video can replace a large fraction of robot-specific action-state data used in imitation learning and vision-language-action models, lowering the cost of building generalist manipulation policies.","Joint training of the three affordance modalities improves each one: visual affordance prediction beats isolated baselines across all reported metrics.","The 68 percent real-world zero-shot success rate indicates that affordances alone, without any robot-specific training, are enough to complete a meaningful fraction of everyday kitchen tasks on two different robot platforms.","Affordance-guided reinforcement learning converges faster when all three modalities are used; removing the visual heatmap makes the policy fail entirely, showing the heatmap is the dominant cue.","The object-centric formulation transfers across embodiments, since grasp and trajectory are expressed relative to the interaction point rather than to a particular robot kinematic frame."],"supporting_citations":[{"why":"Source of egocentric kitchen videos used to build EgoAffordance.","marker":"[8]"},{"why":"Source of large-scale egocentric video used to build EgoAffordance.","marker":"[10]"},{"why":"Used as the high-level VLM for in-context trajectory guidance during inference.","marker":"[13]"},{"why":"Baseline for affordance-from-human-video that VLAff extends and compares against.","marker":"[19]"},{"why":"VidBot, the key zero-shot manipulation baseline that VLAff outperforms by 16 points in the real world.","marker":"[26]"},{"why":"Supplies the 3D hand reconstruction (MANO parameters) used to extract grasp affordances.","marker":"[47]"},{"why":"LISA, the VLM segmentation approach whose <SEG> token and decoder VLAff adapts for visual affordances.","marker":"[54]"},{"why":"DINOv2 self-supervised vision backbone that provides fine-grained visual features for the affordance decoder.","marker":"[55]"},{"why":"Qwen2.5-VL, the base vision-language model VLAff extends.","marker":"[56]"}],"fun_headline_variants":["VLAff unifies visual, grasp, and trajectory affordances to hit 68% zero-shot success","Joint modeling of affordances in VLAff lifts real-robot zero-shot to 68%","VLAff: one model, three affordances, 68% zero-shot robot success","Unified affordance VLM achieves 68% zero-shot manipulation from human video","VLAff predicts where, how to grasp, and how to move—hits 68% on real robots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatically generated contact points (fingertip–object mask intersections) and 3D trajectories (from monocular depth, structure-from-motion, and inpainting) are accurate enough to be treated as ground truth labels, and the visual-affordance test set drawn from the same videos does not hide systematic errors in those labels.","fun_headline_variants_meta":{"raw":{"variants":["VLAff unifies visual, grasp, and trajectory affordances to hit 68% zero-shot success","Joint modeling of affordances in VLAff lifts real-robot zero-shot to 68%","VLAff: one model, three affordances, 68% zero-shot robot success","Unified affordance VLM achieves 68% zero-shot manipulation from human video","VLAff predicts where, how to grasp, and how to move—hits 68% on real robots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001318,"raw_usage":{"total_tokens":5394,"prompt_tokens":996,"completion_tokens":4398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":4275}},"tokens_in":612,"tokens_out":4398,"duration_ms":29027,"temperature":1.0,"reasoning_tokens":4275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:57:10.497100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently acquire 3D ground truth for a sample of EgoAffordance episodes using a calibrated multi-camera or RGB-D motion-capture setup, correct the contact points and trajectories accordingly, retrain VLAff, and also evaluate on a held-out domain not represented in the training videos. If performance drops to near the baselines, the reported gains reflect the label-generation pipeline and training-test overlap rather than transferable skill.","supporting_citations":[{"cited_title":"Scaling egocentric vision: The epic-kitchens dataset,","cited_arxiv_id":null,"evidence_quote":"Source of egocentric kitchen videos used to build EgoAffordance."},{"cited_title":"Ego4d: Around the world in 3,000 hours of egocentric video,","cited_arxiv_id":null,"evidence_quote":"Source of large-scale egocentric video used to build EgoAffordance."},{"cited_title":"Affordances from human videos as a versatile repre- sentation for robotics,","cited_arxiv_id":null,"evidence_quote":"Baseline for affordance-from-human-video that VLAff extends and compares against."},{"cited_title":"Partial Implementation of Max Flow and Min Cost Flow in Almost-Linear Time","cited_arxiv_id":"2407.10034","evidence_quote":"Supplies the 3D hand reconstruction (MANO parameters) used to extract grasp affordances."},{"cited_title":"Lisa: Reasoning segmentation via large language model,","cited_arxiv_id":null,"evidence_quote":"LISA, the VLM segmentation approach whose <SEG> token and decoder VLAff adapts for visual affordances."}],"review_version":1}