{"id":"0e56b9eb-66c6-422a-abb9-6edc5fbcc937","arxiv_id":"2501.06047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using an object-level map during interactive exploration improves the density and accuracy of affordance annotations and leads to better affordance prediction for pick-up and push actions.","lead":"A robot learns what it can pick up or push by exploring rooms and storing the results in a 3D object-level map. The map lets it reuse each interaction across views, giving better training data and more accurate affordance predictions than methods without a map.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Map benefit is confounded with annotation-by-confidence: the 'No Map + Seg' ablation removes both the map and the self-training labeler, so Table 1's pick-up gain may not be caused by the map.","rationale":"The reader's weakest assumption is the reliance on ground-truth segmentation masks in Section 3.3, which is an external-validity concern for real-world transfer. My concern is internal to the paper's own simulation: the ablation intended to isolate the map's contribution also removes the annotation-by-confidence mechanism, because that mechanism depends on cross-frame instance association. Thus the central claim 'the usage of an object-level map improves the quality of the affordance predictions' is not cleanly supported by the Ours-vs-No-Map+Seg comparison in Table 1. The reader did mention the self-training annotation step could introduce circularity in their rationale, but did not identify it as the weakest assumption; hence 'partial' agreement. This confound is more load-bearing than the segmentation issue because it threatens the validity of the headline result even under perfect segmentation. I recommend keeping the CONDITIONAL verdict, but with a specific required condition: add the 'Map + Seg - Confidence' ablation. If that ablation shows no map benefit, the paper should be revised to attribute the gains to the combined pipeline rather than to the object-level map alone.","tokens_in":11777,"tokens_out":8721,"duration_ms":87697,"concrete_test":"Run an additional ablation 'Map + Seg - Confidence' that uses the full object-level map and ground-truth segmentation masks exactly as in Ours, but disables annotation-by-confidence (Section 3.3, Eq. 5), so labels come only from successful and failed interactions. Compare its Table 1 pick-up F1 and Fig. 5 interaction-success curves against 'No Map + Seg' (which also lacks confidence labeling). If 'Map + Seg - Confidence' performs close to 'No Map + Seg' (e.g., pick-up F1 within 0.03), the reported map benefit is attributable to the confidence autolabeler rather than the map itself. If it retains most of the gain (e.g., F1 above 0.18), the map independently improves affordance learning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that the object-level map improves affordance prediction quality. The cleanest internal comparison, Ours vs. 'No Map + Seg' in Table 1, is confounded. Section 3.3 describes two annotation mechanisms: annotation by interaction and annotation by confidence. Annotation by confidence requires associating the same object instance across frames, which is exactly what the object-level map provides; a no-map pipeline cannot evaluate the cross-frame percentiles in Eq. (5) without instance tracking. Yet Section 4.1 defines 'No Map + Seg' only as removing the map from the RL state and using a GRU instead. It does not state whether annotation by confidence is disabled, but it must be, because there is no map to support it. Therefore Ours differs from 'No Map + Seg' in two ways: the map-based state/annotation propagation and the extra self-training signal from confident predictions. The pick-up F1 gap (0.22 vs. 0.13) could be caused by the pseudo-labels generated by the model itself, not by the map's object permanence. This is particularly dangerous because self-training can reinforce the model's own biases: the same network creates labels that are then used to retrain it, and the paper acknowledges that neural networks are overconfident and calibration could be needed. Without an ablation that keeps the map but disables confidence labeling, the central claim is not established even in simulation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an interactive exploration pipeline for learning robot-specific affordances (pick up and push) in 3D indoor scenes. A PPO-trained agent explores iTHOR living rooms while a TSDF++ object-level map tracks object instances and stores interaction outcomes. The map is used both as part of the RL state and to propagate sparse interaction labels across viewpoints. An additional 'annotation by confidence' mechanism pseudo-labels objects using the affordance network's own confident predictions. The authors compare against IntExp and two ablations (No Map + Seg, No Map + No Seg) and report that their method achieves higher pick-up affordance F1 and interaction success rates, while push performance is mixed.","tokens_in":12083,"tokens_out":3555,"duration_ms":36021,"significance":"If the central claim holds, the paper makes a useful contribution: it demonstrates that an explicit object-level map can provide object permanence during interactive exploration, yielding denser and more accurate affordance annotations than per-frame backprojection. The pipeline is modular, and the comparison includes several metrics (pixel-level IoU, object-level accuracy, interaction success, annotation rates). The authors also correctly acknowledge limitations of TSDF++ (rigid objects only) and the overconfidence of neural networks in the confidence-labeling step. However, the main comparison is confounded by the simultaneous introduction of the map and of self-training via annotation by confidence, and the push results do not support the broad claim that the map improves affordance prediction quality. The evaluation is entirely in simulation with oracle segmentation masks, so real-world transfer is not demonstrated.","major_comments":[{"comment":"The comparison between 'Ours' and 'No Map + Seg' is confounded. Section 4.1 defines 'No Map + Seg' only as removing the map from the RL state and using a GRU, but it does not state whether annotation by confidence is disabled. Since the percentile computation in Eq. (5) requires associating the same object instance across frames, a no-map pipeline cannot perform this step, so the ablation must implicitly disable it. Consequently, the pick-up F1 gain in Table 1 (0.22 vs. 0.13) could be caused by the self-training pseudo-labels rather than by the object-level map's contribution. The authors should add an ablation that keeps the map but disables annotation by confidence, or otherwise disentangle the two mechanisms, to support the paper's central claim that the map itself improves affordance prediction.","section":"§3.3, Eq. (5) and §4.1, Ablations"},{"comment":"The push-affordance results in Table 2 contradict the broad claim that the object-level map improves affordance prediction quality. For push, 'Ours' has lower Accuracy (0.61 vs. 0.76), lower Recall (0.43 vs. 0.91), and lower F1 (0.58 vs. 0.83) than 'No Map + No Seg' at the object-interaction level. The paper's explanation (precision/recall tradeoff, spherical annotations spilling over) does not resolve the fact that the full method is worse on an aggregate metric. The central claim should be restricted to pick-up, or the authors should provide an additional task-level or cost-sensitive analysis that justifies the precision-focused behavior for push.","section":"§4.3, Table 2"},{"comment":"The re-implementation of the IntExp baseline changes several factors at once: only two affordances are used instead of the original set, the environment is changed from kitchen to living room, the image size is increased from 80x80 to 128x128, and the number of parallel RL processes is halved from 16 to 8. These changes are not controlled, so the comparison does not isolate the effect of the object-level map. The authors should either justify that these changes affect IntExp and their method symmetrically, or provide a version of IntExp trained under the original settings and then evaluated on the new test scenes.","section":"§4.1, Baseline"},{"comment":"The pipeline relies on ground-truth instance segmentation masks during exploration, as stated in Section 3.3 ('the segmentation mask comes directly from the simulator'). The authors note that a separate segmentation network could be used, but they do not evaluate the effect of imperfect masks or TSDF++ tracking errors on label quality. Since all experimental claims are established under oracle perception, the extent to which the approach transfers to real-world settings remains unsubstantiated; this should be acknowledged more prominently and ideally tested with noisy masks.","section":"§3.3, Data generation and labeling"}],"minor_comments":[{"comment":"The baseline is referred to as both 'IntExp(PT)' and 'IntExp'; for consistency, the paper should use one name throughout and define it explicitly at first use.","section":"§2.4 and §4.1"},{"comment":"The caption lists 'No Map' as one of the curves, but the legend in the figure area appears to include 'No Map + No Seg' and 'No Map + Seg'. Please clarify whether 'No Map' is a typo for 'No Map + No Seg' or a separate method.","section":"Fig. 5 caption"},{"comment":"The notation '{Fi}P95' is not defined; please specify that it denotes the 95th percentile of the affordance network's predicted scores for the pixels of the object in frame Fi, and similarly for the 5th percentile.","section":"Eq. (5)"},{"comment":"The caption says 'averaged over 611 frames of the test scenes', but the table reports medians and [10th, 90th] percentiles; please clarify whether the values are medians over frames or means, and adjust the wording.","section":"Table 1 caption"},{"comment":"The sentence 'the usage of both the segmentation masks and the map have a positive impact on all metrics' is not supported by the push results in Table 2, where 'No Map + No Seg' outperforms 'Ours' in accuracy and F1. Please qualify this statement.","section":"§4.1, Ablations"},{"comment":"The phrase '2D convolutional encoders for the image-like inputs with a dimension of (128, 128)' would be clearer as 'input resolution of 128x128'.","section":"§3.2, State space"}],"recommendation":"major_revision","confidential_remarks":"The main issue for the editor is the confound between the object-level map and the annotation-by-confidence self-training mechanism. The paper's headline claim is not cleanly established by the current ablations. If the authors can add the missing ablation (map with confidence labeling disabled) and reframe the push results appropriately, the contribution would be solid for a robotics venue. The comparison to IntExp would also benefit from a more controlled re-implementation or a sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper brings a genuinely new combination to interactive affordance learning: an object-level map (TSDF++) feeding both the RL exploration state and the annotation process. Compared to Nagarajan and Grauman's frame-wise approach and SEAL's semantic 3D propagation, this is a clear step forward, and the pick-up results are encouraging. The map-based annotation propagation across viewpoints is a clean, sensible idea, and the authors are transparent about the modifications they made to the baseline.\n\nThat said, the central claim—that the object-level map is what improves affordance prediction—is not actually isolated by the experiments. The \"No Map + Seg\" ablation removes both the map and annotation-by-confidence, because confidence labeling depends on cross-frame instance association, which requires the map. So the pick-up F1 gap (0.22 vs. 0.13) could be caused by the extra self-training pseudo-labels rather than by object permanence. This is a load-bearing confound, not a nitpick.\n\nSecondary issues are real but more fixable: single training run with no significance testing, heavy reliance on simulator-provided ground-truth segmentation masks, and push results that are mixed—No Map + No Seg gets higher recall and accuracy for push in Table 2. The baseline re-implementation changes several factors, but it is disclosed, so I don't count that as a major sin.\n\nDespite the confound, the paper deserves a serious referee. The core idea is novel and worth pursuing, and the confound can be resolved with one additional ablation: keep the map but disable confidence labeling. Adding that, plus a variance estimate, would make the central claim much stronger. I would not yet treat the map's benefit as proven, but I would cite this as the first object-level-map interactive affordance pipeline.\n\nRecommendation: send to peer review, with the expectation that the authors will add the missing ablation.","headline":"A real new idea in interactive affordance learning, but the headline claim about the map is confounded with the confidence labeler and needs an extra ablation to stand.","tokens_in":12629,"tokens_out":3246,"would_cite":true,"duration_ms":30521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that an object-level map, which tracks each object instance across viewpoints, makes interactive exploration more efficient and the learned affordance predictions more accurate than labeling each frame…","keywords":["affordance learning","interactive exploration","object-level mapping","reinforcement learning","self-supervised labeling","robot manipulation","semantic mapping","instance segmentation"],"falsifier":"Run the identical exploration loop with a learned segmentation model on real RGB-D data instead of simulator-provided masks and compare pick-up F1 on the same test scenes; if the gap over the map-free baseline disappears, the reported improvements come from perfect masks rather than from the object-level map.","tokens_in":11583,"feed_emoji":"🤖","tokens_out":7584,"duration_ms":68656,"temperature":0.7,"pith_summary":"Robots that need to interact with objects must know what they can push or pick up, but those affordances depend on the robot's own body and gripper, not just on the object. This paper tries to let a robot discover those robot-specific affordances by itself: a reinforcement-learning explorer moves through a new scene, attempts interactions, and records which succeed. The proposed addition is an object-level map that remembers each object instance, so a single interaction can label that object in every later frame and the agent can see what it has already tried. The paper argues this map-based memory produces denser and cleaner training data, which in turn makes exploration more efficient and yields more accurate affordance predictions than the comparison method, especially for small pickupable objects.","feed_headline":"Object maps make robot pick-up learning five times better","feed_subtitle":"Storing each object in a map labels many frames per interaction, raising pick-up F1 from 0.04 to 0.22.","key_machinery":"The central mechanism is the object-level map: a volumetric scene representation in which each object instance is stored as its own signed-distance layer, so the robot can re-identify the same object under different viewpoints and after it has been moved. Interaction outcomes, whether a pick-up or push succeeded or failed, are attached to the instance in the map rather than to a single image location. Two labeling strategies then convert the map into training data: annotation by interaction, which labels all frames where the object appears, and annotation by confidence, which pseudo-labels additional objects when the affordance network is very confident above 0.9 or very unconfident below 0.1. The map also supplies extra states to the reinforcement-learning policy, such as occupancy, locations of interacted and non-interacted objects, and action history, which guide the agent toward objects it has not tried yet.","core_discovery":"The paper's central claim is that using an object-level map in the interactive affordance-learning loop improves both the exploration policy and the final affordance model. Instead of marking a fixed sphere around each interaction point in each frame, the pipeline annotates the object instance in the map and propagates that annotation to every frame where the instance is visible. With this mechanism, the full method reaches an F1 of 0.22 for the pick-up affordance versus 0.04 for the map-free baseline on unseen test frames, and a pick-up interaction-success F1 of 0.60 versus 0.21. For push, the map-based method reaches 0.99 precision, meaning it almost never labels non-pushable surfaces as pushable, at the cost of lower recall. The authors conclude that retaining object identity across viewpoints is the reason for the improvement.","pith_inferences":["Beyond the paper, the segmentation masks are the weakest link: if a real-world instance segmenter confuses objects or misses an object after a push, the map will store outcomes against the wrong instance, and the reported gains should diminish.","Beyond the paper, the rigid-object assumption in the map representation sets a boundary: actions like open, slice, or deform would need a different object model, not just more training data.","Beyond the paper, the confidence-threshold pseudo-labeling at 0.9 and 0.1 could be made self-calibrating; measuring the calibration of the affordance network would show whether the pseudo-labels are reliable.","Beyond the paper, the object-level map with interaction history is a reusable scene representation that could support downstream tasks such as object goal navigation, change detection, or semantic editing of the environment."],"forward_implications":["A robot can build a training dataset for its own body and gripper without human labels, so affordance predictions reflect the robot's actual interaction capabilities.","One interaction with an object can train the model on many frames of that object, which makes data collection cheaper and denser.","The learned pick-up model improves interaction success on unseen scenes, with pick-up F1 rising from 0.21 in the baseline to 0.60.","The high precision of the push model means the robot mostly avoids wasting time pushing objects that cannot move.","The same integration of mapping, exploration, and affordance learning could be extended to other rigid-body interactions such as lifting and placing."],"supporting_citations":[{"why":"Defines affordances as action possibilities, the conceptual foundation for labeling interactions as successful or unsuccessful.","marker":"[1]"},{"why":"Supplies the object-level mapping and tracking used to store interaction outcomes per object instance.","marker":"[8]"},{"why":"Provides the interactive simulation environment used for exploration and for the segmentation masks that feed the map.","marker":"[9]"},{"why":"The state-of-the-art interaction-exploration baseline that annotates frames without a map and is the main comparison.","marker":"[10]"},{"why":"Shows map-based label propagation for self-supervised learning, the strategy the annotation module adapts.","marker":"[38]"},{"why":"The proximal policy optimization algorithm used to train the exploration policy.","marker":"[43]"},{"why":"The U-Net architecture used for the affordance prediction network.","marker":"[46]"}],"fun_headline_variants":["Object maps make robot pick-up accuracy five times higher","Map-based object tracking lifts robot pick-up F1 to 0.22","Tracking objects in a map makes robot pick-up more accurate","Object map integration boosts affordance learning efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that accurate per-pixel instance segmentation masks are available during exploration, because every label stored in the map is attached through those masks.","fun_headline_variants_meta":{"raw":{"variants":["Object maps make robot pick-up accuracy five times higher","Map-based object tracking lifts robot pick-up F1 to 0.22","Tracking objects in a map makes robot pick-up more accurate","Object map integration boosts affordance learning efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000914,"raw_usage":{"total_tokens":3878,"prompt_tokens":850,"completion_tokens":3028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2961}},"tokens_in":466,"tokens_out":3028,"duration_ms":20652,"temperature":1.0,"reasoning_tokens":2961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:20.841796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical exploration loop with a learned segmentation model on real RGB-D data instead of simulator-provided masks and compare pick-up F1 on the same test scenes; if the gap over the map-free baseline disappears, the reported improvements come from perfect masks rather than from the object-level map.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines affordances as action possibilities, the conceptual foundation for labeling interactions as successful or unsuccessful."},{"cited_title":"TSDF++: A multi-object formulation for dynamic object tracking and reconstruction,","cited_arxiv_id":null,"evidence_quote":"Supplies the object-level mapping and tracking used to store interaction outcomes per object instance."},{"cited_title":"AI2-THOR: An Interactive 3D Environment for Visual AI,","cited_arxiv_id":null,"evidence_quote":"Provides the interactive simulation environment used for exploration and for the segmentation masks that feed the map."},{"cited_title":"Learning affordance landscapes for interaction exploration in 3d environments,","cited_arxiv_id":null,"evidence_quote":"The state-of-the-art interaction-exploration baseline that annotates frames without a map and is the main comparison."},{"cited_title":"Seal: Self-supervised embodied active learning using exploration and 3d consistency,","cited_arxiv_id":null,"evidence_quote":"Shows map-based label propagation for self-supervised learning, the strategy the annotation module adapts."},{"cited_title":"U-net: Convolutional networks for biomedical image segmentation,","cited_arxiv_id":null,"evidence_quote":"The U-Net architecture used for the affordance prediction network."}],"review_version":1}