{"id":"32547951-0363-4786-b826-b06d8d2a228b","arxiv_id":"2608.03753","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GORDON learns dense RL rewards from unlabeled video by embedding object-centric scene graphs, and uses the reward's temporal profile to automatically split long-horizon manipulation tasks into subtasks.","lead":"This paper presents GORDON, a method that learns reward signals for robot manipulation by viewing each scene as a graph of objects and spatial relations, and it trains policies from unlabeled video demonstrations. The authors report that using these object-graph rewards allows simulated robots to complete long sequences of manipulation steps with 74.4% average success without hand-written subtask rewards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Activity-persistence pooling rule may manufacture step structure: subtask discovery could be an artifact of weight jumps, not semantic transitions.","rationale":"The reader identifies the persistence rule as the weakest assumption, and I agree. The paper's novel contribution centers on automatic subtask discovery from the reward temporal profile; if the step structure is generated by a hard-coded weight persistence rather than by learned semantic transitions, the contribution is diminished. The concern is concrete and directly testable via an ablation. The success numbers remain meaningful because GORDON still outperforms the decomposed env reward oracle, but the interpretation of why it works changes. The baseline comparison with RoboHorizon from a different implementation is a secondary concern that also warrants caution, but the persistence issue is more central to the paper's claimed mechanism. Therefore the reader's CONDITIONAL verdict is appropriate; no verdict change is needed, pending the proposed ablation or similar evidence.","tokens_in":13205,"tokens_out":5886,"duration_ms":76715,"concrete_test":"Ablate the persistence rule on PutShoesInBox. Train the full-task reward with the original persistent activity (Eq. 2) and with a variant where active_{t,i} is recomputed every frame from Eq. (3) and resets to 0 after the object remains static for K frames (e.g., K=20). Use identical hyperparameters and the same automatic boundary detector. Measure (i) the number of discovered boundaries and their median normalized distance to simulator-annotated subtask ends for both variants, and (ii) downstream full-task success under the sequential policy framework. If the persistent variant yields many more boundaries and better alignment only because of weight jumps, the semantic discovery claim is unsupported; if the non-persistent variant still produces aligned steps and comparable success, persistence is not the driving factor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the reward profile's step-like transitions reveal stage-wise semantic boundaries, enabling automatic subtask discovery. However, Eq. (2) introduces a persistence rule: once any non-robot object becomes active, its pooling weight jumps from 1 to α=2 and remains elevated for the rest of the episode. This injects a discrete step into the pooled representation and hence into r_full each time an object first moves. With τ=0.005 (0.5% of box diagonal), even small jitter or incidental contact can permanently upweight an object, producing steps unrelated to task semantics. The reward profile in Fig. 6 is therefore expected to contain one step per moving object regardless of whether those moments correspond to real subtask boundaries. The paper neither quantifies the boundary detection threshold nor reports alignment distances with simulator-annotated subtask ends; it only shows qualitative overlap on three tasks. Because the discovered segments define subgoal states for sequential policy training, spurious boundaries translate into arbitrary intermediate targets. If the step structure is primarily a byproduct of the persistence rule rather than the learned representation, the 'automatic subtask discovery' novelty is substantially weakened. The persistence rule is never ablated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GORDON, a graph-based object-centric reward learning method that learns dense rewards from action-free video demonstrations. Each frame is converted into a graph of detected objects and spatial relations; a GNN is trained with temporal cycle-consistency and reconstruction losses; the reward is a negative latent-space distance to a demonstrated goal embedding, plus a classifier-gated terminal bonus. Activity-aware weighted pooling (Eq. 2) upweights non-robot objects that move, while masking robot nodes. For long-horizon tasks, the temporal profile of the learned reward is used to discover subtask boundaries, train subtask-specific reward models and policies, and compose them sequentially. Experiments cover one MAGICAL task and six ManiSkill3 reimplementations of RLBench tasks, reporting 74.4% average full-task success across four long-horizon tasks, +35.3 p.p. over RoboHorizon published numbers and +25.4 p.p. over a decomposed environmental-reward oracle. Ablations on PutShoesInBox analyze reward discrimination, training objectives, and the terminal bonus.","tokens_in":13533,"tokens_out":6683,"duration_ms":83560,"significance":"If validated, the paper would make a useful contribution: learning dense rewards from unsegmented videos while masking robot motion is a practical need, and automatic subtask discovery from the reward profile would extend prior work on video-only reward learning. The framework is clearly motivated, and the ablations in Tables III–V usefully isolate robot masking, the TCC+reconstruction objectives, and the learned terminal bonus. The ManiSkill3 reimplementation for faster, parallel RL training is also a practical plus, though validated only on one task. However, the central claims currently rest on two unverified pillars: the semantic validity of the discovered boundaries and the fairness of the headline comparison against RoboHorizon. Both are addressable, but they need direct evidence before the claims can be accepted.","major_comments":[{"comment":"The step-like reward profile may be substantially manufactured by the persistence rule. Eq. (2) sets the pooling weight to 1 for inactive objects and to alpha=2 for active objects, and the text states that 'once an object becomes active, we keep it active for the rest of the episode.' With tau=0.005 (0.5% of the bounding-box diagonal), any small motion or incidental contact permanently upweights an object, injecting a discrete step into the pooled representation and hence into r_full each time an object first moves. The paper does not ablate this rule (e.g., alpha=1 or no persistence), does not report the reward-gradient threshold used for boundary detection, and gives only qualitative overlap in Fig. 6 for three tasks. Since the discovered boundaries define the segmentation used to train subtask-specific policies, the 'automatic subtask discovery' claim requires quantitative evidence th","section":"Sec. III-A, Eq. (2)-(3); Sec. III-C; Fig. 6"},{"comment":"The headline improvement of +35.3 p.p. over the 'strongest learned or language-guided baseline' is computed against RoboHorizon numbers taken from Ref. [21], which were obtained on the original RLBench simulator, whereas GORDON is evaluated on a ManiSkill3 reimplementation. The paper acknowledges this as a 'semantically matched benchmark comparison,' but only the short-horizon TakeUmbrellaOut task is used to validate implementation correspondence; none of the four long-horizon tasks is matched. This cross-simulator comparison is not a controlled baseline and should not be the primary quantitative claim. Either re-run RoboHorizon in the same ManiSkill3 environments or restrict the headline improvement claim to baselines evaluated in the same experimental setting.","section":"Sec. IV-A, Table I; Sec. IV-B"},{"comment":"Several long-horizon comparisons show very large standard deviations over only five seeds. Examples: PutShoesOutBox Ours 56.2 +/- 33.1 vs. Env. Rew. (decomp.) 41.2 +/- 39.9; PutItemInDrawer S1->S2 Ours 95.0 +/- 11.2 vs. 62.5 +/- 39.9. No significance tests, per-seed paired comparisons, or confidence intervals are reported, so the average +25.4 p.p. gain over the decomposed environmental reward may be dominated by one or two seeds. Please provide per-seed results and a paired test (or more seeds) for the main full-task success comparisons.","section":"Sec. IV-A, Tables I-II"},{"comment":"The subtask discovery procedure is underspecified. 'Significant changes in the reward gradient' is never defined algorithmically; there is no threshold, no smoothing or peak-picking procedure, and no pseudocode. Moreover, the paper never reports the number of discovered subtasks M per task/demonstration, nor whether M matches the 2-3 ground-truth stages of PutItemInDrawer, PutShoesInBox, PutShoesOutBox, and EmptyContainer. Because automatic decomposition is the paper's central novelty, this information is necessary to judge whether the method genuinely discovers the intended structure.","section":"Sec. III-C"}],"minor_comments":[{"comment":"The table formatting for the XIRL and GraphIRL rows is broken (numbers run together with method names); please fix spacing and alignment.","section":"Table I"},{"comment":"The inferred boundary locations are presented only as dashed lines. A numeric comparison (e.g., mean absolute distance to GT subtask ends, or a table of detected vs. annotated boundaries) would make the qualitative claim reproducible.","section":"Fig. 6"},{"comment":"Object detections are obtained from simulator annotations. The paper should state more explicitly that the method currently assumes access to reliable object detection and discuss how this would be obtained on a real robot, since this affects the practical claim of learning from 'video demonstrations.'","section":"Sec. IV"},{"comment":"Implementation details such as GNN hidden dimensions, number of layers, optimizer, learning rates, batch size, and SAC hyperparameters are not reported. A project page without code or a completeness checklist limits reproducibility.","section":"Sec. IV"},{"comment":"The abstract says 'approximately +35 p.p.' while the body reports '+35.3 p.p.'; use a single consistent number.","section":"Abstract / Sec. IV-A"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is whether the subtask-boundary signal is an artifact of the persistence rule in Eq. (2); this must be ablated and quantified before publication. The RoboHorizon comparison should be either re-run in the same simulator or clearly excluded from the headline performance claim. If the authors address these two points and add per-seed statistics, the paper could become a solid contribution to reward learning for long-horizon manipulation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea worth a look: encode scene graphs with a GNN, learn a task-aligned embedding with TCC and reconstruction, and derive a dense reward that doubles as a segmentation signal for long-horizon tasks. The paper is clearly written and the ablations are honest. The building blocks aren't new, but the specific integration—robot-masked, activity-weighted graph pooling, and using the reward profile's gradient for subtask discovery—is. The method achieves 74.4% average success on four long-horizon tasks, which is strong if it holds up.\n\nThe main soft spot is the persistence rule in Eq. (2). Once an object becomes active, its pooling weight jumps from 1 to 2 and never goes back. That injects a step into the pooled representation and hence into the reward every time an object first moves. The paper claims these steps align with semantic subtask boundaries, but only shows qualitative agreement on three tasks. The boundary detector is under-specified; no threshold or alignment metric is reported, and the persistence rule is never ablated. So the 'automatic subtask discovery' claim rests on the assumption that first-movement events are semantic boundaries. That may be true for the tasks tested, but the paper doesn't demonstrate it.\n\nThe RoboHorizon comparison is also weaker than the headline suggests: those numbers are imported from a different paper on a different simulator. The authors acknowledge this and provide a one-task correspondence check, but it's a thin basis for a 35 p.p. claim.\n\nWhat the paper does well: the ablations that are present are informative. Robot masking improves reward discrimination, the learned terminal bonus helps, and TCC plus reconstruction beats either alone. The reward discrimination metrics (Table III) give concrete evidence. The method is reproducible in principle, though no code is released.\n\nThis deserves a serious referee. The method is coherent and the results are promising, but the subtask-discovery claim needs stronger validation: an ablation of the persistence rule, quantitative boundary alignment, and either a matched RoboHorizon comparison or a clear statement that the comparison is only indicative. I'd send it out, with major revisions expected.","headline":"A well-integrated object-centric reward method with a promising but under-validated subtask discovery mechanism.","tokens_in":13979,"tokens_out":2809,"would_cite":true,"duration_ms":30913,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a dense reward learned from unsegmented, action-free videos of object-centric scene graphs can both train policies and reveal stage boundaries in long-horizon manipulation tasks.","keywords":["robot manipulation","reward learning","graph neural networks","object-centric representation","long-horizon reinforcement learning","automatic subtask discovery","video demonstrations","temporal cycle consistency"],"falsifier":"On a task with known subtask annotations, retrain the reward model with the keep-active-forever persistence term removed (or with activity allowed to decay), then compute the overlap between inferred reward-gradient boundaries and the annotated subtask endings. If overlap collapses, the stage structure is an artifact of the persistence rule; if it survives, the subtask-discovery claim stands. A complementary check: on a task where all object movement happens early but later stages are purely relational, see whether the reward still produces steps at the relational stage boundaries.","tokens_in":13151,"feed_emoji":"🤖","tokens_out":12413,"duration_ms":123930,"temperature":0.7,"pith_summary":"Long-horizon robot manipulation is hard for reinforcement learning because a single reward for the whole task gives too little guidance. This paper claims that a reward learned from unsegmented, action-free videos can supply that guidance and more: it can also show where the task breaks into stages. The method, GORDON, turns each video frame into a graph of detected objects and their spatial relations, trains a graph neural network to embed these graphs so temporally similar states are close and object structure is preserved, and defines reward as negative distance to the demonstrated goal embedding. An activity-aware pooling step upweights objects whose state changes and masks the robot, so the reward tracks object transitions rather than arm motion or visual appearance. In long-horizon tasks the paper reports that the reward's temporal profile rises in steps aligned with subtask endings, and that using those steps to segment demonstrations and train sequential subpolicies reaches 74.4% average full-task success, about 35 percentage points above the best learned baseline and 25 above a manually decomposed oracle reward.","feed_headline":"Reward from demo videos auto-splits long-horizon tasks","feed_subtitle":"Object-graph reward hits 74.4% full-task success, beating a hand-engineered oracle by 25 points.","key_machinery":"The central mechanism is the step-like temporal profile of the full-task reward, produced by a GNN encoder plus activity-aware weighted pooling (Eq. 2). The pooling rule upweights objects that have ever become active and masks robot nodes, so latent distances to the demonstrated goal respond mainly to object-configuration changes; the persistence of activity keeps completed stages represented. Automatic subtask discovery reads significant changes in the reward gradient to identify transition frames, then uses the resulting segments to train subtask-specific encoders, rewards, and policies composed by a sequential executor.","core_discovery":"The paper's central claim is that a self-supervised, object-centric reward learned from unsegmented videos captures both goal proximity and the ordered object-state transitions that define long-horizon progress. GORDON encodes scenes as graphs of detected objects and spatial relations, trains a GNN with temporal cycle-consistency and reconstruction, masks the robot via activity-aware pooling, and uses negative latent distance to the goal as reward. In long-horizon tasks the reward's temporal profile rises in steps aligned with subtask endings, and these boundaries let the method split demonstrations and compose subtask-specific policies. The reported result is 74.4% average long-horizon full","pith_inferences":["If the step boundaries are semantic rather than artifacts, the same reward profile could be used at deployment for online stage detection and subpolicy switching, whereas the paper currently detects boundaries offline from demonstrations.","Removing the 'once active, always active' persistence rule in Eq. (2) is a direct experiment that would tell whether the steps come from object-state semantics or from the rule itself; the paper ablates the loss objectives but not this persistence mechanism.","The method depends on detector-provided object boxes, which suggests a natural next test: replacing simulator annotations with open-vocabulary detection or segmentation models to see how much reward quality degrades under noisy, unlabeled, or occluded object sets.","The reward profile could serve as weak supervision for building a reusable skill library for imitation or planning, independent of RL, since segment boundaries define reusable manipulation phases."],"forward_implications":["Action-free, unsegmented videos become a sufficient supervision source: no expert actions, no subtask labels, and no language or VLM decompositions are required.","The same reward signal serves both dense shaping for RL and a segmentation cue, so long-horizon policy learning can be decomposed into per-stage subpolicies automatically.","Robot masking plus activity weighting makes the reward robust to appearance shifts and moderate localization noise, as shown in the ablations.","Full-task rewards that suffice for early stages are not enough for later stages; the stage structure itself carries information needed for cumulative success.","The ManiSkill3 reimplementation of RLBench-style tasks preserves task semantics while cutting training steps roughly sixfold, making the long-horizon evaluation reproducible."],"supporting_citations":[{"why":"XIRL provides the embedding-based negative-distance-to-goal reward design that GORDON generalizes to object-centric graphs, and serves as a pixel-based baseline.","marker":"[4]"},{"why":"GraphIRL is the prior object-centric reward learner whose feature-level representation GORDON extends and beats; the graph-input construction follows this line.","marker":"[6]"},{"why":"ManiSkill3 is the simulation framework used for the reimplemented long-horizon tasks and fast parallel RL training.","marker":"[7]"},{"why":"Temporal cycle consistency is the self-supervised temporal alignment loss that organizes graph embeddings by task progress.","marker":"[9]"},{"why":"Universal Visual Decomposer is the image-level phase-change decomposition approach that GORDON contrasts with its object-centric stage discovery.","marker":"[18]"},{"why":"RoboHorizon is the language-guided long-horizon baseline whose published success rates anchor the main comparison.","marker":"[21]"},{"why":"EGTR inspires the attention-based graph decoder used in the reconstruction loss that preserves object class and bounding-box information.","marker":"[24]"},{"why":"MAGICAL supplies the MatchRegions 2D benchmark used as a controlled test for reward learning and downstream RL.","marker":"[25]"},{"why":"RLBench is the source of the task semantics and subtask structure that the ManiSkill3 reimplementations preserve.","marker":"[26]"},{"why":"Soft Actor-Critic is the RL algorithm used uniformly across methods with the same policy architecture and training budget.","marker":"[27]"}],"fun_headline_variants":["Object-graph rewards from videos auto-split long-horizon tasks","Self-supervised scene graphs turn demos into subtask rewards","Graph-based object rewards beat oracle in long-horizon","Video-learned object graphs give 74.4% long-horizon success"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the sharp rises in the learned reward mark genuine semantic subtask boundaries; since the activity rule keeps every object that ever begins moving permanently upweighted, a step is created each time an object first moves, and the paper does not quantify the gradient threshold or show quantitative agreement with ground-truth stage annotations beyond a few curves.","fun_headline_variants_meta":{"raw":{"variants":["Object-graph rewards from videos auto-split long-horizon tasks","Self-supervised scene graphs turn demos into subtask rewards","Graph-based object rewards beat oracle in long-horizon","Video-learned object graphs give 74.4% long-horizon success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001138,"raw_usage":{"total_tokens":4609,"prompt_tokens":836,"completion_tokens":3773,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":3697}},"tokens_in":580,"tokens_out":3773,"duration_ms":31732,"temperature":1.0,"reasoning_tokens":3697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:20:54.484606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a task with known subtask annotations, retrain the reward model with the keep-active-forever persistence term removed (or with activity allowed to decay), then compute the overlap between inferred reward-gradient boundaries and the annotated subtask endings. If overlap collapses, the stage structure is an artifact of the persistence rule; if it survives, the subtask-discovery claim stands. A complementary check: on a task where all object movement happens early but later stages are purely relational, see whether the reward still produces steps at the relational stage boundaries.","supporting_citations":[{"cited_title":"Xirl: Cross-embodiment inverse reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"XIRL provides the embedding-based negative-distance-to-goal reward design that GORDON generalizes to object-centric graphs, and serves as a pixel-based baseline."},{"cited_title":"Graph inverse reinforcement learning from diverse videos,","cited_arxiv_id":null,"evidence_quote":"GraphIRL is the prior object-centric reward learner whose feature-level representation GORDON extends and beats; the graph-input construction follows this line."},{"cited_title":"Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,","cited_arxiv_id":null,"evidence_quote":"ManiSkill3 is the simulation framework used for the reimplemented long-horizon tasks and fast parallel RL training."},{"cited_title":"Temporal cycle-consistency learning,","cited_arxiv_id":null,"evidence_quote":"Temporal cycle consistency is the self-supervised temporal alignment loss that organizes graph embeddings by task progress."},{"cited_title":"Universal visual decomposer: Long-horizon manipu- lation made easy,","cited_arxiv_id":null,"evidence_quote":"Universal Visual Decomposer is the image-level phase-change decomposition approach that GORDON contrasts with its object-centric stage discovery."},{"cited_title":"Egtr: Extracting graph from transformer for scene graph generation,","cited_arxiv_id":null,"evidence_quote":"EGTR inspires the attention-based graph decoder used in the reconstruction loss that preserves object class and bounding-box information."},{"cited_title":"The magical benchmark for robust imitation,","cited_arxiv_id":null,"evidence_quote":"MAGICAL supplies the MatchRegions 2D benchmark used as a controlled test for reward learning and downstream RL."},{"cited_title":"Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,","cited_arxiv_id":null,"evidence_quote":"Soft Actor-Critic is the RL algorithm used uniformly across methods with the same policy architecture and training budget."}],"review_version":1}