{"id":"9f6c0b3b-1c12-482f-a7ef-0c4c3e87e51d","arxiv_id":"2501.06605","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RoboHorizon couples LLM-generated staged rewards with key-horizon multi-view masked autoencoders to train world-model-based RL policies for long-horizon manipulation.","lead":"The paper proposes RoboHorizon, a system that combines LLM-generated dense rewards, key-horizon multi-view representation learning, and a world model to improve long-horizon robotic manipulation in simulation. It reports higher success rates than several model-based RL baselines on RLBench and FurnitureBench tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim hinges on unverified GPT-4o reward code; no reproducibility or correctness check is provided.","rationale":"The reader's verdict identifies the LLM-dependent reward generation as the weakest assumption; I agree. The paper's central claim is an empirical performance advantage, and the ablation explicitly shows that LLM-generated rewards are a necessary component. Yet the paper gives no evidence that the reward code was produced autonomously, no error analysis, and no reproducibility procedure. The appendix provides one example, but the presence of custom condition objects indicates substantial environment engineering outside the LLM. The inconsistency between the abstract's 23.35% and Section 4.1's 25.35%, plus the absence of error bars, reinforces the need for caution but are secondary to the reward-generation dependency. The proposed concrete test directly targets whether the headline numbers can be reproduced without manual reward fixing. If they can, conditional acceptance is appropriate; if not, the claim is not supported as stated. Therefore the verdict should remain CONDITIONAL.","tokens_in":18985,"tokens_out":9059,"duration_ms":89103,"concrete_test":"Independently re-run the two-stage LLM reward generation (Appendix A.1) for all 13 tasks using (a) GPT-4o with a fixed snapshot and (b) an open-weight LLM, with no manual editing, and verify that each generated reward function compiles and produces reward values that align with task success (e.g., by checking that reward is higher for successful than failed rollouts). Then train RoboHorizon under each condition and compare success rates to Table 1. If either condition fails to compile or success rates drop significantly (e.g., >10 percentage points on the long-horizon average), the reported results depend on manual reward engineering or a specific proprietary LLM version.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that 'only GPT-4o reliably generates correct plans and rewards for all tasks,' but the paper provides no quantitative evidence of this reliability: no error count, no manual-correction log, and no analysis of how reward-code failures were detected or fixed. Appendix A.1 shows only one successful example. The reward code uses environment-specific condition objects (e.g., _opened_cond_box, _grasped_cond_shoe1, success_sensor), which appear to be hand-added to the environment; the LLM is not generating these definitions. If the generated rewards are misaligned, the world-model training objective (Eq. 1 reward prediction) and the RL policy both optimize the wrong signal, and the headline improvement would not transfer to new tasks or even to a re-run with a different GPT-4o version. The paper states that other LLMs fail, so the method is not robust to LLM choice, and no safety check or fallback is defined. Because the reported gains are large and the ablation (Fig. 6) shows that removing LLM rewards substantially reduces performance, the entire empirical claim is conditioned on the correctness of this unverified component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Recognize-Sense-Plan-Act (RSPA) pipeline and RoboHorizon, a model-based visual reinforcement learning system for long-horizon robotic manipulation. RoboHorizon combines LLM-generated dense reward structures (Recognize), key-horizon multi-view masked autoencoder representation learning (Sense), an RSSM world model (Plan), and actor-critic policy learning (Act). Experiments on RLBench and FurnitureBench report large success-rate improvements over TCN+WM, CLIP+WM, MAE+WM, MWM, and MV-MWM baselines. The paper claims absolute percentage-point gains of 25.35% on four short-horizon tasks and 29.23% on nine long-horizon tasks, with the abstract citing 23.35% for the short-horizon improvement.","tokens_in":19292,"tokens_out":9801,"duration_ms":77913,"significance":"If the reported results are reproducible, RoboHorizon would be a practical demonstration that LLM-generated rewards can be integrated into a model-based visual RL pipeline to substantially improve long-horizon manipulation performance while reducing manual reward engineering. The key-horizon multi-view representation learning is a reasonable extension of MV-MWM and could be useful beyond this paper. However, the central empirical claim is currently undermined by missing statistical uncertainty, contradictory reported numbers, and an unquantified reliance on GPT-4o for reward code generation. These issues are fixable, but they must be addressed before the results can be considered reliable.","major_comments":[{"comment":"The abstract reports a 23.35% improvement on short-horizon tasks, while the introduction and Section 4.1/Table 1 report 25.35%. The correct figure from Table 1 is 25.35 percentage points (75.58 vs. 50.23), so the abstract is numerically wrong. Moreover, describing this as a 25.35% 'improvement' is misleading: it is an absolute percentage-point difference; the relative improvement over MV-MWM is about 50%. Please report both metrics and correct the abstract.","section":"Abstract, Section 4.1, Table 1"},{"comment":"The claim that 'only GPT-4o reliably generates correct plans and rewards for all tasks' is made without any supporting evidence: no error counts, no manual-correction log, no verification protocol, and no discussion of how misaligned reward code was detected or fixed. The example reward code in Appendix A.1 depends on environment-specific condition objects (e.g., `_opened_cond_box`, `_grasped_cond_shoe1`, `success_sensor`) whose definitions are not generated by the LLM. Because the ablation in Fig. 6 shows that removing the LLM rewards substantially degrades performance, the central result is contingent on an unverified component. Please provide a quantitative reliability analysis of the LLM reward generation, including failure cases and any manual interventions.","section":"Section 3.2, Appendix A.1, Fig. 6"},{"comment":"The main text states that 'For all tasks, 50 expert demonstrations are provided for each camera view,' but Appendix A.2 specifies that long-horizon tasks use 100 demonstration trajectories per camera view. This discrepancy is critical because if RoboHorizon and the baselines receive different numbers of demonstrations, the comparison is confounded and the 29.23% long-horizon gain could be partly due to data quantity. Please clarify the actual numbers and ensure all methods are evaluated with the same demonstration budget.","section":"Section 4 'Environmental Setup', Appendix A.2"},{"comment":"The comparison of SPA-driven baselines with LLM-generated stepwise rewards is presented only in Fig. 5, with no numerical table, error bars, or statistical tests. It is unclear whether all baselines used the same LLM-generated reward code and whether the staged rewards are compatible with their architectures. The claim that these baselines 'fail to surpass' RoboHorizon is not supported by any reported uncertainty or significance measure. Please provide a table with means and standard errors.","section":"Section 4.1 and Fig. 5"},{"comment":"The claim of outperforming 'state-of-the-art visual model-based RL methods' is too broad given the baseline set: all compared methods are from the MWM/MV-MWM family or simple representation-learning world-model combinations. No comparison is made with other recent long-horizon model-based RL or LLM-planning methods, such as Plan-Seq-Learn (cited in the paper). Please either narrow the claim or add relevant baselines.","section":"Section 4.1"}],"minor_comments":[{"comment":"There is an unresolved citation placeholder 'PyRep[?]' in the data collection description.","section":"Appendix A.2"},{"comment":"The table header includes an 'Average' column, but the rows appear to report only 'Short Avg.' and 'Long Avg.' with no overall average; please clarify the table formatting.","section":"Table 1"},{"comment":"The appendix provides prompts for only one task; for reproducibility, please include the prompt templates used for all 13 tasks or explain how they are generated from a common template.","section":"Appendix A.1"},{"comment":"The notation uses 'm' both for the number of keyframes and for the mask ratio; please disambiguate these two uses.","section":"Section 3.3"},{"comment":"The paper does not mention whether the LLM-generated rewards are unit-tested offline before being used in RL training; a simple test of the reward code against the environment would be a valuable addition and would partly address the verification concern.","section":"Section 3.2"},{"comment":"Minor typos: 'efficientlyplan' in the abstract and 'an round table' in Appendix A.4.","section":"Abstract and Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The abstract and introduction disagree on the headline number, which is a copyediting issue, but the more serious problems are the missing uncertainty quantification and the unverified LLM reward generation. The authors should also consider releasing code and the full set of LLM prompts to make the results reproducible, especially because the method's name is not associated with a public repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible systems paper combining known pieces in a sensible way, and the key-horizon multi-view MAE is a genuine new integration. But the headline numbers are sloppier than they look and the LLM reward generation is more scaffolded than the 'Recognize' framing implies. Worth refereeing, but needs revision.\n\nWhat's actually here: an RSSM world model that trains a policy on LLM-generated staged reward code, with a masked autoencoder that learns from keyframes across multiple views. The key-horizon view-and-tube masking is new, and the experiments are broad: 4 short- and 6 long-horizon RLBench tasks plus 3 FurnitureBench tasks, RGB-only, 50-100 demos, and a sensible baseline set including TCN, CLIP, MAE, MWM, and MV-MWM. The strongest part is Fig. 5, where they also give the baselines LLM stepwise rewards; that control shows the staged rewards and the key-horizon representation each contribute. The ablation in Fig. 6 supports the same conclusion, though task-wise the pattern is mixed.\n\nThe soft spots are real but not fatal. First, the numbers: abstract says 23.35% improvement on short-horizon tasks, the intro and Table 1 say 25.35%, and the table's 'improvement' is an absolute percentage-point difference, not a relative gain. There are no standard deviations or confidence intervals over the 5 seeds, so I can't judge whether task-level gaps are significant. Second, the LLM reward code in Appendix A.1 calls condition objects like `_opened_cond_box`, `_grasped_cond_shoe1`, and `success_sensor`. Those have to exist in the environment already; the LLM is assembling code around hand-added predicates, not generating the reward function from scratch. That's a legitimate design, but the paper never says how much per-task environment engineering is involved, and it undercuts the 'Recognize' narrative. Third, everything sits on GPT-4o: the paper says other LLMs fail, gives no failure analysis, no verification step, and no code. That's a reproducibility and robustness risk.\n\nWho's this for? People doing model-based RL for manipulation and anyone interested in grafting LLM reward generation onto world models. It's a solid incremental contribution, not a paradigm shift. I'd send it to peer review, but the authors should fix the percentage discrepancy, report variance, clarify the environment scaffolding, and show the full prompt set or release code.","headline":"Credible integration of LLM rewards, key-horizon MAE, and world models; headline numbers and LLM dependence need tightening before I'd trust the gains.","tokens_in":19774,"tokens_out":3268,"would_cite":false,"duration_ms":30377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoboHorizon claims that LLM-generated staged rewards and key-horizon multi-view representation learning let a world model handle long-horizon manipulation better than model-based visual reinforcement learning baselines.","keywords":["long-horizon manipulation","world model","LLM reward generation","multi-view representation learning","masked autoencoder","model-based reinforcement learning","keyframe discovery","RLBench"],"falsifier":"Run RoboHorizon on a new long-horizon manipulation task that is not among the 13 evaluated tasks, supplying only the task description and environment API to the LLM reward generator; if the generated reward code is misaligned with the success metric or success rates collapse to the baseline level, then the LLM reward-generation dependency is not a stable source of the reported gains.","tokens_in":18805,"feed_emoji":"🤖","tokens_out":6696,"duration_ms":140363,"temperature":0.7,"pith_summary":"RoboHorizon argues that the right way to make model-based visual reinforcement learning work on long-horizon manipulation is to give the robot three capabilities that standard pipelines lack: explicit recognition of the task's stages, a representation of the critical moments between stages, and a dense reward for every step. It proposes a Recognize-Sense-Plan-Act pipeline in which a pre-trained language model turns a natural-language task description into multi-stage motion plans and written reward code, a key-horizon multi-view masked autoencoder learns representations from only the frames around discovered keyframes, a world model plans over those representations, and a reinforcement-learning policy acts on imagined trajectories. On four short-horizon tasks and nine long-horizon tasks across two simulated manipulation benchmarks, the paper reports average success rates of about 75.6% on the short tasks and about 45.5% on the long tasks, beating the strongest visual model-based reinforcement-learning baseline by about 25 and 29 percentage points respectively. A reader should care because long-horizon manipulation with sparse rewards is a bottleneck for real robot deployment, and this work offers a concrete path that replaces manual reward engineering with language-model-generated structure.","feed_headline":"LLM-generated rewards boost long-horizon robot success 29%","feed_subtitle":"Staged language-model rewards and key-frame video training let a world model plan multi-step manipulation.","key_machinery":"The load-bearing mechanism is the key-horizon multi-view masked autoencoder (KMV-MAE) coupled with the Recurrent State Space Model (RSSM) world model. KMV-MAE finds keyframes from near-zero joint velocity and unchanged gripper state, treats the video segment between adjacent keyframes as a key-horizon, randomly masks whole viewpoints and most patches in the remaining view, and reconstructs pixels and rewards with a Vision Transformer, so the representation focuses on stage transitions. The RSSM then turns those frozen representations into a latent dynamics model over which the policy is optimized by backpropagating actor-critic losses with an auxiliary behavior-cloning term. The LLM reward generator supplies the dense reward signal that makes both representation learning and world-model optimization feasible under sparse task feedback.","core_discovery":"The central claim is that combining LLM-generated staged dense rewards with key-horizon multi-view representation learning is sufficient to make a learned world model plan long-horizon manipulation. RoboHorizon first asks an LLM to decompose a task described in language into sub-task stages and to write reward functions for each stage, so the environment emits dense stepwise and stage-completion rewards. It then discovers keyframes in multi-view demonstrations, forms key-horizons between consecutive keyframes, and trains a masked autoencoder with view-and-tube masking to reconstruct pixels and predict rewards, capturing gripper-object interactions at the moments that matter. The resulting frozen representations feed a Recurrent State Space Model world model, which predicts future latent states and rewards and trains an actor-critic policy through imagined trajectories. In the paper's experiments, this pipeline reaches 75.58% average success on four short-horizon tasks and 45.47% on nine long-horizon tasks, outperforming the strongest model-based visual RL baseline by 25.35 and 29.23 percentage points on short and long tasks respectively.","pith_inferences":["Because the paper tests only GPT-4o for reward generation, the practical claim about 'LLM-assisted' rewards is really a claim about one specific language model; a verification step would be needed before the method transfers reliably to other models or real robots.","Key-horizon multi-view masking could generalize to other video-prediction and model-based RL settings where important events are sparse in time but dense in information.","If reward code can be produced reliably from language, the bottleneck shifts to how much state information the environment exposes through its API, since the LLM can only write rewards for quantities the interface reveals.","The small discrepancy between the abstract's 23.35% and the results section's 25.35% for short-horizon improvement suggests the exact magnitude is less stable than the qualitative ordering of methods."],"forward_implications":["Model-based visual RL can be applied to multi-stage tasks by replacing sparse environmental feedback with LLM-generated per-stage dense rewards.","Key-horizon representation learning reduces the amount of video the world model must reconstruct, concentrating capacity on stage transitions where gripper-object interaction changes.","The same LLM-written reward code improves existing SPA baselines, although less than the full RSPA pipeline does, showing that staged reward structure and key-horizon representations each contribute.","The approach works on both short and long horizons, suggesting the gains are not specific to one task length.","Frozen multi-view key-horizon representations can serve as inputs and reconstruction targets for a latent dynamics model, enabling policy training through imagined trajectories."],"supporting_citations":[{"why":"Supplies the multi-view masked autoencoder and world-model architecture that RoboHorizon extends into key-horizon form.","marker":"[Seo et al., 2023b]"},{"why":"Provides the Recurrent State Space Model formulation used for latent dynamics and planning.","marker":"[Hafner et al., 2019]"},{"why":"Supplies the DreamerV2 actor-critic training and imagination procedure used for control-policy learning.","marker":"[Hafner et al., 2021]"},{"why":"Provides the keyframe discovery method based on near-zero joint velocities and unchanged gripper states.","marker":"[James and Davison, 2022]"},{"why":"Supplies the language-to-rewards two-stage decomposition that the Recognize phase adapts for staged reward generation.","marker":"[Yu et al., 2023]"},{"why":"Provides the masked autoencoder basis for the multi-view representation learning module.","marker":"[He et al., 2021]"},{"why":"Provides the frozen CLIP visual representation used as a baseline (CLIP+WM) for comparison.","marker":"[Radford et al., 2021]"}],"fun_headline_variants":["LLM rewards boost long-horizon robot success by 29%","Keyframe-aware world model with LLM rewards wins on long tasks","RoboHorizon: LLM rewards and keyframes improve robot planning 29%","Staged LLM rewards help world model beat baselines on long tasks","LLM rewards give robot world model 29% boost on long tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's central performance claim depends on a single pre-trained language model, GPT-4o, reliably generating correct, well-shaped reward code from a task description and environment API, with no verification step in the pipeline.","fun_headline_variants_meta":{"raw":{"variants":["LLM rewards boost long-horizon robot success by 29%","Keyframe-aware world model with LLM rewards wins on long tasks","RoboHorizon: LLM rewards and keyframes improve robot planning 29%","Staged LLM rewards help world model beat baselines on long tasks","LLM rewards give robot world model 29% boost on long tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000622,"raw_usage":{"total_tokens":2935,"prompt_tokens":1053,"completion_tokens":1882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1784}},"tokens_in":669,"tokens_out":1882,"duration_ms":12838,"temperature":1.0,"reasoning_tokens":1784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:56:25.635414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RoboHorizon on a new long-horizon manipulation task that is not among the 13 evaluated tasks, supplying only the task description and environment API to the LLM reward generator; if the generated reward code is misaligned with the success metric or success rates collapse to the baseline level, then the LLM reward-generation dependency is not a stable source of the reported gains.","supporting_citations":[],"review_version":1}