{"id":"bfee43b2-9ed8-437b-90b9-d7c18cf810e2","arxiv_id":"2608.09853","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"RynnValue shows that timestamp-derived temporal distance can replace preference labels for robotic value models, slightly beating a preference-supervised baseline on RBM-EVAL-OOD.","lead":"RynnValue is a language-conditioned reward model that predicts how many seconds remain until a robot task is done, using timestamps instead of human preference labels. The authors report better trajectory ranking than preference-trained models and improved real-world policy success.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD headline rests on an unverified exclusion of RBM-EVAL-OOD from training; without an overlap audit, 0.675 vs 0.655 may reflect leakage.","rationale":"The reader's weakest_assumption identified exactly this issue, and I agree it is the most load-bearing. The benchmark claim is the centerpiece of the paper: a preference-free value model surpassing preference-supervised SOTA while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Every part of that sentence depends on the evaluation data being unseen. The paper's training mixture is unusually broad and includes 693,037 OXE episodes, which is precisely the kind of corpus that could overlap with a public benchmark built from institution-specific robot data. Because no exclusion audit is documented, the numbers in Table 2 cannot be distinguished from leakage. An overlap audit is a single, well-defined check that would settle the issue. Other concerns (statistical significance, shaping-coefficient sensitivity, history-dependent potentials) are secondary: they would weaken the reward-interface claims or require error bars, but they do not undermine the central benchmark claim as directly. If the audit shows no overlap, the main claim stands; if it shows overlap, the headline generalization claim is false. Therefore the appropriate verdict remains CONDITIONAL, pending that audit.","tokens_in":20679,"tokens_out":8685,"duration_ms":87399,"concrete_test":"Perform a systematic overlap audit between the RynnValue training mixture and RBM-EVAL-OOD. For each of the 976 evaluation trajectories, compute exact frame hashes and CLIP/ViT embeddings of sampled frames and compare against all training clips (at minimum the OXE 693,037 episodes and RoboMIND subsets); also compare metadata such as dataset IDs and episode names. If any eval trajectory has an exact or near-duplicate in the training mixture, remove it and recompute Table 2. Independently, publish the exact versioned OXE split and dataset ID list used so the exclusion can be verified externally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RynnValue's central claim is zero-shot out-of-distribution trajectory ranking: trained without preference labels, it surpasses the preference-supervised SOTA on RBM-EVAL-OOD. For that claim to hold, the six evaluation datasets—USC Franka, USC Koch, USC Trossen, USC xArm, MIT Franka, and UTD SO101—must be absent from the training mixture. The paper never demonstrates this. Table 1 lists 693,037 Open X-Embodiment episodes plus large public corpora (AgiBot, EgoDex, Galaxea, InternData-A1, RDT, RoboCOIN, RoboMIND, RoboTwin, Soft-FOLD) totaling over 7,000 hours, but there is no exclusion procedure, no dataset-ID cross-check, and no trajectory-overlap audit. Robometer's RBM-EVAL-OOD is itself assembled from institution-specific robot datasets; if any of those datasets or their episodes are included in the OXE or other sources in Table 1, the model has been trained on evaluation trajectories and the 'unseen' generalization claim collapses. The same unverified assumption appears in §4.5, where real-world tasks are asserted not to appear in training. This is a missing control, not a matter of author intent, and it is load-bearing because the headline SOTA and zero-shot OOD claims depend entirely on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RynnValue, a vision-language value foundation model that predicts temporal distance (remaining time to a language-specified goal) as a replacement for preference-based or normalized-progress reward supervision. Labels are derived from timestamps after subtask segmentation and cutoff relabeling, allowing training on over 7,000 hours and roughly 3M instruction-conditioned clips without preference annotations. The model combines random temporal sampling, temporal-order shuffling, and value-isolation attention to suppress shortcuts, and uses dual distributional heads for absolute and relative temporal estimates. On the RBM-EVAL-OOD trajectory-ranking benchmark, RynnValue-8B reports average Kendall's tau_a of 0.675, surpassing the preference-supervised Robometer baseline at 0.655, and real-world manipulation experiments report improved success rates when RynnValue's temporal-distance potential is converted into a dense reward via potential-based shaping.","tokens_in":20981,"tokens_out":3288,"duration_ms":32805,"significance":"If the central empirical claims hold, the paper makes a strong contribution: it identifies a supervision target for value/reward models that is substantially cheaper than preference labels, demonstrates scaling to a large heterogeneous corpus, and provides architectural components (temporal-order shuffling, value-isolation attention) that are plausibly useful beyond this specific instantiation. The real-world RL results, if reproducible and properly controlled, would be practically important. The paper also ships model releases and a detailed appendix on data curation and experimental protocols, which supports reproducibility. However, the headline zero-shot generalization and state-of-the-art claims rest on an unverified train/eval exclusion and on a benchmark whose quality labels may coincide with the training target; these issues require resolution before the significance of the results can be fully assessed.","major_comments":[{"comment":"The paper does not document that the RBM-EVAL-OOD evaluation datasets (USC Franka, USC Koch, USC Trossen, USC xArm, MIT Franka, UTD SO101) are absent from the training mixture. Table 1 lists 693,037 Open X-Embodiment episodes and several large public corpora, but no exclusion procedure, dataset-ID cross-check, or trajectory-overlap audit is provided. Since the headline claim is zero-shot out-of-distribution performance and superiority over the preference-supervised state of the art, the absence of this audit is load-bearing. The authors should either provide a verifiable exclusion procedure, report overlap statistics, or temper the zero-shot and OOD claims.","section":"§4.2, Table 1"},{"comment":"The RBM-EVAL-OOD ranking score is the negative predicted remaining time -v_end, which is the model's own trained temporal-distance output. The benchmark's ground-truth quality labels are described as failed, suboptimal, and successful executions, which are likely completion-based. If this is so, the evaluation partly validates the training target rather than an independent notion of reward quality, and the comparison against preference-supervised baselines may be less surprising than presented. The authors should clarify what the ground-truth ordering represents and provide evidence, such as a human-preference-based evaluation or an analysis of trajectories where completion status is not aligned with temporal distance, that the benchmark measures more than the training objective itself.","section":"§4.2, Eq. (10)"},{"comment":"The real-world RL comparison uses different shaping coefficients for the two reward models: kappa = 0.1 for RynnValue and kappa = 1.0 for Robometer. Because kappa scales the entire potential-based shaping term, the observed success-rate differences confound reward-model quality with reward scale/gain. The manuscript does not report a sensitivity analysis over kappa for either model. The authors should show that the conclusions are stable across a range of kappa values or justify the asymmetric choice with a principled selection procedure.","section":"§4.5, Eq. (11), Table 4"},{"comment":"The real-world policy results are reported as success rates over only 20 trials per task, with no confidence intervals or significance tests. For example, the offline difference between RynnValue (100%) and Robometer (80%) on Bread Basket Placement corresponds to 20 vs 16 successes out of 20 trials, which is not statistically meaningful at conventional levels. The paper should report confidence intervals, exact binomial tests, or additional trials to support the claimed improvements.","section":"§4.5, Table 4"}],"minor_comments":[{"comment":"The caption contains a duplicated sentence: 'Each row shows a sequence of observations from one manipulation task' appears twice and should be removed once.","section":"Figure 6 caption"},{"comment":"The abstract and introduction state 'over 7,000 hours', but Table 1 reports episode and segmentation counts without a direct hours total. It would help to state how the 7,000-hour figure is computed from the sources.","section":"§3.1.1"},{"comment":"The sentence 'RynnValue-8B ranks first on USC Franka, USC Koch, and USC Trossen' is accurate, but the next sentence about RynnValue-4B on UTD SO101 could be phrased more clearly since the 8B model achieves 0.833 there and the 4B model 0.933.","section":"§4.2.1"},{"comment":"The confusion matrices are reported only as 'normalized diagonal margin' values in the text; it would be helpful to define this metric explicitly in the main text rather than only in the figure caption.","section":"§4.2.2, Figure 3"},{"comment":"The limitations paragraph mentions the short observation window and the minimum-time objective, but it does not discuss the potential issue of temporal-distance labels being unreliable when trajectories contain post-completion motions or when the cutoff relabeling is incorrect. A brief comment on this would be useful.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and the central idea is appealing, but the unverified train/eval overlap and the potential circularity of the RBM-EVAL-OOD metric are serious. These are fixable with additional analysis, but they are not presentation issues. I would also recommend that the editor ask the authors to report confidence intervals for the real-world success rates and to include a kappa sensitivity analysis, since the current asymmetric shaping coefficients make the comparison hard to interpret."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RynnValue is a serious engineering effort: the authors scale temporal-distance reward learning to 7,000+ hours and 3M clips without preference labels, and they get real gains on trajectory ranking and on real-world RL. The shortcut-suppression designs (random sampling, order shuffling, value-isolation attention) are well motivated, and the ablations show each component matters. The open-source release is a plus. If the headline result holds, this removes the most expensive annotation step in reward modeling.\n\nThe problem is the headline \"surpasses preference-supervised SOTA\" depends on an unverified assumption: that RBM-EVAL-OOD datasets are absent from the training mixture. Table 1 lists 693K Open X-Embodiment episodes plus other large corpora. The paper never documents an exclusion procedure for the six evaluation datasets (USC/MIT/UTD). If any of those trajectories are in OXE or another source, the 0.675 vs 0.655 comparison is contaminated. This is a missing control, not a minor omission. The stress-test note is on target.\n\nSecond, the real-world comparisons use different shaping coefficients (κ=0.1 for RynnValue, κ=1.0 for Robometer) with no sensitivity analysis. That makes the 52.5%→72.5% and 63.8%→82.5% improvements confounded: you can't tell how much is the value model vs the coefficient choice. The appendix fixes the values but doesn't justify them.\n\nThird, the paper cites TimeRewarder [18], which already proposes temporal distance as a reward signal, but never compares against it. That's the closest prior work; a comparison would clarify what RynnValue really adds.\n\nOn the benchmark itself: scoring trajectories by -v_end is the model's own output. The ground-truth labels are completion-based, so the benchmark partly validates the training target. That's not fatal—temporal distance is exactly the intended reward—but it means the Kendall's tau numbers are not an independent measure of reward quality. The progress-only baseline at 0.292 shows the method does better than normalized progress, which is credible.\n\nOverall: solid paper, load-bearing flaw in the OOD claim. The math and engineering look sound; the ablated components behave as expected. Who's this for? Anyone working on reward models or generalist robot policies. Should it go to review? Yes, but the authors must provide a training/eval overlap audit, comparative kappa sensitivity, and a TimeRewarder baseline. Without the audit, the central claim should not be taken at face value.\n\nRecommendation: send to peer review with major revision requests; the work deserves referee time.","headline":"A serious scaling recipe for temporal-distance reward, but the headline OOD claim lacks a documented train/eval separation and the real-world gains are confounded by per-method shaping coefficients.","tokens_in":21537,"tokens_out":2564,"would_cite":true,"duration_ms":23730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RynnValue replaces preference annotations with timestamp-derived temporal distance, and the resulting value model beats fully preference-supervised rivals on out-of-distribution trajectory ranking.","keywords":["temporal distance","value foundation model","robot manipulation","reward model","potential-based shaping","preference-free learning","zero-shot generalization","trajectory ranking"],"falsifier":"Match every episode in the six RBM-EVAL-OOD datasets against the training corpora listed in Table 1, for example by source dataset name, episode hash, or URL. If even a fraction of evaluation episodes occur in training, the zero-shot out-of-distribution and state-of-the-art claims would need to be recomputed, and a positive control would be to retrain on a corpus with those episodes explicitly held out to see whether the 0.675 average $\\tau_a$ persists.","tokens_in":1615,"feed_emoji":"🤖","tokens_out":3881,"duration_ms":90041,"temperature":0.7,"pith_summary":"General-purpose reward models are the bottleneck for scaling robot learning, but their usual supervision targets, preferences and normalized progress, tie a value model to trajectory-internal anchors that do not transfer across embodiments and data sources. RynnValue replaces those anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal, and shows that such labels can be read directly from timestamps, needing neither preference pairs nor progress annotations. Trained on over 7,000 hours and roughly 3M instruction-conditioned clips, the resulting value model attains an average Kendall's $\\tau_a$ of 0.675 on the RBM-EVAL-OOD trajectory-ranking suite, surpassing the 0.655 of the fully preference-supervised state of the art while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards through potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. If these results hold, temporal distance is a more scalable supervision target and a practical reward interface for generalist robot policies.","feed_headline":"Timestamps replace preference labels in a top robot reward model","feed_subtitle":"Timestamp-only training outranks preference-supervised models and lifts real-world success from 52.5 to 72.5 percent.","key_machinery":"The central object is temporal distance, defined as the directed cost-to-go from an observation to the language-specified goal, with labels derived directly from timestamps once a semantic completion cutoff is identified; it carries the argument because it replaces preferences and normalized progress with a single transferable supervision target. The supporting mechanism in the model is value-isolation attention, which prevents each group of repeated value-query tokens from attending to other groups' value predictions, together with random temporal sampling and temporal-order shuffling that break the correspondence between sequence position and task progress. At inference, predicted remaining time is negated into a potential, and potential-based shaping turns it into a dense reward for policy learning.","core_discovery":"The paper's central claim is that goal-conditioned temporal distance, rather than preferences or normalized progress, is the right scaling target for robotic value models. Trained without any trajectory-level preference labels, RynnValue predicts the remaining time to a relabeled completion cutoff along with signed temporal displacement between presented observations, using distributional heads over symlog bins and a set of shortcut-suppression designs: random temporal sampling, temporal-order shuffling, and value-isolation attention. The authors report that this recipe, not model scale, carries the gains: a 4B and an 8B variant perform comparably, and ablations show that removing shuffling, isolation, random sampling, or language supervision each degrades the average ranking score. The paper concludes that temporal distance unifies heterogeneous data under a single preference-free value interface and that it serves as a practical dense reward source for downstream online and offline reinforcement learning.","pith_inferences":["Editorial extension: the timestamp-derived temporal-distance target could plausibly be applied to human egocentric video as well as robot data, potentially yielding reward models that judge demonstration quality without preference annotations.","Editorial extension: because the relative temporal head is instruction-independent, it might be repurposed as a general temporal anomaly or regression detector during policy rollouts, even when the language goal is ambiguous.","Editorial extension: the paper's diversity-versus-volume result suggests a testable prediction that a smaller but more task-diverse corpus can beat a larger repetitive one on unseen tasks, which could guide future data-collection budgets for value models."],"forward_implications":["Temporal-distance labels can be produced automatically from timestamps, so training a general-purpose value model no longer requires constructing preference pairs or normalizing every task onto a $[0,1]$ progress scale.","A preference-free value model can outperform preference-supervised models on out-of-distribution trajectory ranking, suggesting that ordering information can be extracted from timestamp structure alone.","The same model serves as a dense reward function for both online and offline RL, improving real-world policy success without any target-domain fine-tuning of the reward model.","The scaling analysis indicates that broadening task coverage improves temporal-value generalization more than adding episodes within known tasks, pointing data collection toward task diversity.","The auxiliary language supervision, including video description, instruction-matching, and success prediction, contributes semantic grounding beyond the continuous temporal-value targets."],"supporting_citations":[{"why":"Supplies the RBM-EVAL-OOD trajectory-ranking benchmark and the main baselines, including the 0.655 preference-supervised and 0.292 progress-only averages that RynnValue must beat.","marker":"[16]"},{"why":"Open X-Embodiment contributes the largest block of training data, 693,037 episodes, which the temporal-distance recipe must convert into timestamp-derived labels.","marker":"[23]"},{"why":"Provides the potential-based shaping theorem used to convert predicted temporal distance into a dense reward without changing the optimal policy.","marker":"[22]"},{"why":"Supplies the symlog two-hot distributional regression technique used to train stable continuous value heads.","marker":"[7]"},{"why":"RynnBrain is the multimodal backbone on which RynnValue is built and which provides the language and embodied representations.","marker":"[5]"},{"why":"AgiBot is a primary training corpus, contributing 167,535 episodes and over a million instruction-conditioned segments to the mixture.","marker":"[3]"},{"why":"EgoDex contributes egocentric human video, providing the cross-embodiment data that supports the zero-shot generalization claims.","marker":"[10]"}],"fun_headline_variants":["Temporal distance beats preference labels in robot rewards","Time-based rewards outrank preference labels in robot learning","Temporal distance: a preference-free path to better robot rewards","Timestamps beat preferences: robot reward model lifts success to 72.5%","Timestamps alone outperform full preference supervision"],"cache_read_input_tokens":23552,"weakest_assumption_plain":"The claim depends on the assumption that none of the six RBM-EVAL-OOD evaluation datasets appear in the training mixture; the paper asserts they are unseen but does not document an exclusion procedure for them.","fun_headline_variants_meta":{"raw":{"variants":["Temporal distance beats preference labels in robot rewards","Time-based rewards outrank preference labels in robot learning","Temporal distance: a preference-free path to better robot rewards","Timestamps beat preferences: robot reward model lifts success to 72.5%","Timestamps alone outperform full preference supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001436,"raw_usage":{"total_tokens":5822,"prompt_tokens":1011,"completion_tokens":4811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":4731}},"tokens_in":627,"tokens_out":4811,"duration_ms":32093,"temperature":1.0,"reasoning_tokens":4731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:27:26.520857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Match every episode in the six RBM-EVAL-OOD datasets against the training corpora listed in Table 1, for example by source dataset name, episode hash, or URL. If even a fraction of evaluation episodes occur in training, the zero-shot out-of-distribution and state-of-the-art claims would need to be recomputed, and a positive control would be to retrain on a corpus with those episodes explicitly held out to see whether the 0.675 average $\\tau_a$ persists.","supporting_citations":[{"cited_title":"Open X-Embodiment: Robotic learning datasets and RT-X models","cited_arxiv_id":null,"evidence_quote":"Open X-Embodiment contributes the largest block of training data, 693,037 episodes, which the temporal-distance recipe must convert into timestamp-derived labels."},{"cited_title":"Policy invariance under reward transformations: Theory and application to reward shaping","cited_arxiv_id":null,"evidence_quote":"Provides the potential-based shaping theorem used to convert predicted temporal distance into a dense reward without changing the optimal policy."},{"cited_title":"Stop regressing: Training value functions via classification for scalable deep RL","cited_arxiv_id":null,"evidence_quote":"Supplies the symlog two-hot distributional regression technique used to train stable continuous value heads."},{"cited_title":"AgiBot World Colosseum: A large-scale manipulation platform for scalable and intelligent embodied systems","cited_arxiv_id":null,"evidence_quote":"AgiBot is a primary training corpus, contributing 167,535 episodes and over a million instruction-conditioned segments to the mixture."}],"review_version":1}