{"id":"b6081564-afda-4485-a318-94b25d1769fa","arxiv_id":"2501.02330","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SR-Reward replaces the environment reward with the L2 norm of a successor representation learned from demonstrations, enabling offline RL without reward labels.","lead":"SR-Reward learns a reward function from offline demonstrations using successor representations, so offline reinforcement learning can run without any hand-designed reward signal. The method competes with true-reward offline RL and behavioral cloning across D4RL and Maniskill2 benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The L2-norm SR reward is not the expert occupancy it is claimed to be; in episodic tasks it is anti-shaped (higher before success, lower at success), so Table 1 does not validate the proxy beyond the tested settings.","rationale":"The most load-bearing condition for the paper's central claim is that the learned scalar reward is a faithful proxy for expert behavior. The reader identified this as an unproven assumption; I sharpen it into a concrete mechanism. The successor-feature vector M(s,a) is an expected future feature count from (s,a), not the expert's occupancy of (s,a). The L2 norm of this vector is a nonlinear aggregate of future feature magnitudes. Because future steps are discounted and finite in episodic tasks, the norm is naturally larger for early states and smaller near terminal states. That means the reward is not merely unproven but potentially misaligned with task completion. This is a correctness risk, not just a missing theorem. The paper's own Limitations section admits the gap, and the experimental section does not include a diagnostic that would detect anti-shaping: it reports aggregate returns on tasks where offline support and non-terminal dynamics may hide the issue. I give credit for the reproducible code, the ablations on data size and quality, and the negative-sampling analysis; those support the empirical claims as reported. However, the reward-proxy concern is not resolved by Table 1 because the ManiSkill2 episodic results could be driven by the dataset's restriction to successful behavior rather than by the reward's alignment. A small exact tabular experiment would settle it. Since the reader already conditioned acceptance on the missing analysis, the verdict remains CONDITIONAL; I do not see grounds to reject given the provided empirical evidence.","tokens_in":19487,"tokens_out":13673,"duration_ms":149361,"concrete_test":"Construct a deterministic tabular gridworld: start at one end, goal at the other, one expert demonstration path of length L, and a second longer corridor also present in the dataset. Compute the exact SR under the expert policy and the resulting r(s,a)=||M(s,a)||_2. Run SparseQL (or f-DVL) with SR-Reward on this dataset. If the learned policy chooses the longer corridor or otherwise delays the terminal state, the reward is anti-shaped and the proxy fails; if it still follows the expert path, the concern is bounded. Also report the exact SR norms along the expert path to show whether they decrease toward the goal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 sets r(s,a)=||M(s,a)||_2, where M is the successor feature vector of the demonstration policy. The paper motivates this as aggregating expert visitation counts, but Appendix C only relates the occupancy measure to an expectation of M over starting states (rho(s')=E_s[rho(s)M(s,s')]); it never shows that the per-state norm of M equals or is monotone in the expert's occupancy of (s,a). In a finite-horizon MDP with terminal success, M(s,a) is the discounted future feature stream from (s,a), so under the demonstrated policy it is largest at episode onset and shrinks near terminal/success states. An RL agent maximizing this reward is thus incentivized to postpone termination, not to reproduce the expert's path. The Limitations section explicitly concedes that no convergence or optimality analysis is provided. Because the D4RL locomotion tasks are continuing and the offline datasets constrain the policy's support, the anti-shaping may not surface in Table 1; the ManiSkill2 tasks are the only episodic setting and the paper does not analyze reward values along the trajectory. The central claim therefore rests on an unvalidated surrogate, and the empirical parity does not distinguish 'good reward' from 'offline data enough to imitate expert.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SR-Reward, a reward function learned offline from expert demonstrations using a successor representation (SR). The SR vector is trained via a Bellman equation with an auxiliary next-feature prediction task, and the L2 norm of the SR vector is used as the reward. A negative-sampling loss lowers rewards for states and actions perturbed away from the demonstrations. The method is evaluated on D4RL (MuJoCo and Adroit) and ManiSkill2 tasks, combined with f-DVL and SparseQL, and compared against behavioral cloning and offline RL with true rewards. The paper reports competitive or better performance for SR-Reward in most settings and includes ablation studies on data size, data quality, and negative sampling.","tokens_in":19823,"tokens_out":9199,"duration_ms":98801,"significance":"If the empirical findings are robust, the paper offers a simple, non-adversarial reward-learning module that can be plugged into existing offline RL algorithms without requiring environment rewards. The code is released and the approach is easy to reproduce, which strengthens its practical value. However, the central theoretical motivation for the reward function is not established, and the empirical evaluation leaves open important alternative explanations for the reported success.","major_comments":[{"comment":"The claim that ||M(s,a)||_2 represents how often the expert has visited (s,a) is not supported by the derivation in Appendix C. Appendix C shows rho(s') = sum_s p(s) M(s,s'), which is an expectation of M over starting states, not a per-state equivalence. In an episodic task, M(s,a) is the expected discounted future feature stream from (s,a), which is typically largest at episode onset and decreases near success. The resulting reward is therefore anti-shaped: it incentivizes the agent to postpone termination rather than to reproduce the expert's behavior. This undermines the paper's core motivation, and the empirical results do not rule out the anti-shaping concern because the paper does not analyze reward values along trajectories in the episodic ManiSkill2 tasks. The authors should either provide a rigorous argument (or empirical evidence) that the reward is not anti-shaped in these tasks, or re-frame the contribution and add a control experiment with an intentionally anti-shaped reward to show that the reported gains are not simply an artifact of the offline RL algorithm's conservatism.","section":"Section 3.2 and Appendix C"},{"comment":"The negative-sampling hyperparameters (beta and sigma) were selected using a grid search on StackCube, as shown in Appendix G, and the selected configuration is used for the main StackCube result in Table 1. This means the reported StackCube performance is an in-sample optimum rather than an unbiased estimate of the method's performance under a fixed hyperparameter choice. The paper should report results with a separate validation split, or at least provide sensitivity analyses for all benchmark environments, to allow a fair assessment of the method's robustness.","section":"Section 4.1 and Appendix G"},{"comment":"The evaluation protocol saves the checkpoint with the highest mean return over 25 evaluation rollouts during training and then reports that checkpoint's performance on 50 fresh rollouts. This is a best-of-training checkpoint selection that can inflate reported performance relative to a fixed training budget. The concern is especially relevant for the ManiSkill2 comparisons against behavioral cloning, which does not use such a selection protocol. The authors should also report the performance of the final checkpoint or an average over the last few checkpoints, or otherwise justify that the selection protocol does not alter the main conclusions.","section":"Section 4.1"},{"comment":"The Limitations section explicitly states that no convergence or optimality analysis is provided for using the L2 norm as a reward. Combined with the anti-shaping property discussed above, the empirical parity in Table 1 does not distinguish a well-designed reward proxy from a scenario where the offline dataset already contains enough signal for imitation. The paper would be substantially strengthened by a direct investigation of what the learned reward actually encodes, e.g., by plotting SR-Reward values along successful and unsuccessful trajectories in an episodic task.","section":"Section 7"}],"minor_comments":[{"comment":"The text refers to \"covariant shift\" in the discussion of behavioral cloning; this should be \"covariate shift.\"","section":"Section 5"},{"comment":"The notation N(0, beta) uses beta as the standard deviation of the Gaussian noise; the standard convention would be N(0, beta^2) if beta is the variance, so the authors should clarify the exact parameterization.","section":"Section 3.3"},{"comment":"Line 22 builds batches by concatenating negative samples into the replay buffer for the RL update; this procedure is described in the text but the pseudocode is a little terse. A brief explanatory sentence in the caption would improve readability.","section":"Algorithm 1"},{"comment":"The heading contains a typo: \"Ardoit Door\" should be \"Adroit Door.\"","section":"Appendix F"},{"comment":"The table header groups f-DVL and SparseQL columns but the grouping is not visually obvious; it would be clearer to repeat the algorithm names above the True Reward and SR-Reward subcolumns.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published in TMLR, but as a reviewer of the arXiv version, I would ask for major revisions if it were under initial review. The central issue is that the proposed reward function is not a visitation count despite the paper's claims, and the anti-shaping property in episodic tasks is a concrete theoretical concern that the current empirical evidence does not address. The authors should be given the opportunity to add the missing analysis and control experiments rather than being rejected outright, since the method is simple, reproducible, and shows promising results on several benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mahdi,\n\nQuick take on arXiv:2501.02330 (SR-Reward). The paper's real contribution is a practical reward module: learn successor features from offline demonstrations, use the L2 norm as a dense reward, and add a negative-sampling loss to suppress OOD overestimation. It works across D4RL locomotion, Adroit, and ManiSkill2, roughly matching true-reward offline RL and beating BC on the harder manipulation tasks. The method is simple to bolt onto existing offline RL algorithms, code is released, and the ablations on data size/quality and on the negative sampling are useful. That is a solid empirical package.\n\nThe soft spots are real but not fatal. The 'SR norm as visitation count' story is loose: the L2 norm of successor features at (s,a) is the magnitude of the expected future feature stream, not the occupancy of (s,a). In episodic tasks with a terminal success, that stream shrinks as the agent approaches the goal, so the reward is anti-shaped (high at the start, low at success). The ManiSkill2 results suggest the offline data and the conservative negative sampling keep this from blowing up, but the paper never analyzes reward values along trajectories, so the stated mechanism is unvalidated. The Limitations section explicitly concedes the missing convergence/optimality analysis, which is honest but doesn't fix the gap.\n\nThe checkpoint protocol (selecting the best training checkpoint per seed) and the fact that the negative-sampling beta/sigma grid was run on StackCube, one of the benchmark environments, are worth flagging. The former can inflate absolute numbers; the latter makes the StackCube result somewhat optimistic. These are common practices in RL, but they mean the headlines in Table 1 should be read with a grain of salt.\n\nBottom line: if I were refereeing this, I would ask for (1) a trajectory-level plot of SR reward in an episodic task, (2) a non-best-checkpoint comparison, and (3) a more careful statement of what the norm actually encodes. None of that kills the paper; it's a useful, reusable method with broad experiments. It deserves a serious referee (and in fact already got one at TMLR). I'd bring it to a reading group as a 'maybe,' and I'd cite it if I were working on offline IRL. The central mechanism claim is overstated, but the empirical contribution stands.\n\nBest.","headline":"Solid empirical offline IRL paper with a useful reward module, but the SR-norm proxy is oversold and the evaluation has a few selection/tuning soft spots.","tokens_in":20286,"tokens_out":4555,"would_cite":true,"duration_ms":46710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The L2 norm of a successor representation learned from expert demonstrations can serve as the reward signal for offline RL, matching the performance of algorithms that use the true reward.","keywords":["offline reinforcement learning","inverse reinforcement learning","successor representation","reward learning","imitation learning","negative sampling","D4RL benchmark","behavioral cloning"],"falsifier":"Build an offline dataset in which expert trajectories contain a frequently visited but task-irrelevant region, such as a corridor the expert passes through many times before reaching the goal, and compare SR-Reward-trained policies against true-reward policies; if the SR-Reward agent lingers in the incidental region and its success rate drops, the visitation-maximization assumption fails. The paper's own online HalfCheetah experiment, where SR-Reward-trained TD3 underperforms true-reward TD3 after exploring out-of-distribution states, already points to this boundary.","tokens_in":1729,"feed_emoji":"🤖","tokens_out":6202,"duration_ms":96695,"temperature":0.7,"pith_summary":"SR-Reward is a way to manufacture a reward signal for offline reinforcement learning from expert demonstrations alone, with no true reward and no environment interaction. The method learns a successor representation of the state-action pairs in the demonstrations and uses its $\\ell^2$-norm as the reward: frequently visited pairs get higher rewards. The paper argues this reward can be trained concurrently with ordinary TD-based offline RL algorithms and reports that on D4RL locomotion, Adroit hand, and ManiSkill2 tasks it matches offline RL with the true reward and often beats behavioral cloning. A negative-sampling term suppresses overestimated rewards for out-of-distribution states near the demonstrations, keeping the learned policy conservative. The stated goal is to make offline RL usable whenever demonstrations are easier to obtain than hand-engineered rewards.","feed_headline":"Offline RL can learn from demonstrations without any reward signal","feed_subtitle":"The L2 norm of a successor representation estimates how often experts visit state-action pairs, and that suffices to train strong policies.","key_machinery":"Successor representation (SR) — a vector whose elements estimate the expected discounted future occupancy of state-action features — is learned from demonstrations using a Bellman target $\\phi(s,a) + \\gamma M(s',a')$, with $\\phi(s,a)$ the concatenation of a learned state encoder output and the action. The reward is the norm $\\|M(s,a)\\|_2$, which the paper treats as a visitation count of the expert. Two stabilizers carry the method: a magnitude-loss cap near 1, and negative sampling that perturbs demonstration pairs with Gaussian noise and regresses their reward down by an exponential distance kernel $\\exp\\left(-\\|\\phi(s,a)-\\phi(\\tilde{s},\\tilde{a})\\|_2/\\sigma^2\\right)$, injecting conservatism for out-of-distribution data.","core_discovery":"The central claim is that the $\\ell^2$-norm of an action-augmented successor representation learned offline from expert demonstrations is itself a usable reward for offline RL: maximizing it over the learned policy amounts to maximizing the expert's state-action visitation, and this surrogate objective yields policies competitive with those trained on the environment's true reward. The paper estimates successor features via the Bellman equation, concatenating learned state features with the action, and converts the vector to a scalar by its norm. Because the reward is decoupled from the policy, it can be trained with TD-based RL algorithms without adversarial optimization, and it is bounded by a magnitude penalty and made conservative by negative sampling with a Gaussian decay kernel. The paper supports this with experiments on D4RL and ManiSkill2, including data-size and data-quality ablations, and shows that the policy trained from SR-Reward acts similarly to one trained from the true reward.","pith_inferences":["A natural extension is to learn SR-Reward from two datasets, expert and non-expert, and use the ratio of SR norms as a distribution-matching reward; the paper's appendix already establishes SR as a proxy for occupancy measure, so this would bypass adversarial discriminators.","The visitation-count interpretation predicts that demonstrations padded with redundant, task-irrelevant loops will distort the reward landscape; an ablation that deliberately adds such loops would test whether SR-Reward can ignore incidental high-frequency states.","The authors' online HalfCheetah result implies SR-Reward's utility is bounded by how well the demonstration coverage matches the region the learner must explore; a testable extension is to combine SR-Reward with an explicit uncertainty penalty for far out-of-distribution states.","Because the reward is a function of state and action only, it could be transferred across agents with different dynamics, but the SR must be relearned whenever the transition dynamics change; this transfer condition is not examined in the paper."],"forward_implications":["Offline RL can be applied to demonstration datasets that contain no reward annotations, as in the ManiSkill2 tasks, without changing the underlying RL algorithm.","Any TD-based offline RL algorithm can, in principle, substitute its reward stream with SR-Reward, since the reward module trains on the same replay buffer with only the next action added.","Because the reward carries a conservative bias from negative sampling, learned policies stay near the demonstrated behavior, which is beneficial when consequences far from the data are unknown.","The reward remains informative when demonstrations are few or mixed with suboptimal data, since performance degrades at the same rate as with the true reward in the ablation studies.","SR-Reward outperforms behavioral cloning on harder manipulation tasks, suggesting that bootstrapping over future states helps where copying actions does not."],"supporting_citations":[{"why":"Introduces successor representation and its Bellman recursion, the formal object the reward is built from.","marker":"Dayan, 1993"},{"why":"Shows the norm of SR counts state visitation and supplies the feature-normalization and auxiliary-task design used here.","marker":"Machado et al., 2020"},{"why":"Extends SR to learned features with an auxiliary reconstruction task, grounding the continuous-state architecture.","marker":"Kulkarni et al., 2016"},{"why":"Provides the negative-sampling idea and the VINS baseline the paper compares its kernel against.","marker":"Luo et al., 2020"},{"why":"Frames imitation as state-action distribution matching, the objective SR-Reward approximates via occupancy.","marker":"Ho & Ermon, 2016"},{"why":"Provides the D4RL datasets and normalized-return procedure used in all main experiments.","marker":"Fu et al., 2020"},{"why":"Supplies ManiSkill2 datasets without rewards, the key test of reward-free offline RL.","marker":"Gu et al., 2023"},{"why":"f-DVL, one of the two offline RL algorithms whose reward is replaced by SR-Reward.","marker":"Sikchi et al., 2023"},{"why":"SparseQL, the other offline RL algorithm used and the one used in ablations.","marker":"Xu et al., 2023"}],"fun_headline_variants":["Successor representation norms turn demonstrations into rewards","Learn rewards from offline demos without explicit reward signals","SR-Reward: take the path more traveled as your reward signal","Offline RL gets reward from successor representation norms"],"cache_read_input_tokens":22528,"weakest_assumption_plain":"The reward proxy assumes that maximizing how often the expert's states and actions are re-visited is the same as solving the task, even though the paper gives no proof of this equivalence.","fun_headline_variants_meta":{"raw":{"variants":["Successor representation norms turn demonstrations into rewards","Learn rewards from offline demos without explicit reward signals","SR-Reward: take the path more traveled as your reward signal","Offline RL gets reward from successor representation norms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3315,"prompt_tokens":934,"completion_tokens":2381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2329}},"tokens_in":550,"tokens_out":2381,"duration_ms":15802,"temperature":1.0,"reasoning_tokens":2329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:42.857041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an offline dataset in which expert trajectories contain a frequently visited but task-irrelevant region, such as a corridor the expert passes through many times before reaching the goal, and compare SR-Reward-trained policies against true-reward policies; if the SR-Reward agent lingers in the incidental region and its success rate drops, the visitation-maximization assumption fails. The paper's own online HalfCheetah experiment, where SR-Reward-trained TD3 underperforms true-reward TD3 after exploring out-of-distribution states, already points to this boundary.","supporting_citations":[{"cited_title":"Improving generalization for temporal difference learning: The successor representation","cited_arxiv_id":null,"evidence_quote":"Introduces successor representation and its Bellman recursion, the formal object the reward is built from."},{"cited_title":"Learning self-correctable policies and value functions from demonstrations with negative sampling","cited_arxiv_id":null,"evidence_quote":"Provides the negative-sampling idea and the VINS baseline the paper compares its kernel against."},{"cited_title":"Generative adversarial imitation learning","cited_arxiv_id":null,"evidence_quote":"Frames imitation as state-action distribution matching, the objective SR-Reward approximates via occupancy."},{"cited_title":"Dual rl: Unification and new methods for reinforcement and imitation learning, 2023","cited_arxiv_id":null,"evidence_quote":"f-DVL, one of the two offline RL algorithms whose reward is replaced by SR-Reward."}],"review_version":1}