{"id":"feead347-0231-48e1-ae8a-99282ada6092","arxiv_id":"2507.08707","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SPLASH learns reward functions from suboptimal, options-level demonstrations in long-horizon adversarial tasks by ranking full downsampled trajectories, pruning pairs with score and success checks, and enforcing progress constraints.","lead":"SPLASH is a new algorithm for learning robot rewards from human demonstrations that are not expert-quality, aimed at long, adversarial games like boat capture-the-flag. It beats the D-REX baseline at tracking score and progress in simulation and shows the learned options policy works on real boats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Preference-label generation is the load-bearing step, but it is both unvalidated and internally inconsistent: Algorithm 1 appears to invert D-REX's noise-based preference direction relative to Eq. 1, and the paper never checks that higher noise yields worse CTF rollouts.","rationale":"Good-faith reading: SPLASH is a synthesis of D-REX-style automatic ranking with options-level demonstrations, full-trajectory comparisons, score/success filtering, initial-final state constraints, and temporal smoothing. The reader's CONDITIONAL verdict is appropriate: the empirical plots are suggestive but the paper lacks error bars, downstream RL policies trained on the learned reward, and released artifacts. I focused on the preference-labeling step because it is the single condition that must hold for the central claim: if the labels are wrong, every downstream loss (Eq. 1, Eq. 6, Eq. 11) is fitting noise. The reader correctly identified the D-REX monotonicity assumption as weak; I agree that score/success pruning mitigates only obvious contradictions and cannot rescue a non-monotonic noise-quality relationship. I add a sharper, more load-bearing problem: Algorithm 1's pseudocode appears to invert the D-REX preference direction relative to the notation fixed in Section III-C and Eq. 1. This is an internal inconsistency, not just a missing experiment. A reviewer cannot tell from the paper whether the implemented system labels low-noise rollouts as preferred (as the prose says) or high-noise rollouts as preferred (as the pseudocode says). This directly affects reproducibility and the interpretation of Fig. 3. I do not think this overturns the conditional verdict: the method may work as intended, and the evaluation may be correct; but the paper must fix the pseudocode, report noise-versus-score statistics, and ideally close the loop with an RL policy trained on the recovered reward. Because the reader already required conditional acceptance, my verdict is unchanged; the concern strengthens the conditions rather than moving the verdict.","tokens_in":12778,"tokens_out":12011,"duration_ms":146201,"concrete_test":"Re-run SPLASH from the current arXiv text: build pairs exactly as Algorithm 1 states with Eq. 1's convention, train Eq. 11, and plot predicted versus ground-truth returns; in the same run, tabulate mean and variance of eta(tau) for each epsilon bin in {0.5, 0.67, 0.83, 1.0}. If the printed preference direction does not reproduce Fig. 3, the paper's algorithm does not match its experiments; if the epsilon-to-score curve is non-monotonic, the automatic preference labels are unreliable even after pruning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that SPLASH's automatically generated preference dataset is correctly labeled, because Eq. 11 is trained on pairs whose labels come from the noise schedule. Two concrete problems combine here. First, D-REX's monotonicity assumption (Section III-C, Eq. 2/4) is not validated for CTF: the paper reports mean scores for the clean options-BC policy (1.89) and vanilla BC (-1.86), but never reports eta(tau) by noise bin for epsilon in {0.5, 0.67, 0.83, 1.0}. If higher-epsilon rollouts are sometimes better, the score/success filter (Algorithm 1 line 10) removes only contradictions, not mislabeled pairs that happen to pass the filter. Second, as printed, Algorithm 1 is internally inconsistent with the paper's own notation. Section III-C defines tau_i ≺ tau_j as 'tau_j is preferential to tau_i', and Eq. 1 trains the reward to prefer tau_j. Algorithm 1 loops over epsilon_i < epsilon_j (so tau_i is the lower-noise, supposedly better rollout) and adds (tau_i ≺ tau_j) when psi(tau_i) <= psi(tau_j) and eta(tau_i) <= eta(tau_j). Under the stated convention this labels the higher-noise trajectory as preferred, the opposite of D-REX's rule; if the condition were meant to filter noise-based labels, it should require eta(tau_i) >= eta(tau_j) and add (tau_j ≺ tau_i). The prose in Section IV-B describes the higher-ranked trajectory correctly, so the pseudocode appears to invert the preference direction. If the implementation follows the pseudocode, Fig. 3's positive alignment is unexplained; if it follows the prose, the paper omits the correction. Either way, the load-bearing preference-labeling step is under-specified and unvalidated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SPLASH, a preference-based inverse reinforcement learning method that extends D-REX to long-horizon, adversarial tasks with suboptimal demonstrations. The main components are options-level demonstrations with behavioral cloning, comparisons over downsampled full trajectories, automatic preference labeling from a noise schedule followed by pruning using task score and success criteria, initial-final state reward constraints, and first/second-order temporal consistency regularization. The method is evaluated on a maritime capture-the-flag simulation and with three hardware games on autonomous surface vehicles, with the claim that SPLASH significantly outperforms D-REX in reward learning from suboptimal demonstrations.","tokens_in":13127,"tokens_out":5711,"duration_ms":66302,"significance":"If the reported results are robust, SPLASH provides a useful practical recipe for reward learning in long-horizon adversarial tasks, an underexplored regime for preference-based IRL. The options-level behavioral cloning result is a concrete and potentially valuable empirical finding: the options-BC policy wins 99/100 simulated games with average score 1.89, whereas vanilla BC wins 2/100 with average score -1.86. The paper also demonstrates sim-to-real transfer of the cloned policy on physical vehicles, and the use of an open-source CTF environment supports reproducibility. However, the strength of the central claim is limited by the evaluation design: the reward is never used to train a downstream RL policy, the reported comparisons lack statistical quantification, and the score/success signal used for training also serves as the evaluation ground truth.","major_comments":[{"comment":"As printed, the preference direction in Algorithm 1 is inverted relative to the paper's own notation. Section III-C defines tau_i ≺ tau_j to mean that tau_j is preferential to tau_i, and Eq. (1) trains the reward to assign higher return to tau_j. Algorithm 1 loops over epsilon_i < epsilon_j (so tau_i is the lower-noise, supposedly better rollout) and adds (tau_i ≺ tau_j) whenever psi(tau_i) <= psi(tau_j) and eta(tau_i) <= eta(tau_j). Under the stated convention this labels the higher-noise trajectory as preferred, the opposite of D-REX's rule and of the prose in Section IV-B. If the implementation follows the pseudocode, the training labels are inverted and Fig. 3 would not reflect the intended algorithm; if the implementation follows the prose, the pseudocode must be corrected. This is a load-bearing point because Eq. (11) is trained entirely on these automatically generated labels.","section":"Algorithm 1, lines 6-11; Section III-C, Eq. (1)"},{"comment":"The evaluation in Fig. 3 is partially circular. The 'ground truth returns' axis is the same score function eta used in Algorithm 1 to prune preference pairs and in Eqs. (5)-(7) to define the success constraint psi. The training loss therefore already encourages consistency with this metric, and the pruning removes the pairs that most strongly violate it. A positive correlation on this axis is partly a self-consistency check rather than an external validation of the learned reward. Please report results against an independent ground truth — for example, human preference labels on held-out trajectory pairs, or the performance of an RL policy trained on the learned reward — and also report the alignment separately for pairs that passed the pruning filter versus those that were removed.","section":"Section V-C, Fig. 3"},{"comment":"The central monotonicity assumption of D-REX — that higher epsilon produces strictly worse trajectories — is not validated for the CTF task. SPLASH relies on this assumption to generate preference labels from the noise schedule, but the paper reports only aggregate scores for options-BC (1.89) and vanilla BC (-1.86) and never reports eta(tau) by noise bin for epsilon in {0.5, 0.67, 0.83, 1.0}. In a long-horizon adversarial game, larger noise could occasionally lead to better exploration outcomes, and the pruning filter in Algorithm 1 removes only pairs whose score/success labels directly contradict the noise-based label, not pairs that are mislabeled but happen to satisfy the filter. Please provide the per-bin score distributions and, if feasible, validate a sample of the automatically generated preference labels against human judgments.","section":"Section III-C and Section IV-B, Eq. (4)"},{"comment":"The IRL loop is never closed: the learned reward function is not used to train a downstream RL policy. The hardware experiments in Section V-D use the options-BC policy directly, so they demonstrate sim-to-real transfer of behavioral cloning, not of the reward learned by SPLASH. To support the claim that SPLASH produces rewards that are useful for RL, the paper should include at least one experiment in which a policy is trained with the SPLASH reward, compared against a policy trained with the D-REX reward and against the demonstrator's performance. Without this, the central motivation of recovering a reward for downstream policy optimization remains untested.","section":"Section V-D and Section V-C"},{"comment":"The empirical support for 'significantly outperform' is statistically thin. Fig. 3 and Fig. 4 show no confidence intervals, no multiple-seed results, and no significance tests, and Table I reports only three hardware games with a 1-1-1 record. The phrase 'significantly lower deviation' and the abstract's 'significantly outperform the state-of-the-art' are not supported by any statistical procedure. Please report means and variances over multiple seeds for the simulated experiments, add a paired test or effect-size measure for the SPLASH versus D-REX comparison, and describe how the single trajectory in Fig. 4 was selected.","section":"Section V, Figs. 3-4, Table I"}],"minor_comments":[{"comment":"The denominator in Eq. (8) reads '|τj| + |τj| − 2' and '|τj| + |τj| − 4'; this appears to be a typo for |τi| + |τj| − 2 and |τi| + |τj| − 4, and it should be corrected.","section":"Eq. (8)"},{"comment":"In Eq. (4), the condition 'if πΩBC(s) = a' uses an action variable a where an option variable ω would be consistent with the option-level notation; please clarify.","section":"Eq. (4)"},{"comment":"The success criterion for demonstrations is described as η(τ) > 0 in Section V-B, while the formal success function ψ is defined earlier; the relationship between ψ and η should be stated explicitly so that 'successful demonstration' is unambiguous.","section":"Section V-B"},{"comment":"The paper states that D-REX was trained with the same number of trajectory pairs (256,000). Given M=100 rollouts per noise level and four noise levels, the number of cross-noise pairs is 60,000 before any augmentation, so the source of 256,000 pairs should be explained (for example, whether pairs are counted per agent or with repetition).","section":"Section V-C"},{"comment":"Fig. 3's normalization of predicted returns is described only briefly; please specify the normalization procedure and whether it is applied identically to SPLASH and D-REX.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable engineering contribution but the central claim of being 'the first' approach for this task class should be checked carefully against D-REX and related preference-based IRL work on long-horizon Atari. The most urgent issue is the apparent inversion of the preference direction in Algorithm 1: if the implementation follows the pseudocode, the reported results may not reflect the algorithm described in the text, and this should be resolved before further review. The lack of any downstream RL training also makes the paper's contribution closer to reward-function fitting than to a validated IRL pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The problem is real, and the paper’s main idea is worth engaging with: extending PBIRL from short-horizon continuous control to long-horizon adversarial tasks like capture-the-flag. The options-level behavioral cloning is a sensible move, and collecting human demonstrations at the options level is a nice practical contribution. The hardware transfer experiments on the Charles River are a genuine extra, even if the sample size is tiny. I also find the full-trajectory comparison argument reasonable: for long-horizon tasks, snippet comparisons can mix incomparable subtasks, so comparing downsampled full rollouts is a plausible fix. The score/success pruning and initial-final progress constraints are also sensible add-ons.\n\nBut the stress-test note lands, and it lands on the load-bearing part of the method. Algorithm 1, as printed, appears to invert the preference direction relative to the notation in Section III-C. You loop over lower-noise τ_i and higher-noise τ_j, and add (τ_i ≺ τ_j) when η(τ_i) ≤ η(τ_j). Under the paper’s own convention, τ_i ≺ τ_j means τ_j is preferred, so you would be labeling the higher-noise trajectory as better exactly when it has a higher score. That is not pruning contradictions; it is silently relabeling based on the score. The prose in Section IV-B describes the correct D-REX-style rule, so the pseudocode is either wrong or the implementation follows the prose. Either way, a paper whose central claim depends on automatically generated preference labels cannot leave this ambiguous. Worse, the paper never validates the D-REX monotonicity assumption for CTF: there is no plot of score by noise bin, so we do not know if the noise schedule actually produces a quality ordering. If it does not, the score filter only removes outright contradictions, not mislabeled pairs that happen to pass.\n\nThe evaluation has other gaps. The learned reward is never used to train a downstream RL policy, so the IRL loop is not closed. Figures lack error bars, and the score/success signals that enter the loss are the same ones used to evaluate fit, which makes the good alignment in Fig. 3 partly by construction. The three hardware games are anecdotal.\n\nI would still send this to review, but the reviewers should insist on three things: a corrected algorithm description with a clear preference convention, a noise-versus-score validation for the CTF environment, and at least one RL policy trained on the learned reward. The paper is a plausible step forward, not a finished result.","headline":"A sensible extension of D-REX to long-horizon adversarial tasks, but the preference-labeling step is under-specified and the evaluation doesn't close the IRL loop.","tokens_in":13718,"tokens_out":2897,"would_cite":false,"duration_ms":32427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces SPLASH, a preference-based inverse reinforcement learning method that recovers reward functions for long-horizon adversarial tasks from suboptimal hierarchical demonstrations.","keywords":["inverse reinforcement learning","preference-based IRL","suboptimal demonstrations","hierarchical demonstrations","options framework","long-horizon tasks","adversarial tasks","maritime capture-the-flag"],"falsifier":"Roll out the cloned options policy under the exact training noise schedule and compare mean ground-truth score per noise bin; if mean score does not strictly decrease as $\\epsilon$ increases, the automatic preference labels are wrong and the recovered reward cannot be trusted. A second check is to compute whether SPLASH's predicted returns rank-order with ground-truth returns on held-out trajectories better than D-REX's; if not, the claimed advantage is absent.","tokens_in":12552,"feed_emoji":"🚤","tokens_out":11638,"duration_ms":122837,"temperature":0.7,"pith_summary":"SPLASH is a preference-based inverse reinforcement learning method for recovering the reward function of a long-horizon adversarial task from demonstrations that are suboptimal and collected at the level of high-level behaviors, called options. The paper's central claim is that this is the first IRL approach designed for suboptimal demonstrations of long-horizon and adversarial tasks, and that it significantly outperforms the existing D-REX method at reward learning in the maritime capture-the-flag setting. If the claim is right, reward learning becomes usable for field robotics problems where expert demonstrations are unavailable and where handcrafted or sparse rewards are unreliable. The paper supports the claim with simulated capture-the-flag experiments, ablations, and a sim-to-real deployment of the learned options-level policy on unmanned surface vehicles.","feed_headline":"SPLASH learns accurate rewards from suboptimal long-horizon demos","feed_subtitle":"Maritime capture-the-flag tests show the learned reward tracks score and transfers from simulation to real boats.","key_machinery":"The load-bearing object is the SPLASH loss function, $L(\\theta) = L_{\\mathrm{PBIRL}}(\\theta) + \\lambda_{\\mathrm{IF}} L_{\\mathrm{IF}}(\\theta) + L_{\\mathrm{dR}}(\\theta)$, which combines three mechanisms. $L_{\\mathrm{PBIRL}}$ is a binary cross-entropy preference loss over pairs of full, downsampled trajectories labeled by a D-REX-style noise schedule; pairs are kept only when the noise ranking agrees with the task's score and success criteria. $L_{\\mathrm{IF}}$ is an initial-final state comparison that encodes the zero-sum structure of adversarial games by requiring the final reward of a winning trajectory to lie above its initial reward and the final reward of a losing trajectory to lie below it. $L_{\\mathrm{dR}}$ penalizes large first and second differences of reward along a trajectory, smoothing the learned reward. Around this sits the options-level demonstration scheme: trajectories are sequences of high-level behaviors such as attack, defend, and retreat, with predefined control primitives, which makes behavioral cloning sample-efficient enough to generate rollouts for preference labeling.","core_discovery":"On its own terms, the paper establishes that a preference-based IRL recipe can turn suboptimal hierarchical demonstrations into a reward function that tracks both task performance and moment-to-moment progress in an adversarial game. The paper presents SPLASH as the first inverse reinforcement learning approach designed to learn from suboptimal demonstrations of long-horizon and adversarial tasks. Its objective is a pairwise preference loss over full downsampled trajectories, with labels generated by a noise schedule and filtered by score and success criteria, plus an initial-final state comparison that forces winning trajectories to end above their start and losing trajectories to end below it, plus temporal consistency regularization. In the maritime capture-the-flag domain, SPLASH's predicted returns align with ground-truth returns better than D-REX's, its per-step reward marks captures by both teams, and the options-level policy learned from fifty demonstrations transfers to real unmanned surface vehicles.","pith_inferences":["Inference: because SPLASH prunes pairs whose noise-based labels contradict measured score or success, it may tolerate some violations of the noise-degradation assumption; deliberately running a non-monotonic noise schedule would quantify that tolerance.","Inference: the initial-final state comparison loss encodes zero-sum progress in a compact way that should port to other competitive tasks with a defined win, loss, or draw condition.","Inference: options-level data collection shifts the human burden from fine-grained low-level control to choosing high-level behaviors, which could make demonstration collection cheaper and safer on physical systems.","Inference: a natural next test is to close the loop by training an agent on SPLASH's learned reward and measuring wins against the baseline, rather than only measuring how well the reward tracks ground-truth returns."],"forward_implications":["SPLASH extends reward learning from suboptimal demonstrations to long-horizon and adversarial tasks, where snippet-based ranking fails because partial trajectories can demonstrate different subtasks.","The reward functions learned by SPLASH extrapolate beyond the performance of the demonstrations, so a downstream reinforcement learning agent should be able to exceed the demonstrator's level.","Options-level demonstration collection is dramatically more sample-efficient than low-level behavioral cloning: fifty options-level demonstrations produced a capture-the-flag policy that won 99 of 100 simulated games, while low-level cloning won 2 of 100.","The score and success filters plus temporal regularization clean the reward signal enough to show captures and losses as clear rises and falls during a game.","The options-level policy that carried the learned reward in simulation also competed on real unmanned surface vehicles, supporting sim-to-real transfer."],"supporting_citations":[{"why":"Defines the ranked-trajectory preference objective and the snippet-based progress assumption that SPLASH replaces with full, downsampled trajectory comparisons.","marker":"[20]"},{"why":"Supplies the automatic noise-injection preference labeling that SPLASH inherits and uses as its baseline for reward learning from suboptimal demonstrations.","marker":"[22]"},{"why":"Represents the alternative noise-performance characterization approach that fits an assumed degradation curve, which SPLASH explicitly avoids.","marker":"[23]"},{"why":"Introduces the options framework used to collect hierarchical demonstrations and to define the gated policy-over-options model for behavior cloning.","marker":"[24]"},{"why":"Provides the open-source maritime capture-the-flag simulation environment used for training and evaluation.","marker":"[25]"},{"why":"Supplies the marine autonomy software stack used to run the real-world capture-the-flag games on unmanned surface vehicles.","marker":"[26]"},{"why":"Documents why comparing snippets that capture different subtasks of a long-horizon task is unreliable, motivating SPLASH's full-trajectory comparison design.","marker":"[33]"},{"why":"Establishes the precedent of using success and failure labels in learning from suboptimal demonstrations, which SPLASH generalizes into trajectory-pair pruning.","marker":"[37]"}],"fun_headline_variants":["SPLASH: Suboptimal demos teach long-horizon adversarial rewards","SPLASH: IRL handles suboptimal demos for long-horizon tasks","Preference-based IRL tackles suboptimal long-horizon demos","SPLASH learns from imperfect demos to play adversarial games","From suboptimal demos to accurate rewards: SPLASH for long-horizon IRL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that injecting more noise into the cloned options policy produces strictly worse trajectories, so the automatically generated preference labels are mostly correct; SPLASH's score and success checks remove only the obvious contradictions and do not test this premise.","fun_headline_variants_meta":{"raw":{"variants":["SPLASH: Suboptimal demos teach long-horizon adversarial rewards","SPLASH: IRL handles suboptimal demos for long-horizon tasks","Preference-based IRL tackles suboptimal long-horizon demos","SPLASH learns from imperfect demos to play adversarial games","From suboptimal demos to accurate rewards: SPLASH for long-horizon IRL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2284,"prompt_tokens":915,"completion_tokens":1369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1266}},"tokens_in":531,"tokens_out":1369,"duration_ms":11777,"temperature":1.0,"reasoning_tokens":1266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:11:55.017219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Roll out the cloned options policy under the exact training noise schedule and compare mean ground-truth score per noise bin; if mean score does not strictly decrease as $\\epsilon$ increases, the automatic preference labels are wrong and the recovered reward cannot be trusted. A second check is to compute whether SPLASH's predicted returns rank-order with ground-truth returns on held-out trajectories better than D-REX's; if not, the claimed advantage is absent.","supporting_citations":[{"cited_title":"Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,","cited_arxiv_id":null,"evidence_quote":"Defines the ranked-trajectory preference objective and the snippet-based progress assumption that SPLASH replaces with full, downsampled trajectory comparisons."},{"cited_title":"Better-than-demonstrator imitation learning via automatically-ranked demonstrations,","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic noise-injection preference labeling that SPLASH inherits and uses as its baseline for reward learning from suboptimal demonstrations."},{"cited_title":"Learning from suboptimal demonstration via self-supervised reward regression,","cited_arxiv_id":null,"evidence_quote":"Represents the alternative noise-performance characterization approach that fits an assumed degradation curve, which SPLASH explicitly avoids."},{"cited_title":"Pyquaticus capture the flag gymnasium,","cited_arxiv_id":null,"evidence_quote":"Provides the open-source maritime capture-the-flag simulation environment used for training and evaluation."},{"cited_title":"Nested autonomy for unmanned marine vehicles with moos-ivp,","cited_arxiv_id":null,"evidence_quote":"Supplies the marine autonomy software stack used to run the real-world capture-the-flag games on unmanned surface vehicles."},{"cited_title":"A survey of preference-based reinforcement learning methods,","cited_arxiv_id":null,"evidence_quote":"Documents why comparing snippets that capture different subtasks of a long-horizon task is unreliable, motivating SPLASH's full-trajectory comparison design."},{"cited_title":"Inverse reinforcement learning from failure,","cited_arxiv_id":null,"evidence_quote":"Establishes the precedent of using success and failure labels in learning from suboptimal demonstrations, which SPLASH generalizes into trajectory-pair pruning."}],"review_version":1}