{"id":"a30dc763-c8ea-45fc-bba4-f7f31a423b57","arxiv_id":"2501.04426","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Dual-Force maximizes offline skill diversity with a Van der Waals force computed from successor features, replacing the mutual-information discriminator with a closed-form off-policy estimator.","lead":"Dual-Force is an offline reinforcement learning method that creates diverse robot skills from demonstration data using a Van der Waals force objective and an off-policy estimator. It removes the skill discriminator used in earlier work and offers more stable training with zero-shot recall of all skills encountered.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The diversity objective in Eqs. 8–9 and Algorithm 1 is defined through successor features E_d_i[phi(s)] for a feature map phi that is never specified anywhere in the paper; without phi the central diversity claim and Fig. 3 are not reproducible or falsifiable.","rationale":"The reader's conditional verdict already flags phi and the coverage assumption. I focus on phi because it is load-bearing: it enters the objective, the reward, the policy updates, and the evaluation figure. Without a specified phi, Eqs. 8-9 and Algorithm 1 do not define a concrete algorithm. The paper contains no baseline comparisons or error bars, so the experimental support rests heavily on the qualitative trajectory figures; these are generated by policies trained with rewards proportional to beta_i^k, so the missing phi is not a cosmetic omission. I do not claim the method is wrong; rather, the submission is under-specified. The proposed ablation would settle whether the observed diversity is due to the VdW term or to other components. The Eq. 11 to Eq. 12 mismatch is a second, theory-level concern: even with phi fixed, the implemented objective does not match the derived Lagrangian. That would require a separate correction and could further reduce confidence, but the missing phi is the primary blocker for evaluating the central claim. Therefore I agree with the reader's weakest assumption and keep the verdict conditional pending these checks.","tokens_in":15150,"tokens_out":11972,"duration_ms":123066,"concrete_test":"Request or inspect the released code to identify the exact feature map phi used in Algorithm 1 for both Solo12 tasks, then run a controlled ablation: (a) phi(s) = constant (or a random permutation of coordinates) and (b) the submitted phi, with all other hyperparameters fixed, and compare the successor-feature distances and the qualitative diversity of the 30 rollouts in Figs. 2 and 4. If (a) still produces the same separated clusters and diverse behaviors, the diversity signal is not actually driven by the stated VdW objective; if the authors cannot provide phi, the central experiment is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Dual-Force maximizes diversity via the VdW objective in Eq. (9), computed from successor features psi_i = E_{d_i(s)}[phi(s)] (Sec. 3.1). Every downstream quantity depends on this feature map: the VdW reward beta_i^k(s,a) = (1-(ell_i^k/ell_0)^3) <phi(s), psi_i^k - psi_{j*}^k> in Sec. 4.1, the successor-feature clusters in Fig. 3, and the weighted behavior-cloning weights w_i^k through the DICE ratios. Yet the paper never defines phi: not in Sec. 3.1, not in Algorithm 1, not in the experimental section, and not in the appendix. Since phi is free, the optimized objective is underspecified: any choice of phi yields a different diversity measure, and Fig. 3's diversity plots are computed from the same unspecified quantities that the algorithm optimizes. The behavioral trajectory plots (Figs. 2 and 4) are informative, but they are generated by policies whose training rewards depend on phi, so the claimed recovery of diverse high-performing behaviors cannot be separated from the choice of phi. A secondary concern is that the move from the Lagrangian Eq. (11) to the practical objective Eq. (12) is heuristic: Eq. (12) contains an unconditional -KL(d_i||d_O) term whose coefficient does not come from Eq. (11), so the formal derivation does not justify the implemented objective. The missing phi is the more fundamental issue because it makes the method unfalsifiable as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dual-Force, an offline algorithm for diversity maximization under imitation constraints. The method replaces the skill discriminator of prior work with an off-policy DICE-based estimator of a Van der Waals (VdW) force objective defined on successor features, and it conditions the value function and policy on a pre-trained Functional Reward Encoding (FRE) to handle non-stationary intrinsic rewards. The authors evaluate on two Solo12 simulation benchmarks (locomotion and obstacle navigation), reporting that the learned skill sets are diverse, match an expert state occupancy, and improve robustness to blocked obstacle paths.","tokens_in":15545,"tokens_out":3097,"duration_ms":34094,"significance":"If the derivation and experiments hold, the paper would make a useful contribution to offline skill discovery: it removes the skill-discriminator training phase, provides a DICE-based off-policy estimator of a physically inspired diversity objective, and leverages FRE for zero-shot skill recall without pre-specifying the number of skills. The appendices contain clean derivations of the state-only KL relaxation and its finite-sample estimator, and the authors are transparent about the role of the FRE as a reward encoder. However, the load-bearing feature map phi is never specified, the Lagrangian-to-implemented-objective step is not justified as written, and the experiments lack quantitative comparisons with existing offline diversity methods; these issues currently prevent the central claims from being fully substantiated.","major_comments":[{"comment":"The feature mapping phi: S -> R^n that defines successor features psi_i = E_{d_i(s)}[phi(s)] is never specified anywhere in the paper. Every downstream quantity depends on it: the VdW reward beta_i^k(s,a) in Sec. 4.1, the successor-feature distance matrix and UMAP plots in Fig. 3, and the weights w_i through the DICE ratios. Because phi is free, the optimized objective is underspecified, and the diversity evidence in Fig. 3 is computed from the same unspecified quantity that the algorithm optimizes. The authors should specify phi precisely (e.g., the full proprioceptive state vector, a learned encoding, or a defined set of features), state where it enters Algorithm 1 as an input, and report sensitivity to the choice of phi.","section":"Sec. 3.1, Sec. 4.1, Algorithm 1"},{"comment":"The transition from the Lagrangian in Eq. (11) to the practical objective in Eq. (12) is not mathematically justified. In Eq. (11), the constraint term is multiplied by lambda_i and the KL term appears as -lambda_i D_KL(d_i||d_O), whereas Eq. (12) has an unconditional -D_KL(d_i||d_O) term with coefficient 1 and a reward R_mu^i(s,a) = (1-sigma(mu_i)) beta_i^k(s,a) + sigma(mu_i) log(c^*(s)/(1-c^*(s))). If sigma(mu_i) is the bounded Lagrange multiplier, the Lagrangian would instead yield a reward beta + sigma log(c/(1-c)) with a KL coefficient sigma, or, after dividing by (1+sigma), weights (1-sigma)beta and sigma log(c/(1-c)) but with KL coefficient (1-sigma). As written, Eqs. (12)-(13) are a heuristic convex combination, not a relaxation of Eq. (11). The authors should either provide a correct derivation, state explicitly that this is a heuristic approximation, or revise the equations so that the KL coefficient and the Lagrange multiplier are consistent.","section":"Sec. 4.1, Eqs. (11)-(13)"},{"comment":"The experimental section does not include quantitative comparisons with the most relevant prior offline diversity method, Vlastelica et al. (2024), nor with standard DICE-based imitation baselines such as SMODICE/DemoDICE beyond the SMODICE-expert. The claims in the abstract and introduction that Dual-Force is 'enhanced', 'more stable', and 'more efficient' than prior work are therefore not directly supported. The authors should add tables reporting, for each task, the number of recovered skills, imitation constraint violation (e.g., estimated KL to the expert state occupancy), diversity metrics, and mean returns, together with at least one baseline that also performs offline diversity maximization under imitation constraints. The trajectory plots in Figs. 2, 4, and 5 are informative but do not by themselves substantiate the quantitative superiority claims.","section":"Sec. 5, Experiments"}],"minor_comments":[{"comment":"Algorithm 1 lists the number of occupancies n, number of state subsets m, number of state-reward pairs t, and Polyak scale alpha, but it does not list the feature map phi, despite phi(s) being used inside the loop. This omission should be fixed.","section":"Algorithm 1, Inputs"},{"comment":"The main text refers to 'Thm. A.2' and 'Theorem A.3', but the appendix labels the corresponding results as 'Lemma A.2' and 'Corollary A.3', and 'Assumption A.1' is sometimes called 'Theorem A.1' inside the appendix proofs. The numbering should be made consistent.","section":"Appendix A, Theorem numbering"},{"comment":"The sentence describing offline behaviors says the dataset contains 'base-heights (low, middle, orange)', where 'orange' appears to be a typo for 'high' or another height category. This should be corrected.","section":"Sec. 5.1, Data Collection"},{"comment":"The axes of the distance matrices and UMAP plots in Fig. 3 are unlabeled, and it is not stated which skill corresponds to which point in the UMAP panels. Adding labels and a legend would improve interpretability.","section":"Fig. 3, captions"},{"comment":"The paper does not report hyperparameters for the value function and policy networks, the discriminator, or the batch sizes and learning rates, nor does it state whether code or trained models will be released. These details are needed for reproducibility.","section":"Sec. 5, Experimental setup"},{"comment":"The statement that the FRE latent representation 'cannot affect the diversity of skills learned' is too strong: the FRE-conditioned value function and policy are trained on rewards R_i^k that depend on the VdW term, so the representation can influence training dynamics and thus indirectly the skill set. The authors should soften this claim or qualify it as referring only to the diversity objective itself.","section":"Sec. 4.3, FRE claim"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the DICE-based estimation machinery is competently assembled, but the unspecified feature map phi and the unsubstantiated Lagrangian-to-implemented-objective step are load-bearing. Both are fixable in a revision: phi must be explicitly defined and its choice justified, and the optimization objective must either be derived correctly or honestly presented as a heuristic with supporting ablations. The lack of quantitative comparison with existing offline diversity baselines is also a significant omission for a methods paper. I would therefore recommend major revision rather than rejection, provided the authors can close these gaps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper combines three known pieces — the VdW diversity objective from Zahavy et al., DICE-style off-policy estimation, and FRE conditioning — into an offline skill discovery method that avoids a learned discriminator. That combination is new relative to the cited literature, and the DICE machinery is applied in a way that is mostly self-consistent. The FRE conditioning for non-stationary rewards is sensible, and the authors are honest that it serves as a hash map rather than a diversity driver. The behavioral trajectory plots do show genuinely different gaits and obstacle strategies, which is evidence that something is working.\n\nThe soft spots are real. The most load-bearing is the feature map phi. It appears in the VdW reward, in Algorithm 1, and in the Figure 3 diversity plots, but the paper never says what phi is for the experiments. That is not a small omission: the entire diversity measure is defined along phi, so without a specified phi the method is not reproducible and the diversity claim is not falsifiable. The successor-feature clustering in Figure 3 is also partly circular, because the algorithm optimizes exactly those successor features; the trajectory plots are more convincing, but they also depend on the unspecified phi. Second, the move from the Lagrangian in Eq. (11) to the practical objective in Eq. (12) is heuristic: the KL penalty coefficient is 1 rather than sigma, and the diversity reward is weighted by (1-sigma) without formal support. Third, there are no baseline comparisons, no seeds, and no error bars anywhere in the experiments. The main empirical claim — stability and efficiency over prior offline diversity methods — is asserted, not demonstrated.\n\nNone of these are fatal in principle. The derivation is not fitted to the target results, the FRE is pretrained on independent random rewards, and the core idea could work with a specified phi and a proper experimental comparison. But as submitted, the paper is under-specified where it matters most. I would send it to reviewers, because the combination is interesting and the math mostly checks out, but I would expect major revision, not acceptance.\n\nFor a reading group, this is a useful case study in how an unspecified design choice can undermine an otherwise clever method. I would not cite it in its current form.","headline":"A plausible recombination of VdW diversity, DICE, and FRE with a real reproducibility gap: the feature map defining diversity is never specified, and the experiments lack baselines and error bars; worth a serious referee but needs substantial revision.","tokens_in":686,"tokens_out":819,"would_cite":false,"duration_ms":22085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dual-Force maximizes offline diversity in imitation-constrained skill learning with a discriminator-free Van der Waals objective estimated by DICE, and recalls every skill via pretrained Functional Reward Encoding.","keywords":["offline reinforcement learning","diversity maximization","imitation constraints","successor features","Van der Waals force","DICE","functional reward encoding","skill discovery"],"falsifier":"Train Dual-Force twice on the same offline datasets with two different feature maps, one encoding behaviorally relevant variables such as base height and route side and one encoding irrelevant variables; if both yield separated successor features but only the first corresponds to visibly distinct rollouts, the diversity claim is an artifact of the chosen $\\phi$. A second check is to remove expert states from the offline mixture so $d_E$ has support where $d_O$ is zero, where Assumption A.1 fails and the constraint estimator should degrade or become undefined.","tokens_in":1706,"feed_emoji":"🤖","tokens_out":1810,"duration_ms":72465,"temperature":0.7,"pith_summary":"The paper sets out to show that offline diversity maximization under imitation constraints can be made stable and practical by replacing the mutual-information objective and its skill discriminator with a Van der Waals force objective whose gradient has closed form in successor features. The entire objective is estimated off-policy through DICE importance ratios, and the value function and policy are conditioned on a pretrained Functional Reward Encoding to handle non-stationary rewards and to recall each encountered skill by its latent code. On two Solo12 simulation benchmarks, the recovered skill sets are claimed to match a target expert state occupancy, recover all expert modes and several offline modes, and include solutions that outperform the expert under adversarial obstacle variations. The motivation is that a demonstration set can then yield a portfolio of distinct, robust behaviors without any environment interaction.","feed_headline":"Offline skill diversity without a skill discriminator","feed_subtitle":"A physical repulsion between skills stabilizes training and recalls every skill by code.","key_machinery":"The VdW force objective over successor features: each skill $i$ is represented by mean feature $\\psi_i = E_{d_i}[\\phi(s)]$, and its diversity reward at state $s$ is $\\beta_i(s,a) = (1 - (\\ell_i/\\ell_0)^3)\\langle\\phi(s), \\psi_i - \\psi_{j^*}\\rangle$, where $j^*$ is the nearest other skill and $\\ell_i$ is the distance to it; close skills repel and far skills attract, with $\\ell_0$ controlling the target spacing. The off-policy engine is the DICE importance ratio $\\eta_i = \\exp\\{\\delta_i\\}/E_{d_O}\\exp\\{\\delta_i\\}$, with $\\delta_i$ the TD error of the KL-regularized dual value function, which turns offline expectations over $d_O$ into expectations under the skill's occupancy. FRE latent codes $z_r$ condition the value function and policy and act as a look-up table for non-stationary rewards.","core_discovery":"The central claim is that all quantities needed for VdW-force diversity — successor features, the dual-conjugate gradient variable, and KL-divergence constraint estimates — admit off-policy estimators, so no skill discriminator or online rollouts are required. Dual-Force relaxes the imitation constraint to a state-only KL bound, computes importance ratios $\\eta_i$ from the DICE dual solution, and uses them to estimate successor features, the VdW reward, and constraint violations. Conditioning the value function and policy on FRE latent codes of the current reward makes the non-stationary reward updates stable and gives zero-shot recall of every skill encountered during training, removing the fixed-number-of-skills requirement. The experimental claim is that on locomotion and obstacle navigation with the Solo12 quadruped, this yields diverse, high-performing skills that imitate the expert and are more robust to adversarial obstacles.","pith_inferences":["A direct ablation that fixes the FRE conditioning and swaps only the diversity objective (VdW versus the mutual-information bound) would isolate whether the stability gain comes from the discriminator-free objective or from conditioning; the paper reports the combined method only.","Because the FRE code is a hash of the reward, the same mechanism could be used after training to compose skills sequentially or to interpolate between latent codes, though the paper does not test composition.","If the feature map is learned or expanded during training, the VdW force would act on behaviorally meaningful dimensions and the method would generalize beyond hand-chosen features; this is an extension, not a claim of the paper."],"forward_implications":["Offline skill discovery no longer needs a separately trained skill discriminator, removing a phase that was unstable under non-stationary rewards.","The number of skills is not a fixed input; every reward encountered during training leaves a latent code that can be recalled at evaluation time.","On Solo12 locomotion and navigation datasets, the recovered skill sets are claimed to match expert state occupancy while spanning different base heights, velocities, and obstacle routes.","In adversarial obstacle configurations, some learned skills outperform the SMODICE-expert, so the portfolio affords robustness to distribution shift.","The estimator generalizes to arbitrary f-divergence imitation constraints, not only KL."],"supporting_citations":[{"why":"Provides the VdW force diversity objective and the iterative Lagrangian heuristic that Dual-Force adapts to the offline setting.","marker":"Zahavy et al. (2023)"},{"why":"Supplies the Fenchel-duality DICE framework used to solve the KL-regularized RL problem off-policy.","marker":"Nachum and Dai, 2020"},{"why":"Introduces Functional Reward Encoding, the pre-trained reward-to-latent map Dual-Force conditions its value and policy on.","marker":"Frans et al. (2024)"},{"why":"The offline diversity-under-imitation baseline whose mutual-information discriminator and fixed skill-set requirement Dual-Force removes.","marker":"Vlastelica et al. (2024)"},{"why":"Provides the state-discriminator and DICE-based imitation machinery that yield the expert-imitation term $\\log c^*(s)/(1-c^*(s))$.","marker":"Kim et al., 2022"},{"why":"Supplies regularized state-occupancy matching and the state-only discriminator formulation used in the imitation constraint.","marker":"Ma et al., 2022a"},{"why":"DO-MiNiC is the online constrained skill-discovery algorithm used to collect the diverse offline dataset from policy checkpoints.","marker":"Cheng et al., 2024"},{"why":"Define successor features, the quantity whose pairwise distances carry the VdW diversity signal.","marker":"Dayan, 1993; Barreto et al., 2016"}],"fun_headline_variants":["Van der Waals forces diversify skills offline","Dual-Force: diverse skills from demonstrations alone","Zero-shot skill recall via reward encoding","Skill diversity without discriminator or preset counts","Off-policy diversity with successor features"],"cache_read_input_tokens":18048,"weakest_assumption_plain":"Diversity is measured entirely through a fixed feature map of the state, so if that map misses the behaviors that matter, the skills can be well-separated in the objective while behaving alike; the estimator also assumes the offline mixture covers every expert state.","fun_headline_variants_meta":{"raw":{"variants":["Van der Waals forces diversify skills offline","Dual-Force: diverse skills from demonstrations alone","Zero-shot skill recall via reward encoding","Skill diversity without discriminator or preset counts","Off-policy diversity with successor features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1781,"prompt_tokens":898,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":817}},"tokens_in":514,"tokens_out":883,"duration_ms":8104,"temperature":1.0,"reasoning_tokens":817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:34:03.088846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Dual-Force twice on the same offline datasets with two different feature maps, one encoding behaviorally relevant variables such as base height and route side and one encoding irrelevant variables; if both yield separated successor features but only the first corresponds to visibly distinct rollouts, the diversity claim is an artifact of the chosen $\\phi$. A second check is to remove expert states from the offline mixture so $d_E$ has support where $d_O$ is zero, where Assumption A.1 fails and the constraint estimator should degrade or become undefined.","supporting_citations":[],"review_version":1}