{"id":"3dc9b6cf-b2d1-4f82-a4ba-de2728dec9e1","arxiv_id":"2505.14820","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MinSubFI directly minimizes subdominance, a margin-based measure of failing to be acceptable, and empirically reports higher demonstrator acceptability than prior imitation methods.","lead":"This paper proposes MinSubFI, an imitation learning method that trains a policy to outperform demonstrations in cost features rather than to match them, based on the idea that human demonstrators 'satisfice' rather than optimize. The authors report higher empirical demonstrator acceptability rates on control tasks and competitive true returns, but the main metric is tied to the training objective and no code is released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's acceptability guarantee is conditional on the demonstrator's aspiration set being representable as positive linear costs over the hand-chosen features; the experiments do not test this, leaving the central 'guaranteed acceptability' claim dependent on an untested representational…","rationale":"I agree with the reader's weakest_assumption: the representation of acceptability as positive linear costs over hand-chosen features is the linchpin of the theoretical guarantee. My stress-test focus is more pointed: the proof of Theorem 2 yields a down-set guarantee, not a general satisficing guarantee, and the experimental metric in Table 3 is circular with the training objective, so the paper never provides independent evidence that real demonstrator acceptability is captured by the feature class. The proposed check directly tests the necessary condition for a concrete environment, and would either confirm the concern (if the true return is not in the positive feature cone) or alleviate it. Because the reader already returned CONDITIONAL and my concern is the same load-bearing assumption, the verdict is unchanged.","tokens_in":20197,"tokens_out":11789,"duration_ms":110245,"concrete_test":"Fit a non-negative linear model of the LunarLander true episodic return on the six cost-feature sums (x², y², vx², vy², θ², ω², control cost) used in the paper, over the 100 PPO demonstrations in Table 1. Compute the residual error and check whether any coefficient needs to be negative to achieve a good fit. If the best non-negative linear fit has large residual (e.g., R²<0.8) or the fit requires negative coefficients for the features to track successful landings, then the true acceptability criterion that generated the demonstrations lies outside the class assumed by Theorem 2, and zero subdominance cannot guarantee acceptability in this evaluation domain; the paper's acceptability claims should then be restricted to the feature-defined aspiration set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MinSubFI provides 'guaranteed acceptability' rests on Theorem 2, which shows zero subdominance implies satisfaction of any aspiration set of the form cost_w(ξ) < ν with w ≥ 0 over the provided features. The proof only delivers this for aspiration sets that are down-sets in the feature space: if the demonstrator's true acceptability criterion is not expressible as a positive linear cost in those features (e.g., it requires speed to be above a minimum, or a successful landing event that is not a linear function of squared state variables), then zero subdominance does not imply acceptability. This is not a mere technicality: the synthetic demonstrations are generated by PPO maximizing the true reward, and the LunarLander true reward includes a discontinuous landing bonus and per-step fuel terms that need not lie in the positive cone generated by the six chosen cost features. The empirical support in Table 3 ('relative γ-satisficing') is computed with respect to the same cost features used in training, so it measures how often the training objective is satisfied, not how often the demonstrator's true acceptability criterion is met. Thus the load-bearing premise is the representability of the true aspiration set in the chosen feature class, and it is never checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MinSubFI, an imitation-learning method based on the idea that demonstrations are satisficing rather than optimal. Trajectories are evaluated through hand-chosen or learned cost features, and the method minimizes the subdominance, a margin-based hinge measure of how far a policy's rollouts are from Pareto-dominating demonstrations in those features. The authors provide a policy-gradient theorem (Theorem 4), a per-state decomposition (Corollary 5), online/offline and snippet-based algorithms, and a preference-based cost-feature learning method (Definition 9). Experiments on CartPole, LunarLander, Hopper, HalfCheetah, and Walker compare MinSubFI with BC, TREX, TREX-CF, AIRL, and GAIL, reporting a relative gamma-satisficing rate and true returns, including a LunarLander setting with human demonstrations.","tokens_in":20523,"tokens_out":9326,"duration_ms":84328,"significance":"If the main theorems were fully established, MinSubFI would be a useful addition to imitation learning: it avoids a scalar reward-function bottleneck, focuses on high-quality snippets, and is designed for suboptimal and noisy demonstrations. The policy-gradient derivation in Theorem 4 is a standard REINFORCE argument and appears correct, and the per-state decomposition in Corollary 5 is algebraically checkable. The true-return results in Table 4, where several MinSubFI variants outperform the demonstrations, provide independent evidence beyond the acceptability metric, and the human-demonstration result is striking. The main reservations are that Theorem 2 is conditional on an untested representability assumption and that Theorem 8 is not proved. The central 'guaranteed acceptability' claim should not be read as a statement about the demonstrator's true acceptability criterion without further evidence.","major_comments":[{"comment":"The acceptability guarantee is conditional on the demonstrator's aspiration set being representable as {xi : w dot f(xi) < nu} with w >= 0 in the specific cost features f. The proof only establishes that zero subdominance gives f(xi) <= f(xi-tilde) coordinatewise, hence w dot f(xi) <= w dot f(xi-tilde) for every w >= 0. If the true acceptability criterion is not a down-set in the positive cone generated by the chosen features, zero subdominance does not imply acceptability. The experiments do not test this: Table 3 evaluates relative gamma-satisficing with the same cost features that define the training objective, so high values largely show that the training objective is satisfied. The manuscript itself concedes this limitation for learned features in Section 3.4 ('it may not be possible to formally guarantee demonstrator acceptance in such settings'). Please add an evaluation against held-out or true acceptability criteria and report how often high-gamma policies violate such criteria.","section":"2.3, Theorem 2; 3.4, Table 3"},{"comment":"Theorem 8 is not proved in the manuscript. Appendix D states that the bound follows from leave-one-out cross-validation, quasiconvexity, and the claim that removing non-support vectors does not change global optima 'when no distinct local optima exist,' but none of these steps is made rigorous. There is no statement of the stability condition needed for leave-one-out bounds, no argument that the sublevel sets have the required structure for support-vector removal, and no derivation of the displayed formula gamma = 1 - (1/N) |union_k Xi_SV^k|. The assumption that the realizable feature set is convex is not verified or discussed in the experiments. Since the introduction advertises 'generalization guarantees for changing acceptability,' this is a load-bearing gap: the theorem should either be proved with explicit constants and conditions, or the claims should be softened to empirical observations.","section":"2.8, Theorem 8; Appendix D"},{"comment":"The proof of Theorem 2 ends with 'xi in satisf_{xi-tilde}' without stating the threshold condition. A complete proof needs to use that the demonstration itself satisfies its aspiration, i.e., w dot f(xi-tilde) < nu, so that w dot f(xi) <= w dot f(xi-tilde) < nu. As written, the theorem conflates Pareto-dominating the demonstration with satisfying an aspiration set that has an explicit threshold nu. This is fixable, but it is part of the formal foundation and should be stated cleanly.","section":"2.3, Theorem 2 proof"},{"comment":"The experimental protocol for Table 3 is under-specified. The text does not clearly state whether the acceptability rate is computed on training demonstrations, held-out demonstrations, or rollouts from the learned policy; it does not define how P(xi' in Omega_{xi-tilde}) is estimated; and it does not report the number of rollouts or variance. Since the paper's headline result is 'much higher rates of guaranteed acceptability,' this protocol needs to be described precisely, including the train/test split and the exact estimator used for the relative gamma-satisficing values.","section":"3.4, Table 3; 2.8, Definition 7"}],"minor_comments":[{"comment":"The text refers to 'pretrained via Offline MinSubFI (Corollary 10)', but no Corollary 10 appears in the paper; this likely refers to Eq. (10) or Corollary 6.","section":"3.3"},{"comment":"The notation for cost weights is inconsistent: Definition 1 uses w, while Theorem 2 and its proof use theta. Please unify the notation.","section":"2.2 and 2.3"},{"comment":"The importance ratio r_{theta,pi-tilde}(xi-tilde) = pi_theta(xi-tilde)/pi-tilde(xi-tilde) is not defined; please specify the trajectory probability formula, e.g., pi_theta(xi) = prod_t pi_theta(a_t|s_t).","section":"2.7, Corollary 6"},{"comment":"'Feature presentation learning method' should be 'feature representation learning method'.","section":"4, Conclusions"},{"comment":"The padding scheme is described with a toy example, but the actual values of h and f_pad used for each environment are not reported; please provide them or a sensitivity analysis, since the degenerate early-termination issue is a real failure mode.","section":"E.3, Trajectory Padding"},{"comment":"Algorithm 2 line 6 says 'Find largest support vector snippets pair(s)', while Eq. (7) defines snippet subdominance as a maximum over snippet pairs; the wording should be aligned (for example, 'find the snippet pair with maximum subdominance').","section":"2.7, Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":"The main new contribution over Ziebart et al. (2022) is the stochastic-policy, policy-gradient, and snippet-level treatment of subdominance. The central risk is representability: the 'guaranteed acceptability' claim is currently evaluated with the same cost features used to define the objective, making it partly self-confirming. The generalization theorem is a second substantive risk. Both issues are fixable within the manuscript's scope, but they require new analysis or experiments rather than copy-editing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading, and it deserves a serious referee rather than a desk reject. The core idea is a natural extension of Ziebart et al.'s subdominance measure: instead of learning a cost function then optimizing it, MinSubFI uses subdominance directly as the policy gradient objective, with a snippet-focused variant and a learned feature representation from pairwise preferences. The policy gradient derivation in Theorem 4 is standard REINFORCE and looks correct; the per-state decomposition in Corollary 5 is useful for applying modern actor-critic methods. The experimental work is reasonably broad: five environments, synthetic and human demonstrations, and an honest ablation on initialization. The true-return numbers are genuinely encouraging, particularly the human lunar lander result and the robustness to degraded demonstration quality. That is real evidence the method does something useful even if the satisficing framing is imperfect.\n\nNow the soft spots, in proportion. The most important is that Theorem 2's acceptability guarantee is conditional on the demonstrator's aspiration set being representable as positive linear costs over the hand-chosen features. The paper states this assumption, but the abstract and several other places say \"guaranteed acceptability\" without the caveat. That overstates things. Table 3 measures the satisficing rate in the same cost-feature space used for training, so it is partly self-confirming; the true-return results are independent and help, but they do not validate the acceptability claim itself. A synthetic experiment with aspiration sets outside the feature cone would clarify the limitation quickly. The second soft spot is Theorem 8: it is only sketched in Appendix D, relying on leave-one-out bounds and a convex realizable-feature-set assumption that is not examined. It may be true, but the paper does not give enough detail to check. Third, no code is released, which makes the snippet construction and padding scheme harder to verify. These are fixable issues, not fatal flaws.\n\nFor whom: researchers in imitation learning and preference-based RL. It offers a new objective that avoids reward-function bottlenecks and seems to handle suboptimal demonstrations well. I would cite it if I worked in this area, and I would send it to peer review with a request to tighten the acceptability claims and expand the generalization bound proof.","headline":"A solid extension of subdominance minimization to policy-gradient imitation learning, but the 'guaranteed acceptability' claim rests on an untested representability assumption.","tokens_in":21017,"tokens_out":2042,"would_cite":true,"duration_ms":19883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imitation learning can guarantee a learned policy will be acceptable to the demonstrator by minimizing subdominance — a margin-based shortfall measure in cost-feature space — without ever estimating the demonstrator's aspirations.","keywords":["imitation learning","satisficing theory","subdominance minimization","policy gradient","demonstrator acceptability","aspiration levels","learning from suboptimal demonstrations","preference-based feature learning"],"falsifier":"Build an environment whose acceptability label is non-monotone in a measured feature — for instance, a lunar lander that is acceptable only if it lands within a time window, so landing too slowly is also a failure — while recording the same features the subdominance uses. Train MinSubFI on demonstrations that land inside the window, then check whether its zero-subdominance rollouts (which may land much faster than any demonstration) are accepted by the labeler; if a substantial fraction are rejected, positive-linear-threshold aspirations are not the right model of acceptability.","tokens_in":20055,"feed_emoji":"🎯","tokens_out":10438,"duration_ms":83475,"temperature":0.7,"pith_summary":"This paper reframes imitation learning around a different model of the demonstrator: not a near-optimal optimizer, but a satisficing agent who produces behavior that meets personal aspiration levels and accepts anything that clears that bar. On that view, the right learning target is not to match or exceed an unknown reward function but to produce trajectories the demonstrator would accept. The paper claims this can be done by minimizing subdominance, a margin-based measure of how far a policy's trajectories lag behind demonstrations in cost-feature space; zero subdominance provably implies the demonstration's aspiration set is satisfied, so the objective directly targets acceptability rather than reward. Experiments show the resulting policy substantially outperforms existing imitation learners on guaranteed demonstrator acceptability and stays competitive on true returns, including with human demonstrations and heavily suboptimal data.","feed_headline":"Satisficing imitation guarantees demonstrator acceptance","feed_subtitle":"A subdominance margin objective makes learned policies beat the demonstrator's own quality bar, with no reward model.","key_machinery":"The engine of the method is the subdominance, a margin-based hinge between an imitator trajectory $\\xi$ and a demonstration $\\tilde{\\xi}$: $\\mathrm{subdom}_\\alpha(\\xi, \\tilde{\\xi}) = \\sum_k [\\alpha_k(f_k(\\xi) - f_k(\\tilde{\\xi})) + 1]_+$, with $[x]_+ = \\max(x, 0)$ and $\\alpha_k$ a per-feature margin slope. It is zero exactly when the imitator is no worse than the demonstration on every cost feature, and because arbitrary positive linear combinations of the features are monotone in each feature, zero subdominance implies acceptability under the whole aspiration set (Theorem 2). A per-state decomposition (Corollary 5) rewrites trajectory subdominance as a sum of state contributions, which lets off-the-shelf policy gradient algorithms assign credit causally, and a support-vector structure (only demonstrations the imitator fails to beat by the margin contribute) automatically down-weights or drops low-quality demonstrations. A snippet variant maximizes subdominance over extracted snippet pairs, focusing learning on the highest-quality portions of trajectories even when whole demonstrations are easy to beat.","core_discovery":"The paper's central claim is that a policy can be trained to be acceptable to a demonstrator without ever estimating the demonstrator's aspirations. Formally, if acceptability is defined by an aspiration set of positive linear cost-threshold conditions $\\mathrm{cost}_w(\\xi) < \\nu$ over trajectory features, then any trajectory with zero subdominance relative to a demonstration Pareto-dominates it in every feature and therefore satisfies every aspiration in its set (Theorem 2). Minimizing expected subdominance as the reinforcement-learning objective (Definition 3) thus maximizes the probability that rollouts fall inside the demonstrator's unknown acceptability set, and the margin built into the subdominance yields a finite-sample guarantee: with $N$ demonstrations the learned policy is on average $\\gamma$-satisficing with $\\gamma = 1 - \\frac{1}{N}\\lvert\\cup_k \\tilde{\\Xi}^{\\mathrm{SV}_k}\\rvert$ (Theorem 8). The paper develops online, offline, and snippet-focused variants, and shows empirically that the approach yields far higher rates of guaranteed acceptability than behavioral cloning, reward-learning (T-REX), and adversarial baselines, with competitive or better true returns.","pith_inferences":["Beyond the paper's claims: if acceptability is what a real user cares about, MinSubFI could be tested in a direct human-subject study where people label the acceptability of rollouts; the paper's environments, with synthetic returns, can only approximate this.","Beyond the paper's claims: the support-set view implies data requirements scale with the 'frontier' of hard demonstrations rather than the total count — a checkable prediction is that subdominance loss plateaus once support vectors stop shrinking.","Beyond the paper's claims: since the paper notes aspirations are dynamic, a natural extension is a time-varying margin $\\alpha$ modeling a demonstrator who grows stricter with skill or looser with fatigue; the paper does not train with a changing $\\alpha$.","Beyond the paper's claims: zero-subdominance policies are safe by construction with respect to the demonstrated feature envelope, which is a useful property in high-stakes settings where reward misspecification is the failure mode, though the paper does not make this safety claim explicitly."],"forward_implications":["When a demonstrator's standards concern safety-relevant qualities such as smoothness, landing precision, or staying near a target, a policy guaranteed acceptable on the demonstration's own feature set is a more direct alignment target than one that merely maximizes estimated reward.","Low-quality and noisy demonstrations stop influencing the learned policy once it beats them by the margin, so the approach needs no explicit noise modeling or demonstration filtering.","Snippet-level subdominance lets the learner extract and imitate high-quality portions of trajectories, so useful behavior is not lost just because a whole demonstration is poor.","The generalization bound ties acceptability on the population directly to the number of support demonstrations left undominated, giving a concrete, checkable statement of when the method will transfer to unseen demonstrations.","Because the objective is defined entirely from cost features and demonstrations, the same machinery works offline from fixed data and online from rollouts, and the learned-feature variant can recover multiple competing objectives from pairwise preference labels alone."],"supporting_citations":[{"why":"Introduces the subdominance measure and its margin slopes $\\alpha$, the objective MinSubFI reuses; Theorem 2's guarantee inherits from this definition.","marker":"[Ziebart et al., 2022]"},{"why":"Foundational satisficing theory: decision makers accept behavior meeting aspiration levels rather than optimizing, which is the paper's model of the demonstrator.","marker":"[Simon, 1956]"},{"why":"Feature-matching apprenticeship learning, the prior guarantee (Equation 1) that MinSubFI contrasts with and extends by requiring margin-based dominance.","marker":"[Abbeel and Ng, 2004]"},{"why":"T-REX, the main extrapolative baseline; its reward-learning pipeline is the alternative that MinSubFI's direct subdominance objective replaces.","marker":"[Brown et al., 2019]"},{"why":"Support-vector leave-one-out generalization bounds that Theorem 8's $\\gamma$-satisficing guarantee is built on.","marker":"[Vapnik and Chapelle, 2000]"},{"why":"Policy gradient theorem formalism that Theorem 4's subdominance policy gradient and the update rules use.","marker":"[Sutton et al., 1999]"},{"why":"PPO, the base algorithm all experiments use for policy optimization in the comparisons.","marker":"[Schulman et al., 2017]"},{"why":"Analytic solution for the optimal margin slopes $\\alpha$ used in Algorithms 1 and 3.","marker":"[Memarrast et al., 2023]"}],"fun_headline_variants":["Satisficing imitation guarantees acceptance without reward models","Imitation via subdominance margins: never estimate aspirations","Focused satisficing beats optimality for imitation learning","Learn to satisfy the demonstrator, not to mimic perfectly","Satisficing imitation: Pareto-dominant policies need no rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a demonstrator's acceptability set is representable as aspiration thresholds on positive linear combinations of the chosen cost features — if the true standard is not monotone in those features (say, 'too slow' is also unacceptable), a trajectory beating the demonstration in every recorded feature can still be judged unacceptable, and the generalization bound additionally presupposes that the set of realizable feature vectors is convex.","fun_headline_variants_meta":{"raw":{"variants":["Satisficing imitation guarantees acceptance without reward models","Imitation via subdominance margins: never estimate aspirations","Focused satisficing beats optimality for imitation learning","Learn to satisfy the demonstrator, not to mimic perfectly","Satisficing imitation: Pareto-dominant policies need no rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1417,"prompt_tokens":939,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":555,"tokens_out":478,"duration_ms":5557,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:28:58.108663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an environment whose acceptability label is non-monotone in a measured feature — for instance, a lunar lander that is acceptable only if it lands within a time window, so landing too slowly is also a failure — while recording the same features the subdominance uses. Train MinSubFI on demonstrations that land inside the window, then check whether its zero-subdominance rollouts (which may land much faster than any demonstration) are accepted by the labeler; if a substantial fraction are rejected, positive-linear-threshold aspirations are not the right model of acceptability.","supporting_citations":[{"cited_title":"Ziebart, Sanjiban Choudhury, Xinyan Yan, and Paul Vernaza","cited_arxiv_id":null,"evidence_quote":"Introduces the subdominance measure and its margin slopes $\\alpha$, the objective MinSubFI reuses; Theorem 2's guarantee inherits from this definition."},{"cited_title":"Rational choice and the structure of the environment","cited_arxiv_id":null,"evidence_quote":"Foundational satisficing theory: decision makers accept behavior meeting aspiration levels rather than optimizing, which is the paper's model of the demonstrator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Feature-matching apprenticeship learning, the prior guarantee (Equation 1) that MinSubFI contrasts with and extends by requiring margin-based dominance."},{"cited_title":"Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations","cited_arxiv_id":null,"evidence_quote":"T-REX, the main extrapolative baseline; its reward-learning pipeline is the alternative that MinSubFI's direct subdominance objective replaces."},{"cited_title":"Policy gradient methods for reinforcement learning with function approx- imation","cited_arxiv_id":null,"evidence_quote":"Policy gradient theorem formalism that Theorem 4's subdominance policy gradient and the update rules use."}],"review_version":1}