Imitation Learning via Focused Satisficing

Ali Farajzadeh; Brian Ziebart; Nikolaos Agadakos; Rushit N. Shah; Sanjiban Choudhury; Synthia Sasulski

REVIEW 4 major objections 6 minor 35 references

Imitation Learning via Focused Satisficing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Imitation learning can guarantee a learned policy will be acceptable to the demonstrator by minimizing subdominance — a margin-based shortfall measure in cost-feature space — without ever estimating the demonstrator's aspirations.

desk verdict A solid extension of subdominance minimization to policy-gradient imitation learning, but the 'guaranteed acceptability' claim rests on an untested representability assumption. read the letter →

arxiv 2505.14820 v2 pith:4AMFVHM4 submitted 2025-05-20 cs.LG cs.AI

Rushit N. Shah , Nikolaos Agadakos , Synthia Sasulski , Ali Farajzadeh , Sanjiban Choudhury , Brian Ziebart This is my paper · ORCID

classification cs.LGcs.AI

keywords imitationlearningsatisficingtheorysubdominanceminimizationpolicygradientdemonstratoracceptabilityaspirationlevelsfromsuboptimaldemonstrationspreference-basedfeature

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reframes imitation learning around a different model of the demonstrator: not a near-optimal optimizer, but a satisficing agent who produces behavior that meets personal aspiration levels and accepts anything that clears that bar. On that view, the right learning target is not to match or exceed an unknown reward function but to produce trajectories the demonstrator would accept. The paper claims this can be done by minimizing subdominance, a margin-based measure of how far a policy's trajectories lag behind demonstrations in cost-feature space; zero subdominance provably implies the demonstration's aspiration set is satisfied, so the objective directly targets acceptability rather than reward. Experiments show the resulting policy substantially outperforms existing imitation learners on guaranteed demonstrator acceptability and stays competitive on true returns, including with human demonstrations and heavily suboptimal data.

What carries the argument

The engine of the method is the subdominance, a margin-based hinge between an imitator trajectory $\xi$ and a demonstration $\tilde{\xi}$: $\mathrm{subdom}_\alpha(\xi, \tilde{\xi}) = \sum_k [\alpha_k(f_k(\xi) - f_k(\tilde{\xi})) + 1]_+$, with $[x]_+ = \max(x, 0)$ and $\alpha_k$ a per-feature margin slope. It is zero exactly when the imitator is no worse than the demonstration on every cost feature, and because arbitrary positive linear combinations of the features are monotone in each feature, zero subdominance implies acceptability under the whole aspiration set (Theorem 2). A per-state decomposition (Corollary 5) rewrites trajectory subdominance as a sum of state contributions, which lets off-the-shelf policy gradient algorithms assign credit causally, and a support-vector structure (only demonstrations the imitator fails to beat by the margin contribute) automatically down-weights or drops low-quality demonstrations. A snippet variant maximizes subdominance over extracted snippet pairs, focusing learning on the highest-quality portions of trajectories even when whole demonstrations are easy to beat.

What would settle it

Build an environment whose acceptability label is non-monotone in a measured feature — for instance, a lunar lander that is acceptable only if it lands within a time window, so landing too slowly is also a failure — while recording the same features the subdominance uses. Train MinSubFI on demonstrations that land inside the window, then check whether its zero-subdominance rollouts (which may land much faster than any demonstration) are accepted by the labeler; if a substantial fraction are rejected, positive-linear-threshold aspirations are not the right model of acceptability.

Watch

Extended reading notes

Core claim

The paper's central claim is that a policy can be trained to be acceptable to a demonstrator without ever estimating the demonstrator's aspirations. Formally, if acceptability is defined by an aspiration set of positive linear cost-threshold conditions $\mathrm{cost}_w(\xi) < \nu$ over trajectory features, then any trajectory with zero subdominance relative to a demonstration Pareto-dominates it in every feature and therefore satisfies every aspiration in its set (Theorem 2). Minimizing expected subdominance as the reinforcement-learning objective (Definition 3) thus maximizes the probability that rollouts fall inside the demonstrator's unknown acceptability set, and the margin built into the subdominance yields a finite-sample guarantee: with $N$ demonstrations the learned policy is on average $\gamma$-satisficing with $\gamma = 1 - \frac{1}{N}\lvert\cup_k \tilde{\Xi}^{\mathrm{SV}_k}\rvert$ (Theorem 8). The paper develops online, offline, and snippet-focused variants, and shows empirically that the approach yields far higher rates of guaranteed acceptability than behavioral cloning, reward-learning (T-REX), and adversarial baselines, with competitive or better true returns.

Load-bearing premise

The load-bearing premise is that a demonstrator's acceptability set is representable as aspiration thresholds on positive linear combinations of the chosen cost features — if the true standard is not monotone in those features (say, 'too slow' is also unacceptable), a trajectory beating the demonstration in every recorded feature can still be judged unacceptable, and the generalization bound additionally presupposes that the set of realizable feature vectors is convex.

Editorial extensions

If this is right

When a demonstrator's standards concern safety-relevant qualities such as smoothness, landing precision, or staying near a target, a policy guaranteed acceptable on the demonstration's own feature set is a more direct alignment target than one that merely maximizes estimated reward.
Low-quality and noisy demonstrations stop influencing the learned policy once it beats them by the margin, so the approach needs no explicit noise modeling or demonstration filtering.
Snippet-level subdominance lets the learner extract and imitate high-quality portions of trajectories, so useful behavior is not lost just because a whole demonstration is poor.
The generalization bound ties acceptability on the population directly to the number of support demonstrations left undominated, giving a concrete, checkable statement of when the method will transfer to unseen demonstrations.
Because the objective is defined entirely from cost features and demonstrations, the same machinery works offline from fixed data and online from rollouts, and the learned-feature variant can recover multiple competing objectives from pairwise preference labels alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims: if acceptability is what a real user cares about, MinSubFI could be tested in a direct human-subject study where people label the acceptability of rollouts; the paper's environments, with synthetic returns, can only approximate this.
Beyond the paper's claims: the support-set view implies data requirements scale with the 'frontier' of hard demonstrations rather than the total count — a checkable prediction is that subdominance loss plateaus once support vectors stop shrinking.
Beyond the paper's claims: since the paper notes aspirations are dynamic, a natural extension is a time-varying margin $\alpha$ modeling a demonstrator who grows stricter with skill or looser with fatigue; the paper does not train with a changing $\alpha$.
Beyond the paper's claims: zero-subdominance policies are safe by construction with respect to the demonstrated feature envelope, which is a useful property in high-stakes settings where reward misspecification is the failure mode, though the paper does not make this safety claim explicitly.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper is worth reading, and it deserves a serious referee rather than a desk reject. The core idea is a natural extension of Ziebart et al.'s subdominance measure: instead of learning a cost function then optimizing it, MinSubFI uses subdominance directly as the policy gradient objective, with a snippet-focused variant and a learned feature representation from pairwise preferences. The policy gradient derivation in Theorem 4 is standard REINFORCE and looks correct; the per-state decomposition in Corollary 5 is useful for applying modern actor-critic methods. The experimental work is reasonably broad: five environments, synthetic and human demonstrations, and an honest ablation on initialization. The true-return numbers are genuinely encouraging, particularly the human lunar lander result and the robustness to degraded demonstration quality. That is real evidence the method does something useful even if the satisficing framing is imperfect.

Now the soft spots, in proportion. The most important is that Theorem 2's acceptability guarantee is conditional on the demonstrator's aspiration set being representable as positive linear costs over the hand-chosen features. The paper states this assumption, but the abstract and several other places say "guaranteed acceptability" without the caveat. That overstates things. Table 3 measures the satisficing rate in the same cost-feature space used for training, so it is partly self-confirming; the true-return results are independent and help, but they do not validate the acceptability claim itself. A synthetic experiment with aspiration sets outside the feature cone would clarify the limitation quickly. The second soft spot is Theorem 8: it is only sketched in Appendix D, relying on leave-one-out bounds and a convex realizable-feature-set assumption that is not examined. It may be true, but the paper does not give enough detail to check. Third, no code is released, which makes the snippet construction and padding scheme harder to verify. These are fixable issues, not fatal flaws.

For whom: researchers in imitation learning and preference-based RL. It offers a new objective that avoids reward-function bottlenecks and seems to handle suboptimal demonstrations well. I would cite it if I worked in this area, and I would send it to peer review with a request to tighten the acceptability claims and expand the generalization bound proof.

Referee Report

4 major / 6 minor

Summary. This paper proposes MinSubFI, an imitation-learning method based on the idea that demonstrations are satisficing rather than optimal. Trajectories are evaluated through hand-chosen or learned cost features, and the method minimizes the subdominance, a margin-based hinge measure of how far a policy's rollouts are from Pareto-dominating demonstrations in those features. The authors provide a policy-gradient theorem (Theorem 4), a per-state decomposition (Corollary 5), online/offline and snippet-based algorithms, and a preference-based cost-feature learning method (Definition 9). Experiments on CartPole, LunarLander, Hopper, HalfCheetah, and Walker compare MinSubFI with BC, TREX, TREX-CF, AIRL, and GAIL, reporting a relative gamma-satisficing rate and true returns, including a LunarLander setting with human demonstrations.

Significance. If the main theorems were fully established, MinSubFI would be a useful addition to imitation learning: it avoids a scalar reward-function bottleneck, focuses on high-quality snippets, and is designed for suboptimal and noisy demonstrations. The policy-gradient derivation in Theorem 4 is a standard REINFORCE argument and appears correct, and the per-state decomposition in Corollary 5 is algebraically checkable. The true-return results in Table 4, where several MinSubFI variants outperform the demonstrations, provide independent evidence beyond the acceptability metric, and the human-demonstration result is striking. The main reservations are that Theorem 2 is conditional on an untested representability assumption and that Theorem 8 is not proved. The central 'guaranteed acceptability' claim should not be read as a statement about the demonstrator's true acceptability criterion without further evidence.

major comments (4)

[2.3, Theorem 2; 3.4, Table 3] The acceptability guarantee is conditional on the demonstrator's aspiration set being representable as {xi : w dot f(xi) < nu} with w >= 0 in the specific cost features f. The proof only establishes that zero subdominance gives f(xi) <= f(xi-tilde) coordinatewise, hence w dot f(xi) <= w dot f(xi-tilde) for every w >= 0. If the true acceptability criterion is not a down-set in the positive cone generated by the chosen features, zero subdominance does not imply acceptability. The experiments do not test this: Table 3 evaluates relative gamma-satisficing with the same cost features that define the training objective, so high values largely show that the training objective is satisfied. The manuscript itself concedes this limitation for learned features in Section 3.4 ('it may not be possible to formally guarantee demonstrator acceptance in such settings'). Please add an evaluation against held-out or true acceptability criteria and report how often high-gamma policies violate such criteria.
[2.8, Theorem 8; Appendix D] Theorem 8 is not proved in the manuscript. Appendix D states that the bound follows from leave-one-out cross-validation, quasiconvexity, and the claim that removing non-support vectors does not change global optima 'when no distinct local optima exist,' but none of these steps is made rigorous. There is no statement of the stability condition needed for leave-one-out bounds, no argument that the sublevel sets have the required structure for support-vector removal, and no derivation of the displayed formula gamma = 1 - (1/N) |union_k Xi_SV^k|. The assumption that the realizable feature set is convex is not verified or discussed in the experiments. Since the introduction advertises 'generalization guarantees for changing acceptability,' this is a load-bearing gap: the theorem should either be proved with explicit constants and conditions, or the claims should be softened to empirical observations.
[2.3, Theorem 2 proof] The proof of Theorem 2 ends with 'xi in satisf_{xi-tilde}' without stating the threshold condition. A complete proof needs to use that the demonstration itself satisfies its aspiration, i.e., w dot f(xi-tilde) < nu, so that w dot f(xi) <= w dot f(xi-tilde) < nu. As written, the theorem conflates Pareto-dominating the demonstration with satisfying an aspiration set that has an explicit threshold nu. This is fixable, but it is part of the formal foundation and should be stated cleanly.
[3.4, Table 3; 2.8, Definition 7] The experimental protocol for Table 3 is under-specified. The text does not clearly state whether the acceptability rate is computed on training demonstrations, held-out demonstrations, or rollouts from the learned policy; it does not define how P(xi' in Omega_{xi-tilde}) is estimated; and it does not report the number of rollouts or variance. Since the paper's headline result is 'much higher rates of guaranteed acceptability,' this protocol needs to be described precisely, including the train/test split and the exact estimator used for the relative gamma-satisficing values.

minor comments (6)

[3.3] The text refers to 'pretrained via Offline MinSubFI (Corollary 10)', but no Corollary 10 appears in the paper; this likely refers to Eq. (10) or Corollary 6.
[2.2 and 2.3] The notation for cost weights is inconsistent: Definition 1 uses w, while Theorem 2 and its proof use theta. Please unify the notation.
[2.7, Corollary 6] The importance ratio r_{theta,pi-tilde}(xi-tilde) = pi_theta(xi-tilde)/pi-tilde(xi-tilde) is not defined; please specify the trajectory probability formula, e.g., pi_theta(xi) = prod_t pi_theta(a_t|s_t).
[4, Conclusions] 'Feature presentation learning method' should be 'feature representation learning method'.
[E.3, Trajectory Padding] The padding scheme is described with a toy example, but the actual values of h and f_pad used for each environment are not reported; please provide them or a sensitivity analysis, since the degenerate early-termination issue is a real failure mode.
[2.7, Algorithm 2] Algorithm 2 line 6 says 'Find largest support vector snippets pair(s)', while Eq. (7) defines snippet subdominance as a maximum over snippet pairs; the wording should be aligned (for example, 'find the snippet pair with maximum subdominance').

Circularity Check

1 steps flagged · score 5.0 of 10

Headline acceptability metric is the training objective; independent true-return benchmarks prevent full circularity.

fitted input called prediction [Section 3.4, Table 3 and Definition 7]
"In Table 3, we evaluate the rate that the imitator satisfices demonstrations (Definition 7), guaranteeing demonstrator satisfaction, relative to the rate that a randomly chosen demonstration satisfices other demonstrations, P (ξ ∈ Ω ˜ξ)/P ( ˜ξ′ ∈ Ω ˜ξ), using trajectory-level cost features. ... In contrast, since MinSubFI minimizes an upper bound on the imitator’s satisficing value, it consistently guarantees demonstrator acceptability much more frequently."

The paper's training objective (Definition 3 / Eq. 9) minimizes expected subdominance over the hand-chosen cost features f. Definition 1 defines acceptability/satisficing as cost_w(ξ) < ν over those same features, and Theorem 2 equates zero subdominance with membership in the corresponding acceptability set. Table 3 then reports 'guaranteed acceptability' as the rate at which imitator trajectories have zero/low subdominance with respect to demonstrations in exactly that same feature space. Thus the headline result—higher rates of guaranteed acceptability—is measured by the very objective being minimized: the policy is fitted to reduce subdominance, and the evaluation counts how often subdominance is zero.

full rationale

The paper's analytic core is not circular: Theorem 2 is an explicitly conditional statement that zero subdominance implies satisfaction of any aspiration set expressible as positive linear costs over the provided features, and the proof is a direct consequence of the definitions. The policy-gradient adaptation of subdominance minimization is a new contribution, and the citation to Ziebart et al. (2022) is normal prior work, not a self-citation chain that forces the result. However, the central experimental evidence for 'guaranteed acceptability to the demonstrator' (Table 3) uses the same cost features and subdominance zero-level sets that MinSubFI is trained to minimize, so the acceptability comparison is partially tautological: optimizing subdominance naturally increases the frequency of zero subdominance. The paper's true-return results (Table 4) are an independent, non-circular benchmark because they do not rely on the learned acceptability model, and they show competitive performance; this independent content is why the overall circularity is moderate rather than total. The unvalidated representational premise—that the demonstrator's true aspiration set is representable as positive linear costs over the chosen features—is acknowledged in the paper, but it is load-bearing for the 'guarantee' wording and is never checked against actual demonstrator preferences.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The method leans heavily on the hand-chosen cost features and the assumption that acceptability is a positive linear threshold over them. The α slopes are fitted during training, and several horizon and padding parameters are tuned per environment. The generalization theory relies on an extra convexity assumption. These choices are load-bearing for the headline acceptability results.

free parameters (4)

Subdominance margin slopes α_k = not reported
Optimized via exponentiated gradient or analytically (Appendix A) to set the margin by which the imitator must outperform demonstrations; directly affects the objective.
Snippet horizon T and snippet count N = T = 10-25% of max trajectory length; N not fixed globally
Chosen per environment to define snippet-based subdominance (Section E.5), affecting training signal.
Trajectory padding horizon h and padding cost f_pad = not specified exactly
Ad hoc fix (Appendix E.3) to prevent policies from terminating early to lower subdominance.
Learned cost feature dimensionality K'=3 and MLP width = K'=3, two hidden layers of width 8
Architecture choices for MinSubFI-LCF, affecting the learned feature representation.

assumptions (5)

domain assumption Demonstrator behavior is acceptable rather than optimal (satisficing theory)
Introduced in Section 2.2, this is the conceptual foundation for defining acceptability rather than optimality.
domain assumption Aspiration sets are representable as cost_w(ξ_{t:t'}) < ν with w ≥ 0 over monotonic transformations of cost features
Needed for Theorem 2, which equates zero subdominance with acceptability.
domain assumption Cost features f are available, additive over trajectories, and monotonic transformations preserve acceptability
Used throughout the formulation and experiments, Section 2.1 and E.2.
ad hoc to paper Realizable feature sets of the policy class are convex
Required for Lemma 10 and Theorem 8 in Appendix D; not generally true for neural policy classes.
standard math Policy gradient theorem and SVM leave-one-out bounds
Used in Theorem 4 and Theorem 8 respectively.

invented entities (2)

Aspiration set Ω
purpose: Formalizes the unknown demonstrator acceptability criteria as a set of (w, ν, t, t') constraints
A formal construct introduced in Definition 1; it is not observable and is not directly tested.
γ-satisficing policy
purpose: Defines a policy that satisfies demonstrations with probability at least γ, used as the evaluation metric
Introduced in Definition 7; computed from the same cost features used in training, so it is not an external benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Imitation Learning via Focused Satisficing." pith.science (2026). https://pith.science/paper/4AMFVHM4

@misc{pith2026250514820,
  author       = {Pith},
  title        = {Pith review of: Imitation Learning via Focused Satisficing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AMFVHM4}},
  note         = {Machine review of arXiv:2505.14820}
}

read the original abstract

Imitation learning often assumes that demonstrations are close to optimal according to some fixed, but unknown, cost function. However, according to satisficing theory, humans often choose acceptable behavior based on their personal (and potentially dynamic) levels of aspiration, rather than achieving (near-) optimality. For example, a lunar lander demonstration that successfully lands without crashing might be acceptable to a novice despite being slow or jerky. Using a margin-based objective to guide deep reinforcement learning, our focused satisficing approach to imitation learning seeks a policy that surpasses the demonstrator's aspiration levels -- defined over trajectories or portions of trajectories -- on unseen demonstrations without explicitly learning those aspirations. We show experimentally that this focuses the policy to imitate the highest quality (portions of) demonstrations better than existing imitation learning methods, providing much higher rates of guaranteed acceptability to the demonstrator, and competitive true returns on a range of environments.

Figures

Figures reproduced from arXiv: 2505.14820 by the authors.

**Figure 3.** Examples of lunarlander cost features, which are computed easily from the environment’s observation vector. Thus, our objective is to better minimize the subdominance by finely optimizing over a more flexible class of policies. To generalize to unseen data, we additionally seek a margin of improvement over the demonstrator, i.e., subdomα, throughout our formulation. With this added margin, the subdominance is a co… view at source ↗

**Figure 4.** Mean true returns of 100 trajectories rolled out from the learned policies and the minimum, average, and maximum reward of the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: Implemented architecture of Online MinSubFI (top) and Offline MinSubFI (bottom) using [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 6.** Figure 6: Impact of behavior cloning (BC) and offline MinSubFI [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

[1]

[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y . Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the International Conference on Machine Learning, pages 1–8,

work page 2004
[4]

(11) where α(j) k = 1 fk(ξ)−fk( ˜ξ(j)) is the hinge slope that makes demonstration ˜ξ(j) exactly where the subdominance becomes zero

Algorithm 4 Online update of α values 1: for each k do 2: αk ← αk exp −η′ t P i, ˜ξi,j ∈˜Ξ SVk i,m f (ξi) k − f ( ˜ξi,j ) k +λ|˜Ξ|αk 3: end for Alternatively, the optimal α values for can be computed analytically [Memarrast et al., 2023]: α∗ k = arg min αk m such that: fk(ξ) + λ ≤ 1 m mX j=1 fk(ξ(j)), . (11) where α(j) k = 1 fk(ξ)−fk( ˜ξ(j)) is the hinge ...

work page 2023
[6]

Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations

[Brown et al., 2019] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Ma- chine Learning, pages 783–792. PMLR,

work page 2019
[8]

The generalization guarantee is based on leave-one-out cross validation error, which is an almost- unbiased estimate of generalization error under IID assump- tions [Vapnik and Chapelle, 2000 ]. Removing non-support vectors does not change global optima of subdominance min- imization when no distinct local optima exist, which is the case for this quasicon...

work page 2000
[9]

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

[Chen et al., 2020] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv:2010.11723,

work page Pith review arXiv 2020
[10]

Deep reinforcement learning from human preferences.Ad- vances in Neural Information Processing Systems , 30,

[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Ad- vances in Neural Information Processing Systems , 30,

work page 2017
[12]

[Kalman, 1964] Rudolf E. Kalman. When is a linear control system optimal? Trans ASME, J. Basic Eng., pages 51–60,

work page 1964
[15]

Superhuman fairness

[Memarrast et al., 2023] Omid Memarrast, Linh Vu, and Brian D Ziebart. Superhuman fairness. In Proceedings of the International Conference on Machine Learning , vol- ume 202, pages 24420–24435. PMLR, 23–29 Jul

work page 2023

Show all 35 references

[16]

The magical number seven, plus or minus two: Some limits on our capacity for pro- cessing information

[Miller, 1956] George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for pro- cessing information. Psychological Review , 63(2):81,

1956
[22]

Prox- imal policy optimization algorithms

[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

2017 arXiv
[23]

Rational choice and the structure of the environment

[Simon, 1956] Herbert A Simon. Rational choice and the structure of the environment. Psychological review , 63(2):129,

1956
[24]

Policy gradient methods for reinforcement learning with function approx- imation

[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. Advances in Neural Information Processing Sys- tems, 12,

1999
[27]

Bounds on error expectation for support vector machines

[Vapnik and Chapelle, 2000] Vladimir Vapnik and Olivier Chapelle. Bounds on error expectation for support vector machines. Neural Computation, 12(9):2013–2036,

2000
[28]

Effective analysis of reac- tion time data

[Whelan, 2008] Robert Whelan. Effective analysis of reac- tion time data. The Psychological Record , 58:475–482,

2008
[30]

Imitation learning from imperfect demonstra- tion

[Wu et al., 2019] Yueh-Hua Wu, Nontawat Charoen- phakdee, Han Bao, V oot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstra- tion. In International Conference on Machine Learning , pages 6818–6827. PMLR,

2019
[31]

Confidence-aware imitation learn- ing from demonstrations with varying optimality

[Zhang et al., 2021] Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learn- ing from demonstrations with varying optimality. arXiv preprint arXiv:2110.14754,

2021 arXiv
[32]

Ziebart, Sanjiban Choudhury, Xinyan Yan, and Paul Vernaza

[Ziebart et al., 2022] Brian D. Ziebart, Sanjiban Choudhury, Xinyan Yan, and Paul Vernaza. Towards uniformly su- perhuman autonomy via subdominance minimization. In Proceedings of the International Conference on Machine Learning, pages 27654–27670,

2022
[35]

This toy examples gives us a peek into the source of this degeneracy

= 30 =⇒ [rel]subdom[Σ] α,β(ξ1, ˜ξ) > [rel]subdom[Σ] α,β(ξ1, ˜ξ) =⇒ ξ1 ≺ ξ2. This toy examples gives us a peek into the source of this degeneracy. This phenomenon is very similar in nature to ’re- ward gaming’ often encountered in other reinforcement learn- ing settings [Armstr...

2021
[1952]

Openai gym,

[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym,

2016
[1956]

Algorithms for inverse reinforcement learning

[Ng and Russell, 2000] Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. InInterna- tional Conference on Machine Learning , pages 663–670,

2000
[1964]

Advantages of robotic assistance over a manual approach in simulated subretinal injections and its relevance for gene therapy

[Ladha et al., 2023] Reza Ladha, Thijs Meenink, Jorrit Smit, and Marc D de Smet. Advantages of robotic assistance over a manual approach in simulated subretinal injections and its relevance for gene therapy. Gene Therapy, 30(3- 4):264–270,

2023
[1991]

Direct preference optimization: Your lan- guage model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36,

[Rafailov et al., 2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your lan- guage model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36,

2024
[1999]

A game-theoretic approach to apprenticeship learning

[Syed and Schapire, 2007] Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. Advances in Neural Information Processing Systems, 20,

2007
[2000]

An algorithmic perspective on imitation learning

[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Pe- ters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711,

2018 arXiv
[2004]

Concrete problems in ai safety

[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´e. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565,

2016 arXiv
[2007]

Robust imitation learning from noisy demonstrations

[Tangkaratt et al., 2021] V oot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learning from noisy demonstrations. In Interna- tional Conference on Artificial Intelligence and Statistics , pages 298–306. PMLR,

2021
[2008]

A survey of preference-based reinforcement learning methods

[Wirth et al., 2017] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes F ¨urnkranz. A survey of preference-based reinforcement learning methods. Jour- nal of Machine Learning Research, 18(136):1–46,

2017
[2016]

Pitfalls of learning a reward function online

[Armstrong et al., 2021] Stuart Armstrong, Jan Leike, Lau- rent Orseau, and Shane Legg. Pitfalls of learning a reward function online. In Proceedings of the International Joint Conference on Artificial Intelligence,

2021
[2017]

Learning robust rewards with adverserial inverse rein- forcement learning

[Fu et al., 2018] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learn- ing Representations,

2018
[2018]

Pomerleau

[Pomerleau, 1991] Dean A. Pomerleau. Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Computation, 3(1):88–97, 03

1991
[2019]

Brown, Wonjoon Goo, and Scott Niekum

[Brown et al., 2020] Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learn- ing via automatically-ranked demonstrations. In Proceed- ings of the Conference on Robot Learning, pages 330–359,

2020
[2020]

Distance minimization for reward learn- ing from scored trajectories.Proceedings of the AAAI Con- ference on Artificial Intelligence, 30(1),

[Burchfiel et al., 2016] Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance minimization for reward learn- ing from scored trajectories.Proceedings of the AAAI Con- ference on Artificial Intelligence, 30(1),

2016
[2021]

Rank analysis of incomplete block designs: I

[Bradley and Terry, 1952] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345,

1952
[2023]

Offline reinforcement learning: Tu- torial, review, and perspectives on open problems

[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,

2020 arXiv
[2024]

Stable-baselines3: Reliable reinforcement learning implementations

[Raffin et al., 2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8,

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y . Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the International Conference on Machine Learning, pages 1–8,

work page 2004

[2] [4]

(11) where α(j) k = 1 fk(ξ)−fk( ˜ξ(j)) is the hinge slope that makes demonstration ˜ξ(j) exactly where the subdominance becomes zero

Algorithm 4 Online update of α values 1: for each k do 2: αk ← αk exp −η′ t P i, ˜ξi,j ∈˜Ξ SVk i,m f (ξi) k − f ( ˜ξi,j ) k +λ|˜Ξ|αk 3: end for Alternatively, the optimal α values for can be computed analytically [Memarrast et al., 2023]: α∗ k = arg min αk m such that: fk(ξ) + λ ≤ 1 m mX j=1 fk(ξ(j)), . (11) where α(j) k = 1 fk(ξ)−fk( ˜ξ(j)) is the hinge ...

work page 2023

[3] [6]

Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations

[Brown et al., 2019] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Ma- chine Learning, pages 783–792. PMLR,

work page 2019

[4] [8]

The generalization guarantee is based on leave-one-out cross validation error, which is an almost- unbiased estimate of generalization error under IID assump- tions [Vapnik and Chapelle, 2000 ]. Removing non-support vectors does not change global optima of subdominance min- imization when no distinct local optima exist, which is the case for this quasicon...

work page 2000

[5] [9]

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

[Chen et al., 2020] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv:2010.11723,

work page Pith review arXiv 2020

[6] [10]

Deep reinforcement learning from human preferences.Ad- vances in Neural Information Processing Systems , 30,

[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Ad- vances in Neural Information Processing Systems , 30,

work page 2017

[7] [12]

[Kalman, 1964] Rudolf E. Kalman. When is a linear control system optimal? Trans ASME, J. Basic Eng., pages 51–60,

work page 1964

[8] [15]

Superhuman fairness

[Memarrast et al., 2023] Omid Memarrast, Linh Vu, and Brian D Ziebart. Superhuman fairness. In Proceedings of the International Conference on Machine Learning , vol- ume 202, pages 24420–24435. PMLR, 23–29 Jul

work page 2023