Pith. sign in

REVIEW 5 minor 57 references

Partial Identifiability and Misspecification in Inverse Reinforcement Learning

T0 review · 0 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard inverse reinforcement learning models recover rewards only up to shaping transformations, and they are not robust to even slight misspecification of the environment dynamics or discount factor.

desk verdict A transparent consolidation of prior IRL theory that makes the negative transfer and misspecification results precise, with the main caveat being that the impossibility lives in the unrestricted reward space. read the letter →

arxiv 2411.15951 v1 pith:P4AZO3LS submitted 2024-11-24 cs.LG cs.AI

Joar Skalse , Alessandro Abate This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0590C40
keywords inversereinforcementlearningpartialidentifiabilitymisspecificationrobustnessrewardambiguitypotentialshapingBoltzmannrationalitymaximalcausalentropySTARCmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Inverse reinforcement learning (IRL) aims to infer a reward function from a demonstrated policy, but the same policy can be optimal, Boltzmann-rational, or entropy-maximising for many different rewards. This paper proves that, within the training environment, the Boltzmann-rational and maximal-causal-entropy behavioural models pin the reward down exactly up to potential shaping and S'-redistribution, while the optimality model pins it down only up to optimality-preserving transformations. It then asks how much misspecification these models tolerate, and answers: for the transition function and the discount factor, essentially none. The paper shows that any behavioural model invariant to these transformations is not $\epsilon$-robust for any $\epsilon < 0.5$ under the STARC metric to misspecified $\tau$ or $\gamma$, and that no continuous behavioural model is robust to arbitrarily small perturbations of the observed policy. If correct, this means that a small error in the environment model used by an IRL algorithm can produce a reward function that is nearly orthogonal to the true reward.

What carries the argument

The machinery has two parts. First, reward transformations: potential shaping (adding $\gamma\Phi(s') - \Phi(s)$), S'-redistribution (changing $R(s,a,s')$ without changing its expectation under $\tau$), and optimality-preserving transformations. These transformations generate the invariance partitions of the standard behavioural models, and the paper proves that Boltzmann-rational and MCE policies are invariant exactly to potential shaping composed with S'-redistribution. Second, STARC (Standardised Reward Comparison) metrics, constructed by canonicalising away potential shaping and S'-redistribution, normalising away positive linear scaling, and then measuring distance; the paper shows these metrics are sound and complete, meaning small STARC distance is necessary and sufficient for low worst-case regret, and that any metric with this property is bilipschitz equivalent to them. The negative misspecification theorems work by showing that invariance to a transformation forces the diameter of the invariance partition to be 1 under the STARC metric of the misspecified environment.

What would settle it

On a small MDP with a non-trivial transition function, compute the diameter, under the STARC metric $d^{\mathrm{STARC}}_{\tau,\gamma_2}$, of the equivalence class of an arbitrary reward under potential shaping with discount $\gamma_1$; Theorem 55 predicts this diameter is exactly 1, so any computed diameter below 1 would refute the transfer-impossibility result.

Watch

Extended reading notes

Core claim

The central discovery is a complete characterisation of reward ambiguity and misspecification tolerance for the three standard behavioural models in IRL. A behavioural model is treated as a function $f : \mathcal{R} \to \Pi$, and its invariance partition $\mathcal{Am}(f)$ describes which reward functions are indistinguishable from a given policy. The paper shows that Boltzmann-rational and maximal-causal-entropy policies have the same invariance partition: rewards are identified only up to potential shaping (with discount $\gamma$) and S'-redistribution (with transition $\tau$), so under the same environment the learned reward has the same policy ordering as the true reward and zero STARC distance. The optimality model is more ambiguous: it determines the reward only up to optimality-preserving transformations, which preserve optimal policies but not the full policy ordering. The main negative results are that any model invariant to S'-redistribution is not robust to misspecification of $\tau$, and any model invariant to potential shaping is not robust to misspecification of $\gamma$, for any $\epsilon < 0.5$ under a STARC metric, even when the misspecification is arbitrarily small. The paper also proves that continuous behavioural models are not $\epsilon/\delta$-separating, hence not robust to arbitrarily small perturbations of the observed policy.

Load-bearing premise

The load-bearing premise is that IRL algorithms are analysed in the asymptotic limit in which they may converge to any reward function in the unrestricted space of all possible rewards; if the true reward is known to lie in a restricted class, such as state-action rewards, the negative results can fail, as the paper notes in Appendix A.5.

Editorial extensions

If this is right

  • Within the training environment, Boltzmann-rational and MCE IRL recover a reward with the same policy ordering as the true reward, so the ambiguity is harmless if the learned reward is deployed in the same MDP.
  • The optimality model preserves optimal policies but not the full policy ordering, so its ambiguity has positive upper diameter under any sound and complete metric.
  • Any behavioural model invariant to S'-redistribution is not $\epsilon$-robust to a misspecified transition function for any $\epsilon < 0.5$ under a STARC metric.
  • Any behavioural model invariant to potential shaping is not $\epsilon$-robust to a misspecified discount factor for any $\epsilon < 0.5$ under a STARC metric, for any non-trivial transition function.
  • If the true reward is known to lie in a restricted class, such as rewards depending only on state and action, the negative transfer and misspecification results can fail, as shown in Appendix A.5.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, these results imply that an IRL pipeline that estimates the discount factor or transition dynamics from data should treat those estimates as safety-critical: a small estimation error can yield a recovered reward that is nearly orthogonal to the true one under the STARC metric.
  • A testable extension would be to compute, on small MDPs, the diameter of potential-shaping equivalence classes under a slightly different discount factor; Theorem 55 predicts the diameter is 1 even when the discount error is arbitrarily small.
  • The framework also suggests a design principle for robust reward learning: explicitly breaking invariance to potential shaping or S'-redistribution, for example by anchoring rewards to a fixed reference transition or combining data from multiple environments with different dynamics, may evade the negative theorems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. This paper develops a formal theory of partial identifiability and misspecification robustness for inverse reinforcement learning (IRL). It introduces reward objects, invariance partitions, and two notions of misspecification robustness, one based on equivalence relations and one based on pseudometrics. For the three standard behavioural models (optimal policies, Boltzmann-rational policies, and maximal causal entropy policies), it characterizes the ambiguity of the inferred reward exactly: the first two determine the reward up to potential shaping and S'-redistribution, while the optimality model determines it up to optimality-preserving transformations. The paper also introduces STARC metrics, proves they are sound and complete (inducing both upper and lower regret bounds) and unique up to bilipschitz equivalence, and uses them to quantify ambiguity diameters and misspecification sensitivity. The main negative results (Theorems 52-55, 62-63, 73-74) show that any behavioural model invariant to S'-redistribution or potential shaping cannot guarantee transfer to a different transition function or discount factor, and is not epsilon-robust to arbitrarily small misspecification of those parameters for epsilon < 0.5 under the STARC metric. Appendices A.3-A.5 explicitly examine how the results change under inductive-bias assumptions, prior assumptions about the true reward, and restrictions of the reward space.

Significance. If correct, this is a substantial theoretical contribution. The STARC framework provides a canonical, regret-grounded way to compare reward functions, and the invariance theorems give exact characterizations of partial identifiability under the most common IRL behavioural models. The negative results are sharp conditional impossibilities rather than vague warnings, and the paper is unusually transparent about its scope: the asymptotic convergence model, the unrestricted reward space R, and worst-case quantification are all stated in Section 3, and Appendix A.5 explicitly shows that the transfer and misspecification impossibility results can fail when the reward space is restricted to state-action rewards or single-transition rewards. The proofs are detailed and parameter-free, and the appendices provide reusable tools for analysing new behavioural models. The heavy reliance on the authors' own prior conference papers is acknowledged in Section 1.2, and the novel parts (Section 6.2, Appendix A, and parts of Section 5) are clearly identified.

minor comments (5)
  1. [Abstract and Section 1] The abstract's 'comprehensive mathematical analysis' should be qualified by the unrestricted-reward-space assumption. Appendix A.5 shows that Theorems 52-55, 62-63, and 73-74 can fail when the reward space is restricted, for example to state-action rewards or single-transition rewards, so the reader should be told up front that the impossibility results are worst-case over R.
  2. [Section 3.2] The paragraph beginning 'Do do this, we will fist give' contains typos; it should read 'To do this, we will first give'.
  3. [Theorem 59] The stated condition 'g ≠ c_{τ,γ,ψ}' uses an undefined ψ; it should presumably read 'g ≠ c_{τ,γ,α}'.
  4. [Theorem 62] The first clause 'If f_τ is invariant to S′-redistribution with τ' does not quantify τ; the intended reading is that each f_τ is invariant to S′-redistribution with its own τ, and this should be stated explicitly.
  5. [Definition 28] The soundness inequality places the normalizing range term on the right-hand side to avoid division by zero; a parenthetical remark clarifying this would help readers who might otherwise misread the bound as depending on the scale of R1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: main theorems are proved in Appendix C; self-citations are provenance, not load-bearing.

full rationale

The paper's central claims are conditional impossibility theorems derived from explicit definitions (Definitions 7, 8, 22-24, 34, and 44) and proved in Appendix C. The negative results about misspecification of the transition function and discount factor follow from the invariance of the standard behavioural models to S'-redistribution and potential shaping, combined with properties of STARC metrics; these are mathematical consequences, not fits, renamed predictions, or ansatz smuggled in by citation. The paper does cite prior work by the same authors for some results, but it also states that 'The proofs of all theorems stated in the main text are provided in Appendix C' (Section 1.3), so the self-citations serve as provenance rather than as the sole evidence for load-bearing claims. The canonicity of STARC metrics is stated with proof (Proposition 31), and the paper's own Appendix A.5 explicitly scopes the negative results by showing they can fail when the reward space is restricted, which is an honest limitation rather than a hidden circular step. No fitted parameters, no data-derived predictions, and no definitional collapse were found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical fitting is performed, and no new physical entities are postulated. STARC metrics and reward transformations are formal definitions, not entities with independent empirical handles. The main assumptions are standard finite-MDP assumptions plus the paper's own asymptotic, unrestricted-reward modeling choices.

assumptions (5)
  • domain assumption The environment is a finite, reachable MDP with stationary policies and discount factor gamma in (0,1).
    Stated in Section 2.1; all results are relative to this setting, so they do not directly cover POMDPs, non-stationary policies, or infinite-state environments.
  • domain assumption Reward functions are unrestricted over S x A x S and the analysis quantifies over all of R.
    The negative transfer and misspecification results rely on unrestricted rewards; Appendix A.5 shows that restricting R can remove counterexamples.
  • domain assumption An IRL algorithm is modeled as converging, in the infinite-data limit, to any reward function compatible with the observed data under its behavioural model.
    This is the setup for Definitions 7 and 8; the paper explicitly limits scope to asymptotic behavior and discusses finite-sample bounds as future work.
  • domain assumption Behavioural models are deterministic functions from rewards to policies or other reward objects.
    Definition 1; Appendix A.2 discusses set-valued alternatives and why they reduce to this case.
  • ad hoc to paper The useful way to compare reward functions is via a pseudometric that is sound and complete, meaning it induces upper and lower worst-case regret bounds.
    Definitions 28-29 are introduced by the authors as normative criteria; this choice underlies the canonicity claim for STARC metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Partial Identifiability and Misspecification in Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/P4AZO3LS

@misc{pith2026241115951,
  author       = {Pith},
  title        = {Pith review of: Partial Identifiability and Misspecification in Inverse Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4AZO3LS}},
  note         = {Machine review of arXiv:2411.15951}
}
abstract

The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function $R$ from a policy $\pi$. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only *partially identifiable*, and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer $R$ from $\pi$, an IRL algorithm must have a *behavioural model* of how $\pi$ relates to $R$. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be *misspecified*, which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function $R$. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms.

Figures

Figures reproduced from arXiv: 2411.15951 by the authors.

Figure 1
Figure 1. This figure illustrates Definition 1-3 visually. Specifically, suppose [PITH_FULL_IMAGE:figures/full_fig_p021_1.png] view at source ↗
Figure 2
Figure 2. This figure gives a simple illustration of how Definition 1-3 induces a partial [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. This figure illustrates the conditions in Definition 7. Both [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: This figure illustrates the conditions in Definition 8. Both [PITH_FULL_IMAGE:figures/full_fig_p029_4.png]
Figure 5
Figure 5. Figure 5: This figure summarises our results from Section 5 (also incorporating some [PITH_FULL_IMAGE:figures/full_fig_p048_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 57 canonical work pages

  1. [2]

    Note that f (I(x)) = x for all x ∈ X

    I(f (R)) ≡P R for all R. Note that f (I(x)) = x for all x ∈ X. This means that I(g(R)) is a re- ward function such that f (I(g(R))) = g(R). Since R1 ≡P R2 whenever f (R1) = g(R2), this means that I(g(R)) ≡P R. Similarly, I(f (R)) is a re- ward function such that f (I(f (R))) = f (R). Since Am( f ) ⪯ P , this means that I(f (R)) ≡P R. This completes the fi...

    work page

  2. [4]

    Definition 78 is simply directly analogous to Definition 7, except that it makes the assumption that the learning algorithm L uses the inductive bias described by L

    f ̸= g. Definition 78 is simply directly analogous to Definition 7, except that it makes the assumption that the learning algorithm L uses the inductive bias described by L. Using this definitions, we then get the following result: Theorem 79. Let P be any partition of R, let f, g: R →X be any two reward objects, and let I : X → Rbe any inductive bias for...

    work page

  3. [6]

    To show that f is P -robust to misspecification with g using I, we must show that the following two conditions hold:

    f ̸= g. To show that f is P -robust to misspecification with g using I, we must show that the following two conditions hold:

    work page

  4. [8]

    I(g(R)) ≡P R for all R

    work page

  5. [9]

    I(f (R)) ≡P R for all R

    work page

  6. [10]

    To show that f is P -robust to misspecification with g, we must show that the following two conditions hold:

    f ̸= g. To show that f is P -robust to misspecification with g, we must show that the following two conditions hold:

    work page

  7. [11]

    If f (R1) = g(R2) then R1 ≡P R2

    work page

  8. [12]

    low complexity

    Am( f ) ⪯ P . First, suppose f (R1) = f (R2). Since I(f (R)) ≡P R for all R, we have that I(f (R1)) ≡P R1 and I(f (R2)) ≡P R2. Moreover, since f (R1) = f (R2), this means that I(f (R1)) = I(f (R2)). By transitivity, this then implies that R1 ≡P R2, and so Am( f ) ⪯ P . Similarly, suppose f (R1) = g(R2). Since I(g(R)) ≡P R for all R, we have that I(g(R2)) ...

    work page

Show all 57 references
  1. [13]

    Indiscernibility of Identicals: d(ξ1, ξ2) = 0 if ξ1 = ξ2

  2. [14]

    Identity of Indiscernibles: d(ξ1, ξ2) = 0 only if ξ1 = ξ2

  3. [15]

    Positivity: d(ξ1, ξ2) ≥ 0

  4. [16]

    Symmetry: d(ξ1, ξ2) = d(ξ2, ξ1)

  5. [17]

    slippery

    Triangle Inequality: d(ξ1, ξ3) ≤ d(ξ1, ξ2) + d(ξ2, ξ3). It is straightforward to see that 1-4 hold. For 5, let t be the smallest index on which ξ1 and ξ3 differ. Note that if d(ξ1, ξ3) > d(ξ1, ξ2) and d(ξ1, ξ3) > d(ξ2, ξ3), then it must be the case that ξ1[i] = ξ2[i] for all i...

  6. [18]

    If f (R1) = g(R2) then dR(R1, R2) ≤ ϵ

  7. [19]

    If f (R1) = f (R2) then dR(R1, R2) ≤ ϵ

  8. [20]

    For the first condition, suppose f (R1) = g(R2), which implies that f (R1) = f ◦ t(R2)

    f ̸= g. For the first condition, suppose f (R1) = g(R2), which implies that f (R1) = f ◦ t(R2). By assumption, we have that if f (R) = f (R′), then dR(R, R′) = 0. This implies that dR(R1, t(R2)) = 0. Moreover, we have that dR(R, t(R)) ≤ ϵ for all R; this implies that dR(R2, t(...

    1999

  9. [21]

    G2(ξ) = G1(ξ) − Φ(s0),

  10. [22]

    Qπ 2 (s, a) = Qπ 1 (s, a) − Φ(s),

  11. [23]

    V π 2 (s) = V π 1 (s) − Φ(s),

  12. [24]

    Aπ 2 (s, a) = Aπ 1 (s, a), and

  13. [25]

    R1(s, a, S′) + γα log X a′∈A exp 1 α QS α,1(S′, a′) # for all s, a. Since R2(s, a, s′) = R1(s, a, s′) + γ · Φ(s′) − Φ(s), we can rewrite the right-hand side of this equation as E

    J2(π) = J1(π) − ES0∼µ0 [Φ(S0)] for all trajectories ξ, policies π, states s, actions a, transition functions τ , and initial state distributions µ0. In (1), s0 is the first state of ξ. Proof. To prove (1), first consider a finite trajectory fragment ζ with n transitions. It is...

  14. [26]

    We next note a few basic algebraic properties of our transformations: Proposition 92

    + |S|)-dimensional affine space. We next note a few basic algebraic properties of our transformations: Proposition 92. If T is PSγ, S′Rτ , LS, CS, or OPτ,γ , then

  15. [28]

    For all t ∈ T there is a t− ∈ T such that t ◦ t− = id

  16. [29]

    For all t, t′ ∈ T , we have that t ◦ t′ ∈ T . Proof. For (1), first note that id satisfies the conditions for potential shaping with the function Φ such that Φ( s) = 0 for all s; hence id ∈ PSγ. Next, since trivially ES′∼τ (s,a) [R(s, a, S′)] = ES′∼τ (s,a) [R(s, a, S′)], we ha...

  17. [30]

    The identity transformation, id, is in T

  18. [31]

    We then have that

    For all t ∈ T there is a t− ∈ T such that t ◦ t− = id. We then have that

  19. [32]

    The identity transformation, id, is in T1 J T2

  20. [33]

    For all t ∈ T1 J T2 there is a t− ∈ T1 J T2 such that t ◦ t− = id

  21. [34]

    For all t, t′ ∈ T1 J T2, we have that t ◦ t′ ∈ T1 J T2. Proof. For (1), note that T1, T2 ⊂ T1 J T2. For (2), note that if t ∈ T1 J T2, then t = t1 ◦ · · · ◦tn, where each transformation ti is in either T1 or T2. Let t− = t− n ◦ · · · ◦t− 1 . Now t ◦ t− = id, and t− ∈ T1 J T2. ...

  22. [35]

    The case where T1 or T2 is CS is covered by the above cases, since CS ⊆ PSγ

    The case where T1 = LS and T2 = PSγ is analogous, and likewise for the case where T1 and T2 are S′Rτ and LS. The case where T1 or T2 is CS is covered by the above cases, since CS ⊆ PSγ. This completes the proof. Proposition 94 means that we do not have to be very careful about...

  23. [36]

    U is an open subset of Rn, and 117

  24. [37]

    We will show that m and Π+ satisfy the requirements of this theorem

    f : U → Rn is an injective continuous map, then f (U ) is open in Rn, and f is a homeomorphism between U and f (U ). We will show that m and Π+ satisfy the requirements of this theorem. We begin by noting that Π can be represented as a set of points in R|S|(|A|−1). We do this ...

  25. [38]

    that c(R) and R differ by potential shaping and S′-redistribution, and

  26. [39]

    122 We first show that c is linear

    that c(R1) = c(R2) if R1 and R2 differ by potential shaping and S′- redistribution. 122 We first show that c is linear. Given a state s, let vs be the |S||A||S| - dimensional vector where the ( s′, a, s′′)’th dimension is given by ∞X i=0 γi · P(Si = s′, Ai = a, Si+1 = s′′), wh...

  27. [40]

    c(R) and R differ by potential shaping and S′-redistribution for all R,

  28. [41]

    c(R1) = c(R2) for all R1 and R2 which differ by potential shaping and S′-redistribution, and

  29. [42]

    It follows directly from the construction that c is linear

    n(c(R)) ≤ n(R′) for all R′ such that c(R) = c(R′). It follows directly from the construction that c is linear. To see that c(R) and R differ by potential shaping and S′-redistribution, simply note that c(R) = R − RY, where RY is given by a combination of potential shaping and ...

  30. [43]

    c(R) and R differ by potential shaping and S′-redistribution for all R, and

  31. [44]

    It is straightforward that c is linear, since it is a projection map

    c(R1) = c(R2) for all R1 and R2 which differ by potential shaping and S′-redistribution. It is straightforward that c is linear, since it is a projection map. To see that R and c(R) differ by potential shaping and S′-redistribution, note that there is a constant k such that ηπ...

  32. [45]

    n(R) ≥ 0 for all R ∈ Im(c). 125

  33. [46]

    n(R) = 0 if and only if R = R0 for all R ∈ Im(c)

  34. [47]

    n(α · R) = α · n(R) for all R ∈ Im(c) and all scalars α

  35. [48]

    if” part holds straightforwardly. For the “only if

    n(R1 + R2) ≤ n(R1) + n(R2) for all R1, R2 ∈ Im(c). Here R0 is the reward function that is 0 everywhere. It is trivial to show that Axioms 1 and 3 are satisfied by n. For Axiom 2, note that n(R) = 0 exactly when max π J(π) = min π J(π). If R is R0, then J(π) = 0 for all π, and ...

  36. [49]

    d(R1, R2) = d(R2, R1)

  37. [50]

    ∞X t=0 γtR1(St, At, St+1) # − Eξ∼π

    d(R1, R3) ≤ d(R1, R2) + d(R2, R3) 1 follows from the fact that m is a metric, and 2 follows directly from the fact that the definition of STARC metrics is symmetric in R1 and R2. For 3, the fact that m is a metric again implies that d(R1, R3) = m(s(R1), s(R3)) ≤ m(s(R1), s(R2)...

  38. [51]

    R(s, a, S′) + γα log X a′∈A exp 1 α QS α,1(S′, a′) # . This can be rewritten as E [R1(s, a, S′)] = QS α,1(s, a) − E

    In other words, if bτ,γ,β (R1) = bτ,γ,β (R2) then aτ,γ (R1) = aτ,γ (R2), which means that there exists a function h such that aτ,γ = h ◦ bτ,γ,β . Thus, by Lemma 4, we have that Am( bτ,γ,β ) ⪯ Am(aτ,γ ). Since Am( aτ,γ ) ⪯ Am(bτ,γ,β ) and Am( bτ,γ,β ) ⪯ Am(aτ,γ ), we have that ...

  39. [52]

    150 Proof

    If dR is a pseudometric on R that is both sound and complete, then the upper and lower diameter of Am(f ) under dR is 0. 150 Proof. As per Theorem 45 and 46, if f is either bτ,γ,β or cτ,γ,α , and f (R1) = f (R2), then R1 and R2 differ by a transformation in PS γ J S′Rτ . As pe...

  40. [53]

    Am( o⋆ τ,γ ) ̸⪯ ORDτ,γ

  41. [54]

    Am( o⋆ τ,γ ) ⪯ OPTτ,γ

  42. [55]

    state-valued

    If dR is a pseudometric on R that is both sound and complete, then the lower diameter of Am(o⋆ τ,γ ) under dR is 0, but the upper diameter is greater than 0. Proof. The first part follows from Proposition 49, and the second part follows from Theorem 47. For the third part, fir...

  43. [56]

    Since the hypotenuse is 1, we have that sin(θ/2) = ϵ′/2

    Here 19This can be seen by bisecting the triangle along the vertex between sSTARC τ,γ (R) and sSTARC τ,γ (t(R)), to form two right triangles. Since the hypotenuse is 1, we have that sin(θ/2) = ϵ′/2. 165 elementary trigonometry again tells us that x/(1 + y) = sin(θ) = sin(2 arc...

  44. [57]

    Since R and cSTARC τ,γ (R) differ by potential shaping and S′-redistribution, this transformation can be expressed as a combina- tion of potential shaping andS′-redistribution

    Apply cSTARC τ,γ . Since R and cSTARC τ,γ (R) differ by potential shaping and S′-redistribution, this transformation can be expressed as a combina- tion of potential shaping andS′-redistribution. Call the resulting vector R′

  45. [58]

    This transformation is an instance of positive linear scaling

    Normalise R′, so that its magnitude is 1. This transformation is an instance of positive linear scaling. Call the resulting vector R′′

  46. [59]

    This trans- formation is an instance of positive linear scaling

    Scale R′′ until it forms a right triangle with sSTARC τ,γ (t(R)). This trans- formation is an instance of positive linear scaling. Call the resulting vector R′′′

  47. [60]

    This will move R′′′ by a distance equal to (1 + y) sin(2 arcsin(ϵ′/2)), where (1 + y) = L2(R′′′)

    Move from R′′′ to sSTARC τ,γ (t(R)). This will move R′′′ by a distance equal to (1 + y) sin(2 arcsin(ϵ′/2)), where (1 + y) = L2(R′′′). Moreover, since R′′′ is in the image of cSTARC τ,γ , we have that R′′′ = cSTARC τ,γ (R′′′), and so L2(R′′′) = L2(cSTARC τ,γ (R′′′)). This mean...

  48. [61]

    Since sSTARC τ,γ (t(R)) is simply a normalised version of cSTARC τ,γ (t(R)), this is an instance of positive linear scaling

    Move from sSTARC τ,γ (t(R)) to cSTARC τ,γ (t(R)). Since sSTARC τ,γ (t(R)) is simply a normalised version of cSTARC τ,γ (t(R)), this is an instance of positive linear scaling

  49. [62]

    Since t(R) and cSTARC τ,γ (t(R)) differ by potential shaping and S′-redistribution, this transformation can be expressed as a combination of potential shaping and S′-redistribution

    Move from cSTARC τ,γ (t(R)) to t(R). Since t(R) and cSTARC τ,γ (t(R)) differ by potential shaping and S′-redistribution, this transformation can be expressed as a combination of potential shaping and S′-redistribution. Thus, for an arbitrary reward function R, we can find a se...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.