pith. sign in

arxiv: 2604.07016 · v1 · submitted 2026-04-08 · 💻 cs.LG

Predictive Representations for Skill Transfer in Reinforcement Learning

Ruben Vereecken , Luke Dickens , Alessandra Russo This is my paper

Pith reviewed 2026-05-10 17:23 UTC · model grok-4.3

classification 💻 cs.LG
keywords reinforcement learningtransfer learningstate abstractionpredictive representationsoptions frameworkskill transferoutcome predictionsabstract actions
0
0 comments X

The pith

Outcome predictions create abstractions that let reinforcement learning skills transfer across tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Outcome-Predictive State Representations as agent-centered abstractions built from predictions of compact, task-independent environmental outcomes. It establishes formally and through experiments that these representations support optimal but limited transfer. To remove the limitation the authors define OPSR-based skills as abstract actions, constructed from options, that become reusable precisely because of the underlying state abstraction. Skills are learned from demonstrations and then shown to accelerate learning in entirely new tasks with no task-specific preprocessing.

Core claim

Outcome-Predictive State Representations (OPSRs) are formed from predictions of task-independent compact observations of the environment. These representations permit optimal yet limited transfer. By constructing skills as options grounded in OPSRs the authors obtain reusable abstract actions that overcome the prior trade-off, enabling substantial speed-ups when the same skills are applied to previously unseen tasks.

What carries the argument

Outcome-Predictive State Representations (OPSRs): task-independent abstractions consisting of predictions about compact environmental outcomes, which support state abstraction and allow skills (abstract actions derived from options) to be reused across tasks.

If this is right

  • Skills learned once can be applied directly to entirely new and unseen tasks.
  • Learning time in novel tasks decreases considerably without task-specific preprocessing.
  • State abstraction through outcome predictions combines with action abstraction through options to improve transfer.
  • Agents avoid restarting learning from scratch on each new task.
  • Formal conditions are established under which transfer remains optimal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same predictive-abstraction approach could be tested in continuous or high-dimensional domains once suitable outcome variables are identified.
  • OPSR-style predictions might be learned jointly with deep function approximators to scale beyond tabular settings.
  • The framework points toward hierarchical reinforcement learning in which skills at multiple levels of abstraction are composed automatically.

Load-bearing premise

Compact task-independent observations of environmental outcomes exist and can be turned into predictions that support both formal optimality results and empirical transfer without any task-specific preprocessing.

What would settle it

A controlled experiment on a new task in which an agent using learned OPSR-based skills requires the same number of episodes or steps to reach optimal performance as an agent learning from scratch.

Figures

Figures reproduced from arXiv: 2604.07016 by Alessandra Russo, Luke Dickens, Ruben Vereecken.

Figure 1
Figure 1. Figure 1: A simple Gridworld labelled to construct a representation. The shaded area is inaccessible to [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: On the left hand side is depicted an MDP with 5 states and the action set [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Diagram detailing the multi-task transfer setting. Rectangles depict state sets while squares [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Two transfer settings, each showing the same two states with two different state abstractions. In [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two gridworld tasks with different layouts that belong to the same domain. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An outcome sequence–test for the action sequence [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Five tasks of a small Gridworld domain. 3.4 Illustration of a Domain In this section we illustrate the usefulness of outcome equivalent state abstractions for relating tasks. We consider a domain of tasks with Gridworld semantics: 4 actions move the agent in the 4 cardinal directions, unless something is blocking its path. There is only one goal location which incurs a positive reward: there is only one ou… view at source ↗
Figure 10
Figure 10. Figure 10: Two tasks combined contain all of the subtasks that make up a third task. [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Three tasks are depicted. Each task consists of subtasks, the abstractions of which are shown [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: An example hierarchical trace is depicted, produced by an option-enabled agent with primitive [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Left: depiction of a Craftworld task. The diagram on the right illustrates crafting dependencies. [PITH_FULL_IMAGE:figures/full_fig_p034_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Learning curves for an agent without options and agents with three discovered OPSR [PITH_FULL_IMAGE:figures/full_fig_p036_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Craftworld for OPSR 1-3. Since we pose no restrictions on the behavior of options aside from the bias implicit in parameterisa￾tion, it is interesting to investigate the different behaviors emerging with different amounts of options in play. To describe the performance of an agent we can condense and summarize a single learning curve by simply looking at the area under that curve. We can then look at the … view at source ↗
Figure 16
Figure 16. Figure 16: Option occupancy for 100 traces following the same story. [PITH_FULL_IMAGE:figures/full_fig_p037_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: A Lightworld task with three rooms, implemented in VGDL 2.0. The same task description [PITH_FULL_IMAGE:figures/full_fig_p038_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Distribution of areas under the reward curve for target Lightworld tasks with different numbers [PITH_FULL_IMAGE:figures/full_fig_p039_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Distribution of areas under the reward curve in the Lightworld domain with transfer between [PITH_FULL_IMAGE:figures/full_fig_p040_19.png] view at source ↗
read the original abstract

A key challenge in scaling up Reinforcement Learning is generalizing learned behaviour. Without the ability to carry forward acquired knowledge an agent is doomed to learn each task from scratch. In this paper we develop a new formalism for transfer by virtue of state abstraction. Based on task-independent, compact observations (outcomes) of the environment, we introduce Outcome-Predictive State Representations (OPSRs), agent-centered and task-independent abstractions that are made up of predictions of outcomes. We show formally and empirically that they have the potential for optimal but limited transfer, then overcome this trade-off by introducing OPSR-based skills, i.e. abstract actions (based on options) that can be reused between tasks as a result of state abstraction. In a series of empirical studies, we learn OPSR-based skills from demonstrations and show how they speed up learning considerably in entirely new and unseen tasks without any pre-processing. We believe that the framework introduced in this work is a promising step towards transfer in RL in general, and towards transfer through combining state and action abstraction specifically.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Outcome-Predictive State Representations (OPSRs) as agent-centered, task-independent abstractions formed from predictions over compact environmental observations called outcomes. It formally shows that OPSRs support optimal but limited transfer, then defines OPSR-based skills (abstract actions derived from options) to overcome the trade-off. Skills are learned from demonstrations and empirically shown to accelerate learning in entirely new, unseen tasks without pre-processing.

Significance. If the formal results and empirical speedups hold, the work offers a concrete route to transfer in RL by combining state abstraction (via predictive representations) with action abstraction (via options). The explicit use of demonstrations for skill acquisition and the claim of no pre-processing are practical strengths. The framework's emphasis on reusable abstractions could inform subsequent research on generalizable agents.

major comments (2)
  1. [Abstract and formalism section] The central claim that OPSRs enable transfer 'without any pre-processing' in unseen tasks (Abstract) rests on the existence of task-independent outcomes whose predictions yield reusable state abstractions. The formalism must demonstrate that outcome selection or learning is fully automatic and environment-agnostic; otherwise the no-pre-processing guarantee and cross-task reusability do not follow. Clarify the precise definition and acquisition procedure for outcomes, including any assumptions about observability or demonstration data.
  2. [Formal results section] The formal result on 'optimal but limited transfer' with plain OPSRs versus improved transfer with OPSR-based skills is load-bearing. The manuscript should supply the key definitions, theorem statements, and proof sketches (or reference to an appendix) showing how the limitation arises and is overcome by the skill construction; without these the empirical speedups cannot be rigorously linked to the claimed abstraction properties.
minor comments (1)
  1. The abstract refers to 'a series of empirical studies' and 'considerable' speedups but supplies no environment names, baseline algorithms, or quantitative metrics. The full manuscript must include these details, together with statistical significance and ablation controls, to allow assessment of the transfer claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognizing the potential of combining predictive state abstractions with action abstractions for transfer in RL. We address each major comment below and indicate the revisions we will make to the manuscript.

read point-by-point responses

  1. Referee: [Abstract and formalism section] The central claim that OPSRs enable transfer 'without any pre-processing' in unseen tasks (Abstract) rests on the existence of task-independent outcomes whose predictions yield reusable state abstractions. The formalism must demonstrate that outcome selection or learning is fully automatic and environment-agnostic; otherwise the no-pre-processing guarantee and cross-task reusability do not follow. Clarify the precise definition and acquisition procedure for outcomes, including any assumptions about observability or demonstration data.

    Authors: We agree that the definition and acquisition of outcomes require explicit clarification to support the no-pre-processing claim. Outcomes are defined as a fixed, compact, task-independent set of environmental observations (e.g., specific event indicators or sensor thresholds) that do not depend on any particular task. Their selection is environment-agnostic in that the same outcome set applies across tasks; acquisition occurs automatically by direct observation from raw sensory data or demonstration trajectories, without task-specific feature engineering or pre-processing. We assume the outcomes are fully observable to the agent. In the revision we will add a dedicated subsection (3.1) spelling out this definition, the automatic extraction procedure from demonstrations, and the observability assumption, and we will update the abstract to reference this clarification. revision: yes

  2. Referee: [Formal results section] The formal result on 'optimal but limited transfer' with plain OPSRs versus improved transfer with OPSR-based skills is load-bearing. The manuscript should supply the key definitions, theorem statements, and proof sketches (or reference to an appendix) showing how the limitation arises and is overcome by the skill construction; without these the empirical speedups cannot be rigorously linked to the claimed abstraction properties.

    Authors: The key definitions (OPSR as a predictive state abstraction over outcomes, the induced abstract MDP, and OPSR-based skills as options whose initiation and termination sets are defined over the OPSR), the theorem establishing optimal but limited transfer (Theorem 1: any optimal policy in the abstract MDP is optimal in the ground MDP for tasks whose reward depends only on outcomes, yet transfer is limited because the abstraction may collapse distinctions needed for new tasks), and the extension showing that skills overcome the limitation by enabling composition (Theorem 2) are stated with full proofs in Appendix A. In the revised manuscript we will insert concise theorem statements and proof sketches into Section 4, together with an explicit pointer to the appendix, so that the link between the abstraction properties and the empirical speed-ups is self-contained in the main text. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper's abstract introduces OPSRs as agent-centered abstractions composed of predictions over task-independent compact observations (outcomes), then separately claims formal results showing optimal but limited transfer potential, followed by the introduction of OPSR-based skills to address the trade-off. No equations, derivations, or self-referential definitions appear in the provided text that would make any claimed prediction or result equivalent to its inputs by construction. The central premise relies on the existence of task-independent outcomes as a stated foundation rather than a derived quantity, and empirical transfer results are presented as separate validation. No self-citation chains, fitted parameters renamed as predictions, or ansatzes smuggled via prior work are visible. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Framework rests on domain assumptions about outcomes being task-independent and predictive; introduces two new entities with no independent evidence supplied.

axioms (1)
  • domain assumption Outcomes provide task-independent compact observations of the environment that support predictions for state abstraction
    Invoked to define OPSRs and enable transfer without task-specific information.
invented entities (2)
  • Outcome-Predictive State Representations (OPSRs) no independent evidence
    purpose: Agent-centered task-independent abstractions composed of predictions of outcomes for state abstraction and transfer
    Newly introduced formalism; no external validation or prior reference given.
  • OPSR-based skills no independent evidence
    purpose: Abstract actions based on options that reuse state abstractions across tasks
    Introduced to overcome the limited-transfer limitation of plain OPSRs.

pith-pipeline@v0.9.0 · 5474 in / 1273 out tokens · 44649 ms · 2026-05-10T17:23:00.178535+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

  1. [1]

    doi: 10.1109/IROS.2018.8594201

    ISSN 21530866. doi: 10.1109/IROS.2018.8594201. Saul Amarel. On representations of problems of reasoning about actions. InReadings in artificial intelligence, pages 2–22. Elsevier, 1981. David Andre and Stuart J Russell. State Abstraction for Programmable Reinforcement Learning Agents. AAAI, pages 119–125, 2002. ISSN 1049-5258. URLhttp://www.aaai.org/Paper...

    work page doi:10.1109/iros.2018.8594201 2018

  2. [2]

    doi: 10.1016/j.cobeha.2018.11.005

    ISSN 23521546. doi: 10.1016/j.cobeha.2018.11.005. URLhttps://doi.org/10.1016/j. cobeha.2018.11.005. George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement learning. InInternational Joint Conference on Artificial Intelligence, pages 895–900, 2007. George Konidaris, Ilya Scheidwasser, and Andrew G. Barto. Transfer in r...

    work page doi:10.1016/j.cobeha.2018.11.005 2018

  3. [3]

    Britton Wolfe and Satinder Singh

    ISSN 10450823. Britton Wolfe and Satinder Singh. Predictive state representations with options.Proceedings of the 23rd international conference on Machine learning - ICML ’06, pages 1025–1032, 2006. doi: 10.1145/ 1143844.1143973. URLhttp://portal.acm.org/citation.cfm?doid=1143844.1143973. Sen Wu, Hongyang R. Zhang, and Christopher R ´e. Understanding and ...

    work page arXiv 2006

  4. [4]

    𝑛∑︁ 𝑘=1 𝛾 𝑘−1 𝜎(𝑆 𝑡+𝑘−1 , 𝐴𝑡+𝑘−1 , 𝑆𝑡+𝑘 ) 𝑆𝑡 =𝑠 # ⊤ 𝑤 𝑟 = E 𝑝

    𝜙 ′ (𝑠)=𝜙 ′ (𝑡) 3.∃𝛿∈Π 𝛿 (𝜙, 𝑀):𝛿(· |𝑠)≠𝛿(· |𝑡) 4.∀𝛿 ′ ∈Π 𝛿 (𝜙 ′, 𝑀):𝛿 ′ (· |𝑠)=𝛿 ′ (· |𝑡) Proof(2.3).Let𝑀=⟨S,A, 𝑝, 𝑟⟩be an MDP and𝜙:S ↦→Φand𝜙 ′ :S ↦→Φ ′ be two state abstractions. In addition we have: ∃𝛿∈Π 𝛿 (𝜙, 𝑀):∀𝛿 ′ ∈Π 𝛿 (𝜙 ′, 𝑀):𝑣 𝛿 > 𝑣 𝛿′ (46) For simplicity we will assume deterministic policies in this proof. Instead of distributions over actions,...

    work page

  5. [5]

    all optimal abstract policies𝜋 ∗ 𝜙𝛼 :Φ 𝛼 ↦→ P (A)of𝑀 𝜙𝛼 arederived value–compatiblewith respect to𝑀 𝛽; and

    work page

  6. [6]

    no option

    theguaranteed transfer valueof𝑀 𝜙𝛼 for𝑀 𝛽 is maximal (i.e. the optimal policy for𝑀𝛽 can be partially derived from every optimal abstract policy for𝑀 𝜙𝛼). First we provederived value–compatibilityfor all optimal abstract policies. This is a direct consequence of Lemma A.4, which states that all abstract policies arederived value–compatible, which is a stro...

    work page 2016