Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

Andrea Fraschini; Davide Tenedini; Marcello Restelli; Mirco Mutti; Riccardo Zamboni

REVIEW 4 major objections 6 minor 15 references

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

T0 review · 4 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Matching policies by the long-horizon states they visit, not by immediate actions, yields low-dimensional latent manifolds that retain rare high-performing behaviors and speed continuous-control adaptation.

desk verdict Solid occupancy upgrade to the authors' own APC: better curation and mixture matching give cleaner latent manifolds and real Hopper gains, with the usual random-pool caveats. read the letter →

arxiv 2603.27044 v3 pith:BGEVVV3Q submitted 2026-03-27 cs.LG cs.AI

Andrea Fraschini , Davide Tenedini , Riccardo Zamboni , Mirco Mutti , Marcello Restelli This is my paper

classification cs.LGcs.AI

keywords policycompressionstateoccupancyunsupervisedreinforcementlearninglatentbehaviormanifoldsmixturematchingcontinuouscontrolPGPE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep RL is sample-inefficient partly because neural policy parameters are high-dimensional and functionally redundant: many weight vectors produce the same behavior. Prior compression methods built latent manifolds by matching immediate actions, a myopic proxy that compounds errors over sequential decisions and discards rare useful behaviors. This paper shows that two occupancy-centric changes fix the problem: select the training policies by how much unique state-space coverage they add, then train a generative model so the reconstructed population matches the original population’s mixture of state-visitation distributions. The resulting low-dimensional latent space organizes true long-horizon functional similarity. Navigating it with simple zeroth-order search then matches or beats standard deep-RL baselines on continuous-control tasks, sometimes recovering returns above any policy present in the random pre-training set. A sympathetic reader cares because the work reframes unsupervised pre-training as learning a compact behavior atlas rather than a parameter atlas, lowering the cost of multi-task and few-shot adaptation.

What carries the argument

Mixture-occupancy matching: find a generative map g and latent codes that minimize the KL divergence between the true mixture occupancy of the policy population and the mixture occupancy of the reconstructed population; the divergence is estimated end-to-end by a differentiable non-parametric k-NN importance-sampling estimator whose trajectory likelihood ratios cancel the environment dynamics. Policies are first filtered by the uniqueness score that adds individual occupancy entropy to divergence from the population mixture.

What would settle it

On Hopper’s backward-locomotion task, train both occupancy-matched and action-matched manifolds from the same random pool and run warm-start PGPE; if the occupancy manifold never exceeds the best dataset policy and remains statistically indistinguishable from the action-matched collapse while SAC or TD3 still succeed, the claim that occupancy matching organizes a more useful manifold is falsified.

Watch

Extended reading notes

Core claim

Occupancy-based Policy Compression (OPC) improves latent policy manifolds by replacing action-matching with an information-theoretic uniqueness score for dataset curation and a fully differentiable objective that matches the mixture state-occupancy of the original policy population to that of the reconstructed population. The latent space that results is organized around long-horizon functional similarity, preserves rare high-performing modes, and supports efficient downstream optimization that competes with standard actor-critic methods and with action-based compression, especially on harder locomotion tasks.

Load-bearing premise

The method assumes a finite pool of randomly initialized neural policies, filtered by a GMM-based uniqueness score and matched only at the population occupancy level, already contains or can interpolate the behaviors needed for arbitrary downstream rewards; if the random pool misses functional modes, the manifold cannot invent them.

Editorial extensions

If this is right

State-space uniqueness curation preserves high-reward outlier policies that action-novelty search discards.
Latent spaces colored by reward form continuous, clustered high-performance regions rather than scattered or collapsed structure.
Zeroth-order search inside the occupancy manifold reaches returns competitive with DDPG, TD3 and SAC at far lower search dimensionality.
On some tasks the method recovers policies whose returns exceed the best policy present in the random pre-training set.
Immediate action matching alone is insufficient for complex sequential control; long-horizon occupancy is the right behavioral similarity for compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If population-level occupancy is the right inductive bias, the same matching objective could compress multi-agent joint policies or hierarchical options without any extrinsic reward.
The asymptotic ceiling is set by the diversity of the random initialization pool; replacing it with active maximum-entropy exploration would likely raise the returns the manifold can reach.
Variance from the k-NN estimator and zeroth-order search could be reduced by gradient-aware latent optimizers that exploit the differentiability of the decoder.
The occupancy objective could be used online as a regularizer inside skill discovery rather than only as a post-hoc compressor of random policies.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a direct, well-executed extension of the authors' APC framework. The two concrete moves—an information-theoretic uniqueness score that keeps rare high-reward policies via occupancy entropy plus KL to the mixture, and a fully differentiable mixture-occupancy reconstruction loss using the importance-sampling k-NN estimator—are the real additions. They replace the myopic action-matching that APC used, and the ablations (score alone, loss alone, both) plus the reward-colored latent grids make the case cleanly: both pieces matter, and together they produce traversable high-return regions that action-matching scatters or misses.

Empirically the paper is careful. Histograms show the OPC filter preserves the heavy tail that APC discards; learning curves with 95 % CIs over ten seeds cover four tasks each on Mountain Car, Reacher and Hopper; Warm-Start PGPE often helps; and on Hopper backward they even beat the best policy in the training set. Motivation from compounding error (citing Metelli) is sound, the math of the score and the log-sum-exp weights is standard and correctly derived, and the limitations section owns the random-init coverage problem and the deterministic-rollout / stochastic-likelihood bias.

Soft spots are real but proportionate. Everything still rests on a finite random-weight pool plus GMM and mini-batch k-NN estimators; if those miss modes the manifold cannot invent them. Domains are low-dimensional continuous control, free parameters are plentiful, and no code is released. None of that collapses the central claim—the Hopper qualitative jump over APC is visible and the ablations isolate the cause—but it does cap how far the result currently travels.

This is for anyone already working on unsupervised policy compression, skill discovery, or latent-space fine-tuning in continuous control. The citation pattern is heavy on the authors' prior line but the new measurements stand on their own against external baselines. I would send it to peer review; the contribution is clear enough and the evidence sharp enough to deserve referee time. Worth a look if the topic is on your desk.

Referee Report

4 major / 6 minor

Summary. The paper proposes Occupancy-based Policy Compression (OPC), an unsupervised pipeline that compresses high-dimensional neural policy parameters into a low-dimensional latent manifold by matching long-horizon state occupancy rather than immediate actions. Building on Action-based Policy Compression (APC), OPC replaces action-space novelty search with an information-theoretic uniqueness score ρ_OPC = H(d_θi) + D_KL(d_θi ∥ d_Ψ) for dataset curation, and replaces action-matching reconstruction with a fully differentiable mixture-occupancy KL objective (Eq. 6) estimated via importance-sampling k-NN. Downstream fine-tuning uses PGPE (optionally with a latent warm-start) in the learned manifold. Empirical evaluation on Mountain Car Continuous, Reacher, and Hopper, with ablations of score vs. loss, reward-distribution histograms, latent-space visualizations, and multi-seed learning curves against DDPG/TD3/SAC/PPO and APC, supports improved retention of rare high-reward behaviors and competitive or superior few-shot adaptation, especially on Hopper.

Significance. If the empirical picture holds, OPC supplies a practical, behaviorally grounded alternative to action-matching and parameter-space search for unsupervised policy-space compression in continuous control and URL. The separation of curation score from compression loss, the end-to-end differentiable occupancy objective, and the multi-domain ablations with 95% CIs over 10 seeds are genuine strengths. The occasional recovery of returns above the best training-set policy (Hopper backward) is particularly interesting if it can be cleanly attributed to the occupancy-matched manifold. The work is incremental relative to APC and the Mutti/Restelli occupancy line, but the shift from myopic action proxies to mixture occupancy is a clear, testable advance with direct implications for sample-efficient adaptation in redundant policy spaces.

major comments (4)

[§4, Eq. (6); Fig. 2] §4, Eq. (6) and the surrounding argument: the central objective is mixture-occupancy matching, not 1-to-1 policy reconstruction. The text claims this organizes the latent space around “true functional similarity” and retains rare high-performing modes. Fig. 2 shows smoother reward topologies under the full OPC pipeline, yet there is no quantitative reconstruction metric (e.g., per-policy occupancy KL or return correlation for the top-percentile policies) showing that individual rare behaviors survive the mixture relaxation. Without that, the attribution of Hopper gains (and super-dataset returns) to occupancy matching remains partly inferential.
[Appendix D; Eq. (1); §4] Appendix D (Practical Implementation of Policy Stochasticity): the pipeline collects deterministic mean-action trajectories but evaluates stochastic policy likelihoods for the importance weights. The authors call this a “necessary trade-off,” yet provide no diagnostic of the resulting bias (e.g., weight variance, effective sample size, or sensitivity of ρ_OPC / LB to the stochasticity schedule). Because both the uniqueness score and the compression loss rest on these weights, unquantified bias is load-bearing for the claim that OPC’s advantage over APC is due to occupancy rather than estimator artifacts.
[§5.3, Fig. 3; §6] §5.3 / Fig. 3 (HP, backward) and §6: O-PGPE sometimes exceeds the best policy present in the curated dataset. This is a strong claim. The manuscript does not analyze whether the decoder is synthesizing novel occupancy modes outside the training support, whether PGPE is simply refining latent codes near high-reward clusters already present, or whether warm-start sampling is responsible. A short occupancy or trajectory comparison of the super-dataset policies against the training set would make the generalization claim falsifiable rather than anecdotal.
[§5.1; §6; Appendix B] §5.1 / Q1 and Limitations §6: the curation advantage of ρ_OPC is demonstrated via heavier high-reward tails (Fig. 1, Fig. 4). However, the free assumption that a finite random-init pool filtered by GMM-based ρ_OPC covers the functionally relevant behaviors of the CMP is never stress-tested against an active exploration baseline (e.g., max-entropy or DIAYN-style skills). For Hopper this coverage gap is acknowledged but left unquantified; it directly limits the “retains most of the original parameter space’s expressivity” claim.

minor comments (6)

[§2–§4] Notation: d_Ψ and d_g(Z) are clear, but the switch between θ, π_θ, and “policy” as interchangeable objects is occasionally dense; a short glossary or consistent shorthand would help.
[Fig. 3] Fig. 3 caption and legend: several curves share similar colors; ensuring that “Best in Dataset” and warm-start variants remain distinguishable in grayscale would improve readability.
[§4.3; §5.3] Warm-start is introduced only for OPC (WS-O-PGPE). A brief note on whether an analogous warm-start was tried for A-PGPE (or why it is OPC-specific) would preempt fairness questions.
[Appendix E] Hyperparameter tables in Appendix E list many free parameters (k-NN k, batch size 5, GMM details, PGPE β1, warm-start sample counts). A short sensitivity check on the most critical ones (especially k and batch size, which affect collapse vs. mode capture) would strengthen reproducibility claims.
[Appendix A] Related work is thorough on URL and weight-space learning; a one-sentence contrast with recent occupancy-feature or steerable-policy representation work (e.g., Li et al. 2026, already cited) on what is new in the mixture objective would sharpen positioning.
[Introduction; Appendix A] Minor typos: “policiestabula rasa” (p. 1), occasional missing spaces around citations, and “V AEs” spacing in related work.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity; self-citations to the authors' prior APC framework serve only as the improved baseline, while all performance claims rest on independent empirical measurements against external DRL algorithms and ablations.

self citation load bearing [Section 3 (Background: Action-based Policy Compression) and Abstract]
"A recent framework, which we refer to as Action-based Policy Compression (APC, Tenedini et al., 2026) ... its performance is severely constrained by relying on immediate action-matching as a reconstruction loss"

APC is prior work by an overlapping author set and is used as the sole conceptual foil. The citation is not load-bearing: the paper supplies independent ablations (OPC score vs APC score, OPC loss vs APC loss) and external baselines; the performance numbers do not reduce to any claim inside the APC paper.

full rationale

The paper is an algorithmic/empirical contribution that defines a new uniqueness score ho_OPC via a standard entropy decomposition of the mixture occupancy (Appendix B) and a mixture-occupancy matching objective (Eq. 6) optimized with a known non-parametric k-NN estimator. Neither quantity is defined in terms of the downstream returns it later reports, nor is any free parameter fitted to a subset of the evaluation tasks and then re-presented as a prediction. The only self-reference is the citation of APC (Tenedini et al., 2026) as the action-matching baseline being replaced; that citation is not load-bearing for the correctness of the new estimators or for the reported gains over DDPG/TD3/SAC/PPO/parameter-space PGPE. Those gains are measured on held-out reward functions with independent random seeds and are therefore externally falsifiable. The acknowledged estimator biases (deterministic rollouts + stochastic likelihoods) and the finite random-policy pool are limitations of coverage, not circular reductions of the claimed results. Consequently the derivation chain contains no self-definitional, fitted-as-prediction, or uniqueness-imported steps.

Assumptions & free parameters 6 free parameters · 4 assumptions · 3 invented entities

The paper rests on standard CMP/occupancy measure definitions, the authors' prior APC pipeline, particle estimators from Ajgl & Šimandl / Mutti et al., and a collection of hand-chosen hyperparameters that control curation, estimation and optimization. No new physical entities are postulated; the invented constructs are algorithmic (OPC score, mixture-matching loss, warm-start PGPE).

free parameters (6)

top-percentile threshold for policy curation = 5%
Fixed at 5 % of 50 000 random policies; directly determines which behavioral modes enter the manifold.
k-NN parameter for importance-sampling KL estimator = 30
Fixed at k=30 for all domains; controls bias-variance of the occupancy divergence loss.
autoencoder mini-batch size and inner iterations = batch=5
Batch size 5 chosen explicitly to avoid mean-policy collapse; number of inner gradient steps I is free.
latent dimensionality k = task-dependent
Swept over {1,2,3,5,8}; reported best curves depend on this choice.
PGPE learning rates, initial σ, β1, warm-start sample counts = per-task tables in App. E
Environment- and task-specific schedules (e.g., center LR 0.11/0.01, warm-start 40–80 policies) are hand-tuned.
GMM components / downsampling for occupancy density estimates
Used to approximate H(d_θi) and D_KL for the uniqueness score; exact fitting details affect ranking.

assumptions (4)

domain assumption State occupancy d_π fully characterizes long-horizon behavioral similarity for the purpose of manifold learning (stronger than action distributions).
Core premise of OPC vs APC; justified by compounding-error argument and Metelli et al. bound but not proved optimal.
ad hoc to paper Uniform mixture occupancy of a finite random-policy population is a sufficient proxy for the feasible behavior set of the CMP.
Dataset generation starts from random weight init; Limitations section concedes this caps asymptotic performance.
ad hoc to paper Deterministic mean-action rollouts evaluated under stochastic policy likelihoods yield usable importance weights for the k-NN estimator.
Explicit empirical trade-off stated in App. D; introduces acknowledged bias.
standard math Standard definitions of CMP, occupancy measure, KL, differential entropy, and PGPE gradient estimator.
Section 2 and background citations.

invented entities (3)

Occupancy-based Policy Compression (OPC) pipeline
purpose: Replace APC's action-matching stages with occupancy uniqueness scoring and mixture-occupancy autoencoder loss.
Central algorithmic contribution; no independent existence outside this paper and its APC predecessor.
Information-theoretic uniqueness score ρ_OPC = H(d_θi) + D_KL(d_θi ∥ d_Ψ)
purpose: Rank random policies for dataset curation so that high-coverage and rare modes are retained.
Derived from entropy decomposition of the mixture (App. B); new scoring rule for this setting.
Mixture-occupancy matching objective (Eq. 6) and its mini-batch k-NN estimator
purpose: Train the generative mapping g so reconstructed and original policy populations cover the same states.
Relaxes 1-to-1 reconstruction; fully differentiable via importance weights that cancel dynamics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching." pith.science (2026). https://pith.science/paper/BGEVVV3Q

@misc{pith2026260327044,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGEVVV3Q}},
  note         = {Machine review of arXiv:2603.27044}
}

abstract

Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space. A recent framework, which we refer to as Action-based Policy Compression (APC), mitigates this issue by compressing the parameter space $\Theta$ into a low-dimensional latent manifold $\mathcal Z$ using a learned generative mapping $g:\mathcal Z \to \Theta$. However, its performance is severely constrained by relying on immediate action-matching as a reconstruction loss, a myopic proxy for behavioral similarity that suffers from compounding errors across sequential decisions. To overcome this bottleneck, we introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage. Specifically, we propose two principal improvements: (1) we curate the dataset generation with an information-theoretic uniqueness metric that delivers a diverse population of policies; and (2) we propose a fully differentiable compression objective that directly minimizes the divergence between the true and reconstructed mixture occupancy distributions. These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors while retaining most of the original parameter space's expressivity. Finally, we empirically validate the advantages of our contributions across multiple continuous control benchmarks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

[1]

Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

arXiv
[2]

Learning policy representations for steerable behavior synthesis.arXiv preprint arXiv:2601.22350,

Beiming Li, Sergio Rozada, and Alejandro Ribeiro. Learning policy representations for steerable behavior synthesis.arXiv preprint arXiv:2601.22350,

arXiv
[3]

Mirco Mutti, Stefano Del Col, and Marcello Restelli

DOI: 10.1609/aaai.v35i10.17091. Mirco Mutti, Stefano Del Col, and Marcello Restelli. Reward-free policy space compression for reinforcement learning. InInternational Conference on Artificial Intelligence and Statistics, pp. 3187–3203. PMLR,

work page doi:10.1609/aaai.v35i10.17091
[4]

Learning to learn with generative models of neural network checkpoints.arXiv preprint arXiv:2209.12892,

10 William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints.arXiv preprint arXiv:2209.12892,

arXiv
[5]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv
[6]

11 Supplementary Materials The following content was not necessarily subject to peer review

DOI: 10.1109/DDCLS49620.2020.9275169. 11 Supplementary Materials The following content was not necessarily subject to peer review. A Extended Related Work Our framework sits at the intersection of unsupervised behavior discovery, weight-space learning, and occupancy matching, thereby unifying these domains to learn a functionally grounded, low- dimensiona...

work page doi:10.1109/ddcls49620.2020.9275169 2020
[7]

In supervised learning, hyper-representations(Schürholt et al., 2021; 2022

or performance checkpoints (Peebles et al., 2022). In supervised learning, hyper-representations(Schürholt et al., 2021; 2022

2022
[8]

To circumvent this, recent works in supervised learning supplement standard parameter reconstruction losses with behavioral output matching (Meynent et al., 2025)

face a critical hurdle: the vast number of parameter-space symmetries (e.g., neuron permutations and scaling) inherent in neural architectures (K˚ urková & Kainen, 1994). To circumvent this, recent works in supervised learning supplement standard parameter reconstruction losses with behavioral output matching (Meynent et al., 2025). In Unsupervised Reinfo...

1994

Show all 15 references

[9]

Similar architectures have been applied in Quality Diversity to improve the sample efficiency of diversity-based search (Rakicevic et al.,

and explored using Variational Autoencoders (V AEs) to analyze pre-trained expert embeddings (Chang et al., 2019). Similar architectures have been applied in Quality Diversity to improve the sample efficiency of diversity-based search (Rakicevic et al.,

2019
[10]

Notably, these methods rely onparameter-reconstruction losses, which fundamentally restrict their compression ratios (e.g., up to 19 : 1 in Hegde et al

or to distill large policy archives into generative models (Hegde et al., 2023). Notably, these methods rely onparameter-reconstruction losses, which fundamentally restrict their compression ratios (e.g., up to 19 : 1 in Hegde et al. (2023)). By optimizing a behavioral objecti...

2023
[11]

and Wasserstein-GANs (Goodfellow et al., 2020; Arjovsky et al., 2017; Gulrajani et al., 2017; Zhang 12 et al.,

2020
[12]

Our contribution scales these concepts from single-expert matching to population-level alignment by developing a mixture-occupancy matching objective

minimize the divergence between an agent’s behavior and a target expert distribution. Our contribution scales these concepts from single-expert matching to population-level alignment by developing a mixture-occupancy matching objective. Once this manifold is learned, we optimi...

2008
[13]

traditionally struggle to scale in highly redundant parameter spaces, executing PGPE within our highly compressed, behaviorally organized latent space resolves these scalability bottlenecks, avoiding the complex decoder Jacobian calculations required by prior manifold optimiza...

2021
[14]

Proof.By the definition of the mixture, m(x) =w ipi(x)+P j̸=i wjpj(x)

For any component i, the KL divergence is bounded: DKL(pi ∥ m)≤ −logw i. Proof.By the definition of the mixture, m(x) =w ipi(x)+P j̸=i wjpj(x). Because density functions and weights are non-negative,m(x)≥w ipi(x), which implies pi(x) m(x) ≤ 1 wi . 14 Since the logarithm is mon...

2021
[15]

Reacher (RC).RC features a two-jointed robotic arm moving in a 2D plane

Episodes terminate upon reaching the standard goal or after 999 steps. Reacher (RC).RC features a two-jointed robotic arm moving in a 2D plane. To focus strictly on task-agnostic behavioral discovery, we remove all target-related information from the observations. The normaliz...

2026

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.

[1] [1]

Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

arXiv

[2] [2]

Learning policy representations for steerable behavior synthesis.arXiv preprint arXiv:2601.22350,

Beiming Li, Sergio Rozada, and Alejandro Ribeiro. Learning policy representations for steerable behavior synthesis.arXiv preprint arXiv:2601.22350,

arXiv

[3] [3]

Mirco Mutti, Stefano Del Col, and Marcello Restelli

DOI: 10.1609/aaai.v35i10.17091. Mirco Mutti, Stefano Del Col, and Marcello Restelli. Reward-free policy space compression for reinforcement learning. InInternational Conference on Artificial Intelligence and Statistics, pp. 3187–3203. PMLR,

work page doi:10.1609/aaai.v35i10.17091

[4] [4]

Learning to learn with generative models of neural network checkpoints.arXiv preprint arXiv:2209.12892,

10 William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints.arXiv preprint arXiv:2209.12892,

arXiv

[5] [5]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv

[6] [6]

11 Supplementary Materials The following content was not necessarily subject to peer review

DOI: 10.1109/DDCLS49620.2020.9275169. 11 Supplementary Materials The following content was not necessarily subject to peer review. A Extended Related Work Our framework sits at the intersection of unsupervised behavior discovery, weight-space learning, and occupancy matching, thereby unifying these domains to learn a functionally grounded, low- dimensiona...

work page doi:10.1109/ddcls49620.2020.9275169 2020

[7] [7]

In supervised learning, hyper-representations(Schürholt et al., 2021; 2022

or performance checkpoints (Peebles et al., 2022). In supervised learning, hyper-representations(Schürholt et al., 2021; 2022

2022

[8] [8]

To circumvent this, recent works in supervised learning supplement standard parameter reconstruction losses with behavioral output matching (Meynent et al., 2025)

face a critical hurdle: the vast number of parameter-space symmetries (e.g., neuron permutations and scaling) inherent in neural architectures (K˚ urková & Kainen, 1994). To circumvent this, recent works in supervised learning supplement standard parameter reconstruction losses with behavioral output matching (Meynent et al., 2025). In Unsupervised Reinfo...

1994