Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Adding just 60 expert trajectories to a memory buffer that an offline RL policy can query improves performance over a strong baseline, by up to 10.7 normalized points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A plug-in that retrieves expert actions from a small associative memory buffer improves offline Decision Transformer performance on three of four D4RL MuJoCo tasks, with gains up to 10.7 points.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Re:Frame is a plausible new combination of retrieval + Decision Transformer, but the missing union-trained baseline and a contradicting control result leave the mechanism unproven. the 3 major comments →

arxiv 2508.19344 v1 pith:BT226H54 submitted 2025-08-26 cs.LG cs.AI

Re:Frame -- Retrieving Experience From Associative Memory

Daniil Zelezetsky , Egor Cherepanov , Alexey K. Kovalev , Aleksandr I. Panov This is my paper
classification cs.LG cs.AI
keywords offline reinforcement learningassociative memoryDecision Transformerexpert demonstrationslatent-space retrievalD4RL MuJoCodata-efficient RLsequence modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Re:Frame claims that offline RL agents stuck with mostly low-quality data can be substantially improved by attaching a small external memory filled with expert experience, even when that memory holds only 0.1% of the dataset. The paper builds this memory as a compressed autoencoder latent space over expert (return, observation, action) triplets, then lets a Decision Transformer policy retrieve the nearest expert code to its current context and add the decoded action as a correction to its own prediction. In D4RL MuJoCo locomotion experiments, this lifted performance on three of four medium and medium-replay tasks, by up to +10.7 normalized points, using the same 60 expert trajectories that fail to help when used merely to fine-tune the baseline. The authors' central point is that how scarce expert data is stored and accessed—associative retrieval rather than direct training—is what turns a handful of demonstrations into a usable signal.

Core claim

The central claim is that a frozen Associative Memory Buffer (AMB), built from a few expert trajectories and queried by nearest-neighbor search in an autoencoder latent space, can inject expert knowledge into an offline RL policy without changing the backbone architecture. At each step the agent encodes its return-to-go and observation, projects this query into the AMB space, retrieves the closest expert latent vector, decodes it into a candidate action, and adds a learned linear correction to the Decision Transformer's action embedding before the action head. With an AMB of 60 expert trajectories, Re:Frame consistently outperforms fine-tuned DT on Hopper-M (+10.7), Walker2d-MR (+3.2), and W

What carries the argument

The Associative Memory Buffer (AMB): an autoencoder is trained on expert (return-to-go, observation, action) triplets; the encoder maps each triplet into a compact latent code stored in the buffer, while decoders reconstruct the three components. Retrieval is nearest-neighbor over these latent codes using the squared Euclidean distance between a projected query and stored codes. The retrieved code is decoded into an expert action candidate, passed through a linear layer into a correction vector, and added to the transformer's action embedding. This 'retrieve, decode, correct, add' loop is what carries the argument that scarce expert experience can be reused contextually throughout training a

Load-bearing premise

That the nearest expert code in the learned latent space is useful guidance for every state the policy encounters, even though the 60 expert trajectories cover only a tiny fraction of the state distribution.

What would settle it

Run Re:Frame with an AMB that stores 60 expert trajectories from a different task or from the same task with the agent's dynamics altered; if performance stays high, the retrieval is not using content-based correspondence. Alternatively, on a state far from any AMB neighbor, remove the correction and measure whether adding it moves the DT action closer to or further from the expert action; a net increase in action error would falsify the claim that nearest-neighbor retrieval helps.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Re:Frame needs no environment interaction and no backbone modification, so the same memory module can be attached to any offline policy that outputs action embeddings, not just DT.
  • The dataset-AMB control matching DT shows that storing low-quality trajectories provides no gain; the benefit comes specifically from expert content in memory.
  • Because the AMB is frozen after pretraining, retrieval is cheap (nearest-neighbor over at most 60 vectors), so the method adds little compute at deployment.
  • The AMB-size sweep implies that task-dependent latent coverage sets a floor: Walker2d tolerates 30 expert trajectories, while Hopper collapses, so applications must check whether the expert set spans the states the agent will encounter.
  • Fine-tuning on the same 60 trajectories gives only small or negative gains, which the paper interprets as evidence that associative retrieval, not extra exposure, drives improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension implied but not run: Re:Frame should also lift other sequence-modeling offline agents, such as an RNN or a transformer variant, by the same additive-correction mechanism, since it never touches the backbone's weights.
  • The Hopper-MR failure suggests the latent space learned from 60 expert trajectories does not cover the medium-replay state distribution; one could test whether populating the AMB with a handful of diverse medium-replay-plus-expert trajectories, rather than pure expert, restores gains.
  • If retrieval works by content, then corrupting the AMB with expert trajectories from a different task or a shifted dynamics should erase or reverse the gains — a clean control the paper did not include.
  • The additive-correction view implies a continuum: with a sufficiently large expert AMB, Re:Frame should asymptote toward expert behavior even if the training dataset remains low-quality, making it a cheap data-centric alternative to collecting more expert demonstrations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Re:Frame, a plug-in module that equips an offline RL policy such as Decision Transformer (DT) with a small external Associative Memory Buffer (AMB) of expert trajectories. An autoencoder is trained on the expert data to produce latent memory entries; during training and evaluation the agent retrieves the nearest latent entry for the current return-to-go and observation, decodes the stored action, and adds a learned correction to the DT action embedding. Experiments on D4RL Hopper and Walker2d Medium/Medium-Replay report that with 60 expert trajectories Re:Frame improves over vanilla DT in three of four settings (up to +10.7 normalized points), with ablations on AMB size and memory source, plus a fine-tuned DT control. The central empirical claim is supported on those three settings, but the paper's mechanistic claim—that associative retrieval, rather than mere additional expert data, is responsible—is not fully established because the control condition does not include the full training data plus the expert trajectories.

Significance. If confirmed, the contribution is practically useful: it is a simple, architecture-agnostic mechanism for injecting a handful of expert demonstrations into offline RL, with a transparent retrieval procedure and no environment interaction. The paper's strengths include a clearly specified algorithm (Algorithm 1), a buffer-size ablation (Table 2), a memory-source ablation (dataset AMB vs expert AMB in Table 1), and an explicit limitations section. The paper does not claim theoretical guarantees, and the results are empirical; the main weakness is that the control experiments do not isolate the retrieval mechanism from the effect of adding expert data to the training set. With the additional control and more careful statistical reporting, the paper's claims would be credible.

major comments (3)
  1. [§5.2, Table 1] The fine-tuned DT control is insufficient to support the central mechanistic claim. The control fine-tunes DT only on the 60 expert trajectories, rather than training DT on the union of the full M/MR dataset and those 60 expert trajectories. Re:Frame's DT backbone is trained on the full non-expert dataset and receives expert information only through a latent retrieval/correction path. As a result, 'fine-tuned DT' differences could reflect either the absence of the retrieval mechanism or the absence of the full low-quality training data during expert exposure. Please add a DT baseline trained (or fine-tuned, with appropriate steps) on the full dataset plus the same 60 expert trajectories. If that baseline matches Re:Frame, the claim in §5.2 that the benefit 'does not come from sheer exposure to expert data' would be unsupported.
  2. [§5, Table 1] The quantitative evidence for 'consistent improvement' is thin: all results are mean ± std over three seeds, with no significance tests or confidence intervals. For Walker2d-MR, the improvement of Re:Frame over DT (69.8±1.7 vs 66.6±3.0) is within the variability one would expect from three seeds. Please report per-seed results or bootstrap/t-tests, and ideally increase the number of seeds. This is material because the paper's headline claim is 'consistent' improvement across tasks.
  3. [§5, Tables 1 and 2; §6] The Hopper-MR results complicate the 'few as 60 trajectories' narrative. With 60 trajectories Re:Frame is 13 points below DT (69.7 vs 82.7); at 45 trajectories it drops to 40.0 and at 30 to 3.0. The paper should explicitly state that the method does not work in one of four settings and is highly sensitive to buffer size/coverage. Currently the text says 'DT achieves the best mean return' for Hopper-MR, but Table 1 shows Fine-tuned DT achieves 84.8, which is higher than DT's 82.7; this needs correction. This does not invalidate the three positive results, but it qualifies the 'data-efficient' claim.
minor comments (6)
  1. [Abstract, §1, §5] The paper states that 60 trajectories are '0.1% of a 6000-trajectory dataset'. 60/6000 = 1%, not 0.1%. Please correct either the percentage or the dataset size; as written, the quantitative data-efficiency claim is arithmetically wrong.
  2. [Table 1] The caption says 'Best per row in bold', but in the Hopper-MR row the highest value is 84.8 (Fine-tuned DT), not 82.7 (DT). Ensure the bold formatting matches the stated rule and the text in §5.
  3. [§4, Algorithm 1] The AMB is introduced as B ∈ R^{T×N}, but T is also used for the trajectory length in the same algorithm. Clarify the dimensions: the buffer contains a set of latent vectors, not necessarily T rows tied to the current trajectory.
  4. [§4, AMB construction] The autoencoder is described as training on expert trajectories, but the text says 'sampling the triplet (Rt, ot, at) at timestep t from a dataset D' without specifying that D is the expert dataset. Make this explicit.
  5. [Figures 2 and 3] The labels 'Encoder (1)' and 'Encoder (2)' are not explained in the main text. Define these in the captions or text so the reader can follow which encoder belongs to the AMB and which to the DT.
  6. [Reproducibility] No code or data link is provided. For an empirical paper of this type, a public implementation would substantially aid reproducibility and adoption.

Circularity Check

0 steps flagged

No significant circularity: the improvements are empirical comparisons, not derived from fitted parameters or self-citation chains.

full rationale

The paper's central claim is empirical: Re:Frame (autoencoder + AMB retrieval) improves over a Decision Transformer baseline on D4RL when 60 expert trajectories are stored in memory. No step in Sections 3-5 derives the reported score from a fitted parameter or from a self-citation. The AMB is built by training an autoencoder with reconstruction losses L_R, L_o, L_a (Section 4), and the policy is trained on non-expert data with an additive correction a''_t = Linear(Decoder(argmin_h ||h*_t - h||^2)); the reported gains are measured against DT in Table 1. There is no equation in which the output equals the training input by construction: retrieval is a nearest-neighbor lookup and the correction is a learned module, not a copy of the label. The self-citations ([13], [14], [24]) appear only in related-work or introductory context and do not supply a load-bearing theorem or ansatz. The legitimate methodological concern, raised by the skeptic, is that the fine-tuned DT control in Section 5.2 trains only on the 60 expert trajectories rather than on the full non-expert dataset plus those 60 trajectories, leaving open whether the gain is from extra expert data rather than associative retrieval. That is an experimental-control gap, not a circularity: it does not make any predicted quantity equivalent to an input by construction. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method has no invented physical or conceptual entities; the AMB is a neural module. The load-bearing assumptions are about the quality and transferability of the latent space learned by the autoencoder, and the chosen buffer size. The paper does not report the autoencoder's latent dimension or training data, leaving a significant unstated degree of freedom.

free parameters (3)
  • AMB size (number of expert trajectories) = 60
    Chosen by hand; Table 2 shows severe performance collapse for Hopper when reduced to 45 or 30, so the central result is sensitive to this choice.
  • AE latent dimension and architecture = not reported
    The paper does not specify the latent dimensionality or the autoencoder architecture beyond dedicated encoders and a linear projection; this is a hidden degree of freedom that could affect retrieval quality.
  • Context length K = 60
    Transformer context length set to 60, matching the AMB size; no justification is given for this value.
axioms (3)
  • domain assumption Euclidean nearest-neighbor retrieval in the learned latent space is a valid content-based association criterion
    The method relies on the premise that similarity in the AE latent space corresponds to useful behavioral similarity for action correction. This is not derived or externally validated, and the Hopper-MR collapse at small AMB sizes suggests it can fail.
  • domain assumption D4RL Expert trajectories used to build the AMB are behaviorally compatible with the Medium and Medium-Replay evaluation distributions
    The paper uses expert trajectories from the same task but does not quantify the distribution shift between expert and medium/replay states; retrieval assumes the expert memory is relevant at query states.
  • domain assumption The frozen autoencoder provides stable, information-preserving representations for both memory storage and query encoding
    Section 4 states the AE parameters are frozen after stage one; if the AE features are not stable across the training distribution shift, the projection W must compensate, which is not verified.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Re:Frame -- Retrieving Experience From Associative Memory." pith.science (2026). https://pith.science/paper/BT226H54

@misc{pith2026250819344,
  author       = {Pith},
  title        = {Pith review of: Re:Frame -- Retrieving Experience From Associative Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BT226H54}},
  note         = {Machine review of arXiv:2508.19344}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame (Retrieving Experience From Associative Memory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them into decision-making; the same AMB is queried at evaluation. This requires no environment interaction and no modifications to the backbone architecture. On D4RL MuJoCo tasks, using as few as 60 expert trajectories (0.1% of a 6000-trajectory dataset), Re:Frame consistently improves over a strong Decision Transformer baseline in three of four settings, with gains up to +10.7 normalized points. These results show that Re:Frame offers a simple and data-efficient way to inject scarce expert knowledge and substantially improve offline RL from low-quality datasets.

Figures

Figures reproduced from arXiv: 2508.19344 by Aleksandr I. Panov, Alexey K. Kovalev, Daniil Zelezetsky, Egor Cherepanov.

Figure 1
Figure 1. Figure 1: Re:Frame overview. A standard offline RL agent trained on suboptimal data is augmented with an Associative Memory Buffer holding a few expert data. At each step, the agent queries mem￾ory, retrieves and decodes an expert data, and in￾tegrates it to guide the next action, enriching low￾quality training with scarce expert knowledge. Memory is fundamental to human intelligence, enabling us to accumulate exper… view at source ↗
Figure 2
Figure 2. Figure 2: Construction of the Associative Memory Buffer (AMB). Expert trajectories (Rt, ot, at) are encoded by an autoencoder into latent vectors (R′ t , o′ t , a′ t ), and concatenated into a hidden state ht. A linear projection maps ht into a compact latent representation, which is then decoded back to reconstruct (Rˆ t, oˆt, aˆt) with component-wise reconstruction losses. This training yields stable, information-… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 27 canonical work pages

  1. [1]

    Episodic memory: From mind to brain

    Endel Tulving. Episodic memory: From mind to brain. Annual Review of Psychology, 53(V ol- ume 53, 2002):1–25, 2002

    work page 2002

  2. [2]

    Larry R. Squire. Memory systems of the brain: A brief history and current perspective. Neu- robiology of Learning and Memory, 82(3):171–177, 2004. Multiple Memory Systems

    work page 2004

  3. [3]

    Working memory

    Alan Baddeley. Working memory. Current Biology, 20(4):R136–R140, 2010

    work page 2010

  4. [4]

    Memory: organization and control

    Howard Eichenbaum. Memory: organization and control. Annual review of psychology , 68(1):19–45, 2017. 7

    work page 2017

  5. [5]

    Prefrontal compu- tation as active inference

    Thomas Parr, Rajeev Vijay Rikhye, Michael M Halassa, and Karl J Friston. Prefrontal compu- tation as active inference. Cerebral Cortex, 30(2):682–695, 2020

    work page 2020

  6. [6]

    Active inference: the free energy principle in mind, brain, and behavior

    Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022

    work page 2022

  7. [7]

    Peter G. Polson. The American Journal of Psychology, 88(1):131–140, 1975

    work page 1975

  8. [8]

    The seven sins of memory - insights from psychology and cognitive neuro- science

    Daniel Schacter. The seven sins of memory - insights from psychology and cognitive neuro- science. The American psychologist, 54:182–203, 04 1999

    work page 1999

  9. [9]

    Rusu, Joel Veness, Marc G

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Ku- maran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein...

    work page 2015

  10. [10]

    Lillicrap, Fan Hui, L

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, L. Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354–359, 2017

    work page 2017

  11. [11]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    Pith/arXiv arXiv 2005

  12. [12]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:1273– 1286, 2021

    work page 2021

  13. [13]

    Recurrent action transformer with memory

    Egor Cherepanov, Alexey Staroverov, Dmitry Yudin, Alexey K Kovalev, and Aleksandr I Panov. Recurrent action transformer with memory. arXiv preprint arXiv:2306.09459, 2023

    work page arXiv 2023

  14. [14]

    A new perspective on transformers in online reinforcement learning for continuous control

    Nikita Kachaev, Daniil Zelezetsky, Alexey Kovalev, and Aleksandr Panov. A new perspective on transformers in online reinforcement learning for continuous control. In7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025

    work page 2025

  15. [15]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179– 1191, 2020

    work page 2020

  16. [16]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    Pith/arXiv arXiv 2006

  17. [17]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Plas: Latent action space for offline reinforcement learning

    Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719–1735. PMLR, 2021

    work page 2021

  19. [19]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  20. [20]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    Pith/arXiv arXiv 2004

  21. [21]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982

    1982

  22. [22]

    Neural turing machines

    Alex Graves. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014

    Pith/arXiv arXiv 2014

  23. [23]

    Associative recurrent memory transformer

    Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer. arXiv preprint arXiv:2407.04841, 2024. 8

    Pith/arXiv arXiv 2024

  24. [24]

    Unraveling the complexity of memory in rl agents: an approach for classification and evalua- tion

    Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K Kovalev, and Aleksandr I Panov. Unraveling the complexity of memory in rl agents: an approach for classification and evalua- tion. arXiv preprint arXiv:2412.06531, 2024

    work page arXiv 2024

  25. [25]

    Associative mem- ory based experience replay for deep reinforcement learning

    Mengyuan Li, Arman Kazemi, Ann Franchesca Laguna, and X Sharon Hu. Associative mem- ory based experience replay for deep reinforcement learning. In Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, pages 1–9, 2022

    work page 2022

  26. [26]

    Learning associative infer- ence using fast weight memory

    Imanol Schlag, Tsendsuren Munkhdalai, and J ¨urgen Schmidhuber. Learning associative infer- ence using fast weight memory. arXiv preprint arXiv:2011.07831, 2020

    Pith/arXiv arXiv 2011

  27. [27]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):1735–1780, nov 1997

    work page 1997

  28. [28]

    Self-attentive associative memory

    Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In Interna- tional conference on machine learning, pages 5682–5691. PMLR, 2020

    work page 2020

  29. [29]

    Associative search network: A reinforce- ment learning associative memory

    Andrew Barto, Richard Sutton, and Peter Brouwer. Associative search network: A reinforce- ment learning associative memory. Biological Cybernetics, 40:201–211, 05 1981

    work page 1981

  30. [30]

    Episodic reinforcement learning with associative memory

    Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations, 2020

    work page 2020

  31. [31]

    A minimalist approach to offline reinforcement learn- ing

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learn- ing. Advances in neural information processing systems, 34:20132–20145, 2021

    work page 2021

  32. [32]

    Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance

    Qisen Yang, Shenzhi Wang, Qihang Zhang, Gao Huang, and Shiji Song. Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance. IEEE Transactions on Neural Networks and Learning Systems, 35(11):16288–16300, 2023

    work page 2023

  33. [33]

    Hybrid policy optimization from imperfect demon- strations

    Hanlin Yang, Chao Yu, Siji Chen, et al. Hybrid policy optimization from imperfect demon- strations. Advances in Neural Information Processing Systems, 36:4653–4663, 2023

    work page 2023

  34. [34]

    Optimal transport for offline imitation learning

    Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisen- roth. Optimal transport for offline imitation learning. arXiv preprint arXiv:2303.13971, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Equivari- ant offline reinforcement learning

    Arsh Tangri, Ondrej Biza, Dian Wang, David Klee, Owen Howell, and Robert Platt. Equivari- ant offline reinforcement learning. arXiv preprint arXiv:2406.13961, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Guided data augmentation for offline reinforcement learning and imitation learning

    Nicholas E Corrado, Yuxiao Qu, John U Balis, Adam Labiosa, and Josiah P Hanna. Guided data augmentation for offline reinforcement learning and imitation learning. arXiv preprint arXiv:2310.18247, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal represen- tations by error propagation. 1986

    work page 1986

  38. [38]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026–5033. IEEE, 2012

    2012

  39. [39]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Herv ´e J ´egou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547, 2019

    work page 2019

  40. [40]

    Hybrid computing using a neural network with dynamic external memory

    Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi´nska, Sergio G´omez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016. 9 A S UPPLEMENTARY MATERIALS Table 3: Re:Frame hyperparameters. Hyperp...

    work page 2016

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.