pith. sign in

arxiv: 2606.21188 · v1 · pith:PEMFJ2UNnew · submitted 2026-06-19 · 💻 cs.RO

Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation

Kuancheng Wang , Seungho Yeom , Jinglin Cao , Yuheng Zhi , Nikhil Shinde , Michael Yip This is my paper

Pith reviewed 2026-06-26 14:20 UTC · model grok-4.3

classification 💻 cs.RO
keywords partially observable manipulationbehavioral memoryaction historycontact-rich tasksrobot policy learningmemory modulevisuomotor control
0
0 comments X

The pith

CAMP learns a compressed latent memory from a robot's own past actions to succeed at contact-rich manipulation tasks that are only partially observable from vision.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon contact-rich manipulation is partially observable because one camera view rarely shows prior attempts or full interaction history. Standard visuomotor policies therefore fail when they cannot remember what they have already tried. The paper introduces CAMP, which trains a memory module on the robot's own action sequence to produce a compact latent representation of behavioral history. This representation is used to contextualize the next action, implicitly tracking task progress and recovering from failures with no extra labels or external memory. On four real-robot setups and two new simulation benchmarks the method shows large gains over baselines that lack such memory.

Core claim

By training a memory module to maintain a compressed representation of past actions, CAMP encodes a latent behavioral memory of all prior interactions that can then be used to better contextualize future actions, allowing the policy to implicitly track generalized task progress and learn from failed attempts without any additional supervision.

What carries the argument

Compressed Action Memory Policy (CAMP), a memory module that compresses the robot's action history into a latent behavioral representation for use in the policy.

If this is right

  • Robots can perform long-horizon contact-rich tasks under partial observability without explicit state tracking.
  • Learning occurs directly from failed attempts encoded in the action history.
  • No external supervision or additional sensors are required beyond the standard action stream.
  • The same memory module works across real-robot setups and the introduced simulation benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same compression idea could be tested on navigation or assembly tasks where action sequences also encode progress.
  • Combining CAMP memory with vision-language-action models might reduce their reliance on long context windows.
  • If the latent memory proves task-general, it could serve as a drop-in replacement for recurrent state in other robot learning pipelines.

Load-bearing premise

A robot's own action history is a sufficiently rich self-supervised signal that can be compressed into a latent memory capable of tracking task progress.

What would settle it

CAMP produces no substantial improvement over vision-only baselines on the Memory-T-Bench or Memory-Manip-Bench contact-rich tasks.

Figures

Figures reproduced from arXiv: 2606.21188 by Jinglin Cao, Kuancheng Wang, Michael Yip, Nikhil Shinde, Seungho Yeom, Yuheng Zhi.

Figure 1
Figure 1. Figure 1: From 0% to 70% success: a compressed memory of past actions resolves partial observability. Long horizon, contact-rich manipulation tasks are partially observable, so memoryless policies act on an aliased signal and fail on tasks like Push-T-Multi-Goals (left). CAMP (right) pretrains a memory module Eθ to reconstruct the past action trajectory, yielding a compressed code z that conditions a diffusion actio… view at source ↗
Figure 2
Figure 2. Figure 2: The Training Pipeline of CAMP. We model vision-based manipulation as a partially observ￾able Markov decision process (POMDP) (S, A, T , O, Z). At time t, the environment is in an underlying state st ∈ S. When the robot applies an action at ∈ A, the state evolves under a transition function T (st+1 | st, at), and the robot receives only an observation ot+1 ∈ O drawn from an observation function Z(ot+1 | st+… view at source ↗
Figure 4
Figure 4. Figure 4: Memory-Manip-Bench partially observable 3D manipulation tasks Models Push-T Multi-Goals Swap-Direct Swap-Shuffle Find-Track DP 94.0 56.0 74.0 48.0 62.0 CAMP (ours) 98.0 94.0 94.0 92.0 90.0 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Memory-T-Bench tasks’ initial states. Memory-T-Bench. Four PushT-derived vari￾ants share the same contact-rich dynamics. The first three are track-task-progress tasks: in Multi￾Goals, the agent pushes the T-block into three goal regions in any order without repetition; Swap-Direct and Swap-Shuffle require two T￾blocks exchange positions, but mid-episode frames do not encode the original assignment, which S… view at source ↗
Figure 5
Figure 5. Figure 5: Success rate of DP and CAMP with increasing number of demos. CAMP converts more demonstrations into higher success; the memoryless policy does not [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Real-robot tasks visualized in multi-stage or with potential failed attempts. and action head from scratch reaches only 66%, since the action loss drives the hidden state toward immediate prediction and never installs long-horizon memory. Pretraining the memory module and then freezing it when stitched to DP does better at 80%, confirming that the pretrained representation is already informative, but it le… view at source ↗
Figure 7
Figure 7. Figure 7: Real-robot setup. The white bounding boxes show the real-world camera set-up. We use a Franka Emika Panda 7-DoF arm with two Intel RealSense D435 cameras: a third￾person view mounted on a fixed stand and a wrist-mounted view rigidly attached to the end-effector ( [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
read the original abstract

Long horizon, contact-rich manipulation is inherently partially observable. This is as a single visual observation rarely captures a robot's full action context, including prior attempts, interactions, or progress. Consequently, standard visuomotor policies or vision-language-action models are prone to struggle in such tasks due to a lack of memory. To address this, we introduce Compressed Action Memory Policy (CAMP) based on the insight that a robot's own action history serves as a highly informative, self-supervised signal, enabling the policy to learn a robust, compact history representation. In our approach, we train a memory module to maintain a compressed representation of past actions, forcing it to encode a latent behavioral memory of all the robot's past interactions that can then be used to better contextualize future actions. This allows our approach to implicitly track generalized task progress and learn from failed attempts without any additional supervision, or external oversight. We evaluate CAMP across four real-robot setups and two novel simulation benchmarks: Memory-T-Bench and Memory-Manip-Bench. By demonstrating substantial gains over state-of-the-art baselines, CAMP is, to our knowledge, the first policy to demonstrate substantial success on contact-rich partially observable manipulation tasks purely through learned memory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper introduces Compressed Action Memory Policy (CAMP), a visuomotor policy that trains a memory module to compress a robot's action history into a compact latent behavioral memory. This representation is intended to implicitly track task progress and recover from failures in long-horizon, contact-rich, partially observable manipulation without external supervision. The method is evaluated on four real-robot setups plus two new simulation benchmarks (Memory-T-Bench and Memory-Manip-Bench), with the central claim that CAMP achieves substantial gains over state-of-the-art baselines and is the first policy to demonstrate substantial success on such tasks purely through learned memory.

Significance. If the empirical claims hold, the work would be significant for robotics because it supplies a self-supervised mechanism for maintaining behavioral memory in POM settings, potentially reducing reliance on privileged state or additional sensors for contact-rich tasks.

minor comments (2)
  1. [Abstract] Abstract: the claims of 'substantial gains' and 'substantial success' are stated without any numerical results, baseline names, or task-success metrics; adding at least one quantitative highlight would strengthen the summary.
  2. [Abstract] The two novel benchmarks (Memory-T-Bench, Memory-Manip-Bench) are introduced but their task definitions, observation spaces, and success criteria are not summarized in the abstract; a brief description would improve accessibility.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our work on Compressed Action Memory Policy (CAMP) and the recommendation for minor revision. The recognition of its potential significance for self-supervised behavioral memory in POM manipulation tasks is appreciated.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The abstract and available context present CAMP as an empirical policy that compresses action history into a latent memory representation for partially observable tasks. No equations, loss functions, architectural derivations, or self-citation chains are supplied that would allow any claimed prediction or uniqueness result to reduce to its own inputs by construction. The central contribution is framed as a training procedure and experimental evaluation on real-robot and simulation benchmarks, with no load-bearing step that equates outputs to fitted parameters or prior self-referential definitions. The derivation chain is therefore self-contained as an architectural and empirical proposal.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no information on free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5761 in / 1022 out tokens · 30456 ms · 2026-06-26T14:20:18.283410+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 1 canonical work pages

  1. [1]

    H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan. Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation, 2025. URLhttps://arxiv.org/abs/2501.18564

    arXiv 2025

  2. [2]

    Y . Dai, H. Fu, J. Lee, Y . Liu, H. Zhang, J. Yang, C. Finn, N. Fazeli, and J. Chai. Robomme: Benchmarking and understanding memory for robotic generalist policies, 2026. URL https: //arxiv.org/abs/2603.04639

    Pith/arXiv arXiv 2026

  3. [3]

    T. Chen, Y . Wang, M. Li, Y . Qin, H. Shi, Z. Li, Y . Hu, Y . Zhang, K. Wang, Y . Chen, H. Wang, R. Xu, R. Wu, Y . Mu, Y . Yang, H. Dong, and P. Luo. Rmbench: Memory-dependent robotic manipulation benchmark with insights into policy design, 2026. URL https://arxiv.org/ abs/2603.01229

    arXiv 2026

  4. [4]

    Cherepanov, N

    E. Cherepanov, N. Kachaev, A. K. Kovalev, and A. I. Panov. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning, 2026. URL https: //arxiv.org/abs/2502.10550

    arXiv 2026

  5. [5]

    Chung, T

    N. Chung, T. Hanyu, T. Nguyen, H. Le, F. Bumgarner, D. M. H. Nguyen, K. V o, K. Yamazaki, C. Rainwater, T. Kieu, A. Nguyen, and N. Le. Rethinking progression of memory state in robotic manipulation: An object-centric perspective, 2025. URL https://arxiv.org/abs/ 2511.11478

    arXiv 2025

  6. [6]

    M. Lin, X. Liang, B. Lin, L. Jingzhi, Z. Jiao, K. Li, Y . Sun, W. Liufu, Y . Ma, Y . Liu, S. Zhao, Y . Zhuang, and X. Liang. Echovla: Synergistic declarative memory for vla-driven mobile manipulation, 2026. URLhttps://arxiv.org/abs/2511.18112

    arXiv 2026

  7. [7]

    H. Shi, B. Xie, Y . Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang. Memo- ryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation,

  8. [8]

    URLhttps://arxiv.org/abs/2508.19236

    Pith/arXiv arXiv

  9. [9]

    H. Li, S. Yang, Y . Chen, X. Chen, X. Yang, Y . Tian, H. Wang, T. Wang, D. Lin, F. Zhao, and J. Pang. Cronusvla: Towards efficient and robust manipulation via multi-frame vision-language- action modeling, 2025. URLhttps://arxiv.org/abs/2506.19816

    arXiv 2025

  10. [10]

    R. Li, W. Guo, Z. Wu, C. Wang, H. Deng, Z. Weng, Y .-P. Tan, and Z. Wang. Map-vla: Memory- augmented prompting for vision-language-action model in robotic manipulation, 2025. URL https://arxiv.org/abs/2511.09516

    arXiv 2025

  11. [11]

    Torne, K

    M. Torne, K. Pertsch, H. Walke, K. Vedder, S. Nair, B. Ichter, A. Z. Ren, H. Wang, J. Tang, K. Stachowicz, K. Dhabalia, M. Equi, Q. Vuong, J. T. Springenberg, S. Levine, C. Finn, and D. Driess. Mem: Multi-scale embodied memory for vision language action models, 2026. URL https://arxiv.org/abs/2603.03596

    arXiv 2026

  12. [12]

    Sridhar, J

    A. Sridhar, J. Pan, S. Sharma, and C. Finn. Memer: Scaling up memory for robot control via experience retrieval, 2025. URLhttps://arxiv.org/abs/2510.20328

    arXiv 2025

  13. [13]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong. Vision-language foundation models as effective robot imitators, 2024. URL https://arxiv.org/abs/2311.01378

    Pith/arXiv arXiv 2024

  14. [14]

    Long short-term memory

    S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural Computation, 9(8): 1735–1780, Nov 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.8.1735

    work page doi:10.1162/neco.1997.9.8.1735 1997

  15. [15]

    M. S. Mark, J. Liang, M. Attarian, C. Fu, D. Dwibedi, D. Shah, and A. Kumar. Bpp: Long- context robot imitation learning by focusing on key history frames, 2026. URL https:// arxiv.org/abs/2602.15010

    arXiv 2026

  16. [16]

    Torne, A

    M. Torne, A. Tang, Y . Liu, and C. Finn. Learning long-context diffusion policies via past-token prediction, 2025. URLhttps://arxiv.org/abs/2505.09561

    arXiv 2025

  17. [17]

    M. Lin, P. Ding, S. Wang, Z. Zhuang, Y . Liu, X. Tong, W. Song, S. Lyu, S. Huang, and D. Wang. Hif-vla: Hindsight, insight and foresight through motion representation for vision-language- action models, 2026. URLhttps://arxiv.org/abs/2512.09928. 10

    Pith/arXiv arXiv 2026

  18. [18]

    Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. Univla: Learning to act anywhere with task-centric latent actions, 2025. URL https://arxiv.org/abs/2505. 06111

    2025

  19. [19]

    Zheng, Y

    R. Zheng, Y . Liang, S. Huang, J. Gao, H. D. III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies,

  20. [20]

    URLhttps://arxiv.org/abs/2412.10345

    Pith/arXiv arXiv

  21. [21]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024. URL https://arxiv.org/ abs/2303.04137

    Pith/arXiv arXiv 2024

  22. [22]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. URLhttps://arxiv.org/abs/2304.13705

    Pith/arXiv arXiv 2023

  23. [23]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. V...

    Pith/arXiv arXiv 2025

  24. [24]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models, 2020. URL https: //arxiv.org/abs/2006.11239

    Pith/arXiv arXiv 2020

  25. [25]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models, 2022. URL https: //arxiv.org/abs/2010.02502

    Pith/arXiv arXiv 2022

  26. [26]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  27. [27]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023. URL https: //arxiv.org/abs/2212.09748. 11 A Training Details A.1 Visuomotor Policies The CAMP, Diffusion Policy (DP), and ACT baselines are trained from scratch independently per task on a single H100 GPU, with the same recipe applied across simulation and real-world tasks. For CAMP...

    Pith/arXiv arXiv 2023