REVIEW 3 major objections 4 minor 56 references
A robot policy can keep minute-long visual history by turning each past camera frame into one token from its own vision encoder, lifting success from about 32% to 84% without new memory modules.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 23:33 UTC pith:MFESFH7G
load-bearing objection Solid systems paper: one native token per past frame-view, action-supervised under a frozen VLA, with large sim/real gains and clean efficiency numbers; residual risk that Stage-1 under-specifies abstract progress on a narrow suite. the 3 major comments →
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Native Memory Compression—compressing each past frame-view into one token produced by the VLA’s own vision encoder and trained under the frozen policy’s action loss—lets a pretrained single-frame VLA attend over minute-scale visual history with negligible latency, converting it into a memory-augmented policy that substantially outperforms both VLM-driven and external-module baselines.
What carries the argument
Native Memory Compression: a memory tokenizer derived from the pretrained vision encoder that, via a learnable query token and linear projection, maps each historical frame-view to one action-supervised summary token; a two-stage pipeline freezes the VLA while learning the tokenizer, then freezes the tokenizer while fine-tuning the policy with the cached tokens appended to its input sequence.
Load-bearing premise
That the ordinary action-prediction loss of a frozen policy is enough supervision to force each single-token summary to keep the fine-grained historical details (counts, original pads, scanned items) that later actions actually need.
What would settle it
On a held-out long-horizon task whose critical cue is a subtle visual detail early in the episode, measure whether success collapses when that cue is removed or scrambled after the tokenizer has already produced its single tokens; if performance stays high, the compression is not truly retaining the claimed information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. NativeMEM converts a pretrained single-frame VLA into a memory-augmented policy by Native Memory Compression: the VLA’s own vision encoder is repurposed (with a learnable query and projection) to map each historical frame–view into one native memory token (Eqs. 5–7), which is appended to the original token sequence. Stage 1 freezes the VLA and trains the tokenizer with the native action loss on mixed standard and memory-demanding data (Eq. 9); Stage 2 freezes the tokenizer and finetunes the VLA on task demos with cached tokens (Eq. 10). On simulation (RMBench + button tasks) average success rises from 32.4% (Mem-0) to 84.0%; on real robots (including an unseen checkout task) overall success reaches 98.7%, with low latency/GPU cost and competitive results using ~20% of prior training data. Ablations (Tab. 3) and attention maps (Figs. 5–6) support the design choices.
Significance. If the result holds, the paper offers a practical, architecture-light path to long-horizon memory for pretrained VLAs: one-token-per-frame compression that reuses the existing visual pathway and attention stack, avoiding external planners or freshly initialized memory modules. The two-stage pipeline, strong sim/real gains, inference-efficiency curves (Fig. 4a), data-efficiency results (Fig. 4b), and qualitative attention evidence are concrete contributions that the community can build on. The work is empirical rather than theoretical; its value is the demonstrated conversion recipe and the efficiency–horizon trade-off it unlocks for non-Markovian manipulation.
major comments (3)
- §3.2 Eqs. 5–9 and Limitations: the central claim that action loss on a frozen VLA is sufficient to force each frame–view into a single token that retains fine-grained non-Markovian cues (counts, original pads, scanned items) is only partially stress-tested. Tab. 3 shows Stage-1 and long/dense history matter, and Figs. 5–6 are consistent with task-relevant focus, but there is no controlled probe that the one-token summary preserves the specific detail required rather than a coarse progress proxy that works on this suite. A short diagnostic (e.g., recovery of button-press counts or original-pad identity from memory tokens alone, or a held-out task that forces finer discrimination) would make the compression claim load-bearing rather than inferred from end-task success.
- Tables 1–2: success rates are reported as point estimates with no trial counts, standard errors, or confidence intervals. Given the large jumps (32.4%→84.0% sim; ~98.7% real) and the fact that several baselines are reproduced (HAMLET*, MEM-short*) on the same backbone and limited demos, uncertainty quantification is needed to support the ranking and the claim of consistent outperformance.
- §4.1 Baselines and Tab. 1–2: fairness of the reproduced external-memory baselines is asserted but under-specified. MEM-short is noted as originally designed for large-scale retraining; training it only on the same limited demos as NativeMEM may understate that line of work. A short sensitivity note (or an upper-bound run with more data where available) would clarify how much of the gap is method vs. training regime.
minor comments (4)
- Typo consistency: “Externel Memory Modules” in Tab. 1 header; “NATIVEMEM” vs “NativeMEM” capitalization varies across abstract, title, and body.
- Fig. 4(a) axis units and exact history lengths used for the 100 ms / 32 GB operating points should be stated in the caption or text so the efficiency claim is reproducible.
- §3.2: the composition of I_t (first frame + recent window + current) is described qualitatively; giving the default window length and sampling rate used in Stage 1 would help replication.
- Related Work: a brief explicit contrast with concurrent compressed-history VLAs (beyond HAMLET/MEM) would situate the “native” claim more sharply.
Circularity Check
No circularity: standard two-stage supervised finetuning of a memory tokenizer under frozen-VLA action loss, evaluated on held-out task success.
full rationale
NativeMEM is an empirical systems/method paper, not a first-principles derivation. The load-bearing chain is: (i) define one-token-per-frame memory via the VLA vision encoder plus a learnable query (Eqs. 5–7); (ii) train that tokenizer with the frozen VLA’s native action loss on mixed data (Eq. 9); (iii) cache tokens and finetune the VLA on target demos (Eq. 10); (iv) report success rates vs. baselines on simulation and real-robot memory tasks (Tabs. 1–2), plus ablations (Tab. 3) and efficiency plots. None of these steps reduces a claimed prediction to a fitted constant or to a definition of the target. Action-supervised tokens are optimized and then measured on independent task success; that is ordinary supervised learning, not self-definitional circularity. Reproducing HAMLET*/MEM-short* on the same π0.5 backbone and demo budget is experimental control, not a self-citation uniqueness theorem. No uniqueness result is imported from overlapping authors to force the design. The Limitations section’s own caveat about action-loss under-specifying abstract memory is a correctness/generalization risk, not circularity. Score 0 with empty steps is the honest finding.
Axiom & Free-Parameter Ledger
free parameters (4)
- Per-frame memory token budget M =
1
- History index set I_t (first frame + recent window + current) =
first frame + recent window + current
- Memory update frequency / horizon length at inference =
dense updates; minute-level horizon claimed
- Stage-1 data mixture (standard + memory-demanding, sim + real)
axioms (4)
- domain assumption Long-horizon manipulation is non-Markovian: identical current (o_t, s_t, ℓ) can require different optimal actions given different histories H_t (Eq. 2).
- ad hoc to paper Action prediction loss of a frozen pretrained VLA is sufficient supervision to learn compact, action-relevant single-token frame summaries (Eq. 9).
- domain assumption Memory tokens produced by a vision encoder initialized from the VLA remain interpretable by that VLA’s attention when appended to the original token sequence (Eqs. 5–8).
- ad hoc to paper Reproduced external-memory baselines (HAMLET*, MEM-short*) on π0.5 with the same limited demos are fair comparators for the reported gains.
invented entities (2)
-
Native memory token m^v_τ (one summary per frame-view)
no independent evidence
-
Memory tokenizer (E_mem, q_mem, W_mem, b_mem) derived from the VLA vision encoder
no independent evidence
read the original abstract
How can pretrained Vision-Language-Action (VLA) models retain long-horizon visual histories with high-frequency updates without sacrificing efficiency? Existing approaches rely on external memory management, which restrains either the memory horizon or the reactiveness of pretrained policies. To this end, we present NativeMEM, a VLA policy that features long-term and real-time updated memory. At its core is an efficient memory encoding scheme, Native Memory Compression, which repurposes the VLA's own vision encoder to compress each historical frame from each camera view into a single token. Appended to the input sequence, these memory tokens enable the pretrained VLA to attend over long-term history with negligible latency overhead, requiring neither an external planner nor a freshly initialized memory module. To align the memory tokens with the pretrained policy, we first develop a generic memory tokenizer under the supervision of a frozen VLA on memory-demanding data, and then unfreeze the VLA for task-specific fine-tuning. NativeMEM consistently outperforms prior methods, boosting success rates from 32.4% to 84.0% in simulation and up to 98.7% on real robots, while maintaining low inference latency and GPU memory usage. Notably, NativeMEM exhibits high data efficiency by achieving competitive results with prior arts using only 20% of the training data.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, 2023
work page 2023
-
[2]
M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[3]
$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[4]
$\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al.π 0.5: a vision-language-action model with Open-World generalization. arXiv preprint arXiv:2504.16054, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[5]
Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. UniVLA: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
- [6]
-
[7]
C. Ni, C. Chen, X. Wang, Z. Zhu, W. Zheng, B. Wang, T. Chen, G. Zhao, H. Li, Z. Dong, et al. SwiftVLA: Unlocking spatiotemporal dynamics for lightweight vla models at minimal overhead.CVPR 2026, 2026
work page 2026
- [8]
- [9]
- [10]
-
[11]
PaliGemma: A versatile 3B VLM for transfer
L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdul- mohsin, M. Tschannen, E. Bugliarello, et al. PaliGemma: A versatile 3b VLM for transfer. arXiv preprint arXiv:2407.07726, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[12]
SmolVLM: Redefining small and efficient multimodal models
A. Marafioti, O. Zohar, M. Farr ´e, M. Noyan, E. Bakouch, P. Cuenca, C. Zakka, L. B. Allal, A. Lozhkov, N. Tazi, et al. SmolVLM: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[13]
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[14]
S. Zhou, A. Vilesov, X. He, Z. Wan, S. Zhang, A. Nagachandra, D. Chang, D. Chen, X. E. Wang, and A. Kadambi. VLM4D: Towards spatiotemporal awareness in vision language mod- els. InProceedings of the IEEE/CVF international conference on computer vision, 2025
work page 2025
- [15]
- [16]
-
[17]
Y . Wu, S. Liang, C. Zhang, Y . Wang, Y . Zhang, H. Guo, R. Tang, and Y . Liu. From human memory to AI memory: A survey on memory mechanisms in the era of LLMs.arXiv preprint arXiv:2504.15965, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
- [18]
-
[19]
A. Sridhar, J. Pan, S. Sharma, and C. Finn. MemER: Scaling up memory for robot control via experience retrieval.arXiv preprint arXiv:2510.20328, 2025
- [20]
-
[21]
T. Chen, Y . Wang, M. Li, Y . Qin, H. Shi, Z. Li, Y . Hu, Y . Zhang, K. Wang, Y . Chen, et al. RM- Bench: Memory-Dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026
work page internal anchor Pith review arXiv 2026
- [22]
- [23]
-
[24]
Z. Zeng, F. Ding, H. Yang, and X. Li. HELM: Harness-Enhanced long-horizon memory for vision-language-action manipulation.arXiv preprint arXiv:2604.18791, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
- [25]
-
[26]
M. Koo, D. Choi, T. Kim, K. Lee, C. Kim, Y . Seo, and J. Shin. HAMLET: Switch your vision-language-action model into a history-aware policy.arXiv preprint arXiv:2510.00695, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[27]
M. Lin, P. Ding, S. Wang, Z. Zhuang, Y . Liu, X. Tong, W. Song, S. Lyu, S. Huang, and D. Wang. HiF-VLA: Hindsight, Insight and Foresight through motion representation for Vision-Language-Action models.arXiv preprint arXiv:2512.09928, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
- [28]
-
[29]
G. Nangue Tasse, M. Riemer, B. Rosman, and T. Klinger. Beyond sliding windows: Learning to manage memory in non-markovian environments.arXiv e-prints, 2025
work page 2025
-
[30]
H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan. SAM2Act: In- tegrating visual foundation model with a memory architecture for robotic manipulation.arXiv preprint arXiv:2501.18564, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[31]
Y . Yang, H. Yang, J. Zhou, P. Chen, H. Zhang, Y . Du, and C. Gan. 3D-mem: 3D scene memory for embodied exploration and reasoning. InProceedings of the Computer Vision and Pattern Recognition Conference, 2025. 11
work page 2025
-
[32]
Y . Gao, J. Liu, S. Li, and S. Song. Gated memory policy.arXiv preprint arXiv:2604.18933, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[33]
Z. Chen, Y . Hu, Z. Fu, Z. Li, J. Huang, Q. Huang, and Y . Wei. INTENT: Invariance and discrimination-aware noise mitigation for robust composed image retrieval. InProceedings of the AAAI Conference on Artificial Intelligence, 2026
work page 2026
-
[34]
Z. Li, Y . Hu, Z. Chen, S. Zhang, Q. Huang, Z. Fu, and Y . Wei. HABIT: Chrono-synergia robust progressive learning framework for composed image retrieval. InProceedings of the AAAI Conference on Artificial Intelligence, 2026
work page 2026
-
[35]
Z. Li, Y . Hu, Z. Chen, Q. Huang, G. Qiu, Z. Fu, and M. Liu. ReTrack: Evidence-driven dual- stream directional anchor calibration network for composed video retrieval. InProceedings of the AAAI Conference on Artificial Intelligence, 2026
work page 2026
- [36]
-
[37]
J. Yang, K. Lin, J. Li, W. Zhang, T. Lin, L. Wu, Z. Su, H. Zhao, Y .-Q. Zhang, L. Chen, et al. RISE: Self-improving robot policy with compositional world model.arXiv preprint arXiv:2602.11075, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
- [38]
- [39]
- [40]
- [41]
- [42]
-
[43]
J. Wen, Y . Zhu, J. Li, M. Zhu, Z. Tang, K. Wu, Z. Xu, N. Liu, R. Cheng, C. Shen, et al. TinyVLA: Towards fast, data-efficient vision-language-action models for robotic manipula- tion.IEEE Robotics and Automation Letters, 2025
work page 2025
-
[44]
DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
W. Zhang, H. Liu, Z. Qi, Y . Wang, X. Yu, J. Zhang, R. Dong, J. He, H. Wang, Z. Zhang, et al. DreamVLA: a vision-language-action model dreamed with comprehensive world knowledge. arXiv preprint arXiv:2507.04447, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[45]
B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. SpatialVLM: En- dowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024
work page 2024
-
[46]
RT-1: Robotics Transformer for Real-World Control at Scale
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. RT-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[47]
M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025. 12
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[48]
J. Cen, C. Yu, H. Yuan, Y . Jiang, S. Huang, J. Guo, X. Li, Y . Song, H. Luo, F. Wang, et al. WorldVLA: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[49]
AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models
Y . Hu, J.-N. Zaech, N. Nikolov, Y . Yao, S. Dey, G. Albanese, R. Detry, L. Van Gool, and D. Paudel. AR-VLA: True autoregressive action expert for Vision-Language-Action models. arXiv preprint arXiv:2603.10126, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[50]
F. Peller-Konrad, R. Kartmann, C. R. Dreher, A. Meixner, F. Reister, M. Grotz, and T. As- four. A memory system of a robot cognitive architecture and its implementation in ArmarX. Robotics and Autonomous Systems, 2023
work page 2023
-
[51]
H. Shi, B. Xie, Y . Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic ma- nipulation.arXiv preprint arXiv:2508.19236, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
- [52]
- [53]
-
[54]
X. Guo, C. Jiang, H. B. Kim, Y . Sun, Y . Xiao, Y . Han, and J. Yang. Chameleon: Episodic memory for long-horizon robotic manipulation.arXiv preprint arXiv:2603.24576, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[55]
X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y . Feng, Y . Zheng, J. Zou, Y . Chen, J. Zeng, et al. X- VLA: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
- [56]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.