Addressable Memory for Video World Models

Aljo\v{s}a O\v{s}ep; Despoina Paschalidou; James Lucas; Jonathan Lorraine; Laura Leal-Taix\'e; Olga Russakovsky; Sven Elflein; Xindi Wu

REVIEW 3 major objections 4 minor 120 references

Addressable Memory for Video World Models

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Long-horizon video memory fails because the cache stops being addressable; WorldTrace fixes it by assigning every compressed slot a distinct in-distribution virtual position, improving temporal consistency by +15.5% and episodic recall by…

desk verdict A credible training-free memory patch for video world models; addressability story is plausible and partly supported, but the paper never directly shows the model reads synthetic slots as real frames. read the letter →

arxiv 2608.07408 v1 pith:E3EX2SGQ submitted 2026-08-07 cs.CV cs.LG

Xindi Wu , Sven Elflein , James Lucas , Olga Russakovsky , Laura Leal-Taixé , Despoina Paschalidou , Jonathan Lorraine , Aljoša Ošep This is my paper · ORCID

classification cs.CVcs.LG

keywords videoworldmodelsvisualpersistenceKVcachecompressionrotarypositionembeddingaddressablememoryepisodicrecalltemporalcoherencetraining-free

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive video world models carry visual history in a Key-Value cache, but once generation exceeds the training horizon, the rotary position offsets used by attention leave the trained range and the model can no longer reliably read past frames. The paper argues this is fundamentally an addressability problem, not a storage problem, and that naive cache compression makes it worse by averaging RoPE-rotated keys whose phases cancel. WorldTrace is a training-free cache design that keeps each compressed memory slot at a fixed, in-distribution virtual position relative to the current frame, and it couples that with two content writers: WorldTrace-Field averages keys in the canonical unrotated domain for temporal coherence, and WorldTrace-Landmark stores verbatim scene-entry frames for episodic recall. On the new LoopBench revisit benchmark, WorldTrace-Field improves temporal consistency by +15.5% and WorldTrace-Landmark improves episodic recall by +19.5%, extending visually persistent generation without any retraining.

What carries the argument

The central mechanism is the WorldTrace slot indexing and canonical key domain. Each summary slot is assigned a virtual position $t^v_s = q - (L_{\text{attn}} - 1 - s)$, fixed by slot rank relative to the current query position $q$, keeping all summary offsets inside the trained range at any horizon and preventing the collapse that plagues block-relative schemes that cap offsets. Keys are unrotated to their canonical form, averaged or frozen there, and re-rotated at attention time to the slot's virtual position; this avoids the phase cancellation of naive RoPE-space averaging and preserves mean attention scores. WorldTrace-Field instantiates this with canonical key averaging over contiguous temporal groups, while WorldTrace-Landmark freezes verbatim canonical keys at detected scene-entry events.

What would settle it

Measure attention weights on the summary slots at horizons N=32 to N=256: if WorldTrace slots do not receive attention comparable to what verbatim frames at the same virtual positions receive, the 'read as real' premise fails. Alternatively, run WorldTrace-Field at N=48 while randomizing only the virtual position assignment (but keeping slot content identical): if TempSSIM does not degrade against the slot-rank assignment, addressability is not the binding constraint claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is that long-horizon failure in autoregressive video world models is fundamentally a problem of addressability: no memory scheme can improve visual persistence if past observations cannot be reliably accessed once they fall outside the context window. Because temporal Rotary Positional Embeddings (RoPE) are queried at offsets beyond the trained range, the model struggles to retrieve stored visual information through attention, and naively compressing the cache in the RoPE-rotated space corrupts memory by averaging incompatible positional phases. WorldTrace restores addressability by assigning each summary slot a distinct, in-distribution virtual position, and it maintains informativeness by writing compressed content in the canonical (unrotated) key domain. The result is a training-free cache that extends the effective memory horizon: WorldTrace-Field improves temporal consistency by +15.5% and WorldTrace-Landmark improves episodic recall by +19.5% on LoopBench, without retraining.

Load-bearing premise

The central premise is that a pretrained model, trained on verbatim frames at every cache position, will treat a compressed summary key placed at a fixed in-distribution offset as if it were a real frame at that position; the model must extract meaning from averaged keys even though such averaged keys never appeared in training.

Editorial extensions

If this is right

Sliding-window caches and naive compression fail not because history is missing but because distant keys are unreadable; fixing position alone recovers coherence, and fixing content alone is insufficient.
A fixed-size cache can sustain visually persistent generation to minute-scale rollouts without any training, which is relevant for interactive world models and closed-loop simulators.
In one shared cache, the slot budget can be split between coherence-oriented summaries and recall-oriented landmarks, so a single rollout benefits from both temporal consistency and episodic recall.
The addressability view extends beyond video: any autoregressive model with temporal RoPE and a fixed KV budget faces the same OOD-offset failure, so slot-rank position assignment is a general training-free remedy.
The streaming summary writer variant shows that the coherence gain is attributable to slot-rank positions rather than to extra retained statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's framing suggests a practical design rule: when compressing KV caches for RoPE-based models, content should be merged in the canonical domain and each merged token should receive a distinct, horizon-independent virtual position; this is a testable recipe for other model families.
The 'read as real' premise suggests a diagnostic experiment the paper does not run: measuring attention weights on summary slots across horizons and per-frequency retrieval to confirm the model actually reads averaged keys as if they were verbatim frames at that position.
The addressability analysis implies that training-time position-aware fine-tuning on the slot-rank offsets, as the authors suggest in their future directions, could tighten the approximation further and allow larger cache budgets.
A direct extension would apply WorldTrace to non-temporal RoPE axes (spatial height/width or camera-conditioned positions), since the same phase-cancellation and OOD-offset issues may arise in those dimensions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the punchline: this is a credible training-free fix for a real failure mode in autoregressive video world models—RoPE offsets beyond the training horizon make cached tokens unreadable, and naive averaging in the rotated domain cancels phases. WorldTrace's combination of slot-rank virtual positions and canonical-domain key averaging is genuinely new, and the formal parts (Props 1 and 2) are correct. The experiments support the qualitative story: position assignment matters even when content is held fixed (Table 1), and the gains on LoopBench are consistent.

What the paper does well: it identifies two coupled problems—addressability and informativeness—and designs a unified framework. The math is elementary but sound, and the ablation isolating position assignment from content is the right kind of evidence. The LoopBench benchmark is a useful addition, with controlled geometry and difficulty tiers. The cross-architecture result on LingBot-World adds confidence that the finding isn't specific to one model.

Where are the soft spots? The paper never directly measures attention weights on the summary slots, so we don't know if the model actually reads the averaged keys as real frames. The strongest indirect evidence is Table 1, where only positions vary, and WorldTrace beats the alternatives. But Table 9 in the appendix complicates the Landmark story: Landmark+Block-relative already reaches PAC 0.929 at N=32, so most of the recall gain comes from verbatim content, not from the slot-rank positions. That doesn't kill the paper—Field is still a position-driven result—but it means the 'addressability is restored' claim is better supported for averaged summaries than for landmarks.

Other gaps: Table 2 has no error bars at N=32 and shows a mixed signal (higher TempSSIM but higher Scene Drift than centroid variants); the scene-entry threshold tau is not reported in the main text; no code is provided. These are fixable but should be addressed.

Who is this for? People working on long-horizon video generation, especially interactive world models. It's a solid idea worth engaging with, and the benchmark could be useful to the community.

My recommendation: yes, send it to serious peer review. The core mechanism is plausible, the formal claims hold up, and the empirical gaps are addressable in revision. The reviewers should push for direct attention analysis and a clearer separation of position vs. content effects for the Landmark variant.

Referee Report

3 major / 4 minor

Summary. This paper argues that long-horizon visual persistence failures in autoregressive video world models are fundamentally an addressability problem: temporal RoPE offsets between the current query and cached keys grow beyond the trained range, so stored content becomes unreadable even when it remains in the KV cache, and compressing keys in RoPE-rotated space further corrupts memory through phase cancellation. The authors propose WorldTrace, a training-free cache scheme that assigns each summary slot a fixed in-distribution virtual position determined by slot rank (Def. 1) and stores compressed content in the canonical pre-RoPE domain, with two writers: WorldTrace-Field (canonical key averaging) for temporal coherence and WorldTrace-Landmark (frozen verbatim keys at detected scene entries) for episodic recall. They also introduce LoopBench, a benchmark of loop-shaped paths (ABA, ABCA, ABCDA) that scores return frames against the model's own first-visit generation. Experiments on MG2-1.3B (with a LingBot-World cross-check) report +15.5% TempSSIM at N=48 for WorldTrace-Field and +19.5% PAC on ABA loops for WorldTrace-Landmark, alongside ablations isolating position assignment and compression domain.

Significance. The paper's cleanest contributions are analytic and reproducible: Proposition 1 is a correct l1/l-infinity bound on projected-attention error, Proposition 2 is a parameter-free identity showing that canonical key averaging preserves mean pre-softmax attention scores, and Definition 1 guarantees distinct in-distribution slot positions at any horizon. The empirical study is unusually transparent for the area: all twelve LoopBench geometries are enumerated (Fig. 8), memory accounting is reported (Tab. 13), slot-count sensitivity is swept (Tab. 12), the norm-reduction side effect is disclosed (App. C), and the repositioning error is honestly left unbound (App. B.3). If the addressability framing survives, the paper provides a useful design vocabulary for fixed-budget memory in world models. However, the significance is moderated by the evidence gap discussed in Major Comment 1: the largest measured gains are driven by verbatim landmark content, and the incremental contribution of the position mechanism is small relative to the causal weight placed on it.

major comments (3)

[Abstract, Sec. 2.1, Sec. 5, Tables 9-10] The headline causal claim—that long-horizon failure is "fundamentally a problem of addressability"—is stronger than the evidence presented, and the stress-test concern about the unverified "read as real" mechanism lands. In the episodic-recall regime, Table 9 (App. E.2) shows Landmark+Block-relative (verbatim keys, capped positions, no WorldTrace indexing) achieves PAC 0.929 at N=32 and 0.934 at N=48, versus 0.964/0.972 for WorldTrace-Landmark and 0.401/0.388 for the sliding window; content retention accounts for nearly all of the gain, and the incremental contribution of the slot-rank position mechanism is roughly 0.03-0.04. In the compression-only tier (Table 10), WorldTrace-Field beats the sliding window at N=32-64 (0.434-0.442 vs 0.401-0.412) but falls below it at N=128 and N=256 (0.495 vs 0.504; 0.602 vs 0.631). Table 1, the cleanest position-only test, shows modest gains (0.545 vs 0.530 at N=16) with no error bars. The paper never measures attention weights assigned to summary slots or per-frequency retrieval success, so the proposed mechanism remains unconfirmed. Either add direct evidence of retrieval at virtual positions or reframe the claim as "position adjustment provides a modest, additive benefit on top of content retention," which is what the data currently support.
[Sec. 3.4; Algorithm 1; App. G.3] The scene-entry threshold tau is a free hyperparameter that determines which frames WorldTrace-Landmark stores as landmarks, yet its numeric value is never reported and no sensitivity analysis is given; the hyperparameter listing in App. G.3 mentions only slot counts and temporal grouping. This is not a cosmetic omission: App. E.5 (Table 12) shows recall is highly sensitive to the closely related slot-allocation choice (PAC drops from 0.964 to 0.419 at N=32 when N_s goes from 4 to 2), so tau plausibly has similar leverage on the landmark tier's results. The paper should report tau (per model, if tuned) and provide a sweep, or remove tau from the method by fixing an explicit quantile-based criterion.
[Table 2; Sec. 4.2; Tables 1, 4, 5] The Table 2 caption states that "WorldTrace-Field achieves the highest TempSSIM at both horizons and the lowest Scene Drift," which is contradicted by the table's own N=32 column: Centroid-linear has lower Scene Drift (0.0211 vs 0.0250), and the sliding window also has lower Scene Drift than WorldTrace-Field (0.0229 vs 0.0250) at that horizon. The body text acknowledges the first of these but the caption does not, and the abstract's unqualified "improves temporal consistency" is only accurate at N=48. In addition, Tables 1, 2, 4, and 5 report no error bars or significance tests, so the headline +15.5% gain and the position-assignment differences (e.g., 0.545 vs 0.530 in Table 1) cannot be separated from noise; Table 3 shows the authors can compute standard errors (n=100 per condition), so their absence from the headline tables should be corrected.

minor comments (4)

[Sec. 4.2 (Centroid-linear discussion) vs Eq. (1)] The text claims Centroid-linear "shift[s] old summaries into positional ranges the model was not trained to use as the horizon grows," but Centroid-linear is defined as mapping positions into [t_v_min, t_v_max], which is in-distribution by construction (Eq. 1); the causal explanation of the baseline's failure is internally inconsistent and should be revised to match the definition (e.g., instability of the mapping over time rather than OOD positions).
[Sec. 4.4, Table 5] The claim that canonical averaging is "unaffected" by the number of slots overstates the table: canonical LatentDiff rises from 0.224 (N_s=1) to 0.257 (N_s=2), a 15% increase similar in size to naive averaging's increase; the text should describe the canonical curves as more stable, not unaffected.
[Abstract; App. F] LoopBench scores return frames against the model's own first-visit generation, so the metric measures self-consistency over time, not fidelity to an external ground-truth scene; the paper discloses this in App. F, but the abstract's phrasing ("reconstruct a previously visited scene") should be qualified in the main text, since a model that consistently redraws its own hallucinated scene can still score high on PAC.
[Abstract; Remark 1] Typos: "corrupt memory" should be "corrupts memory" in the abstract, and "WorldTr aceslot" in Remark 1 should be "WorldTrace slot."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the addressability framework, virtual-position formula, and canonical averaging are definitions or proven identities, not fitted targets; the unmeasured 'read-as-real' mechanism is an evidence gap rather than a circular reduction.

full rationale

WorldTrace's derivation chain is self-contained. Def. 1 (slot indexing) is a definition satisfying the stated in-distribution and distinctness properties, Eq. (3) is an algebraic construction, and Prop. 2 is a linearity identity (rotation and inner product linearity) proven in App. C; none of these import the empirical results as inputs. The near-optimality statements in App. B are explicitly conditional: 'Eqn (7) and Eqn (9) are conditional characterizations, true by construction for the query families that Eqn (6) and Eqn (8) define, not unconditional optimality claims.' No baseline metric is fit by a learned parameter: the +15.5%/+19.5% numbers are measured on the authors' own LoopBench, whose target frames are generated by the same model ('the rollout itself produces both the target and the prediction'), and the landmark threshold tau is not documented; these are external-validity and reproducibility concerns, not cases where a prediction is equivalent to an input by construction. The canonical-key mechanism is explicitly attributed to concurrent MemRoPE [42], not to the present authors, and the paper's self-citations (e.g., [58] for benchmark-design principles) are not load-bearing evidence for the central claim. The acknowledged limitations in App. H.1 (lossy compression, at most N_s revisitable scenes) and App. C (norm reduction of averaged keys, kept by design) further show the authors do not conceal the method's dependence on its own choices. The unmeasured 'read-as-real' premise, that pretrained attention treats compressed keys at virtual positions as real frames, is a significant evidence gap, but the paper does not derive that premise from the target results or cite it into existence; it is a testable empirical hypothesis. Accordingly, no circular step meets the quoting-and-reduction bar.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced; virtual positions are a re-indexing of existing cache slots, not a new mechanism with independent handles. The free parameters are the landmark threshold tau and the slot allocation, both hand-chosen rather than fitted to the target metrics.

free parameters (2)

scene-entry threshold tau (WorldTrace-Landmark) = not reported
Used in Sec 3.4 to mark cosine-distance spikes in canonical keys as scene-entry events; the value and sensitivity are not given in the main text, and recall depends on detecting the right entries.
slot allocation (N_s, N_r) = N_s=2, N_r=4 (coherence); N_s=4, N_r=2 (recall)
Chosen per task in Sec 4.1; total cache size is fixed at L_attn=6, and the split is a hand-selected hyperparameter that favors either coherence or recall.

assumptions (5)

standard math RoPE attention scores decompose per frequency pair as a_f = Re(A_f exp(i theta_f delta)) (Eq. 2), and rotations are linear and preserve inner products under unitary transforms.
Invoked in Sec 2.1 and used in Props 1 and 2.
domain assumption The pretrained model was trained with a fixed local attention window L_attn and max offset Delta t_train, and the KV cache is the only memory channel available at inference.
Stated in Sec 2.1 and App G.1 for MG2-1.3B; the method assumes this configuration is known.
domain assumption A key placed at an in-distribution RoPE offset is readable by the pretrained attention as if it were a real frame, even when it is an average of many distant frames.
This is the load-bearing premise behind Def 1 and the slot-rank virtual position assignment in Sec 3.1.
domain assumption The 3D-RoPE configuration of the backbone (2c_t=44 temporal dims, base theta=10000) is known and fixed.
Used in App C to analyze per-frequency phase degradation on MG2-1.3B.
ad hoc to paper LoopBench's recall reference is the model's own first-visit generation, so the benchmark measures self-consistency over time, not fidelity to an external ground-truth scene.
App F states that the rollout itself produces both the target and the prediction, which is a deliberate design choice for world-model evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Addressable Memory for Video World Models." pith.science (2026). https://pith.science/paper/E3EX2SGQ

@misc{pith2026260807408,
  author       = {Pith},
  title        = {Pith review of: Addressable Memory for Video World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3EX2SGQ}},
  note         = {Machine review of arXiv:2608.07408}
}

read the original abstract

We study visual persistence in interactive video world models. These models rely on a Key-Value (KV) cache as a growing visual memory to carry forward previously generated frames. However, we find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal Rotary Positional Embeddings (RoPE) offsets then fall outside the range seen during training and the model struggles to retrieve the relevant visual information through attention. Moreover, naively compressing the cache in the RoPE-rotated space corrupts memory by averaging together incompatible positional phases. To address this, we propose WorldTrace, a training-free memory framework for long-horizon visual persistence. WorldTrace keeps compressed memory addressable by assigning each summary slot a distinct, in-distribution virtual position. Within this addressable cache, we study two memory compression approaches: WorldTrace-Field compresses history for temporal coherence, while WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall. We further introduce LoopBench, a benchmark evaluating whether a compressed cache can reconstruct a previously visited scene after a long detour. WorldTrace-Field improves temporal consistency by +15.5%, and WorldTrace-Landmark improves episodic recall by +19.5% on LoopBench, extending visually persistent generation without retraining.

Figures

Figures reproduced from arXiv: 2608.07408 by the authors.

**Figure 1.** Addressable memory maintains long-horizon visual persistence. (Left) Loop topology A→B→C→D→A. (Right) Representative frames on Matrix-Game-2 along the path, covering the initial scene A (blue border), waypoints B, C, and D, and the return to A. Sliding window (top) mismatches the reference appearance at the return (red border). Our WorldTrace with frozen landmark keys (green border) matches scene A, confirming addre… view at source ↗

**Figure 2.** WorldTrace overview. WorldTrace partitions the context window into a compressed summary cache 𝒮 for the distant past and a verbatim recent window ℛ (top). Each slot occupies one latent-frame position in the cache, with recent slots holding one verbatim frame and summary slots holding compressed history. WorldTrace-Field (bottom left): Keys are unrotated and averaged in canonical space, then re-rotated at each slot’s… view at source ↗

**Figure 3.** Three LoopBench geometries. We propose LoopBench, a memory benchmark with diverse geometries, including ABA (straight reversal), ABCA (approximate L-triangle with diagonal 5 √ 2 ≈ 7), and ABCDA (square). 𝑁 is rollout length in AR chunks. Each path returns to scene A. We provide a full gallery of edge-length, orientation, and multi-revisit settings in App. F ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: LoopBench qualitative results on the three revisit geometries in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Projection matrices 𝑃 for the four cache structures. (a) Sliding window: last 𝐿attn frames verbatim. (b) Sliding window with attention sink: initial frame held as an attention sink, remaining slots verbatim recent. (c) WorldTrace-Field: recent rows verbatim newest, sum…

**Figure 7.** Figure 7: Fast RoPE components go out of distribution at long horizons. Per-frequency RoPE phase 𝜃𝑓 |𝛿𝑞,𝑘| on MG2-1.3B over frequency index 𝑓 and inference distance |𝛿𝑞,𝑘|. The log color scale marks the severity thresholds, and the dashed line shows the training max ∆𝑡train=5. T…

**Figure 8.** Figure 8: LoopBench benchmark gallery. Solid and dashed blue arrows mark the outbound and return legs, lighter arcs mark a second traversal, and dotted arrows mark shortcuts that skip intermediate waypoints. Row 1: topology (ABA, ABCA, ABCDA). Row 2: rollout length on the ABA to…

**Figure 9.** Figure 9: ABA qualitative results. Each sample group of three frames shows trajectory keyframes at chunks 0 (A) / 7 (B) / 15 (A return). The two rows compare the sliding-window baseline (top) vs. WorldTraceLandmark (bottom). The sliding window drifts away from the scene-A appea…

**Figure 10.** Figure 10: Pan 90∘ qualitative results. In camera-orientation Tier 3, the agent stays at A while the camera pans right by ∼90∘ and then pans back. Each sample group of three frames shows the start (0 ∘ ), the pan peak (∼90∘ ), and the return to 0 ∘ . The two rows compare the sli…

**Figure 11.** Figure 11: Camera trajectory used for [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

120 extracted references · 3 canonical work pages

[1]

Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

arXiv 2026
[2]

Round and round we go! what makes rotary positional encodings useful? InInternational Conference on Learning Representations (ICLR), 2025

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2410.06205

arXiv 2025
[3]

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

Aarti Basant, Amlan Kar, Despoina Paschalidou, Fangyin Wei, Francesco Ferroni, Guillermo Garcia Cobo, Haithem Turki, Huan Ling, Jaewoo Seo, James Lucas, et al. Nvidia omnidreams: Real-time generative world model for closed-loop autonomous vehicle simulation.arXiv preprint arXiv:2606.03159, 2026

work page Pith review arXiv 2026
[4]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

arXiv 2004
[5]

Variance Reduction for Expectations with Diffusion Teachers

JesseBettencourt, XindiWu, MatanAtzmon, JamesLucas, andJonathanLorraine. Variancereductionforexpectations with diffusion teachers.arXiv preprint arXiv:2605.21489, 2026

work page Pith review arXiv 2026
[6]

Token merging: Your ViT but faster

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InInternational Conference on Learning Representations (ICLR), 2023

2023
[7]

Recurrentmemorytransformer

AydarBulatov, YuriKuratov, andMikhailS.Burtsev. Recurrentmemorytransformer. InAdvancesinNeuralInformation Processing Systems (NeurIPS), 2022

2022
[8]

Mixture of contexts for long video generation

Shengqu Cai, Ceyuan Yang, Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang, Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas, Maneesh Agrawala, Lu Jiang, and Gordon Wetzstein. Mixture of contexts for long video generation. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2508.21058, OpenReview: https://openreview.net/forum?...

arXiv 2026

Show all 120 references

[9]

PyramidKV: Dynamic KV cache compression based on pyramidal information funneling

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Conference on Language Modeling (COLM), 2025

2025
[10]

Diffusion forcing: Next-token prediction meets full-sequence diffusion

Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

2024
[11]

Past- and future-informed kv cache policy with salience estimation in autoregressive video diffusion.arXiv preprint arXiv:2601.21896, 2026

Hanmo Chen, Chenghao Xu, Xu Yang, Xuan Chen, and Cheng Deng. Past- and future-informed kv cache policy with salience estimation in autoregressive video diffusion.arXiv preprint arXiv:2601.21896, 2026

2026
[12]

Grounded forcing: Bridging time-independent semantics and proximal dynamics in autoregressive video synthesis.arXiv preprint arXiv:2604.06939, 2026

Jintao Chen, Chengyu Bai, Junjun Hu, Xinda Xue, and Mu Xu. Grounded forcing: Bridging time-independent semantics and proximal dynamics in autoregressive video synthesis.arXiv preprint arXiv:2604.06939, 2026

2026 arXiv
[13]

Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

2023 arXiv
[14]

Context forcing: Consistent autoregressive video generation with long context.arXiv preprint arXiv:2602.06028, 2026

Shuo Chen, Cong Wei, Sun Sun, Ping Nie, Kai Zhou, Ge Zhang, Ming-Hsuan Yang, and Wenhu Chen. Context forcing: Consistent autoregressive video generation with long context.arXiv preprint arXiv:2602.06028, 2026

2026
[15]

Learning world models for interactive video generation.arXiv preprint arXiv:2505.21996, 2025

Taiye Chen, Xun Hu, Zihan Ding, and Chi Jin. Learning world models for interactive video generation.arXiv preprint arXiv:2505.21996, 2025. Project pagehttps://sites.google.com/view/vrag

2025 arXiv
[16]

FINCH: Prompt-guided key-value cache compression for large language models

Giulio Corallo and Paolo Papotti. FINCH: Prompt-guided key-value cache compression for large language models. Transactions of the Association for Computational Linguistics, 12:1517–1532, 2024

2024
[17]

Lol: Longer than longer, scaling video generation to hour.arXiv preprint arXiv:2601.16914, 2026

Justin Cui, Jie Wu, Ming Li, Tao Yang, Xiaojie Li, Rui Wang, Andrew Bai, Yuanhao Ban, and Cho-Jui Hsieh. Lol: Longer than longer, scaling video generation to hour.arXiv preprint arXiv:2601.16914, 2026

2026
[18]

Self-Forcing++: Towards minute-scale high-quality video generation

Justin Cui, Jie Wu, Ming Li, Tao Yang, Xiaojie Li, Rui Wang, Andrew Bai, Yuanhao Ban, and Cho-Jui Hsieh. Self-Forcing++: Towards minute-scale high-quality video generation. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2510.02283

2026 arXiv
[19]

Le, and Ruslan Salakhutdinov

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. InAnnual Meeting of the Association for Computational Linguistics (ACL), pages 2978–2988, 2019. arXiv:1901.02860

2019 arXiv
[20]

Oasis: A universe in a transformer.https://oasis-model.github.io, 2024

Decart, Julian Quevedo, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a transformer.https://oasis-model.github.io, 2024. 12 Addressable Memory for Video World Models

2024
[21]

WorldScore: A unified evaluation benchmark for world generation

Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. WorldScore: A unified evaluation benchmark for world generation. InInternational Conference on Computer Vision (ICCV), 2025

2025
[22]

A unified framework for approximating and clustering data

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. InProceedings of the 43rd Annual ACM Symposium on Theory of Computing (STOC), pages 569–578, 2011

2011
[23]

Memcam: Memory- augmented camera control for consistent video generation.arXiv preprint arXiv:2603.26193, 2026

Xinhang Gao, Junlin Guan, Shuhan Luo, Wenzhuo Li, Guanghuan Tan, and Jiacheng Wang. Memcam: Memory- augmented camera control for consistent video generation.arXiv preprint arXiv:2603.26193, 2026

2026
[24]

Contextual position encoding: Learning to count what’s important.arXiv preprint arXiv:2405.18719, 2024

Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position encoding: Learning to count what’s important.arXiv preprint arXiv:2405.18719, 2024

2024 arXiv
[25]

Genie 3: A new frontier for world models

Google DeepMind. Genie 3: A new frontier for world models. DeepMind Blog,https://deepmind.google/blog/ genie-3-a-new-frontier-for-world-models, 2025

2025
[26]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. InConference on Language Modeling (COLM), 2024

2024
[27]

Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025

Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025

2025 arXiv
[28]

Memorize when needed: Decoupled memory control for spatially consistent long-horizon video generation.arXiv preprint arXiv:2604.18215, 2026

Yanjun Guo, Zhengqiang Zhang, Pengfei Wang, Xinyue Liang, Zhiyuan Ma, and Lei Zhang. Memorize when needed: Decoupled memory control for spatially consistent long-horizon video generation.arXiv preprint arXiv:2604.18215, 2026

2026 arXiv
[29]

Recurrent world models facilitate policy evolution

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. InAdvances in Neural Information Processing Systems (NeurIPS), 2018

2018
[30]

Mastering diverse control tasks through world models.Nature, 640:647–653, 2025

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640:647–653, 2025

2025
[31]

A2ATS: Retrieval-based KV cache reduction via windowed rotary position embedding and query-aware vector quantization

Junhui He, Junna Xing, Nan Wang, Rui Xu, Shangyu Wu, Peng Zhou, Qiang Liu, Chun Jason Xue, and Qingan Li. A2ATS: Retrieval-based KV cache reduction via windowed rotary position embedding and query-aware vector quantization. InFindings of the Association for Computational Lingu...

2025 arXiv
[32]

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model.arXiv preprint arXiv:2508.13009, 2025

Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Size Wu, Wei Li, Xuchen Song, Yang Liu, Yangguang Li, and Yahui Zhou. Matrix-game 2.0: An open-source, real-time, ...

2025 arXiv
[33]

StreamingT2V: Consistent, dynamic, and extendable long video generation from text

Roberto Henschel, Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. StreamingT2V: Consistent, dynamic, and extendable long video generation from text. InConference on Computer Vision and Pattern Recog...

2025 arXiv
[34]

Rotary position embedding for vision transformer

Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision (ECCV), 2024. arXiv:2403.13298

2024 arXiv
[35]

RELIC: Interactive video world model with long-horizon memory.arXiv preprint arXiv:2512.04040, 2025

Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, Kalyan Sunkavalli, Feng Liu, Zhengqi Li, and Hao Tan. RELIC: Interactive video world model with long-horizon memory.arXiv preprint arXiv:2512....

2025
[36]

Fourier position embedding: Enhancing attention’s periodic extension for length generalization

Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Youbang Sun, Yuchen Fan, Xuekai Zhu, Biqing Qi, Ning Ding, and Bowen Zhou. Fourier position embedding: Enhancing attention’s periodic extension for length generalization. In International Conference on Machine Learning (ICML), 202...

2025 arXiv
[37]

Vid2World: Crafting video diffusion models to interactive world models

Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. Vid2World: Crafting video diffusion models to interactive world models. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2505.14357

2026
[38]

Self forcing: Bridging the train-test gap in autoregressive video diffusion

Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight; arXiv:2506.08009

2025 arXiv
[39]

Block-recurrent transformers

DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2203.07852

2022 arXiv
[40]

Transformers are RNNs: Fast autore- gressive transformers with linear attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autore- gressive transformers with linear attention. InInternational Conference on Machine Learning (ICML), 2020

2020
[41]

The impact of positional encoding on length generalization in transformers

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.19466

2023 arXiv
[42]

Jay Kuo, and Peter A

Youngrae Kim, Qixin Hu, C.-C. Jay Kuo, and Peter A. Beerel. MemRoPE: Training-free infinite video generation via evolving memory tokens.arXiv preprint arXiv:2603.12513, 2026

2026
[43]

Robust nonnegative matrix factorization using l21-norm

Deguang Kong, Chris Ding, and Heng Huang. Robust nonnegative matrix factorization using l21-norm. InProceedings of the 20th ACM International Conference on Information and Knowledge Management, pages 673–682, 2011

2011
[44]

Lee and H

Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization.Nature, 401(6755):788–791, 1999

1999
[45]

Understanding and improving length generalization in hierarchical sparse attention models.arXiv preprint arXiv:2510.17196, 2025

Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, and Yucheng Lu. Understanding and improving length generalization in hierarchical sparse attention models.arXiv preprint arXiv:2510.17196, 2025. 13 Addressable Memory for Video World Models

2025 arXiv
[46]

Rolling sink: Bridging limited-horizon training and open-ended testing in autoregressive video diffusion.arXiv preprint arXiv:2602.07775, 2026

Haodong Li, Shaoteng Liu, Zhe Lin, and Manmohan Chandraker. Rolling sink: Bridging limited-horizon training and open-ended testing in autoregressive video diffusion.arXiv preprint arXiv:2602.07775, 2026

2026 arXiv
[47]

Train short, inference long: Training-free horizon extension for autoregressive video generation.arXiv preprint arXiv:2602.14027, 2026

Jia Li, Xiaomeng Fu, Xurui Peng, Weifeng Chen, Youwei Zheng, Tianyu Zhao, Jiexi Wang, Fangmin Chen, Xing Wang, and Hayden Kwok-Hay So. Train short, inference long: Training-free horizon extension for autoregressive video generation.arXiv preprint arXiv:2602.14027, 2026

2026
[48]

PackCache: A training-free acceleration method for unified autoregressive video generation via compact KV-cache.arXiv preprint arXiv:2601.04359, 2026

Kunyang Li, Mubarak Shah, and Yuzhang Shang. PackCache: A training-free acceleration method for unified autoregressive video generation via compact KV-cache.arXiv preprint arXiv:2601.04359, 2026

2026
[49]

Long-horizon streaming video generation via hybrid attention with decoupled distillation.arXiv preprint arXiv:2604.10103, 2026

Ruibin Li, Tao Yang, Fangzhou Ai, Tianhe Wu, Shilei Wen, Bingyue Peng, and Lei Zhang. Long-horizon streaming video generation via hybrid attention with decoupled distillation.arXiv preprint arXiv:2604.10103, 2026

2026 arXiv
[50]

Cameras as relative positional encoding

Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, and Angjoo Kanazawa. Cameras as relative positional encoding. Advances in Neural Information Processing Systems, 38:15984–16009, 2026

2026
[51]

VMem: Consistent interactive video scene generation with surfel-indexed view memory

Runjia Li, Philip Torr, Andrea Vedaldi, and Tomas Jakab. VMem: Consistent interactive video scene generation with surfel-indexed view memory. InInternational Conference on Computer Vision (ICCV), 2025. arXiv:2506.18903

2025 arXiv
[52]

Stable video infinity: Infinite-length video generation with error recycling.arXiv preprint arXiv:2510.09212, 2025

Wuyang Li, Wentao Pan, Po-Chien Luan, Yang Gao, and Alexandre Alahi. Stable video infinity: Infinite-length video generation with error recycling.arXiv preprint arXiv:2510.09212, 2025

2025
[53]

SnapKV:LLMknowswhatyouarelookingforbeforegeneration

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, andDemingChen. SnapKV:LLMknowswhatyouarelookingforbeforegeneration. InAdvancesinNeuralInformation Processing Systems (NeurIPS), 2024. arXiv:2404.14469

2024 arXiv
[54]

Toward memory-aided world models: Benchmarking via spatial consistency.arXiv preprint arXiv:2505.22976, 2025

Kewei Lian, Shaofei Cai, Yilun Du, and Yitao Liang. Toward memory-aided world models: Benchmarking via spatial consistency.arXiv preprint arXiv:2505.22976, 2025. Loop-based Minecraft navigation benchmark for spatial consistency in world models

2025 arXiv
[55]

Rolling forcing: Autoregressive long video diffusion in real time

Kunhao Liu, Wenbo Hu, Jiale Xu, Ying Shan, and Shijian Lu. Rolling forcing: Autoregressive long video diffusion in real time. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2509.25161

2026 arXiv
[56]

KIVI: A tuning-free asymmetric 2-bit quantization for KV cache

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2-bit quantization for KV cache. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2402.02750

2024 arXiv
[57]

JacNet: Learning functions with structured Jacobians

Jonathan Lorraine and Safwan Hossain. JacNet: Learning functions with structured Jacobians. InICML Workshop on Invertible Neural Networks and Normalizing Flows (INNF), 2019

2019
[58]

Task selection for AutoML system evaluation.arXiv preprint arXiv:2208.12754, 2022

Jonathan Lorraine, Nihesh Anderson, Chansoo Lee, Quentin De Laroussilhe, and Mehadi Hassen. Task selection for AutoML system evaluation.arXiv preprint arXiv:2208.12754, 2022

2022 arXiv
[59]

ATT3D: Amortized text-to-3D object synthesis

Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. ATT3D: Amortized text-to-3D object synthesis. InInternational Conference on Computer Vision (ICCV), 2023

2023
[60]

Flow caching for autoregressive video generation.arXiv preprint arXiv:2602.10825, 2026

Yuexiao Ma, Xuzhe Zheng, Jing Xu, Xiwei Xu, Feng Ling, Xiawu Zheng, Huafeng Kuang, Huixia Li, Xing Wang, Xuefeng Xiao, Fei Chao, and Rongrong Ji. Flow caching for autoregressive video generation.arXiv preprint arXiv:2602.10825, 2026

2026
[61]

TriAttention: Efficient long reasoning with trigonometric KV compression.arXiv preprint arXiv:2604.04921, 2026

Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, and Yukang Chen. TriAttention: Efficient long reasoning with trigonometric KV compression.arXiv preprint arXiv:2604.04921, 2026

2026 arXiv
[62]

PackForcing: Short video training suffices for long video sampling and long context inference.arXiv preprint arXiv:2603.25730, 2026

Xiaofeng Mao, Shaohao Rui, Kaining Ying, Bo Zheng, Chuanhao Li, Mingmin Chi, and Kaipeng Zhang. PackForcing: Short video training suffices for long video sampling and long context inference.arXiv preprint arXiv:2603.25730, 2026

2026
[63]

Landmark attention: Random-access infinite context length for trans- formers

Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for trans- formers. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.16300

2023 arXiv
[64]

Leave no context behind: Efficient infinite context transformers with Infini-attention.arXiv preprint arXiv:2404.07143, 2024

Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with Infini-attention.arXiv preprint arXiv:2404.07143, 2024

2024 arXiv
[65]

KVPress: A compression library for transformer KV caches.https://github.com/NVIDIA/kvpress, 2024

NVIDIA. KVPress: A compression library for transformer KV caches.https://github.com/NVIDIA/kvpress, 2024

2024
[66]

WorldPack: Compressed memory improves spatial consistency in video world modeling.arXiv preprint arXiv:2512.02473, 2025

Yuta Oshima, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. WorldPack: Compressed memory improves spatial consistency in video world modeling.arXiv preprint arXiv:2512.02473, 2025

2025 arXiv
[67]

YaRN: Efficient context window extension of large language models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2309.00071

2024 arXiv
[68]

Long-context state-space video world models

Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models. In2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8733–8744. IEEE, 2025

2025
[69]

Smith, and Mike Lewis

Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InInternational Conference on Learning Representations (ICLR), 2022

2022
[70]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conf...

2021
[71]

Rae, Anna Potapenko, Siddhant M

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. InInternational Conference on Learning Representations (ICLR), 14 Addressable Memory for Video World Models 2020

2020
[72]

Score distillation sampling for audio: Source separation, synthesis, and beyond

Jessie Richter-Powell, Antonio Torralba, and Jonathan Lorraine. Score distillation sampling for audio: Source separation, synthesis, and beyond. InICML Workshop on AI Heard That!, 2025. arXiv:2505.04621

2025 arXiv
[73]

Fast autoregressive video diffusion and world models with temporal cache compression and sparse attention.arXiv preprint arXiv:2602.01801, 2026

Dvir Samuel, Issar Tzachor, Matan Levy, Michael Green, Gal Chechik, and Rami Ben-Ari. Fast autoregressive video diffusion and world models with temporal cache compression and sparse attention.arXiv preprint arXiv:2602.01801, 2026

2026
[74]

LongRoPE2: Near-lossless LLM context window scaling

Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. LongRoPE2: Near-lossless LLM context window scaling. InInternational Conference on Machine Learning (ICML),
[75]

History-guided video diffusion

Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. InInternational Conference on Machine Learning (ICML), 2025. arXiv:2502.06764

2025 arXiv
[76]

Multi-student diffusion distillation for better one-step generators

Yanke Song, Jonathan Lorraine, Weili Nie, Karsten Kreis, and James Lucas. Multi-student diffusion distillation for better one-step generators. InICML Workshop on Efficient Systems for Foundation Models (ES-FoMo), 2025. arXiv:2410.23274

2025 arXiv
[77]

Composition of memory experts for diffusion world models

Sebastian Stapf, Pablo Acuaviva, Aram Davtyan, and Paolo Favaro. Composition of memory experts for diffusion world models. InInternational Conference on Learning Representations (ICLR), 2026. OpenReview: https:// openreview.net/forum?id=sUEdpZCHdp

2026
[78]

RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

2024
[79]

WorldPlay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

Wenqiang Sun, Haiyu Zhang, Haoyuan Wang, Junta Wu, Zehan Wang, Zhenwei Wang, Yunhong Wang, Jun Zhang, Tengfei Wang, and Chunchao Guo. WorldPlay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

2025 arXiv
[80]

Learning to (learn at test time): RNNs with expressive hidden states

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states. InInternational Conference on Machine ...

2025
[81]

A length-extrapolatable transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2023. arXiv:2212.10554

2023 arXiv
[82]

Advancing open-source world models.arXiv preprint arXiv:2601.20540, 2026

Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, Yihang Chen, Jie Liu, Yansong Cheng, Yao Yao, Jiayi Zhu, Yihao Meng, Kecheng Zheng, Qingyan Bai, Jingye Chen, Zehong Shen, Yue Yu, Xing Zhu, Yujun ...

2026 arXiv
[83]

KeepKV: Achieving periodic lossless KV cache compression for efficient LLM inference

Yuxuan Tian, Zihan Wang, Yebo Peng, Aomufei Yuan, Zhiming Wang, Bairen Yi, Xin Liu, Yong Cui, and Tong Yang. KeepKV: Achieving periodic lossless KV cache compression for efficient LLM inference. InAAAI Conference on Artificial Intelligence, 2026. arXiv:2504.09936

2026
[84]

Diffusion models are real-time game engines

Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In International Conference on Learning Representations (ICLR), 2025

2025
[85]

Pavlo Vasylenko, Hugo Pitorro, André F. T. Martins, and Marcos Treviso. Long-context generalization with sparse attention.arXiv preprint arXiv:2506.16640, 2025

2025
[86]

Fast transformers with clustered attention

Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

2020
[87]

Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

Wan Team. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

2025 arXiv
[88]

When precision meets position: BFloat16 breaks down RoPE in long-context training.Transactions on Machine Learning Research (TMLR), 2025

Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, and Tianyu Pang. When precision meets position: BFloat16 breaks down RoPE in long-context training.Transactions on Machine Learning Research (TMLR), 2025. arXiv:2411.13476

2025 arXiv
[89]

Li, Madian Khabsa, Han Fang, and Hao Ma

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

2006 arXiv
[90]

LLaMA-Mesh: Unifying 3D mesh generation with language models.arXiv preprint arXiv:2411.09595, 2024

Zhengyi Wang, Jonathan Lorraine, Yikai Wang, Hang Su, Jun Zhu, Sanja Fidler, and Xiaohui Zeng. LLaMA-Mesh: Unifying 3D mesh generation with language models.arXiv preprint arXiv:2411.09595, 2024

2024 arXiv
[91]

Bovik, Hamid R

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity.IEEE Transactions on Image Processing, 13(4):600–612, 2004

2004
[92]

Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory.arXiv preprint arXiv:2604.08995, 2026

Zile Wang, Zexiang Liu, Jiaxing Li, Kaichen Huang, Baixin Xu, Fei Kang, Mengyin An, Peiyu Wang, Biao Jiang, Yichen Wei, Yidan Xietian, Jiangbo Pei, Liang Hu, Boyi Jiang, Hua Xue, Zidong Wang, Haofeng Sun, Wei Li, Wanli Ouyang, Xianglong He, Yang Liu, Yangguang Li, and Yahui Zh...

2026 arXiv
[93]

VideoRoPE: What makes for good video rotary position embedding? In International Conference on Machine Learning (ICML), 2025

Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. VideoRoPE: What makes for good video rotary position embedding? In International Conference on Machine Learning (ICML), 2025. Or...

2025 arXiv
[94]

Video world models with long-term spatial memory

Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, and Gordon Wetzstein. Video world models with long-term spatial memory. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.05284, 15 Addressable Memory for Video World Models project p...

2025 arXiv
[95]

Corgi: Cached memory guided video generation

Xindi Wu, Uriel Singer, Zhaojiang Lin, Andrea Madotto, Xide Xia, Yifan Xu, Paul Crook, Xin Luna Dong, and Seungwhan Moon. Corgi: Cached memory guided video generation. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4585–4594. IEEE, 2025

2025
[96]

Motion attribution for video generation

Xindi Wu, Despoina Paschalidou, Jun Gao, Antonio Torralba, Laura Leal-Taixé, Olga Russakovsky, Sanja Fidler, and Jonathan Lorraine. Motion attribution for video generation. InInternational Conference on Machine Learning (ICML),
[97]

Rabe, DeLesley Hutchins, and Christian Szegedy

Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. InInternational Conference on Learning Representations (ICLR), 2022

2022
[98]

Efficient streaming language models with attention sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations (ICLR), 2024

2024
[99]

DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads

Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2410.10819

2025 arXiv
[100]

WorldMem: Long-term consistent world simulation with memory

Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. WorldMem: Long-term consistent world simulation with memory. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

2025
[101]

LATTE3D: Large-scale amortized text-to-enhanced 3D synthesis

Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. LATTE3D: Large-scale amortized text-to-enhanced 3D synthesis. InEuropean Conference on Computer Vision (ECCV), 2024

2024
[102]

Nyströmformer: A Nyström-based algorithm for approximating self-attention

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A Nyström-based algorithm for approximating self-attention. InAAAI Conference on Artificial Intelligence (AAAI), 2021

2021
[103]

Sparse forcing: Native trainable sparse attention for real-time autoregressive diffusion video generation.arXiv preprint arXiv:2604.21221, 2026

Boxun Xu, Yuming Du, Zichang Liu, Siyu Yang, Ziyang Jiang, Siqi Yan, Rajasi Saha, Albert Pumarola, Wenchen Wang, and Peng Li. Sparse forcing: Native trainable sparse attention for real-time autoregressive diffusion video generation.arXiv preprint arXiv:2604.21221, 2026

2026 arXiv
[104]

UCM: Unifying camera control and memory with time-aware positional encoding warping for world models

Tian-Xing Xu, Zi-Xuan Wang, Guangyuan Wang, Li Hu, Zhongyi Zhang, Peng Zhang, Bang Zhang, and Song-Hai Zhang. UCM: Unifying camera control and memory with time-aware positional encoding warping for world models. arXiv preprint arXiv:2602.22960, 2026

2026 arXiv
[105]

Anchor forcing: Anchor memory and tri-region RoPE for interactive streaming video diffusion.arXiv preprint arXiv:2603.13405, 2026

Yang Yang, Tianyi Zhang, Wei Huang, Jinwei Chen, Boxi Wu, Xiaofei He, Deng Cai, Bo Li, and Peng-Tao Jiang. Anchor forcing: Anchor memory and tri-region RoPE for interactive streaming video diffusion.arXiv preprint arXiv:2603.13405, 2026

2026
[106]

MIND: Benchmarking memory consistency and action control in world models.arXiv preprint arXiv:2602.08025, 2026

Yixuan Ye, Xuanyu Lu, Yuxin Jiang, Yuchao Gu, Rui Zhao, Qiwei Liang, Jiachun Pan, Fengda Zhang, Weijia Wu, and Alex Jinpeng Wang. MIND: Benchmarking memory consistency and action control in world models.arXiv preprint arXiv:2602.08025, 2026

2026
[107]

∞-RoPE:Action-controllable infinite video generation emerges from autoregressive self-rollout

HidirYesiltepe,TunaHanSalihMeral,AdilKaanAkan,KaanOktay,andPinarYanardag. ∞-RoPE:Action-controllable infinite video generation emerges from autoregressive self-rollout. InConference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2511.20649

2026
[108]

Deep forcing: Training-free long video generation with deep sink and participative compression.arXiv preprint arXiv:2512.05081, 2025

Jung Yi, Wooseok Jang, Paul Hyunbin Cho, Jisu Nam, Heeji Yoon, and Seungryong Kim. Deep forcing: Training-free long video generation with deep sink and participative compression.arXiv preprint arXiv:2512.05081, 2025

2025
[109]

Freeman, Frédo Durand, Eli Shechtman, and Xun Huang

Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Frédo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. InConference on Computer Vision and Pattern Recognition (CVPR), 2025

2025
[110]

Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141, 2025

Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141, 2025

2025 arXiv
[111]

Chakaravarthy, Dennis Anthony, Yang Ye, Yidi Li, Weiwei Wan, and Animesh Garg

Wei Yu, Runjia Qian, Yumeng Li, Liquan Wang, Songheng Yin, Sri Siddarth P. Chakaravarthy, Dennis Anthony, Yang Ye, Yidi Li, Weiwei Wan, and Animesh Garg. MosaicMem: Hybrid spatial memory for controllable video world models.arXiv preprint arXiv:2603.17117, 2026

2026
[112]

VideoSSM: Autoregressive long video generation with hybrid state-space memory.arXiv preprint arXiv:2512.04519, 2025

Yifei Yu, Xiaoshan Wu, Xinting Hu, Tao Hu, Yang-Tian Sun, Xiaoyang Lyu, Bo Wang, Lin Ma, Yuewen Ma, Zhongrui Wang, and Xiaojuan Qi. VideoSSM: Autoregressive long video generation with hybrid state-space memory.arXiv preprint arXiv:2512.04519, 2025

2025
[113]

Frame context packing and drift prevention in next-frame-prediction video diffusion models

Lvmin Zhang, Shengqu Cai, Muyang Li, Gordon Wetzstein, and Maneesh Agrawala. Frame context packing and drift prevention in next-frame-prediction video diffusion models. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight; arXiv:2504.12626

2025
[114]

SimLayerKV: A simple framework for layer-level KV cache reduction.arXiv preprint arXiv:2410.13846v1, 2024

Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, and Min Lin. SimLayerKV: A simple framework for layer-level KV cache reduction.arXiv preprint arXiv:2410.13846v1, 2024

2024 arXiv
[115]

H2O: Heavy-hitter oracle for efficient generative inference of large language models

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InAdvances in Neural I...

2023
[116]

RIFLEx: A free lunch for length extrapolation in video diffusion transformers

Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. RIFLEx: A free lunch for length extrapolation in video diffusion transformers. InInternational Conference on Machine Learning (ICML), 2025. arXiv:2502.15894

2025 arXiv
[117]

Relax forcing: Relaxed KV-memory for consistent long video generation.arXiv preprint arXiv:2603.21366, 2026

Zengqun Zhao, Yanzuo Lu, Ziquan Liu, Jifei Song, Jiankang Deng, and Ioannis Patras. Relax forcing: Relaxed KV-memory for consistent long video generation.arXiv preprint arXiv:2603.21366, 2026

2026
[118]

VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness

Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755, 2025

2025 arXiv
[119]

EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection.arXiv preprint arXiv:2503.01586, 2025

Yuhao Zhou, Sirui Song, Boyang Liu, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Zhihao Zhang, Wei Li, and Xuanjing Huang. EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection.arXiv preprint arXiv:2503.01586, 2025

2025 arXiv
[120]

SANA- WM: Efficient minute-scale world modeling with hybrid linear diffusion transformer.arXiv preprint arXiv:2605.15178, 2026

Haoyi Zhu, Haozhe Liu, Yuyang Zhao, Tian Ye, Junsong Chen, Jincheng Yu, Tong He, Song Han, and Enze Xie. SANA- WM: Efficient minute-scale world modeling with hybrid linear diffusion transformer.arXiv preprint arXiv:2605.15178, 2026. 17 Appendices A Notation 19 BWorldTraceas St...

2026 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

arXiv 2026

[2] [2]

Round and round we go! what makes rotary positional encodings useful? InInternational Conference on Learning Representations (ICLR), 2025

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2410.06205

arXiv 2025

[3] [3]

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

Aarti Basant, Amlan Kar, Despoina Paschalidou, Fangyin Wei, Francesco Ferroni, Guillermo Garcia Cobo, Haithem Turki, Huan Ling, Jaewoo Seo, James Lucas, et al. Nvidia omnidreams: Real-time generative world model for closed-loop autonomous vehicle simulation.arXiv preprint arXiv:2606.03159, 2026

work page Pith review arXiv 2026

[4] [4]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

arXiv 2004

[5] [5]

Variance Reduction for Expectations with Diffusion Teachers

JesseBettencourt, XindiWu, MatanAtzmon, JamesLucas, andJonathanLorraine. Variancereductionforexpectations with diffusion teachers.arXiv preprint arXiv:2605.21489, 2026

work page Pith review arXiv 2026

[6] [6]

Token merging: Your ViT but faster

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InInternational Conference on Learning Representations (ICLR), 2023

2023

[7] [7]

Recurrentmemorytransformer

AydarBulatov, YuriKuratov, andMikhailS.Burtsev. Recurrentmemorytransformer. InAdvancesinNeuralInformation Processing Systems (NeurIPS), 2022

2022

[8] [8]

Mixture of contexts for long video generation

Shengqu Cai, Ceyuan Yang, Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang, Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas, Maneesh Agrawala, Lu Jiang, and Gordon Wetzstein. Mixture of contexts for long video generation. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2508.21058, OpenReview: https://openreview.net/forum?...

arXiv 2026