{"id":"1b1244e8-573f-4667-93eb-baaf0e294b24","arxiv_id":"2505.20171","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.","lead":"This paper builds a video world model that combines a compressed state-space memory (Mamba) with small local attention windows, so it can remember places seen hundreds of frames ago while keeping per-frame computation constant. It reports better long-term memory than attention-only baselines on Maze and Minecraft tests, and shows the approach scales linearly instead of quadratically with video length.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Long-term memory is only demonstrated within the training context length; the paper's own Limitations admit no memory beyond training context, so the headline claim is not yet supported for indefinite world modeling.","rationale":"The paper makes a plausible architecture contribution: the block-wise SSM scan gives each spatial block its own state, frame local attention preserves short-term coherence, and the efficiency measurements in Fig. 8 support constant-cost inference. The ablations in Table 5 and the FVD result in Table S1 are real evidence that the components help at the tested scale. I do not think the argument is internally invalid; the problem is a scope mismatch. The evaluations are all bounded by the training context, while the abstract and Fig. 1 imply a persistent, indefinitely interactive world. Since the paper itself acknowledges no memory beyond the training context, the load-bearing condition for the broad claim is untested. This is a missing support, not a demonstrated contradiction. Supplying the beyond-training-context test would settle it, so the reader's Conditional verdict is appropriate; the concern does not require rejection, but the headline claim should be scoped until that test is run.","tokens_in":15722,"tokens_out":5388,"duration_ms":59734,"concrete_test":"Retrain or fine-tune the Memory Maze model on 800-frame sequences as in Sec. S2, then run the retrieval task with a horizon that exceeds the training context, for example 600 context frames followed by 600 retrieval frames (1200 total), and plot retrieval PSNR versus distance from the target beyond frame 800. If PSNR after frame 800 drops toward the Mamba2-with-local-attention curve in Fig. 7, the fixed-state capacity assumption fails and the long-term memory claim must be scoped to memory within the training context; if PSNR stays flat, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the fixed-dimensional per-block SSM state (Sec. 4.1) provides long-term memory at constant per-frame cost. The load-bearing assumption is that this state can retrieve exact spatial layout over the whole rollout. The paper's own Limitations state that the method cannot effectively handle memory longer than the training context length, and the experimental protocol never challenges this: Maze retrieval is 400 context frames plus 400 generated frames, Maze reasoning is 576 plus 224 (800 total), and training uses 400 then 800 frames (Sec. S2); Minecraft uses 100 plus 50 frames with 300-frame training. Every evaluation is therefore within the training context. For an interactive world model the rollout is indefinite; once the horizon exceeds training context, the per-block state must either overwrite earlier spatial information or compress it further, and there is no evidence it can. This is the same failure mode shown for Mamba2 with local attention in Table 3 and Fig. 6, where state capacity was insufficient to recall ball positions; block-wise scanning increases capacity only by a fixed factor (H W divided by bh bw) and does not eliminate saturation at longer horizons. Thus the headline 'long-term memory with constant per-frame inference speed' overclaims relative to the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an autoregressive video diffusion world model that replaces full-sequence attention with a block-wise causal state-space model (SSM) scan plus local frame attention, and introduces a training scheme that keeps a random clean prefix during diffusion forcing. The authors claim constant per-frame inference cost and linear training cost while retaining long-term memory, and they evaluate spatial retrieval and reasoning on Memory Maze and Minecraft, reporting that the proposed method outperforms sub-quadratic baselines and approaches a full-context causal transformer.","tokens_in":16003,"tokens_out":8663,"duration_ms":81148,"significance":"The architecture addresses a real bottleneck: video world models based on causal attention either lose memory under sliding-window inference or incur linear per-frame costs with full context. The block-wise scan is a sensible way to increase SSM state capacity for spatial detail, and the clean-prefix training heuristic is a plausible mechanism for encouraging long-range dependence. The complexity analysis in Table 1 and Fig. 8 is clear, and the retrieval/reasoning tasks are well chosen to expose memory failures. The paper is also honest about its limitations (no extrapolation beyond training context, low-resolution experiments). If the empirical margins survive a proper variance analysis, the method is a useful step toward efficient long-horizon world models.","major_comments":[{"comment":"All memory evaluations are conducted within the training context length: Maze retrieval uses 400 context + 400 generated frames, Maze reasoning uses 576 + 224, Minecraft uses 100 + 50, while training uses 400/800 frames for Maze and 300 frames for Minecraft (Sec. S2). The paper's own Limitation states that 'our method cannot effectively handle memory longer than the training context length.' Consequently, the 'Long Memory ✓' in Table 1 and the abstract's claim of maintaining long-term memory at constant per-frame cost are not supported for rollouts exceeding the training context. Because a sliding-window causal transformer also has constant per-frame cost but no memory, the claimed advantage hinges on memory beyond the window, yet the experiments never test memory beyond the training context. Please either qualify the claim to 'memory up to the training context length' or add an experiment that generates past this horizon (e.g., train on 400 frames and generate 800, or apply a length-extrapolation method).","section":"Sec. 5.3, Sec. 6, Table 1"},{"comment":"No error bars, confidence intervals, or number of seeds are reported, despite diffusion sampling being stochastic. The margins over the strongest sub-quadratic baseline are small in the reasoning task (Table 3: PSNR 28.2 vs. 27.5 for Mamba2 + Frame Local Attn; Table 4: 17.8 vs. 17.1 for DFoT). Without variance estimates, these differences cannot be distinguished from sampling noise, which weakens the central claim that the method 'surpasses baselines.' Please report mean and standard deviation over at least three seeds, or provide per-trajectory scatter plots.","section":"Tables 2-5, Table S1"},{"comment":"The ablation row 'Ours w/o block-wise scan' reports exactly the same scores as 'Mamba2 + Frame Local Attn' in Table 3 (SSIM 0.845, LPIPS 0.113, PSNR 27.5), so the entire improvement over the strongest sub-quadratic baseline is attributable to the block-wise scan, and that improvement is only 0.7 PSNR. Moreover, the block sizes (bh, bw) per layer are never specified in Sec. 4.1 or Sec. S2, so the method is not reproducible and the claimed temporal-memory versus spatial-coherence trade-off cannot be checked. Please specify the per-layer block sizes and, ideally, show sensitivity to this hyperparameter.","section":"Table 5 / Sec. 5.4"}],"minor_comments":[{"comment":"The second sentence of the caption says 'retrieval task' but the task uses 576 context frames and 224 generated frames, which is the reasoning task described in Sec. 5.2; please correct the caption.","section":"Fig. 6 caption"},{"comment":"The table header says 'Reasoning (200 Frames)' while the corresponding results in Table 3 and Sec. 5.2 use 224 frames; please align the numbers.","section":"Table 5"},{"comment":"The text says 'For both tasks, we compare our method against baselines with sub-quadratic training complexity' immediately before describing the full-context causal transformer as a reference; please clarify that the reference model is not sub-quadratic.","section":"Sec. 5.3"},{"comment":"The row for 'Bidirectional attention' lists 'Cubic' for the AR inference complexity, but the text in Sec. 1 says bidirectional attention has quadratic complexity when generating a single frame; please reconcile the table with the text.","section":"Table 1"},{"comment":"Please state the number of evaluation trajectories and the random seed(s) used for the retrieval and reasoning metrics, since the variance of these metrics is not otherwise assessable.","section":"Sec. 5.2 / Tables 2-4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core architecture idea is promising, but the main claim of long-term memory is currently supported only within the training context, and the small empirical margins lack error bars. The authors should be encouraged to add variance estimates and consider an extrapolation experiment or a more careful qualification of the claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The new thing is the block-wise causal SSM scan combined with frame-local attention and the clean-prefix diffusion forcing training scheme. That combination is, as far as I can tell, not in the cited prior work, and it pays off empirically: the model beats Mamba2, Mamba2+local attention, and a 192-frame causal transformer on both Memory Maze tasks, and comes within a couple dB of a full-context transformer. The FVD result over 560 generated frames (38.9 vs 45.1) is actually better than full context, which is striking. The ablations show each piece contributes, which is the right way to make the case. The complexity story in Table 1 is correct: constant per-frame inference with a fixed state.\n\nThe soft spots are real but not fatal. No error bars anywhere, and the margins over the strongest sub-quadratic baseline in the reasoning task are small. No code. Block sizes, window sizes, and the chunking scheme in the appendix are hand-set; I'd want a sensitivity analysis. And the abstract oversells \"interactive applications\"—the Limitations section says they don't hit interactive frame rates yet.\n\nThe biggest issue is the stress-test point, and it lands: the model's memory is only demonstrated within the training context length. Maze retrieval uses 400 context + 400 generated frames, reasoning 576 + 224, Minecraft 100 + 50; training uses 400 then 800 frames for Maze and 300 for Minecraft. The Limitations explicitly says the method \"cannot effectively handle memory longer than the training context length.\" So \"long-term memory\" here means \"within training context,\" not indefinite rollout. That undercuts the strongest version of the claim. But it doesn't undercut the core contribution: a fixed-cost architecture that uses a compressed state to retrieve spatial layout over hundreds of frames is still a meaningful advance over sliding-window transformers. The paper is honest about the limit, just not loud enough in the abstract.\n\nWho benefits: anyone working on autoregressive video diffusion, world models, or SSM-based generation. I'd send it to referees—the architecture and results deserve scrutiny, and the authors need to add seeds, release code, and temper the abstract. Not a desk reject.","headline":"A genuinely useful architecture for long-context video world models, with honest but under-emphasized limits on memory beyond the training context.","tokens_in":719,"tokens_out":744,"would_cite":true,"duration_ms":29029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a block-wise state-space scan plus local causal attention lets a video world model recall scenes from hundreds of frames earlier at constant per-frame cost.","keywords":["video world models","state-space models","diffusion forcing","long-term memory","autoregressive video generation","spatial retrieval","Memory Maze","Mamba"],"falsifier":"Run the same retrieval protocol on a maze with much denser visual detail (more objects, textures, higher resolution) and check whether PSNR stays flat as retrieval distance grows to 800 frames; a clear drop with scene complexity or horizon would show the fixed block state is the bottleneck. A complementary probe is to decode the SSM state and test whether object positions can be linearly read out from it.","tokens_in":15539,"feed_emoji":"🧠","tokens_out":9681,"duration_ms":96570,"temperature":0.7,"pith_summary":"This paper claims that an autoregressive video world model can keep long-term memory without paying attention's quadratic price, by replacing global temporal attention with a block-wise state-space model (SSM) scan while retaining dense local causal attention between neighboring frames. The architecture keeps per-frame inference time and memory constant as the generated video grows, and training cost grows only linearly with sequence length. On the Memory Maze retrieval and reasoning benchmarks, the model recovers previously visited regions with quality close to a full-context causal transformer (retrieval PSNR 30.8 versus 32.6), and on Minecraft reasoning it beats the diffusion-forcing transformer baseline. If correct, this removes the main scaling obstacle to interactive world models that simulate persistent environments.","feed_headline":"Video world model retrieves scenes seen 800 frames earlier","feed_subtitle":"Block-wise state-space scan plus local attention keeps near-full-context quality at linear training cost.","key_machinery":"The load-bearing object is the block-wise SSM scan. The flattened token sequence of each video is partitioned into blocks of size $(b_h, b_w, T)$, and each block is scanned independently by a selective state-space model, so temporally adjacent tokens are only a block apart in scanning order instead of a full frame apart, and each spatial block owns a separate state that increases the total memory capacity. This is paired with frame-local causal attention that lets tokens in one frame attend to the same frame and the previous $k$ frames, compensating for the well-known weak associative recall of SSMs and keeping adjacent frames coherent. A modified diffusion-forcing training scheme, where a random-length prefix of frames is kept completely clean and loss is computed only on noised frames, forces the model to read distant clean context. Together these mechanisms make training linear in sequence length and inference constant per frame.","core_discovery":"The paper's central claim is that the temporal memory of a video world model should be carried by causal state-space models, not by attention, provided the scan order is restructured so that memory and spatial coherence are traded off deliberately. A block-wise scan partitions the flattened spatio-temporal token sequence into spatial blocks of size $(b_h, b_w, T)$ and runs an independent SSM scan along time within each block; temporally adjacent tokens become close in scanning order, and each block gets its own fixed-size state, which enlarges total state capacity. A frame-local causal attention module (a window of the previous $k=10$ frames) then restores short-range coherence and per-frame visual quality. The authors also modify diffusion forcing so that a random-length prefix of frames is kept completely clean during training, forcing the model to learn to rely on distant clean context rather than only nearby noisy frames. On Memory Maze and TECO Minecraft, this combination yields long-horizon spatial retrieval and reasoning that approaches full-context causal transformers; on a long-horizon FVD evaluation the paper reports 38.9, below the 45.1 of the full-context causal transformer.","pith_inferences":["Editorial: the block-wise scan is effectively a way to enlarge the SSM's fixed state without changing per-frame cost, so the memory ceiling should scale with the number of blocks and the information density each block must compress; a natural next test is whether the gains hold in high-resolution, object-dense scenes.","A testable extension: vary the random clean-prefix length in training and measure retrieval accuracy at different retrieval distances; the paper's design implies the prefix distribution sets the memory horizon the model learns.","An implication the paper leaves implicit is that the same hybrid scan could be applied to other causal sequence generators beyond video diffusion, such as token-based autoregressive world models, wherever long-range spatial consistency matters.","Because the retrieval evaluation uses exactly invertible action sequences, an open question is how memory behaves under partial reversals or noisy actions; the Minecraft reasoning experiment only partially addresses this."],"forward_implications":["A world model built this way can reproduce an area the agent saw hundreds of frames earlier, as the retrieval and reasoning experiments show.","Per-frame inference time and memory stay constant as rollout length grows, so indefinite interactive generation no longer gets slower over time.","Training on longer context costs only linearly, making longer memory affordable instead of quadratically prohibitive.","The remaining gap to a full-context causal transformer is small (about 1.8 dB PSNR on retrieval and 0.6 dB on reasoning), and on FVD the hybrid model is reported ahead.","A practical consequence is that game-like environments can keep a persistent layout when the camera turns away and later returns, rather than regenerating inconsistent content."],"supporting_citations":[{"why":"supplies the diffusion forcing training scheme that the paper extends with a clean-prefix variant.","marker":"[10]"},{"why":"introduces the selective state-space model that the block-wise scan is built on.","marker":"[20]"},{"why":"provides the Mamba2 SSM backbone used as a baseline and as the base for the hybrid model.","marker":"[13]"},{"why":"the diffusion-forcing transformer baseline whose limited context and quadratic cost motivate the comparison.","marker":"[59]"},{"why":"describes autoregressive video diffusion with causal attention, the sliding-window baseline that loses long memory.","marker":"[86]"},{"why":"defines the Memory Maze dataset and the long-term memory evaluation tasks.","marker":"[50]"},{"why":"provides the TECO Minecraft dataset used for the second evaluation.","marker":"[78]"},{"why":"shares the clean-context training idea that the paper adapts for long-context diffusion forcing.","marker":"[18]"}],"fun_headline_variants":["SSM video world model remembers 800 frames back","State-space video model extends memory without attention cost","Block-wise SSM scan gives video models long-term memory","Causal SSM scan trades spatial detail for temporal depth","Video world model keeps memory via state-space scans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a fixed-size compressed state per spatial block can store exact scene layout, wall geometry, and object positions well enough for hundreds of frames, so that revisits are reproduced rather than guessed.","fun_headline_variants_meta":{"raw":{"variants":["SSM video world model remembers 800 frames back","State-space video model extends memory without attention cost","Block-wise SSM scan gives video models long-term memory","Causal SSM scan trades spatial detail for temporal depth","Video world model keeps memory via state-space scans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3753,"prompt_tokens":929,"completion_tokens":2824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2748}},"tokens_in":545,"tokens_out":2824,"duration_ms":21890,"temperature":1.0,"reasoning_tokens":2748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:57:35.402910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same retrieval protocol on a maze with much denser visual detail (more objects, textures, higher resolution) and check whether PSNR stays flat as retrieval distance grows to 800 frames; a clear drop with scene complexity or horizon would show the fixed block state is the bottleneck. A complementary probe is to decode the SSM state and test whether object positions can be linearly read out from it.","supporting_citations":[{"cited_title":"Mamba: Linear-time sequence mod- eling with selective state spaces","cited_arxiv_id":null,"evidence_quote":"introduces the selective state-space model that the block-wise scan is built on."},{"cited_title":"From slow bidirectional to fast autoregressive video diffusion mod- els","cited_arxiv_id":null,"evidence_quote":"describes autoregressive video diffusion with causal attention, the sliding-window baseline that loses long memory."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the TECO Minecraft dataset used for the second evaluation."}],"review_version":1}