{"id":"2e69cb2e-257c-41c7-aff5-22928394cef0","arxiv_id":"2411.19460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Video-Ma2mba shows that a Mamba-2 backbone with multi-axis gradient checkpointing can train on very long video token sequences with near-linear memory while scoring competitively against much larger video-LLMs.","lead":"This paper replaces the attention layers in a video-understanding language model with Mamba-2 state-space layers and adds a two-axis gradient checkpointing scheme to cut training memory. The resulting 3.1B model processes up to millions of video tokens on one GPU, around 2 hours of frames at 1 FPS, and roughly matches larger 7B-8B models on long-video QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MA-GC's exact recompute premise assumes a first-order Markov recurrence, but standard Mamba-2 blocks include a causal Conv1d; checkpointing only SSD states makes the recomputation inexact unless conv buffers are also stored.","rationale":"The reader's weakest_assumption already flags the strict-Markov premise of the memory analysis, and I agree that this is a genuine weak point. However, I think it is more load-bearing than the accuracy-delta concern: if exact recomputation fails because of the unaccounted causal convolution, then the training procedure itself is unsound and all benchmark numbers become unreliable. The proposed gradient-equivalence test would settle this decisively. The reader's CONDITIONAL verdict is therefore not changed; it is simply reinforced with an additional, more fundamental condition. If the test passes (conv buffers are correctly checkpointed or the conv is absent), the memory claim is likely sound and the paper's remaining issues are the ones the reader already identified.","tokens_in":18151,"tokens_out":22173,"duration_ms":157506,"concrete_test":"Run a gradient-equivalence test on a small official Mamba-2 model (1 block, d=64, conv kernel=4): implement MA-GC exactly as in Algorithm 1 with Sckpt storing only SSD states, set S=1024 and s=128, and compare every parameter gradient against a full-storage reference backprop. If the max relative error exceeds 1e-5, exact recomputation fails. Also inspect the Mamba-2 block source for CausalConv1d; if present and its buffer is not saved at sequence checkpoints, the failure is explained. If the authors' implementation saves conv buffers, the memory constants in Appendix C and the O(S) analysis should be re-derived to include that state.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The memory analysis in Sections 3.2-3.3 is built on Eq. (1), which reduces Mamba-2 to h_t = A_t h_{t-1} + B_t x_t. On that basis, Algorithm 1 saves only the SSD state h as the sequence-wise checkpoint (Sckpt), and Eq. (4) counts no other per-step state. However, the Mamba-2 block defined in the cited reference [11] applies a causal Conv1d (kernel size > 1) to the input before the SSD. The output at time t then depends on x_{t-k+1...t}. If a recomputation cell begins at a sequence checkpoint at time s, computing the conv at time s+1 needs x_{s-k+2...s}; with only h_s and x_s stored, those values are not available. The paper neither states that the conv is removed nor that the checkpoint stores the conv buffer. If the implementation follows the pseudocode, the forward recomputation is inexact and the gradients are wrong, invalidating the trained results. If it instead caches conv buffers, the memory model in Eq. (4) undercounts and the description is incomplete. Either way, the central claim that MA-GC recomputes exactly from a state every s steps is not supported by the written method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Video-Ma2mba, a video large multi-modal model that replaces the Transformer backbone with a Mamba-2 state-space model, and introduces Multi-Axis Gradient Checkpointing (MA-GC) to reduce training activation memory from O(L·S) to O(S). The model is trained in three stages, including a long-video knowledge stage on the authors' SceneWalk dataset, and is evaluated on Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, and MVBench. The main claimed contributions are (i) a linear-complexity backbone for long video, (ii) a bi-axis gradient checkpointing scheme with a memory analysis in Eqs. (4)-(11), and (iii) empirical results showing that a 3.1B model can process sequences up to 0.8M tokens during training and 2M tokens at inference on a single 80GB GPU.","tokens_in":18401,"tokens_out":16828,"duration_ms":139275,"significance":"If the central claims hold, the memory engineering result is useful: the paper provides a parameter-free scaling analysis, reports measured peak memory across sequence lengths, and the qualitative trend in Table 3 is consistent with a sublinear memory improvement over no checkpointing. The system-level demonstration of 2M-token inference on one GPU is also noteworthy. However, the accuracy comparisons are overstated: there is no same-scale Transformer baseline trained under identical conditions, and the headline numbers are essentially tied with existing 7B-8B models (e.g., 45.2 vs 45.3 for SliME on Video-MME). More importantly, the exact-recompute premise of MA-GC is not justified for actual Mamba-2 blocks because the causal Conv1d is ignored, and the claimed advantage over existing sequence-axis checkpointing is not established. The novelty and validity of the core method therefore require substantive revision.","major_comments":[{"comment":"The exact-recompute premise of MA-GC is not supported for real Mamba-2 blocks. Eq. (1) simplifies the block to h_t = A_t h_{t-1} + B_t x_t, and Algorithm 1 saves only the SSD hidden state h as the sequence-wise checkpoint. In the cited Mamba-2 architecture [11], a causal Conv1d is applied to the input before the SSD, so the block output at time t depends on x_{t-k+1},...,x_t. Starting from a checkpoint at time s with only h_s and x_s, the recomputation at s+1 cannot reproduce the convolution. If the implementation caches convolution buffers, Eq. (4) undercounts memory and the description is incomplete; if it does not, the recomputed forward pass and the resulting gradients are inexact. The paper never states that the Conv1d is removed from the backbone. Since the trained results in Tables 1-2 and the memory measurements in Table 3 rest on this step, the central methodological claim needs either a corrected memory model or an explicit statement of the implemented variant.","section":"Section 3.2, Algorithm 1"},{"comment":"The claimed improvement over existing gradient checkpointing omits the sequence-axis checkpointing for RNNs from Chen et al. [6], which the paper itself cites. For a fixed stack of L layers, storing sequence checkpoints every s steps requires Θ(S + LS/s + Ls) peak memory (input activations, hidden states, and recomputation cell); choosing s = √S gives Θ(S), and the constant is no larger than what MA-GC achieves with its optimal layer interval l = L in Eq. (4), namely S + L√S + L√S. Thus the layer axis of MA-GC does not reduce the asymptotic memory below the existing sequence-axis method; it only adds the S-term for layer-wise checkpoints. The comparison in Table 3 is only against layer-grouping baselines ('GC on' and 'Sqrt GC'), not against time-axis checkpointing from [6]. The paper should compare against sequence-axis-only checkpointing and clarify what the layer axis contributes beyond Chen et al.","section":"Section 3.3, Eqs. (4)-(11)"},{"comment":"The accuracy claims are overstretched relative to the evidence. Table 1 compares only against published numbers from models with different training data, backbones, and evaluation protocols; no same-scale Transformer baseline trained under identical conditions is provided. On Video-MME, Video-Ma2mba-3.1B (45.2) is statistically tied with SliME-8B (45.3), so the abstract's 'substantial advantages over existing frameworks' is unsupported. In Table 4, the key motivational comparison between full 1 FPS input and 32-frame sampling (45.2 vs 44.6) is a 0.6-point gap with no error bars or repeated runs. The text in Section 4.2 additionally reports a '+1.7 points (4.1%) improvement' that does not match any clean row of Table 4 (the closest all-stage comparison is 45.2 vs 43.7, a 1.5-point gap). The evidence that processing all frames at 1 FPS materially improves accuracy over dense-but-sampled input is not yet convincing.","section":"Section 4.2, Tables 1 and 4"}],"minor_comments":[{"comment":"In the 1.3B MA-GC row, the entry at S=2^13 appears as '.2.1' and should be '2.1'.","section":"Table 3"},{"comment":"The text states that models are trained for one epoch at each stage, while Table 6 reports 'train epochs 2' for all stages; this contradiction should be resolved.","section":"Section 3.5 vs Table 6"},{"comment":"The reported '+1.7 points (4.1%)' improvement does not correspond to a direct comparison in the table; please specify which rows are being compared.","section":"Section 4.2, Table 4"},{"comment":"The measured MA-GC values in Table 3 increase by a factor of roughly 1.6 per doubling of S, which appears more consistent with a sublinear Θ((LS)^{2/3}) trend than with the claimed Θ(S) boundary solution; the authors should report the actual l and s used for each row and reconcile the measured scaling with Eq. (10).","section":"Section 3.3 vs Table 3"},{"comment":"The evaluation uses gpt-3.5-turbo-0125 for response scoring, but no temperature, decoding parameters, or prompt template for the scorer are given; this limits reproducibility.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The Stage 1.5 training relies on the authors' own SceneWalk dataset [19], which is not publicly available and is from the same group; the paper should clarify its availability and any potential overlap with the evaluation benchmarks. The technical core of the paper, if corrected, would be a memory-engineering contribution; the current framing as a breakthrough in long-video understanding exceeds the empirical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the MA-GC memory analysis is the real contribution: checkpointing on both the layer and sequence axes for a Mamba-2 backbone, with a clean calculus/AM-GM derivation of the optimal intervals and an asymptotic Theta(S) training-activation bound. The measured memory numbers in Table 3 track the claimed scaling, and the throughput trade-off in Appendix D is reported honestly. That is worth referee time.\n\nThe second thing is the accuracy story. The paper claims \"substantial advantages\" over existing frameworks, but the tables show parity at best: 45.2 on Video-MME versus 45.3 for SliME (8B), and the ablation in Table 4 gives 1 FPS only +0.6 over 32-frame sampling when Stage 1.5 is on. There is no same-scale Transformer baseline and no run-level variance. Those claims should be cut in revision.\n\nThe softest spot is the checkpointing write-up. The analysis is built on Eq. (1), which drops the causal conv1d that is part of the Mamba-2 block. Algorithm 1 stores only the SSD state h at sequence checkpoints. To recompute exactly from time s onward, the conv at time s+1 needs the conv inputs from times s-k+2...s. The paper never says the conv is removed, nor that its buffer is checkpointed. If the implementation follows the pseudocode, the recomputation is inexact and the backward pass is wrong; if it caches conv buffers, Eq. (4) undercounts the memory and the method description is incomplete. This is a fixable but substantive gap, and it should be resolved before publication.\n\nTwo smaller issues: the pseudocode initializes h inside the time loop, which would reset the recurrent state each step (I assume that is a typo), and Stage 1.5 relies on SceneWalk, a same-group dataset that is not released, with no code release either. That keeps the benchmark results from being independently checked.\n\nBottom line: the memory-engineering core is plausible and worth engaging; the empirical framing is oversold and the exact-recompute claim needs an explicit conv-state answer. Send to review.","headline":"The memory analysis for the proposed bi-axis checkpointing is the real contribution, but the accuracy claims are oversold and the exact-recompute description leaves the causal conv in Mamba-2 unaccounted for.","tokens_in":18985,"tokens_out":6009,"would_cite":true,"duration_ms":58679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Mamba-2 backbone plus multi-axis gradient checkpointing cuts long-video LMM activation memory from $\\Theta(LS)$ to $\\Theta(S)$, enabling 0.8M-token training and 2M-token inference on one 80 GB GPU.","keywords":["video understanding","state space models","Mamba-2","gradient checkpointing","long-context language models","large multimodal models","memory efficiency","1 FPS video sampling"],"falsifier":"Measure peak activation memory for Mamba-2-2.7B at $S=2^{19}$ with MA-GC on an 80 GB A100 in BF16; the paper predicts roughly 42.2 GB, so a measured value above 80 GB (or failure at $S=2^{20}$) would refute the $\\Theta(S)$ bound. Separately, retraining Video-Ma2mba-3.1B with 32-frame sampling under otherwise identical conditions and comparing Video-MME would test whether the 0.6-point advantage of 1 FPS over 32 frames is real.","tokens_in":17941,"feed_emoji":"🎥","tokens_out":7520,"duration_ms":64185,"temperature":0.7,"pith_summary":"Video-Ma2mba claims that a large multimodal model for video can stop sampling a handful of frames and instead ingest every frame at 1 FPS, because replacing the Transformer backbone with Mamba-2 turns quadratic memory into linear memory. To make that feasible in training, the paper introduces Multi-Axis Gradient Checkpointing (MA-GC), which stores activations on a grid along both the layer axis and the sequence axis and reduces activation memory from $\\Theta(LS)$ to $\\Theta(S)$ for a fixed number of layers $L$. If the claim holds, a 3.1B-parameter model can train on 0.8M-token sequences and answer from up to 2M tokens on a single 80 GB GPU, corresponding to more than two hours of video at 1 FPS. On Video-MME and LongVideoBench it scores 45.2 and 43.0 respectively, matching or beating several 7B-8B Transformer-based video models.","feed_headline":"New checkpointing lets a 3.1B video model watch 2 hours at 1 FPS","feed_subtitle":"Replacing attention with Mamba-2 cuts memory from quadratic to linear, lifting Video-MME to 45.2.","key_machinery":"The key mechanism is the multi-axis checkpoint grid. Layer-wise checkpoints store the input to every $l$-th layer; sequence-wise checkpoints store the hidden state every $s$-th time step; their intersections define grid cells that are recomputed forward and then backpropagated through during the backward pass. This is possible only because Mamba-2 (a structured state space model) advances one step at a time and needs the immediately previous state, unlike attention, which needs all past keys and values. The memory formula $M = LS/l + LS/s + ls$ and its minimizer $l=s=\\sqrt[3]{LS}$ are what carry the theoretical claim of $\\Theta(S)$ memory.","core_discovery":"The central discovery is that Mamba-2's Markov hidden-state recurrence, $h_t = A_t h_{t-1} + B_t x_t$, makes gradient checkpointing possible on a second axis: instead of storing activations only every $l$ layers, the model also stores sequence states every $s$ time steps, so gradients are reconstructed cell-by-cell on an $L\\times S$ grid. With the checkpoint intervals chosen to minimize $M = LS/l + LS/s + ls$, the memory bound becomes $\\Theta((LS)^{2/3})$ in general and $\\Theta(S)$ in the long-sequence regime $L^2 \\le S$, down from $\\Theta(LS)$ without such checkpointing. The paper then builds Video-Ma2mba on this memory engine, training a 3.1B Mamba-2-based LMM on dense 1-FPS video data, and reports that it processes up to 0.8M tokens during training, up to 2M at inference, and achieves 45.2 on Video-MME and 43.0 on LongVideoBench.","pith_inferences":["The bi-axis checkpointing recipe should transfer to any recurrent or linear-RNN backbone with a Markov state, so the $O(S)$ memory result is likely a general tool for SSM-based LMMs, not a video-specific trick.","An independent repeated-run comparison is needed to see whether the 0.6-point gap between 1 FPS and 32-frame sampling is stable; if it is not, the main proven contribution would be memory engineering rather than accuracy.","Combining MA-GC with targeted frame selection or memory banks could address the paper's own note that salience-based sampling may be better for very long videos; MA-GC would then handle memory while retrieval handles relevance.","Because throughput drops 35%, the method's practical value is best tested on tasks where dense temporal coverage matters more than wall-clock speed, such as long-duration activity localization or narrative reasoning."],"forward_implications":["A single 80 GB GPU can train Video-Ma2mba-3.1B on sequences of 0.8M tokens, and the same model can answer with up to 2M input tokens, so videos beyond two hours at 1 FPS are in reach without frame dropping.","Memory for Mamba-2 activations grows linearly with sequence length rather than quadratically, so doubling video duration roughly doubles activation memory.","The 3.1B model reaches 45.2 on Video-MME and 43.0 on LongVideoBench, outperforming most 7B-8B Transformer-based video LMM baselines despite being smaller.","Including the intermediate long-video knowledge stage (Stage 1.5) and 1 FPS full-frame input together add about 1.8 points over the no-Stage-1.5 baseline on Video-MME.","MA-GC costs about 35% throughput but extends trainable sequence length by up to 32 times compared with no checkpointing, making the memory-for-time trade-off explicit."],"supporting_citations":[{"why":"Introduces Mamba-2 and Structured State-Space Duality, the backbone whose linear recurrence Video-Ma2mba replaces attention with and whose state scan constrains checkpoint spacing.","marker":"[11]"},{"why":"Establishes gradient checkpointing for deep networks and RNNs along layer and time axes, the foundation that MA-GC extends to two axes simultaneously.","marker":"[6]"},{"why":"Defines Mamba and selective state spaces, providing the RNN-like recurrence that justifies sequence-wise checkpointing.","marker":"[15]"},{"why":"Supplies the 11.8K-hour long-video dense-caption dataset used in Stage 1.5 to train the model on full 1-FPS sequences.","marker":"[19]"},{"why":"Provides the Video-MME benchmark with short, medium, and long video durations used to measure long-video understanding.","marker":"[13]"},{"why":"Provides LongVideoBench, a second long-context video QA benchmark used for the main comparisons.","marker":"[43]"},{"why":"Motivates adding an intermediate knowledge-learning stage between alignment and SFT, which the paper expands to long video.","marker":"[27]"}],"fun_headline_variants":["Mamba-2 replaces attention to watch 2-hour videos on one GPU","Video-Ma2mba: linear memory for 2-hour video understanding","Multi-axis gradient checkpointing cuts memory for long videos","One GPU processes 2 hours of video at 1 FPS via Mamba-2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument depends on the assumption that a Mamba-2 hidden state is a complete and exact summary of the past, so the backward pass can be rebuilt from checkpoints spaced every $s$ steps, and on the empirical premise that full 1-FPS input is worth the added memory engineering; if the recurrence cannot be resumed exactly, or if 1 FPS gains are negligible, the central claim unravels.","fun_headline_variants_meta":{"raw":{"variants":["Mamba-2 replaces attention to watch 2-hour videos on one GPU","Video-Ma2mba: linear memory for 2-hour video understanding","Multi-axis gradient checkpointing cuts memory for long videos","One GPU processes 2 hours of video at 1 FPS via Mamba-2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3933,"prompt_tokens":1005,"completion_tokens":2928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2847}},"tokens_in":621,"tokens_out":2928,"duration_ms":19540,"temperature":1.0,"reasoning_tokens":2847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:10:21.402940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure peak activation memory for Mamba-2-2.7B at $S=2^{19}$ with MA-GC on an 80 GB A100 in BF16; the paper predicts roughly 42.2 GB, so a measured value above 80 GB (or failure at $S=2^{20}$) would refute the $\\Theta(S)$ bound. Separately, retraining Video-Ma2mba-3.1B with 32-frame sampling under otherwise identical conditions and comparing Video-MME would test whether the 0.6-point advantage of 1 FPS over 32 frames is real.","supporting_citations":[{"cited_title":"Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"Motivates adding an intermediate knowledge-learning stage between alignment and SFT, which the paper expands to long video."}],"review_version":1}