{"id":"786a05bb-6b03-4788-b050-4477d3bdfda3","arxiv_id":"2607.24904","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.","lead":"Mage-VL is a 4B streaming vision-language model that keeps only motion- and residual-rich video patches from standard codecs, cutting visual tokens by about 75%. It matches a same-size dense VLM on images, improves video and spatial scores, and runs up to 3.5× faster wall-clock.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The \"differs only in the visual front-end\" framing overstates causal credit: Mage-VL's flagship gains over Qwen3-VL-4B are confounded by a heavier custom data/curriculum stack, and the tracking block makes the confound visible in the numbers themselves.","rationale":"The reader's verdict (CONDITIONAL, medium correctness risk) already identifies exactly this confound; my pass confirms it is the load-bearing concern and finds in-text evidence that makes it more concrete than the reader stated: the tracking rows (+18–20 J&F) align with Molmo2 tracking SFT data, and the spatial/grounding gains align with dedicated spatial and temporal-grounding supervision, neither of which Qwen3-VL received. I also considered two alternative concerns and judged them secondary: (1) the 3.5× speedup is selectively drawn from NextQA in Table 5, while Mage-VL is actually slower than Qwen3-VL on TempCompass (729s vs 433s), Charades (729s vs 707s), and VSI-Bench (537s vs 255s), and baseline times exclude video-loading time—this qualifies the efficiency claim but the matched-budget Fig. 8 frontier partially addresses it; (2) several Findings (3, 4, 7) are single-pipeline observations, which the reader already discounted in the novelty/confidence scoring. Neither displaces the attribution confound as the central soft spot. Because the reader already priced this in with a CONDITIONAL verdict and medium correctness risk, I recommend no verdict change: the concern argues for keeping (not weakening further) the CONDITIONAL status pending a matched-recipe control, and the within-model chunk-vs-codec test above is cheap enough to run with the released weights.","tokens_in":43350,"tokens_out":1942,"duration_ms":20196,"concrete_test":"Run Mage-VL-4B itself in chunk-wise (dense uniform frame-sampling) mode—which Mage-ViT natively supports (Table 1 evaluates both modes at probe level)—at a matched 4096-token budget on VideoMME, MLVU-dev, Timelens-QVHighlight, VSI-Bench, and Ref-DAVIS17, and compare against the codec-mode numbers in Tables 4–5. If the same-model codec-vs-chunk delta is much smaller than the Mage-VL-vs-Qwen3-VL delta (e.g., the tracking gap collapses or the VSI/grounding gains mostly survive in chunk mode), then the headline gains are predominantly data/recipe effects and the \"front-end\" framing must be downgraded. A stronger but costlier variant: retrain the full five-stage curriculum with a dense ViT front-end and re-measure Table 4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest-assumption call is correct, and the full text sharpens it. Table 4's caption asserts Mage-VL and Qwen3-VL-4B \"share the 4B Qwen3 LLM backbone and differ only in the visual front-end.\" But §§4.2–4.3 show the models also differ in: ~350M AI4AI-recaptioned image captions (vs. Qwen3-VL's own recipe), 7.95M duration-stratified video captions, Stage 4 codec-stream supervision, Stage 5 gate data, and—critically—task-specific SFT data that Qwen3-VL did not train on. The tracking rows of Table 4 are the tell: Mage-VL scores 25.83/22.55/25.57 J&F on Ref-DAVIS17/MeViS/Ref-YT-VOS versus Qwen3-VL's 7.48/3.16/5.28, an +18–20 point gap. §4.2.2 and §4.3 state Mage-VL is explicitly trained on Molmo2-VideoTrack and Molmo2-VideoPoint. A front-end tokenization change cannot plausibly produce a 3–8× jump in referring-video segmentation; supervised tracking data can and does. Similarly, the +11.0 VSI-Bench gain coincides with 4M spatial samples and dedicated spatial supervision in Stages 2–4 (§4.3), and the Timelens grounding gains (+7.6/+17.1/+22.5) coincide with temporal-grounding data (TimeLens, VideoChat-Flash) added in Stage 3. The paper's own §5.3.2 acknowledges \"rigorous single-variable ablations remain challenging due to the tightly coupled joint pre-training recipe\"—but only for Finding 4, not for the Table 4 framing. The encoder-level claim has independent support (Table 1 probes and Fig. 8 matched-budget curves), so the codec-native design is not empty; what is unsupported is attributing the *head-to-head VLM gains* primarily to it rather than to the data recipe. This lands directly on the strongest claim's \"matches Qwen3-VL on static, improves video/spatial\" causal framing, and is a correctness-risk issue (attribution), not a consensus disagreement.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper presents Mage-VL-4B, a streaming vision-language model whose visual front-end (Mage-ViT) tokenizes video codec-natively: I-frame patches are kept dense while P-frame patches are selected by codec-derived importance (motion-vector magnitude + residual energy in HEVC, or negative log-likelihood in the neural codec DCVC-RT), cutting visual tokens by ~75%. Mage-ViT is trained from scratch with cluster discrimination against MetaCLIP-derived K-means prototypes on ~560M images and 100M video frames. On top, a five-stage curriculum (captions → instruction → long video → codec-stream → streaming gate) builds the full model, with a System-1 event gate deciding when a frozen System-2 decoder should speak. Reported results: Mage-ViT matches SigLIP2-class encoders on linear/attentive probes (Table 1); Mage-VL-4B matches Qwen3-VL-4B on static image tasks and exceeds it on most video QA, temporal grounding, spatial, and tracking benchmarks (Tables 3–5), with up to 3.5× wall-clock speedups; streaming results on SoccerNet-Caption and OVO-Bench (Tables 6–7); plus ablations on caption-pipeline optimization and a Zero-Vision SFT + RL study. Support is broad: frozen-encoder probes, matched-budget scaling curves (Fig. 8), cross-codec transfer without retraining (Table 2), and a standardized lmms-eval protocol. The central weaknesses are causal attribution: the Table 4 claim that Mage-VL and Qwen3-VL-4B \"differ only in the visual front-end\" is confounded by large training-data/curriculum","tokens_in":43883,"tokens_out":3974,"duration_ms":155050,"significance":"If the results hold, the work makes three useful contributions: (i) evidence that a codec-aligned sparse front-end preserves representation quality at ~75% fewer tokens, supported by encoder-level probes rather than only end-task scores; (ii) a falsifiable efficiency claim (Fig. 8, Table 5) with matched-token-budget curves and wall-clock numbers on stated hardware; and (iii) an open release of encoder, model, and code, plus a set of clearly enumerated, individually testable empirical findings. The cross-codec transfer experiment and the Zero-Vision SFT result are each of independent interest to the community. Significance is tempered by the causal-attribution problems above: the headline 'front-end-only' comparison is actually a full-recipe comparison, and Finding 1 omits that the pretraining targets are distilled from a web-scale supervised teacher. These are framing/attribution defects rather than evidence that the engineering results are wrong.","major_comments":[{"comment":"The caption asserts Mage-VL and Qwen3-VL-4B 'differ only in the visual front-end,' but §4.2–4.3 document large training-mixture differences: ~350M AI4AI-recaptioned images, 7.95M video captions, Stage 4 codec-stream supervision, and task-specific SFT data Qwen3-VL did not use. The numbers make the confound concrete: the +18–20 J&F tracking gains (Ref-DAVIS17 25.83 vs 7.48) coincide with explicit Molmo2-VideoTrack/VideoPoint supervision in Stage 4; the +11.0 VSI-Bench gain coincides with 4M spatial samples; the Timelens gains (+7.6/+17.1/+22.5) coincide with TimeLens/VideoChat-Flash data in Stage 3. A front-end change cannot plausibly yield a 3–8× referring-segmentation jump; supervised tracking data can. The authors themselves concede in §5.3.2 (Finding 4) that single-variable ablations are infeasible — the same caveat must be applied to the Table 4 framing. Either retitle the comparison","section":"Table 4 caption and §5.3.2"},{"comment":"Mage-ViT is trained by cluster discrimination against K-means prototypes computed over MetaCLIP features (§3.2, 'Objective'). MetaCLIP is itself trained on billions of image–text pairs, so the representation claim 'trained from scratch on ~560M unlabeled images... matches encoders trained on billions of image–text pairs' (Abstract; Finding 1, §5.2.1) describes distillation from a web-scale-supervised teacher, not label-free learning from unlabeled data alone. This materially weakens Finding 1 as stated ('web-scale pre-training is not essential'). The claim is recoverable in a weaker form (codec-aligned objectives plus prototype distillation are data-efficient), but the text should state the MetaCLIP dependence prominently — abstract, Finding 1, and §5.2.1 — and ideally report a control with self-supervised prototypes (e.g., DINOv2 features) to isolate the effect.","section":"§3.2 (Objective) and Finding 1 (§5.2.1)"},{"comment":"The 'matched budget' in Fig. 8 matches only encoder token workload: tc-N is built from 8N source frames while frame-N sees N frames, so the codec arm consumes 8× more raw visual information plus an HEVC encode/decode pass whose cost is not accounted for. Table 5 says Mage-VL times are 'full measured wall-clock' — please state explicitly whether codec extraction and patchification are included; if not, the '3.5× speedup' (NextQA: tc8/415s vs Qwen/1460s) is not end-to-end. Also note the flagship speedup compares tc8 (64 source frames) against Qwen at 32 frames, i.e., different information budgets in the opposite direction from Fig. 8. A per-stage timing breakdown (codec, ViT, LLM) and at least one accuracy comparison at equal raw-frame count would make Finding 5 sound.","section":"§5.5, Fig. 8, and Table 5"}],"minor_comments":[{"comment":"Table 6 mixes evaluation conditions: StreamMind numbers are taken from its publication, JoyAI is evaluated at 1 Hz with a relaxed ±1s window while Mage-VL uses zero-tolerance canvas matching. The TriggerAcc/TimVal comparison across these protocols is not apples-to-apples; please flag this in the caption or re-run baselines under a single protocol.","section":"Table 6"},{"comment":"The caption says Mage-VL runs at '1fps,' but the text says queries are answered under the SimpleStream recent-window protocol (four most recent frames). Clarify whether Mage-VL was evaluated on codec-token streams or on the 4-frame window, and how the gate/EPFE memory was used (or bypassed) in this offline-style protocol. Backward-Tracing accuracy (48.15%) lagging Real-Time (79.84%) also deserves one sentence of analysis given the streaming-memory design.","section":"Table 7 (OVO-Bench)"},{"comment":"Several regressions vs Qwen3-VL-4B are unexplained: CC-OCR Doc (32.25 vs 39.69), DUDE (46.44 vs 50.98), WebSRC (92.80 vs 95.40), TextVQA (77.28 vs 80.55), Charades in Table 5 (31.4 vs 45.9). Given the OCR/document emphasis of the caption pipeline, the CC-OCR gap is surprising and worth a remark.","section":"Table 3"},{"comment":"'Comprehensively surpasses the 15B Phi-4-reasoning-vision baseline' (Abstract) should be qualified: Phi-4-R-V is a reasoning-tuned model not optimized for video/streaming, and it beats Mage-VL on some rows (MMOU 51.9 vs 39.3, ERQA, MMSI in part). The comparison is fine to include but the abstract phrasing overstates it.","section":"Abstract / Tables 3–4"},{"comment":"Typos/formatting: 'a16× 16patch level' (Abstract); 'Duringlatervideostages' (§4.2.1); Fig. 1 label 'VSI-Bench 64.0' vs Table 4's 64.3; double citation [72]/[100] for StreamMind with different venues; several 2026-dated arXiv references lack links. Appendix A prompt rendering has artifacts (e.g., '‘**bold**‘').","section":"Throughout"},{"comment":"PR-AUC of 9.30% (Table 6) is very low in absolute terms; a sentence contextualizing it against the speak-event base rate on SoccerNet-Caption would help readers calibrate the streaming claim.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The author list overlaps substantially with the teams behind several cited baselines and components (OV-Encoder, LLaVA-OneVision-2, Phi-4 family, DCVC-RT, lmms-eval). This is not improper, but reviewers should note that baseline numbers, the evaluation harness, and parts of the comparator ecosystem originate from closely affiliated groups; independent replication of the headline Table 4 numbers would be valuable. The AI4AI pipeline depends on proprietary components (GPT-5 scorer, Copilot), so that portion of the recipe is not fully reproducible even with the released code."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core here is a from-scratch Mage-ViT that allocates 16×16 patches from codec motion/residual (or neural NLL) signals, plus a light speak/silent gate on rolling codec windows. They actually ship the stack, report wall-clock on 8×B200, show HEVC→DCVC-RT selector transfer without retraining, and give matched-budget tc-N vs frame-N curves. That is more than another “we sample fewer frames” paper.\n\nWhat is new is not the idea of codec bits as importance—CoViAR, Video-LaVIT, and especially OneVision-Encoder / LLaVA-OneVision-2 already sit on that line, and the author overlap is obvious. What is new is the end-to-end 4B streaming system: sparse I/P canvases, Stage-4 codec adaptation, Stage-5 frozen-backbone gate, and the practical 75% token cut with up to ~3.5× measured speedup while staying competitive on static image tables. Encoder probes (Table 1) and the resolution-scaling plot are clean: variable-res pretrain on ~560M images + 100M video frames gets you near SigLIP2-class features without billion-scale image-text pairs. Caption-only long VideoQA and the Zero-Vision SFT→RL side experiment are worth reading even if you never touch codecs.\n\nThe soft spot is attribution, and it is real but localized. Table 4’s caption says Mage-VL and Qwen3-VL-4B “differ only in the visual front-end.” They do not. Mage-VL also gets ~350M AI4AI dense captions, duration-stratified video captions, codec-stream Stage 4, gate data, Molmo2 track/point SFT, TimeLens-style grounding, and heavy spatial mixtures. The tracking block gives the game away: +18–20 J&F on Ref-DAVIS/MeViS/Ref-YT-VOS is what supervised referring-VOS data does, not what a patchifier does. Same pattern on VSI-Bench and Timelens. The paper quietly admits single-variable ablations are hard—only under Finding 4—while the flagship framing still sells front-end causality. Encoder-level and matched-budget evidence still stands; the “sparsity beat Qwen on video/spatial” slogan does not without matched-data controls.\n\nMath is ordinary engineering, not load-bearing theory. Citations are appropriate. Code/weights/prompts are promised; full data regen is still heavy.\n\nFor people building long-context or live VLMs this is worth a careful read and a cite on the efficiency frontier. I would send it to referees: ask for honest ablation language and a clearer split between tokenizer gains and recipe gains. Engage; do not swallow the causal headline whole.","headline":"Solid codec-native streaming stack with real efficiency numbers; the Qwen3-VL head-to-head overclaims causal credit for the front-end because data and task SFT are confounded.","tokens_in":44786,"tokens_out":696,"would_cite":true,"duration_ms":16140,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A 4B streaming vision-language model cuts visual tokens by ~75% via codec motion/residual patches and still matches stronger static baselines while speeding video inference up to 3.5×.","keywords":["codec-native tokenization","streaming vision-language models","Mage-ViT","motion vectors and residual energy","proactive event gate","visual token efficiency","video spatial reasoning","AI4AI data pipelines"],"falsifier":"Train an otherwise identical Mage-VL stack on the same five-stage data but with uniform dense frame sampling at matched token budgets; if video, spatial, and wall-clock advantages over the dense twin disappear, the codec-sparsity claim fails.","tokens_in":44248,"feed_emoji":"🎬","tokens_out":1101,"duration_ms":22724,"temperature":0.7,"pith_summary":"Standard vision-language models are strong at offline image reasoning but wasteful and slow on continuous video because they re-encode mostly static frames. Mage-VL argues the fix is to make the visual front-end codec-native: keep full anchor frames and, on predicted frames, keep only 16×16 patches that motion vectors and residual energy mark as informative, dropping over three-quarters of the tokens while holding spatiotemporal layout via 3D rotary positions. The authors train that encoder, Mage-ViT, from scratch on far less data than flagship image encoders yet match them on probes, then stack a five-stage curriculum and a light System-1 gate that decides when a System-2 language decoder should speak. The resulting 4B model matches a same-scale dense baseline on static tasks, improves video QA, temporal grounding, and 2D/3D spatial scores, and reports up to 3.5× wall-clock speedup. Along the way they package seven empirical recipes: small unlabeled pre-training can suffice, variable resolution scales cleanly, dense captions can replace long VideoQA SFT, motion training helps static spatial skill, closed-loop AI caption pipelines lift quality, and pure-text SFT before multimodal RL can unlock agentic gains.","feed_headline":"Codec patches cut VLM video tokens 75% with 3.5× speedup","feed_subtitle":"A 4B model matches dense static baselines and gains on video and spatial tasks by encoding only motion-rich regions.","key_machinery":"Mage-ViT’s codec-driven patchifier: an importance map from HEVC motion-vector magnitude and residual energy (or a neural codec’s bit-allocation map) that keeps all I-frame patches and top-k P-frame patches under a fixed token budget, assembled into canvases with shared 3D rotary positions so a System-1 event gate and System-2 decoder can run on sparse rolling streams.","core_discovery":"Codec-aligned sparse tokenization—selecting dynamic, entropy-rich 16×16 patches from motion vectors and residual energy on I/P-style frames—lets a from-scratch visual encoder and a 4B dual-system VLM match dense same-scale baselines on static tasks, beat them on video and spatial reasoning, and cut wall-clock inference by up to 3.5× while supporting proactive streaming with a lightweight speak/silent gate.","pith_inferences":["If motion/residual importance is truly the right prior, the same selector could front-end non-VLM video stacks (detection, tracking, robotics) without retraining the compressor.","Confounding of data recipe and architecture suggests a clean ablate-one-factor leaderboard (codec on/off, captions on/off, gate on/off) would become the decisive community check.","Zero-Vision SFT before RL implies post-training compute may be better spent protecting text reasoning than on another round of visual instruction tuning.","Edge and always-on cameras are the natural deployment surface: 75% fewer visual tokens plus a silent-most-of-the-time gate maps directly to battery and bandwidth budgets."],"forward_implications":["Long-video and live perception systems can allocate tokens by temporal unpredictability instead of fixed frame grids and still keep accuracy.","Vision encoders for VLMs need not rely on multi-billion image-text pre-training if objectives and image+video joint training are aligned to the downstream interface.","Dense timestamped video captions plus short SFT can substitute for heavy long-VideoQA instruction tuning for zero-shot long-video QA.","A frozen backbone plus a tiny speak/silent gate is enough to add proactive streaming without a separate online architecture.","Cross-codec patch selection (traditional HEVC vs neural codecs) can stay training-free if only the importance map is consumed."],"fun_headline_variants":["Codec-native patches cut VLM tokens 75% with 3.5× speedup","Mage-VL encodes only motion-rich 16×16 regions for 3.5× faster video","Sparse I/P codec tokens match dense VLMs and win on spatial tasks","From-scratch Mage-ViT slashes visual tokens 75% via residual energy","Dual-system Mage-VL streams perception at 3.5× wall-clock speed"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"Gains against the matched-size dense baseline are treated as evidence for the codec front-end, even though the training data mix, caption pipeline, and five-stage recipe also differ.","fun_headline_variants_meta":{"raw":{"variants":["Codec-native patches cut VLM tokens 75% with 3.5× speedup","Mage-VL encodes only motion-rich 16×16 regions for 3.5× faster video","Sparse I/P codec tokens match dense VLMs and win on spatial tasks","From-scratch Mage-ViT slashes visual tokens 75% via residual energy","Dual-system Mage-VL streams perception at 3.5× wall-clock speed"]},"model":"grok-4.5","effort":"low","cost_usd":0.004841,"raw_usage":{"total_tokens":1486,"prompt_tokens":910,"num_sources_used":0,"completion_tokens":100,"cost_in_usd_ticks":48408000,"prompt_tokens_details":{"text_tokens":910,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":476,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":910,"tokens_out":100,"duration_ms":9785,"temperature":1.0,"reasoning_tokens":476,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T06:15:49.894823+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train an otherwise identical Mage-VL stack on the same five-stage data but with uniform dense frame sampling at matched token budgets; if video, spatial, and wall-clock advantages over the dense twin disappear, the codec-sparsity claim fails.","supporting_citations":[],"review_version":1}