{"id":"689a9657-8e0c-4535-bc7c-2021e921fe07","arxiv_id":"2501.01426","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MERV fuses four frozen specialized visual encoders with spatio-temporal alignment and cross-attention, improving video QA accuracy over single-encoder VideoLLMs by up to 4.6 percentage points.","lead":"Researchers built MERV, a video-language model that combines four different vision systems into one unified representation instead of relying on a single one. It answers video questions more accurately than single-encoder models on several standard benchmarks, with modest extra computing cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hold-out gains on Perception Test are fragile because the largest single-encoder (ViViT) gains on exactly the temporal subset where MERV's advantage is credited, and MERV's reported gain over SeViLA collapses to 1.2% when MERV (frozen) is used; the central fusion benefit needs an error-bar and…","rationale":"The reader's weakest assumption was about ChatGPT-scoring noise on open-ended QA. I agree that this is a real issue, but I locate the most load-bearing version of it in the held-out Perception Test comparisons, where the headline claims (surpassing SeViLA, 4.1% gain over Video-LLaVA) and the fusion-architecture choice (cross-attention vs channel-concat) all rest on single-run differences that are smaller than observed projector/token-count variations in the appendix. The paper is otherwise well-executed: same-data-mix controls, held-out evaluation, ablation completeness, efficiency analysis, and qualitative skill decomposition are all genuine strengths. The central qualitative claim (different encoders contribute different skills, and MERV captures both axes) is well supported by the SSv2 temporal vs general analysis, which is internally consistent and not merely consensus-disagreeing. The lack of error bars and seed variation is the load-bearing weakness. I recommend UNCHANGED rather than CONDITIONAL because the reader already flagged the evaluation-protocol issue and the paper's fixable reporting gap is unlikely to overturn the overall direction of the result; however, if the proposed test shows fusion-strategy differences are noise, the architectural contribution would need to be restated. My concern is therefore with the confidence of the numerical claims, not with the validity of the multi-encoder hypothesis.","tokens_in":32343,"tokens_out":1852,"duration_ms":17166,"concrete_test":"Re-run the MERV (frozen) and MERV (full) evaluations on Perception Test, NExT-QA, VLEP, and TVQA with 3 training seeds and 3 ChatGPT scoring calls per answer (or a fixed-seed local judge like an open LLM with temperature 0), and report mean ± std, plus a paired bootstrap over scored answers. Additionally, re-run cross-attention versus channel-concatenation fusion with 3 seeds on Perception Test to see if channel-concat's 47.36 vs 46.21 advantage persists; if the fusion-strategy gap is within seed noise, the paper's architectural claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fusing four specialized encoders delivers a robust, generalizable improvement over single-encoder VideoLLMs. The most load-bearing evidence is the held-out evaluations on NExT-QA, VLEP, TVQA, and Perception Test. On TVQA, MERV (frozen) beats Video-LLaVA by +4.62 (37.66 to 42.28), and on Perception Test MERV (full) beats SeViLA by 2.2 (46.2 to 48.4). But the latter comparison is not apples-to-apples: MERV (frozen) gets 46.21 on Perception Test, only 0.01 above SeViLA's 46.2, and the 48.4 comes from MERV (full), which uses a different training recipe (Stage 1 with LLM unfrozen). The paper trains only a single run per configuration and reports no variance, no repeated evaluation, and no seed sensitivity; for open-ended benchmarks, the ChatGPT-based scoring itself (one GPT-3.5-turbo-0613 call per answer) is stochastic. The paper's own appendix (Table 6c) shows that on Perception Test, channel-wise concatenation (47.36) outperforms the cross-attention fusion default (46.21), and the difference between 64-token and 100-token projections in Table 6b is 0.78 points (46.34 vs 45.56), larger than several headline gaps. With single runs and no bootstrap or repeated-measure analysis, the reported advantages of the specific fusion module over simpler alternatives are within plausible measurement noise. The core idea of multi-encoder fusion is supported by the ablation removing one encoder (all degrade), but the magnitude and the best-fusion choice are not statistically secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MERV, a VideoLLM that fuses four frozen visual encoders (DINOv2, ViViT, SigLIP, LanguageBind) through spatio-temporal alignment, per-encoder linear projection, and a cross-attention mixer, before feeding the unified visual representation into LLaMA-2 7B. The authors fix the training data to match Video-LLaVA, introduce two training recipes (MERV frozen, a Stage-2-only recipe, and MERV full, which unfreezes the LLM during Stage 1), and evaluate on open-ended and multiple-choice video QA benchmarks, including three datasets held out during development. The main claims are that MERV outperforms Video-LLaVA by up to 4.62 points on TVQA, improves over SeViLA on zero-shot Perception Test by 2.2 points in the full recipe, and that the multi-encoder representation captures complementary specializations (spatial, temporal, and contrastive) with modest computational overhead. The appendix reports ablations of projectors, token counts, fusion strategies, training recipes, and per-encoder contributions, plus a Something-Something v2 analysis.","tokens_in":32884,"tokens_out":5885,"duration_ms":59944,"significance":"If the reported gains are statistically robust, MERV is a practically useful and conceptually clean contribution: it shows that combining existing frozen encoders can improve VideoLLMs without retraining the encoders, and it does this with a careful controlled comparison using Video-LLaVA's data mix, released code and weights, and a set of held-out benchmarks. The qualitative and quantitative analysis of encoder specializations (especially the temporal subset of Something-Something v2) is a strength, as is the systematic ablation of projectors, token budgets, and fusion strategies. The paper's main weakness is that the central numerical claims rest on single training runs and a single ChatGPT-based evaluation per answer, with no confidence intervals or repeated evaluations; several ablation differences are of the same magnitude as the headline gains. The significance of the method is therefore conditional on additional uncertainty quantification, but the core idea and the controlled experimental design are sound enough to warrant revision rather than rejection.","major_comments":[{"comment":"The central comparison is reported without any measure of uncertainty. Each configuration appears to be a single training run, and open-ended answers are scored by a single ChatGPT call per answer (Section 4, 'Results using GPT-3.5-turbo for evaluation'), yet several load-bearing gaps are small: the Perception Test gain of MERV (frozen) over Video-LLaVA is 1.99 points, and the Perception Test advantage over SeViLA is 0.01 points for MERV (frozen) and 2.2 points only for MERV (full). The ablations show differences of comparable size, e.g. 0.78 points on Perception Test between 64 and 100 tokens (Appendix Table 6b) and channel concatenation beating the selected cross-attention fusion on Perception Test by 1.15 points (Appendix Table 6c). With single runs and no repeated ChatGPT scoring, the reader cannot determine whether the multi-encoder gains or the specific fusion choice are larger than run-to-run and evaluation noise. I ask for repeated evaluations (multiple seeds or at least repeated ChatGPT scoring with reported means and standard deviations, plus bootstrap confidence intervals on accuracy) for at least the main comparisons and the fusion ablation.","section":"Section 4.1, Table 1; Appendix Tables 6b and 6c"},{"comment":"The claim of outperforming SeViLA by 2.2% on zero-shot Perception Test is not apples-to-apples. MERV (full) uses a different training recipe (Stage 1 with the LLM unfrozen) than SeViLA, and Perception Test is one of the five benchmarks explicitly described as used during development, not one of the three held-out benchmarks. With the default MERV (frozen) recipe, Perception Test accuracy is 46.21, essentially tied with SeViLA's 46.2, and on the other SeViLA-comparable benchmarks MERV (frozen) is below SeViLA on VLEP (58.66 vs 64.4) and NExT-QA (63.09 vs 63.6). The 'improve upon SeViLA by 2.2%' statement should be qualified to the full-recipe variant and to the development benchmark, and the generalization claim should rest primarily on the held-out results.","section":"Section 4.1, Table 1; Section 4.2.3; Section 4 (Dataset paragraph)"},{"comment":"The selection of cross-attention as the final fusion module is not strongly supported by the reported numbers. Table 2c gives an average accuracy of 56.83 for cross-attention versus 56.64 for channel-wise concatenation, a 0.19-point difference over four development benchmarks, while Appendix Table 6c shows channel concatenation is better on Perception Test by 1.15 points (47.36 vs 46.21). The paper justifies the choice partly by the interpretability of attention weights, but the claimed performance advantage of the specific fusion mechanism over a simpler concatenation baseline appears to be within the unexplained single-run noise identified above. Please either provide uncertainty estimates that separate these configurations or soften the claim that cross-attention is the best-performing fusion strategy on accuracy.","section":"Table 2c and Appendix Table 6c; Section 4.2.2"},{"comment":"The headline held-out comparisons in Table 1 pair MERV (frozen), a Stage-2-only recipe, against Video-LLaVA, which uses the two-stage recipe. The same-recipe control in Appendix Table 5 partly addresses this concern, but only for the five development benchmarks; the three held-out benchmarks (NExT-QA, VLEP, TVQA) are not reported for the exact Video-LLaVA recipe, so the main held-out gains of +3.48 on NExT-QA and +4.62 on TVQA are not measured under a matched recipe. Reporting the held-out numbers for the same-recipe variant would strengthen the central claim that the multi-encoder representation, rather than the simplified training recipe, drives the improvement.","section":"Section 4, Dataset paragraph; Table 1; Appendix Table 5"}],"minor_comments":[{"comment":"The appendix says 'We plan to release our full code including training and evaluation as well as all model checkpoints for the camera-ready version,' while the footnote on page 2 states the code and pretrained weights are already available; please update the appendix to match the actual release status.","section":"Appendix A.2.2"},{"comment":"The captions and sample text contain the typo 'iteracted' instead of 'interacted' in several places; please correct it.","section":"Figures 10 and 13"},{"comment":"The inconsistent rendering of model names such as 'Video-LLaV A' and 'LLaMA-Vid' (with and without hyphens and spaces) should be unified to match the canonical names used by the cited papers.","section":"Table 1 and throughout"},{"comment":"The text says Stage 1 leaves 'approximately 1.3M vision-text pairs' unused; the cited Stage 1 data totals 558k image-text pairs plus 702k video-text pairs, which is roughly 1.26M, so the approximation should be stated consistently.","section":"Section 4.2.3"},{"comment":"The note that Video-ChatGPT's and Video-LLaVA's author-reported TGIF numbers are on incomparable subsets is useful and should be moved or summarized in the main text near Table 1, since Table 1 reports TGIF numbers without this caveat.","section":"Appendix A.3, Note on TGIF-QA"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is credible; the main barrier is statistical substantiation rather than novelty or internal consistency. If the authors can provide repeated evaluations or confidence intervals for the main comparisons and the fusion ablation, I would expect the central claims to hold. I have no concerns about citation practices or novelty disclosure; the related-work discussion of Eyes Wide Shut, Cambrian-1, and BRAVE is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful empirical paper that deserves a proper refereeing. MERV extends the multi-encoder fusion recipe from image LLMs to video, adding a temporal expert (ViViT) and spatio-temporal alignment, and shows consistent gains over single-encoder VideoLLMs on standard benchmarks. The ablations are thorough—projectors, token counts, fusion strategies, training recipes, encoder removal—and the held-out evaluation on NExT-QA, VLEP, TVQA is a real plus. The efficiency story (parallel encoders, similar step time) is credible and practically relevant.\n\nThe central claim that combining frozen specialized encoders helps holds up under the evidence. The same-recipe control in the appendix (Table 5) shows MERV with the Video-LLaVA recipe still beats Video-LLaVA, which addresses the main-table confound. And the encoder-removal ablation shows each of the four contributes something.\n\nThe soft spots are real but not fatal. The evaluation is dominated by single runs and ChatGPT-based scoring with no error bars, repeated evaluations, or seed sensitivity. On Perception Test, the headline gain over SeViLA relies on MERV (full); the frozen model is essentially tied with SeViLA. Within the ablations, channel-wise concatenation beats cross-attention on Perception Test (47.36 vs 46.21), and a 64 vs 100 token projection swings 0.78 points—both larger than some of the headline gains. So the specific fusion choice and the magnitude of advantages are not statistically secured. Also, the paper says in one place that code and weights are available, and in the appendix that they plan to release them; that needs fixing.\n\nNone of this breaks the paper. The recipe and the qualitative analysis are worth having. I'd send it to peer review and ask for variance estimates, repeated ChatGPT evaluations (or a human-agreement check), and a cleaner comparison that explicitly separates frozen/full and states which recipe each prior model used. The authors should also align the code-release statements.\n\nWho is this for: anyone building VideoLLMs or thinking about multi-encoder fusion. It won't reorganize the field, but it's a practical step forward. A serious referee can help make the claims match the evidence.","headline":"A solid, well-ablated empirical recipe for multi-encoder video LLMs whose exact advantage margins need error bars before the headline numbers are trusted.","tokens_in":33329,"tokens_out":2869,"would_cite":true,"duration_ms":27940,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MERV, a video-language model that fuses four frozen visual encoders, outperforms single-encoder models on standard video QA benchmarks.","keywords":["MERV","video language models","multi-encoder fusion","spatio-temporal alignment","cross-attention feature mixing","frozen visual encoders","video question answering","Something-Something v2"],"falsifier":"Re-run MERV and Video-LLaVA on Perception Test, TVQA, and the open-ended QA sets several times with the same GPT-3.5-turbo-0613 scorer and compute the variance across runs. If the 1.99-point Perception Test gap, and similarly sized gaps elsewhere, falls inside the run-to-run noise, the paper's advantage over its base model is not established.","tokens_in":32141,"feed_emoji":"🎥","tokens_out":7182,"duration_ms":63825,"temperature":0.7,"pith_summary":"This paper argues that video-language models unnecessarily limit themselves by letting a single vision encoder supply all visual information, and that combining several specialized encoders into one representation is a cheap way to broaden what the language model can see. The proposed model, MERV (Multi-Encoder Representation of Videos), freezes four encoders — DINOv2 for spatial detail, ViViT for temporal structure, SigLIP for image–text associations, and LanguageBind for video–text semantics — and aligns, projects, then cross-attention-mixes their features before feeding them to a LLaMA-2 7B backbone. On standard video QA benchmarks, MERV outperforms the single-encoder Video-LLaVA model it was adapted from, with gains up to 4.62 accuracy points on TVQA, and the fully trained version reaches 48.4% zero-shot accuracy on the Perception Test, ahead of SeViLA's 46.2%. The added parameters are minimal, and parallel visual processing keeps the per-step training time close to that of a single-encoder model. The paper's intended lesson is that scaling video understanding should include widening the visual encoder set, not just enlarging the language model.","feed_headline":"Four visual encoders beat one in video question answering","feed_subtitle":"MERV fuses four frozen specialized encoders and gains up to 4.62 points over Video-LLaVA on standard benchmarks.","key_machinery":"The load-bearing mechanism is the spatio-temporal pre-fusion projection followed by a cross-attention mixer. A 2D adaptive average pool resizes every encoder's spatial feature map to one common grid, and a per-encoder linear layer maps all feature dimensions to the LLM's dimension; the temporal dimension is matched by choosing input frame counts so each encoder outputs the same number of time steps. The aligned tokens are then averaged over the sequence dimension and used as keys and values in a cross-attention layer with a single learnable query $Q\\in\\mathbb{R}^{1\\times d}$, so the attention weights $O=\\mathrm{Softmax}(QX^\\top/\\sqrt{d})X$ define a linear, input-dependent mixture of the four encoders' representations. This design keeps trainable overhead small and lets the LLM receive one compact token set per frame.","core_discovery":"The central claim is that heterogeneous frozen visual encoders can be unified into a single additive video representation that inherits each encoder's specialty without sacrificing the others. MERV aligns the four encoders spatio-temporally: it picks input frame counts so temporal outputs match, adaptively average-pools the spatial grids to a common size, and linearly projects all channels into the LLM embedding dimension. A single learnable query then cross-attends over the averaged encoder tokens, so the attention weights form an input-dependent linear mixture of the four representations. The result, MERV, matches or beats every one of its single-encoder counterparts across open-ended and multiple-choice video QA benchmarks, including held-out sets, and on Something-Something v2 it captures both the temporal sensitivity of ViViT and the scene understanding of the contrastive encoders. Removing any of the four encoders degrades performance, which the paper takes as evidence that the fusion itself, not one dominant expert, carries the gains.","pith_inferences":["A consequence the authors leave implicit is that the cross-attention weights could serve as a cheap routing signal at inference time, letting a deployment skip encoders whose specialties are clearly irrelevant for a given video.","A testable extension is that the benefit of adding encoders should follow the diversity of their training objectives, not their number: adding a second video-language contrastive model added little, while the four-way mix of distinct paradigms gave the gains.","Another observable prediction is that models built this way should show their largest advantage on benchmarks that mix temporal and static reasoning in one dataset, because single encoders trade one off against the other.","Because the reported evaluation is a single GPT-scored pass, a natural check is whether these accuracy gaps persist under repeated scoring runs or under human agreement scoring."],"forward_implications":["If the central claim holds, any LLaVA-style video model can be upgraded by adding frozen specialized encoders behind a spatio-temporally aligned projector, without redesigning the LLM or its training data.","Performance on benchmarks that emphasize temporal reasoning, such as Perception Test and TVQA, should keep improving as more complementary frozen encoders are added, provided the parallelization overhead stays small.","Because encoders remain frozen, new visual backbones can be swapped into MERV without retraining the language side from scratch, separating visual representation research from language-model training.","The Something-Something v2 analysis implies that video benchmarks should measure temporally reversed actions separately from general scene understanding, since single encoders differ sharply on those two axes."],"supporting_citations":[{"why":"Supplies the single-encoder Video-LLaVA baseline, the LLaVA-style architecture, and the identical training data mix used for fair comparison.","marker":"Lin et al., 2024"},{"why":"Provides DINOv2, the frozen spatial expert that contributes fine-grained object and part understanding.","marker":"Oquab et al., 2023"},{"why":"Provides ViViT, the frozen temporal expert that contributes motion and temporal-structure features.","marker":"Arnab et al., 2021"},{"why":"Provides SigLIP, the frozen image-language contrastive expert used for vision-language associations.","marker":"Zhai et al., 2023"},{"why":"Provides LanguageBind, the frozen video-language contrastive expert and the encoder used by the video segment of the training mix.","marker":"Zhu et al., 2024a"},{"why":"Supplies SeViLA, the previous best zero-shot Perception Test baseline that MERV (full) surpasses.","marker":"Yu et al., 2024b"},{"why":"Supplies the frozen single-stage training recipe and codebase that MERV builds on for efficiency.","marker":"Karamcheti et al., 2024"},{"why":"Provides the Perception Test diagnostic benchmark where MERV (full) reaches 48.4% zero-shot accuracy.","marker":"Pătrăucean et al., 2023"}],"fun_headline_variants":["Four visual encoders beat one in video QA","MERV fuses four experts to boost video understanding","VideoLLMs gain up to 4.6 points with multi-encoder fusion","More encoders, better video reasoning: MERV wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ChatGPT-based scoring protocol used for open-ended answers is stable enough that accuracy differences of roughly two to five points, reported from a single evaluation pass, reflect real model ability rather than evaluator noise.","fun_headline_variants_meta":{"raw":{"variants":["Four visual encoders beat one in video QA","MERV fuses four experts to boost video understanding","VideoLLMs gain up to 4.6 points with multi-encoder fusion","More encoders, better video reasoning: MERV wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1184,"prompt_tokens":980,"completion_tokens":204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":135}},"tokens_in":596,"tokens_out":204,"duration_ms":2831,"temperature":1.0,"reasoning_tokens":135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:27:35.921113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MERV and Video-LLaVA on Perception Test, TVQA, and the open-ended QA sets several times with the same GPT-3.5-turbo-0613 scorer and compute the variance across runs. If the 1.99-point Perception Test gap, and similarly sized gaps elsewhere, falls inside the run-to-run noise, the paper's advantage over its base model is not established.","supporting_citations":[],"review_version":1}