{"id":"1108ccd4-ec8f-4022-851b-6bc90afe3d95","arxiv_id":"2501.01709","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoVE-KD distills multiple visual encoders into one student encoder using mixture-of-LoRA-experts and CLS-attention-guided distillation, improving LLaVA-family VLMs at single-encoder cost.","lead":"A method that squeezes the strengths of several vision encoders into one small encoder for vision-language models, cutting compute to that of a single encoder while keeping multi-encoder accuracy. It fine-tunes a CLIP-based student with low-rank expert modules and uses CLIP's own attention to decide which parts of an image matter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed CLIP [CLS]-attention weighting is the weakest load-bearing assumption; the TextVQA degradation under MoVE-KD is consistent with it down-weighting OCR-relevant tokens, so the claim that attention-guided KD enhances performance needs qualification.","rationale":"The reader's weakest assumption is exactly the CLS-attention weighting prior, and I agree it is the most load-bearing point. The method's contribution is explicitly the attention-guided KD regularization; if that prior is wrong for a class of tasks, the method actively suppresses useful tokens. The paper's own Table 1 provides direct evidence: TextVQA degrades on LLaVA-NeXT while most other benchmarks improve, and the paper's Sec. 4.2 explanation ('questions not related to vision') is weak because TextVQA is a visual question-answering task centered on reading text in images. A uniform-weighting ablation would isolate whether the degradation is caused by the CLS-attention prior or by multi-teacher distillation more generally. Since the reader's CONDITIONAL verdict already accounts for this uncertainty, I do not change the verdict; I also note a minor reproducibility point that Eq. 3 writes W^V where standard attention would use W^K, which should be corrected so the stated 'attention map' is actually the attention map.","tokens_in":13642,"tokens_out":6873,"duration_ms":71175,"concrete_test":"Run MoVE-KD-v1.0 (LLaVA-NeXT 7B) with the CLS-attention weighting disabled: set W_tok=0 (uniform token weight) and W_tea=1/m for non-CLIP teachers, keeping the fixed CLIP weight 0.8 and all other hyperparameters identical. Compare TextVQA, OCRBench, and the Table 1 average against the published MoVE-KD-v1.0 numbers. If TextVQA recovers toward the 64.9 baseline while the average does not collapse, the CLS-attention prior is causing the degradation and should be made task-adaptive; if TextVQA stays low under uniform weighting, the drop is due to another component such as multi-teacher KD, and the weighting concern is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is the attention-guided KD in Sec. 3.3: Eq. 3 weights individual visual tokens by the frozen CLIP [CLS] attention, and Eq. 4 weights teachers by average similarity of CLIP's [CLS] token to each teacher's tokens. This assumes CLIP attention is a reliable, task-independent saliency map for all vision-language tasks. The paper asserts this in Sec. 5.1 ('the definition of foreground and background in distillation should be fixed... human-like') but provides no benchmark-based validation of this prior. The strongest counter-evidence is in the paper's own Table 1: on LLaVA-NeXT 7B, MoVE-KD-v1.0 drops TextVQA from 64.9 to 63.7 while improving most other tasks; on 13B it drops from 67.1 to 65.8. TextVQA is an OCR/reading task where CLIP attention is known to under-emphasize text; a fixed CLS-attention weighting would systematically down-weight the tokens needed for reading. The paper's explanation that such questions are 'not related to vision' is ad hoc and contradicted by TextVQA being a visual-OCR benchmark. If this prior is wrong for a task class, the 'attention-based distillation strategy... enhances performance' claim is not universal, and the state-of-the-art framing is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MoVE-KD proposes to distill multiple pre-trained visual encoders (CLIP, EVA-02, ConvNeXt, and optionally SAM-L) into a single student visual encoder inside LLaVA and LLaVA-NeXT, preserving single-encoder inference efficiency. The student is initialized from CLIP and fine-tuned with the standard VLM loss plus a distillation loss (Eq. 2) from the teacher encoders. Per-teacher encoder adapters align the teacher feature spaces, a mixture-of-LoRA-experts (MoLE) router selects specialized student parameters per input, and the CLIP [CLS] cross-attention map is used to weight visual tokens (Eq. 3) and teacher contributions (Eq. 4) in the distillation loss. The paper reports results on VQAv2, GQA, TextVQA, VizWiz, POPE, SQA, MME, and MMB, showing improvements over LLaVA-1.5 and LLaVA-NeXT baselines on most benchmarks, with TextVQA degradations in some configurations.","tokens_in":13965,"tokens_out":8483,"duration_ms":73522,"significance":"If the empirical claims hold, the paper makes a useful practical contribution: it offers a way to obtain multi-encoder visual knowledge at near-single-encoder inference cost, with only a small parameter overhead from LoRA experts, and it provides an interesting template for multi-teacher distillation into VLMs. The paper also contains well-structured ablations and useful controls (MoLE without KD, freeze versus unfreeze), and the public code release supports reproducibility. However, the headline state-of-the-art claim is not yet established because the comparison with AM-RADIO is not data-matched, the results are single-run scores without error bars, and the central assumption that CLIP [CLS] attention is a fixed task-independent saliency map is plausible but unvalidated and is in tension with the paper's own TextVQA results.","major_comments":[{"comment":"The paper's own results contradict the unqualified claim that the attention-based distillation strategy enhances performance: on LLaVA-NeXT-7B, MoVE-KD-v1.0 lowers TextVQA from 64.9 to 63.7, and on LLaVA-NeXT-13B from 67.1 to 65.8. The explanation that TextVQA questions are 'not related to vision' is inaccurate, since TextVQA is a visual OCR benchmark; a more consistent reading is that CLIP [CLS] attention underweights text regions, so Eq. 3 down-weights the tokens needed for reading. Because this weighting is the central novelty, the authors should either qualify the claim, analyze the failure mode, or compare Eq. 3 against uniform, text-guided, or learned token weighting on TextVQA and related OCR tasks.","section":"Sec. 4.2, Table 1"},{"comment":"The comparison with AM-RADIO is not matched: RADIO is pre-trained on DataComp-1B with 1.4B image-text pairs, while MoVE-KD uses only LCS-558K, and the paper acknowledges this as 'a bit unfair' but still concludes state-of-the-art performance. Without a same-data, same-compute baseline (for example, training RADIO or a comparable single-encoder distillation baseline on the same data budget), the SOTA claim and the claim that MoVE-KD overcomes RADIO's knowledge forgetting are not established. Please also report the actual training and inference costs, including FLOPs and added parameter counts.","section":"Sec. 4.2, Table 1"},{"comment":"The teacher and token weights are derived from the frozen CLIP [CLS] attention map, and this choice is justified only by qualitative visualization and an appeal to a 'human-like' foreground/background definition. No benchmark-based validation is provided for the assumption that one fixed saliency map is reliable across tasks. The paper reports (Sec. 5.1) that text-driven foreground weighting hurt, but it does not compare against uniform token weighting or a learned weighting, so the specific contribution of Eqs. 3-4 beyond MoLE and adapters is not isolated. Since Eq. 2 multiplies these weights into the loss, an unreliable prior can actively suppress useful tokens; the TextVQA drops in Table 1 are consistent with this risk. Please add ablations with alternative token/teacher weighting schemes and a per-task reliability analysis.","section":"Sec. 3.3, Eqs. (2)-(4); Sec. 5.1"},{"comment":"All results are reported as single numbers without error bars or significance tests, and many reported differences are within 0.5-1.5 points (for example, LLaVA-1.5-7B GQA 62.0 vs 63.2, and MME 1510.7 vs 1524.5). Given that the central claims are quantitative (comprehensive improvements and state-of-the-art performance), the manuscript should report variance over multiple seeds or otherwise quantify uncertainty; otherwise the improvements may not be distinguishable from training noise.","section":"Tables 1, 2, 4, 5"},{"comment":"The teacher weight in Eq. 4 is computed as a dot product between CLIP's [CLS] token V_cls and each teacher's tokens V_t^i. Because the teacher encoders have different output dimensionalities, this dot product is only defined if V_t denotes the adapter-projected teacher tokens in the student/CLIP space; the text and Figure 3 do not state this explicitly. Please specify the exact tensors used in Eq. 4 and confirm whether the teacher weight reflects CLIP-teacher agreement after projection rather than arbitrary adapter scaling.","section":"Eq. (4), Sec. 3.3"}],"minor_comments":[{"comment":"The notation is inconsistent: W(V) in Eq. 3 is used for the key projection, not the value projection; please rename it W(K) and align with the accompanying text.","section":"Eq. (3)"},{"comment":"The constant 1/n added to W(tok) is not motivated; please explain its role or clarify whether it is absorbed by re-normalization.","section":"Eq. (2)"},{"comment":"The versions MoVE-KD-v1.0 and MoVE-KD-v1.1 are used in Table 1 but defined only later in Sec. 4.3; please define them at first use.","section":"Sec. 4.3, Table 1"},{"comment":"Several baseline rows (RADIO) have missing entries for TextVQA, VizWiz, SQA, MME, and MMB, which makes the comparison incomplete; please fill in the available numbers or state explicitly that they are unavailable.","section":"Table 1"},{"comment":"There are several typos and formatting slips (for example, 'eperts', 'V iunca', 'attetion', and inconsistent capitalization of Softmax) that should be corrected in the revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent empirical systems contribution with a clear mechanism and useful ablation controls. My main concerns are about the strength of the claims rather than the method itself: the SOTA comparison is not data-matched, there are no error bars, and the TextVQA degradation is a counter-example to the universal claim for the attention weighting. I would be willing to accept after the authors add matched comparisons, robustness/uncertainty analysis, and a qualification or fix for the CLIP-attention prior. The text-similarity weighting experiment mentioned in Sec. 5.1 is a good starting point but needs to be reported with numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuinely useful empirical contribution, but the SOTA framing oversells it. The new bit is the combination: distilling three or four visual encoders (CLIP, EVA-02, ConvNeXt, optional SAM-L) into one CLIP-initialized student inside LLaVA and LLaVA-NeXT, using Mixture-of-LoRA-Experts to avoid teacher conflict and CLIP's [CLS] attention to weight both tokens and teachers. That specific integration is not in the prior work I know.\n\nThe experiments are better than average for this area. The ablations include the right controls: MoLE without KD (Table 3), freeze vs unfreeze encoder (Table 5), and incremental additions (Table 2). The claim that MoLE adds only 0.3% parameters is credible. The authors also honestly report the TextVQA degradation, which most teams would bury.\n\nNow the soft spots. No error bars, and the AM-RADIO comparison is not matched on data—they admit RADIO trained on DataComp-1B, so the 'better than RADIO' sentence is weaker than it looks. The bigger issue is the attention-weighting assumption in Sec. 3.3. Eqs. 3 and 4 fix distillation weights to CLIP's [CLS] attention, treated as a task-independent saliency map. The paper's own Table 1 contradicts that: TextVQA drops on both LLaVA-NeXT 7B (64.9→63.7) and 13B (67.1→65.8). TextVQA is an OCR/reading benchmark; CLIP attention is known to under-weight text, so the method down-weights exactly the tokens needed for reading. The authors' explanation that such questions are 'not related to vision' is ad hoc and not credible. That is not a fatal flaw—most tasks improve—but it makes the 'attention-based strategy enhances performance' claim conditional on task class.\n\nIf I were refereeing, I'd ask for error bars, a matched-data comparison with AM-RADIO if the compute allows, and a discussion or fix for the OCR blind spot. Those are tractable. The central mechanism—multi-teacher KD with MoLE and a saliency prior—is supported by the controls. This deserves peer review and likely acceptance after revision. I'd send it to a good CV conference rather than desk reject. I'd also cite it if I work on VLM efficiency.\n\nBest.","headline":"Solid empirical integration—multi-teacher KD with MoLE works—but the CLIP-attention weighting has a real blind spot on OCR tasks, and the SOTA claim needs matched-data and error bars.","tokens_in":14510,"tokens_out":2459,"would_cite":true,"duration_ms":21958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the distinct strengths of several pre-trained visual encoders can be distilled into a single student encoder inside a vision-language model, yielding multi-encoder benefits at single-encoder inference cost, as…","keywords":["knowledge distillation","vision-language models","visual encoders","mixture-of-experts","LoRA","CLIP attention","LLaVA","multi-teacher distillation"],"falsifier":"Build a test set whose answers depend on text or objects that CLIP's [CLS] attention scores low, and train MoVE-KD on it with both CLIP-attention token weights and uniform token weights; if uniform weighting matches or beats attention weighting on that set, the claim that CLIP attention reliably identifies the valuable tokens fails.","tokens_in":13464,"feed_emoji":"🖼️","tokens_out":10643,"duration_ms":95144,"temperature":0.7,"pith_summary":"MoVE-KD sets out to prove that a vision-language model can inherit the combined strengths of several visual encoders—CLIP, EVA-02, ConvNeXt, and optionally SAM-L—without running any of them at inference time. The method distills their visual tokens into one student encoder initialized from CLIP, using per-teacher MLP adapters to align token spaces, a Mixture-of-LoRA-Experts module to keep conflicting teacher knowledge separate, and CLIP's [CLS] attention to reweight both image tokens and teachers during distillation. The paper reports consistent gains over the LLaVA and LLaVA-NeXT baselines across eight benchmarks, with the added MoLE parameters accounting for only about 0.3% of the total. If the claim holds, upgrading a VLM's perception reduces to swapping in one distilled encoder rather than adding a mixture of encoders.","feed_headline":"MoVE-KD distills several vision encoders into one LLaVA tower","feed_subtitle":"The distilled student brings CLIP, EVA-02, and ConvNeXt strengths to a single encoder at ~0.3% parameter overhead.","key_machinery":"The load-bearing device is the attention-weighted distillation loss $L_{kd} = \\sum_{i=1}^{m} W^{(tea)}_i \\sum_{j=1}^{n} (W^{(tok)}_j + \\frac{1}{n}) \\mathrm{MSE}(V^{(t)}_{i,j}, V^{(s)}_j)$, where token weights come from CLIP's [CLS]-to-patch attention and teacher weights come from the average CLIP attention to each teacher's tokens. This loss tells the student which image regions and which teachers carry valuable knowledge for a given input. The Mixture-of-LoRA-Experts (MoLE) router, which selects one of several low-rank LoRA adapters inside the student's feed-forward layers, is what keeps each teacher's specialized knowledge from being destroyed by interference during joint distillation.","core_discovery":"The central claim is that multi-encoder visual knowledge can be transferred into a single visual tower through a three-part distillation procedure. Dedicated two-layer MLP adapters project each teacher's tokens into a common space; a Mixture-of-LoRA-Experts module inside the student's feed-forward layers routes each input to specialized low-rank experts so that conflicting teacher knowledge is separated rather than averaged; and the distillation loss is reweighted by CLIP's [CLS] attention, which up-weights informative visual tokens and gives each teacher a per-image importance score. The student is initialized from CLIP, and CLIP is kept as one teacher with a fixed high weight to anchor the student's own knowledge. The paper reports gains on eight benchmarks and shows that adding a fourth teacher (SAM-L) improves results further, supporting the scalability of the design.","pith_inferences":["A natural test the paper does not run: replace the fixed CLIP [CLS] attention weights with a learned or task-conditioned weighting and measure TextVQA; if the text-reading loss disappears, the attention prior is the cause rather than the vision improvement.","The paper's conclusion that the projector becomes the bottleneck at larger LLM scales implies a testable consequence: MoVE-KD's gains should grow when the distilled student is paired with a stronger vision-language projector.","The MoLE router's expert choices can be analyzed post hoc; if experts specialize by teacher or by visual domain, the routing signal could be reused at inference to skip unused experts and save computation.","Because the weighting inherits CLIP's blind spots, tasks whose answers live in low-CLIP-attention regions (such as dense OCR) are the places where a multi-attention or ensemble-salience weighting would plausibly be more robust."],"forward_implications":["A VLM with the MoVE-KD student encoder matches or exceeds the single-encoder and multi-encoder baselines on most of the eight benchmarks, so users get multi-encoder benefits without multi-encoder inference cost.","The v1.1 variant with SAM-L as a fourth teacher improves over v1.0 on several benchmarks, indicating the approach continues to gain from additional teachers rather than saturating immediately.","Unfreezing the vision encoder alone lowers performance (Table 5), so the reported gains come from the distillation modules themselves rather than from simply letting the encoder train.","The MoLE module adds about 0.3% of total parameters, so the parameter overhead of reconciling multiple teacher knowledge is small.","CLIP needs a fixed high teacher weight (0.8), so the student's inherited CLIP knowledge must be anchored during distillation to avoid forgetting."],"supporting_citations":[{"why":"Supplies the general knowledge-distillation objective that the method extends from one teacher to a mixture of visual encoders.","marker":"[15]"},{"why":"Supplies the LLaVA-1.5 student VLM baseline, its two-stage training procedure, and the observation that linear interpolation of tokens is insufficient for aligning encoder spaces.","marker":"[27]"},{"why":"Supplies the LLaVA-NeXT high-resolution VLM baseline and its training and inference settings used in the main experiments.","marker":"[28]"},{"why":"Supplies the CLIP encoder that initializes the student, serves as one teacher, and provides the [CLS] attention used for token and teacher weighting.","marker":"[35]"},{"why":"Supplies EVA-02, one of the teacher encoders whose visual proficiencies are distilled into the student.","marker":"[10]"},{"why":"Supplies ConvNeXt, one of the teacher encoders whose visual proficiencies are distilled into the student.","marker":"[31]"},{"why":"Provides the closest multi-encoder distillation baseline; the paper compares against it and argues MoVE-KD avoids its knowledge-forgetting problem.","marker":"[36]"},{"why":"Supplies the low-rank adaptation mechanism that the mixture-of-experts module uses to build parameter-efficient experts.","marker":"[16]"},{"why":"Supplies the mixture-of-local-experts routing idea that the MoLE module adapts for selective activation of specialized knowledge.","marker":"[19]"}],"fun_headline_variants":["MoVE-KD distills multiple vision encoders into one efficient model","One tower to replace many: MoVE-KD distills vision encoders","MoVE-KD: LoRA and MoE merge visual encoders via distillation","Attention-weighted distillation: MoVE-KD blends vision encoders into one","MoVE-KD fuses CLIP, EVA-02, and ConvNeXt into one encoder"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the attention CLIP's [CLS] token pays to image patches is a correct, fixed map of what is worth learning for every downstream vision-language task, so that reweighting tokens and teachers by this map always helps rather than hurting.","fun_headline_variants_meta":{"raw":{"variants":["MoVE-KD distills multiple vision encoders into one efficient model","One tower to replace many: MoVE-KD distills vision encoders","MoVE-KD: LoRA and MoE merge visual encoders via distillation","Attention-weighted distillation: MoVE-KD blends vision encoders into one","MoVE-KD fuses CLIP, EVA-02, and ConvNeXt into one encoder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001944,"raw_usage":{"total_tokens":7605,"prompt_tokens":945,"completion_tokens":6660,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":6554}},"tokens_in":561,"tokens_out":6660,"duration_ms":45331,"temperature":1.0,"reasoning_tokens":6554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:13.397508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set whose answers depend on text or objects that CLIP's [CLS] attention scores low, and train MoVE-KD on it with both CLIP-attention token weights and uniform token weights; if uniform weighting matches or beats attention weighting on that set, the claim that CLIP attention reliably identifies the valuable tokens fails.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP encoder that initializes the student, serves as one teacher, and provides the [CLS] attention used for token and teacher weighting."},{"cited_title":"Am-radio: Agglomerative vision foundation model reduce all domains into one","cited_arxiv_id":null,"evidence_quote":"Provides the closest multi-encoder distillation baseline; the paper compares against it and argues MoVE-KD avoids its knowledge-forgetting problem."}],"review_version":1}