{"id":"bf824131-d122-4cd7-a6e9-beb7957a1d7e","arxiv_id":"2501.04322","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 1.8B-parameter vision-language model, Eve, uses elastic visual experts and type-aware token routing to reach a 68.87% average on six VLM benchmarks while preserving language performance.","lead":"A 1.8-billion-parameter vision-language model called Eve is trained in three stages with special visual experts so it can answer questions about images without losing its language skills. For small models, it reports the best average score so far on six vision-language benchmarks, beating much larger models on some tasks, which matters for putting capable assistants on phones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA attribution is untested on the final configuration: Table 6 never runs SigLIP/PanGu-Pro without EVF/Img-GBPR, so the 4.35-point Stage-3 data gain could be independent of the elastic visual experts, whose measured contribution is only 1.56 points on a weaker backbone.","rationale":"The reader's conditional verdict correctly identifies the Stage-3 dataset as the largest unconfounded-looking step in Table 6. My reading of the same table agrees: the only clean architectural measurements are on ResNet50/PanGu and total +1.56 points, while the final SOTA step is a +4.35-point data addition on an upgraded backbone/LLM. The paper does not provide the missing cells, so the abstract's causal claim ('strategically incorporating adaptable visual expertise... results in... state-of-the-art results') is not yet established. This is an addressable experimental gap, not an internal inconsistency, and it does not undermine the value of the paper as a reproducible training recipe with clear ablations. I also note that the language-preservation claim is guaranteed by construction (the language FFN is excluded at inference), so it is not evidence for the architecture's multimodal contribution. The remaining evaluation issues (no error bars, thin 0.08-point margin, benchmark-train overlap) are secondary to the missing factorial controls. Verdict remains CONDITIONAL; no adjustment needed.","tokens_in":20161,"tokens_out":7289,"duration_ms":68580,"concrete_test":"Run the 2x2 factorial on the final stack: (A) SigLIP-L + PanGu-Pro + EVF + Img-GBPR, Stage-2 data only (expect 64.52 at top of Table 6); (B) same with Stage-3 data (expect 68.87); (C) SigLIP-L + PanGu-Pro, no EVF and no Img-GBPR, Stage-2 data only; (D) SigLIP-L + PanGu-Pro, no EVF and no Img-GBPR, Stage-3 data. If D reaches roughly 68.5, the elastic-visual-expert architecture contributes at most a few tenths and the 'SOTA via elastic experts' claim is not supported. If D stays near 64.5 and B - D is near 4.3, the architecture carries the gain. Report at least two training seeds for A and B to interpret the 0.08-point margin over LLaVA-1.5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Table 5, 68.87 vs. LLaVA-1.5's 68.79) is the endpoint of the cumulative ablation in Table 6. In that table, the only rows that isolate the proposed architecture are on the ResNet50 + PanGu-1.5B base: adding EVF raises VLM-AVG from 53.36 to 53.83, and adding Img-GBPR raises it to 54.92, a combined 1.56-point effect. The next two rows swap in SigLIP-L (+8.11) and PanGu-1.5B-Pro (+1.49), and the final row adds the 3.2M-sample Stage-3 dataset (+4.35). No row runs the final SigLIP/PanGu-Pro stack without EVF/Img-GBPR, and no row runs Stage-3 data on a dense (non-EVF) control. The central claim that elastic visual experts are responsible for the SOTA result therefore depends on an untested additivity assumption: that the 1.56-point architectural gain measured at the weak-backbone point survives the +9.6-point backbone upgrade, and that the 4.35-point data gain is orthogonal to architecture. The paper's own Limitations section warns that training-data effectiveness 'may not be universally transferable,' which cuts in the same direction. Because the final margin over LLaVA-1.5 is only 0.08 points with no error bars, the attribution question is not a minor caveat: it determines whether Table 5 supports the abstract's causal claim about elastic visual experts or simply reports a data/backbone recipe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Eve is a three-stage training framework for small vision-language models. The first two stages train a lightweight adapter on top of a frozen choice of vision encoder (ResNet, ViT, or SigLIP) while the LLM is either frozen or lightly tuned with LoRA; the third stage adds an 'elastic vision FFN' (EVF) that duplicates the original FFN into language and vision branches, routes tokens via a learned router, and uses an importance-based token allocation scheme (Img-GBPR) with a capacity constraint and a load-balancing auxiliary loss. The paper reports that the best Eve variant (SigLIP-L + PanGu-pi-1.5B-Pro, 1.5B activated) reaches 68.87% average on GQA/SQA/TextVQA/POPE/MME/MMBench, surpassing DeepSeek-VL-1.6B and matching or exceeding LLaVA-1.5, while retaining language-task accuracy. The appendix details the 3.2M-sample Stage-3 dataset and additional ablations.","tokens_in":20479,"tokens_out":7467,"duration_ms":70844,"significance":"If fully supported, the paper would make a useful contribution to edge-friendly VLMs: it demonstrates a sparse dual-FFN design that adds visual capacity without retraining the language FFN, and it shows a systematic path from a small dense baseline to a competitive 1.5B-parameter model. The manuscript is unusually explicit about dataset composition and training cost (15 GPU-days, Table 5), and it ships code. The main empirical chain in Table 6 is logically ordered, and the language-preservation comparison in Tables 3 and 11 addresses a real weakness of MoE-based VLMs. However, the central attribution of the final performance to the elastic visual experts is not yet established: the architecture is only ablated on a weak backbone, the Stage-3 dataset gain is not measured on a dense control, and the final comparisons lack statistical uncertainty. With the missing control experiments, the paper is a strong engineering report but not yet a supported causal claim.","major_comments":[{"comment":"The ablation chain in Table 6 supports the paper's component ordering, but it does not support the causal attribution in the abstract. The EVF+Img-GBPR contribution is measured only on ResNet50+PanGu-pi-1.5B (53.36 to 54.92, +1.56 points). The next rows replace the backbone (SigLIP-L: +8.11, PanGu-pi-1.5B-Pro: +1.49) and then add the Stage-3 dataset (+4.35). There is no row that runs the final SigLIP/PanGu-Pro stack without EVF/Img-GBPR, and no row that runs the dense baseline on the Stage-3 dataset. The 68.87% result could therefore be produced largely by the stronger backbones plus the 3.2M-sample dataset, with the elastic experts contributing only a small or non-additive effect. Because the Stage-3 dataset contains 12K ScienceQA and 35K TextVQA training samples, two of the six evaluation benchmarks, the data effect is especially confounded. The Limitations section's warning that training-data effectiveness 'may not be universally transferable' reinforces that this missing control is load-bearing.","section":"Table 6 / Section 3.3"},{"comment":"The headline comparison depends on an unstated averaging protocol. The caption of Table 5 calls the last column a 'weighted mean of 6 VLM benchmarks', but no weights are given, and the MME column is in raw points (1466.14 for Eve), which is on a completely different scale from the other five accuracy columns. The reported 68.87 is reproduced only if MME is divided by 20 and then all six values are averaged with equal weights: (60.45+71.49+60.26+84.92+1466.14/20+62.80)/6 = 68.87. The paper should state this rescaling and the exact evaluation prompts and decoding settings; otherwise the 0.08-point margin over LLaVA-1.5 is not verifiable.","section":"Table 5 / Section 3.4"},{"comment":"The final result is an optimized score, not a pre-specified model. Table 6 shows a cumulative path that selects the components and the 3.2M-sample dataset to maximize the same six-benchmark average used in Table 5, and Table 5 reports a single run with no error bars or multiple seeds. With a 0.08-point advantage over LLaVA-1.5 and an approximately 1-point advantage over the best sub-3B baselines, the wording 'state-of-the-art' and 'outstrips' is stronger than the evidence supports. I would ask for at least three seeds or a small variance estimate on the final configuration, plus a statement of whether the final choice was made on a held-out split or on the reported test sets.","section":"Table 5 / Section 3.4"}],"minor_comments":[{"comment":"The values alpha=0.001, C=1.5, and the redistribution proportion W are fixed without any sensitivity study; a brief ablation or discussion of stability would improve reproducibility.","section":"Section 2.3 / Eq. (3)"},{"comment":"The DeepSeek-VL row is labeled 'D-1B' and reports a 45.70 language average, while the main text and Figure 1 describe DeepSeek-VL-1.6B; the model identity and benchmark subset should be aligned.","section":"Appendix A.3 / Table 11"},{"comment":"There are several typographical inconsistencies: Table 4 has 'Imapct' instead of 'Impact', Section 3.2 writes 'FNN ei' where 'FFN' is meant, and the dataset is called both 'LLaVA-mixed-665' and 'LLaVA-mixed-665k'.","section":"Throughout"},{"comment":"The acronym 'Eve' is used both for the full framework and for the 'Elastic Vision Encoder (EVE)' in the contributions list, which makes the component naming in Section 3.1 confusing; a different acronym for the encoder would help.","section":"Contributions bullet and Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: this is a competent engineering paper with a useful design and a detailed appendix, but the abstract's causal claim about elastic visual experts is not supported by the current ablation design. The missing dense control on the final stack and the unstated MME rescaling should be addressed before publication. The paper's fit to the conference is otherwise reasonable given the practical interest in efficient VLMs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on Eve. It's a credible empirical systems paper: a 1.8B VLM built from a frozen vision encoder, LoRA in stage 2, and a new vision FFN (EVF) with the language FFN frozen, plus Img-GBPR token allocation. The three-stage pipeline is clearly described, the third-stage dataset is itemized in the appendix, and the code is public. The language-preservation trick is neat: for text-only inference the EVF is simply excluded, so the original FFN weights remain untouched. That part holds up.\n\nThe ablation chain in Table 6 is logically ordered as a cumulative recipe. The biggest soft spot is exactly the one the stress-test note names: the measured architectural contribution of EVF + Img-GBPR is 1.56 points on the weak ResNet50/PanGu-1.5B base, and the final 4.35-point jump comes from adding the 3.2M-sample Stage-3 dataset. No row runs the final SigLIP/PanGu-Pro stack without EVF/Img-GBPR, and no row runs Stage-3 data on a dense control. So the abstract's causal language—that elastic visual experts deliver the SOTA—rests on an additivity assumption that is untested. The paper's own Limitations section says the training data's effectiveness may not transfer to new language models, which cuts the same way. Since the final margin over LLaVA-1.5 is 0.08 points with no error bars, this is not a minor caveat.\n\nOther soft spots are minor: MME appears to be rescaled by 1/20 to form the six-way average but that is never stated; exact evaluation prompts are not given; the MoE-LLaVA citation is inconsistent (Lin et al. in one place, Jiang et al. in another); and results are single-seed. None of these are fatal.\n\nWho is this for? Anyone building sub-3B VLMs for edge deployment. The paper is a usable recipe, and the recipe likely works even if the attribution is fuzzy. I would not take 68.87 as evidence that EVF is the cause; I would take it as evidence that a bigger vision encoder, a stronger LLM, and a large third-stage dataset give you a good small VLM. That is still useful.\n\nRecommendation: send it to review. The flaws are addressable—run the missing control rows, add error bars, clarify the averaging—and the empirical content deserves referee time.","headline":"A credible efficient-VLM recipe whose headline SOTA claim depends on an untested additivity assumption between the big third-stage dataset and the elastic experts.","tokens_in":21118,"tokens_out":2894,"would_cite":true,"duration_ms":23376,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1.8B vision-language model beats a 7B rival while keeping language skills.","keywords":["small vision-language models","efficient multimodal learning","elastic vision experts","mixture of experts","token allocation","language capability preservation","VLM benchmarks","three-stage training"],"falsifier":"Run the full 3.2M-sample Stage-3 dataset on a baseline without EVF and Img-GBPR, for example the ResNet50/PanGu-pi-1.5B setup from the ablation, and compare with the reported 68.87; if the dataset alone produces most of the gain, the architecture's role is not what the paper claims.","tokens_in":19893,"feed_emoji":"🖼️","tokens_out":7908,"duration_ms":67329,"temperature":0.7,"pith_summary":"The paper tries to show that a small vision-language model does not have to choose between language skills and multimodal ability. It proposes Eve, a 1.8B-parameter model whose training embeds elastic visual experts in all three stages, ending with a dual-FFN layer that routes image tokens to a trainable vision feed-forward network while leaving the language path frozen. On six VLM benchmarks Eve averages 68.87%, above the 1.3B DeepSeek-VL and the 7B LLaVA-1.5 model, and it keeps a higher average on eight language benchmarks than other sub-3B VLMs. The result matters because it suggests capable multimodal assistants can run on edge devices with modest training budgets.","feed_headline":"1.8B model beats 7B LLaVA-1.5 on vision-language tests","feed_subtitle":"Elastic visual experts let a 1.8B model top bigger rivals on six benchmarks while keeping language skills.","key_machinery":"The load-bearing object is the Elastic Vision Feed-Forward Network (EVF): each LLM transformer layer is split into two FFNs, one for language, kept at its Stage-2 weights, and one for vision, trained in Stage 3, plus a router that computes a softmax score over the two FFNs. The companion mechanism, Img-GBPR, adds a modality prior to the router score so image tokens are preferentially assigned to the vision FFN and text tokens to the language FFN, with overflow tokens redistributed rather than dropped. At inference for text-only tasks the vision FFN and router are bypassed, so the language model runs exactly as before. This dual-FFN routing is what lets the paper claim multimodal gains without language degradation, in contrast to the MoE layers it compares against.","core_discovery":"The central claim is that visual expertise can be added to a pretrained small language model without eroding its linguistic ability, by separating visual processing into its own trainable experts at each stage. Concretely, Eve keeps the vision encoder frozen in the first two stages while training only the adapter and a LoRA pass, then in the third stage duplicates each FFN into a frozen language FFN and a trained vision FFN, with an Img-GBPR token-allocation scheme that gives image tokens priority for the vision FFN and text tokens for the language FFN. With 1.8B parameters, this design reports 68.87% average accuracy on GQA, SQA, TextVQA, POPE, MME, and MMBench, beating the 7B LLaVA-1.5 (68.79) and the 1.3B DeepSeek-VL (67.90), while scoring a 58.56 language-task average versus DeepSeek-VL's 45.70. The authors attribute the gain to the EVF layer, Img-GBPR, the stronger SigLIP-L encoder, and the upgraded PanGu-pi-1.5B-Pro backbone, with a curated 3.2M-sample Stage-3 dataset adding 4.35 points.","pith_inferences":["The paper's strongest architectural evidence would be an ablation that runs the Stage-3 dataset on the earlier ResNet50/PanGu-pi-1.5B stack without EVF and Img-GBPR; that test is missing, so the 4.35-point dataset gain could be largely data-driven.","Because the language FFN is never trained after Stage 2, the recipe should extend to other modalities, such as audio or video tokens, by supplying a matching token-type prior instead of only image and text priors.","A resolution-matched comparison against LLaVA-1.5, which uses 336 by 336 inputs while Eve uses 384 by 384, would clarify how much of the win comes from higher input resolution rather than the expert architecture.","Inference cost is not fully reported: the paper gives activated parameters but not FLOPs at 384 resolution, so the efficiency claim should be read as training-efficiency and parameter-efficiency rather than necessarily fastest run time."],"forward_implications":["Sub-3B VLMs can match or exceed 7B models on standard multimodal benchmarks, which would make edge-device deployment more practical.","Language ability can be preserved during multimodal training without massive text-data mixing, cutting training from 896 to 15 GPU-days in the DeepSeek-VL comparison.","Swapping in a better frozen vision encoder should improve multimodal accuracy without retraining the language model, because the encoder is never tuned.","If the Stage-3 dataset recipe transfers, the same corpus plus the EVF/Img-GBPR recipe could be applied to other small language models.","The benchmark result depends on the exact combination of SigLIP-L, PanGu-pi-1.5B-Pro, and Stage-3 data, so changing any piece would require re-validation."],"supporting_citations":[{"why":"Supplies the MobileVLM baseline and the stage-1/stage-2 training data recipe that Eve starts from.","marker":"(Chu et al. 2023)"},{"why":"Introduces the MoE layers and auxiliary loss whose architecture EVF modifies and outperforms in the ablation.","marker":"(Lin et al. 2024)"},{"why":"Provides the PanGu-pi-1.5B-Pro backbone and the language benchmark suite used for evaluation.","marker":"(Tang et al. 2024)"},{"why":"Contributes ShareGPT4V, the main Stage-3 general multimodal dataset with 665K samples.","marker":"(Chen et al. 2023)"},{"why":"DeepSeek-VL is the main sub-3B baseline outperformed, and its 896 GPU-days is the training-cost comparison.","marker":"(Lu et al. 2024)"},{"why":"LLaVA-1.5 supplies the 7B performance level Eve is compared against as well as the LLaVA-558k/665k data.","marker":"(Liu et al. 2023a)"},{"why":"LoRA is the Stage-2 tuning method used to adapt the adapter and language model without large language degradation.","marker":"(Hu et al. 2021)"},{"why":"SigLIP-L is the vision encoder whose replacement in the ablation gives the largest single accuracy jump.","marker":"(Zhai et al. 2023)"}],"fun_headline_variants":["1.8B model with elastic visual experts beats 7B LLaVA-1.5","Elastic visual experts let 1.8B model top 7B LLaVA-1.5","Eve: 1.8B VLM with elastic experts beats 7B LLaVA-1.5","1.8B Eve keeps language skills while beating 7B LLaVA-1.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes the Stage-3 dataset's 4.35-point gain is additive with the EVF and Img-GBPR components, since the paper never tests the dataset without those components and the dataset alone could explain most of the improvement.","fun_headline_variants_meta":{"raw":{"variants":["1.8B model with elastic visual experts beats 7B LLaVA-1.5","Elastic visual experts let 1.8B model top 7B LLaVA-1.5","Eve: 1.8B VLM with elastic experts beats 7B LLaVA-1.5","1.8B Eve keeps language skills while beating 7B LLaVA-1.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001055,"raw_usage":{"total_tokens":4461,"prompt_tokens":1009,"completion_tokens":3452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":3343}},"tokens_in":625,"tokens_out":3452,"duration_ms":23712,"temperature":1.0,"reasoning_tokens":3343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:14.174035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full 3.2M-sample Stage-3 dataset on a baseline without EVF and Img-GBPR, for example the ResNet50/PanGu-pi-1.5B setup from the ablation, and compare with the reported 68.87; if the dataset alone produces most of the gain, the architecture's role is not what the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SigLIP-L is the vision encoder whose replacement in the ablation gives the largest single accuracy jump."}],"review_version":1}