{"id":"291312e1-410c-4ce3-b01f-b5e829889c08","arxiv_id":"2501.05901","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Valley2, a 7B-scale open-source multimodal model, reports second-best OpenCompass average (67.4) among sub-10B models and the highest score (79.66) on its own in-house Ecom-VQA benchmark.","lead":"Valley2 is an open-source vision-language model that combines a Qwen2.5 language backbone with several efficiency-focused visual components, and reports competitive benchmark scores including second place on OpenCompass among sub-10B models. It also introduces an in-house e-commerce benchmark and claims state-of-the-art results on it, though that benchmark is not publicly released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ecom-VQA SOTA rests on an unreleased 536-item benchmark used for model selection, with no leakage analysis; the OpenCompass ranking is externally checkable.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the unreleased, small in-house Ecom-VQA benchmark is the sole basis for the headline SOTA claim, and it is used both as a training-domain source and as a model-selection signal without a leakage analysis. The paper's external OpenCompass claim is independently checkable because the weights are open-sourced, and the architectural components are described with ablations, so the overall structure of the paper remains credible. However, the central e-commerce result should not be accepted as stated without releasing the benchmark and demonstrating that the 79.66 number survives a held-out evaluation. Since the reader already returned CONDITIONAL and this concern reinforces that condition rather than changing the verdict category, the appropriate output is UNCHANGED: the paper should be accepted only under the condition that the benchmark and leakage audit are provided.","tokens_in":15744,"tokens_out":4450,"duration_ms":45155,"concrete_test":"Ask the authors to release Ecom-VQA with full provenance and run a leakage audit: for each of the 536 test items, detect exact or near-duplicate images (e.g., pHash or embedding similarity) and n-gram/question overlap against the in-house training corpus described in Table 1; any nonzero overlap invalidates the comparison. Then, with the final frozen checkpoint, evaluate on a freshly annotated held-out set of 536 e-commerce QA pairs drawn from the same distribution but never used for model selection. If held-out accuracy is not above Qwen2-VL-7B by more than the bootstrap 95% CI width, the abstract's large-margin SOTA should be downgraded to a within-noise result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is the e-commerce SOTA: Valley2 (79.66) vs. Qwen2-VL-7B (72.76) on Ecom-VQA. That claim rests on an in-house benchmark of only 536 QA pairs. Three concrete weaknesses make it fragile. (1) Train/eval separation is not demonstrated: Section 3.2 states the proposed benchmarks are used to track and assess the development of capabilities, while Table 1 lists in-house Ecom data inside every training category (General, Reasoning, OCR, OneVision). If the Ecom-VQA questions or images come from the same annotation pool as the in-house training data, the comparison is contaminated. No overlap or leakage analysis is reported. (2) The same Ecom-VQA numbers are used as the model-selection signal: Table 5 chooses ConvAdapter vs. PixelShuffle, MLP hidden size, Eagle Module, and annealing by E-VQA/OC score on this 536-item set. Selecting hyperparameters on a tiny benchmark and then reporting the selected score overstates expected performance on new data. (3) No error bars or seeded repeats are given. For 536 binary pass/fail questions, the standard error of the headline gap is roughly 2.6 points, and the reported margin is 6.9 points, so the comparison is not robust to modest contamination or selection effects. The OpenCompass rank-second claim is not the problem: it uses an external leaderboard and released weights and can be rechecked independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Valley2, a 7B-parameter multimodal large language model built on Qwen2.5 and SigLIP, with a large-MLP projector combined with a ConvAdapter for token compression, an optional Eagle Module for high-resolution inputs, and a chain-of-thought (CoT) post-training stage. The authors claim state-of-the-art performance on an in-house e-commerce benchmark (Ecom-VQA 79.66 vs. Qwen2-VL-7B 72.76), second place among sub-10B models on the OpenCompass leaderboard (average 67.4), and an average training sequence length under 1,000 tokens. The paper also describes data curation for e-commerce and short-video domains, a packing strategy for training efficiency, and ablations for each architectural component. Code and model weights are released.","tokens_in":16005,"tokens_out":4659,"duration_ms":41209,"significance":"If the claims hold, Valley2 is a competitive and efficient design for practical multimodal applications, and the open-source release allows independent verification of the OpenCompass ranking, which is a concrete strength. The architectural contributions (ConvAdapter, large MLP, Eagle Module) are clearly described and ablated. However, the headline e-commerce SOTA rests on a small, unreleased 536-item benchmark that is also used for model selection, so the e-commerce claim is not currently established to the standard expected for a journal publication. The external OpenCompass result is the more defensible claim and is readily checkable.","major_comments":[{"comment":"The headline Ecom-VQA result is based on only 536 QA pairs, and the benchmark is not publicly released. For 536 independent binary items, the standard error of the difference between two proportions near 0.75 is roughly 2.6 percentage points, so the reported margin of 6.9 points (79.66 vs. 72.76) is not robust to modest contamination or selection effects. Please report confidence intervals (or bootstrap replicates) and provide the benchmark, or at minimum a detailed statistical analysis, to support the claimed margin.","section":"Section 3.2 and Table 4"},{"comment":"The ablation study selects hyperparameters (MLP hidden size, ConvAdapter vs. PixelShuffle, Eagle Module, and annealing) using the same Ecom-VQA and OpenCompass benchmarks that are later reported as headline results. This makes the reported Ecom-VQA score an optimistically biased estimate of performance on new data. The authors should either use a held-out validation set for model selection or report selection-adjusted estimates, such as nested cross-validation or a clear statement of how many configurations were tried.","section":"Section 6.1 and Table 5"},{"comment":"The manuscript states that the proposed benchmarks are used to track and assess the development of capabilities, while Table 1 lists Inhouse Ecom-Data in every training category, including General, Reasoning, OCR, and OneVision tasks. The paper does not demonstrate that Ecom-VQA questions and images are disjoint from the in-house training data, so the comparison against Qwen2-VL-7B could be contaminated by training on the same distribution or even the same instances. Please provide a detailed overlap/leakage analysis and clarify the provenance of the benchmark items relative to the training corpus.","section":"Section 3.2 and Table 1"}],"minor_comments":[{"comment":"The average score for Valley2 is listed as 67.40, but the table column header includes an unused 'A VG' column; please clarify how the average is computed and whether it matches the OpenCompass aggregate.","section":"Section 5.1 and Table 3"},{"comment":"The CoT ablation row reports an OpenCompass improvement but leaves the E-VQA cell blank; please state explicitly whether CoT was evaluated on Ecom-VQA and, if not, why.","section":"Section 6.4 and Table 8"},{"comment":"The text says the intermediate MLP dimension (Dh) is set to '1/5 of the LLM vocabulary size', but Table 5 reports a default MLP hidden size of 3584, which equals the Qwen2.5-7B hidden dimension rather than one-fifth of its vocabulary size. This appears to be a typo and should be corrected.","section":"Section 6.1"},{"comment":"The packing ablation reports a 2.2x training speedup, while the text in Section 6.3 says training time is reduced 'by a factor of approximately 2–3'; please reconcile these numbers for consistency.","section":"Section 4.2 and 6.3"},{"comment":"Several references contain 'arXiv preprint arXiv:unknown' (e.g., [24], [46], [73], [87]); these entries are incomplete and should be updated with the correct identifiers or venues.","section":"References"},{"comment":"The 'Coming Soon' section listing future work is unusual for a journal article; consider removing it or moving it to a separate document to keep the paper focused.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The OpenCompass ranking is externally verifiable and the open-source release is a clear strength. The e-commerce SOTA, however, rests on an unreleased 536-item benchmark that is also used for hyperparameter selection, with no leakage analysis. For a journal publication, the authors should either release the benchmark with an overlap analysis or substantially temper the e-commerce claim. The numerical inconsistency in the MLP hidden size description should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the OpenCompass result is the trustworthy part of this paper; the e-commerce SOTA is not. Valley2's rank-second among sub-10B models is externally checkable, the weights are released, and the ConvAdapter is a simple, sensible efficiency tweak that I can believe helps. The ablations are systematic and the writing is mostly clear. That is real value.\n\nThe soft spot is exactly where the stress-test puts it: Ecom-VQA has 536 questions, is not released, no leakage analysis is reported, and the same benchmark is used to select the MLP hidden size, ConvAdapter, Eagle Module, and annealing schedule. You cannot report a 79.66 SOTA on a set you tuned on, especially with no error bars and a cyclic testing protocol that multiplies the effective sample size without adding information. The standard error on 536 binary items is around 2 points, so the 6.9-point gap over Qwen2-VL-7B is not robust to modest contamination or selection effects. The paper itself says the proposed benchmarks are used to track development while also listing in-house Ecom data inside every training category, which raises a real overlap risk. This is not a fatal flaw in the model; it is a fatal flaw in the headline claim as written.\n\nThe OpenCompass average of 67.4 is plausible and independently verifiable. The efficiency claim about under 1,000 average training tokens is interesting but lacks a systematic comparison against alternatives, so treat it as anecdotal. The references have some sloppiness (e.g., the LRV entry mispointing to LLaVA-OneVision, and several 'arXiv:unknown' placeholders), but nothing that changes the technical content. The 'Coming Soon' section is filler and should be cut.\n\nWho is this for? People building open-source MLLMs for e-commerce or short-video applications will want the weights and the domain recipes. The paper deserves a serious referee, but the e-commerce benchmark has to be released, with error bars, a leakage analysis, and either a held-out split or a corrected reporting protocol that separates model selection from final evaluation. Without that, the central comparison is not defensible.","headline":"OpenCompass claim is solid and the model release is useful, but the e-commerce SOTA is built on a tiny unreleased benchmark used for model selection.","tokens_in":16610,"tokens_out":1643,"would_cite":true,"duration_ms":17876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Valley2 claims a 7B model beats Qwen2-VL-7B by 6.9 points on e-commerce VQA and ranks second on OpenCompass among sub-10B models.","keywords":["multimodal large language model","vision-language model","e-commerce benchmark","ConvAdapter","Eagle Module","chain-of-thought post-training","OpenCompass","token compression"],"falsifier":"Releasing Ecom-VQA and having an independent lab re-run Valley2 alongside Qwen2-VL-7B and InternVL2-8B under identical settings would settle it; if the 79.66 versus 72.76 margin shrinks to near parity or the benchmark leaks training data, the claim loses its basis.","tokens_in":15513,"feed_emoji":"🛒","tokens_out":3471,"duration_ms":29791,"temperature":0.7,"pith_summary":"This paper introduces Valley2, a 7B-parameter multimodal large language model that accepts text, images, and videos. The authors claim it achieves state-of-the-art accuracy on e-commerce question-answering benchmarks, 79.66 versus 72.76 for Qwen2-VL-7B, and ranks second among models under 10B parameters on the OpenCompass leaderboard with an average score of 67.4. The purpose of the work is to show that careful token compression and a second vision encoder can let a small open model match or exceed much larger systems on practical, real-world tasks. The authors also open-source code and weights so others can verify these results.","feed_headline":"7B model tops e-commerce VQA benchmark","feed_subtitle":"Valley2 also ranks second on OpenCompass among models under 10B parameters.","key_machinery":"The load-bearing mechanism is the token-budget compromise: a ConvAdapter that halves the spatial grid of vision tokens while preserving channel dimension, a large hidden-size MLP that benefits from that compact representation, and the Eagle Module that adds a second vision encoder's tokens only when needed. This trio keeps the average training sequence length under 1,000 tokens while allowing single-image inference to scale to 1,000-2,000 tokens, matching models that use 4,000-8,000 tokens per image.","core_discovery":"Valley2's central claim is that a 7B model can reach competitive general multimodal performance and best-in-class e-commerce understanding without paying the token cost of high-resolution tiling on every input. The design pairs a large two-layer MLP projector with a ConvAdapter that uses a 2x2 convolution to fold four spatial tokens into one, keeping the MLP input dimension at the vision encoder's width instead of quadrupling it as PixelShuffle would. An additional parallel vision encoder, the Eagle Module, supplies extra tokens for extreme aspect ratios and OCR-heavy inputs while remaining constrained to the same token budget during training. On the authors' in-house Ecom-VQA benchmark Valley2 scores 79.66, and on OpenCompass it averages 67.4 across eight benchmarks. The paper further shows that chain-of-thought post-training adds about 1.3 points to the OpenCompass average.","pith_inferences":["If the token-budget compromise generalizes, the same ConvAdapter-style compression could extend to audio inputs or longer video sequences, where token count is the main bottleneck.","The Ecom-VQA benchmark's four-way cyclic testing checks answer-position consistency, but an independent evaluation without the authors' hyperparameter tuning on the same data would clarify whether the margin over Qwen2-VL-7B is robust.","A testable extension would be to apply Valley2's ConvAdapter to other vision encoders while omitting the Eagle Module, isolating whether the compression or the extra encoder drives the e-commerce gains.","The paper's planned omni-model incorporating audio suggests the authors expect the same architectural pattern to transfer across modalities, but that claim is not yet supported by results here."],"forward_implications":["Valley2 reaches state-of-the-art accuracy on e-commerce VQA among open models of comparable size, as measured by the authors' Ecom-VQA benchmark.","It ranks second on the OpenCompass leaderboard among models with fewer than 10B parameters, with an average score of 67.4.","The ConvAdapter and Eagle Module design keeps average training sequence length under 1,000 tokens, and packing improves training efficiency by a factor of 2.2.","Chain-of-thought post-training raises the OpenCompass average from 66.11 to 67.40, showing that systematic reasoning data helps even on general benchmarks.","Code and model weights are open-sourced, allowing independent reproduction and application to other domains."],"supporting_citations":[{"why":"Qwen2.5 serves as the LLM backbone and provides the language reasoning and world knowledge that the ablation shows improves both Ecom-VQA and OpenCompass.","marker":"[79]"},{"why":"SigLIP-384 is the primary vision encoder used with the large MLP and ConvAdapter projector.","marker":"[85]"},{"why":"Ovis is the design source for the large two-layer MLP with enlarged hidden dimensions and a learnable visual vocabulary.","marker":"[52]"},{"why":"InternVL2 supplies the high-resolution tiling strategy that Valley2 adapts for single-image inputs, with best-ratio matching for aspect-ratio preservation.","marker":"[11]"},{"why":"Eagle provides the parallel vision encoder module that extends token capacity and reduces distortion in extreme-aspect-ratio and OCR scenarios.","marker":"[66]"},{"why":"LLaVA-CoT supplies the 100k chain-of-thought samples used in Stage-3 post-training to improve reasoning performance.","marker":"[76]"},{"why":"OpenCompass is the evaluation platform and leaderboard that establishes Valley2's second-place ranking among sub-10B models.","marker":"[12]"},{"why":"Qwen2-VL-7B is the main comparable open model that Valley2 surpasses on Ecom-VQA, serving as the headline baseline.","marker":"[74]"}],"fun_headline_variants":["Valley2 7B: top e-commerce VQA, second on OpenCompass","Efficient token folding boosts 7B multimodal model to SOTA","Open-source Valley2 7B ranks second on OpenCompass","Token-efficient design lifts 7B model to e-commerce SOTA","Valley2: scalable vision-language with token-efficient training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline e-commerce result rests on the validity of the in-house Ecom-VQA benchmark, which contains 536 multiple-choice pairs, has not been publicly released, and was used to tune the very hyperparameters the paper then evaluates.","fun_headline_variants_meta":{"raw":{"variants":["Valley2 7B: top e-commerce VQA, second on OpenCompass","Efficient token folding boosts 7B multimodal model to SOTA","Open-source Valley2 7B ranks second on OpenCompass","Token-efficient design lifts 7B model to e-commerce SOTA","Valley2: scalable vision-language with token-efficient training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001346,"raw_usage":{"total_tokens":5417,"prompt_tokens":845,"completion_tokens":4572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":4478}},"tokens_in":461,"tokens_out":4572,"duration_ms":28471,"temperature":1.0,"reasoning_tokens":4478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:39.715661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Releasing Ecom-VQA and having an independent lab re-run Valley2 alongside Qwen2-VL-7B and InternVL2-8B under identical settings would settle it; if the 79.66 versus 72.76 margin shrinks to near parity or the benchmark leaks training data, the claim loses its basis.","supporting_citations":[],"review_version":1}