{"id":"a428904f-e0ed-4456-b950-9ec31d5bba54","arxiv_id":"2507.05934","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"BlueLM-2.5-3B is a small multimodal model with a switchable thinking mode that reportedly matches larger models like Qwen3-4B and comes close to Kimi-VL-A3B-16B on many benchmarks.","lead":"This report describes BlueLM-2.5-3B, a 2.9-billion-parameter vision-language model that can switch between a fast answering mode and a longer 'thinking' mode. The authors report it matches or beats several larger open models on common benchmarks while using less training data, making it a candidate for phones and other edge devices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline reasoning comparisons rely on self-reported evaluations and small, high-stakes test sets; a few leaked or memorized AIME/LiveCodeBench problems could explain the reported parity, and the paper's qualitative decontamination description does not rule this out.","rationale":"The paper's strongest value claim is that a 2.9B dense MLLM reaches near-Qwen3-4B text reasoning and trails Kimi-VL by about 5% on multimodal benchmarks. For that to hold, the evaluation must be uncontaminated and comparable. The report asserts a decontamination pipeline but supplies no evidence specific enough to audit: Section 3.7.4 lists n-gram and MinHash levels without benchmark names, overlap counts, or hashes, and Section 3.3 shows that training and RL data are drawn from open-source corpora and teacher-model generations that can contain web- and PDF-derived reasoning problems. AIME24/25 have only 30 items, so a single memorized item changes a score by 3.3 points; the observed AIME24 gap to Qwen3-4B is exactly one item, and the AIME25 lead is one item. This makes the reasoning-parity claims extremely sensitive to a small amount of leakage. I am not claiming leakage occurred; I am claiming the report cannot currently exclude it, and the absence of released artifacts makes independent verification impossible. The 10-benchmark multimodal average is more robust, so the 'about 5% behind Kimi-VL' claim is less fragile than the small-set reasoning claims. The reader's weakest assumption covers this broad area; I partially agree and sharpen it to contamination risk in small high-stakes test sets. The reader's CONDITIONAL verdict already reflects the appropriate level of confidence, so no verdict change is needed.","tokens_in":28090,"tokens_out":8066,"duration_ms":88958,"concrete_test":"Release the training-data deduplication manifest for AIME24, AIME25, and LiveCodeBench, including exact and near-duplicate match counts with the evaluation sets; then independently run the final checkpoint on temporally future reasoning problems and on paraphrased versions of the AIME items. If any exact or near-duplicate matches are found, or if the score drops by more than one question under paraphrase, the reported reasoning parity is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims rest on self-reported evaluations with no released artifacts. The most fragile components are AIME24/AIME25 (30 items each) and LiveCodeBench: a single memorized or leaked problem shifts AIME scores by about 3.3 percentage points. BlueLM trails Qwen3-4B on AIME24 by one problem (73.3 vs 76.7) and leads on AIME25 by one problem (66.7 vs 65.6), so the reasoning-parity claim is sensitive to at most one or two items. The decontamination descriptions in Section 3.7.4 and Section 3.7.8 are qualitative: no hash lists, benchmark-version details, overlap counts, or held-out checks are provided. Because answer-generation data is synthesized from Qwen/DeepSeek teachers and sampled from web and PDF corpora (Section 3.3), near-duplicate contamination in these small high-value sets is not addressed. If one or two AIME problems appear in training or teacher-generated data, the reported parity with Qwen3-4B and the large AIME24 margin over Kimi-VL would be materially overstated. The concern is not that the results are false, but that the report currently provides no way to distinguish a genuinely superior 2.9B model from test-set memorization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents BlueLM-2.5-3B, a 2.9B-parameter dense multimodal LLM developed by vivo AI Lab. The authors describe a three-stage pretraining pipeline (pure-text distillation, joint text-image pretraining with 40% text, reasoning-enhanced and long-context stages), a post-training pipeline with SFT and hybrid RL, infrastructure engineering, and an evaluation on 20+ benchmarks. The headline claims are that the model supports both thinking and non-thinking modes through the [BlueThink] control token, that thinking mode is comparable to Qwen3-4B on text benchmarks and within roughly 5% of Kimi-VL-A3B-16B on multimodal benchmarks, and that this is achieved with substantially less total pretraining data than Qwen2.5-VL-3B and Qwen3-4B.","tokens_in":28460,"tokens_out":10052,"duration_ms":100947,"significance":"If the empirical claims hold, the contribution is significant for edge deployment of small MLLMs. The paper's strengths are the unusually detailed data-curation and infrastructure descriptions, the explicit mode-switch mechanism with a length-penalty objective aimed at token-budget control, and the breadth of benchmarks considered. However, the manuscript does not release model weights, benchmark prompts, or evaluation harnesses; all central comparisons are self-reported, and several quantitative claims are internally inconsistent or lack measurement detail. The soundness of the headline parity claims therefore cannot be assessed at the level required for a research contribution until the contamination and uncertainty issues below are addressed.","major_comments":[{"comment":"The central reasoning claims (AIME24 73.3 vs 76.7 for Qwen3-4B; AIME25 66.7 vs 65.6; LiveCodeBench-v1 81.5) are separated by at most one or two items on 30-problem tests, yet the decontamination evidence is only qualitative. The manuscript states that rigorous deduplication is performed but reports no overlap counts, hash lists, benchmark-version identifiers, or held-out checks for the teacher-generated data described in Section 3.3 or for the web- and PDF-derived corpora. Since one leaked AIME problem changes the score by about 3.3 points, the parity claim cannot be distinguished from test-set memorization without released artifacts or quantitative contamination reporting.","section":"§3.7.4, §3.7.8, §3.3, Tables 4-7"},{"comment":"All benchmark numbers are single-point estimates with no error bars, no repeated sampling, no significance tests, and no released evaluation harness. The comparable-performance claims are based on differences of one to a few benchmark items on small sets. The manuscript only states that OpenCompass was used; it does not report decoding parameters, numbers of samples, or per-benchmark prompts. Please provide confidence intervals or repeated-sample statistics, at least for the 30-item AIME sets, and state the evaluation protocol for each score.","section":"§5, Tables 4-8"},{"comment":"The token counts for the reasoning-enhanced stage are internally inconsistent. Section 2.2.3 reports 2.5T tokens (1.3T textual + 1.2T multimodal); Section 3.3 reports approximately 950B pre-training tokens plus 450B synthetic tokens, totaling 1.4T; Section 3.6 and the Introduction state that 3.3T tokens of reasoning-enhanced data were used across these stages. Although the 4T + 2.5T + 1.3T entries in Table 2 do sum to the 7.8T MLLM pretraining number in Table 3, the descriptions in Sections 2.2.3, 3.3, and 3.6 cannot all be correct. Since Section 3.6 uses these totals to support the data-efficiency claim, a single consistent breakdown by stage and by data type must be provided.","section":"§2.2.3, §3.3, §3.6, Table 3"},{"comment":"The claim that the [BlueThink] switch has a failure rate below 1 PPM is not supported by any definition of failure, sample size, or testing protocol. This quantitative reliability claim is used to justify the mode-switch design and the practical usability of the model. Please report the evaluation set, the measurement procedure, and the observed failure count.","section":"§3.5.1"},{"comment":"The GUI comparison rests in part on ScreenSpot vivo, an in-house Chinese GUI grounding benchmark that is neither released nor described. The table provides no sample size, task format, annotation protocol, or metric definition for this benchmark. Without this information, the statement that BlueLM-2.5-3B outperforms Qwen2.5-VL-7B and UI-TARS-7B on this benchmark cannot be independently checked. Please release the benchmark or replace it with public GUI benchmarks for the headline GUI claims.","section":"Table 8"},{"comment":"The text says that the length difference is capped as DeltaL = max(500, DeltaL). The max operator places a floor, not a cap, on the length difference, which contradicts the stated purpose of preventing large reward fluctuations. If the intended operation is min(500, DeltaL), please correct the equation and any downstream derivation; if the max is intentional, please explain why a floor is needed and how the resulting denominator is used in the length reward.","section":"§2.3.5, Eq. (1)"}],"minor_comments":[{"comment":"The model name LLL-base-3B appears to be a typo for LLM-base-3B.","section":"§2.2.1"},{"comment":"The thinking-mode control token is typeset inconsistently across the paper, for example [ |BlueThink|], [|BlueThink|], and [BlueThink]; please unify the notation.","section":"§2.3.3, §3.5.1"},{"comment":"The pure-text pretraining token counts are given as 6T plus 3T in Section 2.2.1, 6.3T plus 3T in Section 3.1, and 9.3T in Table 3; please reconcile these numbers.","section":"§2.2.1, §3.1, Table 3"},{"comment":"References [54] and [55] both cite the Kimi-VL Technical Report; consolidate them to avoid duplicate citation.","section":"References"},{"comment":"The description of the in-house assessment model for long-chain reasoning reports speed and cost improvements but gives no scoring-agreement metric; please either provide the consistency number or state that it is unavailable.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an industry technical report with no public release of model weights, evaluation harness, or contamination certificates. The main risk is not fraud but unverifiable evaluation and internally inconsistent data accounting. If the journal requires released artifacts for empirical model claims, the appropriate outcome would be rejection; under the current policy I recommend major revision with emphasis on quantitative contamination reporting and uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: this is a serious small-model engineering report, and the switchable thinking/non-thinking mode with explicit token-budget control is a genuinely useful capability for edge deployment. The headline parity claims with Qwen3-4B and Kimi-VL are plausible but not proven; the numbers rest on self-reported evaluation, small high-stakes test sets, and no released artifacts.\n\nWhat's actually new: the [|BlueThink|] control token plus length-penalty RL gives users a mode switch and a token budget at 2.9B dense scale. The training recipe—SigLIP2 + AnyRes, 40% pure text in joint pretraining, pruning+distillation from a 7B, curriculum-style reasoning data, and hybrid RL with a dynamic length penalty—is a coherent and informative combination. The data-efficiency claim (17.1T total vs 22.1T for Qwen2.5-VL-3B) is worth taking seriously, and the infrastructure sections are concrete (context parallelism, async RL, load balancing). The citation pattern is fair: they credit Qwen, Kimi, and the RL systems they build on.\n\nWhere it frays: no weights, code, or data are released, so all benchmark results are self-reported. The decontamination description (Sections 3.7.4/3.7.8) is qualitative—no hash lists, overlap counts, or benchmark-version details—and the reasoning-eval conclusions are sensitive: AIME24 and AIME25 are 30 problems each, so one memorized or leaked problem is worth about 3.3 points. The reported AIME24 gap to Qwen3-4B is exactly one problem, and the AIME25 edge is one problem, so the \"comparable to Qwen3-4B\" claim hinges on a couple of items. There's also an in-house GUI benchmark (ScreenSpot vivo) with no public description, and the \"<1 PPM\" thinking-switch reliability number has no methodology behind it. Token accounting is inconsistent: Section 3.3 lists 950B + 450B = 1.4T reasoning tokens, while Section 2.2.3 says the reasoning-enhanced stage used 2.5T, and Section 3.6 claims 3.3T across stages. Those differ by a wide margin and the report doesn't reconcile them.\n\nWho it's for: people building small MLLMs for phones and robots, and anyone working on training recipes for efficient multimodal reasoning. It deserves a serious referee, but the referee should ask for artifacts, a detailed contamination report, and fixed token accounting. As it stands, treat the benchmark tables as promising but unverified.","headline":"A serious small-model systems report with a genuinely useful thinking-mode switch, but the headline benchmark parity rests on self-reported, contamination-sensitive numbers that need artifacts before they convince me.","tokens_in":29250,"tokens_out":3943,"would_cite":true,"duration_ms":40340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BlueLM-2.5-3B claims a 2.9B dense multimodal model can switch between fast and deep reasoning modes.","keywords":["multimodal large language models","edge deployment","thinking mode","token budget control","reinforcement learning","data efficiency","reasoning","vision-language model"],"falsifier":"Run the released checkpoint, if it becomes available, on AIME24 and LiveCodeBench-v1 under the stated sampling settings, then audit whether near-duplicates of test problems appear in the described training data; if the scores drop materially or any overlap surfaces, the central comparison claim is falsified.","tokens_in":27892,"feed_emoji":"🤖","tokens_out":7579,"duration_ms":75365,"temperature":0.7,"pith_summary":"The paper aims to establish that a small dense multimodal model can serve as an edge-deployable assistant without sacrificing either fast response mode or deep reasoning. It presents BlueLM-2.5-3B, a 2.9B-parameter multimodal model that toggles between a thinking mode and a non-thinking mode through a control token, with explicit control over how many tokens the model may spend thinking. If the claims hold, a single small model could run on phones or vehicles and in thinking mode match a 4B text-only reasoning model on text benchmarks while trailing a 16B multimodal model by only about 5% on multimodal benchmarks. In non-thinking mode, it is reported to outperform a similarly sized vision-language model on most multimodal benchmarks, and the whole recipe reportedly uses less total training data than comparable models.","feed_headline":"A 2.9B AI model adds a thinking switch for edge devices","feed_subtitle":"One small multimodal model toggles between instant answers and long reasoning, matching bigger models on less data.","key_machinery":"The load-bearing mechanism is a special control token, written [|BlueThink|], placed at the end of the query to switch the model between a long chain-of-thought thinking mode and a fast non-thinking mode. Around that core, the training recipe stacks a multi-phase pipeline: pure-text pretraining of a 2.5B backbone via pruning and distillation from a 7B teacher, joint text-image pretraining with 40% pure-text data, a reasoning-enhanced stage with long and short chain-of-thought synthetic data, a fast-decay long-context stage, and then a hybrid reinforcement learning stage. Within the RL stage, a dynamic length penalty drawn from a per-group comparison the authors call Group Overlong encourages the model to compress its reasoning so it stays accurate under a tight token budget. This combination of mode control, length-aware RL, and data curation is what the paper argues lets a dense 2.9B model close the gap to far larger systems.","core_discovery":"The paper claims that a 2.9B-parameter dense multimodal model can simultaneously offer a fast non-thinking mode and a long-reasoning thinking mode, with the user controlling the reasoning budget through a dedicated token. The authors report that in thinking mode the model is comparable to a 4B-parameter text-only reasoning model on text benchmarks and trails a much larger 16B MoE multimodal reasoning model by only about 5% on average across ten multimodal benchmarks. In non-thinking mode, it is said to outperform a similarly scaled vision-language model on most multimodal metrics and across the text benchmarks shown. The report also claims especially strong accuracy at constrained token budgets under 8K tokens on AIME24 and LiveCodeBench-v1, and a measured thinking-switch failure rate below one in a million uses. The contribution is framed as the first 3B-scale multimodal model with both thinking and non-thinking modes plus explicit token-budget control, achieved with 17.1T total pre-training tokens, about 23% less than a comparable model.","pith_inferences":["Editorial inference: If the reported switch reliability of fewer than one failure per million uses replicates outside the authors' own data distribution, the end-of-query control-token pattern is a portable design that other small dense models could adopt without architectural change.","Editorial inference: The Group Overlong length penalty and the long-CoT data selection are separable interventions; an ablation holding data fixed while toggling the penalty would isolate how much of the under-8K-token accuracy gain comes from RL length control.","Editorial inference: The data-efficiency claim would become stronger or weaker with an explicit scaling-curve comparison at matched token budgets across training, which the report does not provide.","Editorial inference: If these results replicate, they would push edge vision-language design toward dense models with explicit mode control rather than mixture-of-experts architectures, since close parity is achieved at a lower parameter count and lower inference complexity."],"forward_implications":["A single 2.9B dense model can serve both fast-response and deliberate-reasoning workloads, with the user choosing the mode by adding or omitting a control token.","In thinking mode, the model's text-only benchmark scores are comparable to a 4B-parameter reasoning model while using about 60% of that model's parameters.","Across ten multimodal benchmarks in thinking mode, the reported gap to a 16B MoE reasoning model averages about 5%, and the model beats a 72B multimodal model on two math-vision suites.","Under constrained token budgets below 8K tokens, the model reports large accuracy gains over both baselines on AIME24 and LiveCodeBench-v1, which matters for latency-limited edge deployment.","The recipe uses substantially less total pre-training data than comparable open models, suggesting the gains come from data curation and training design rather than raw scale."],"supporting_citations":[{"why":"It supplies the main similar-scale vision-language baseline used in the non-thinking mode comparisons.","marker":"[3]"},{"why":"It supplies the 4B-parameter text reasoning baseline and the thinking-mode decoding configuration used in evaluations.","marker":"[65]"},{"why":"It supplies the larger 16B MoE multimodal reasoning baseline against which the about-5% thinking-mode gap is measured.","marker":"[54]"},{"why":"It supplies the vision transformer initialization that defines the model's visual encoder.","marker":"[57]"},{"why":"It supplies the tile-based resolution processor used for dynamic input resolutions up to 1512 by 1512 pixels.","marker":"[26]"},{"why":"It is the source of the length-penalty idea that the report adapts as Group Overlong to control thinking token budgets.","marker":"[53]"},{"why":"It supplies the reinforcement learning training framework that the authors adapt for asynchronous multimodal RL training.","marker":"[45]"}],"fun_headline_variants":["2.9B MLLM toggles fast and reasoning modes on edge","Small multimodal AI matches bigger on text, beats same-scale","Budget-controlled thinking in a 2.9B on-device model","First 3B MLLM with thinking and instant modes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported benchmark numbers are clean, complete, and measured under conditions identical to those of the baselines, since the report supplies no externally verified decontamination certificate and no released artifacts.","fun_headline_variants_meta":{"raw":{"variants":["2.9B MLLM toggles fast and reasoning modes on edge","Small multimodal AI matches bigger on text, beats same-scale","Budget-controlled thinking in a 2.9B on-device model","First 3B MLLM with thinking and instant modes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3140,"prompt_tokens":1028,"completion_tokens":2112,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2036}},"tokens_in":644,"tokens_out":2112,"duration_ms":19693,"temperature":1.0,"reasoning_tokens":2036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:14:29.056662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released checkpoint, if it becomes available, on AIME24 and LiveCodeBench-v1 under the stated sampling settings, then audit whether near-duplicates of test problems appear in the described training data; if the scores drop materially or any overlap surfaces, the central comparison claim is falsified.","supporting_citations":[],"review_version":1}