{"id":"830f3d45-b28a-4fae-8778-8a3d53f290e5","arxiv_id":"2412.08443","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"POINTS1.5-7B, a vision-language model with a NaViT-style encoder, bilingual data, and filtered instruction tuning, ranks first on OpenCompass among sub-10B models.","lead":"POINTS1.5 is a 7-billion-parameter vision-language model trained on less than 5 billion tokens that tops the OpenCompass leaderboard among models under 10 billion parameters. Its three claimed innovations are native dynamic-resolution image encoding, Chinese bilingual support, and filtering of visual instruction data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Token-budget claim is internally inconsistent: the abstract says <4B tokens while Table 3 sums to ~4.4B; the ranking claim also needs a robustness check.","rationale":"The reader's weakest-assumption concern is that OpenCompass leaderboard scores may not be comparable across models. That is a legitimate concern, but it is not the most decisive issue: the paper contains an internal arithmetic contradiction in its own token-budget claim. Table 3 reports about 2.1B pre-training tokens and about 2.3B instruction-tuning tokens, totaling about 4.4B, which contradicts the abstract's '<4B' statement and is also inconsistent with the conclusion's 'less than 5B' wording. Since the central claim explicitly includes the token budget, this is a load-bearing flaw that does not depend on any external benchmark protocol. The ranking claim is also fragile because model soup was selected on the OpenCompass benchmarks themselves, and the reported margin over the closest sub-10B competitor is only about 0.36 average points; without confidence intervals or a held-out evaluation, 'ranks first' is not firmly established. These issues do not warrant rejection, because the model weights are released and the ranking is externally checkable, but they do warrant a conditional acceptance pending verification. I therefore keep the reader's conditional verdict.","tokens_in":16035,"tokens_out":6706,"duration_ms":70124,"concrete_test":"Use the released training code and data configs to exactly count all tokens consumed in pre-training and visual instruction tuning, including prompt and vision tokens. If the total is >=4B, the abstract's 'fewer than 4 billion tokens' claim is unsupported and must be revised; if the authors intended to count only newly created Chinese tokens, that exclusion must be stated explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the paper's own arithmetic for the efficiency claim. The abstract asserts POINTS1.5-7B is 'trained on fewer than 4 billion tokens,' but Section 5 states the total is 'less than 5 billion tokens,' and Table 3 lists ~2.1B pre-training tokens plus ~2.3B visual instruction tuning tokens, which sums to ~4.4B. Unless some tokens are excluded without explanation, the abstract's '<4B' statement is false under the paper's own numbers. This is independent of any leaderboard-comparability issue and directly undercuts half of the headline claim. Separately, the ranking claim is fragile: the final model is a model soup selected on the OpenCompass benchmarks (Section 5), and Table 4's margin over Qwen2-VL-7B is only ~0.36 average points, so the leaderboard result needs a robustness check. Both issues support a conditional verdict, but the token-count contradiction is the one already settled by the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"POINTS1.5 is a 7B-parameter vision-language model built on the authors' earlier POINTS1.0. The paper replaces a fixed-resolution CLIP encoder with a NaViT-style dynamic-resolution encoder, adds Chinese pre-training and instruction-tuning data, and applies LLM-based filtering to visual instruction datasets. The headline claims are that the model is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among sub-10B models. The paper reports architecture details, dataset construction, training configurations in Table 3, and evaluation against open-source and proprietary models in Tables 4-5.","tokens_in":16240,"tokens_out":5331,"duration_ms":50732,"significance":"If the headline results held, the contribution would be practically significant: a competitive sub-10B open VLM with a very small training-token budget, bilingual support, and native arbitrary-resolution image processing. The paper also releases model weights and code links, which supports reproducibility. However, the efficiency claim is internally inconsistent with the paper's own token counts, and the ranking claim is weakened by model selection on the evaluation benchmark itself. These issues need to be resolved before the headline claims can be taken at face value.","major_comments":[{"comment":"The abstract states that POINTS1.5-7B is 'trained on fewer than 4 billion tokens,' but Section 5 states the total is 'less than 5 billion tokens,' and Table 3 reports about 2.1B pre-training tokens plus about 2.3B visual instruction tuning tokens, which sums to about 4.4B. Under the paper's own accounting the '<4B' claim is false; the authors should either correct the abstract or explain which tokens are excluded from the count.","section":"Abstract; Section 5; Table 3"},{"comment":"The final model is selected by averaging models with the best OpenCompass scores, and the same OpenCompass benchmark suite is then used in Table 4 to support the ranking claim. Because the model soup composition is chosen from these very scores, the reported margin over Qwen2-VL-7B (about 0.3 average points) is not an unbiased estimate. The authors should report robustness, e.g., multiple soup seeds, or evaluation on a held-out benchmark not used in selection.","section":"Section 5, 'Model Soup over Best Performing Model'; Section 6, Tables 4-5"},{"comment":"The abstract claims that the filtering methods were 'comprehensively evaluate[d]' and the best ones selected, but Section 4 provides no ablation table or quantitative comparison. In particular, the image-free-answer filtering is reported to slightly degrade performance, yet the paper does not state whether the degraded variant was used in the final model. Adding ablations or removing the 'comprehensive evaluation' claim is required to support the contribution.","section":"Section 4, 'Visual Instruction Tuning Set Filtering'"}],"minor_comments":[{"comment":"The caption contains a typo: 'converntional' should be 'conventional.'","section":"Figure 3 caption"},{"comment":"The table lists two-number entries (e.g., 20.4/18.4 for InternVL2-8B) without defining what the two values represent; a footnote or legend is needed.","section":"Table 5"},{"comment":"The phrase 'image sequence[Dao, 2024]' should have a space before the citation bracket.","section":"Section 2, batch forwarding"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the headline result is real in the sense that the weights are released and the OpenCompass numbers are externally checkable; a 7B model trained on roughly 4.4B tokens does place near the top of the sub-10B pack on that leaderboard as of Dec 2024. Second, the paper's own arithmetic undercuts its headline: the abstract says 'fewer than 4 billion tokens' while Table 3 lists 2.1B pre-training plus 2.3B instruction tuning, which sums to 4.4B. The conclusion says 'less than 5 billion,' which is consistent with the table, so the abstract appears to be simply wrong.\n\nWhat's actually new: not a new architecture or objective. It recombines three known pieces—NaViT-style dynamic resolution, a Chinese data curation pipeline using CapFusion and perplexity filtering, and LLM-based filtering of instruction data—and shows that this combination trains a competitive 7B VLM with a small budget. That's a useful empirical recipe, especially for teams that can't afford large-scale pre-training. The paper also deserves credit for reporting some negative results: unfreezing the vision encoder hurt, and removing image-free questions slightly degraded performance, which they honestly leave in rather than spin.\n\nSoft spots, in order of severity. The token-count mismatch is the clearest, and it should be fixed before anything else. Second, the 'rigorous filtering methods' are described with examples and a couple of numbers, but there is no ablation table showing the contribution of each filter to the final benchmark score. The phrase 'comprehensively evaluate' is doing work that a table should do. Third, the final model is selected via model soup on the OpenCompass average itself (66.5 to 67.4), and then the same benchmark is used for the ranking claim. That's a mild form of benchmark fitting, and the margin over Qwen2-VL-7B is about 0.36 points on average, so the 'first' claim is not robust to small perturbations. I'd want to see variance across runs or at least a note that the ranking holds under different seeds.\n\nOverall, this is a competent technical report from an industry lab, not a methodological breakthrough. The reader who gets value is someone training or fine-tuning a VLM on a budget who wants a concrete recipe and a released checkpoint. The citation pattern is fine; self-citing POINTS1.0 is appropriate. It deserves a serious referee, but the referee should hold the authors to fixing the arithmetic and giving at least one filtering ablation. I'd treat the leaderboard claim as 'reported on Dec 8, 2024' rather than a timeless fact.","headline":"Useful training recipe and released weights, but the abstract's token count doesn't match the paper's own table, and the leaderboard claim is thinner than it looks.","tokens_in":16783,"tokens_out":3082,"would_cite":true,"duration_ms":30560,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"POINTS1.5-7B, trained on under 4 billion tokens, claims first place among sub-10B vision-language models on the OpenCompass leaderboard.","keywords":["vision-language model","NaViT","dynamic resolution","bilingual model","visual instruction tuning","data filtering","model soup","OpenCompass leaderboard"],"falsifier":"Run the released POINTS1.5 checkpoint through the public VLMEvalKit evaluation on the eight leaderboard benchmarks with the same generation settings used by the leaderboard, and compare the reproduced average with the top competing sub-10B models; if another sub-10B model scores higher, or if the reproduced POINTS1.5 score falls below 67.4, the ranking claim is false. Separately, checking whether the token counts in the training logs match the abstract's 'fewer than 4 billion' figure would settle the token-budget sub-claim.","tokens_in":15825,"feed_emoji":"🤖","tokens_out":9063,"duration_ms":79299,"temperature":0.7,"pith_summary":"POINTS1.5 is a 7-billion-parameter vision-language model built for real-world image tasks such as reading receipts, extracting formulas, describing charts, and solving visual math problems, in both English and Chinese. The paper's central claim is that this model reaches the top of a public multimodal leaderboard among models under 10 billion parameters while being trained on fewer than 4 billion tokens. The authors attribute the result to three changes over their previous model: a vision encoder that natively accepts images at any resolution without splitting them into tiles, newly collected Chinese training data, and systematic filtering of the instruction-tuning data. If the claim holds, a comparatively small training budget can match or beat much larger open models, and the recipe can be reused by other groups.","feed_headline":"7B vision-language model tops sub-10B leaderboard","feed_subtitle":"Native multi-resolution vision, bilingual data, and data filtering get it ahead of larger rivals on under 5B tokens.","key_machinery":"The architecture is the conventional LLaVA-style pipeline: a NaViT-style vision encoder (a Vision Transformer that patchifies images of any resolution and aspect ratio into a single patch sequence), a two-layer MLP projector with GELU, and Qwen2.5-7B-Instruct as the language backbone. The carrying mechanism is native dynamic resolution: instead of tiling a large image into fixed-size crops, the encoder processes the whole image and packs multiple image sequences into one long forward pass, recording start and end indices so self-attention stays within each image (via FlashAttention-2). On the data side, CapFusion merges original and VLM-generated captions, perplexity filtering selects the lowest-perplexity portion of the Chinese captions, and instruction data is filtered by LLM-based grammar detection and by discarding answers that are answerable without the image. Model soup averages the best checkpoints to produce the final weights.","core_discovery":"The paper claims that POINTS1.5-7B—a LLaVA-style model with a frozen NaViT vision encoder, an MLP projector, and Qwen2.5-7B-Instruct as its language model—achieves the highest OpenCompass average among models with fewer than 10 billion parameters as of December 8, 2024. The claimed result comes from a training budget of about 2.1 billion pretraining tokens plus 2.3 billion instruction-tuning tokens, built from two million captions (roughly half English, half Chinese) and a filtered visual instruction set. The authors report that model soup raises the best single checkpoint from 66.5 to 67.4 on the leaderboard average, and that the model's mathematical performance on MathVista, MATH-Vision, and MathVerse is particularly strong. They also state that filtering out samples whose questions can be answered without the image slightly degraded performance, so those samples were retained.","pith_inferences":["The abstract says fewer than 4 billion tokens while the training section says less than 5 billion and the table sums to about 4.4 billion; reconciling this figure would sharpen the headline claim.","The paper's observation that dropping no-image questions slightly hurt performance is a clue that pure-text data acts as a regularizer, which a controlled study could test by adding a small text-only set to a VLM training run.","The leaderboard claim is a snapshot from December 8, 2024, so it is necessarily time-limited; re-checking after later sub-10B models are submitted would confirm whether the ranking is durable.","An ablation isolating the NaViT encoder from the data filtering would show how much of the gain comes from resolution handling versus data curation."],"forward_implications":["Other groups can reproduce the recipe on a 7B language model—NaViT-style encoder, about two million pretraining captions, filtered instruction data, and model soup—without needing tens of billions of tokens.","Native dynamic resolution removes the tile-splitting step, so documents, screenshots, and charts of arbitrary sizes can be served in a single packed forward pass, simplifying deployment.","The Chinese data pipeline, built by translating existing sets, generating answers with a strong VLM, and verifying by human labelers, offers a path to bilingual capability when open Chinese instruction data is scarce.","The slightly negative effect of removing image-free questions suggests that some text-only samples belong in visual instruction tuning, making data filtering a question of balance rather than simple elimination.","A 7B model beating several-times-larger models on the leaderboard would lower the compute bar for real-world multimodal applications."],"supporting_citations":[{"why":"Introduces NaViT, the native arbitrary-resolution patch-and-pack vision transformer that POINTS1.5 adopts as its vision encoder.","marker":"[Dehghani et al., 2024]"},{"why":"Qwen2-VL, the NaViT-style model whose dynamic-resolution approach and vision-encoder design this work follows.","marker":"[Wang et al., 2024b]"},{"why":"POINTS1.0, the predecessor whose architecture, datasets, and affordable training strategy POINTS1.5 extends.","marker":"[Liu et al., 2024d]"},{"why":"CapFusion, the caption-merging method used to construct the pretraining captions from LAION data.","marker":"[Yu et al., 2024]"},{"why":"The OpenCompass leaderboard whose ranking underlies the claim of first place among sub-10B models.","marker":"[Contributors, 2023]"},{"why":"Model soup, the weight-averaging technique that raises the final leaderboard score to 67.4.","marker":"[Wortsman et al., 2022]"},{"why":"FlashAttention-2, used to restrict self-attention to packed image-sequence boundaries in the NaViT batch forward.","marker":"[Dao, 2024]"},{"why":"Qwen2.5 technical report, the LLM used for grammatical-error detection in instruction data filtering.","marker":"[Yang et al., 2024]"}],"fun_headline_variants":["7B VLM tops sub-10B leaderboard with under 5B tokens","Native resolution, bilingual data: 7B VLM takes #1","POINTS1.5: Small model, big win on OpenCompass","Dynamic-res vision and Chinese boost 7B VLM to top","Data filtering plus soup: 7B VLM leads sub-10B pack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking claim assumes that all scores quoted from the public leaderboard were produced under identical, fair evaluation settings for every model; if some sub-10B entries were evaluated under different protocols, being first may be an artifact of protocol differences rather than model quality.","fun_headline_variants_meta":{"raw":{"variants":["7B VLM tops sub-10B leaderboard with under 5B tokens","Native resolution, bilingual data: 7B VLM takes #1","POINTS1.5: Small model, big win on OpenCompass","Dynamic-res vision and Chinese boost 7B VLM to top","Data filtering plus soup: 7B VLM leads sub-10B pack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4109,"prompt_tokens":1010,"completion_tokens":3099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":626,"tokens_out":3099,"duration_ms":23501,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:48:13.789748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released POINTS1.5 checkpoint through the public VLMEvalKit evaluation on the eight leaderboard benchmarks with the same generation settings used by the leaderboard, and compare the reproduced average with the top competing sub-10B models; if another sub-10B model scores higher, or if the reproduced POINTS1.5 score falls below 67.4, the ranking claim is false. Separately, checking whether the token counts in the training logs match the abstract's 'fewer than 4 billion' figure would settle the token-budget sub-claim.","supporting_citations":[],"review_version":1}