{"id":"9b6549b1-c0b2-49a3-93d3-6a6ba9d0955e","arxiv_id":"2505.11945","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A visual instruction tuning model that combines global fusion and dual-expert token selection to cut visual tokens by 75 to 95 percent with comparable or better benchmark scores.","lead":"This paper introduces LLaVA-Meteor, a system that compresses visual tokens by 75 to 95 percent before they enter a language model, while keeping benchmark accuracy roughly the same or better. A generalist reader might care because it addresses the practical cost bottleneck in image-plus-language AI systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual-Native Selection's instruction token may receive no gradient through the discrete Top-K step, so the claimed instruction-aware selection is untestable and possibly not learning what the paper claims; the +1.3 gain over visual-only may not come from instruction relevance.","rationale":"The reader correctly identified that the instruction token's ability to learn instruction-relevant cues is the weakest assumption. My pass sharpens this into a concrete mechanism: the discrete Top-K selection in Eq. (6) likely prevents gradients from reaching the native-expert scores, so the instruction token may be trained only as a generic sequence element through FGF. This is an internal consistency issue, not just a missing ablation. However, the paper's headline empirical claim—comparable or superior accuracy at reduced token counts—could still hold even if the native expert is not instruction-aware, because the visual expert and FGF may drive most of the gains. The absence of code and checkpoints makes it impossible to verify the gradient path or reproduce Table 3, so the appropriate verdict remains conditional rather than a rejection. The proposed control experiment and the two-question invariance check would settle whether the native expert contributes anything beyond a random scorer.","tokens_in":15618,"tokens_out":5339,"duration_ms":59449,"concrete_test":"Run the Visual-Native variant with the native scores replaced by a fixed, randomly initialized and frozen INS (or by random scores with the same distribution), keeping FGF and all other training conditions identical. If the average over the 12 benchmarks is statistically indistinguishable from the reported Visual-Native row of Table 3, the native expert is not learning instruction-aware cues. As an additional direct check, feed the same image with two different questions (e.g., counting vs. color) into the trained model and record the selected token indices; because INS is not prompt-conditioned, the selected set must be identical, confirming that selection is question-agnostic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism claimed to make selection instruction-aware cannot be trained as described. In Section 3.3, the native score NS^i is computed from the instruction token INS and is used only to choose the Top-K indices in Eq. (6); the Top-K operation is discrete, so no gradient flows from the language-model loss back through AS^i or NS^i to INS. The only gradient INS receives is through the FGF sequence output F^i, which is shared with the visual-expert pathway. Since INS is also not conditioned on the actual text prompt (the paper concedes this in Section 1), it can at best learn a generic image summary, not per-query instruction relevance. Consequently, the +1.3 average improvement of Visual-Native over Visual-only in Table 3 cannot be attributed to instruction-aware token selection; it may come from the extra INS parameters or from the FGF module. The phrase 'training-free Visual-Native Selection' in Table 4 is also in tension with the trained INS token, suggesting the selection is treated as a non-learned masking step. If no gradient reaches the native scores, the paper's headline novelty—instruction-aware compression—rests on an untested assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaVA-Meteor, a vision-language model built on LLaVA-UHD, in which a Top-Down Compression pipeline replaces the standard projector. The pipeline has two stages: Flash Global Fusion (FGF), a lightweight SSM-based module with a local-to-single scanning pattern and a shared learnable instruction token (INS), and Visual-Native Selection (VNS), which scores tokens using both a CLS-attention-based visual expert and an INS-similarity-based native expert, then keeps the Top-K tokens per sub-image for the LLM. Experiments across 12 benchmarks report that reducing visual tokens by 75--95% yields results comparable to or better than LLaVA-UHD, including a +2.0 average gain at 144 tokens and only a 1.1 average drop at 32 tokens. The paper also reports a lightweight projector (37M parameters) and higher token throughput relative to LLaVA-UHD.","tokens_in":15811,"tokens_out":6755,"duration_ms":69657,"significance":"If the reported results hold, the proposed architecture is practically relevant: it offers a nontrivial accuracy/efficiency trade-off, strong benchmark coverage, and a simple fusion/selection decomposition that could be reused by other efficient VLM pipelines. The local-to-single scanning idea is a straightforward and plausible improvement over plain Mamba scanning. However, the paper's central novelty is the claim that the native expert makes selection instruction-aware. That claim is not supported by the training signal: the discrete Top-K step prevents gradients from flowing to the native scores, and the instruction token is shared across images and prompt-independent. The reported gains could therefore arise from a generic learned query rather than from instruction-conditioned selection. The lack of released code, model weights, or multi-seed statistics further weakens confidence in the headline comparisons.","major_comments":[{"comment":"The Top-K selection is non-differentiable, so the native scores NS^i and the aggregation AS^i never receive a direct gradient from the language-model loss. The instruction token INS is trained only through its influence on the fused features F^i inside FGF, and it is shared across all images and not conditioned on the text prompt (as the paper acknowledges in Section 1). Consequently, the claim that the native expert \"captures instruction preferences\" and that Visual-Native Selection is \"instruction-aware\" is not justified by the training procedure. The +1.3 average gain of Visual-Native over Visual-only in Table 3 could come from the extra INS parameters, from the FGF pathway, or from a generic learned query rather than from instruction-conditioned token selection. Please provide direct evidence of instruction dependence: e.g., show that selected token subsets change with different questions on the same image, or compare against a version with a randomly initialized and frozen INS, or replace the top-K with a differentiable relaxation that provides supervision to the scores.","section":"Section 3.3, Eqs. (4)--(6)"},{"comment":"The text accompanying Table 4 describes Visual-Native Selection as \"training-free,\" but Section 3.2 defines INS as a \"shared learnable instruction token\" optimized in both pretraining and instruction tuning, and the native expert score is computed from this token. This is a direct contradiction. If the selection scores are never updated by any loss, the authors should state this explicitly and explain how the native expert is supposed to acquire instruction knowledge; if they are updated indirectly through FGF, then calling the module \"training-free\" is misleading. The efficiency comparison and the parameter-count argument depend on this distinction, so it needs to be resolved.","section":"Section 4.4 and Table 4"},{"comment":"The central empirical claims rest on single runs without released code, model weights, or evaluation scripts. Several of the key deltas are small: for example, in Table 1 many individual benchmarks differ by only 0.1--0.7 points between LLaVA-Meteor and LLaVA-UHD, and the +1.3 gain in Table 3 is driven by a few benchmarks (MMB, MMVet, POPE, SEED) while other benchmarks are nearly unchanged. Without seed variance or error bars, the reader cannot tell whether the headline +2.0 average is a real effect or noise. Efficiency is reported only as TPS and projector parameters; actual inference latency and memory usage for the LLM are not reported, although those are the quantities the token reduction is supposed to improve. Please report multi-seed statistics, release the artifacts, and add end-to-end latency/memory measurements.","section":"Tables 1--4 and Section 4.3"},{"comment":"The claim of 95% token compression with \"comparable or superior\" performance is overstated for the 32-token configuration. In Table 1, the 32-token row is on average 1.1 points lower than LLaVA-UHD, and it drops by 2.7 points on TextVQA and 4.2 points on DocVQA, both of which are text-oriented benchmarks where compression is most consequential. This is not \"comparable\" in the strong sense implied by the abstract. The authors should either soften the claim to describe the accuracy-efficiency trade-off explicitly or provide additional evidence, such as task-specific analysis, that the losses are acceptable.","section":"Abstract and Section 4.3"}],"minor_comments":[{"comment":"The word \"ouputs\" appears in the sentence after Eq. (2); it should be \"outputs.\"","section":"Section 3.2"},{"comment":"The phrase \"144 tokens for on sub-images\" appears to contain a typo; it should probably be \"144 tokens for one sub-image.\"","section":"Section 4.3"},{"comment":"The summation notation in Eq. (3) is corrupted (\"PHuWu\" appears where a summation symbol is intended); please fix the formula so that the normalization is unambiguous.","section":"Section 3.3, Eq. (3)"},{"comment":"The token counts in Table 1 (\"~256,\" \"~114,\" \"~56\") and Table 4 (\"144,\" \"64,\" \"32\") are confusing at first glance because the former are average totals across sub-images while the latter are per-sub-image counts. Please clarify this in the table captions and refer to the same quantity consistently in the text.","section":"Tables 1 and 4"},{"comment":"The progressive weighting scheme for the native expert (linearly increasing its contribution from 0 to 1-lambda over early training) is mentioned but never ablated or analyzed. Since this is a training-dynamics choice, a small ablation or a sentence explaining its effect would strengthen the paper.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The central difficulty is that the paper's main novelty, instruction-aware token selection, is not supported by the training signal as described. I do not see evidence that the native expert learns per-query instruction relevance, and the +1.3 gain over the visual expert alone can be explained by confounding factors. This is fixable within the scope of a revision if the authors add a proper training-signal analysis or a differentiable selection mechanism, but it is load-bearing for the claimed contribution. The absence of code and seeds also needs to be addressed for a journal-level claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a plausible and well-engineered token-compression method that delivers consistent gains over its chosen baseline at the same token budget. The headline result—144 tokens per sub-image, +2.0 average over LLaVA-UHD—is believable, and the degradation at 32 tokens is small. The novelty is the specific combination: SSM-based global fusion, a local-to-single scan, and a dual-expert selection that blends class-token saliency with similarity to a learnable instruction token. Each part has precedent, but the integrated pipeline is new, and the ablations support the design choices (local-to-single beats single scan; visual-native beats visual-only).\n\nNow the soft spots, in proportion. The stress-test worry about the instruction token is only half right. It is true that the Top-K selection is discrete, so no gradient flows through the scores to the instruction token. But INS is inserted into the token sequence and processed by the SSM, so it absolutely receives gradient from the language-model loss through the fused token representations F^i. What it does not receive is direct pressure to make good selection decisions. That means the 'instruction-aware selection' claim is indirect: INS is trained to produce useful F^i features, and then the selection is a fixed cosine-similarity heuristic on those features. The +1.3 gain in Table 3 could come from the richer F^i representation rather than from genuinely instruction-relevant selection. The paper's language overstates what is actually trained. That is worth asking the authors to clarify, but it is not a fatal flaw—the method still works as a compression strategy.\n\nThe bigger practical gaps are the missing code/checkpoints (the Table 1 numbers are specific, but nothing is independently checkable) and the absence of direct comparisons with FocusLLaVA and LLaVA-Mini, two closely related methods they cite. The fixed lambda=0.8 is under-analyzed, but the authors do mention dataset-specific tuning. The efficiency table reports projector parameters and TPS but not end-to-end latency or memory; minor.\n\nWho is this for? People working on efficient vision-language models or token reduction. They will find the architecture clean and the results useful. It deserves a serious referee, but the reviewers should push on the selection training dynamics and demand the artifacts. If those are provided, I would take the conditional verdict to accept.","headline":"A credible efficiency-focused compression paper with consistent gains over LLaVA-UHD; the instruction-aware selection claim is indirect, but the method itself is solid and worth refereeing.","tokens_in":16388,"tokens_out":2182,"would_cite":true,"duration_ms":26373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLaVA-Meteor cuts visual tokens by up to 95% without losing accuracy","keywords":["visual instruction tuning","token compression","vision-language model","state space model","token selection","instruction-aware selection","high-resolution image","efficient multimodal LLM"],"falsifier":"Take the same model and compare the Top-K token sets selected on one image under two different questions; if the native expert carries instruction information, the selected sets should diverge toward regions relevant to each question. If the sets are nearly identical across questions, the instruction token is not instruction-aware and the gains come from the fusion module or extra parameters.","tokens_in":1614,"feed_emoji":"🖼️","tokens_out":5713,"duration_ms":83992,"temperature":0.7,"pith_summary":"This paper claims that a new projection design, LLaVA-Meteor, breaks the usual accuracy-versus-efficiency trade-off in visual instruction tuning by compressing visual tokens before they enter a large language model. It first runs a cheap global fusion over all tokens using a selective state space operator and a local-to-single scan, then selects a small set of the most important tokens based on both visual saliency and instruction-related cues. Across 12 benchmarks, LLaVA-Meteor matches or exceeds the LLaVA-UHD baseline while using 75–95% fewer tokens: at 144 tokens per sub-image it averages 62.4 versus 60.4, and even at 32 tokens it stays within about one point.","feed_headline":"LLaVA-Meteor cuts visual tokens by up to 95% without losing accuracy","feed_subtitle":"Fusing then selecting beats dense tokens while using 75–95 percent fewer.","key_machinery":"The load-bearing object is the Top-Down Compression paradigm, composed of Flash Global Fusion (FGF) and Visual-Native Selection (VNS). FGF uses a bidirectional selective state space operator with a local-to-single scanning strategy that summarizes each token's 3×3 spatial neighborhood before global propagation, and it inserts one learnable instruction token in the center of each sub-image's sequence to absorb instruction-related cues. VNS then computes a visual importance score from normalized class-token attention and a native importance score from softmax-normalized dot-product similarity to the instruction token, aggregates them as λ·visual + (1-λ)·native, and selects the Top-K tokens by that fused score.","core_discovery":"The central claim is that a two-stage Top-Down Compression – global fusion before local selection – lets a vision-language model keep essentially all the accuracy of a dense token input at a fraction of the compute. The fusion stage enriches every token with holistic context and distills instruction-relevant cues into a shared learnable instruction token; the selection stage then scores each token twice, once from the frozen vision encoder's class-token attention and once from similarity to the instruction token, and merges the scores with λ = 0.8. Because selection is applied to enriched tokens rather than raw features, the retained subset carries both global semantic context and task-aware emphasis, which the experiments show is enough to beat the uncompressed baseline at the same token budget and remain competitive at extreme compression.","pith_inferences":["The native expert's contribution likely depends on the diversity of instruction patterns in the training mixture: on counting and color questions the visual expert should dominate, while OCR and attribute questions should shift λ toward the native expert; this is a testable tuning axis the paper leaves implicit.","Because the instruction token never sees the explicit question text, its 'instruction preference' is really a learned prior over common user attention patterns; a natural extension is to condition the token on the actual instruction embedding to make selection genuinely query-dependent.","The local-to-single scan is a cheap spatial-locality plug-in that other token-pruning pipelines could adopt, and the paper itself notes the neighborhood window can grow to 5×5 or 7×7 on higher-resolution feature maps for multi-scale context.","If the native expert truly encodes instruction-relevant cues, the same fusion-then-select pipeline should transfer to video by sharing the instruction token across frames and selecting spatiotemporally important tokens."],"forward_implications":["At the same 144-token budget as LLaVA-UHD, LLaVA-Meteor improves the average across 12 benchmarks by 2.0 points, so token reduction need not cost accuracy.","Dropping to 64 tokens loses only about 1.4 points relative to the 144-token version, and 32 tokens stays within 1.1 points of LLaVA-UHD, enabling roughly 95% compression for resource-limited deployment.","The projector uses 37.08 million trainable parameters versus 137.84 million in the LLaVA-UHD projector, and throughput in tokens per second rises as the number of retained tokens falls.","The dual-expert selection helps most on reasoning and attribute-discrimination tasks, while the visual expert alone is nearly sufficient for dense prediction tasks.","Because the fusion and selection steps are separable, the same Top-Down Compression recipe could be dropped into other high-resolution vision-language frameworks that currently use heavier fusion or saliency-only pruning."],"supporting_citations":[{"why":"LLaVA-UHD supplies the high-resolution image partitioning, the training datasets, and the baseline whose compression layer is replaced.","marker":"[16]"},{"why":"The selective state space operator provides the linear-complexity global context propagation used inside Flash Global Fusion.","marker":"[14]"},{"why":"CLIP ViT-L/336px is the frozen vision encoder that produces the visual tokens and the class token used as the visual expert.","marker":"[53]"},{"why":"LLaVA-1.5 establishes the pretraining and instruction-tuning recipe that the model follows.","marker":"[40]"},{"why":"FocusLLaVA's dual-expert sampling is the closest prior attempt to combine visual and textual attention for token compression, motivating the native expert design.","marker":"[73]"},{"why":"VisionZip's class-token attention selection is the direct saliency-only baseline that the visual expert mirrors.","marker":"[66]"},{"why":"Vicuna-13B is the language model that receives the compressed tokens, so its input length determines the efficiency gain.","marker":"[8]"}],"fun_headline_variants":["LLaVA-Meteor: fuse then select—75-95% fewer tokens, same accuracy","Top-Down Compression: LLaVA-Meteor trims 95% of visual tokens without loss","LLaVA-Meteor: global fusion first, then pick—keeps accuracy, cuts compute","Fuse-then-select: LLaVA-Meteor achieves 95% token reduction with no accuracy drop","LLaVA-Meteor: top-down compression cuts visual tokens by 75-95%, accuracy holds"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The claim that instruction-aware selection improves accuracy rests on the assumption that the learnable instruction token, trained only through the vision-language loss and never given the explicit question, actually learns instruction-relevant cues rather than acting as a generic image summary.","fun_headline_variants_meta":{"raw":{"variants":["LLaVA-Meteor: fuse then select—75-95% fewer tokens, same accuracy","Top-Down Compression: LLaVA-Meteor trims 95% of visual tokens without loss","LLaVA-Meteor: global fusion first, then pick—keeps accuracy, cuts compute","Fuse-then-select: LLaVA-Meteor achieves 95% token reduction with no accuracy drop","LLaVA-Meteor: top-down compression cuts visual tokens by 75-95%, accuracy holds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3244,"prompt_tokens":894,"completion_tokens":2350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2220}},"tokens_in":510,"tokens_out":2350,"duration_ms":17847,"temperature":1.0,"reasoning_tokens":2220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:43:08.768856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same model and compare the Top-K token sets selected on one image under two different questions; if the native expert carries instruction information, the selected sets should diverge toward regions relevant to each question. If the sets are nearly identical across questions, the instruction token is not instruction-aware and the gains come from the fusion module or extra parameters.","supporting_citations":[{"cited_title":"Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images","cited_arxiv_id":null,"evidence_quote":"LLaVA-UHD supplies the high-resolution image partitioning, the training datasets, and the baseline whose compression layer is replaced."}],"review_version":1}