{"id":"e7274402-de37-41ab-a1af-cc3a35d2844f","arxiv_id":"2411.14228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FocusLLaVA compresses visual tokens to 39% using a vision-guided region sampler plus a text-guided attention sampler, beating its LLaVA-NeXT baseline on 9 of 10 benchmarks.","lead":"A new compression method for multimodal AI models keeps only 39% of visual tokens while preserving or slightly improving accuracy on most benchmarks. It uses two learned samplers: one drops low-information image regions, the other keeps regions that match the user's question.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison is the load-bearing weak point: small uncertified benchmark deltas and a single unverified speed measurement do not establish that the compression modules cause the improvement.","rationale":"The reader correctly identifies the baseline/no-error-bars issue as the weakest assumption. Our stress-test reinforces it and adds the unverified efficiency measurement. However, the method is well-motivated, ablations show both modules contribute, and the concerns are fixable by releasing code/data and running seeded repeats; therefore the conditional verdict stands. The missing data-config table and the train/test mismatch in Eq. (2) (scaling by the Top1 probability) are additional but secondary; the decisive test is whether the small deltas survive variance analysis.","tokens_in":16221,"tokens_out":8397,"duration_ms":80070,"concrete_test":"Run FocusLLaVA and the baseline with 5 random seeds (or at least 3) on GQA, TextVQA, ScienceQA, and MME-P, using identical hyperparameters and reporting mean and standard deviation, and report the speed benchmark with a fixed batch size of 1, a fixed input resolution distribution, and a fixed maximum generation length. If the mean delta between FocusLLaVA and baseline is smaller than one standard deviation on more than two of the four benchmarks, the claim that the compression modules improve performance is not supported by this evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that removing visual redundancy simultaneously improves efficiency and performance. The performance side rests entirely on Table 1's comparison between FocusLLaVA and the authors' self-implemented LLaVA-NeXT baseline. Most deltas are small (GQA +0.3, MMBench-CN +0.2, MME-C −6) and no error bars or multiple seeds are reported; for instruction-tuned 8B LLMs these differences are within typical run-to-run variance. Hyperparameters such as the balance-loss weight (Table 5), the text-guided threshold γ, and the placement layer are selected on the same evaluation benchmarks, so the small observed advantages could be tuning artifacts rather than evidence that compression improves accuracy. The baseline itself is trained on a substituted data mixture (ALLaVA-Instruct-VFLAN-4V in place of the unreleased LLaVA-NeXT data), and the promised data-config table is absent, so the claimed 'strict alignment' is not independently verifiable. The efficiency side is also thinly supported: one speed number (2.85 vs. 4.01 samples/s) with no batch size, input resolution, or generation-length details, and the '39% visual tokens' figure is an average over layers that does not by itself imply the measured latency ratio. If the benchmark gaps are within noise, the paper's dual-improvement claim reduces to a modest, possibly non-real speedup at matched performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FocusLLaVA, a visual-token compression method for high-resolution multimodal LLMs built on LLaVA-NeXT. It combines a vision-guided sampler, which performs region-level multi-scale downsampling before the LLM, with a text-guided sampler, which prunes visual tokens inside an intermediate LLM layer based on attention relevance to the instruction. The authors report that keeping about 39% of visual tokens improves the model over their LLaVA-NeXT baseline on 9 of 10 benchmarks and raises throughput from 2.85 to 4.01 samples per second. The paper includes implementation details, ablations of the two modules, sensitivity studies on window size, balance-loss weight, threshold, and layer placement, and qualitative analyses of selected regions.","tokens_in":1659,"tokens_out":1875,"duration_ms":68541,"significance":"If the empirical claims withstand scrutiny, the paper makes a useful contribution to efficient multimodal inference: it addresses visual redundancy with a learned coarse-to-fine scheme rather than a hand-crafted heuristic, and the disentangled placement of the two samplers is clearly motivated. The ablations are informative and the qualitative visualizations support the intuition that the vision-guided sampler focuses on information-dense regions while the text-guided sampler focuses on instruction-relevant regions. The paper also compares against a heuristic token-dropping baseline in the supplementary, which helps position the learned selector. However, the headline result depends on small benchmark deltas against a self-implemented baseline, with no error bars, no released code or checkpoints, and no full speed-measurement protocol, so the reproducibility and statistical strength of the central claim are not yet established.","major_comments":[{"comment":"The central claim that removing visual redundancy improves both efficiency and accuracy rests on Table 1, where FocusLLaVA exceeds the self-implemented LLaVA-NeXT baseline by margins such as +0.3 on GQA, +0.2 on MMBench-CN, +0.4 on ScienceQA, and -6 on MME-Cognition. For instruction-tuned 8B MLLMs these differences are within typical run-to-run variance, yet no repeated seeds, error bars, or significance tests are reported. Please provide multiple runs with mean and standard deviation, or paired comparisons where possible; if that is infeasible, the performance claim should be weakened to matched performance at reduced token count and higher throughput.","section":"Section 4.3, Table 1"},{"comment":"The 'strict alignment' between FocusLLaVA and its baseline is not independently verifiable. The fine-tuning data replaces the unreleased LLaVA-NeXT mixture with ALLaVA-Instruct-VFLAN-4V and other open-source sets, and Section 4.2 states that the specific data configuration is available in the supplementary, but the supplementary contains no such table. In addition, the speed comparison (2.85 vs 4.01 samples/s) is reported without input resolution, batch size, generation length, decoding method, hardware details, number of runs, or whether the measurement includes prefill and decoding phases. Please release the exact data recipe, code and model checkpoints, and a complete speed-measurement protocol so that the efficiency claim can be reproduced.","section":"Sections 4.2 and 4.3, Table 2"},{"comment":"Several key hyperparameters are selected by varying them on the same benchmark suite that is later used for the headline results in Table 1: the window size w (Table 4), balance-loss weight alpha (Table 5), text-guided threshold and random-layer schedule (Table 6), and the insertion layer (Figure 2). Because the reported improvements over the baseline are small, the observed advantages could be artifacts of tuning on the evaluation benchmarks rather than evidence of a general property of the compression modules. Please state whether any held-out split or separate validation set was used for hyperparameter selection, or show that the conclusions are stable across a range of settings.","section":"Section 4.4, Tables 2-6 and Figure 2"},{"comment":"The training-time formula for the vision-guided sampler multiplies the selected downsampled token by Top1(Softmax(Z)), while inference uses the raw selected token without this probability scaling. This train/test mismatch is not discussed. Additionally, because the value passed forward is DS[argmax(Z)](Xr) scaled by a scalar probability, gradients reach the selector only through the probability of the chosen scale and never through the unselected downsampled features. Please clarify whether this is intended as a straight-through estimator, and report whether using hard selection with the balance loss alone, or removing the probability multiplication, changes the results.","section":"Section 3.2, Eq. (2)"}],"minor_comments":[{"comment":"The axis labels in Figure 2 are garbled: the x-axis appears to be labeled with both benchmark score values and samples/sec values, making the accuracy-versus-speed trade-off impossible to read. Please split the two curves into separate panels with clear axis labels.","section":"Section 4.4, Figure 2"},{"comment":"The formula m - n + i*n/32 for converting remaining visual tokens should define m, n, and i explicitly, and should state that this is an average over the 32 LLM layers rather than the instantaneous token count.","section":"Section 4.4, Table 2 caption"},{"comment":"The summation limits are typeset inconsistently (for example, 'nX0' and 'N-1X0'), and the indicator function in Eq. (4) should be written with standard indicator notation; please also clarify the relation between n in Eq. (3) and the number of scales S.","section":"Section 3.2, Eqs. (3)-(5)"},{"comment":"There are minor language issues: 'harmfull' should be 'harmful', '31th' should be '31st', and the sentence describing the balance loss as ensuring 'each expert is selected uniformly' should be qualified, since the auxiliary loss encourages balanced fractional usage rather than exact uniform selection per sample.","section":"Section 4.4"},{"comment":"Reference [4] has a malformed author string ('Sagnak Ta, and sırlar'); the author list should be corrected. Also, references [11] and [30] appear to point to the same Open-LLaVA-NeXT repository with different numbering and should be unified or disambiguated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the architectural ideas are reasonable, but the headline empirical claim is not yet supported with statistical rigor or full reproducibility artifacts. The small benchmark deltas, the unverified speed measurement, and the use of the same benchmarks for hyperparameter selection are the main barriers. I would encourage the authors to add variance-aware evaluation, release the data configuration and code, and provide a complete speed protocol. If they cannot substantiate the accuracy improvement beyond noise, the claim should be reframed as matched accuracy with substantial token reduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new coarse-to-fine token compression design, and the authors do the right internal comparison. But the accuracy gains over their LLaVA-NeXT baseline are small and reported without error bars, so the 'compression helps accuracy' claim is plausible, not yet nailed down.\n\nWhat's new: the vision-guided sampler does region-level multi-scale max-pooling with an MoE-style learned scale selector, trained with a balance loss; the text-guided sampler uses attention scores inside the LLM to drop tokens, with stochastic layer/threshold training to keep the model robust. That combination is not in the earlier work I know. The paper also does a sensible analysis of where to put the text-guided sampler, and the ablations are reasonably thorough: window size, scale count, balance loss weight, textual guidance variants, plus a heuristic-dropping comparison in the supplementary.\n\nThe soft spots are real but fixable. First, the deltas in Table 1 are small (GQA +0.3, MMBench-CN +0.2) and there are no multiple seeds or error bars. For instruction-tuned 8B models, those could be run-to-run noise. That said, the ablations in Table 2 show each module helps, and removing the balance loss tanks the model, so the modules are not inert. Second, hyperparameters like the balance loss weight, window size, and gamma are selected on the same benchmarks used in the main table, which inflates the apparent gains. A held-out benchmark would help. Third, the speed figure (2.85 vs 4.01 samples/s) is one measurement with no batch size or generation-length details. It's consistent with the token reduction, but I'd like to see a proper latency curve. Fourth, the baseline uses a substituted data mixture; the internal comparison is fair, but the data config table promised in the paper is missing from what I saw.\n\nWho gets value: anyone working on efficient MLLMs or visual token compression. The method is clean and the write-up is honest about the data substitution. I'd send this to a serious referee. It's an accept-shaped paper, but the authors need to add uncertainty quantification, release the data config (and ideally code), and show at least one benchmark that was not used in ablations.","headline":"A well-motivated coarse-to-fine compression method with a fair internal comparison, but the accuracy gains are small and uncertified, so the 'compression improves accuracy' claim needs more evidence before I'd trust it.","tokens_in":17059,"tokens_out":3821,"would_cite":true,"duration_ms":36365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that removing visual redundancy via a coarse-to-fine, vision-then-text guided token sampler makes a high-resolution multimodal LLM both faster and more accurate.","keywords":["visual token compression","multimodal large language models","high-resolution image understanding","vision-guided sampling","text-guided attention selection","coarse-to-fine compression","efficient inference","instruction-aware token pruning"],"falsifier":"Run FocusLLaVA and its exact baseline several times with different random seeds on the same 1M-sample training data; if the average accuracy gap on GQA, MME, ScienceQA, and TextVQA falls inside the run-to-run variance, then the claim that compression improves performance is not confirmed.","tokens_in":16026,"feed_emoji":"⚡","tokens_out":6102,"duration_ms":51997,"temperature":0.7,"pith_summary":"FocusLLaVA argues that the visual tokens produced from high-resolution images contain a large amount of redundancy, and that removing this redundancy with learned, instruction-aware selection can make a multimodal language model faster and more accurate at the same time. The method compresses in two stages: a vision-guided sampler first chooses a downsampling scale for each local image region based only on image content, and a text-guided sampler later keeps the visual tokens that the user's question actually attends to. With about 39 percent of the original visual tokens, FocusLLaVA outperforms its full-token LLaVA-NeXT baseline on 9 of 10 benchmarks while increasing throughput from 2.85 to 4.01 samples per second. If the claim holds, it means high-resolution visual understanding does not have to be bought at the price of quadratic token growth.","feed_headline":"Cutting visual tokens to 39% also raises accuracy on most benchmarks","feed_subtitle":"Two learned samplers drop redundant regions and keep instruction-relevant tokens; high-res models get faster and sharper.","key_machinery":"The two load-bearing components are the vision-guided sampler and the text-guided sampler. The vision-guided sampler partitions the global visual feature map into 4x4 local blocks, max-pools each block at three scales (4x4, 2x2, 1x1), and uses a learned selector that scores each block against the global image features to choose one scale per region; a switch-transformer-style balance loss prevents the selector from always choosing the same branch. The text-guided sampler, inserted at layer 8 of the LLM, computes per-token importance $S_i$ as the average over text tokens of the max over attention heads of the text-to-visual attention scores, then keeps the smallest number of top tokens whose cumulative normalized importance exceeds $\\gamma$; during training it randomly samples a layer and a $\\gamma$ value to make the model robust to token dropping. The coarse-to-fine structure matters because visual-only compression is applied early in the projector, where it is cheap, while text-guided selection is deferred to a middle LLM layer where the language signal has become reliable.","core_discovery":"The paper's central claim is that removing visual redundancy can improve both efficiency and performance, contradicting the common trade-off view. Concretely, the authors show that a coarse-to-fine pipeline—vision-guided region-level scale selection in the projector followed by text-guided, attention-based token selection inside the LLM—keeps only about 39 percent of the visual tokens and still raises scores on nearly all evaluated benchmarks relative to an identically trained baseline. The vision-guided sampler treats each 4x4 local feature block as a unit and selects among 1x1, 2x2, and 4x4 max-pooled representations using a learned correlation with the global image; the text-guided sampler computes an importance score from multi-head attention between text and visual tokens and drops tokens whose cumulative normalized importance falls below a threshold $\\gamma$. The authors interpret the result as evidence that low-information regions and instruction-irrelevant tokens are not needed for correct answers, and that a model trained with stochastic token dropping can learn to answer without them.","pith_inferences":["If the two-stage separation holds up, visual redundancy and instruction relevance are partly independent axes, so each sampler could be tuned or retrained separately without retraining the whole model.","The text-guided sampler's dependence on the question suggests the same architecture could serve as a dynamic per-query token budget, spending more tokens when the question demands fine detail and fewer for global questions.","A natural testable extension is to apply the same coarse-to-fine recipe to video or document inputs, where temporal and layout redundancy is even larger than in single images.","The reliance on attention-map importance implies the method's ceiling is tied to how well the LLM's self-attention locates answers; if attention is noisy or hallucinated, the selected tokens may be wrong even though they are few."],"forward_implications":["High-resolution multimodal models can be made faster without a performance penalty if redundancy is removed both before and inside the LLM.","A learned per-region downsampling choice beats handcrafted token-importance metrics: replacing the selector with a fixed similarity heuristic drops scores as token budgets shrink.","The balance loss is necessary: without it the scale selector collapses onto one branch and performance falls well below baseline.","Stochastic training with a random layer and random threshold converts text-guided token dropping from a performance loss into a small gain on several benchmarks.","More downsampling scales help fine-grained reading tasks such as TextVQA, suggesting the framework benefits from richer scale choices."],"supporting_citations":[{"why":"Supplies the LLaVA-NeXT-style baseline architecture and training recipe that FocusLLaVA modifies and compares against.","marker":"[30]"},{"why":"Defines the original LLaVA-NeXT high-resolution adaptive-slicing baseline whose performance FocusLLaVA is measured against.","marker":"[35]"},{"why":"Provides the switch-transformer balance loss used to keep the vision-guided scale selector from collapsing onto one branch.","marker":"[16]"},{"why":"Supplies the CLIP ViT-L/14 visual encoder used to produce the visual tokens.","marker":"[42]"},{"why":"A token-reduction baseline using merging rather than learned selection; the comparison shows the advantage of the proposed learned approach.","marker":"[43]"},{"why":"A heuristic attention-guided token-dropping baseline representing the handcrafted approach FocusLLaVA argues against.","marker":"[2]"},{"why":"Provides the ALLaVA instruction data used as a substitute for the unreleased real user data of the original LLaVA-NeXT training set.","marker":"[8]"}],"fun_headline_variants":["Dropping 61% of visual tokens boosts LLaVA accuracy","Two-stage pruning cuts tokens to 39% and improves accuracy","Coarse-to-fine drop: 39% tokens, better results","Less visual data, more accuracy: token pruning works","Visual token pruning: retain 39%, beat the trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the self-implemented LLaVA-NeXT baseline is identical to FocusLLaVA except for the compression modules, so the small benchmark differences (for example +0.3 on GQA and +0.2 on MMBench-CN) are caused by those modules rather than by training noise or implementation details, and the paper reports a single run without multiple seeds.","fun_headline_variants_meta":{"raw":{"variants":["Dropping 61% of visual tokens boosts LLaVA accuracy","Two-stage pruning cuts tokens to 39% and improves accuracy","Coarse-to-fine drop: 39% tokens, better results","Less visual data, more accuracy: token pruning works","Visual token pruning: retain 39%, beat the trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":1997,"prompt_tokens":895,"completion_tokens":1102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1014}},"tokens_in":511,"tokens_out":1102,"duration_ms":10423,"temperature":1.0,"reasoning_tokens":1014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:23:58.205859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FocusLLaVA and its exact baseline several times with different random seeds on the same 1M-sample training data; if the average accuracy gap on GQA, MME, ScienceQA, and TextVQA falls inside the run-to-run variance, then the claim that compression improves performance is not confirmed.","supporting_citations":[{"cited_title":"Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-NeXT-style baseline architecture and training recipe that FocusLLaVA modifies and compares against."},{"cited_title":"Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity","cited_arxiv_id":null,"evidence_quote":"Provides the switch-transformer balance loss used to keep the vision-guided scale selector from collapsing onto one branch."},{"cited_title":"Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model,","cited_arxiv_id":null,"evidence_quote":"Provides the ALLaVA instruction data used as a substitute for the unreleased real user data of the original LLaVA-NeXT training set."}],"review_version":1}