{"id":"dadbc8fd-f216-4976-8787-967a6e4d2c60","arxiv_id":"2412.04317","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FlashSloth compresses visual input to 90 tokens with attention pooling and an embedded cross-attention query module, achieving 2-5x faster response than tiny MLLM baselines with competitive benchmark scores.","lead":"FlashSloth is a compact AI model that turns an image into only 90 small pieces of information before answering questions about it, instead of hundreds or thousands. If the reported numbers hold, it responds two to five times faster than similarly sized models while staying close to them in accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's TFLOPs for FlashSloth are about 10x too low under standard prefill accounting, so the headline 70–98% inference-computation reduction is not established by the paper's own numbers.","rationale":"The reader's weakest assumption was that Table 1 lacks a stated measurement protocol, making the speedup hard to attribute to the architecture. I agree with that concern, but I found a more specific and more damaging internal problem: the TFLOPs reported for FlashSloth are not consistent with standard prefill-cost accounting, while the Qwen2-VL and FlashSloth-HD rows roughly are. This is not an attack on the authors' honesty; it is a quantitative discrepancy that can be checked from the architecture alone. If the TFLOPs are recomputed on a consistent basis, the 70–98% inference-computation reduction may shrink substantially, and the headline '2–5x response time' claim should also be re-examined under identical hardware and decoding conditions. That said, the qualitative direction of the result is plausible: compressing visual tokens from ~400–1700 to 90 should reduce prefill cost and latency. The paper's architecture and accuracy results could still be valuable after the efficiency accounting is corrected. I therefore keep the reader's CONDITIONAL verdict but with a sharper condition: the authors must either reproduce Table 1 under a consistent, documented FLOPs/latency protocol or weaken the efficiency claims to what the architecture can actually support. The proposed concrete test is a re-derivation of prefill FLOPs followed by profiling; it would settle whether the discrepancy is a labeling/accounting artifact or a real error in the central efficiency claim.","tokens_in":24774,"tokens_out":8587,"duration_ms":88168,"concrete_test":"Recompute first-round FLOPs for FlashSloth and FlashSloth-HD from the released checkpoints/configs using one consistent formula: prefill FLOPs = 2 * N_params * total_tokens for the LLM plus the vision encoder, using actual token counts per benchmark (e.g., GQA: 81 or 414 SAP tokens + 9 query tokens + prompt tokens), and compare with Table 1. Independently profile with torch.profiler on the same A800, batch=1, FP16, identical max_new_tokens and decoding settings for all models. If FlashSloth's recomputed prefill FLOPs exceed 0.5 TFLOPs (vs the reported 0.09), then Table 1 and the abstract's reduction percentages should be revised; if 0.09 is reproduced, the paper must state the exact accounting basis so that the comparison is apples-to-apples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim in Sec. 4.3.1/Table 1 is that FlashSloth cuts first-round inference TFLOPs to 0.08–0.10, versus 4.4 for LLaVA-1.5-7B. This number is internally implausible. FlashSloth uses a 3.2B total model (Phi-2 2.7B + SigLIP) with 81 SAP tokens, 9 query tokens, and a text prompt. A standard prefill FLOPs estimate, roughly 2 * N_params * total_tokens, gives about 0.6 TFLOPs for the LLM alone with ~110–150 tokens, and roughly 1.2 TFLOPs if the vision encoder is included. The same formula applied to Qwen2-VL rows (e.g., 977 tokens → 4.14 TFLOPs) matches Table 1, but the FlashSloth rows do not. FlashSloth-HD reports 2.71 TFLOPs for 414 tokens, which is consistent with 2 * N * T; FlashSloth reporting 0.09 for 90 tokens implies a factor-of-30 TFLOP drop for a 4.6x token reduction, which is not possible in a linear prefill-cost model. The 0.09 value appears consistent with counting only the SAP/EmbQ compression modules or only a tiny subset of tokens, not the full first-round inference. Because the paper's contribution is efficiency and Table 1 is the sole quantitative support for '70–98% inference computation reduction' and part of the '2–5x response time' claim, this inconsistency is load-bearing: the architecture may still be faster, but the headline magnitude is not supported by the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FlashSloth introduces a 3.2B multimodal LLM that compresses visual tokens with a spatial attention pooling (SAP) module and an embedded query (EmbQ) module. SAP reduces the token count by attention-weighted pooling over spatial regions, while EmbQ uses 9 learnable query tokens to cross-attend to the uncompressed visual tokens at an interior LLM layer, supplying instruction-related visual information. The default model uses Phi-2 2.7B and SigLIP, with 81 visual tokens plus 9 query tokens; a high-resolution variant, FlashSloth-HD, uses 414 tokens. The paper reports accuracy on 14 benchmarks and efficiency comparisons (visual token count, TFLOPs, GPU memory, response time, throughput) against several tiny MLLMs. It claims reductions of 80–89% in visual tokens, 61–80% in training memory, and 70–98% in inference computation, with roughly 2×–5× faster response time while remaining competitive. The central efficiency claim rests on Table 1, and the review focuses on whether the numbers in that table support it.","tokens_in":25074,"tokens_out":8308,"duration_ms":77406,"significance":"The architecture is well motivated and clearly described, and the ablation study in Tables 3–6 is systematic, including comparisons with average pooling, pixel shuffle, and LDP-based compression. The training budget is small (3.7M samples), the code is released, and the accuracy results suggest that a 3.2B MLLM with 90 visual tokens can be competitive with 2–3B MLLMs on many benchmarks. If the efficiency numbers were verified under controlled conditions, the contribution would be meaningful: it would demonstrate that embedded visual compression can preserve accuracy while drastically shortening the visual token sequence. However, the paper's primary claim is about efficiency, and that claim currently rests on Table 1, whose TFLOPs are not reproducible under standard prefill accounting and whose measurement conditions are not stated. The accuracy side is generally sound, but the headline efficiency contribution needs a major correction or clarification.","major_comments":[{"comment":"The reported first-round TFLOPs for FlashSloth are not reproducible under standard prefill accounting. For a 3.2B-parameter model (Phi-2 ~2.7B plus SigLIP ~0.4B) and a sequence of roughly 110–150 tokens, the usual 2·N·T estimate gives about 0.6 TFLOPs for the LLM alone and about 1.2 TFLOPs if the vision encoder is included; the same estimate applied to the Qwen2-VL rows is consistent with the table, but the FlashSloth rows (0.08–0.10 TFLOPs) are about an order of magnitude too low. FlashSloth-HD reports 2.71 TFLOPs for 414 tokens, so the 30-fold drop in TFLOPs for a 4.6-fold token reduction is internally inconsistent. Table 3's 729-token baseline at 0.30 TFLOPs is similarly below the ~4 TFLOPs expected for that sequence length. Because the abstract and Sec. 4.3.1 use these numbers for the headline '70–98% inference computation reduction', the central efficiency claim is not established. Please state the exact FLOPs counting formula and which components (vision encoder, attention, prefill vs. decode) are included, and re-derive all affected percentages.","section":"Sec. 4.3.1, Table 1 and Table 3"},{"comment":"The response-time, throughput, and GPU-memory comparisons omit the measurement conditions: GPU model, batch size, numerical precision, maximum generated token length, decoding configuration, and whether numbers are medians over repeated runs. These settings can dominate the reported 2–5× speedups, especially because the benefit of token reduction is concentrated in prefill. Please report a controlled comparison with identical settings across all rows, and separate time-to-first-token from decode throughput.","section":"Sec. 4.3.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'FlashSloth can even achieve new SOTA performance among tiny MLLMs on several benchmarks, such as MMB, GQA and AI2D' is not supported by the table: Qwen2-VL-2B scores 74.9 on MMB, InternVL2 scores 61.6 on GQA and 74.1 on AI2D, all above FlashSloth's 73.0, 61.1, and 72.5.","section":"Sec. 4.3.2, Table 2"},{"comment":"Benchmark scores are reported as single point estimates without confidence intervals or repeated-evaluation statistics; given the small margins in some comparisons, please include variance information or state that the differences are within evaluation noise.","section":"Tables 2–6"},{"comment":"The text says the framework is illustrated in Fig. 1, but Fig. 1 is the motivation/efficiency plot and the framework is in Fig. 2; the cross-reference should be corrected.","section":"Sec. 3.1, Fig. 2"},{"comment":"The claimed reduction ranges (80–89% visual tokens, 61–80% training memory, 70–98% inference computation) do not match the per-model percentages in Table 1; for example, 90 tokens versus InternVL2's 1561 tokens is a 94% token reduction, which is outside the stated range. Please specify the reference model for each range.","section":"Abstract and Sec. 4.3.1"}],"recommendation":"major_revision","confidential_remarks":"The architecture and accuracy results are interesting, but the paper's main selling point is efficiency, and Table 1 currently contains internally inconsistent TFLOPs figures and omits the measurement protocol. I would encourage the authors to redo the efficiency comparison with a clear, controlled setup and to recompute all FLOP-based percentages. If the corrected numbers still show a substantial speedup, the contribution is publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The embedded visual compression idea is real: spatial attention pooling plus an embedded query module, trained without dedicated VL alignment, and the ablations are the most solid part of the paper. But the efficiency table has a numerical problem that undercuts the main headline. FlashSloth reports 0.08–0.10 TFLOPs for 90 tokens, which is roughly an order of magnitude below a standard prefill estimate for a 3.2B model, and it sits oddly next to the paper's own FlashSloth-HD numbers (2.71 TFLOPs for 414 tokens). The FlashSloth rows appear to be counting only part of the computation, not the full first-round inference. The same 'too low' pattern shows up for IMP, so the TFLOPs column is not consistently defined across models. Without a definition and a measurement protocol (hardware, batch size, precision, decoding settings), the claimed 70–98% inference-computation reduction is not established. The response-time and throughput numbers are also not reproducible from the text.\n\nWhat is genuinely new: the specific combination of SAP and EmbQ, with evidence that the query module adds instruction-related detail beyond what attention pooling retains. The ablations cover pooling type, query count, initialization, insertion layer, and fusion method, and they are systematic. On 14 benchmarks, a 90-token model is competitive with 2–4B baselines that use hundreds or thousands of tokens; that result is plausible and useful even if the exact speedups are not.\n\nSoft spots: no error bars in Table 2, sub-1-point differences used for 'best' claims, and no artifacts to verify the measurements. The GitHub link is given, but nothing in the paper lets a reader check the protocol. None of this is fatal to the architecture; the design could still be effective. But the paper's framing makes efficiency the load-bearing contribution, and that part is not yet credible. I'd send this to review, but the authors need to fix the FLOPs accounting, define the measurement setup, and ideally release code and evaluation scripts. The compression design and benchmark results are enough to justify referee time.","headline":"Plausible compression design with systematic ablations, but the TFLOPs accounting in Table 1 is inconsistent and the headline efficiency claims need verification.","tokens_in":25681,"tokens_out":4954,"would_cite":false,"duration_ms":44533,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims FlashSloth, a 3.2B multimodal model, can cut visual tokens by 80–89%, training memory by 61–80%, and inference computation by 70–98% while shortening response time by 2–5x and staying competitive with advanced tiny MLLMs.","keywords":["FlashSloth","multimodal large language models","visual token compression","spatial attention pooling","embedded visual query","efficient inference","tiny MLLMs"],"falsifier":"Run FlashSloth, FlashSloth-HD, InternVL2-2B, and Qwen2-VL-2B on the same GPU with matched batch size, input resolution, precision, prompts, and maximum generated tokens; if the time-to-first-token gap against InternVL2 does not remain near 2–5x, the architecture-specific efficiency claim fails.","tokens_in":24515,"feed_emoji":"⚡","tokens_out":7151,"duration_ms":68132,"temperature":0.7,"pith_summary":"FlashSloth claims that the main obstacle to fast tiny multimodal LLMs is not the small language model but the excessive number of visual tokens, and that token compression can be made safe by improving the descriptive power of the tokens that survive. To test this, the paper builds a 3.2B model that squeezes 729 raw visual tokens into 81 salient tokens via spatial attention pooling, then adds just 9 learnable query tokens that pull instruction-relevant image details through an embedded cross-attention module inside the LLM. Against advanced tiny MLLMs such as InternVL2-2B and Qwen2-VL-2B, FlashSloth reports 80–89% fewer visual tokens, 61–80% lower training memory, 70–98% lower inference computation, and 2–5x faster response time while remaining competitive on 14 vision-language benchmarks. If the efficiency measurements hold up, the result would make high-accuracy multimodal assistants practical on edge hardware without sacrificing much capability. A higher-resolution variant, FlashSloth-HD, is also presented to recover fine-grained OCR performance while still beating the baselines' token counts.","feed_headline":"A 3.2B model runs 2–5x faster with 84% fewer visual tokens","feed_subtitle":"Embedded visual compression keeps a 3.2B model competitive with InternVL2 and Qwen2-VL on most benchmarks.","key_machinery":"The load-bearing mechanism is a two-part compression pipeline embedded inside the MLLM. Spatial Attention Pooling (SAP) divides the visual token grid into $s \\times s$ regions, predicts a softmax weight $\\alpha_i = \\mathrm{Softmax}(\\mathrm{mlp}(F^i_v))$ for each token in a region, and produces one salient token per region via the weighted sum $f^s_v = \\sum_i \\alpha_i f^i_v$, cutting 729 tokens to 81. The Embedded Query Module (EmbQ) pads 9 learnable query tokens into the LLM input; after a few transformer layers, those queries first cross-attend to text tokens to become instruction-conditioned, then cross-attend to the uncompressed visual tokens, and the resulting features are added back into the query token stream. This lets the model recover task-relevant image details that saliency-only pooling loses, without introducing a separate language model or a dedicated vision-language alignment stage.","core_discovery":"The central discovery claim is that attention-based pooling combined with an embedded instruction-aware query module can replace a large visual-token sequence almost losslessly. On the paper's terms, visually salient semantics and instruction-related semantics are complementary: SAP captures what stands out in each image region, EmbQ captures what the question cares about, and together they let a 3.2B model use only 90 visual tokens while matching or exceeding advanced tiny MLLMs on many benchmarks. The paper calls this design embedded visual compression and contrasts it with Q-Former-style bridges that require another language model and dedicated alignment pretraining. With the high-resolution FlashSloth-HD variant, the same compression also narrows the gap on OCR-heavy tasks such as DocVQA and ChartQA.","pith_inferences":["The reported speedups likely act mainly on prefill and first-token latency; benchmarks with long generated answers would probably show a smaller relative response-time gap, since token-count savings matter less once decoding is dominated by autoregressive generation. ","Combining architectural compression like EmbQ with runtime token pruning could compound the savings, because the paper's compression works at the network-structure level while pruning methods are orthogonal inference-time additions. ","The saliency weights produced by SAP could double as cheap interpretability maps for debugging hallucinations or grounding failures, a use the paper does not explore. ","The near-optimality of 9 query tokens suggests coarse instruction grounding, not fine alignment, is what these benchmarks reward; tasks requiring dense spatial detail may need more queries or a resolution-aware query allocation. "],"forward_implications":["A 3.2B model with 90 visual tokens can match or exceed advanced tiny MLLMs on MMB, GQA, SQA, and AI2D while using far fewer tokens. ","Average time-to-first-token drops to 0.05 seconds, enabling interactive and mobile deployments where response latency is the main constraint. ","Training with the LLaVA-665k split costs only 6.4 GPU-hours for pretraining, substantially lowering the barrier to custom tiny MLLMs. ","FlashSloth-HD recovers most of the OCR and document-understanding gap while still using fewer visual tokens than Qwen2-VL and InternVL2. ","EmbQ is reported as a reusable module: adding it to average pooling, pixel shuffle, or LDP compression consistently improves their benchmark scores. "],"supporting_citations":[{"why":"Supplies the LLaVA-1.5 training splits and the 7B reference baseline used for efficiency and performance comparisons.","marker":"[33]"},{"why":"Supplies the InternVL2 baseline whose pixel-shuffle compression is compared and whose efficiency and accuracy FlashSloth is measured against.","marker":"[10]"},{"why":"Supplies the Qwen2-VL baseline, the strongest accuracy competitor on OCR-heavy tasks.","marker":"[58]"},{"why":"Provides the Q-Former query-based compression paradigm that EmbQ replaces without a second language model or alignment pretraining.","marker":"[28]"},{"why":"Supplies the SigLIP vision encoder whose raw visual tokens are compressed by SAP.","marker":"[67]"},{"why":"Supplies the Phi-2 2.7B LLM backbone that makes FlashSloth a tiny model.","marker":"[20]"},{"why":"Supplies the high-resolution image tiling strategy used by FlashSloth-HD.","marker":"[34]"},{"why":"Supplies the Imp tiny MLLM baseline used in efficiency and performance comparisons.","marker":"[51]"},{"why":"Supplies the MobileVLM-V2 baseline with LDP compression, which EmbQ is shown to improve.","marker":"[12]"}],"fun_headline_variants":["FlashSloth: 3.2B model uses 90 tokens, runs 2-5x faster","Tiny MLLM cuts visual tokens 84%, holds its own on benchmarks","Embedded visual compression lets 3.2B model beat bigger ones","FlashSloth: fast tiny MLLM via instruction-aware visual pooling","90 visual tokens enough? FlashSloth says yes, runs faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline efficiency gains rest on the assumption that every model in Table 1 was measured under identical inference conditions—same hardware, batch size, precision, and decoding length—which the paper does not report.","fun_headline_variants_meta":{"raw":{"variants":["FlashSloth: 3.2B model uses 90 tokens, runs 2-5x faster","Tiny MLLM cuts visual tokens 84%, holds its own on benchmarks","Embedded visual compression lets 3.2B model beat bigger ones","FlashSloth: fast tiny MLLM via instruction-aware visual pooling","90 visual tokens enough? FlashSloth says yes, runs faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1602,"prompt_tokens":913,"completion_tokens":689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":529,"tokens_out":689,"duration_ms":6230,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:32:00.354107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FlashSloth, FlashSloth-HD, InternVL2-2B, and Qwen2-VL-2B on the same GPU with matched batch size, input resolution, precision, prompts, and maximum generated tokens; if the time-to-first-token gap against InternVL2 does not remain near 2–5x, the architecture-specific efficiency claim fails.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-1.5 training splits and the 7B reference baseline used for efficiency and performance comparisons."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Provides the Q-Former query-based compression paradigm that EmbQ replaces without a second language model or alignment pretraining."},{"cited_title":"Phi-2: The surprising power of small language models","cited_arxiv_id":null,"evidence_quote":"Supplies the Phi-2 2.7B LLM backbone that makes FlashSloth a tiny model."},{"cited_title":"Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the high-resolution image tiling strategy used by FlashSloth-HD."}],"review_version":1}