{"id":"3db36f96-aa5d-4240-a9bb-c598e898ce88","arxiv_id":"2501.16297","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FALCON embeds learned register tokens in a vision transformer to compress high-resolution visual representations and enable cross-crop interaction, achieving strong benchmark scores with 9 times fewer visual tokens.","lead":"FALCON is a new multimodal AI model that cuts the number of visual tokens sent to a language model by about 9 times while keeping strong performance on high-resolution image understanding benchmarks. It works by adding a small set of learned register tokens inside the vision encoder, which absorb key visual information and allow different crops of an image to communicate during encoding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 6 shows accuracy still rises from 64 to 144 registers, contradicting the claim that 64 tokens give 'comprehensive' visual info; since the LLM sees only registers, this lossy 9x compression is a tradeoff, not redundancy-free.","rationale":"After reading the paper in good faith, the central claim is that a tiny set of visual registers placed inside a pretrained ViT can serve as the sole visual representation without loss. The most load-bearing assumption is that 64 registers per sub-image aggregate all task-relevant information, because the LLM never accesses the original image tokens. The paper's own Fig. 6—the register-count ablation—shows that increasing registers from 36 to 64 to 144 yields monotonic gains on MME-RealWorld and POPE. This is direct evidence that 64 registers are not a sufficient statistic; additional registers recover useful information. The paper acknowledges the taper after 64 but still frames 64 as achieving 'comprehensive visual information' and claims SOTA in Sec. 4.1, which Table 1 contradicts (LLaVA-OneVision is higher on Avg and Avg-C). Therefore the mechanism is best described as a lossy compression with an efficiency-accuracy tradeoff, not a redundancy-eliminating transformation. This is the single most load-bearing concern because it undercuts the conceptual novelty (compression 'for free') and the empirical headline (SOTA). The reader's CONDITIONAL verdict remains appropriate: the contribution is plausible, and the controlled LLaVA-v1.5 ablation (Table 3) and ReAtten ablation (Table 5) provide support, but the lossiness evidence and absent code/error bars require revision. My proposed test—publishing full M=144 (and M=256) results with error bars—would settle whether 64 tokens are indeed lossy on the tasks the paper targets.","tokens_in":17268,"tokens_out":8027,"duration_ms":73997,"concrete_test":"Re-run the same training pipeline with register counts M=64, M=144, and (if feasible) M=256, holding all other hyperparameters and data fixed, and report full MME-RealWorld, V*, DocVQA, TextVQA, ChartQA, and POPE results with at least 3 random seeds and error bars. If M>=144 significantly improves fine-grained benchmarks (e.g., V* and OCR-heavy subsets) over M=64, the 64-token representation is lossy, falsifying the 'comprehensive information' claim. As a stronger control, also compare against a variant that feeds all uncompressed image tokens to the LLM under the same training data; if it outperforms FALCON, the register bottleneck is confirmed as a source of information loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core mechanism ReCompact (Sec. 3.1) asserts that visual registers 'adaptively aggregate essential information while discarding redundant content' and that the output is a 'comprehensive' representation, eliminating the need for an additional compression module. This is the load-bearing assumption: the LLM receives only the M=64 register outputs per sub-image, so any information not moved into a register is permanently lost. The paper's own ablation in Fig. 6 directly contradicts the 'comprehensive' characterization: increasing the register count from 36 to 64 to 144 monotonically improves both MME-RealWorld and POPE scores, with only a 'noticeable taper' after 64. If 64 registers captured all task-relevant information, then 144 registers should not help. The monotonic gain proves that 64 registers are a lossy compression; the chosen 9x reduction trades away measurable accuracy. This is not merely a theoretical risk: the paper shows the tradeoff in its own figure, yet the abstract and Sec. 4.1 claim that FALCON 'convey[s] comprehensive visual information' and 'achiev[es] state-of-the-art performance.' Table 1 also shows LLaVA-OneVision outperforming FALCON on both Avg and Avg-C (perception 59.6/55.8 vs 50.3/48.0; reasoning 44.2 vs 40.7), so the SOTA claim is not supported by the paper's own results. Because the mechanism's central value proposition is that 9x compression removes redundancy without losing essential information, the empirical evidence of lossiness—and the absence of error bars—makes the principal claim conditional at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FALCON, a high-resolution multimodal large language model that introduces a set of learned visual registers inside a pretrained ViT. The ReCompact mechanism aggregates image information into M registers per sub-image, yielding a 9x reduction in visual tokens and eliminating the need for a separate compression module, while a ReAtten module exchanges information across registers of different sub-images to mitigate the fragmentation caused by cropping. The model is trained with a four-stage progressive pipeline and evaluated on MME-RealWorld, DocVQA, TextVQA, ChartQA, V*, and several general benchmarks. The authors claim state-of-the-art performance with 64 tokens per sub-image and argue that the representation is comprehensive and redundancy-free.","tokens_in":17573,"tokens_out":4748,"duration_ms":42424,"significance":"If validated, the idea of performing token compression inside the vision encoder via learnable registers is a useful contribution that could be more parameter- and data-efficient than post-hoc abstractors, and the ReAtten module offers a lightweight mechanism for cross-sub-image information exchange. The manuscript includes several strengths: ablations that control for training data (Table 3), comparisons against alternative compression methods (Fig. 4), and an ablation with and without ReAtten (Table 5). These internal comparisons support the effectiveness of the proposed modules within the authors' pipeline. However, the headline state-of-the-art claim is contradicted by the paper's own Table 1, where LLaVA-OneVision achieves higher average scores, and the 'comprehensive representation' claim is not fully supported by the register-count ablation in Fig. 6, which shows monotonic improvement with more registers. The core idea is defensible as a lossy-compression tradeoff, but the paper currently overstates the results.","major_comments":[{"comment":"The statement that FALCON 'achieves state-of-the-art performance' is not supported by the numbers in Table 1: LLaVA-OneVision obtains higher weighted average scores in both perception (Avg 59.6 vs 50.3; Avg-C 55.8 vs 48.0) and reasoning (Avg 44.2 vs 40.7; Avg-C 43.4 vs 40.7). The authors either need to restrict the state-of-the-art claim to a clearly defined comparison class (e.g., models with at most 1024 visual tokens) or directly compare with and discuss the tradeoff against higher-token models such as LLaVA-OneVision.","section":"Sec. 4.1, Table 1"},{"comment":"The abstract and Section 3.1 describe the register output as a 'comprehensive' and 'non-redundant' visual representation, but Fig. 6 shows that increasing the register count from 64 to 144 continues to improve both MME-RealWorld and POPE scores. This indicates that 64 registers discard task-relevant information, so the 9x compression is a lossy tradeoff rather than a redundancy-free aggregation. The authors should explicitly acknowledge this tradeoff and soften the wording accordingly, or provide evidence that the residual loss is negligible for downstream tasks.","section":"Sec. 3.1 and Fig. 6"},{"comment":"The choice of the register count M = 64 is derived from an ablation on MME-RealWorld and POPE, which are the same benchmarks used in the final evaluation (Tables 1 and 4). This constitutes a form of test-set tuning for a hyperparameter. The authors should either perform the ablation on a held-out validation set or demonstrate that the results are stable for a range of M values and not an artifact of the specific benchmarks chosen for the selection.","section":"Sec. 4.2, Fig. 6"}],"minor_comments":[{"comment":"There is a typo: 'effectivly' should be 'effectively'.","section":"Sec. 3.2, paragraph 1"},{"comment":"'Base on these observations' should be 'Based on these observations'.","section":"Sec. 4.2, paragraph 4"},{"comment":"The word 'maximun' should be 'maximum'.","section":"Table 1 caption"},{"comment":"The label 'Extra Crompression' should be 'Extra Compression'.","section":"Fig. 1"},{"comment":"The phrase 'FALCON significantly outperform other advanced MLLMs' is grammatically incorrect; it should be 'FALCON significantly outperforms'.","section":"Sec. 4.1, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper reports promising internal ablations, but the SOTA claim is contradicted by its own Table 1. The editing should be careful with the framing: the grayed-out models (especially LLaVA-OneVision) are exactly the ones that beat FALCON, and graying them out by token count obscures the fact that the central claim is a Pareto tradeoff, not dominance. Additionally, the selection of M=64 on the same benchmarks used for final reporting is a test-set leakage issue that should be addressed explicitly in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is simple and worth taking seriously: append a set of learnable register tokens to each sub-image in a ViT, and after encoding, feed only the register outputs to the LLM. That is a real departure from Darcet et al., who introduced registers but discarded them, and from abstractor modules that run cross-attention after the encoder. The ReAtten module, which lets registers from different crops talk to each other, is also a reasonable way to address fragmentation without quadratic cost. The controlled ablation on the exact LLaVA-v1.5 training data (Table 3) shows consistent gains over that baseline, and the compression-method comparison in Figure 4 favors ReCompact over pooling, pixel shuffle, and an abstractor. This is honest, reproducible-in-spirit evidence that the mechanism itself helps.\n\nWhere the paper goes soft is in its own framing. The abstract and Section 4.1 call the 64-token representation \"comprehensive\" and claim state-of-the-art performance. Table 1 shows LLaVA-OneVision beating FALCON on both average perception (59.6 vs 50.3) and reasoning (44.2 vs 40.7). Graying out models with more tokens does not change that. “State-of-the-art” is only defensible if qualified as “among token-efficient models.” Likewise, Figure 6 shows accuracy still climbing from 64 to 144 registers. That means 64 tokens are a lossy compression, not a redundancy-free summary. The tradeoff may be a good one, but the paper should say “favorable accuracy/efficiency tradeoff,” not “comprehensive.” There is also a mild selection issue: the register count was chosen on MME-RealWorld and POPE, the same benchmarks used for final evaluation. No error bars, and cross-model comparisons use different backbones and training data.\n\nNone of this sinks the paper. The mechanism, the ablation design, and the training pipeline are all legitimate. The claims just need to be pulled back to match the evidence. The GitHub link is listed, but without actual artifacts the benchmark numbers are hard to verify.\n\nThis is a paper for researchers working on efficient high-resolution MLLMs and visual token compression. It deserves a serious referee who will push on the claims and ask for the code. I would send it to review, but with the expectation that the authors tone down the SOTA language and acknowledge the lossiness of their own compression.","headline":"A genuinely useful token-compression trick for high-res MLLMs that is buried under a SOTA claim its own Table 1 contradicts.","tokens_in":18152,"tokens_out":2617,"would_cite":true,"duration_ms":25937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FALCON's central claim is that 64 learned visual registers per sub-image can replace the full token output of a high-resolution vision encoder, cutting visual tokens 9x while preserving cross-crop continuity through register interaction.","keywords":["visual registers","high-resolution multimodal LLMs","visual token compression","cropping-based encoding","visual fragmentation","ReAtten","fine-grained perception","document understanding"],"falsifier":"Take a high-resolution document or scene image, crop it, and run FALCON with 64 registers; then identify cases where a small text string or tiny object is fully contained in one crop and compare transcription or detection accuracy against a variant with 144 registers or against the same model receiving uncompressed image tokens. If accuracy degrades on those fine-grained cases even when overall benchmark scores stay high, the 64-register compaction is lossy and the central claim fails for the exact detail regime it targets.","tokens_in":17052,"feed_emoji":"🦅","tokens_out":7670,"duration_ms":65173,"temperature":0.7,"pith_summary":"The paper argues that visual redundancy in high-resolution multimodal LLMs can be removed at the encoding stage rather than by a separate compression module. It introduces learned visual registers, a small set of extra tokens fed into a pretrained vision transformer alongside the image patches of each crop; self-attention naturally concentrates the information worth keeping into these registers, so the model sends only 64 register tokens per sub-image to the language model. A second component, Register Interactive Attention (ReAtten), lets registers from different crops attend to one another at every transformer layer, countering the broken words and objects that cropping can cause. FALCON reports large gains across high-resolution benchmarks with a 9x reduction in visual tokens, and its ablations argue that this beats pooling, pixel shuffle, and query-based abstractors while needing far less training data.","feed_headline":"64 learned registers replace thousands of tokens in high-res MLLMs","feed_subtitle":"Cropping-based high-res models get a compact, unbroken visual representation that beats larger-token baselines.","key_machinery":"The central object is the visual register: $M=64$ learnable tokens appended to the image-token sequence of each sub-image before it enters every ViT self-attention layer. Because no attention mask constrains them, the registers act as information sinks, able to read every image token while the image tokens mostly exchange local information, so at the output the register states are a compact digest of the whole crop. The same register set is shared across crops, and after each self-attention step the register states from all crops are concatenated and passed through ReAtten, a self-attention layer over registers only, before the FFN; this is the mechanism that stitches sub-images back together. ReAtten is initialized from the ViT attention weights at the corresponding depth, and the model is trained in four stages that progress from low to high resolution, ending with instruction tuning at high resolution with the ViT frozen.","core_discovery":"FALCON's central claim is that a small set of learned visual registers placed inside a pretrained ViT can serve as the sole visual representation for a high-resolution multimodal LLM. The registers are concatenated with the image tokens of each crop; after self-attention, the register outputs are treated as the complete visual input to the LLM, replacing the roughly 576 tokens a standard 384px patch encoder would emit per crop. The paper calls this Register-based Representation Compacting (ReCompact) and contends that it removes the need for any post-encoder compression module, because a well-pretrained ViT already gathers global information into a few tokens when given register tokens to fill that role. To address fragmentation, Register Interactive Attention (ReAtten) collects the register states of all crops at each ViT layer, runs a self-attention layer over them, and returns the mixed states to their crops; initializing these layers from the ViT's own attention weights lets the mechanism start from learned behavior. With 64 registers per sub-image and a 9x token reduction, FALCON is reported to match or beat models that carry 2-16x more visual tokens across MME-RealWorld, DocVQA, TextVQA, ChartQA, V*, POPE, ScienceQA, MMBench, and SEED-Bench.","pith_inferences":["A testable extension the paper does not run is a reconstruction or probing study: feed a crop through ReCompact, then ask a decoder or linear probe to recover small text or object boundaries from the 64 register states; this would quantify exactly how much fine detail the aggregation loses and predict the failure cases.","The register recipe looks transferable beyond high-resolution crops: video frames, multi-view images, or any input with many patches could use in-encoder registers as a parameter-efficient alternative to resamplers, provided the training-then-frozen progression in the paper carries over.","Because ReAtten is added only in the final instruction-tuning stage with the ViT frozen, a natural control experiment is to add ReAtten to an already-trained cropping-based model without the earlier FALCON stages; if continuity gains persist, the module can be retrofitted to existing systems.","The shared-register design also suggests a cheap way to attend over many crops: because interactions happen only over $M \\cdot N_c$ register tokens rather than all image patches, the cost of global context grows linearly in the number of crops, which is what makes the 16-crop setting practical."],"forward_implications":["A high-resolution MLLM no longer needs a separate query-based compression stage: adapting a pretrained ViT with under 3 million samples teaches the registers to aggregate, where cited abstractor-style modules were pretrained on 129 million to 1.4 billion samples.","The token budget drops 9x at 64 registers per sub-image, so a 16-crop high-resolution image enters the LLM as about 1,024 visual tokens rather than around 9,216 image tokens.","Cross-crop register interaction recovers visual continuity lost to cropping: FALCON reports higher MME-RealWorld perception and reasoning scores than shifted-window attention or a complementary image pyramid, and higher POPE scores, indicating fewer hallucinations.","Register count acts as a direct efficiency-performance dial: raising 36 to 64 registers improves accuracy, while 64 to 144 yields sharply diminishing returns, making 64 the paper's chosen operating point.","The combination transfers across diverse tasks, including OCR in the wild, remote sensing, diagrams and tables, monitoring, autonomous driving, document VQA, chart QA, fine-grained attribute recognition, and general reasoning, suggesting the mechanism is not specialized to one image type."],"supporting_citations":[{"why":"Supplies the observation that pretrained ViTs can gather global information into a few tokens through self-attention, which motivates using registers as information sinks.","marker":"[10]"},{"why":"Provides the Llama-3.1-8B-Instruct language backbone that consumes the register outputs.","marker":"[12]"},{"why":"Provides the SigLIP-L/16-384px vision encoder that FALCON adapts with the visual registers.","marker":"[61]"},{"why":"Supplies the MME-RealWorld benchmark whose perception and reasoning scores anchor the paper's main performance claim.","marker":"[65]"},{"why":"Supplies the LLaVA-v1.5 dataset and training pipeline used in the same-data ablation that isolates the method's contribution.","marker":"[30]"},{"why":"Represents the Abstractor-style query-based compression module that FALCON is contrasted against and claims to outperform with far less pretraining data.","marker":"[59]"},{"why":"Provides the shifted-window attention baseline (W-Atten) for cross-sub-image information exchange that ReAtten is compared with.","marker":"[33]"},{"why":"Provides the complementary image pyramid baseline (CIP) for handling fragmentation, which the paper argues adds redundancy rather than removing it.","marker":"[16]"},{"why":"Provides a cropping-based high-resolution MLLM with query compression that serves as a comparison and background for the high-resolution design space.","marker":"[28]"}],"fun_headline_variants":["Visual registers slash token count 9x in high-res multimodal LLMs","FALCON: small registers, big vision – 9x fewer tokens","Register-based compacting shrinks visual tokens 9-fold","High-res MLLMs get 9x token cut via learned registers","No more crop artifacts: registers keep high-res vision intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pretrained vision transformer can, after light adaptation, fold every task-relevant detail of a high-resolution image into a fixed small set of 64 register tokens per crop without dropping small text, fine boundaries, or spatial relations.","fun_headline_variants_meta":{"raw":{"variants":["Visual registers slash token count 9x in high-res multimodal LLMs","FALCON: small registers, big vision – 9x fewer tokens","Register-based compacting shrinks visual tokens 9-fold","High-res MLLMs get 9x token cut via learned registers","No more crop artifacts: registers keep high-res vision intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1769,"prompt_tokens":1078,"completion_tokens":691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":694,"tokens_out":691,"duration_ms":5783,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:33:53.124847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a high-resolution document or scene image, crop it, and run FALCON with 64 registers; then identify cases where a small text string or tiny object is fully contained in one crop and compare transcription or detection accuracy against a variant with 144 registers or against the same model receiving uncompressed image tokens. If accuracy degrades on those fine-grained cases even when overall benchmark scores stay high, the 64-register compaction is lossy and the central claim fails for the exact detail regime it targets.","supporting_citations":[{"cited_title":"Vision transformers need registers","cited_arxiv_id":null,"evidence_quote":"Supplies the observation that pretrained ViTs can gather global information into a few tokens through self-attention, which motivates using registers as information sinks."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the SigLIP-L/16-384px vision encoder that FALCON adapts with the visual registers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MME-RealWorld benchmark whose perception and reasoning scores anchor the paper's main performance claim."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-v1.5 dataset and training pipeline used in the same-data ablation that isolates the method's contribution."},{"cited_title":"mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration","cited_arxiv_id":null,"evidence_quote":"Represents the Abstractor-style query-based compression module that FALCON is contrasted against and claims to outperform with far less pretraining data."},{"cited_title":"Mini-monkey: Alleviating the semantic saw- tooth effect for lightweight MLLMs via complementary im- age pyramid","cited_arxiv_id":null,"evidence_quote":"Provides the complementary image pyramid baseline (CIP) for handling fragmentation, which the paper argues adds redundancy rather than removing it."},{"cited_title":"Mon- key: Image resolution and text label are important things for large multi-modal models","cited_arxiv_id":null,"evidence_quote":"Provides a cropping-based high-resolution MLLM with query compression that serves as a comparison and background for the high-resolution design space."}],"review_version":1}