{"id":"21be1c27-ac1d-429b-9ea4-b99fdff5937f","arxiv_id":"2506.03643","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DOVE adaptively generates a variable number of continuous visual tokens per image via a learned EOS signal, improving efficiency and semantic quality over fixed-length autoencoders.","lead":"A new vision encoder, DOVE, gives each image a token count suited to its complexity, stopping early on simple images and later on detailed ones. In tests it often beats fixed-length tokenizers at equal or fewer tokens, and a query-guided version focuses tokens on question-relevant regions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Token-count reduction is supported only by a single aggregate average; per-benchmark EOS behavior and reconstruction quality at EOS lengths are not reported.","rationale":"The reader's weakest assumption targeted the EMA threshold heuristic as potentially brittle. My concern is closely related but more specific to the evaluation of the central claim: the paper never disaggregates the reported average token count by benchmark, nor does it evaluate reconstruction quality at the actual EOS-truncated lengths. This makes it impossible to verify that the efficiency-quality trade-off holds across the full range of claimed downstream tasks. The paper does provide some supporting evidence that the EOS policy tracks image complexity, e.g., a Pearson correlation of 0.742 with Laplacian variance on COCO (Section 3.2, Figure 6c), and the qualitative examples in Figure 10 show meaningful differences with token count. Those are real points in its favor. However, this evidence is limited to COCO and reconstruction-focused analyses, not to the document, chart, and infographic benchmarks central to Table 3. The missing per-benchmark token counts and EOS-length reconstruction metrics are a concrete, checkable gap. If the token counts on TextVQA/DocVQA/ChartQA turn out to be close to 256, the abstract's 'significantly reduces the average number of tokens' would be an artifact of pooling easy and hard domains. Even if the counts are favorable, the paper should report them to support the claim. The under-specified λ_eos schedule and EMA window further motivate a CONDITIONAL verdict, but they do not by themselves refute the approach. The reader's CONDITIONAL verdict remains appropriate; my concern reinforces it rather than moving it.","tokens_in":14913,"tokens_out":14171,"duration_ms":161039,"concrete_test":"Using the released checkpoint, compute per-benchmark token statistics on each dataset in Table 3 (VQAv2, GQA, OK-VQA, TextVQA, DocVQA, InfoVQA, ChartQA, ScienceQA): mean and median EOS position, histogram, and fraction of images truncated below 128 tokens. Also compute reconstruction FID/L1 on each benchmark for the EOS-truncated sequences and compare against the 256-token fixed-length baseline. If any benchmark shows a mean token count near 256 or a substantial quality drop relative to 256 tokens, the headline average-token reduction is not uniform and the claim should be scoped to the benchmarks that actually show compression.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract is that DOVE 'significantly reduces the average number of tokens while maintaining high reconstruction quality.' The only evidence for token-count reduction is one aggregate number (121.6 tokens, Table 3) with no per-benchmark breakdown, and no quantitative reconstruction metric is reported for sequences truncated at the EOS positions. The EOS policy is trained with an EMA threshold over ImageNet reconstruction losses (Eq. 3, Table 1), so the learned stopping rule is calibrated to ImageNet's difficulty distribution. On downstream benchmarks with very different image statistics (TextVQA, DocVQA, ChartQA, InfoVQA), the EOS positions may be much longer, meaning the 121.6 average could be dominated by VQAv2 and GQA. If document and chart images are rarely truncated before 200 tokens, the 'significantly reduces' claim would not hold for those tasks, and the VLM gains in Table 3 might partly reflect a larger effective token budget rather than a learned notion of complexity. The paper also does not report FID or L1 at EOS-truncated lengths, so the 'maintaining high reconstruction quality' half of the claim is verified only for fixed lengths, not for the actual dynamic-length outputs. Both halves of the headline efficiency-quality claim therefore rest on an unverified aggregate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DOVE, a dynamic vision encoder that generates a variable-length sequence of continuous tokens for each image and terminates the sequence at a predicted EOS position, trained with a reconstruction loss plus a heuristic EOS loss based on an EMA threshold. A query-conditioned variant (Q-DOVE) refines this idea by supervising reconstruction with bounding boxes around answer-relevant regions. The authors evaluate DOVE on image reconstruction FID, token-length distributions, frozen-feature classification and linear probing, and as the vision encoder of a Vicuna-7B VLM across eight VQA benchmarks, reporting large gains over TiTok, VQGAN, and ALIT at equal or smaller token counts.","tokens_in":15208,"tokens_out":10326,"duration_ms":105474,"significance":"The idea of allowing token count to vary with image complexity is timely, and a working dynamic tokenizer that can be dropped into VLM pipelines would be practically valuable. The paper's breadth is a strength: it evaluates reconstruction, classification, linear probing, and downstream VQA, and it promises code and checkpoints. If the efficiency-quality trade-off is confirmed, DOVE would be a useful alternative to fixed-length tokenizers for multimodal systems. The main weaknesses are that the central claim is not directly evidenced at the dynamic operating point, and several technical details of the EOS mechanism and evaluation protocol are underspecified.","major_comments":[{"comment":"The EOS training objective is not reproducible as written. Table 1's pseudocode treats the generated sequence D as containing an EOS token ('Find the first index j such that D[j]=EOS') even though the model is described as producing continuous visual tokens, and it does not say how p_eos(i) is produced. After the while loop, i is K+1, so the branch 'L_eos ← p_eos(i)' in Table 1 uses an invalid position, whereas Eq. (3) is written in terms of the EOS position m. The pseudocode also has no branch for the case where no EOS token is generated before the maximum length K. Please specify the EOS prediction head, the exact index used in the loss, and the inference-time behavior when no EOS is emitted.","section":"Sec. 2.2, Table 1, Eq. (3)"},{"comment":"The headline claim that DOVE 'significantly reduces the average number of tokens while maintaining high reconstruction quality' is supported only by a single aggregate token count (121.6 in Table 3). No per-benchmark breakdown of EOS lengths is given, and the token-length distribution in Figure 6a is not tied to the downstream benchmarks; the reconstruction-loss curve in Figure 6b is for fixed lengths on COCO, not for sequences truncated at predicted EOS positions. Table 2 reports FID only at fixed lengths. As a result, the reader cannot determine whether the dynamic-length output actually maintains reconstruction quality or whether the 121.6 average is dominated by particular datasets. Please report per-dataset (or at least per-benchmark) average token counts, the EOS-length distribution on those benchmarks, and a reconstruction metric (FID or L1) for the EOS-truncated sequences.","section":"Sec. 3.2-3.3, Table 3"},{"comment":"The linear probing results in Table 5 are not interpretable without specifying which hidden layer is probed and the training protocol. The main text says probing is done on 'model's hidden layers' and Appendix C refers to 'a selected hidden layer,' but neither the layer index nor the classifier training details (epochs, learning rate, pooling) are given. Since linear probing accuracy varies substantially across layers, the cross-model comparison may be unfair, and this weakness directly affects the 'emergent semantics' claim.","section":"Sec. 3.4, Appendix C"},{"comment":"The EOS policy is determined by a training heuristic with several unexamined free parameters: the EMA window W (set to 100), the λ_eos schedule, and, for Q-DOVE, λ_o=1e-10. Because the token-length distribution is entirely a function of the EMA threshold, the robustness of the efficiency claim requires a sensitivity analysis or, at minimum, a report of the variance of the average token count across training runs and datasets. In addition, Section 2.3's description of L_pen is ambiguous: it says L_irr is compared to a threshold after defining L_eos via L_rel, and the penalty term appears identical to the second branch of Eq. (3). Please clarify whether L_pen is an extra term and which loss is used for the threshold comparison.","section":"Sec. 2.2-2.3"}],"minor_comments":[{"comment":"Table 2's column layout is garbled; the values cannot be unambiguously assigned to (dataset, token length) pairs. Please reformat the table so each FID score has a clear header.","section":"Table 2"},{"comment":"The classification subsection says accuracy improves with token count but no quantitative accuracy table is provided; please add numeric results or a table in addition to Figure 5.","section":"Sec. 3.2"},{"comment":"Please define what '121.6 (Avg)' and '82.4 (Avg)' average over, and report the standard deviation or per-benchmark values; the Q-DOVE rows labeled '256#' and '256' should also be disambiguated in the table header.","section":"Table 3"},{"comment":"Please report speed and FLOPs at the EOS-truncated average length in addition to fixed lengths, since that is the claimed operating point.","section":"Table 4"},{"comment":"Please specify which parameters are frozen versus fine-tuned (VQGAN encoder/decoder, the two Pythia models) and give the exact training configurations used for the baselines (TiTok, ALIT) to support the 'same configuration' claim.","section":"Sec. 3.1"},{"comment":"The word 'significantly' is used throughout without statistical tests or error bars; at minimum, state the number of seeds and report standard deviations for the main tables.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the central idea is interesting. The main risk is not novelty or soundness of the approach per se, but that the reported efficiency-quality trade-off currently rests on an unverified aggregate and an underspecified EOS training procedure. I recommend major revision rather than rejection because these gaps can be closed with additional experiments and clarifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution here is an EOS-based stopping mechanism for variable-length visual tokenization, plus a query-conditioned variant. ALIT already had adaptive lengths, but DOVE lets the model emit an EOS token at any position, supports arbitrary lengths in a single forward pass, and adds a text-query-conditioned loss that focuses reconstruction on boxed regions. That is a legitimate, useful step for VLM token efficiency. The linear-probing results are strong and the VQA sweep is wide, which gives the paper surface credibility.\n\nWhere it wobbles: the headline claim—'significantly reduces average tokens while maintaining high reconstruction quality'—rests on a single aggregate number (121.6 tokens in Table 3). There is no per-benchmark token-count breakdown, so you cannot tell whether TextVQA/DocVQA/ChartQA images are also truncated at ~120 tokens or near 256. Those datasets have very different statistics than ImageNet, and the EOS policy is shaped during training by an EMA threshold over ImageNet reconstruction losses. If the policy runs long on documents and charts, the aggregate looks good while the efficiency win is mostly on natural images. Similarly, reconstruction quality at the actual EOS positions is never reported—FID and L1 are only given for fixed lengths. The 'maintaining quality' half of the claim is therefore indirectly supported at best.\n\nThe EOS loss (Eq. 3) is a heuristic with a hand-picked EMA threshold, and the pseudocode in Table 1 leaves the 'no EOS found' case unhandled. That is not fatal, but it is the sort of corner that matters for reimplementation. No error bars or code artifact are provided, which limits confidence in the magnitude of the gains.\n\nOn balance, I think the paper deserves a serious referee. The mechanism is novel enough relative to ALIT, the downstream gains are consistent, and the missing pieces are fixable: report token counts and reconstruction metrics per benchmark, add error bars, and clarify the EOS behavior. I'd send it to review, expecting a minor-to-moderate revision, not a desk reject.","headline":"A genuine EOS-based dynamic tokenizer with broad experiments, but the headline efficiency claim needs per-dataset token counts and EOS-quality numbers before I'd trust the aggregate.","tokens_in":15701,"tokens_out":3596,"would_cite":true,"duration_ms":40805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision encoder that lets each image choose its own token count produces better reconstructions and VQA answers with fewer tokens than fixed-length tokenizers.","keywords":["dynamic vision encoder","variable-length tokenization","end-of-sequence prediction","image reconstruction","vision-language models","visual question answering","token compression","emergent semantics"],"falsifier":"Run DOVE on a set of solid-color or otherwise trivially simple images and record its EOS positions; the central claim predicts that these images will terminate at very few tokens, so if the average length stays near the general training average (for example above 100), the threshold rule is not tracking image complexity.","tokens_in":14752,"feed_emoji":"🖼️","tokens_out":9343,"duration_ms":85990,"temperature":0.7,"pith_summary":"The paper argues that assigning every image the same number of tokens is wasteful, because images carry different amounts of information. It introduces DOVE (Dynamic Output Vision Encoder), a vision encoder that generates a variable-length sequence of continuous tokens per image and learns to emit an end-of-sequence token once its reconstruction is good enough. The central claim is that this dynamic policy uses fewer tokens on average than fixed-length tokenizers while matching or improving reconstruction, classification, and visual-question-answering performance. A query-conditioned variant, Q-DOVE, concentrates tokens on regions relevant to a text query and compresses further. If the paper is right, token count stops being a fixed hyperparameter and becomes a per-image, even per-question, decision.","feed_headline":"Variable-length image tokens beat fixed-length encoders in QA","feed_subtitle":"DOVE cuts average tokens to ~122 while lifting VQAv2 and GQA accuracy over VQGAN, TiTok, and ALIT.","key_machinery":"The central mechanism is a learned end-of-sequence (EOS) token inside an autoregressive token generator. The generator is a 70M-parameter transformer (Pythia) that, given VQGAN encoder features plus timestamp encodings, produces continuous visual tokens and can output EOS at any of up to K positions; all slots after EOS are replaced with zero vectors, and a second transformer decodes the padded sequence for the VQGAN decoder. The EOS training loss compares the current sample's reconstruction loss to an exponential moving average of the last 100 losses, rewarding earlier termination when reconstruction already meets the threshold and later termination when it does not. This threshold rule is what turns token length into a learned, per-sample decision rather than a constant.","core_discovery":"DOVE extends an autoencoder-style tokenizer with a transformer-based dynamic token generator. For each image, the generator produces tokens one at a time and may stop at any position by emitting an end-of-sequence symbol; tokens after that point are zeroed and a fixed decoder reconstructs the image from the truncated sequence. Training couples sequence length to reconstruction quality by comparing each image's reconstruction loss with a moving average of recent losses: when the loss is already below the threshold the model is encouraged to end sooner, and when it is above the threshold the model is encouraged to continue. The paper reports an average of about 121.6 tokens per image instead of 256, a Pearson correlation of 0.742 between EOS position and image complexity, and higher scores than VQGAN, TiTok, and ALIT on most VQA benchmarks and linear-probing sets even at 32 tokens. Query-conditioned Q-DOVE adds a bounding-box weighted loss so tokens focus on query-relevant regions, reducing the average to about 82.4 tokens while improving VQA accuracy further.","pith_inferences":["The same EOS-plus-threshold recipe could be carried over to video, audio, or 3D tokenizers, where per-sample information varies even more than in images.","The moving-average threshold is a training heuristic; a learned complexity estimator or a reward model for token length might make the policy transfer more reliably across datasets, which the paper does not test.","If the reported semantic emergence is real and not an artifact of the threshold rule, variable-length autoencoders could serve as cheap unsupervised feature extractors for segmentation and retrieval, not only as QA backbones.","Q-DOVE's bounding-box supervision could be relaxed to attention or saliency signals, which would let query-conditioned compression apply where box annotations are unavailable."],"forward_implications":["A vision-language model using DOVE needs roughly half the visual tokens (121.6 average versus 256) and still scores higher than VQGAN and TiTok on VQAv2, GQA, OK-VQA, and ScienceQA in the paper's experiments.","Because the encoder can be truncated at any position in a single forward pass, one model can serve different compute budgets at inference time without retraining.","Q-DOVE shows that conditioning token generation on a text query further cuts token count (82.4 average) while improving VQA accuracy, making task-driven compression a working alternative to fixed token budgets.","DOVE's semantic advantage appears in linear probing, where it outranks fixed-length autoencoder tokenizers on all seven reported benchmarks, and in PCA visualizations that resemble semantic segmentation."],"supporting_citations":[{"why":"Supplies the pretrained VQGAN encoder/decoder and the MSE, perceptual, and GAN reconstruction losses DOVE adapts.","marker":"[21]"},{"why":"Closest prior dynamic tokenizer; DOVE's EOS mechanism and single-pass generation are positioned against ALIT's recurrent distillation.","marker":"[20]"},{"why":"Fixed-length tokenizer baseline at 32/64/128 tokens used for reconstruction, classification, and VQA comparisons.","marker":"[60]"},{"why":"Provides the AIM V2 training recipe for fine-tuning the VLM adapter and language model in downstream evaluations.","marker":"[22]"},{"why":"Supplies the 665K mixed VQA dataset and the Vicuna-7B/MLP interface used to evaluate DOVE as a vision encoder.","marker":"[37]"},{"why":"The Pythia-70M transformer is the backbone for both the dynamic token generator and the token decoder.","marker":"[7]"},{"why":"Visual Genome's region descriptions and question-answer pairs provide the text queries and bounding boxes for Q-DOVE training.","marker":"[32]"},{"why":"Open Images relationship annotations are turned into text queries and target regions for Q-DOVE fine-tuning.","marker":"[34]"},{"why":"Motivates the premise that vision tokens are information-redundant, so variable-length representation is worth pursuing.","marker":"[11]"}],"fun_headline_variants":["DOVE adapts token count per image, cutting compute and beating fixed-length rivals","Variable token lengths from DOVE match fixed-length accuracy with fewer tokens","Adaptive image tokens: DOVE uses ~120 instead of 256, yet wins on VQA","DOVE's dynamic token counts shrink to ~120 while boosting VQA accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing an image's reconstruction loss to a moving average of the last 100 training losses reliably indicates how many tokens that image needs, and that this rule continues to work on new images and datasets.","fun_headline_variants_meta":{"raw":{"variants":["DOVE adapts token count per image, cutting compute and beating fixed-length rivals","Variable token lengths from DOVE match fixed-length accuracy with fewer tokens","Adaptive image tokens: DOVE uses ~120 instead of 256, yet wins on VQA","DOVE's dynamic token counts shrink to ~120 while boosting VQA accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2816,"prompt_tokens":939,"completion_tokens":1877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1790}},"tokens_in":555,"tokens_out":1877,"duration_ms":15241,"temperature":1.0,"reasoning_tokens":1790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:58:08.579139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DOVE on a set of solid-color or otherwise trivially simple images and record its EOS positions; the central claim predicts that these images will terminate at very few tokens, so if the average length stays near the general training average (for example above 100), the threshold rule is not tracking image complexity.","supporting_citations":[{"cited_title":"Adaptive length image tok- enization via recurrent allocation","cited_arxiv_id":null,"evidence_quote":"Closest prior dynamic tokenizer; DOVE's EOS mechanism and single-pass generation are positioned against ALIT's recurrent distillation."},{"cited_title":"Multimodal autoregressive pre-training of large vision encoders, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the AIM V2 training recipe for fine-tuning the VLM adapter and language model in downstream evaluations."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the 665K mixed VQA dataset and the Vicuna-7B/MLP interface used to evaluate DOVE as a vision encoder."},{"cited_title":"Pythia: A suite for analyzing large language models across training and scaling","cited_arxiv_id":null,"evidence_quote":"The Pythia-70M transformer is the backbone for both the dynamic token generator and the token decoder."},{"cited_title":"Efficient large multi-modal models via visual context compression","cited_arxiv_id":null,"evidence_quote":"Motivates the premise that vision tokens are information-redundant, so variable-length representation is worth pursuing."}],"review_version":1}