{"id":"73f3d942-f355-4672-aab8-b600b6e78dc9","arxiv_id":"2506.23639","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Priority-guided byte-pair encoding of quantized image patches plus curriculum training yields an 8B discrete-token MLLM competitive with continuous-embedding models on VQA and multimodal benchmarks.","lead":"A team applies byte-pair encoding, the tokenization method behind LLMs, to visual tokens from a VQ-GAN, then trains an 8B multimodal model with curriculum stages. On standard vision-language benchmarks the model nears continuous-encoder baselines, suggesting discrete visual tokens can work competitively without CLIP-style encoders.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The priority function in Eq. 5 is not implemented as described: with F normalized as a probability and S in [0,1], alpha*S dominates P, and Algorithm 2's u_i only encodes orientation, so the claimed frequency-plus-spatial tradeoff is unverified.","rationale":"The reader's verdict is CONDITIONAL and I agree with the direction. The most load-bearing issue is that the central algorithmic contribution, priority-guided encoding, is not faithfully expressed by the provided pseudocode and equations. The paper's headline result depends on the BPE ablation, not on a clean isolation of spatial consistency; but the method section claims that both frequency and spatial consistency are considered, and that claim is the novelty. Algorithm 2's F normalization plus the near-constant S makes the priority score dominated by the spatial term, and the spatial term itself is only orientation consistency. This is not merely a philosophical disagreement about 'spatial': it makes the algorithm's behavior underdetermined, because with alpha*S approximately 0.3 and F small, the top-k list is essentially selected by a score that varies by less than 0.06 across pairs. The reported improvements could therefore come from the training curriculum or the extra BPE embeddings rather than the spatial-consistency objective. I would keep CONDITIONAL because the concern is addressable by logging and ablations; the lack of code and variance data is a secondary reproducibility issue. The concern is about an internal mismatch between equations and pseudocode, not about author integrity.","tokens_in":16531,"tokens_out":11051,"duration_ms":119131,"concrete_test":"Instrument Algorithm 2 on the actual quantized training corpus (or a 1M-image subset) and log F, S, and P for all candidate pairs in the first 200 merge rounds. Check (i) the Spearman rank correlation between P and F, and the share of pairs with alpha*S > 10*F; (ii) whether recomputing S with true position-aware vectors (normalized patch coordinates plus orientation) changes the top-100 pair rankings by more than a threshold. If P is uncorrelated with F or nearly constant, or if the orientation-only S yields the same ranking as a position-aware S, the claimed priority-guided spatial encoding is not load-bearing, and the paper's mechanism must be re-derived or re-ablated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 5 defines P(a,b)=F(a,b)+alpha*S(a,b). Algorithm 2 (Appendix D) normalizes F by the corpus-wide pair count (line 14), so typical F values are small probability masses, while S is an average of exponentials in [0,1] (lines 21-26). With alpha=0.3, alpha*S is about 0.3, several orders of magnitude larger than F for most pairs; the ranking is therefore dominated by S, not frequency. Compounding this, lines 16-18 set u_i(a,b) to (0,1) for horizontal adjacency and (1,0) for vertical adjacency only. Eq. 6's 'relative positioning' then depends only on the horizontal/vertical mix, not on where the pair occurs; for sigma=2.0, S is confined to roughly [0.94, 1.0]. So P is approximately a near-constant orientation-consistency signal, not the 'both frequency and spatial consistency' priority claimed. The 80.6-to-54.3 w/o-BPE ablation in Table 1 shows BPE tokens matter, but it cannot attribute the gain to spatial consistency: no ablation varies alpha, replaces S with a position-aware metric, or reports the selected vocabulary. As written, the central mechanism is internally inconsistent and unvalidated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a framework for training multimodal LLMs on discrete visual tokens obtained by applying byte-pair encoding (BPE) to VQ-GAN indices. It introduces a priority-guided vocabulary construction that combines co-occurrence frequency with a spatial consistency term, a curriculum-based data composition strategy, and a three-stage training procedure with progressive parameter unfreezing. The authors train Being-VL-0.5/0.5+ on top of Llama-3.1-8B and report competitive results on VQAv2, MMBench, MME-P, POPE, and VizWiz relative to continuous-embedding models, with an ablation showing a large drop when the BPE vocabulary is removed.","tokens_in":16909,"tokens_out":6755,"duration_ms":65311,"significance":"If the claims hold, the paper would provide evidence that discrete visual tokens can approach continuous-encoder performance in multimodal understanding, a valuable result for unified architectures. The within-model w/o-BPE ablation (80.2 to 54.3 on VQAv2, Table 1) is a strong demonstration that the BPE vocabulary matters. The paper also includes useful analyses of embedding weight distributions and scaling/efficiency trade-offs. However, the specific mechanism claimed to drive the improvement, the spatial consistency term in the priority function, is not actually implemented as described and is not isolated by ablation, substantially weakening the contribution.","major_comments":[{"comment":"The implementation of the spatial consistency term does not measure relative positioning. For each occurrence, u_i(a,b) is set to (0,1) for horizontal adjacency and (1,0) for vertical adjacency, discarding the actual coordinates (i,j) of the occurrence. Consequently S(a,b) measures whether a pair appears consistently in one orientation, not whether it appears consistently at a particular spatial location or with a particular spatial offset. The claim in Section 3.4 that 'token pairs with consistent spatial relationships across images receive higher scores' is therefore not supported by the algorithm as written.","section":"Appendix D, Algorithm 2 vs Section 3.4, Eq. (6)"},{"comment":"The combined priority P(a,b)=F(a,b)+alpha*S(a,b) is dominated by S in the implementation. F is normalized by the corpus-wide sum of pair counts (Algorithm 2 line 14), making typical F values very small probability masses, while S is an average of exponentials that lies in [0.94, 1.0] for sigma=2.0 given the two-orientation encoding. With alpha=0.3, alpha*S is on the order of 0.28-0.3, several orders of magnitude larger than F for most pairs, so the priority ranking effectively ignores frequency. Since the paper provides no ablation varying alpha or replacing S with a position-aware metric, the claimed frequency-spatial tradeoff is unverified.","section":"Section 3.4, Eq. (5) with Algorithm 2"},{"comment":"The experimental evidence does not isolate the contributions of the priority-guided encoding. The only controlled ablation is 'w/o BPE', which removes the entire BPE vocabulary and thus changes token sequence length, vocabulary size, and training dynamics simultaneously; it does not test whether the spatial consistency term (or the priority function at all) is responsible for the gains over frequency-only BPE, which is the article's primary methodological novelty. The comparisons to published baselines also do not control for training data composition and compute, so the claim of being 'competitive with continuous embedding-based models' rests on uncontrolled comparisons.","section":"Table 1, Section 4.2"}],"minor_comments":[{"comment":"alpha=0.3 and sigma=2.0 are said to be determined through ablation studies on a validation set, but no ablation table for these parameters is provided, so their optimality cannot be assessed.","section":"Section A.1"},{"comment":"The diversity filter applies a threshold tau, but neither the similarity measure nor the value of tau is defined anywhere in the paper; this is a missing hyperparameter for a step that is stated to be part of the method.","section":"Algorithm 2, line 30"},{"comment":"The baselines are said to have 'similar training data scales', but no quantitative comparison of training data composition or volume is given; the appendix lists datasets for the proposed model but not for the baselines, hindering a fair assessment.","section":"Section 4.1"},{"comment":"The text refers to 'activation magnitude' during inference, but the visualization shows embedding weight values; these are different quantities and the wording should be reconciled.","section":"Section 4.3"},{"comment":"The claim that the 16K vocabulary 'eventually outperforms' 4K/8K beyond 1.5x data appears to be based on an extrapolation; please state which data points are measured and which are inferred, and clarify the 'same checkpoint' description.","section":"Section 4.4 and Figure 4"},{"comment":"The subset of LAION-2B-en used as Foundation Data is not quantified and the selection criterion is not described, which impedes reproducibility of the data composition.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The first page states 'Accepted by ICCV 2025.' If this submission is intended for a journal, the editor should confirm that it represents a substantial extension of the conference version, since the current arXiv v1 appears to be the conference preprint. The main technical concern, the mismatch between Eq. (6) and Algorithm 2, should be verified against the released code if available; if the code implements the algorithm as written, the spatial-consistency claim would need to be revised or the algorithm corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable systems/empirical paper that shows a training recipe for discrete visual BPE tokens can reach continuous-encoder quality, but the paper's headline spatial-consistency mechanism doesn't look like it does what the text says, and the benchmark comparisons are not controlled.\n\nWhat's actually new: the authors extend their earlier visual BPE work (ICLR 2025) with three engineering pieces: a priority score that adds a spatial term to pair frequency, a curriculum over data types, and progressive unfreezing of the LLM. The internal ablations are the strongest part: removing BPE entirely drops VQAv2 from 80.2 to 54.3, and the curriculum/unfreezing ablations (Tables 2–4) show consistent, sensible differences. That evidence supports the broad claim that BPE tokens plus a staged curriculum help.\n\nThe soft spot is the spatial-consistency module. Reading Algorithm 2 carefully, u_i(a,b) is set to (0,1) for horizontal adjacency and (1,0) for vertical adjacency. That is orientation, not position. Eq. 6 is described as relative positioning, but the implementation never uses where the pair occurs in the image. Worse, F is normalized as a probability mass, so alpha*S (about 0.3) dominates P numerically for any realistic pair count. The priority ranking is therefore driven by a near-constant orientation consistency signal, not by the frequency/spatial tradeoff claimed. The paper doesn't ablate alpha or replace S with a position-aware metric, so we don't know if the spatial term contributes anything over plain frequency BPE. That's a load-bearing gap in the method description, though not in the overall empirical claim.\n\nAlso missing: code, seeds, error bars, and a controlled comparison against continuous baselines using the same data and compute. The Table 1 numbers are useful but not conclusive.\n\nBottom line: the training recipe is worth reading, and the unified-token direction is promising. But treat the spatial-consistency story skeptically. If this crosses a desk for review, the spatial term needs to be either fixed and validated or dropped, and the authors need to release code and multiple seeds. I'd send it to review rather than desk reject, because the ablations and the discrete-vs-continuous gap question are genuinely interesting.","headline":"Solid training-recipe paper on visual BPE tokens, but the claimed spatial-consistency mechanism is not implemented as described and the controlled comparison is missing.","tokens_in":17390,"tokens_out":3169,"would_cite":false,"duration_ms":34214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Byte-pair encoding of vector-quantized image patches, guided by a priority blending co-occurrence frequency with spatial consistency and trained with a three-stage curriculum, lets a discrete-token multimodal model match…","keywords":["multimodal large language models","byte-pair encoding","visual tokenization","discrete token representation","vector quantization","curriculum learning","vision-language understanding","spatial consistency"],"falsifier":"Train two BPE vocabularies on the same quantized corpus, one with $S(a,b)$ computed from true two-dimensional relative positions $(dx,dy)$ as equation 6 reads and one with the orientation-only version in Algorithm 2; if the vocabularies essentially coincide and downstream VQAv2 and MMBench scores are indistinguishable, then the priority term carries no position-dependent spatial information and the structural-encoding claim is unsupported.","tokens_in":16361,"feed_emoji":"🖼️","tokens_out":8992,"duration_ms":85243,"temperature":0.7,"pith_summary":"The paper tries to establish that byte-pair encoding (BPE), the tokenization method behind text language models, can be carried over to images in a way that lets a multimodal model work entirely with discrete tokens instead of a continuous visual encoder. The proposed pipeline vector-quantizes image patches, then merges frequently co-occurring and spatially consistent token pairs into a compact visual vocabulary, and trains the model with a three-stage curriculum schedule. If the claim is right, the modality gap and information bottleneck of CLIP-style encoders disappear because image and text share one token space. The key empirical evidence is that the 16K-vocabulary variant reaches 80.6 on VQAv2 and 72.1 on MMBench, nearly matching the continuous-embedding model VILA-1.5 (80.9 and 72.3), while the same model without the BPE vocabulary collapses to 54.3 and 38.2.","feed_headline":"Discrete visual tokens now rival continuous encoders","feed_subtitle":"An 8B model fed BPE-coded image tokens hits 80.6 on VQAv2, matching CLIP-based models.","key_machinery":"The central mechanism is the priority-guided visual BPE vocabulary construction. Starting from an 8,192-entry VQ-GAN codebook, the algorithm iteratively merges adjacent token pairs into new tokens, scoring each pair with $P(a,b)=F(a,b)+\\alpha S(a,b)$: $F$ is the normalized frequency of adjacent co-occurrence and $S$ is the average over occurrences of $\\exp(-\\|u_i - \\bar{u}\\|^2/2\\sigma^2)$, where $u_i$ records the relative direction of the pair in each image. The merged tokens extend the vocabulary, and the process repeats until a target size is reached. A second load-bearing piece is the three-stage curriculum training: embeddings are aligned alone, then the first 25 percent of transformer layers are unfrozen, then all parameters, with data ratios shifting from foundation captions to instruction data across stages.","core_discovery":"In the paper's own terms, the discovery is that discrete visual tokens, when built with a priority that mixes co-occurrence frequency and spatial consistency, can carry enough two-dimensional structure for a transformer language model to perform vision-language understanding at the level of continuous-embedding systems. The priority function $P(a,b) = F(a,b) + \\alpha S(a,b)$ scores each candidate token pair, with $F$ the normalized co-occurrence frequency and $S$ the average spatial similarity of the pair's relative offset across the training corpus; the highest-scoring pairs are merged into new vocabulary entries and their occurrences replaced. A three-stage training procedure then aligns the new visual embeddings with frozen language parameters, selectively fine-tunes the early transformer layers, and finally fine-tunes everything, using a curriculum that shifts from foundation captions to complex reasoning and instruction data. The paper reports that this combination closes most of the gap to continuous-embedding baselines on VQAv2, MMBench, MME-P, SciQA-IMG, POPE, and VizWiz, and that removing the BPE vocabulary causes a large performance drop.","pith_inferences":["Editorial inference: because Algorithm 2 sets $u_i$ to $(0,1)$ for horizontal and $(1,0)$ for vertical adjacency, the spatial consistency term $S(a,b)$ as implemented measures orientation consistency, not true position-dependent structure; a direct re-implementation with position-aware offsets would test whether the claimed structural encoding changes the learned vocabulary.","Editorial inference: the framework's logic suggests a testable extension to video, where temporal adjacency can be treated as a third merge direction, predicting similar gains on video QA benchmarks.","Editorial inference: the many near-zero embeddings in the 16K variant indicate a utilization ceiling, implying that an adaptive vocabulary schedule—growing the token count only as data grows—could recover the scaling benefit at lower cost.","Editorial inference: if unified discrete tokens are the cause of the gains, hallucination rates under identical data and compute budgets should differ systematically from continuous-encoder models; comparing POPE-style scores with controlled training would isolate the tokenization effect."],"forward_implications":["Discrete token models can reach the performance level of continuous-embedding models on standard vision-language benchmarks without a CLIP-style encoder.","The BPE vocabulary itself, not just added parameters or training time, is the load-bearing component: deleting it drops VQAv2 from 80.2 to 54.3.","Vocabulary size should be matched to available data: the 16K vocabulary overtakes 8K only after more than 1.5× the standard training data, and is less efficient per unit cost.","The curriculum order and the unfreezing order both matter, with single-stage training falling 12.8 and 14.1 points behind on perception and reasoning averages.","Because all modalities live in one token space, the framework is positioned as a stepping stone toward unified autoregressive generation of images and text."],"supporting_citations":[{"why":"Supplies the theoretical framework for BPE on quantized visual data and the Being-VL-0 baseline this work extends.","marker":"[15]"},{"why":"Provides the CLIP continuous visual encoder that the discrete-token approach is compared against.","marker":"[10]"},{"why":"Serves as the discrete-token baseline (Chameleon) the method must surpass.","marker":"[12]"},{"why":"Provides the strongest discrete-token baseline (Unified-IO-2) and the comparison for SciQA-IMG.","marker":"[13]"},{"why":"Provides the LLaVA continuous-embedding baseline and the visual instruction data used in training.","marker":"[23]"},{"why":"Provides the VILA-1.5 continuous baseline whose scores (80.9/72.3) the proposed model nearly matches.","marker":"[65]"},{"why":"Supplies the VQ-GAN codebook that quantizes image patches into the initial discrete tokens.","marker":"[39]"},{"why":"Provides the Llama-3.1-8B base language model whose embedding layer is expanded for visual tokens.","marker":"[66]"}],"fun_headline_variants":["Byte-pair visual tokens rival continuous encoders","Discrete tokens close the gap to continuous encoders","BPE vision encoding matches CLIP-based MLLMs","8B model with BPE visual tokens hits 80.6 VQAv2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the spatial-consistency score in the priority function actually measures what it claims; as implemented in Algorithm 2, it records only whether a token pair is horizontal or vertical, not where in the image the pair occurs, so the asserted structural information may reduce to orientation consistency.","fun_headline_variants_meta":{"raw":{"variants":["Byte-pair visual tokens rival continuous encoders","Discrete tokens close the gap to continuous encoders","BPE vision encoding matches CLIP-based MLLMs","8B model with BPE visual tokens hits 80.6 VQAv2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2255,"prompt_tokens":912,"completion_tokens":1343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1273}},"tokens_in":528,"tokens_out":1343,"duration_ms":11770,"temperature":1.0,"reasoning_tokens":1273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:35:39.080057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two BPE vocabularies on the same quantized corpus, one with $S(a,b)$ computed from true two-dimensional relative positions $(dx,dy)$ as equation 6 reads and one with the orientation-only version in Algorithm 2; if the vocabularies essentially coincide and downstream VQAv2 and MMBench scores are indistinguishable, then the priority term carries no position-dependent spatial information and the structural-encoding claim is unsupported.","supporting_citations":[],"review_version":1}