{"id":"d8e2905a-87f1-4889-bf61-d0d2b1b954bd","arxiv_id":"2412.16491","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ImagePiece merges the bottom 30% of low-attention image tokens into semantically meaningful chunks before pruning, reporting 54% faster DeiT-S inference with a 0.39% ImageNet accuracy gain.","lead":"This paper proposes ImagePiece, a method that first merges the least important image patches into bigger chunks before pruning or merging tokens in a vision transformer. On ImageNet it reports 54% faster DeiT-S inference with a 0.39% accuracy gain, suggesting token reduction can be done more carefully.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablation: conv stem alone may account for the 0.39% ImageNet accuracy gain; the paper never isolates retokenization from the overlapping-conv stem.","rationale":"The reader identified the same load-bearing assumption, and I agree. The paper's central result is a combined speed-accuracy claim. The missing ablation directly tests the causal attribution stated in the abstract: that retokenization 'enhances inference speed ... while achieving a 0.39% improvement.' The paper's only ablation removes the whole local coherence bias module, which simultaneously removes the conv stem and the similarity-bias mechanism; it cannot separate the stem's architectural effect from the retokenization effect. This matters because the paper cites Xiao et al. 2021 (Early Convolutions Help Transformers See Better), which shows that replacing the patchify stem with overlapping convolutions improves ViT accuracy on ImageNet. Without a stem-only control, the 0.39% improvement could be a known architectural effect rather than a benefit of the proposed re-tokenization. The speed claim (54%) is less at risk because token pruning clearly reduces compute, but the accuracy claim is central to the paper's framing as an improvement. I do not think this invalidates the engineering value of the full method; the compatibility results and hyper-speed robustness are interesting, and the method may still be a valid engineering contribution. However, the paper's central accuracy attribution is unproven without the stem-only ablation, so the reader's CONDITIONAL verdict is appropriate. No other concern is more load-bearing than this missing control.","tokens_in":13313,"tokens_out":3769,"duration_ms":32649,"concrete_test":"Train DeiT-S from scratch with the exact same overlapping conv stem (four 3x3 convs plus one 1x1 conv) used by ImagePiece, with retokenization and token pruning disabled, using the same 300-epoch ImageNet recipe. Report top-1 accuracy and throughput on the same hardware. If this baseline reaches >= 80.22%, the claimed accuracy gain over DeiT-S is attributable to the conv stem, not retokenization; if it stays near 79.8%, retokenization is responsible for the gain. Also report the throughput of this baseline to assess how much of the 54% speedup comes from the stem itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's reported accuracy improvement over DeiT-S (79.83 -> 80.22, +0.39%) is presented as the benefit of the ImagePiece retokenization strategy. However, the full system also replaces the standard patchify stem with an overlapping-convolution 'local coherence bias' stem. The only ablation removes this entire module, dropping accuracy to 79.81, but no model is trained with the conv stem and without retokenization/pruning. Since early conv stems are known to improve ViT accuracy (Xiao et al. 2021 is cited but not discussed), the conv stem alone could plausibly reach about 80.2%. In that case the accuracy gain over DeiT-S is an architecture change, and the retokenization's contribution is primarily speed, at best accuracy parity with the stem-only baseline. Comparisons against DynamicViT, EViT, and ToMe also use standard patchify stems, so the comparison conflates the stem change with retokenization. Without this ablation, Tab. 1's accuracy claim and the abstract's wording overstate the mechanism. The compatibility results in Tab. 9 and the analyses in Tabs. 5-8 are useful, but they do not repair this missing attribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ImagePiece, a retokenization strategy for Vision Transformers that groups inattentive (bottom-k) tokens into semantically meaningful chunks via bipartite soft matching, reassesses their [CLS]-attention after merging, and prunes tokens that remain unattentive. To make nearby non-semantic patches more similar, the standard patchify stem is replaced by an overlapping-convolution local coherence bias module. On ImageNet-1k, the authors report that with DeiT-S the method reaches 80.22% top-1 accuracy at 3891.9 img/s, versus the DeiT-S baseline of 79.83% at 2531.1 img/s (a 54% speedup and +0.39% accuracy), and they further demonstrate compatibility with existing pruning and merging methods and robustness under hyper-speed inference and random masking.","tokens_in":13685,"tokens_out":8206,"duration_ms":61978,"significance":"The core idea of retokenizing image patches in analogy to subword tokenization is well motivated and potentially useful. The compatibility experiments (Table 9) and the analyses of token attentiveness and similarity during merging (Tables 5-8) provide interesting evidence about why merging inattentive tokens first can help. If the accuracy gain were properly isolated to the retokenization mechanism, this would be a meaningful contribution to efficient ViT inference. However, as the paper stands, the central accuracy claim is confounded with an architecture change, and no variance information is given, so the significance level is not yet established. The speed benefit of the method is plausible, but the advertised accuracy advantage requires additional controlled experiments.","major_comments":[{"comment":"The paper's headline result, +0.39% accuracy over DeiT-S while improving throughput by 54%, is not attributable to retokenization because the full model also replaces the standard patchify stem with an overlapping-convolution local coherence bias module. The only ablation removes this entire module, giving 79.81%, which is slightly below DeiT-S's 79.83%. This strongly suggests that the entire accuracy gain comes from the conv stem, not from the retokenization mechanism. The authors must add a model that uses the conv stem with the original DeiT tokenizer (no retokenization and no pruning) to separate the stem's contribution; without that control, the central accuracy claim is unsupported.","section":"Ablation Study (near end) and Tables 1-2"},{"comment":"The comparisons against DynamicViT, EViT, ToMe, and other baselines are unfair because those baselines use the standard patchify stem, whereas ImagePiece uses a different overlapping-convolution stem. Since early convolutional stems are known to improve ViT accuracy (the paper itself cites Xiao et al. 2021 but does not discuss this), the accuracy advantage over the baselines could be due to the stem rather than the retokenization strategy. To make the comparison meaningful, the authors should either integrate the same stem into the baselines or report a stem-only baseline (DeiT with conv stem, no token reduction) for all comparisons.","section":"Tables 1, 2 and Section \"Main Results\""},{"comment":"The method description is underspecified for reproducibility. The three-stage retokenization procedure does not state at which transformer layers the retokenization is applied, how many retokenization iterations are run per layer, or how the hyperparameters (p = 0.3, similarity merging ratio = 0.08, pruning keep rate r = 0.8) are scheduled across layers. For example, Table 5 reports statistics at layers 2, 3, 5, 6, 8, and 9, but the text never explains why these layers are chosen. A precise algorithm description or pseudocode, plus a table of hyperparameter schedules, is needed before the experiments can be reproduced.","section":"Section \"ImagePiece\", Subsection \"Re-tokenizing Non-semantic Tokens\""},{"comment":"The main claims of +0.39% accuracy and 54% speedup are reported without error bars or multiple-seed trials. ImageNet-1k training from scratch typically has run-to-run variation of at least ±0.2%, which is comparable to the reported accuracy improvement. The authors should report the mean and standard deviation over at least two or three seeds for the main models, and state the measurement conditions for throughput (e.g., batch size, hardware, software versions). Without such information, the accuracy gain cannot be distinguished from noise.","section":"Tables 1-4 and the Abstract"}],"minor_comments":[{"comment":"When describing the ablation that removes the local coherence bias module, the paper should explicitly state that this returns to the standard patchify stem, so that readers understand the two confounded changes.","section":"Ablation Study"},{"comment":"The caption for Table 4 says \"16×16 masks\" but it is not clear whether each mask is a 16×16 pixel block or a patch, nor how many masks are applied for each column besides the listed counts. Please clarify the mask generation procedure.","section":"Table 4"},{"comment":"The header \"Ratioinattn→attn(%)\" is missing spacing and should be written as \"Ratio (inattn→attn) (%)\" for readability.","section":"Table 5"},{"comment":"The local coherence bias module is described as \"four 3×3 convolutions and a single 1×1 convolution\" but the stride, padding, number of channels, and output resolution are not specified. Please provide the full architecture details.","section":"Section \"Implementation details\""},{"comment":"The abstract's claim that the approach surpasses other baselines \"by an accuracy over 8%\" is vague; the Experiments section reports the average gap as 8.15% over three baselines. Please state the average explicitly in both places or give the range of per-baseline gaps.","section":"Abstract and Section \"Hyper-speed Inference Results\""},{"comment":"The paper uses \"re-tokenization\" and \"retokenization\" inconsistently; please pick one and use it consistently. There are also minor hyphenation errors such as \"bottom-k\" and \"bottom-k\" appearing inconsistently.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2412.16491 (ImagePiece). The core idea—before pruning or merging, merge only the bottom-k CLS-inattentive tokens, then re-evaluate the merged abstractions—is a clean, sensible twist on EViT and ToMe. The compatibility experiments in Tab. 9 (plugging ImagePiece into DynamicViT, EViT, ToMe) are genuinely useful, and the hyper-speed results (2.51x speedup with a 6.28% accuracy drop vs 9-20% for baselines) are striking if they hold up. This paper is worth a look for anyone working on token reduction for ViTs.\n\nWhat it does well: it identifies a real failure mode of both pruning and merging—discarding tokens before context forms, or smoothing attentive tokens with background—and addresses it in a simple, model-agnostic way. The analysis tables (Tab. 5-8) give some empirical support for the mechanism. The novelty claim is mostly fair: restricting merging to inattentive tokens and then re-scoring them is not in EViT or ToMe.\n\nThe soft spot is load-bearing. The abstract and Tab. 1 state that ImagePiece improves DeiT-S accuracy by 0.39% (79.83 to 80.22) while being 54% faster. But the full system replaces the standard patchify stem with an overlapping-convolution 'local coherence bias' module. The only ablation removes that entire module, and accuracy drops to 79.81. There is no model trained with the conv stem but without retokenization/pruning. Since early conv stems are known to improve ViT accuracy (Xiao et al. 2021, which they cite but do not discuss), the conv stem alone could plausibly account for the 0.39% gain. In that case the retokenization's contribution is primarily speed, at accuracy parity with a stem-only baseline. This conflation also affects Tab. 9, where the baselines use patchify stems, so the +0.79% average improvement may be mostly the stem, not the retokenization. The authors need to train a conv-stem DeiT-S with standard EViT/ToMe pruning or merging, and show that ImagePiece still beats it. Without that, the central accuracy claim is not established.\n\nOther, smaller issues: no error bars on the main throughput/accuracy numbers; no code release; the MaxMatch analogy is decorative (no algorithmic link to WordPiece); and the precise pruning schedule for hyper-speed inference is deferred to a supplement not present in this version. None of these are fatal, but they add to the incompleteness.\n\nBottom line: good idea, useful compatibility data, but the paper overstates what is proven. It deserves a serious referee, and the missing ablation should be requested. If the authors run that experiment, I'd want to see the result before building on it.","headline":"The retokenization idea is neat and the compatibility results are useful, but the paper overclaims the accuracy gain because it never isolates the conv stem from the retokenization mechanism.","tokens_in":14095,"tokens_out":2388,"would_cite":true,"duration_ms":20214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grouping low-attention image patches into coherent chunks before pruning or merging—called ImagePiece—makes DeiT-S run 54% faster while improving ImageNet accuracy by 0.39%.","keywords":["image recognition","vision transformer","efficient inference","token pruning","token merging","retokenization","ImageNet classification","attention scoring"],"falsifier":"Train a DeiT-S variant that uses the same local coherence bias stem and the same training recipe, but keeps ordinary patch tokens and performs no ImagePiece grouping or pruning; if its ImageNet-1k top-1 accuracy matches or exceeds 80.22%, the central accuracy claim of the paper collapses to an architecture change.","tokens_in":13121,"feed_emoji":"🖼️","tokens_out":6042,"duration_ms":46476,"temperature":0.7,"pith_summary":"ImagePiece claims that the standard way of cutting tokens in vision transformers—pruning low-attention patches or merging similar patches—works on the wrong unit: an individual image patch often carries no meaning by itself, so it is discarded or smoothed away before it can contribute. The paper proposes treating image patches like subwords in NLP and regrouping the least-attended tokens into locally coherent chunks, using a MaxMatch-style procedure lifted from WordPiece. Those chunks are then scored again against the class token, and only the ones that have become meaningful survive. This re-tokenization, together with a lightweight overlapping-convolution stem, is claimed to make DeiT-S 54% faster on ImageNet while raising its top-1 accuracy by 0.39%, and to give an 8% accuracy margin over prior token-reduction baselines in hyper-speed regimes. If true, it means token reduction for ViTs should be a content-aware re-segmentation problem, not just a selection or averaging problem.","feed_headline":"Retokenize patches like words: ViT runs 54% faster, gains accuracy","feed_subtitle":"ImagePiece groups weak image tokens into meaningful chunks, then prunes the rest, adding 0.39% ImageNet accuracy at higher speed.","key_machinery":"The mechanism is a three-stage retokenization loop supervised by class-token attention, preceded by a local coherence bias module. The local coherence bias replaces the ViT patchify stem with overlapping 3x3 convolutions and a 1x1 convolution, raising the first-layer similarity among nearby low-attention tokens from 0.5293 to 0.8091 (Tab. 8). Then MaxMatch-style bottom-k grouping: from the bottom 30% of tokens by [CLS]-attention, tokens are divided into two groups A and B by alternating rank order; each A-token is bipartite-soft-matched to its most similar B-token and averaged into an abstraction. The merged tokens are re-scored, and those that have become attentive are retained while the rest are pruned (bottom 20%). This machinery is compatible as a drop-in replacement: applying ImagePiece to DynamicViT, EViT, and ToMe raises their ImageNet accuracy by 0.79% on average.","core_discovery":"The central claim is that the standard ViT patch tokenizer produces tokens that are not atomic semantics: many patches are meaningless in isolation, so pruning or merging them prematurely destroys information. ImagePiece addresses this by repeatedly applying a three-step loop: score each token by its attention to the class token, take the bottom 30% of tokens, split them into two alternating groups, and merge each token from one group with its most similar partner in the other; then recompute attention on the merged 'abstractions' and discard those that remain inattentive. This lets the model delay its pruning decision until tokens have had a chance to become meaningful when grouped with local neighbors, and it preserves highly attentive tokens from being diluted by similarity-based merging. On ImageNet-1k with DeiT-S, the method reports 80.22% top-1 accuracy (up from 79.83% for DeiT-S) at 3891.9 img/s (up from 2531.1), a 54% speedup; with DeiT-Ti it reports 72.61% (up from 72.13%) at 9450.2 img/s. At a 2.51x speedup it claims to lose only 6.28% accuracy, while DynamicViT, EViT, and ToMe lose 20.05%, 13.99%, and 9.24% respectively.","pith_inferences":["The reported accuracy gain over DeiT-S rests on a single training run; a multi-seed replication would establish whether the +0.39% is systematic or partly random variation, since the paper reports no variance.","The retokenize-then-rescore loop is not specific to images; the same logic could be applied to other transformer inputs whose elementary tokens are context-dependent, such as point clouds or audio frames, with a locality bias appropriate to each domain.","Because the bottom-30% fraction and pruning ratio are fixed hyperparameters, an adaptive policy that varies how many tokens are grouped per image based on its attention distribution could push the speed-accuracy frontier further; the paper does not explore this."],"forward_implications":["Token pruning in ViTs becomes safer: decisions to discard tokens are postponed until low-attention patches have been grouped with their local neighbors and re-scored, so fewer prematurely discarded tokens lose useful context.","ImagePiece is a plug-in rather than a replacement: attaching it to DynamicViT, EViT, and ToMe improves their ImageNet accuracy by 0.79% on average, so existing efficient-ViT methods can be upgraded without rewriting their pruning or merging logic.","At extreme compression (26 output tokens, 13% of the original 197), ImagePiece preserves accuracy comparable to baselines that keep two to three times as many tokens, pointing to substantially cheaper deployment for batch inference.","The local coherence bias stem is a separate, transferable ingredient: it raises first-layer similarity among low-attention tokens from 0.5293 to 0.8091, which can be adopted by other token-reduction schemes to improve merge quality."],"supporting_citations":[{"why":"Supplies the WordPiece MaxMatch tokenization concept that ImagePiece adapts to images.","marker":"Devlin et al. 2018"},{"why":"Defines class-attention scoring (EViT) used to evaluate token importance, and serves as pruning baseline.","marker":"Liang et al. 2022"},{"why":"Provides bipartite soft matching algorithm used in the merging step, and serves as token-merging baseline (ToMe).","marker":"Bolya et al. 2023"},{"why":"Supplies the DeiT training recipe and baseline models (DeiT-S/Ti) that ImagePiece builds on and compares against.","marker":"Touvron et al. 2021a"},{"why":"DynamicViT, a learned-projection pruning baseline that ImagePiece is compared with and integrated into.","marker":"Rao et al. 2021"},{"why":"ImageNet-1k dataset used for all experiments.","marker":"Deng et al. 2009"}],"fun_headline_variants":["Retokenize like NLP: ViT 54% faster, accuracy up","ImagePiece: speed up ViT by 54% with smarter tokens","ViT runs 54% faster and gains accuracy with ImagePiece","Group weak ViT patches to speed up 54% and boost accuracy","ViT patches as words: ImagePiece groups to go 54% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the reported accuracy improvement over DeiT-S comes from the retokenization logic, but the model also changes the input stem to overlapping convolutions; without a control that uses the new stem but no retokenization, the gains could be architectural rather than due to token grouping.","fun_headline_variants_meta":{"raw":{"variants":["Retokenize like NLP: ViT 54% faster, accuracy up","ImagePiece: speed up ViT by 54% with smarter tokens","ViT runs 54% faster and gains accuracy with ImagePiece","Group weak ViT patches to speed up 54% and boost accuracy","ViT patches as words: ImagePiece groups to go 54% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2836,"prompt_tokens":1054,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1685}},"tokens_in":670,"tokens_out":1782,"duration_ms":11554,"temperature":1.0,"reasoning_tokens":1685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:30:57.664381+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a DeiT-S variant that uses the same local coherence bias stem and the same training recipe, but keeps ordinary patch tokens and performs no ImagePiece grouping or pruning; if its ImageNet-1k top-1 accuracy matches or exceeds 80.22%, the central accuracy claim of the paper collapses to an architecture change.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines class-attention scoring (EViT) used to evaluate token importance, and serves as pruning baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides bipartite soft matching algorithm used in the merging step, and serves as token-merging baseline (ToMe)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DynamicViT, a learned-projection pruning baseline that ImagePiece is compared with and integrated into."}],"review_version":1}