{"id":"98553679-84ac-40a4-b26b-4ee8bc26fbb3","arxiv_id":"2411.18092","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A noise-allocation training trick learns token relevance scores that can prune vision transformer tokens at test time, beating some baselines in some regimes but not all claimed settings.","lead":"This paper proposes a way to train vision transformers to drop unimportant image tokens at test time by adding tuned noise during training. The method is simple and needs no classification token, but its claimed performance edge is inconsistent across the paper's own experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Softmax normalization in Eq. 4 prevents α from being a selective relevance score for more than a few tokens, so the learned noise allocation cannot support the claimed hard-dropping mechanism.","rationale":"Read in good faith, the paper's central claim is that TNT provides state-of-the-art token pruning by learning α as a relevance score through a softmax-constrained noise allocation. The most load-bearing condition for that claim is that α can actually be a selective per-token relevance estimate. The math of Eq. 4-5 does not permit this selectivity at scale: because α is a probability distribution over N≈196 tokens, only a handful of tokens can receive α large enough to reduce their noise substantially; all other tokens receive nearly identical noise during training. This is an internal structural problem, not a disagreement with external consensus. It is closely related to, but sharper than, the reader's training/test mismatch concern: even before considering test-time hard dropping, the training-time allocator cannot behave as the paper describes. The check of recording α distributions and ablating the softmax would settle whether any residual ranking signal survives. Additionally, even taking the paper's own tables at face value, the broad SOTA claim is contradicted in the high-token regime: for DeiT-B multi-layer at GFLOPs≈8.0, Table 7 shows EViT at 78.46 versus TNT at 77.31, and for ViT/16 multi-layer at GFLOPs≈4.1, Table 19 shows DynamicViT at 67.63 versus TNT at 58.96. Credit is due for the detailed supplement, released code, ablations, and the honest limitation statement in §4.4, but those do not repair the central mechanism or the overclaimed headline. The verdict should remain REJECT for the central SOTA claim as stated.","tokens_in":22149,"tokens_out":10982,"duration_ms":111385,"concrete_test":"Run the released TNT checkpoint on ImageNet validation and, at the pruning layer, record the per-image α vector (Eq. 4) before similarity pruning. For K=0.5, compute the fraction of the top-K kept tokens that have α_i < 0.1. At N=196 this fraction is at least ~89/98 by the sum-to-one constraint, so a value near 1 confirms that the softmax allocator is not selectively protecting the kept tokens. Then retrain with the softmax replaced by an independent sigmoid, α_i = σ((Wf)_i), and compare K=0.5 accuracy; a material gain over the reported DeiT-S value of 78.65 would show that the sum-to-one softmax, not the noise-relaxation idea, is the weak link.","verdict_should_be":"REJECT","load_bearing_attack":"The central premise of TNT is that α (Eq. 4) is a learned relevance estimate: tokens with high α receive little noise in training and are kept at test time. But Eq. 4 defines α_i = Softmax(W f)_i over all N token embeddings, so Σ_i α_i = 1. For N=196, typical α_i ≈ 0.005, and the training noise multiplier in Eq. 5, (1−α_i), is therefore ≈0.995 for nearly every token. Softmax allows at most one token with α≈1, and at most nine tokens with α>0.1; for any keep rate K above a few tokens, the kept tokens are mostly ones whose training-time noise was almost identical to that of the dropped tokens. Thus the gradient signal for the α ranking does not come from selectively suppressing noise on the tokens that test-time pruning keeps; it comes from a near-uniform perturbation, and the claimed per-token relevance mechanism is not what the training procedure implements. This compounds the acknowledged gap in §4.4 that neither similarity pruning nor discrete dropping is simulated in training. The reported gains over random dropping, if reproducible, cannot be attributed to the noise-allocation mechanism as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Training Noise Token (TNT) Pruning, a method for vision transformers that relaxes discrete token dropping into a continuous additive noise process. During training, a lightweight noise allocator computes per-token scores alpha via a softmax over a learned linear projection of token embeddings (Eq. 4) and perturbs each token with Gaussian noise scaled by (1 - alpha) (Eq. 5). At test time, the same alpha scores are used to rank tokens and prune the lowest-ranked ones, optionally followed by a similarity-based redundancy-removal step using random partitioning. The authors claim theoretical connections to Rate-Distortion / Variational Information Bottleneck, and evaluate on ImageNet with DeiT-T/S/B and ViT/16, reporting accuracy, GFLOPs, and throughput under single-layer and multi-layer pruning schedules. The paper includes extensive supplementary tables, an ablation study, qualitative visualizations, and a PyTorch implementation.","tokens_in":22379,"tokens_out":10615,"duration_ms":88156,"significance":"If the claims were substantiated, TNT would be a practically relevant contribution: it prunes tokens using only a small learned allocator on a frozen backbone, needs no CLS token, and could be applied to architectures without a classification token. The paper is commendable for providing code, extensive ImageNet experiments, qualitative pruning maps, and ablations over several design choices. However, the headline claims of state-of-the-art performance and of a principled information-theoretic method are not supported by the evidence in the manuscript. The paper's own tables show that TNT is frequently outperformed by EViT and DynamicViT in the low-token regimes where the authors claim superiority, and the softmax-based noise allocation has a structural property that undermines the claimed per-token relevance learning. The theoretical connection to the information bottleneck is asserted rather than derived. The practical value of the method is therefore uncertain outside a narrow set of configurations.","major_comments":[{"comment":"The abstract and conclusion claim that TNT 'demonstrates advantages over previous pruning methods' and 'excels in low-token retention rates.' These claims are contradicted by the paper's own data. In single-layer DeiT-B-Distil. at K=0.25 (Table 6), EViT achieves 73.54 while TNT achieves 70.76, and at K=0.2 EViT achieves 70.4 vs TNT 62.62. In multi-layer ViT/16 (Table 20), at GFLOPs ≈ 3.9 DynamicViT achieves 60.74 vs TNT 51.40, and at GFLOPs ≈ 3.6 DynamicViT achieves 50.15 vs TNT 39.81. The 'state-of-the-art' claim is therefore not supported by the authors' own measurements; the conclusion overstates the method's performance.","section":"Section 5 and Tables 6, 19-20"},{"comment":"The softmax normalization in Eq. (4) makes alpha a probability distribution over all N tokens, so the sum of alpha_i is 1. For N=196, the average alpha is about 0.005, and the noise multiplier (1 - alpha_i) in Eq. (5) is approximately 0.995 for almost every token. At keep rates K=0.5 or K=0.25, most of the tokens retained at test time therefore received nearly the same noise magnitude during training as the tokens that were dropped. The gradient signal for ranking the retained tokens is consequently very weak, and the claim that alpha is a learned per-token relevance estimate is not supported by the mechanism as written. The reported gains over random dropping cannot be attributed to the noise-allocation mechanism without further analysis, especially given the acknowledged train/deploy gap in Section 4.4.","section":"Section 3, Eqs. (4)-(5)"},{"comment":"The paper claims 'theoretical connections to Rate-Distortion literature' and positions TNT 'within the Information Bottleneck framework,' but no information-bottleneck objective is actually optimized. The training loss is the standard cross-entropy, there is no rate term, no mutual-information estimator, and the cited channel-capacity bound is not used to derive any result. The sentence 'This solution can also be directly mapped onto the Deep Variational Information Bottleneck' is an assertion, not a derivation. As a result, the information-theoretic framing is rhetorical rather than substantive, and the paper's methodological novelty rests entirely on the heuristic noise-allocation scheme.","section":"Section 3, 'Classical results...' and Section 1"},{"comment":"The multi-layer comparisons do not appear to be on equal footing. In Table 11 (DeiT-S-Distil. multi-layer), the EViT parameter is listed as ρ=.2 for every row while GFLOPs vary from 3.48 to 2.05, which suggests the EViT keep rate was held fixed rather than swept; this does not produce a genuine accuracy-computation trade-off curve for EViT. Additionally, in Table 19 (ViT/16 multi-layer), DynamicViT's GFLOPs are systematically higher than TNT's GFLOPs within the same nominal bucket (e.g., 7.23 vs 6.73 for 'GFLOPs ≈ 6.8'), so the comparison is not at matched compute. These issues undermine the claim that TNT 'consistently shows strong performance' in the multi-layer setting.","section":"Section 4.2, Tables 7, 11-12, 19-20"},{"comment":"Section 4.4 admits that redundant tokens are not removed during training, and the code in Figure 6 adds noise at layers 1-5 during training, whereas deployment prunes at a single layer (Section 4.1) or at layers [3,4,5] (Table 4). Thus the alpha scores used at the pruning layer are trained under a distribution in which earlier layers also inject noise, but at test time no noise is added before pruning. The paper does not analyze how this train/deploy mismatch affects the validity of alpha as a relevance ranking for the deployment procedure, which is a load-bearing gap given that the method's only training signal is the noise allocation.","section":"Section 4.4 and Figure 6"}],"minor_comments":[{"comment":"The caption says the ablation is for 'DeiT-S-Distil.' but the table rows list 'Deit-B-Distil.'; please correct the inconsistency.","section":"Table 3 caption"},{"comment":"Several references contain stray page numbers (e.g., [4] and [16] end with '1, 8, 9, 10, ...' and similar strings); please clean the bibliography.","section":"References"},{"comment":"EViT is cited inconsistently as both [10] and [14]; use a single reference consistently throughout.","section":"Section 1 and Tables"},{"comment":"The supplementary uses 'TNT+MLP' for DeiT-Tiny, but the main text does not describe this variant; please explain the difference between TNT and TNT+MLP.","section":"Supplementary Figure 7"},{"comment":"The number of similarity-pruned tokens s is stated as 25 and 30 for DeiT and ViT respectively in Section 4.1, but as 40 for all experiments in Section 4.2; please clarify which setting applies to which configuration.","section":"Section 4.1 and Section 4.2"},{"comment":"The text says 'Both I(x, y) and I(s, x) measure interaction information'; this terminology is imprecise, as these are mutual information quantities, not interaction information.","section":"Section 3.1"}],"recommendation":"reject","confidential_remarks":"The paper's central claim of state-of-the-art performance is contradicted by its own tables, which is a serious issue in its current form. The softmax-based noise allocation argument, if correct, means the method's mechanism does not do what is claimed, and fixing it would require a methodological change rather than a textual revision. The authors should be encouraged to resubmit a substantially revised version that either adopts a different normalization (e.g., sigmoid) with a proper optimization objective, or reframes the contribution as a heuristic with clearly delimited regimes of success."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The noise-based continuous relaxation for token pruning is a genuinely new idea, and the paper is unusually honest about its own gaps — they explicitly admit that redundancy pruning is not simulated in training and that multi-layer tuning is inexact. The supplementary material is extensive, code is provided, and the CLS-free evaluation on mean-pooled ViT is a nice contribution. This is not a sloppy paper.\n\nThat said, the central SOTA claim fails against the paper's own tables. In DeiT-B single-layer at K=0.25, EViT gets 73.54 versus TNT's 70.76; in ViT/16 multi-layer around 3.9 GFLOPs, DynamicViT gets 60.74 versus TNT's 51.40. Those are not corner cases. The conclusion's statement that the method \"excels in low-token retention rates\" is simply not what the results show in several regimes. The paper is competitive elsewhere, but competitive is not state-of-the-art.\n\nThe stress-test about softmax normalization is worth taking seriously. Since α sums to 1 over 196 tokens, most α values are tiny, so (1−α) is close to 1 for almost every token. That means the training-time noise is nearly uniform, and the gradient signal for distinguishing kept from dropped tokens is weak. The paper's own explanation — that the model adds the least noise to the most predictive tokens — is not really what the equations implement when you keep 25% or 30% of tokens. The authors should have discussed this directly. Still, the method does beat random dropping in most experiments, so some signal exists; the mechanism is just less clean than the paper suggests.\n\nOther soft spots: no error bars, single runs, and the AViT baseline is excluded due to a failed reproduction — that is acceptable but weakens the comparison. The Rate-Distortion connection is asserted as motivation, not proven.\n\nWho is this for? People working on token pruning for ViTs, especially those interested in relaxation-based training tricks. The idea has merit, and the paper is worth reading. I would send it to peer review, but it needs major revision: the claims need to be scaled back to what the data actually show, and the softmax mechanism needs an empirical analysis (e.g., measuring how much α actually varies across tokens, and whether the ranking is stable across runs).","headline":"Genuinely new noise-relaxation idea, honest limitations, but SOTA claim overreaches and the softmax normalization blurs the claimed mechanism.","tokens_in":22954,"tokens_out":3289,"would_cite":false,"duration_ms":30294,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a noise-allocation training signal — adding graded noise to tokens instead of dropping them — learns token relevance well enough to beat CLS-attention pruning on frozen vision transformers, with no CLS token required.","keywords":["token pruning","vision transformer","information bottleneck","additive noise relaxation","rate-distortion","frozen backbone","similarity-based pruning","ImageNet-1K"],"falsifier":"Keep the full TNT evaluation protocol but replace the trained noise-allocator head with an untrained, randomly initialized head and compare top-$K$ accuracies at the same keep rates: if the gap between trained and untrained rankings is small at $K = 0.3$ or below, the 40-epoch noise training is not the source of the reported gains. A complementary check is to inspect the learned score distribution, since the softmax over roughly 196 tokens pushes each $\\alpha_i$ near $1/N$; if the spread of $(1 - \\alpha_i)$ is negligible, the training-time noise is nearly uniform and the learned scores cannot be carrying the pruning signal.","tokens_in":21898,"feed_emoji":"✂️","tokens_out":13520,"duration_ms":112467,"temperature":0.7,"pith_summary":"This paper claims that a lightweight add-on module can learn which image patches a vision transformer can afford to lose, purely by adding graded noise to tokens during training: no hard dropping, no fine-tuning of the backbone, and no reliance on the CLS token. If the claim holds, the payoff is a cheap way to speed up any existing vision transformer — train one small head for 40 epochs on a frozen model, then at deployment rank tokens by the learned score, keep the top K, and remove redundant patches with a similarity pass. The reported evidence is strongest at aggressive keep rates, where the method is claimed to preserve accuracy far better than stochastic-dropout, token-merging, and CLS-attention baselines at matched FLOPs, while also working on a mean-pooled ViT where CLS-attention methods cannot be applied at all.","feed_headline":"Noise-trained allocator beats CLS-attention pruning at low keep rates","feed_subtitle":"A softmax head learns which image tokens matter by adding noise in training, then drops the rest at test time on frozen backbones.","key_machinery":"The central object is the noise allocator: a linear map $W \\in \\mathbb{R}^{D \\times 1}$ followed by a softmax that converts each token embedding into a relevance score $\\alpha_i$, with training noise $\\eta_i = (1 - \\alpha_i)\\varepsilon$ for $\\varepsilon \\sim \\mathcal{N}(0, \\beta I)$ and a fixed noise scale $\\beta = 0.02$. Its companion at deployment is similarity-based pruning by random partition — tokens are split into two random groups, the closest pair across groups is found by embedding similarity, and the most similar pairs are pruned — which removes redundancy that element-wise relevance scores miss. The information-bottleneck framing ties the two together: the token embeddings stand in for the latent means and $(1 - \\alpha_i)$ for the element-wise standard deviations of the Deep Variational Information Bottleneck, so the softmax-constrained noise budget is the rate term and classification accuracy is the relevance term.","core_discovery":"TNT relaxes discrete token dropping into a continuous optimization problem: a per-token relevance score $\\alpha_i$, computed by a linear layer and softmax attached to a frozen transformer block, controls the amplitude of Gaussian noise $\\eta_i = (1 - \\alpha_i)\\varepsilon$, $\\varepsilon \\sim \\mathcal{N}(0, \\beta I)$, added to that token's embedding. Because the softmax keeps the total noise budget fixed, the network can only protect the most predictive tokens, and so $\\alpha$ becomes a learned estimate of token relevance. At test time the noise is replaced by hard pruning: tokens are ranked by $\\alpha$, the top $K$ are kept, and a random-partition similarity step removes the most redundant among them, recovering the discrete computational savings that the continuous training relaxation hides. The paper frames the procedure as an information-bottleneck trade-off in which the number of kept tokens is the transmission rate and classification accuracy is the relevance, with layer-wise normalization bounding the signal power so the noise budget acts as a genuine rate constraint. On ImageNet-1K with ViT and DeiT backbones — frozen except for the noise-allocator heads — the paper reports accuracy at matched compute that matches or exceeds the CLS-attention Top-K baseline while also succeeding on a mean-pooled ViT, and its largest margins come at the lowest token keep rates.","pith_inferences":["Beyond the paper: the same noise-allocator recipe should transfer to non-classification tasks — detection, segmentation, captioning — that use ViT-style backbones without a CLS token, since the only requirements are a frozen backbone and a differentiable task loss.","Beyond the paper: the training noise scale $\\beta$ and the test-time keep rate $K$ are only loosely connected; sweeping $\\beta$ per layer so the training-time rate constraint matches the deployment keep rate would likely sharpen the transfer of the learned ranking.","Beyond the paper: because redundancy removal is never simulated in training (a limitation the paper states), a soft, differentiable merge during training could teach the allocator to favor tokens with low mutual redundancy, directly addressing that limitation.","Beyond the paper: the rate-distortion framing suggests replacing the hand-tuned per-layer keep rates with a distortion-based allocation — give more kept tokens to layers where marginal accuracy loss is highest — which is a testable alternative schedule."],"forward_implications":["Pruning no longer needs a CLS token or attention supervision: any frozen ViT-style backbone can be retrofitted with a short-trained noise-allocator head, which is what lets the method run on mean-pooled, CLS-free models.","The paper's largest reported margins come at the most aggressive keep rates — multi-layer DeiT-B at about 5.9 GFLOPs holds 59.9% top-1 accuracy versus 56.0% for Top-K, 54.3% for EViT, 43.4% for ToMe, 19.8% for Zero-TP, and 11.2% for DynamicViT.","Because training is limited to the new head on a frozen backbone, the procedure can be applied to already-pretrained checkpoints rather than requiring a new model to be trained from scratch.","Test-time inference still enjoys discrete-dropping savings: the allocator scores tokens, drops the losers, and only the kept tokens flow through the rest of the network.","The information-bottleneck derivation reframes the accuracy/compute trade-off as a rate-distortion choice, giving a principled vocabulary for setting how many tokens each layer may keep."],"supporting_citations":[{"why":"It supplies the deep variational information-bottleneck recipe that TNT maps its noise-allocation objective onto.","marker":"[1]"},{"why":"It provides the information-bottleneck formulation that frames token pruning as a rate-versus-relevance trade-off.","marker":"[17]"},{"why":"It establishes the CLS-attention Top-K baseline that the paper identifies as the strongest prior method and sets out to match without a CLS token.","marker":"[10]"},{"why":"It is the source of the similarity-based pruning step that TNT adopts with random partitioning, and it is also a primary baseline.","marker":"[20]"},{"why":"It is the prototypical stochastic token-dropout method used as the main learned-pruning baseline.","marker":"[16]"},{"why":"It provides the token-merging baseline (ToMe) and the merger alternative tested in TNT's ablation.","marker":"[4]"},{"why":"It contributes the reparameterization trick that makes sampling the token noise differentiable.","marker":"[13]"},{"why":"It defines the ViT architecture, one of the two base model families on which the method is evaluated.","marker":"[7]"}],"fun_headline_variants":["TNT Pruning: train with noise, prune hard at test","Additive noise learns which tokens to keep in ViTs","Rate-distortion optimal token selection via noise","Noise-based token pruning beats CLS-attention at low rates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A relevance ranking learned under gentle additive noise on a frozen backbone remains the correct ranking for removing whole tokens at test time, even when far more tokens are removed than the noise schedule ever simulated and the similarity-pruning step is applied without ever having been part of training.","fun_headline_variants_meta":{"raw":{"variants":["TNT Pruning: train with noise, prune hard at test","Additive noise learns which tokens to keep in ViTs","Rate-distortion optimal token selection via noise","Noise-based token pruning beats CLS-attention at low rates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3177,"prompt_tokens":878,"completion_tokens":2299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2231}},"tokens_in":494,"tokens_out":2299,"duration_ms":14606,"temperature":1.0,"reasoning_tokens":2231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:31:28.373280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Keep the full TNT evaluation protocol but replace the trained noise-allocator head with an untrained, randomly initialized head and compare top-$K$ accuracies at the same keep rates: if the gap between trained and untrained rankings is small at $K = 0.3$ or below, the 40-epoch noise training is not the source of the reported gains. A complementary check is to inspect the learned score distribution, since the softmax over roughly 196 tokens pushes each $\\alpha_i$ near $1/N$; if the spread of $(1 - \\alpha_i)$ is negligible, the training-time noise is nearly uniform and the learned scores cannot be carrying the pruning signal.","supporting_citations":[{"cited_title":"Which tokens to use? investigating token reduction in vision transformers","cited_arxiv_id":null,"evidence_quote":"It establishes the CLS-attention Top-K baseline that the paper identifies as the strongest prior method and sets out to match without a CLS token."},{"cited_title":"Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers","cited_arxiv_id":null,"evidence_quote":"It is the source of the similarity-based pruning step that TNT adopts with random partitioning, and it is also a primary baseline."},{"cited_title":"Dynamicvit: Efficient vision trans- formers with dynamic token sparsification","cited_arxiv_id":null,"evidence_quote":"It is the prototypical stochastic token-dropout method used as the main learned-pruning baseline."}],"review_version":1}