{"id":"f18c57ce-69ce-449d-ae61-6800c1d26705","arxiv_id":"2411.15787","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a vision transformer with multiple auxiliary tokens and distilling them into a single global token improves self-supervised representation quality with no additional inference cost.","lead":"MTE adds several extra helper tokens to a vision transformer during training, learns richer image features from them, and folds their knowledge into the original token so the helpers can be discarded at inference. The paper reports consistent gains over strong self-supervised baselines like DINO, iBOT, and MoCo V3, with no extra inference cost and only about 5% more training compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: MTE's mechanism is internally coherent and gains are plausible, but missing code, missing error bars, and an in-house BYOL baseline keep the empirical claim conditional.","rationale":"I agree with the reader's CONDITIONAL verdict but for a slightly different reason. The reader's weakest assumption centers on token complementarity and distillation completeness; those are the paper's stated mechanism, but the strongest empirical evidence in Tabs. 5-11 and 13-16 shows the method works end-to-end, and the attention mask makes auxiliary-token removal lossless for ViT. The real residual risk is that the comparison is not yet reproducible: no code or checkpoints are released, no variance is reported, and one baseline (BYOL) is implemented in-house. Because the strongest effect sizes are on BYOL, and because the paper claims consistency across all methods, this is the most load-bearing uncertainty. I would not reject the paper; the method is plausible and well-ablated, but the empirical claim should be verified with official baselines and error bars before full acceptance. Hence UNCHANGED relative to the CONDITIONAL reader verdict.","tokens_in":16825,"tokens_out":18204,"duration_ms":177832,"concrete_test":"Reproduce all entries of Tab. 5 with each baseline's official code, identical pre-training schedule (epochs, batch, augmentation, multi-crop), and at least three seeds; run BYOL with its official implementation and predictor. If the official BYOL baseline exceeds 56.1% k-NN / 65.1% linear, recompute BYOL+MTE's gain. Additionally, report seed-level mean and std for k-NN and linear for DINO and iBOT 100-epoch runs; if the 0.3-1.2 point linear gains are within one standard deviation, strengthen the CONDITIONAL verdict.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Reading in good faith, the core construction is sound: the Fig. 4 attention mask makes the ViT global/patch path independent of the auxiliary CLS tokens, and the adaptively pooled tokens are post-encoder, so discarding them at inference is lossless by construction. Eq. 6 is a standard online-distillation objective, and Tabs. 13-16 ablate each component consistently. I do not find an internal inconsistency that would overturn the claim.\n\nThe load-bearing risk is empirical reliability. All headline gains are single runs with no error bars, and the BYOL baseline in Tab. 5 is explicitly implemented with the DINO codebase. Since BYOL uses a predictor head and a different loss, that in-house baseline may be weaker than the official one; the +9.4 k-NN gain over BYOL is the largest in the table and the most vulnerable to this. The other baselines mix official-paper numbers with in-house reproductions, which can introduce recipe mismatch. If the faithful BYOL baseline is closer to DINO, the claim that MTE 'consistently' improves over all five methods weakens.\n\nThe complementarity of auxiliary tokens is only demonstrated on a 300-class subset with qualitative CKA and small linear/NMI gains; however, even if that explanation is imperfect, the method could still improve via extra capacity and ensemble averaging of the training signal. Thus I do not treat complementarity as the decisive fault.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Multi-Token Enhancing (MTE), a plug-in module for vision Transformers that adds M auxiliary CLS tokens and K adaptively pooled tokens during pre-training. These auxiliary tokens are used in the self-supervised or supervised loss through independent projection heads, and their fused representation is distilled online into the global CLS token (Eq. 6). At inference the auxiliary tokens are discarded; for plain ViTs an attention mask prevents the global/patch tokens from depending on the auxiliary CLS tokens, making removal lossless (Table 16). The method is evaluated on ImageNet with DINO, iBOT, MoCo V3, BYOL, MoBY, and Swin backbones, as well as on fine-tuning, detection/segmentation transfer, and a supervised classification setting, reporting consistent gains over baselines.","tokens_in":17192,"tokens_out":9819,"duration_ms":80872,"significance":"If the empirical claims hold, MTE is a low-cost, architecture-agnostic way to obtain ensemble-like improvements in self-supervised pre-training without inference overhead. The paper's main strengths are the breadth of the evaluation (five SSL losses, two architectures, and dense downstream tasks) and the clean construction: the adaptively pooled tokens are post-encoder, and the attention-mask design in Figure 4 is validated by Table 16, so the 'no additional inference costs' claim is convincingly supported. The ablation studies in Tables 13–15 are consistent with the proposed mechanism. The method is not a derivation with fitted constants; M=4 and K=6 are selected by ablation and reported as such. The main uncertainties are empirical reliability: there are no error bars, and the BYOL baseline is in-house. The paper promises code but does not provide it, which limits reproducibility.","major_comments":[{"comment":"The BYOL baseline is implemented using the DINO codebase, as stated in Table 5, rather than the official BYOL recipe. BYOL uses an asymmetric predictor head and stop-gradient on the target, whereas DINO uses a prototype/clustering loss with a momentum teacher. Because the reported +9.4 k-NN improvement over BYOL is the largest in Table 5, this gain may partly reflect a weak baseline. Please either replace the BYOL numbers with the official BYOL ViT-S/16 results under the same 100-epoch protocol, or provide a faithful re-implementation with the BYOL loss and report the exact hyperparameters and code configuration. Without this, the claim that MTE consistently improves over all five self-supervised baselines is not fully supported.","section":"Table 5"},{"comment":"All reported results are single pre-training runs with no variance estimates. Several headline gains are small relative to typical run-to-run variability, e.g., +0.8 linear probing for MoCo V3, +0.6 for iBOT (300 epochs), +0.8 for MoBY in Table 5, and +0.3 for ViT-S/16 fine-tuning in Table 7. Please provide standard deviations over at least three independent runs for the primary comparisons (Table 5 and Table 7), or otherwise justify that the differences are beyond run-to-run variance. If computational budget is limited, reporting two seeds for the smallest gains and the corresponding spread would already strengthen the consistency claim.","section":"Tables 5–11"}],"minor_comments":[{"comment":"Please state explicitly whether the CKA, per-category expertise, and fusion analyses are performed on the 300-class subset used for ablations or on full ImageNet; if the former, note that the mechanism analysis is illustrative and not carried out on the full benchmark.","section":"Section 4.1 / Figures 7–10"},{"comment":"The fine-tuning epoch counts are contradictory: Section 5.2 says ViT-S/16 and ViT-B/16 are fine-tuned for 100 and 200 epochs, respectively, while the Table 7 caption says 200 and 100. Please correct the inconsistency.","section":"Section 5.2 and Table 7 caption"},{"comment":"The unit label 'FLOPS' should be 'FLOPs' (floating-point operations); the table reports quantities in GFLOPs, not operations per second. Please fix the unit.","section":"Table 12"},{"comment":"The paper states that source code will be made publicly available, but no code or detailed pseudocode is included in the submission. Given the many implementation details (attention mask placement, adaptive pooling convolution, TEN, per-token heads), providing code or at least pseudocode in a supplement would aid reproducibility.","section":"Abstract / Section 1"},{"comment":"There is a subscript typo: the classifier weights are written as 'W i 1 ∈ R D×C' but should presumably be 'W i c ∈ R D×C'. Please correct.","section":"Equation (7)"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical methods paper. The two main risks are the in-house BYOL baseline and the absence of error bars; both are addressable within a revision and do not undermine the core construction. I would not require full 300-epoch re-runs for all experiments; error bars on the smallest gains in Table 5 and Table 7 would suffice. Code release should be a condition of acceptance. The self-citation pattern is not problematic; the cited works are used as components (large-kernel convolutions, HSSL). The paper fits the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper earns its conditional accept. The idea is simple—train a ViT to output multiple auxiliary CLS tokens and adaptively pooled tokens, let them share the SSL loss, and distill them into the global token during pre-training—and the attention mask in Fig. 4 is the piece that makes it work: it prevents the global token and patch tokens from attending to the auxiliary tokens, so removing them at inference is lossless. Table 16 validates exactly that, and it is the right experiment to run.\n\nWhat is genuinely new is the specific combination. Extra CLS tokens, adaptive pooling, and distillation are all known ingredients, but putting them together with the masking trick into a no-inference-cost boost is not in the cited literature. The paper does a broad job: DINO, iBOT, MoCo V3, BYOL, MoBY, Swin, plus supervised learning, and the gains are consistent—roughly 0.8 to 4.0 points on k-NN/linear and solid improvements on COCO and ADE20K. The ablation structure (components, pooling type, token counts, independent heads) is coherent and matches the story.\n\nThe soft spots are real but not fatal. No code and no error bars; every headline number is a single run. The BYOL baseline in Tab. 5 is implemented in-house on the DINO codebase, and the +9.4 k-NN gain over it is the largest in the table, so that number is the most suspect. If a faithful BYOL baseline is closer to DINO, 'consistently improves over all five' loses some force, but the DINO, iBOT, and MoCo V3 gains are more credible because those baselines are official or near-official. The complementarity analysis is qualitative and on a 300-class subset, but even if that explanation is imperfect, the method could still help through extra capacity and ensemble-averaged training signal, so I do not treat that as a decisive fault. The self-citations for large-kernel convolutions and HSSL are component attributions, not circular.\n\nBottom line: this paper is for anyone doing SSL pre-training on ViTs. It deserves a serious referee and should be accepted after code release and error bars—or at minimum a statement of how many seeds. Desk rejection would be wrong.","headline":"MTE is a well-tested plugin for SSL pre-training: multiple auxiliary tokens distilled into a single global token with a clean attention-masking trick, and the main weakness is missing reproducibility artifacts, not the mechanism.","tokens_in":17640,"tokens_out":2121,"would_cite":true,"duration_ms":19332,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single vision transformer can create its own ensemble with extra tokens, then distill them away, leaving a stronger model at zero inference cost.","keywords":["multi-token ensemble","self-supervised learning","vision transformer","auxiliary tokens","online distillation","adaptive pooling","representation learning","knowledge distillation"],"falsifier":"Train the same MTE recipe but with all auxiliary CLS tokens sharing one initialization and all pooling convolutions sharing one weight initialization, keeping every other detail fixed; if the reported k-NN gain over DINO (61.2 to 66.6 in Table 5) largely disappears, the claim that auxiliary tokens are complementary by construction is falsified.","tokens_in":16665,"feed_emoji":"🧩","tokens_out":6066,"duration_ms":50842,"temperature":0.7,"pith_summary":"The paper claims that a single vision transformer can supply its own ensemble: by adding several auxiliary tokens (extra CLS tokens with different initializations and adaptively pooled tokens from patch features), and fusing their losses during pre-training, the model learns richer representations. It then distills the fused auxiliary knowledge into the global CLS token online, so the auxiliary tokens can be discarded with no loss at inference and no added inference cost. Across DINO, iBOT, MoCo V3, BYOL, and MoBY, the paper reports consistent k-NN and linear-probing gains on ImageNet, and improved transfer to iNaturalist, COCO, and ADE20K. The contribution is a general plugin rather than a new loss family, minimally invasive to existing self-supervised pipelines.","feed_headline":"Extra tokens inside one model give free accuracy gains","feed_subtitle":"MTE distills a self-ensemble into the global token, boosting self-supervised vision without extra inference cost.","key_machinery":"The load-bearing mechanism is the set of auxiliary tokens plus the distillation loss. Auxiliary CLS tokens are additional trainable class tokens with distinct random initializations, fed through the same transformer blocks; adaptively pooled tokens are computed from patch tokens using $K$ different learned pooling-weight maps generated by large-kernel convolutions. A Token Enhancing (TEN) module applies cross-attention to auxiliary CLS tokens to produce a stronger teacher. The total pre-training loss is $\\mathcal{L}=\\mathcal{L}(\\hat{h}_t,\\tilde{h}_t)+\\mathcal{L}(\\hat{h}_t,\\tilde{h}_c)$ (Eq. 6), where the second term distills the fused auxiliary-token representation into the global token; an attention mask prevents the global token and patches from depending on the auxiliary tokens, so discarding them at inference is lossless.","core_discovery":"The central discovery is that ensemble-like diversity can be manufactured inside one encoder. With $M$ auxiliary CLS tokens and $K$ adaptively pooled tokens, each token develops its own expertise on different categories, shown by CKA differences and per-category accuracy analyses; fusing their outputs gives a stronger teacher than any single token. The online distillation term $\\mathcal{L}(\\hat{h}_t,\\tilde{h}_c)$ makes the global CLS token imitate that teacher, and because the global token shares the encoder with the auxiliary tokens, the optimization is reciprocal: distillation improves the auxiliary tokens as well as the global token. After pre-training the auxiliary parts are removed, leaving an unchanged model whose representations are stronger than the baseline's.","pith_inferences":["Because the paper only tests complementarity at the final token level, a natural next experiment is to apply the same auxiliary-token ensemble at intermediate transformer blocks; if per-token expertise is generic, mid-level tokens might give even larger dense-prediction gains.","The per-category expertise analysis suggests MTE could be a cheap fix for fine-grained and long-tail recognition; the largest relative gains might appear where different tokens specialize on rare classes, though the paper only measures aggregate accuracy.","The auxiliary tokens are removed after pre-training, but the same online-distillation idea could be reused during fine-tuning to transfer a teacher's diversity into a student without changing the deployed architecture."],"forward_implications":["Adopting MTE changes only the pre-training forward pass (roughly 6.1G to 6.4G FLOPs for a 224×224 image); the deployed model is identical to the baseline, so any downstream pipeline inherits the gains for free.","On DINO, 300 epochs with MTE already match the 800-epoch baseline's k-NN and linear accuracy, so the technique compresses long pre-training schedules.","The gains transfer beyond classification: iBOT+MTE pre-trained for 100 epochs improves COCO detection and segmentation mAP by about 0.9 and 1.0 points, respectively, and ADE20K mIoU by 1.2 points.","The method extends to supervised training and to non-CLS architectures such as Swin, where six adaptively pooled tokens are used with an average-pooled global token."],"supporting_citations":[{"why":"Supplies the clustering-based DINO baseline and the prototype-layer evaluation that MTE builds on.","marker":"[5]"},{"why":"Provides the vision transformer architecture and the CLS-token design that MTE extends with auxiliary tokens.","marker":"[12]"},{"why":"The iBOT baseline whose pre-training and fine-tuning recipes MTE adopts and improves.","marker":"[66]"},{"why":"The instance-discrimination formulation of two augmented views that Section 3.1 uses as the base learning setup.","marker":"[23]"},{"why":"A contrastive baseline (MoCo V3) that MTE improves by 0.8% linear probing in Table 5.","marker":"[9]"},{"why":"A siamese-network baseline (BYOL) whose loss is implemented in the DINO codebase to test MTE with similarity losses.","marker":"[18]"},{"why":"The Swin-transformer baseline (MoBY) that MTE boosts by 4.0% k-NN in Table 5.","marker":"[58]"},{"why":"Supplies the multi-crop augmentation setting used in Table 6 and the self-clustering initialization.","marker":"[4]"},{"why":"The CKA measure used in Section 4.1 to quantify auxiliary-token differences.","marker":"[28]"}],"fun_headline_variants":["One model, many tokens, zero extra inference cost","Token ensemble in a single encoder boosts self-supervised learning","Self-distilled token ensemble sharpens vision representations","Multi-token trick fuses ensemble power at no inference overhead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire no-inference-cost gain rests on the empirical premise that differently initialized auxiliary tokens stay complementary instead of collapsing into the same representation, and that the online distillation term transfers all of their task-relevant knowledge into the global token.","fun_headline_variants_meta":{"raw":{"variants":["One model, many tokens, zero extra inference cost","Token ensemble in a single encoder boosts self-supervised learning","Self-distilled token ensemble sharpens vision representations","Multi-token trick fuses ensemble power at no inference overhead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000115,"raw_usage":{"total_tokens":1035,"prompt_tokens":869,"completion_tokens":166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":101}},"tokens_in":485,"tokens_out":166,"duration_ms":2667,"temperature":1.0,"reasoning_tokens":101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:53:30.927149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same MTE recipe but with all auxiliary CLS tokens sharing one initialization and all pooling convolutions sharing one weight initialization, keeping every other detail fixed; if the reported k-NN gain over DINO (61.2 to 66.6 in Table 5) largely disappears, the claim that auxiliary tokens are complementary by construction is falsified.","supporting_citations":[{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the clustering-based DINO baseline and the prototype-layer evaluation that MTE builds on."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Provides the vision transformer architecture and the CLS-token design that MTE extends with auxiliary tokens."},{"cited_title":"ibot: Image bert pre-training with online tokenizer","cited_arxiv_id":null,"evidence_quote":"The iBOT baseline whose pre-training and fine-tuning recipes MTE adopts and improves."},{"cited_title":"Momentum contrast for unsupervised visual rep- resentation learning","cited_arxiv_id":null,"evidence_quote":"The instance-discrimination formulation of two augmented views that Section 3.1 uses as the base learning setup."},{"cited_title":"An empiri- cal study of training self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"A contrastive baseline (MoCo V3) that MTE improves by 0.8% linear probing in Table 5."},{"cited_title":"Richemond, Elena Buchatskaya, Carl Doersch, Bernardo ´Avila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko","cited_arxiv_id":null,"evidence_quote":"A siamese-network baseline (BYOL) whose loss is implemented in the DINO codebase to test MTE with similarity losses."},{"cited_title":"Unsupervised learn- ing of visual features by contrasting cluster assignments","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-crop augmentation setting used in Table 6 and the self-clustering initialization."},{"cited_title":"Similarity of neural network representa- tions revisited","cited_arxiv_id":null,"evidence_quote":"The CKA measure used in Section 4.1 to quantify auxiliary-token differences."}],"review_version":1}