{"id":"47f72614-7df3-4ab8-abed-bec0279f6cdf","arxiv_id":"2505.04375","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under symmetric label noise, the larger ViTl32 model consistently outperforms smaller and higher-token-count variants in both accuracy and calibration, while Swin transformers lag behind.","lead":"This paper benchmarks Vision Transformer and Swin Transformer configurations under symmetric label noise in active learning, finding that the large ViT with 32x32 patches (ViTl32) is the most robust and efficient choice. It also reports that information-based active learning strategies improve accuracy at moderate noise levels but generally hurt calibration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed 20-epoch protocol and single-run results make the ViTl32-vs-ViTl16 ranking unproven; a per-model tuned multi-seed comparison could reverse the headline.","rationale":"The reader's conditional verdict correctly identifies the load-bearing weakness: the comparison protocol described in Section 4.2 is shared across models of very different token counts, and all headline tables (Tables 2-9) report single-run numbers without variance estimates. The 'consistently' language in the abstract and conclusion is therefore not supported for the smaller effect sizes, especially the ViTl16-vs-ViTl32 comparison, where the ViTl16 model has 4x the tokens and a higher reported training time (228s vs 90s in Table 1) yet is claimed to be uniformly worse. My analysis found no internal contradiction in the reported aggregate numbers, and the efficiency argument for ViTl32 over ViTl16 is plausible. However, because the entire practical recommendation hinges on this ranking, the missing seed-level variance and the unspecified early-stopping criterion constitute a genuine correctness risk, not merely a stylistic issue. I also note secondary concerns (the Swin complexity equations (3)-(6) are inconsistent with standard Swin attention, and the GCI_ViTAL comparison may use label smoothing while other strategies do not), but neither is as load-bearing as the protocol fairness issue. The proposed concrete test directly targets the central claim and would settle whether the ViTl32 advantage is robust or a training-budget artifact. Since the reader's verdict is already CONDITIONAL, I recommend keeping it unchanged rather than escalating to REJECT: the findings are plausible and potentially useful, but they require the additional evidence before being accepted as guidance.","tokens_in":14630,"tokens_out":4664,"duration_ms":50645,"concrete_test":"Rerun the ViTl16 and ViTl32 comparison on CIFAR100 under noise rates 0.0, 0.3, and 0.6 with random acquisition, using at least 5 seeds and per-model hyperparameter selection: search learning rate over {1e-5, 3e-5, 1e-4}, use a proper held-out validation split for early stopping, and allow training until validation convergence rather than a fixed 20 epochs. Report mean and 95% CI for test accuracy and Brier score with compute budget matched or explicitly reported. If ViTl16's mean accuracy and calibration match or exceed ViTl32 under this tuned protocol, the paper's central ranking is an artifact of the shared under-training schedule; if ViTl32 still wins, the claim is substantially strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ViTl32 'consistently' outperforms ViTl16 in accuracy, calibration, and efficiency rests on a single, shared fine-tuning protocol described in Section 4.2: 20 epochs, early stopping with 10-epoch tolerance, batch size 256, and an unspecified learning-rate schedule. The results in Tables 2-9 are reported without variance, error bars, or repeated seeds. This is not a neutral protocol for comparing models with 4x different token counts: ViTl16 processes 196 tokens per image versus 49 for ViTl32, so a fixed epoch budget can systematically under-train the token-heavy model. The reported accuracy gap between ViTl16 and ViTl32 is often small (e.g., 0.42 percentage points on CIFAR10 at zero noise, 0.60 on CIFAR100 at zero noise), and ViTl16 may simply need more training or a different learning rate to realize its finer-grained representations. The early-stopping criterion is also not specified (no validation split is mentioned), so it is unclear whether the recorded numbers reflect fair convergence or an artifact of the stopping rule. A modest amount of extra training for ViTl16, or a different LR schedule, could plausibly close or reverse the gap, which would directly undermine the paper's primary practical recommendation. The direction of the reported effect is plausible and consistent across datasets and noise levels, so this is not a refutation; it is an unverified assumption about the fairness of the comparison protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper experimentally investigates how Vision Transformer (ViT) and Swin Transformer configurations of different model sizes and patch sizes behave under symmetric label noise in a deep active learning setting. The authors fine-tune four ViTs (base/large, 16x16/32x32 patches) and three SwinV2 variants on CIFAR10 and CIFAR100, using random, entropy-based, and GCI_ViTAL acquisition strategies at noise rates from 0% to 90%. They report top-1 accuracy, Brier score, and training time, and conclude that larger ViTs, especially ViTl32, outperform smaller counterparts in accuracy and calibration even under moderate to high label noise; that ViTl16 is inefficient relative to ViTl32; that SwinV2 models are less robust; and that information-based active learning yields only modest accuracy gains at moderate noise while worsening calibration at high noise.","tokens_in":14935,"tokens_out":5958,"duration_ms":55223,"significance":"The question addressed is practically relevant: selecting a transformer model for fine-tuning under label noise and limited labeling budgets is a common real-world problem, and the paper provides a broad empirical sweep across model sizes, patch sizes, noise rates, and acquisition strategies. The inclusion of calibration (Brier score) and training time alongside accuracy is a strength, as is the explicit evaluation of active learning beyond the usual random/entropy baselines. If the findings are confirmed with proper statistical support, the recommendation to prefer ViTl32 over ViTl16 under these constraints and the caution about information-based strategies at high noise would be useful guidance. However, the current single-run protocol and the absence of uncertainty quantification mean that the headline ranking is not yet established; the paper also contains a technical error in its complexity analysis of SwinV1 that needs correction.","major_comments":[{"comment":"All accuracy and Brier-score results are reported as single-run values without error bars or significance tests. The headline model ranking rests on small differences, for instance ViTl32 exceeds ViTl16 by 0.42 percentage points on CIFAR10 and 0.60 percentage points on CIFAR100 at zero noise; these gaps are within the variation expected across random seeds and DAL cycles. The claim that ViTl32 'consistently outperforms' its smaller counterpart is therefore not statistically supported in the present form. The authors should provide repeated-seed averages with standard deviations or confidence intervals, and ideally significance tests, for the main model-size and patch-size comparisons.","section":"§5.1, Tables 2 and 3"},{"comment":"The comparison relies on a single shared fine-tuning protocol: 20 epochs, early stopping with a 10-epoch tolerance, batch size 256, and an unspecified learning-rate schedule. Because ViTl16 sees four times as many tokens as ViTl32 (196 vs. 49 per image), a fixed epoch budget can systematically under-train the token-heavy model, so the observed ranking may reflect an artifact of the training protocol rather than a property of the architectures. The paper should either tune hyperparameters per model on a validation split, report learning curves showing that all models reach convergence, or conduct a sensitivity analysis over training budgets; the validation split used for early stopping is also not described.","section":"§4.2, §5.1"},{"comment":"The complexity analysis misrepresents SwinV1: Equation (3) attributes O(N^2) complexity to SwinV1 self-attention, but SwinV1 already uses windowed self-attention with per-window complexity O(W^2) and overall complexity O(N·W^2), not full quadratic complexity in the number of tokens N. Equation (6)'s expression O(N·W'^2) is essentially the same windowed-attention complexity, not a new linear-attention mechanism. This technical error undermines the motivation for choosing SwinV2 and should be corrected or the passage removed.","section":"§3.1, Eqs. (3)–(6)"},{"comment":"The abstract and conclusion state that information-based active learning strategies provide 'meaningful accuracy improvements' at moderate label noise rates, but the measured improvements over random are at most 0.38 percentage points on CIFAR10 and 0.48 percentage points on CIFAR100, with several negative entries in the moderate noise range. Without repeated runs and significance testing, these differences are not distinguishable from run-to-run noise. The claim should be qualified or supported with statistical evidence.","section":"§5.1, Tables 4 and 5; §6"}],"minor_comments":[{"comment":"The captions of Tables 4 and 5 contain the typo 'Secion'; it should be 'Section'.","section":"Tables 4 and 5"},{"comment":"Reference [14] lists the first author as Kolesnikov, but 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale' is normally attributed to Dosovitskiy et al.; please correct the author order and the citation.","section":"References, [14]"},{"comment":"The sentence citing reference [27] as a previous work that adopts ViTs for DAL appears mismatched, since [27] is a survey of transformers in medical image analysis and does not describe an active-learning query strategy; please revise the sentence or cite an appropriate primary source.","section":"§2.2"},{"comment":"The notation for the concatenation of attention heads is ambiguous: the symbol with a superscript h is not defined as concatenation; please introduce standard notation explicitly.","section":"§3.1, Eq. (7)"},{"comment":"The appendix figures (Figures 5–15) are not referenced in the main text; add pointer sentences where the corresponding results are discussed.","section":"Appendix figures"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the authors' prior work [13] (GCI_ViTAL) and partly re-confirms that this self-cited method beats random at moderate noise; the novelty over [13] should be clarified. The absence of code, seeds, and variance estimates is a reproducibility concern for an empirical study of this scope. Given the load-bearing issues in the training protocol and statistical support, I recommend major revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a practical benchmark, not a new method, and the headline finding — ViTl32 beats ViTl16 under symmetric label noise in active learning — is plausible but rests on a single-run, under-specified training protocol. Worth refereeing, but not worth taking at face value.\n\nWhat is actually new: the interaction between model size, patch size, symmetric label noise, and active learning budget for ViT and Swin models on CIFAR-10/100. The ViTl16/ViTl32 inversion and the calibration ordering across acquisition strategies are not in the cited prior work. The comparison is systematic: seven models, noise rates 0–0.9, three query strategies, several labeled-data proportions, and both accuracy and Brier score reported. That is a reasonable scope for a benchmark.\n\nSoft spots, roughly in order.\n\nFirst, the central comparison may be unfair. Section 4.2 gives one fixed fine-tuning protocol: 20 epochs, early stopping with 10-epoch tolerance, batch size 256, and no learning-rate schedule. ViTl16 sees 196 tokens per image; ViTl32 sees 49. A shared budget can under-train the finer-grained model. There are no repeated seeds, error bars, or significance tests. The zero-noise CIFAR10 gap is 0.42 points, which is below what you can distinguish from a single run. The direction is consistent across noise levels and datasets, so the result is credible, but “consistently outperform” is stronger than the reported evidence allows.\n\nSecond, no code or data are included. For an empirical-claims paper that is a real obstacle; the tables cannot be checked or extended.\n\nThird, the Swin attention-complexity discussion in Section 3.1 is wrong. Equation (3) assigns O(N^2) to SwinV1, which uses windowed attention and is effectively linear in N for fixed windows. The models are called SwinV2 but the citation is the original SwinV1 paper. This is peripheral to the experiments, but it suggests the methodology section was not carefully reviewed.\n\nThe self-citation of GCI_ViTAL is not a problem per se, but since the paper partly re-confirms that prior method, independent validation would help.\n\nWho this is for: practitioners choosing a ViT for low-budget fine-tuning with noisy labels. The practical advice — prefer ViTl32 over ViTl16, treat Swin as less robust — is reasonable and likely useful even if the exact ordering needs confirmation. A serious referee should ask for multi-seed results, a protocol that controls for convergence (per-model early stopping on a held-out split), learning-rate details, and code/data.\n\nI would not desk-reject this. I would send it to review with a request for major revision.","headline":"A useful but under-powered empirical benchmark; the ViTl32-over-ViTl16 result is plausible yet not proven by single-run, fixed-protocol numbers.","tokens_in":15467,"tokens_out":4249,"would_cite":false,"duration_ms":40967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under symmetric label noise, the large-patch ViTl32 sustains higher accuracy and calibration than smaller ViTs and Swin transformers while using less compute than ViTl16.","keywords":["Vision Transformer","Active Learning","Label Noise","Symmetric Label Noise","Model Calibration","Brier Score","Patch Size","Model Efficiency"],"falsifier":"Re-run the same experiments with per-model hyperparameter optimization and multiple random seeds, then check whether ViTl32 still beats ViTl16 on both accuracy and Brier score by a margin larger than the run-to-run spread; if not, the claimed ranking rests on the fixed training recipe.","tokens_in":14434,"feed_emoji":"🎯","tokens_out":7650,"duration_ms":71036,"temperature":0.7,"pith_summary":"This paper asks how model size and patch size change a vision transformer's behavior when training labels are noisy and the labeling budget is small. By injecting symmetric label noise at rates from 0% to 90% into CIFAR-10 and CIFAR-100 active-learning runs, it compares four ViT configurations (base/large, 16×16 and 32×32 patches) and three SwinV2 configurations. The intended claim is that the largest ViT configuration with the larger patch, ViTl32, is the most robust and efficient choice: it beats smaller ViTs on accuracy and calibration under moderate to high noise, while ViTl16 buys no accuracy with its extra tokens and higher compute. The paper also argues that SwinV2 models lag behind ViTs under noise, and that information-based active-learning strategies help accuracy only at moderate noise while harming calibration relative to random selection at high noise. If these results hold, they give practitioners a concrete model-selection rule for low-budget, noisy-label fine-tuning.","feed_headline":"ViTl32 beats smaller ViTs and Swin under noisy labels","feed_subtitle":"With up to 90% label noise, the large 32-patch ViT stays accurate and well-calibrated at lower cost than ViTl16.","key_machinery":"The load-bearing comparison is the model grid: four ViT configurations that vary embedding dimension (768 base vs 1024 large) and patch size (16×16 vs 32×32), plus three SwinV2 configurations, all fine-tuned from ImageNet-1k under one shared protocol (20 epochs, early stopping, batch size 256) while symmetric label noise is injected into the training set at rates 0.0–0.9. The mechanism that carries the argument is the interaction between token count (patch size) and model capacity (embedding/layer count), measured through top-1 accuracy, Brier score for calibration, and per-cycle training time. Three acquisition strategies — random, entropy, and the ViT-specific GCI_ViTAL — are used to show that query strategy changes the accuracy/calibration trade-off less than model choice does under noise.","core_discovery":"The paper's central discovery is a model-size ranking under symmetric label noise: within the ViT family, capacity in embedding dimension matters more than token density, so ViTl32 (1024-dim embedding, 32×32 patches) consistently beats ViTl16 (same embedding, 16×16 patches) and both base ViTs in top-1 accuracy and Brier-score calibration across noise rates on CIFAR-10 and CIFAR-100, even as noise reaches 90%. ViTl16's smaller patch creates more tokens and roughly two and a half times the training time of ViTl32 without improving results. SwinV2 transformers train at comparable speed but remain less accurate and less calibrated under noise. When active-learning acquisition is varied, random selection is the calibration-safe baseline: entropy and GCI_ViTAL improve accuracy only in the 30–60% noise band and degrade calibration at high noise. The paper states this as evidence that choosing the largest, highest-capacity model is not automatically the best strategy; the best practical choice is the one that balances accuracy, calibration, and compute.","pith_inferences":["Inference: The ViTl32 advantage may be partly a regularization effect of fewer tokens and larger patches under noise, rather than raw capacity; a controlled ablation that keeps embedding size fixed while varying patch size could separate these causes.","Inference: Because the shared fine-tuning protocol was not tuned per model, per-model hyperparameter optimization might change the ranking; testing with tuned schedules would tell whether ViTl32's dominance is robust.","Inference: The calibration penalty of information-based acquisition suggests a natural extension: acquisition functions that optimize a proper scoring rule or expected calibration error, rather than accuracy alone, may preserve both gains at high noise.","Inference: The results were obtained on 32×32 natural images resized to 224; on higher-resolution or domain-shifted data where local detail matters, smaller patches may regain their usual advantage, so the ViTl32 result should be tested beyond CIFAR."],"forward_implications":["A practitioner with a low labeling budget and noisy annotations should prefer ViTl32 over ViTl16, since it matches or beats the smaller-patch model while cutting training time.","Designers should not assume smaller patches are always better: the 32×32 large ViT dominates the 16×16 large ViT across noise levels.","Choosing an information-based acquisition strategy such as entropy or GCI_ViTAL is only justified in the moderate-noise band (roughly 30–60%); at high noise it sacrifices calibration for little accuracy gain.","SwinV2 hierarchies are not a drop-in robustness upgrade over plain ViTs under symmetric noise; they offer comparable speed but weaker accuracy and calibration.","The data show size differences persist at 70–90% noise, especially on CIFAR-100, so extreme noise does not fully erase model-capacity effects."],"supporting_citations":[{"why":"Supplies the GCI_ViTAL acquisition strategy used in the experiments and the prior result that ViTs outperform CNNs in active learning under label noise.","marker":"[13]"},{"why":"Defines the ViT architecture and the base/large, 16×16/32×32 configurations whose size comparisons carry the paper's main claim.","marker":"[14]"},{"why":"Defines the Swin transformer family used as the comparison architecture; its SwinV2 variants are the baseline that lags in accuracy and calibration.","marker":"[15]"},{"why":"Introduces the Brier score, the proper scoring rule used to measure calibration throughout the experiments.","marker":"[16]"},{"why":"Establishes the patch-size/compute scaling relationship that the paper tests under label noise and uses to interpret the ViTl16 vs ViTl32 result.","marker":"[29]"},{"why":"Shows how architecture and data selection interact in deep active learning, motivating the paper's controlled comparison of model size under noise.","marker":"[10]"}],"fun_headline_variants":["Bigger ViT beats finer patches for noisy-label accuracy","ViTl32 outshines ViTl16 under 90% label noise","Active learning hurts calibration at high label noise","Swin transformers lag ViTs in noisy-label robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes that one fixed fine-tuning recipe with a single run per setting is enough to compare models fairly, so if a different learning rate or schedule favored ViTl16, the ordering could change.","fun_headline_variants_meta":{"raw":{"variants":["Bigger ViT beats finer patches for noisy-label accuracy","ViTl32 outshines ViTl16 under 90% label noise","Active learning hurts calibration at high label noise","Swin transformers lag ViTs in noisy-label robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2878,"prompt_tokens":1065,"completion_tokens":1813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":1744}},"tokens_in":681,"tokens_out":1813,"duration_ms":14239,"temperature":1.0,"reasoning_tokens":1744,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:30:14.889954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same experiments with per-model hyperparameter optimization and multiple random seeds, then check whether ViTl32 still beats ViTl16 on both accuracy and Brier score by a margin larger than the run-to-run spread; if not, the claimed ranking rests on the fixed training recipe.","supporting_citations":[{"cited_title":"GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise","cited_arxiv_id":"2411.05939","evidence_quote":"Supplies the GCI_ViTAL acquisition strategy used in the experiments and the prior result that ViTs outperform CNNs in active learning under label noise."},{"cited_title":"Kolesnikov, A","cited_arxiv_id":null,"evidence_quote":"Defines the ViT architecture and the base/large, 16×16/32×32 configurations whose size comparisons carry the paper's main claim."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Defines the Swin transformer family used as the comparison architecture; its SwinV2 variants are the baseline that lags in accuracy and calibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Brier score, the proper scoring rule used to measure calibration throughout the experiments."},{"cited_title":"Scaling vision transformers","cited_arxiv_id":null,"evidence_quote":"Establishes the patch-size/compute scaling relationship that the paper tests under label noise and uses to interpret the ViTl16 vs ViTl32 result."},{"cited_title":"On the interdependence between data selection and architecture optimization in deep active learning","cited_arxiv_id":null,"evidence_quote":"Shows how architecture and data selection interact in deep active learning, motivating the paper's controlled comparison of model size under noise."}],"review_version":1}