{"id":"48194dd7-1a1b-4357-97fc-397ccdf69f56","arxiv_id":"2501.06040","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MSCViT, a small hybrid ViT with multi-scale self-attention and wavelet-based convolutional fusion, reaches 84.68% on CIFAR-100 from scratch with 14.0M parameters.","lead":"This paper presents MSCViT, a compact vision transformer that combines multi-scale attention, depthwise convolutions, and wavelet convolution for image classification on tiny datasets. It reports 84.68% top-1 accuracy on CIFAR-100 with 14.0M parameters and 2.5 GFLOPs, trained from scratch without large-scale pretraining.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts its own 'same setting' claim: four baselines are copied from original papers with different resolutions and schedules, so the claimed superiority over similar-size models is unsupported.","rationale":"The reader identified the load-bearing weakness as baseline training parity in Table 3, and I agree that this is the most direct threat to the central claim. My review finds additional supporting evidence: the asterisked entries make the 'same setting' statement false as written, and the ablation baseline in Table 5 exactly matches the Shunted-T published result, which casts doubt on whether the experimental pipeline is a controlled comparison. I also noted that the kernel-size ablation (Table 6) concludes the best configuration is 5x5 in the first two stages and 3x3 in the last two, whereas Table 2's architecture uses 3x3 in the first two and 5x5 in the last two; this internal inconsistency reinforces the need for code release and reproducibility checks. These issues do not prove the central claim false, but they do mean the paper, as submitted, does not substantiate its headline superiority over comparable models. The reader's CONDITIONAL verdict already requires code, parity clarification, and multiple seeds, so I do not change the verdict; I only sharpen the reason: the comparison is not currently controlled, and at least one ablation row appears to reproduce a competitor's published numbers exactly.","tokens_in":13721,"tokens_out":14138,"duration_ms":130392,"concrete_test":"Request the authors' training code and logs, and independently reproduce Table 3 by retraining the four asterisked baselines (CCT-7/3x1, HSViT-C3A4, HSViT-C4A8, Swin-T) under the protocol described in Section 4.2: 224x224 input, DeiT augmentation, AdamW, lr 5e-4, weight decay 0.05, 300 epochs, batch 128. If the reproduced accuracies differ materially from the copied values (e.g., Swin-T moves above 80 on CIFAR-100), the same-setting claim is false and the claimed margins shrink. Additionally, check whether Table 5's 'none' backbone reproduces Shunted-T's exact 2.1 GFLOPs and 81.66 accuracy when run with the released code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4.3.1) that MSCViT 'exceeds all other models of similar sizes' on tiny datasets rests entirely on Table 3, which states 'All methods are tested and compared under the same setting.' This is contradicted by the table itself: CCT-7/3x1, HSViT-C3A4, HSViT-C4A8, and Swin-T are marked with an asterisk, indicating their results are taken from the original papers. Those papers used their own training recipes and, typically, native input resolutions (e.g., 32x32 or 64x64 for CIFAR) rather than the 224x224 resolution used for MSCViT (Section 4.2, Table 2). The reported GFLOPs for the asterisked entries therefore are not on a comparable computational basis with MSCViT's 2.5 GFLOPs at 224x224. If those baselines were retrained under the paper's exact protocol, their accuracies and the margins in Table 3 would change, potentially erasing the claimed superiority. A second red flag is that the 'none' ablation row in Table 5 reports GFLOPs 2.1 and accuracy 81.66, which exactly match the Shunted-T row in Table 3; this suggests the ablation numbers were not produced by running the paper's own backbone as described. Without a demonstration of training parity, the headline superiority claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MSCViT, a compact vision-transformer backbone designed for image classification on small ('tiny') datasets. The architecture combines three modules: a Local Feature Extraction (LFE) block that replaces positional encoding, a Lightweight Multi-scale Self-Attention (LMSSA) module that reduces token counts via per-head fusion coefficients, and a Convolutional Feature Fusion (CFF) module that mixes wavelet-based and standard convolutions on selected channels. The authors report top-1 accuracies of 80.11%, 83.44%, and 84.68% on CIFAR-100 for MSCViT-T, -XS, and -S with 3.8M, 7.8M, and 14.0M parameters and 0.5, 1.0, and 2.5 GFLOPs, respectively, all trained from scratch with DeiT augmentation. Results are also reported on CIFAR-10, Flowers102, Chaoyang, Oxford-IIIT Pet, and Tiny ImageNet, together with ablations of the three modules. The central claim is that MSCViT exceeds all comparable-size ViT and hybrid models on tiny datasets (Section 4.3.1).","tokens_in":14053,"tokens_out":4759,"duration_ms":45484,"significance":"If the reported comparisons are fair, the paper offers a parameter-efficient ViT variant that is competitive on small data, a practically useful direction. The paper has several strengths: it evaluates on multiple tiny datasets, includes ablations that support a contribution from each proposed module, reports computational costs explicitly, and trains from scratch without large-scale pretraining. The architecture is clearly described and the reported numbers are internally consistent. However, the load-bearing comparison claim currently rests on a table that is not fully self-consistent about training protocols, and the ablation baseline appears to coincide with an existing model, so the evidence for the headline superiority claim is not yet convincing. Reproducibility would also be materially improved by releasing code and reporting variance over multiple runs.","major_comments":[{"comment":"The statement 'All methods are tested and compared under the same setting' is contradicted by the table itself: the entries for CCT-7/3x1, HSViT-C3A4, HSViT-C4A8, and Swin-T are marked with an asterisk, indicating they are taken from the original papers. Those papers use different training schedules, augmentations, and native input resolutions (for example, 32x32 or 64x64 for CIFAR), so their GFLOPs and accuracies are not on the same computational or training basis as the 224x224 MSCViT runs. Because the headline claim that MSCViT 'exceeds all other models of similar sizes' rests entirely on this table, that claim is unsupported until the asterisked baselines are retrained under the paper's exact protocol (Section 4.2) or the comparison is reworked to state the protocol differences explicitly and the claim is narrowed accordingly.","section":"Section 4.3.1, Table 3"},{"comment":"The first row of the ablation table reports GFLOPs of 2.1 and accuracy of 81.66, which are exactly the values listed for Shunted-T in Table 3. This exact coincidence suggests that the ablation baseline is not the paper's own backbone as described in Section 3.1 but rather a Shunted-T model. The paper must specify the exact backbone used in Section 4.4.1; if it is Shunted-T, the authors need to justify why that is the appropriate base for isolating the contributions of LFE, LMSSA, and CFF, and the interpretation of the ablation attributions would need to be revised. As written, the ablation does not clearly demonstrate gains over the proposed architecture's own default configuration.","section":"Table 5, row 'none'"},{"comment":"All accuracy values are reported as single numbers with no error bars, no multiple seeds, and no significance tests. Several comparisons involve small margins, such as the 0.18-point difference in Table 7 and the 0.08-point difference in Table 6. On small datasets like CIFAR-100, single-run differences of this size are typically within run-to-run variance. The paper should report mean and standard deviation over at least three independent training runs for the main comparisons and the ablations; otherwise, the claimed margins, especially the fine-grained ones, are not statistically grounded.","section":"Tables 3-5, 7"},{"comment":"The design choices central to the architecture, including the fusion coefficients R_i in LMSSA, the CFF kernel-size schedule (5x5 in early stages and 3x3 in later stages), and the stage depths, appear to have been selected using experiments on CIFAR-100 (for example, Table 6 chooses the kernel schedule and Table 2 fixes the R_i values). CIFAR-100 is also the dataset on which the main superiority claim is made. This creates a selection loop: the architecture is tuned to the headline dataset. The paper should state explicitly whether the Flowers102, Chaoyang, Oxford-IIIT Pet, and Tiny ImageNet results were obtained with these hyperparameters fixed before seeing those test sets, or report a validation-based selection procedure. Without this clarification, the cross-dataset generalization claim is weakened.","section":"Sections 3.5, 4.4.2, 4.4.3, Table 2"}],"minor_comments":[{"comment":"The notation 'i ∈ 1, 2, 3, 4' should be 'i ∈ {1, 2, 3, 4}', and the dimension description following Equation (4) is garbled: 'turning the dimension of the feature map X into HW R2 R2C' should be written using standard tensor shapes with explicit division symbols.","section":"Equations (5)-(6)"},{"comment":"There is a typo: 'Sof tmax' should be 'Softmax'. In addition, the sentence after Equation (6), 'The, the calculation', is missing a clause and should be rephrased.","section":"Equation (7)"},{"comment":"The Tiny ImageNet results are reported without the same-setting caveat that appears for Table 3; it is unclear whether the baseline numbers (cmt_Ti, cmt_XS, BiFormer_T, Shunted_T) were rerun under the paper's protocol or taken from their original papers. This should be clarified.","section":"Table 4"},{"comment":"The sentence 'In section, we analyze the impact of lightweight self-attention mechanism on LMSSA' is incomplete; it presumably refers to Section 4.4.3 but reads as a missing cross-reference.","section":"Section 4.4.4"},{"comment":"The model name 'Hybird' in Table 4 is a typo for 'Hybrid', and the figure caption 'The comparison of the models sizes and accuracies' should be 'model sizes'.","section":"Table 4 and Figure 3"},{"comment":"The Grad-CAM discussion states 'we captures the attention' and 'we captures the attention scattered in the background', which should be 'we capture' or 'our model captures'. More importantly, the qualitative heatmap comparison would be stronger if accompanied by a quantitative metric, such as the average drop or increase in confidence over a set of test images.","section":"Section 4.3.1 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is on a relevant topic for a computer-vision journal, but the integrity of the comparison table and the ablation baseline is essential. The exact match between the 'none' ablation row (Table 5) and the Shunted-T row (Table 3) needs to be clarified by the authors; this could be a simple coincidence in reporting, but it must be addressed directly. Given the absence of code and the importance of exact training details, I would recommend requiring the authors to release code, or at least detailed training logs, for a revised version. The selection-loop issue around CIFAR-100 is also worth probing in the revision, though it is not disqualifying if the other datasets were held out."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the architecture is a reasonable recombination of known ideas—multi-scale token aggregation, convolution in attention, wavelet conv—and the authors show consistent gains across several small datasets. But I would not trust the headline claim until they release code and fix the comparison table. Table 3 says 'All methods are tested and compared under the same setting,' yet CCT, HSViT, and Swin-T are marked with asterisks and the numbers are copied from the original papers, which used different resolutions and training recipes. That's not a minor caveat; it eliminates the basis for 'exceeds all other models of similar sizes.' The GFLOPs for those asterisked rows are computed at native input sizes, not the 224x224 used for MSCViT, so the comparison is not on a common computational basis.\n\nA second red flag: the 'none' row of the ablation (Table 5) shows 2.1 GFLOPs and 81.66% on CIFAR-100, which exactly matches the Shunted-T row in Table 3. If their base architecture without LFE, LMSSA, and CFF is Shunted-T, that needs to be stated. As written, it suggests the ablation numbers were either produced by a different backbone than the one described, or simply copied. Either way, the ablation study doesn't validate the modules against an honestly identified baseline.\n\nWhat's genuinely useful: the paper addresses a real problem—ViT from scratch on tiny datasets—and the design logic is coherent. The LFE replacing positional encoding, the multi-scale fusion coefficients, and the channel-selective convolution are plausible and the results on Flowers102 and Chaoyang are consistent with the story. The authors also report results on five datasets, which is more than many papers, and the Tiny ImageNet comparison is at least a fairer setting. The scaling strategy and computational analysis are fine.\n\nMissing: code, error bars or multiple seeds, and a direct statement of how each baseline was actually trained. The tuning on CIFAR-100 (fusion coefficients, kernel sizes) means the headline CIFAR-100 number is mildly circular, though the other datasets provide some out-of-loop evidence.\n\nBottom line: the paper deserves a serious referee because the idea is worth checking, but in its current form the central claim is unsupported. I would ask for code, retrained baselines under identical protocol, and an honest ablation baseline before accepting the conclusions.","headline":"A plausible hybrid ViT for small datasets, but the headline accuracy claim rests on a comparison table that contradicts its own 'same setting' statement.","tokens_in":14533,"tokens_out":3053,"would_cite":false,"duration_ms":28151,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes MSCViT, a compact hybrid vision transformer that trains from scratch on tiny datasets and claims to outdo comparable CNNs and transformers, reaching 84.68% on CIFAR-100 with 14M parameters and no pretraining.","keywords":["Vision Transformer","Convolutional Neural Networks","Multi-scale self-attention","Tiny datasets","Wavelet convolution","Local feature extraction","Lightweight attention"],"falsifier":"Retrain the unmarked baselines in Table 3 (e.g., PVT-T, CMT-Ti, BiFormer-T) on CIFAR-100 with exactly the MSCViT training recipe — DeiT augmentation, 300 epochs, AdamW, 224x224 resolution — and check whether any reaches 84.68% or higher; if so, the claim of universal superiority under the same setting is refuted.","tokens_in":13513,"feed_emoji":"🖼️","tokens_out":7668,"duration_ms":61118,"temperature":0.7,"pith_summary":"This paper argues that a small Vision Transformer (ViT) can be trained from scratch on tiny datasets if it is redesigned around multi-scale attention and convolutional feature fusion. The authors propose MSCViT, which replaces positional encoding with a local feature extraction block, merges tokens at different rates per attention head, and injects wavelet-extracted high-frequency features into selected channels. On CIFAR-100, MSCViT-S reaches 84.68% top-1 accuracy with 14.0M parameters and 2.5 GFLOPs, beating comparable-size CNNs, hybrid models, and transformer baselines without any large-scale pretraining. The practical payoff is a parameter-efficient ViT for small-data applications where pretraining data or compute is scarce.","feed_headline":"Small ViT hits 84.68% on CIFAR-100 without pretraining","feed_subtitle":"A 14M-parameter hybrid with wavelet convolution and multi-scale attention outperforms larger models on tiny datasets.","key_machinery":"The load-bearing idea is multi-scale token fusion inside each attention block, coupled with two local-feature injections. LMSSA spatially reduces K and V by different factors (8, 4, 2, 1) in different heads, so coarse heads capture global structure while fine heads retain detail, at roughly $4nd^2 + \\sum_i 2n^2d/R_i^2$ FLOPs. LFE replaces the learnable positional encoding with a residual depthwise-convolution block that carries translation equivariance into the transformer. CFF selects a subset of channels, applies wavelet convolution followed by a convolution with stage-dependent kernel size, and concatenates the result with standard attention output. Together they let the network induce locality without pretraining and keep the token count low.","core_discovery":"MSCViT is a hybrid vision transformer that outperforms existing models of similar size on tiny image datasets when trained from scratch. The architecture combines three modifications to the standard ViT block: a Local Feature Extraction (LFE) module that replaces positional encoding with a pair of depthwise convolutions; a Lightweight Multi-scale Self-Attention (LMSSA) module that applies different token fusion coefficients ($R=8,4,2,1$) across attention heads and reduces key/value spatial size with depthwise convolutions; and a Convolutional Feature Fusion (CFF) module that selects redundant channels and processes them with wavelet convolution to capture high-frequency shape information. On CIFAR-100 the small variant achieves 84.68% top-1 accuracy with 14.0M parameters and 2.5 GFLOPs, and the tiny variant reaches 80.11% with only 3.8M parameters and 0.5 GFLOPs, both without pretraining. The paper concludes that the gaps between ViT and CNNs on small data can be closed by architectural changes that inject locality and multi-granularity attention, rather than by larger data or pretraining.","pith_inferences":["The success of wavelet-based CFF suggests a broader testable recipe: frequency-split convolutions in transformer channels may be a general inductive bias for fine-grained classification under scarce data.","Because the scaling factors and kernel sizes were hand-searched (8/4/2/1 and 5x5/3x3), a systematic ablation on fusion schedules might yield further gains or transfer to other dataset regimes.","MSCViT could plausibly be combined with modern self-supervised pretraining or knowledge distillation; the paper only trains from scratch, so the component's value under those regimes is unknown.","The medical Chaoyang result hints that the architecture may be well suited to histopathology-like tasks with few samples per class, but that would need dedicated evaluation."],"forward_implications":["On CIFAR-100, MSCViT-T (3.8M params) already surpasses several larger baselines, suggesting that small-data ViTs need not trade much accuracy for compactness.","The reported gains on Flowers102 and Chaoyang (medical) indicate the recipe transfers across domains with very different class counts and sample imbalances.","Replacing positional encoding with a convolution block did not degrade accuracy, so future ViT designs for small data can omit PE without penalty.","The multi-scale attention with fusion coefficients offers a cheap way to model scale: a 10.7% parameter reduction and 5.2% FLOP reduction for only a 0.18% accuracy drop (200-epoch setting).","On Tiny ImageNet, MSCViT-S reaches 72.11%, outperforming CMT, BiFormer, and Shunted baselines, supporting the same claim at slightly larger scale."],"supporting_citations":[{"why":"Establishes the ViT architecture and its reliance on large-scale pretraining, which is the problem the paper addresses.","marker":"[1]"},{"why":"Supplies evidence that ViT underperforms on local feature learning without sufficient data, motivating LFE and CFF.","marker":"[2]"},{"why":"Provides the training recipe (DeiT augmentation, optimizer, epochs) used for all models in the comparison.","marker":"[6]"},{"why":"The multi-stage pyramid structure and scaling factors that LMSSA adapts, and a key baseline.","marker":"[12]"},{"why":"A hierarchical ViT baseline with shifted windows, used as a comparison point on tiny datasets.","marker":"[13]"},{"why":"A compact transformer baseline for tiny datasets, used as a comparison point.","marker":"[19]"},{"why":"A multi-scale token aggregation baseline, used as a comparison point and motivation for multi-scale attention.","marker":"[34]"}],"fun_headline_variants":["MSCViT: ViT that works on tiny data without pretraining","No pretraining needed: 14M ViT hits 84.68% on CIFAR-100","Small hybrid ViT closes gap with CNNs on tiny datasets","Multi-scale attention ViT beats CNNs on small data, no pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim that MSCViT exceeds all similar-size models on tiny datasets assumes that every baseline in Table 3 was trained under the same recipe as MSCViT (DeiT-style augmentation, 300 epochs, AdamW, 224x224 input), even though some reported numbers are quoted from the original papers rather than reproduced, so unequal training settings could explain the accuracy gaps instead of architecture.","fun_headline_variants_meta":{"raw":{"variants":["MSCViT: ViT that works on tiny data without pretraining","No pretraining needed: 14M ViT hits 84.68% on CIFAR-100","Small hybrid ViT closes gap with CNNs on tiny datasets","Multi-scale attention ViT beats CNNs on small data, no pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1287,"prompt_tokens":1046,"completion_tokens":241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":154}},"tokens_in":662,"tokens_out":241,"duration_ms":3037,"temperature":1.0,"reasoning_tokens":154,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:06:01.349383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the unmarked baselines in Table 3 (e.g., PVT-T, CMT-Ti, BiFormer-T) on CIFAR-100 with exactly the MSCViT training recipe — DeiT augmentation, 300 epochs, AdamW, 224x224 resolution — and check whether any reaches 84.68% or higher; if so, the claim of universal superiority under the same setting is refuted.","supporting_citations":[{"cited_title":"Swintransformer:Hierarchicalvision transformerusingshiftedwindows","cited_arxiv_id":null,"evidence_quote":"A hierarchical ViT baseline with shifted windows, used as a comparison point on tiny datasets."},{"cited_title":"Shunted self-attention via multi-scale token aggregation","cited_arxiv_id":null,"evidence_quote":"A multi-scale token aggregation baseline, used as a comparison point and motivation for multi-scale attention."}],"review_version":1}