{"id":"e57e9deb-544f-4566-a906-a4e4089b035e","arxiv_id":"2508.18960","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A compact convolutional transformer replacing SDPA with token mixing attention reports higher CIFAR100 accuracy, but the method is essentially prior token mixing and the comparison lacks error bars, code, and a strong baseline.","lead":"The paper assembles token mixing attention, sequence pooling, and a convolutional tokenizer into a compact vision transformer and reports a large CIFAR100 accuracy gain over its own SDPA baseline with 40% fewer parameters. The core attention is a renamed version of Hosseini et al.'s token mixing, and the evaluation is a single run against a low, possibly undertrained baseline, so the headline result is not yet established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 10-point CIFAR100 gain rests on an unvalidated, badly underperforming SDPA baseline; the central comparison is not yet trustworthy.","rationale":"The reader's REJECT verdict is supported by the strongest available evidence: the paper's headline empirical claim depends on a single comparison against a baseline that scores far below any published CCT result. The proposed method is essentially Hosseini et al.'s token mixing combined with CCT's sequence pooling and tokenizer, so even if the architecture were identical, the novelty is thin. However, the decisive problem is not novelty but correctness: the 36.50% baseline is implausibly low, and the paper provides no code, no error bars, no multiple seeds, and no standard-recipe control. The absence of error bars is especially relevant because Figure 4 shows the baseline diverging, which suggests training instability rather than a fair comparison. I also considered whether the lack of SOTA comparison should be the lead concern, but the central claim as stated by the authors is the improvement over their own SDPA baseline; if that baseline were solid, the relative comparison could stand even without SOTA. Since it is not solid, the empirical foundation collapses. The recommended verdict remains REJECT, i.e., unchanged from the reader's verdict.","tokens_in":5671,"tokens_out":6900,"duration_ms":64878,"concrete_test":"Obtain the authors' code (the paper says it is on GitHub) and train the identical SDPA CCT-6/3x1 under the standard CCT recipe of Hassani et al. (2022): e.g., 300 epochs, cosine LR schedule, and the augmentations used in that paper, while keeping the rest of the architecture unchanged. Train the super-attention model under the same improved recipe as well. If the SDPA baseline rises to the normal CCT range (roughly 75-80% top-1) or the top-1 gap shrinks materially, then the Table 2 improvement is an artifact of the undertrained baseline. If the gap persists under the standard recipe, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that replacing SDPA with 'super attention' yields a ~10-point top-1 gain on CIFAR100 (36.50% to 46.29%) while reducing parameters by 40%. This inference is only valid if the SDPA control is a fair, reasonably trained representative of the compact convolutional transformer. That control is a self-implemented CCT-6/3x1 trained for 75 epochs with constant lr=0.01, batch size 1024, no augmentation, and no LR schedule; it reaches 36.50%, far below published CCT-6/3x1 numbers (roughly 75-80% on CIFAR100 under standard training recipes). Figure 4 shows the SDPA baseline diverging while super attention converges. The reported gap could therefore be an interaction between the new module and a poor training configuration: super attention may merely be less sensitive to the degenerate schedule, not better in a meaningful architectural sense. A single run without error bars or a working code link leaves no way to separate these explanations. The 'state-of-the-art' wording in the abstract is also unsupported because no SOTA comparison is provided, but the load-bearing issue is the unvalidated baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a variant of the Compact Convolutional Transformer (CCT) in which the standard scaled dot-product attention (SDPA) is replaced by 'super attention' (token mixing with a learned l x l matrix, following Hosseini et al. 2025), combined with sequence pooling and a convolutional tokenizer. On CIFAR-100, the authors report top-1/top-5 validation accuracies of 46.29%/76.31% for their model versus 36.50%/66.33% for a self-implemented CCT-6/3x1 with SDPA, while reducing parameters from 17.7M to 10.6M. Training uses 75 epochs, constant AdamW lr=0.01, batch size 1024, no data augmentation, and no learning-rate schedule. The paper claims state-of-the-art performance, high training stability, and efficiency gains when context length is below embedding dimension.","tokens_in":6010,"tokens_out":5004,"duration_ms":44937,"significance":"If correct, the result would be an interesting data point: token-mixing attention could beat SDPA in a low-budget CCT setting with 40% fewer parameters and without augmentation or LR schedules. The manuscript is transparent about its limitations and provides explicit equations and hyperparameters, which is useful. The main claim, however, is not established by the current evidence: the comparison uses a single seed, a baseline that is far outside the normal operating range of CCT, and no code or error bars. The efficiency claim is also not measured. The practical significance is therefore conditional on a properly controlled comparison.","major_comments":[{"comment":"The central 10-point gain rests on an unvalidated SDPA baseline. CCT-6/3x1 implemented by the authors reaches only 36.50% top-1 on CIFAR-100, far below typical published CCT-6/3x1 results (often >75% under standard training recipes). The training configuration in Table 1 (constant lr=0.01, batch 1024, no augmentation, no schedule, 75 epochs) is atypical; Figure 4 shows the SDPA model's validation loss increasing while training loss drops, i.e., severe overfitting. The comparison therefore conflates the attention mechanism with training robustness: super attention may simply tolerate the degenerate schedule better. Please report (a) a properly tuned SDPA baseline using standard CCT training practices, (b) published CCT-6/3x1 numbers in the same table, and (c) multi-seed mean±std for both models.","section":"§4, Table 2; Figure 4"},{"comment":"The abstract and introduction describe the result as 'state-of-the-art performance', but Table 2 contains no state-of-the-art comparison: only a ViT-12/16 entry from a different training setup and the authors' own low-scoring baseline. The title and abstract overstate the contribution. Either add a comparison table with published compact transformer results on CIFAR-100, or remove the SOTA wording.","section":"Abstract; §1; §4"},{"comment":"The evidence consists of a single run. No seeds, error bars, or code URL appear in the manuscript despite the statement 'We make our code available on Github.' Single-run comparisons cannot exclude initialization or run-to-run variance, especially with lr=0.01 constant. Provide the code repository and at least three seeds.","section":"§4, Table 1"},{"comment":"The efficiency claim ('more efficient ... when context length is less than embedding dimension') is not substantiated. Parameter count is reported (10.6M vs 17.7M), but no latency, memory, FLOPs, or throughput measurements are given. The theoretical complexity tradeoff should be stated explicitly and, if possible, measured on the T4 used for training.","section":"Abstract; §4"}],"minor_comments":[{"comment":"'Top 1%' and 'top 5%' should be 'top-1' and 'top-5' accuracy; the current phrasing is misleading.","section":"Figures 1-2; throughout"},{"comment":"The text says the baseline 'diverges', but the curves show overfitting: training loss decreases while validation loss increases. Please rephrase.","section":"Figure 4"},{"comment":"The notation for the token-mixing matrix W^A is ambiguous: is W^A shared across heads or per-head? Equations (2)-(5) suggest per-head, but the description says W^A ∈ R^{l×l} without a head subscript.","section":"§3.1"},{"comment":"Typo: 'CIF AR100' should be 'CIFAR-100'. Also mention the dataset resolution consistently.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is not ready as submitted. The central issue is the baseline: 36.50% top-1 is far below representative CCT-6/3x1 performance, so the claimed gain may be an artifact of an unfairly trained SDPA control. If the authors cannot reproduce the gain against a properly trained CCT in a multi-seed comparison, the paper should be rejected. I would not support acceptance based on the current single-run evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central claim—that replacing SDPA with \"super attention\" buys a 10-point top-1 gain on CIFAR100 with 40% fewer parameters—is not supported by the evidence as presented. The comparison baseline is a self-implemented CCT-6/3x1 trained for 75 epochs with constant lr=0.01, batch 1024, no augmentation, no LR schedule, and it lands at 36.5% top-1. Published CCT-6/3x1 numbers under standard training recipes are roughly double that. So the reported gap is as consistent with \"super attention is more robust to a degenerate schedule\" as with any real architectural advantage. Single run, no error bars, and the Github link in the arXiv version didn't resolve for me.\n\nWhat's actually new: not much. The attention block is exactly the token mixing mechanism from Hosseini et al. (2025) (their equations 1–5), which the authors cite honestly. Sequence pooling and the convolutional tokenizer come from Hassani et al. (2022). The combination is straightforward and the hyperparameters are their own. That's a reasonable engineering experiment, but it doesn't add a result absent from the cited literature.\n\nWhat the paper does well: the writing is unusually clear about provenance. They cite the source of every component, state the training configuration precisely, include loss curves, and explicitly list limitations (fixed context length, compute constraints). The loss curves do show the SDPA baseline diverging while super attention converges, which is worth a look. The claim that token mixing is cheaper than SDPA when context length < embedding dimension is from Hosseini et al., not new here.\n\nThe \"state-of-the-art\" in the abstract is unsupported: no comparison to any recent SOTA, only a ViT-12/16 score quoted from Hassani et al. (2022). That should be cut.\n\nBottom line: the architecture might be fine, but the one experiment meant to demonstrate it is not trustworthy yet. If the authors retrain a proper CCT baseline with standard augmentation, LR schedule, and a few seeds, and compare directly against Hosseini et al.'s token mixing as the control, this could become a useful empirical note. As it stands, the load-bearing comparison fails. I wouldn't cite it, and I wouldn't bring it to reading group. For peer review: a serious editor should desk reject this version, with an invitation to resubmit with a valid baseline. The thinking is clear and the paper is honest, so it is still a \"serious thinker\" despite the errors.","headline":"A compact transformer variant that swaps SDPA for Hosseini et al.'s token mixing reports a 10-point CIFAR100 gain against a self-implemented baseline that scores 36.5%; the claim is not supported until the baseline is fixed.","tokens_in":6449,"tokens_out":2246,"would_cite":false,"duration_ms":20255,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing scaled dot-product attention with a token-mixing 'super attention' in a compact convolutional transformer improves CIFAR-100 top-1 accuracy from 36.50% to 46.29% while cutting parameters to 10.6M from 17.7M.","keywords":["compact convolutional transformer","super attention","token mixing","scaled dot product attention","CIFAR-100","sequence pooling","vision transformer","parameter efficiency"],"falsifier":"Train the same CCT-6/3x1 backbone with a properly tuned SDPA attention module (standard augmentation, learning-rate schedule, original CCT hyperparameters) on CIFAR-100 for 75 epochs. If that baseline reaches or exceeds 46.29% top-1 validation accuracy, the claimed advantage of super attention over SDPA is falsified.","tokens_in":5606,"feed_emoji":"🧠","tokens_out":7568,"duration_ms":59496,"temperature":0.7,"pith_summary":"The paper tries to establish that the standard scaled dot-product attention (SDPA) used in vision transformers contains redundant value projections, and that replacing it with a token-mixing 'super attention' block yields both higher accuracy and lower parameter count on fixed-context-length vision tasks. Using a compact convolutional transformer backbone with a convolutional tokenizer and sequence pooling, the authors report a jump from 36.50% to 46.29% top-1 validation accuracy on CIFAR-100 (66.33% to 76.31% top-5) in 75 epochs, while total parameters drop from 17.7M to 10.6M. They further claim these gains come without any data augmentation, positional embeddings, or learning-rate scheduling, and that super attention is more efficient than SDPA when the token sequence is shorter than the embedding dimension. If true, this would show that attention modules can be made substantially simpler and cheaper for small-scale vision tasks without sacrificing accuracy.","feed_headline":"Token-mixing attention beats SDPA on CIFAR-100 by ~10 points","feed_subtitle":"A compact convolutional transformer with super attention reaches 46.3% top-1 with 40% fewer parameters and no augmentation.","key_machinery":"Super attention, the paper's name for token mixing: instead of computing each head's values with a learned linear projection V_i = X W_i^V, it applies a single learned sequence-mixing matrix W^A ∈ R^{ℓ×ℓ} to the value vectors from the left, multiplying across the token dimension. This mixes information across the ℓ tokens and eliminates per-head value projections. The block also uses compact convolutional transformer pieces: a one-layer 3×3 convolutional tokenizer that embeds patches, and sequence pooling (softmax-weighted pooling over the sequence) in place of a class token. The mechanism's claimed effect is to reduce attention parameters by 25% and total parameters by 40% while producing m","core_discovery":"On CIFAR-100, a six-layer compact convolutional transformer with super attention reaches 46.29% top-1 and 76.31% top-5 validation accuracy, versus 36.50% and 66.33% for the same backbone with standard SDPA, within the same 75-epoch budget. The model uses 10.6M parameters, about 60% of the 17.7M SDPA baseline; attention parameters drop by roughly 25% and total parameters by 40%. The authors attribute the gain to token mixing: a learned ℓ×ℓ matrix applied to the value vectors from the left, replacing per-head value projections in SDPA. They also report that the SDPA baseline overfits and diverges in validation loss while super attention converges stably, without augmentation, positional embedd","pith_inferences":["A natural next experiment is to re-run the SDPA baseline with the original CCT hyperparameters and standard augmentation to see how much of the 10-point gap is recovered; the paper's 36.50% baseline sits far below typical published CCT-6/3x1 numbers on CIFAR-100, so the size of the true token-mixing gain is not yet pinned down.","If the efficiency analysis carries over, token-mixing attention could lower the inference cost of small vision transformers on edge hardware, since it avoids per-head value projections and has favorable complexity when tokens are few.","The authors' removal of positional embeddings aligns with earlier work showing convolutional tokenizers can encode locality; an ablation that swaps the tokenizer for plain patching would separate the tokenizer's contribution from the token-mixing block's.","Scaling the model up (more layers, larger datasets like ImageNet) would test whether the stability and accuracy gains persist beyond the fixed 75-epoch CIFAR-100 regime."],"forward_implications":["If the result holds, token-mixing attention is a drop-in replacement for SDPA in compact transformers, improving CIFAR-100 accuracy by roughly 10 points at 60% of the parameter count.","Super attention becomes the preferred attention when context length is smaller than embedding dimension (e.g., 32×32 images with 768-dim embeddings), since its cost is lower than SDPA's in that regime.","Training stability without mixup, positional embeddings, or a learning-rate schedule suggests the model could be easier to deploy in low-resource settings.","The same convolutional tokenizer plus sequence pooling plus token-mixing design may transfer to other fixed-length recognition benchmarks beyond CIFAR-100, as the authors expect."],"supporting_citations":[{"why":"Supplies the token-mixing formulation (equations 1-5) and the cost-effectiveness argument that super attention replaces SDPA with.","marker":"Hosseini et al. (2025)"},{"why":"Supplies the compact convolutional transformer backbone, the sequence pooling mechanism (Equation 8), the convolutional tokenizer, and the ViT-12/16 baseline result the paper compares against.","marker":"Hassani et al. (2022)"},{"why":"Defines the original scaled dot product attention that serves as the SDPA baseline and as the efficiency comparison point.","marker":"Vaswani et al. (2017)"}],"fun_headline_variants":["Token-mixing super attention lifts compact transformer to 46% accuracy","Compact transformer with super attention: 40% fewer params, 10-point gain","No augmentation or positional embeddings: super attention still tops baselines","Super attention replaces SDPA, cuts parameters 40%, boosts CIFAR-100 accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The SDPA baseline is a correctly implemented, reasonably tuned compact transformer, so the 10-point accuracy gain comes from replacing attention with token mixing rather than from an undertrained or misconfigured baseline.","fun_headline_variants_meta":{"raw":{"variants":["Token-mixing super attention lifts compact transformer to 46% accuracy","Compact transformer with super attention: 40% fewer params, 10-point gain","No augmentation or positional embeddings: super attention still tops baselines","Super attention replaces SDPA, cuts parameters 40%, boosts CIFAR-100 accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1535,"prompt_tokens":702,"completion_tokens":833,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":759}},"tokens_in":446,"tokens_out":833,"duration_ms":7672,"temperature":1.0,"reasoning_tokens":759,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:02:47.578770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same CCT-6/3x1 backbone with a properly tuned SDPA attention module (standard augmentation, learning-rate schedule, original CCT hyperparameters) on CIFAR-100 for 75 epochs. If that baseline reaches or exceeds 46.29% top-1 validation accuracy, the claimed advantage of super attention over SDPA is falsified.","supporting_citations":[],"review_version":1}