{"id":"8c362c00-b923-469f-8521-eba9655eb52a","arxiv_id":"2501.00619","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Smaller image patches improve transformer accuracy on biomedical tasks, attention-window size matters less, and Hyena/MambaVision operators match attention with up to 80% faster training.","lead":"This study benchmarks transformer models on six biomedical imaging datasets, testing how changing patch sizes or attention windows affects accuracy and speed. It finds that smaller patches help most tasks, and that two attention-free operators, Hyena and MambaVision, match or beat standard attention while training much faster on long inputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Patch-size gains are credited to 'context length,' but the paper's own window-size control (Fig. 5) shows minimal benefit from longer context at fixed resolution; the central claim rests on a resolution confound.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the patch-size manipulation is confounded with resolution, and the Swin window-size experiment—the paper's own control for context length at fixed resolution—shows little benefit. My stress-test confirms this is the most serious threat to the central scientific claim. The abstract asserts a 'strong relationship between context length and performance,' but the evidence for a context-length effect per se comes only from the ViT patch-size sweep, where smaller patches also produce a higher-resolution tokenization. The Swin window sweep, which varies the attention span without changing tokenization or input resolution, is flat for most tasks, so the paper's data actively undercut the context-length interpretation. This is an internal inconsistency, not a matter of external consensus, and it directly affects the headline claim rather than only the reproducibility or efficiency details. The efficient-transformer findings (comparable performance and large speedups at long context) are better supported: the speedup numbers are concrete, the performance comparisons are internally consistent on average, and the ability to run configurations that attention cannot fit is a practical advantage. Those claims could also be scrutinized—for instance, the 'over 80% speedup' is measured at configurations where attention sometimes cannot run, so the performance comparison and the speedup comparison are not always aligned—but that is secondary to the context-length framing. Because the reader already identified this same concern and assigned a CONDITIONAL verdict, my analysis does not move the verdict; it reinforces the condition. The proposed factorial test would provide a decisive empirical separation of resolution effects from token-count effects on a single task, and the simpler reanalysis of the existing figures would already reveal whether the Swin window control contradicts the abstract's claim.","tokens_in":17410,"tokens_out":8809,"duration_ms":87941,"concrete_test":"Conduct a factorial control on one pixel-level task (e.g., microscopy denoising) with four configurations: (A) 1024x1024 input, patch 16 (4096 tokens, 256 px/patch); (B) 1024x1024 input, patch 8 (16384 tokens, 64 px/patch); (C) 512x512 input, patch 8 (4096 tokens, 64 px/patch); (D) 512x512 input, patch 4 (16384 tokens, 16 px/patch). Compare A vs C and B vs D to isolate input-resolution (pixels per patch) at fixed token count, and A vs C and B vs D against A vs B to isolate token count at fixed input resolution. If performance tracks pixels per patch rather than token count, the abstract's context-length claim is unsupported; if it tracks token count, the concern is refuted. Alternatively, recompute the reported curves at matched token counts and show that the Swin window-size effect (Figure 5) lies within the 95% confidence intervals while the ViT patch-size effect (Figure 4) is large.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—'a strong relationship between context length and performance' (Abstract, Section 1)—is supported almost entirely by the ViT patch-size sweep (Figure 4). But changing patch size changes two quantities jointly: context length (number of tokens) and the spatial resolution of the token grid (image pixels per patch). The paper itself notes in Section 2 that 'image resolution and context length are closely linked.' The one manipulation that isolates context length while holding input resolution and tokenization fixed is the Swin attention-window sweep (Section 3.3, Figure 5). That experiment shows 'only minor impacts': most tasks are flat or slightly decreasing as the window grows from 4 to 16 tokens, with CMR denoising the sole clear gain (11.37%). If longer context were the driver of the ViT patch-size gains, larger windows should reproduce them; they do not. The Appendix's Swin patch-size comparison (Table 9) again varies tokenization resolution together with context length, so it does not resolve the confound. The Discussion itself reframes the finding as 'preserving high-resolution information' (Section 5), which is a different claim from the abstract's context-length claim. The data are therefore consistent with a resolution effect, and the 'strong relationship between context length and performance' is not internally supported. This is not a dispute with outside consensus; it is an inconsistency between the paper's own controlled manipulation and its headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of how transformer context length affects performance on biomedical image analysis, using six curated datasets spanning 2D/3D data and segmentation, denoising, and classification tasks. The authors vary ViT patch size and Swin attention window size as two ways to change context length, and they evaluate Hyena and MambaVision as drop-in replacements for self-attention in ViT/Swin backbones. They report that smaller patches strongly improve performance, especially for pixel-level tasks; that attention window size has only minor effects in Swin; and that Hyena and MambaVision achieve comparable performance to attention while providing large training-speedups at long context lengths and enabling configurations that self-attention cannot fit in memory.","tokens_in":17696,"tokens_out":7093,"duration_ms":64297,"significance":"If the efficiency results hold, the paper provides a practically useful benchmark for biomedical imaging: it demonstrates that Hyena and MambaVision can substitute for self-attention with modest accuracy changes and large training-speed gains at long context lengths, and that smaller-patch tokenization improves ViT/Swin performance across several tasks. Strengths include the diversity of tasks and modalities, bootstrapped confidence intervals, the reporting of parameter counts and learning rates, and appendix control experiments such as the Swin shift-removal study. The main interpretive limitation is that the central claim—that there is a strong relationship between context length and performance—is not supported by the paper's own controlled manipulation, because the patch-size sweep confounds context length with input resolution; the Discussion itself attributes the findings to preserving high-resolution information. This reduces the strength of the abstract's headline claim but does not eliminate the value of the empirical measurements.","major_comments":[{"comment":"The claim of a \"strong relationship between context length and performance\" is not supported by the paper's own window-size control experiment. Varying the ViT patch size changes both the number of tokens (context length) and the spatial resolution of each token, so the two effects are never separated. The Swin attention-window sweep, which holds resolution and the patch embedding fixed and isolates context length, shows only minor impacts for most tasks in Figure 5, with CMR denoising the only clear gain. The Discussion in Section 5 instead attributes the patch-size results to \"preserving high-resolution information,\" which is a different mechanism from the abstract's framing. Please reframe the abstract and introduction to claim that token granularity, rather than context length per se, drives the improvements, or add an experiment that varies context length while holding resolution fixed (for example, by padding token sequences or by testing larger input images at a fixed window size).","section":"Section 3.3, Figures 4 and 5; Abstract and Section 5"},{"comment":"The ViT patch-size comparison is additionally confounded by parameter-count changes and by per-configuration learning-rate tuning. Table 7 shows, for example, the Abdominal CT attention backbone having 33.9M parameters at patch 32 but 23.2M at patch 16, and Tables 5-6 show learning rates varying across patch sizes for the same operator and task (e.g., CMR ViT with attention uses 1e-3 at patch 4 but 1e-2 at patches 8, 16, and 32). Because both the model size and the optimizer setting vary, the observed performance differences cannot be attributed solely to context length or even to resolution. Please quantify the effect of the parameter-count drift, hold the learning rate fixed across the sweep as a sensitivity check, or explicitly justify that per-configuration tuning does not change the qualitative conclusions.","section":"Section 3.3, Tables 5-7"},{"comment":"The reported \"over 80% speedup\" at ViT patch size 4 is presented without specifying which configurations are included in the average. For several tasks, self-attention cannot run at patch 4 (indicated by X in Figure 4), so the average may be computed over a subset of tasks where both operators run, or it may compare against attention at a different patch size. Please state explicitly which tasks contribute to each average in Tables 3-4, whether self-attention was feasible at that configuration, and note in the main text that the timings are for the backbone only with batch size one; this determines how the headline efficiency claim should be interpreted.","section":"Section 4.3, Tables 3-4 and Figures 6-9"}],"minor_comments":[{"comment":"The caption says \"We visualize performance for each task, operator, and patch size\" but the x-axis is the attention window size, not the patch size; please correct the caption to avoid misleading readers.","section":"Figure 5 caption"},{"comment":"The text \"MaMambaVisionmba\" appears to be a typo and should read \"MambaVision\".","section":"Table 7, Embolism row"},{"comment":"The text states that smaller Swin patches improved performance on all six tasks, but the Microscopy row shows identical SSIM values (0.60 with overlapping confidence intervals) for patch 4 and patch 2; please correct the sentence to reflect the actual pattern, noting that five of six tasks improved.","section":"Appendix B.2.1, Table 9"},{"comment":"The sentence \"these operators can achieve comparable performance to self-attention while improving efficiency by over 80%\" is broader than the evidence; the 80% figure applies only to the longest-context ViT configuration (patch 4) and is based on backbone-only timings, so the sentence should be qualified with those conditions.","section":"Section 1"},{"comment":"The phrase \"how many tokens are processed concurrently\" is imprecise for Swin, where attention is computed within each local window rather than over the whole image; consider phrasing such as \"the number of tokens attended to within a window.\"","section":"Section 3.3, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"This paper is a useful empirical benchmark and the code availability, bootstrapped confidence intervals, and appendix controls are commendable. The main barrier is the mismatch between the abstract's context-length claim and the evidence from the window-size control; this is fixable by reframing the central claim as a tokenization-resolution finding and by adding the requested clarifications on the efficiency comparisons. I would encourage the editor to invite a revision rather than reject, since the underlying experiments appear sound and the practical efficiency results are valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical benchmark, but the headline claim about context length doesn't survive its own control experiment. The patch-size gains are real; attributing them to context length is not supported by the paper's own data.\n\nWhat's new: a systematic comparison of ViT/Swin with self-attention, Hyena, and MambaVision across six biomedical tasks spanning 2D/3D data, with bootstrapped confidence intervals and efficiency curves. The finding that Hyena and MambaVision match or beat attention on pixel-level tasks while cutting training time by over 80% at long context is solidly supported and directly actionable for practitioners. The paper is also honest about limitations: it flags the absence of window shifting for alternative operators, reports the shift ablation in the appendix, and acknowledges the resolution/context-length link in Section 2.\n\nSoft spots: the central framing. The abstract claims a 'strong relationship between context length and performance,' but the only manipulation that isolates context length while holding resolution and tokenization fixed is the Swin window sweep (Figure 5), which shows mostly flat or decreasing performance across tasks. The ViT patch-size sweep changes resolution and context length jointly, so the large gains there cannot be attributed to context length alone. The paper's own Discussion reframes the result as 'preserving high-resolution information,' which is a different and more defensible claim. This is a framing problem rather than a fatal flaw, but it should be fixed before publication.\n\nOther concerns are minor: efficiency numbers are averaged over ten runs but no variance is reported; parameter counts drift by up to 10M across ViT patch sizes (Table 7); code is promised but not yet released; one dataset is private. None of these undercut the main efficiency result.\n\nBottom line: practitioners will get a direct recipe — smaller patches plus Hyena or MambaVision — and a measured map of trade-offs across tasks they care about. The citation pattern is fine and the paper engages with relevant prior work. It deserves a serious referee, who should ask the authors to separate the resolution confound, temper the abstract, and release the code.\n\nI'd accept it for peer review.","headline":"A useful empirical benchmark with a real framing problem: the patch-size gains are solid, but attributing them to context length is undercut by the paper's own window-size control.","tokens_in":18237,"tokens_out":1505,"would_cite":true,"duration_ms":15505,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing self-attention with attention-free operators like Hyena and MambaVision maintains accuracy while cutting training time by over 80 percent on long-context biomedical imaging tasks, and that smaller patches…","keywords":["biomedical imaging","context length","self-attention","efficient transformers","Hyena","MambaVision","Vision Transformer","Swin Transformer"],"falsifier":"Run a ViT on the same datasets with (a) fixed image resolution and variable patch count and (b) fixed patch count and variable image resolution; if performance tracks resolution rather than token count, the paper's conclusion that longer context drives the gains would be refuted, and the Swin window-size sweep already provides a control pointing in that direction.","tokens_in":17177,"feed_emoji":"🩺","tokens_out":5171,"duration_ms":45459,"temperature":0.7,"pith_summary":"This paper asks whether biomedical image analysis benefits from longer transformer context, and whether recent attention-free operators can deliver that benefit cheaply. Across six datasets spanning 2D and 3D segmentation, denoising, and classification, the authors find that shrinking the ViT patch size, which lengthens the token sequence, consistently improves performance, most strongly for pixel-level prediction. Increasing Swin's attention window size, by contrast, yields only minor gains. The paper then shows that replacing self-attention with Hyena or MambaVision maintains or slightly improves accuracy while speeding up training by over 80 percent at long context lengths, and fits configurations that self-attention cannot run on the available hardware.","feed_headline":"Attention-free transformers cut medical imaging training time by 80%","feed_subtitle":"Hyena and MambaVision match self-attention accuracy while enabling high-resolution patches that otherwise exceed GPU memory.","key_machinery":"The paper's central object is the context length of a vision transformer, the number of image tokens processed together, and the two levers used to change it: patch size in ViT (smaller patches give more tokens) and attention window size in Swin (larger windows attend over more tokens at once). The argument that efficiency can be gained without losing accuracy is carried by two attention replacements: Hyena, which uses long convolutions with gating and scales as O(n $log^{2}$ n), and MambaVision, a selective state-space operator that scans tokens through a learnable hidden state. These operators are swapped into the same transformer blocks in place of self-attention, so any performance difference is attributed to the operator itself rather than to architecture changes. This comparison is what supports the claim that the operators replicate attention's accuracy while removing its quadratic cost.","core_discovery":"On the paper's own terms, the central discovery is that context length is a major lever for transformer performance in biomedical imaging, but only when it is increased by preserving image resolution through smaller patches; enlarging attention windows gives little benefit. As a practical corollary, the paper argues that attention-free operators, Hyena (built on long convolutions with subquadratic O(n $log^{2}$ n) scaling) and MambaVision (built on a selective state-space scan), can serve as drop-in replacements for self-attention in ViT and Swin backbones. In its experiments these operators match or exceed self-attention accuracy on segmentation, denoising, and classification, cut training time by up to roughly 80 percent at the longest contexts, and allow the use of patch sizes that exceed the memory available to self-attention. This is offered as evidence that efficient long-context processing, not quadratic attention itself, is what matters for high-resolution biomedical imaging.","pith_inferences":["A testable extension the paper does not run is to decouple resolution from context length directly, for instance by keeping patch count fixed while varying input resolution; the Swin window results already hint that resolution, not token count, is the active ingredient.","If the resolution interpretation is right, then the efficiency gains of Hyena and MambaVision become even more useful: they let models afford resolution-preserving tokenization that quadratic attention simply cannot fit in memory.","The same operator swap is likely to transfer to other dense prediction tasks such as instance segmentation or image restoration, where fine detail matters more than global reasoning.","The classification gap for Swin with attention-free operators is likely attributable to the removed shift operation, which suggests that an efficient shifted-window implementation for Hyena and MambaVision would close most of that gap."],"forward_implications":["Smaller patch sizes improve performance across segmentation, denoising, and classification, with the largest gains on pixel-level tasks, so future backbones for medical imaging should prioritize resolution-preserving tokenization.","Hyena and MambaVision can replace self-attention in ViT and Swin with little or no accuracy loss, which would let laboratories train high-resolution models that currently exceed GPU memory.","At long context lengths these operators speed up training by over 80 percent relative to self-attention, making them a practical choice for clinical-scale workloads.","The Swin results suggest that local windows plus hierarchical merging already capture enough context for many tasks, so effort spent enlarging windows is less valuable than effort spent preserving resolution."],"supporting_citations":[{"why":"Defines the ViT architecture whose patch-size sweep is the paper's primary way of varying context length.","marker":"(Dosovitskiy et al., 2020)"},{"why":"Defines the Swin transformer and its shifted-window scheme, which provides the second context-length lever and the shift operation studied in the appendix.","marker":"(Liu et al., 2021)"},{"why":"Introduces the Hyena operator with long convolutions and subquadratic scaling, one of the two attention-free replacements the paper evaluates.","marker":"(Poli et al., 2023)"},{"why":"Introduces the selective state-space model, Mamba, that underlies the MambaVision operator.","marker":"(Gu and Dao, 2023)"},{"why":"Introduces the MambaVision operator adapted to vision tasks and supplies the implementation used in the experiments.","marker":"(Hatamizadeh and Kautz, 2024)"},{"why":"Supplies the quadratic-complexity result for self-attention that motivates the need for efficient alternatives.","marker":"(Keles et al., 2023)"},{"why":"Provides the Hyena implementation used in the paper, a load-bearing piece for reproducing the efficiency comparisons.","marker":"(Arora et al., 2023)"}],"fun_headline_variants":["Context length boosts medical image transformers, but only via smaller patches","Attention-free operators match self-attention on medical images, cut training 80%","Long-context models enable high-res medical imaging that self-attention can't handle","Context length is the key lever for transformer performance in biomedical imaging","Hyena and MambaVision replace self-attention in medical imaging with 80% speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline claim rests on treating smaller patch size as a pure way to lengthen context, but smaller patches also preserve more image detail; since larger attention windows, which lengthen context without changing resolution, show only minor gains, the patch-size results may be driven by resolution rather than by context length.","fun_headline_variants_meta":{"raw":{"variants":["Context length boosts medical image transformers, but only via smaller patches","Attention-free operators match self-attention on medical images, cut training 80%","Long-context models enable high-res medical imaging that self-attention can't handle","Context length is the key lever for transformer performance in biomedical imaging","Hyena and MambaVision replace self-attention in medical imaging with 80% speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2631,"prompt_tokens":936,"completion_tokens":1695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1594}},"tokens_in":552,"tokens_out":1695,"duration_ms":12486,"temperature":1.0,"reasoning_tokens":1594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:46:10.907371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a ViT on the same datasets with (a) fixed image resolution and variable patch count and (b) fixed patch count and variable image resolution; if performance tracks resolution rather than token count, the paper's conclusion that longer context drives the gains would be refuted, and the Swin window-size sweep already provides a control pointing in that direction.","supporting_citations":[],"review_version":1}