{"id":"b56e98f5-7506-4bdd-a2ee-51c53871edff","arxiv_id":"2412.20601","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adaptive patch sizes chosen by local variance let spatiotemporal transformers match uniform high-resolution accuracy with about half the token sequence length.","lead":"This paper introduces MATEY, an AI model that predicts how fluids and other physical systems evolve by zooming in on the important active regions and using coarse detail elsewhere. It matters because it tackles the main cost of building foundation models for science, the huge computing bill from high-resolution simulations, and shows a way to cut that cost by roughly half without losing accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed ~2x compute reduction for Adap Mix is quantified by average sequence length (Eq. 10), but training uses variable-length sequences with padding masks; because ViT attention is quadratic, average length can understate actual batched cost, so a wall-clock timing check is needed.","rationale":"The reader's weakest assumption, the variance-based refinement proxy in Eq. (4), is a real concern about generality: low-variance but dynamically important regions could be missed, and the paper does not validate the proxy against other complexity measures. That would be the right concern if the paper only claimed that the method works broadly on multiscale physics. However, the central empirical claim is that adaptive tokenization achieves comparable accuracy at roughly half the compute of the uniformly fine case. The paper supports this with average sequence length rather than measured training cost, and variable-length sequences are handled with padding masks. Since attention is quadratic, average token count can materially understate actual per-step cost. This concern applies directly to the demonstrated experiment on colliding thermals, not just to extrapolation to other systems, and it is checkable with a straightforward timing reproduction. I therefore flag it as the most load-bearing issue. The reader's conditional verdict remains appropriate: code and timing evidence are needed before the compute-reduction claim is accepted, but the concern does not warrant rejection without running the check.","tokens_in":18934,"tokens_out":5097,"duration_ms":58263,"concrete_test":"Run the Section 4.2 Adap Mix experiment for Ti-ViT with (px1, pxsts, gamma) = (32, 16, 0.1) (the highlighted Pareto point) and for constant ps=16x16, using the same batch size (128), optimizer, and 20k steps as Appendix A.2, on the same GPU. Record wall-clock time per step and total training time, and also record the per-batch maximum and mean-squared padded sequence length. If the adaptive run's wall-clock time per step is not close to half of the ps=16x16 run (say within 0.4-0.6x), then the claimed 2x compute reduction is an artifact of averaging unpadded token counts; report the ratio.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that adaptive patch sizes lead to 'patches of varying length across samples, which are handled with padding masks,' and Figure 2 shows masked batched token sequences. Section 4.2 and Eq. (10) define Lavg,mix as the per-sample average sequence length over time and plot it against NRMSE in Figure 5; the text then concludes a '2x reduction' in compute. That inference is not supported. For ViT and SViT, attention cost is quadratic in sequence length, and padded batches are processed at the maximum sequence length in the batch unless a sparse kernel is used (none is described). The average of L over samples/times is not the average of L^2, and neither accounts for padding to the batch maximum. If refinement counts vary across the 128-sample batches used in Appendix A.2, the actual per-step FLOPs and GPU time can remain close to the uniformly fine ps=16x16 case even when Lavg,mix is halved. Thus the central accuracy/compute Pareto claim in Figure 5 is currently a claim about average token count, not about measured compute; the headline '2x reduction in compute' from the introduction is not established by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MATEY, a vision-transformer-based framework for forecasting spatiotemporal physical systems, with two contributions: adaptive tokenization schemes (Adap Mul and Adap Mix) that refine patch sizes locally based on a variance threshold, and a comparison of three spatiotemporal attention mechanisms (ViT, SViT, and AViT). The authors evaluate MATEY on a colliding-thermals dataset, show that adaptive tokenization can reach lower NRMSE than uniform fine patch tokenization at smaller average sequence length, and demonstrate that models pretrained on PDEBench outperform randomly initialized models when fine-tuned on out-of-distribution colliding-thermals and MHD tasks, especially with frozen attention in low-data regimes. The paper also reports that fully decoupled axial attention (AViT) is less efficient and expressive than SViT or ViT in their experiments.","tokens_in":19160,"tokens_out":3081,"duration_ms":31785,"significance":"If the compute-efficiency claim is substantiated, adaptive tokenization is a useful contribution for applying ViTs to high-resolution physical simulations, where token counts become prohibitive. The paper offers a simple, interpretable refinement rule (local variance thresholding), guarantees a smooth interpolation between coarse and fine uniform tokenization for Adap Mix, and evaluates on genuinely out-of-distribution fine-tuning tasks that include variables absent from pretraining. The planned public release of data, code, and trained models is a strength. The main limitation is that the central '2x compute reduction' claim is currently supported only by an average-sequence-length proxy, not by measured cost; and the statistical evidence (error bars, multiple-run variability) is incomplete. With those gaps addressed, the paper would be a solid empirical contribution to scientific machine learning.","major_comments":[{"comment":"The claimed '2x reduction in compute' is based on the average sequence length Lavg,mix, not on measured computational cost. Because the attention cost in ViT and SViT is quadratic in sequence length and because the manuscript states (Section 3, Figure 2) that variable-length samples are handled with padding masks, the actual per-step FLOPs and GPU time depend on the maximum sequence length in each batch, which can remain close to the fine-patch case even when Lavg,mix is halved. The paper should report wall-clock time per training step (and ideally FLOPs) for the adaptive versus uniform fine-patch runs, or provide a clear analysis of batch-level maximum lengths. Without this, the headline compute claim in the introduction and abstract is not established.","section":"Section 4.2, Eq. (10), Figure 5"},{"comment":"Most plots lack error bars: only the ViT panel in Figure 5 shows standard deviations from 3 runs, while SViT in Figure 5 and Figures 6, 7, 8, and 10 do not. The conclusion that Adap Mix 'achieves lower predictive errors than the more expensive ps=16x16 case despite requiring only half the average sequence length' is based on selected gamma values, and the gamma sweep is then presented with the best cases highlighted. The authors should provide repeated-seed variability for all central Pareto claims, and either report all gamma values in the sweep or justify the selection rule to avoid cherry-picking.","section":"Figures 5-8, 10; Section 4.2"},{"comment":"The refinement criterion assumes that the variance of the solution field inside a patch is a sufficient proxy for where smaller patches are needed. This assumption is load-bearing for the entire accuracy/compute tradeoff, but it is not validated against alternative complexity measures or against error maps of the trained models. Physical features such as thin boundary layers, smooth wide-amplitude gradients, or near-uniform regions that still require fine resolution for accurate long-time prediction may not be refined, potentially collapsing the claimed benefit. The paper should either provide evidence that the selected patches correlate with prediction error on the test problem, or explicitly discuss this limitation and its consequences for generalization.","section":"Section 3, Eq. (4)"}],"minor_comments":[{"comment":"The notation 'A ViT' is used inconsistently: the text and figures refer to 'AViT' and 'A ViT' interchangeably; please standardize.","section":"Section 3, Eq. (2)"},{"comment":"The caption contains a typo: 'hyperparamters (px1, pxsts, γsts)m' should be 'hyperparameters (px1, pxsts, gamma_sts)'.","section":"Figure 7 caption"},{"comment":"The number of colliding-thermals training trajectories is listed as [1, 6, 12, 24, 48] in Table A1, but Section 4.3 and Figure 8 report 1 to 24 trajectories; please reconcile the discrepancy.","section":"Table A1"},{"comment":"The conclusion that ViT is less expensive than SViT and AViT is drawn from a single dataset with a short token sequence (16x8x8); the text acknowledges this, but the generalized statement in the abstract and Section 5 should be couched as regime-specific.","section":"Section 4.1, Figure 3"},{"comment":"For Adap Mul, the 'dramatically more cost effective' conclusion is based on the linear and quadratic cost indices in Eqs. (11)-(12), not on measured time; the text should clarify that these are analytical estimates, not observed cost.","section":"Section 4.2"},{"comment":"The paper compares only against constant-patch baselines; a comparison with a recent adaptive tokenization method such as PMSViT, or at least a discussion of why it is not used as a baseline, would strengthen the claims.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its empirical design, but the headline compute-efficiency claim needs to be backed by measured wall-clock or FLOPs rather than average sequence length, and the statistical support (error bars, multi-run variability) needs strengthening. The variance-proxy limitation is worth raising in revision but is not necessarily fatal if discussed honestly. The work fits the scope of the journal and, once the cost claim is properly quantified, would be a valuable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MATEY does two useful things—adaptive patch tokenization driven by local variance, and a clean comparison of AViT/SViT/ViT on physical data—but the headline \"2x compute reduction\" is measured in average sequence length, not actual training cost. That distinction matters because variable-length sequences are padded to batch max, and attention is quadratic.\n\nWhat's genuinely new: the AMR-style refinement rule with two ways to fuse coarse and fine tokens (Adap Mul and Adap Mix). The convergence of Adap Mix to the coarse and fine uniform patch solutions as gamma goes to 1 and 0 is a nice property and is verified. The empirical claim that Adap Mix can beat the fine uniform patch case at half the average sequence length holds in the colliding thermals experiments; the contour plots show visibly smoother local structure than the coarse baseline. The attention scheme comparison is also a useful data point—axial attention being less efficient and expressive on this problem is worth knowing. And the OOD fine-tuning setup with thermals and MHD, where new variables appear, is a more honest test of transferability than same-equation fine-tuning; the muted pretraining advantage is a sober result.\n\nWhere it gets soft: the compute claim. Equation (10) defines average sequence length; Figures 5–7 plot NRMSE against that. No wall-clock time or FLOPs measurement is reported anywhere. With padding masks, a batch runs at the max sequence length, so halving the average can leave actual cost close to the uniform fine-patch case. That is a real gap for the paper's central promise. The missing error bars (only Figure 5 ViT has them) and the gamma sweep with best cases highlighted also make the accuracy advantage look firmer than it is. The AViT conclusion is drawn from one dataset with very short token sequences (16x8x8); generalizing to \"axial attention is less efficient\" from that is too strong. And the variance criterion assumes local variance tracks where refinement is needed; that heuristic is plausible but unvalidated against error maps or alternatives. None of this kills the method, but it should be scoped precisely.\n\nWho it's for: people building transformer surrogates for PDEs and anyone thinking about tokenization for high-res spatiotemporal inputs. It deserves a serious referee. I would accept it for review and ask for wall-clock timing or FLOPs, error bars, and a scoped AViT claim, plus code/data release.","headline":"Variance-based adaptive tokenization is a plausible win, but the headline 2x compute reduction rests on average sequence length, not measured cost.","tokens_in":19716,"tokens_out":2363,"would_cite":true,"duration_ms":23775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive tokenization lets a vision transformer match fine-grid accuracy at roughly half the sequence length.","keywords":["adaptive tokenization","vision transformer","spatiotemporal attention","multiscale physical systems","foundation models","pretraining","fine-tuning","MATEY"],"falsifier":"A concrete test: on a holdout set, build an oracle refinement mask from the actual prediction error of the coarse-patch model and compare it with the variance mask from Eq. (4); if the variance-selected model needs substantially more tokens than the error-selected oracle to reach the same NRMSE, or fails on a low-variance thin boundary layer or smooth wide-amplitude gradient, the central tradeoff collapses.","tokens_in":18697,"feed_emoji":"🧩","tokens_out":10881,"duration_ms":100770,"temperature":0.7,"pith_summary":"In a vision transformer, the number of tokens, the patches the attention mechanism processes, grows quickly with resolution, making accurate multiscale forecasting of physical fields expensive. MATEY instead starts from coarse patches and splits only the highest-variance patches into smaller sub-patches, producing a mixed-resolution token sequence of adjustable length. On the colliding-thermals test problem, the paper reports that this adaptive scheme (Adap Mix) reaches lower prediction error than a uniformly fine patch model while using roughly half the average token sequence length, and that the same tradeoff holds for a second coarse/fine patch pair. It further reports that a time-then-space attention scheme is more efficient and expressive than full spatiotemporal attention or fully axial attention, and that pretraining on a multi-physics benchmark transfers to new physics mainly when attention is frozen and fine-tuning data are scarce.","feed_headline":"Adaptive patching beats uniform fine patches at half the tokens.","feed_subtitle":"Vision-transformer model for physical fields refines only high-variance patches, cutting token count without sacrificing accuracy.","key_machinery":"The mechanism that carries the accuracy-compute claim is variance-threshold adaptive tokenization. For each coarse patch, the model computes the variance of the solution field inside it; patches whose variance exceeds $\\gamma_{\\mathrm{sts}}$ times the maximum patch variance are split into smaller sub-token-scale (STS) patches, per Eq. (4). Two fusion strategies are defined: Adap Mix inserts the refined STS tokens directly into the token sequence so coarse and fine tokens attend together, and Adap Mul keeps coarse and refined tokens as separate sequences, merging them only at reconstruction. The attention variants, full spatiotemporal attention (ViT), time-then-space attention (SViT), and fully axial attention (AViT), are the comparators, and the paper's cost indices $L_{\\mathrm{lin}}$ and $L_{\\mathrm{quad}}$ make the sequence-length tradeoff quantitative.","core_discovery":"On its own terms, the central finding is that adaptive tokenization is not merely a compute-saving approximation: the paper reports cases where it improves accuracy over the uniformly fine patch model while roughly halving the average token sequence length. For Adap Mix settings such as $(p_x^1,p_x^{\\mathrm{sts}},\\gamma_{\\mathrm{sts}})=(32,16,0.2)$ and similar intermediate values, the final NRMSE falls below the $16\\times16$ or $8\\times8$ uniform cases at about half the sequence length. A second finding is about attention structure: SViT, which factors attention into a time pass followed by one full spatial pass, reaches lower test error than full spatiotemporal attention (ViT) and fully axial attention (AViT) for comparable parameter counts, while also costing less training time than AViT. The third finding is that pretraining transfer to genuinely new physical equations with new variables is real but muted: pretrained models beat randomly initialized models in low-data, frozen-attention fine-tuning, and the advantage largely disappears under full fine-tuning with sufficient data.","pith_inferences":["The variance criterion is the load-bearing choice; an editor would expect it to be tested against alternative refinement signals, such as local gradients, wavelet coefficients, or the actual error map of the coarse model.","If the $2\\times$ reduction scales with resolution, the method's practical payoff is largest in high-resolution CFD and climate applications, where uniform fine tokenization is the bottleneck, not in the small grids shown here.","Adap Mix's advantage over Adap Mul suggests that letting coarse and fine tokens attend to each other, cross-scale correlation, matters more than just having more tokens; ablating variance selection against random refinement would isolate this.","The fine-tuning results imply that foundation-model benchmarks for physics should routinely include tasks with entirely new variables, such as thermal or electromagnetic fields, rather than only same-equation out-of-distribution flows, because that is where pretraining gains become visible and measurable."],"forward_implications":["At the demonstrated $2\\times$ sequence-length reduction, adaptive tokenization makes ViT-based forecasting feasible for spatial grids where uniformly fine patch sizes would be computationally prohibitive.","By sweeping $\\gamma_{\\mathrm{sts}}$ from 1 to 0, Adap Mix interpolates continuously between a coarse-patch model and a fine-patch model, giving practitioners a single scalar knob to trade cost against accuracy.","The comparisons among attention schemes imply that fully decoupled axial attention should be avoided for spatiotemporal physics, while decoupling time only (SViT) preserves accuracy with lower cost than full spatiotemporal attention.","Pretrained representations transfer to new physical variables mainly when attention layers are frozen and fine-tuning data are scarce; full fine-tuning with enough data removes most of the pretraining advantage."],"supporting_citations":[{"why":"Supplies the five 2D PDEBench systems used for pretraining; all pretraining and transfer results depend on this data.","marker":"Takamoto et al., 2022"},{"why":"Provides the multi-physics preprocessor/postprocessor and pretraining recipe that MATEY adopts, and is the comparison point for fine-tuning performance.","marker":"McCabe et al., 2023"},{"why":"Introduces axial attention, the decomposition that defines the AViT scheme and its quadratic-cost estimate.","marker":"Ho et al., 2019"},{"why":"Introduces the factorized spatiotemporal attention that the SViT scheme is based on.","marker":"Arnab et al., 2021"},{"why":"Defines the colliding-thermals equations used for the adaptive-tokenization experiments and for one fine-tuning task.","marker":"Norman, 2024"},{"why":"Defines the lid-driven cavity MHD benchmark and its governing equations for the second fine-tuning task.","marker":"Fambri et al., 2023"},{"why":"Supplies the patch position and patch-area embedding method used in the adaptive tokenization module.","marker":"Bodnar et al., 2024"}],"fun_headline_variants":["Adaptive patches beat uniform grids at half the tokens","MATEY: adaptive tokens give better accuracy in half the sequence","Adaptive patch sizes cut tokens and improve physics accuracy","Token-efficient ViT for physics: half tokens, better accuracy","Adaptive tokenization: accuracy gain at half the compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy/compute benefit depends on the assumption that a coarse patch's solution variance is a reliable indicator of where smaller patches are needed, so that refining only the highest-variance patches preserves accuracy while shortening the token sequence.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive patches beat uniform grids at half the tokens","MATEY: adaptive tokens give better accuracy in half the sequence","Adaptive patch sizes cut tokens and improve physics accuracy","Token-efficient ViT for physics: half tokens, better accuracy","Adaptive tokenization: accuracy gain at half the compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3312,"prompt_tokens":970,"completion_tokens":2342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":2260}},"tokens_in":586,"tokens_out":2342,"duration_ms":15926,"temperature":1.0,"reasoning_tokens":2260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:16:31.392127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a holdout set, build an oracle refinement mask from the actual prediction error of the coarse-patch model and compare it with the variance mask from Eq. (4); if the variance-selected model needs substantially more tokens than the error-selected oracle to reach the same NRMSE, or fails on a low-variance thin boundary layer or smooth wide-amplitude gradient, the central tradeoff collapses.","supporting_citations":[],"review_version":1}