{"id":"b914f30d-0405-4798-a753-c083ee42f844","arxiv_id":"2507.03272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"S-NOT, a GRU-transformer hybrid, predicts full-field solutions of time-dependent nonlinear PDEs with lower error than Sequential DeepONet on steel solidification, 3D lug, and dogbone benchmarks.","lead":"This paper introduces S-NOT, a neural network that combines recurrent memory with transformer attention to predict stress and temperature fields in time-dependent simulations. It reports lower prediction errors than the Sequential DeepONet baseline on three engineering test problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The S-NOT-vs-S-DON comparison lacks controlled training settings, matched splits, and repeated-seed/paired statistics, so the 'consistently higher accuracy' claim is not yet established.","rationale":"The reader's weakest assumption—that S-DON was trained with the same hyperparameter budget and that the splits are matched—is exactly the load-bearing point. The paper's architecture and code release are genuine strengths; the concern is not that S-NOT is wrong, but that the comparative evidence is under-specified. The missing training settings, the unexplained dog-bone split, and the absence of repeated-seed or paired statistics all bear directly on the word 'consistently' in the central claim. The proposed check is feasible because the code is public. The reader's CONDITIONAL verdict already captures this uncertainty, so no adjustment to the verdict is needed.","tokens_in":11827,"tokens_out":4445,"duration_ms":55219,"concrete_test":"Using the released code, retrain both S-DON and S-NOT on the dog-bone data with identical hyperparameter tuning budgets (same number of random-search trials, same optimizer and epoch schedule) and the same 3,200/800 split, across at least five random seeds. Then compute per-sample paired L2 errors and report the median difference with a 95% bootstrap confidence interval for each dataset. If the interval includes zero, or if S-DON wins under some seeds, the 'consistently higher accuracy' claim is not supported. As a secondary check, retrain S-DON on the original 12,000/3,000 dog-bone split to see whether the baseline error drops substantially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a comparative one: S-NOT 'consistently achieves a higher prediction accuracy than S-DON' on three benchmarks. For that claim to hold, S-DON must be a fair baseline and the observed differences must be statistically credible. Section 4 gives no training configuration—optimizer, learning rate, epochs, batch size, early stopping, or random seeds—for either model. Moreover, the dog-bone dataset is split 3,200/800 in Section 3.3 while the other two datasets use 80/20, and it is not stated whether the S-DON dog-bone results are reproduced under this smaller split or quoted from He et al. [27] under a different split. Table 1 reports only means and standard deviations across test samples from a single run per model; the large standard deviations (e.g., steel stress 18.1 ± 23.2 for S-DON vs 4.3 ± 15.3 for S-NOT) show that the per-sample error distributions are heavy-tailed. The claim of 'consistent' superiority, especially 'even for data outliers,' requires paired comparison and seed variation to rule out that the gap comes from an undertuned S-DON, an unequal split, or run-to-run stochasticity rather than the attention mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Sequential Neural Operator Transformer (S-NOT), a surrogate model that combines GRU-based sequential encoding with transformer-style self-attention and cross-attention to map time-dependent loading histories to full-field PDE solutions. The model is benchmarked on three FEM-generated datasets: multiphysics steel solidification, a 3D lug specimen, and a dog-bone specimen under path-dependent loading. The central claim, stated in the abstract and Section 1, is that S-NOT consistently achieves higher prediction accuracy than S-DON across all test cases, including outlier samples. Table 1 reports lower mean errors for S-NOT on all datasets, and Table 2 indicates comparable inference time and parameter counts. However, the evaluation is based on a single run per model, reports no training hyperparameters, uses a nonstandard split for the dog-bone dataset, and provides only qualitative evidence for the outlier claim, leaving the comparative claim insufficiently supported.","tokens_in":12108,"tokens_out":4629,"duration_ms":51956,"significance":"The attention-based cross-attention decoder is a natural and potentially impactful modification of S-DON for sequential operator learning, and the three realistic nonlinear PDE benchmarks are well chosen. If the claimed accuracy gains are confirmed under controlled comparison, S-NOT could provide a useful surrogate for path-dependent material simulations. The paper also makes its code and trained models available on GitHub and the datasets on Zenodo, which is a strength for reproducibility. The main weakness is that the empirical comparison is not yet rigorous enough to establish the central claim: missing training configurations, single-run results with large per-sample standard deviations, and an unmatched data split for the dog-bone dataset leave room for alternative explanations of the reported accuracy gap.","major_comments":[{"comment":"The central claim that S-NOT 'consistently achieves a higher prediction accuracy than S-DON' is not supported by the present evaluation because no training hyperparameters (optimizer, learning rate, batch size, epochs, early stopping, random seeds) are reported for either model, and only a single run per model is presented. The per-sample standard deviations are large (e.g., steel stress: 4.3 ± 15.3 for S-NOT vs. 18.1 ± 23.2 for S-DON), so the observed differences may be within run-to-run stochasticity. Please provide full training configurations for both models, repeat each model over multiple seeds, and report paired statistics (e.g., per-sample error-difference distributions or a signed-rank test) to substantiate the consistency claim.","section":"Section 4, Table 1"},{"comment":"The dog-bone dataset uses a 3,200/800 train/test split, which is neither the 80/20 split used for the other two datasets nor justified relative to the original 15,000 samples. The manuscript does not state whether the reported S-DON dog-bone results were reproduced on this exact split or quoted from He et al. [27] under a different split. This makes the S-DON comparison potentially unequal and could bias the reported accuracy gap. Please justify the split and ensure both models are trained and evaluated on identical data partitions.","section":"Section 3.3, Dog-bone split"},{"comment":"The claim of superior accuracy 'even for data outliers' is supported only by qualitative percentile plots (Figs. 6, 9, 10, 12, 13) and error histograms, with no quantitative tail metrics (e.g., 90th/95th percentile errors, tail-mean, or the fraction of test samples where S-NOT outperforms S-DON). Given the heavy-tailed error distributions in Table 1, the robustness claim requires such statistics to be substantiated.","section":"Abstract, Section 4.3, Conclusion"}],"minor_comments":[{"comment":"The abstract contains the phrase 'a architecture' and should read 'an architecture'; it also uses 'S-DeepONet' while the main text predominantly uses 'S-DON', which should be made consistent.","section":"Abstract"},{"comment":"The text says 'the finnal state of von Mises stress' and should be corrected to 'the final state'.","section":"Section 3.3"},{"comment":"The dog-bone row for S-NOT parameter count reads '7,90952', which appears to be a typo for '790,952'.","section":"Table 2"},{"comment":"Equation (7) is introduced with 'given in Eq. (5)', which is incorrect; it should refer to its own equation number.","section":"Section 3.1, Eq. (7)"},{"comment":"The sentence 'S-NOT offers similar inference speed and parameter count to S-DON' is imprecise because Table 2 shows S-NOT has fewer parameters in the 3D Lug case but more in the other two cases; the statement should be qualified.","section":"Section 4"},{"comment":"The attention formula in Eq. (2) is labeled generically; the text should explicitly clarify that cross-attention in the decoder uses the same formula with Q derived from query points and K, V from the encoder, as described in the prose.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound architectural idea and reproducible artifacts, but the comparative evaluation needs to be made rigorous before it is publishable in a serious journal. I recommend major revision and would be open to reviewing a revised version that addresses the training-configuration and split-matching issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible incremental architecture — replace the dot-product merge in S-DON with cross-attention and add self-attention to the GRU encoder — and the reported error reductions on three engineering benchmarks are large. But the empirical claim is not yet nailed down: there are no training settings, no repeated seeds, no paired statistics, and the dog-bone split doesn't match the other datasets. The paper deserves peer review, but the authors need to make the baseline comparison fair.\n\nWhat's actually new: S-NOT is a straightforward combination of two existing ideas, but I don't see it in the cited literature. The GRU encoder from S-DON is kept, and the decoder uses query-point embeddings with cross-attention over the encoder outputs. That's a clean way to let each query point selectively pull from the loading history. The NeRF-style positional encoding for query points is a nice touch. The paper is well written and the architecture is clear. The datasets are real FEM simulations from their own prior work, and the code, trained models, and data are on GitHub/Zenodo. That is reproducible evidence, and I believe the reported numbers are what they say they are.\n\nWhere it's soft: the baseline comparison is under-specified. No optimizer, learning rate, epochs, batch size, or seeds are given for either model. The dog-bone dataset is split 3,200/800 out of 15,000 samples, while the other two are 80/20, and it is not stated whether the S-DON dog-bone numbers were reproduced under this smaller split or quoted from He et al. [27]. The standard deviations in Table 1 are huge — steel stress is 18.1 ± 23.2 for S-DON and 4.3 ± 15.3 for S-NOT — so the error distributions are heavy-tailed. Showing a few percentile samples supports the qualitative direction but not the strong 'even for data outliers' claim. A paired comparison across the whole test set with a handful of seeds would settle it. Also, the title says 'high-fidelity' but the absolute errors are still several percent and the tails are long; I'd tone that down.\n\nWho should read it: people working on surrogate models for path-dependent plasticity will find a useful architecture and a clear baseline to beat. The paper is a decent candidate for peer review, but it should be a conditional accept: the authors need to add training settings, multiple seeds, and a matched dog-bone split. If the gap survives that, the 'consistent' wording is fine; if not, the paper is still a useful incremental study but with a weaker claim.","headline":"S-NOT is a plausible incremental architecture but the empirical claim of consistent superiority over S-DON needs a fairer, multi-seed comparison before it is established.","tokens_in":12605,"tokens_out":4365,"would_cite":true,"duration_ms":44378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","65M99"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing the dot-product fusion in Sequential DeepONet with cross-attention between the encoded loading sequence and spatial query points consistently improves prediction accuracy on three challenging time-dependent…","keywords":["Transformer","Sequential Neural Operator","Deep Learning","Attention Mechanism","Surrogate Modeling","Time-dependent PDEs","GRU","Operator Learning"],"falsifier":"Retrain both models on the same three datasets with each model's hyperparameters tuned separately using equal budgets and several random seeds, then compute mean and 90th-percentile errors with confidence intervals. If S-DON matches or beats S-NOT in mean or tail errors under matched tuning, the claim that attention-based fusion is more accurate would be refuted.","tokens_in":11636,"feed_emoji":"⚙️","tokens_out":5574,"duration_ms":62658,"temperature":0.7,"pith_summary":"The paper introduces Sequential Neural Operator Transformer (S-NOT), a surrogate architecture for predicting full solution fields of time-dependent nonlinear PDEs. The central claim is that replacing the dot-product fusion used by Sequential DeepONet (S-DON) with cross-attention between the encoded loading history and spatial query points yields consistently lower prediction errors on three challenging problems: multiphysics steel solidification, a 3D lug under time-varying pressure, and a dog-bone specimen under sequential displacement. The reported improvements are substantial: mean stress relative $L^2$ error drops from $18.1\\%$ to $4.3\\%$ on steel solidification, from $11.6\\%$ to $5.31\\%$ on the lug, and from $2.01\\%$ to $1.13\\%$ on the dog-bone, while parameter count and GPU inference time stay comparable and remain orders of magnitude faster than finite element simulation. The paper argues this makes S-NOT a practical surrogate for workflows that require many forward evaluations, such as uncertainty quantification and optimization.","feed_headline":"Attention fusion beats dot-product S-DON on nonlinear PDE surrogates","feed_subtitle":"Cross-attention between load history and query points cuts error tails on casting, lug, and dog-bone benchmarks.","key_machinery":"The central mechanism is cross-attention between the encoded loading history and the spatial query points, defined as $\\mathrm{Attention}(Q,K,V) = \\mathrm{softmax}(QK^T/\\sqrt{d_e})V$. The GRU stack preserves temporal causality in the loading sequence, the sinusoidal positional encoding injects time information explicitly, self-attention lets the encoder re-weight the sequence, and cross-attention lets each spatial point aggregate the most relevant parts of the history instead of relying on a single inner product. This replacement of dot-product fusion by attention is the component the paper credits for the accuracy gain.","core_discovery":"S-NOT uses a GRU encoder to process a sequence of loading values in time, adds sinusoidal positional encodings, and refines the representation with self-attention blocks; the resulting sequence serves as keys and values for the decoder. In the decoder, each spatial query point is embedded with a NeRF-style positional encoding followed by an MLP to form a query, and cross-attention selects the relevant portions of the load history for that point. This replaces the fixed dot product of branch and trunk outputs in S-DON. In the paper's tests, the change consistently improves accuracy, and the largest gains appear on high-error outlier samples, so the error tails become shorter.","pith_inferences":["Because the accuracy gain concentrates in the error tails, S-NOT should be most valuable in safety-critical applications such as predicting hot tearing in continuous casting, where rare large errors dominate risk.","The cross-attention decoder is agnostic to the choice of sequence encoder, so S-NOT could be combined with other temporal encoders or with physics-informed loss terms without changing the core architecture.","A natural next test is whether S-NOT retains its advantage on loading histories sampled from distributions held out during training, a stronger out-of-distribution check than the outlier analysis reported here."],"forward_implications":["On all three reported benchmarks, S-NOT attains a lower mean error than S-DON for stress, temperature, and plastic strain fields.","S-NOT's test error distributions have shorter tails, so the model is more reliable on difficult or outlier samples where S-DON's accuracy degrades.","S-NOT keeps roughly the same number of parameters and inference speed as S-DON, so the accuracy gain does not add much deployment cost.","Surrogate inference (about $10^{-3}$ seconds per sample on GPU) is orders of magnitude faster than the reference finite element runs (20 to 333 seconds per sample on CPU), enabling repeated forward evaluations.","The design is supported by the theoretical result that cross-attention transformers are universal operator approximators, giving the attention-based fusion a principled basis beyond the empirical gains."],"supporting_citations":[{"why":"Supplies the S-DON baseline, the sequential GRU branch idea, and the dog-bone dataset used as one of the three benchmarks.","marker":"[27]"},{"why":"Establishes the original DeepONet branch-trunk dot-product fusion that S-NOT replaces.","marker":"[17]"},{"why":"Provides the transformer attention mechanism and sinusoidal positional encoding reused in S-NOT.","marker":"[28]"},{"why":"Gives the theoretical universal-approximation result for cross-attention transformers that motivates replacing the inner product.","marker":"[34]"},{"why":"Supplies the NeRF-style positional encoding used to embed spatial query points in the decoder.","marker":"[36]"},{"why":"Provides the viscoplastic constitutive model for austenite that drives the nonlinearity in the steel solidification benchmark.","marker":"[37]"},{"why":"Provides the power-law model for delta-ferrite that is part of the steel solidification material response.","marker":"[38]"},{"why":"Describes the multiphysics thermo-mechanical solidification model used to generate the first dataset.","marker":"[40, 41]"}],"fun_headline_variants":["S-NOT: cross-attention beats dot-product on nonlinear PDE surrogates","GRU plus attention fusion improves surrogate accuracy on hard PDE tasks","S-NOT cross-attention cuts error tails on casting, lug, and dogbone surrogates","Cross-attention fusion outperforms dot-product for PDE surrogates on outliers","S-NOT: attention-based merging shortens error tails on tough PDE benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that S-DON was trained with the same hyperparameters, training budget, and data splits as S-NOT; the paper does not report those settings, so if S-DON was undertuned the reported accuracy gap could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["S-NOT: cross-attention beats dot-product on nonlinear PDE surrogates","GRU plus attention fusion improves surrogate accuracy on hard PDE tasks","S-NOT cross-attention cuts error tails on casting, lug, and dogbone surrogates","Cross-attention fusion outperforms dot-product for PDE surrogates on outliers","S-NOT: attention-based merging shortens error tails on tough PDE benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001099,"raw_usage":{"total_tokens":4583,"prompt_tokens":937,"completion_tokens":3646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3542}},"tokens_in":553,"tokens_out":3646,"duration_ms":27645,"temperature":1.0,"reasoning_tokens":3542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:13:53.260083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain both models on the same three datasets with each model's hyperparameters tuned separately using equal budgets and several random seeds, then compute mean and 90th-percentile errors with confidence intervals. If S-DON matches or beats S-NOT in mean or tail errors under matched tuning, the claim that attention-based fusion is more accurate would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the S-DON baseline, the sequential GRU branch idea, and the dog-bone dataset used as one of the three benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the original DeepONet branch-trunk dot-product fusion that S-NOT replaces."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Provides the transformer attention mechanism and sinusoidal positional encoding reused in S-NOT."},{"cited_title":"Kozlowski, B","cited_arxiv_id":null,"evidence_quote":"Provides the viscoplastic constitutive model for austenite that drives the nonlinearity in the steel solidification benchmark."},{"cited_title":"Zhu, Coupled Thermo-Mechanical Finite-Element Model with Application to Initial Solidification, Ph.D","cited_arxiv_id":null,"evidence_quote":"Provides the power-law model for delta-ferrite that is part of the steel solidification material response."}],"review_version":1}