{"id":"2c44a3b5-eaed-4c0c-93af-9d9e90866fd2","arxiv_id":"2607.13866","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Benchmarking four neural operators for airfoil and NASA CRM pressure prediction: Transolver best on 2D, BSMS-GNN best on 3D; UPT and GAOT lag.","lead":"This paper evaluates four deep-learning surrogate models for predicting surface pressure on airfoils and a NASA CRM aircraft. It finds Transolver most accurate in 2D and BSMS-GNN in 3D, recommending those two for aerospace use.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"3D ranking of BSMS-GNN over Transolver++ rests on single runs with unequal batch sizes and no repetition; the reported 32–34% gap may not be robust.","rationale":"The reader's verdict is CONDITIONAL, based on the lack of repetition runs and ad hoc 3D hyperparameters. My stress-test identifies the same weakest assumption but sharpens it to a concrete fairness issue: the unequal batch sizes give BSMS-GNN four times more gradient updates than Transolver++, which could materially inflate the observed 32–34% error gap. The 2D experiment is solid (10 seeds, confidence intervals, tuned hyperparameters) and supports Transolver > BSMS-GNN > GAOT > UPT, though even there the Transolver/BSMS gap is modest. For the 3D experiment, the paper's own disclaimer ('no statistical statement') and the batch-size inequality mean the specific ranking of BSMS-GNN over Transolver++ is not established. The broad conclusion that both are promising surrogates is robust because it holds in both experiments, but the precise 3D ordering should not be used for architecture selection without confirmation. Therefore, I do not change the CONDITIONAL verdict: the paper is a useful benchmark, but its 3D ranking needs repetition with matched training budgets before being accepted as definitive.","tokens_in":36505,"tokens_out":4775,"duration_ms":49735,"concrete_test":"Retrain Transolver++ and BSMS-GNN on the NASA CRM dataset with matched training budgets: set Transolver++ batch size to 1 (same as BSMS-GNN), keep all other 3D hyperparameters as in Table 13–14, and run at least 3 seeds each. If the 95% CI for the MAE difference (BSMS-GNN minus Transolver++) overlaps zero, the 3D hierarchy is not supported under equal update counts. If the gap remains significant, the ranking survives this fairness correction. A smaller-scale pilot using 1 epoch of evaluation per seed could be run first.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim — that BSMS-GNN surpasses Transolver++ on the NASA CRM dataset (Section 5.1, Table 9) — is load-bearing for the conclusion highlighting BSMS-GNN as a promising surrogate. However, this result is based on a single training run per model with hyperparameters transferred or ad hoc scaled from 2D, and the paper explicitly disclaims statistical significance (Section 5.1: 'do not allow us to make any statistical statement'). A concrete fairness flaw: Table 10 shows BSMS-GNN, UPT, and GAOT were trained with batch size 1, while Transolver++ used batch size 4. Over 10,000 epochs, this gives BSMS-GNN ~1.05M gradient updates and Transolver++ ~260k updates—a 4x difference. The reported 32–34% error gap could partly reflect this unequal update count. Additionally, Section 6 admits greater prior experience with BSMS-GNN than with the other architectures, which can bias 3D setup choices (e.g., increasing scales to 11, switching seeding heuristic). Since the 2D results with confidence intervals are solid, the concern specifically targets the 3D ranking reversal and the strength of the recommendation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks four operator-learning architectures (BSMS-GNN, Transolver/Transolver++, UPT, GAOT) for surface-pressure prediction on two aerodynamic datasets: a 2D airfoil dataset (598 CST-perturbed RAE2822 geometries) and the industrial-scale 3D NASA CRM dataset (149 samples, 454k surface points each). The 2D experiment performs per-model hyperparameter optimization on a validation split, trains ten instantiations per model, and reports 95% confidence intervals; the ranking is Transolver > BSMS-GNN > GAOT > UPT. The 3D experiment trains a single instantiation per model with 2D-derived or ad hoc hyperparameters and reports BSMS-GNN > Transolver++ > GAOT > UPT, with the two best models separated by 32–34% in several error metrics. The paper concludes that BSMS-GNN and Transolver(++) are the most promising surrogate models.","tokens_in":36843,"tokens_out":4361,"duration_ms":41225,"significance":"If the results hold, the paper provides a valuable practitioner-oriented comparison for a class of problems (aerodynamic surface pressure prediction) where public benchmarks are scarce. The 2D experiment is a model of good empirical practice: hyperparameter search on a held-out validation split, ten retrained instantiations, 95% confidence intervals, a local error analysis as a function of flow parameters, and explicit discussion of failure cases, including shock-capture behavior. The derivation of supernode pooling radii in Appendix A.3 is a useful methodological detail. However, the 3D experiment, which drives the reversal of the ranking and the final recommendation, is based on one run per model without a statistically valid comparison. The paper itself disclaims any statistical statement for the 3D results (Section 5.1). The load-bearing 3D claim therefore needs either additional repetitions or a substantially softened interpretation.","major_comments":[{"comment":"The central claim that BSMS-GNN surpasses Transolver++ on the NASA CRM dataset rests on a single training run per model. Section 5 states that the authors \"abstained from doing repetition runs\" and Section 5.1 states that the results \"do not allow us to make any statistical statement.\" The paper attempts to substitute the lack of confidence intervals by appealing to gaps \"multiple times larger\" than the 2D confidence intervals, but this does not control for run-to-run variance in the 3D setting, where training is more unstable and hyperparameters were not optimized. The ranking reversal and the conclusion \"BSMS-GNN and Transolver(++) as promising surrogate models\" (Section 7) depend on this single-run comparison. At minimum, the 3D ranking should be reported as preliminary, and the conclusion should be correspondingly qualified.","section":"Section 5.1, Table 9"},{"comment":"The comparison between BSMS-GNN and Transolver++ is confounded by unequal numbers of gradient updates. BSMS-GNN, UPT, and GAOT were trained with batch size 1, while Transolver++ used batch size 4 to \"keep its training time feasible.\" Over 10,000 epochs with 105 training samples, this yields roughly 1.05M parameter updates for BSMS-GNN versus roughly 262.5k for Transolver++ — a factor of 4. The reported 32–34% error gap (Section 5.1) may therefore reflect training effort rather than architectural superiority. A fair comparison should equalize the number of gradient updates (or use repetitions over several update budgets) and should report the update count for each model.","section":"Section 5.1, Table 10"},{"comment":"The 3D setup choices are not demonstrably fair to all architectures. Section 6 notes that prior experience with BSMS-GNN \"exceeds the experience present with all other three architectures,\" and Appendix A.1 shows that BSMS-GNN uses the dataset-specific adjacency matrix, while the other models' 3D hyperparameters (e.g., UPT's 20,480 supernodes, GAOT's 131,072 supernodes, Transolver++'s four layers and hidden size 256) are chosen by \"previous experiences\" or \"to the best of our knowledge.\" Because the paper recommends BSMS-GNN, the absence of a documented process for equal tuning effort (e.g., a small validation-based search or ablations for all models) is a load-bearing concern. The 3D ranking should be labeled as conditional on the selected hyperparameters, or the authors should provide evidence that the ranking is robust to reasonable alternative settings.","section":"Section A.1 and Section 6"}],"minor_comments":[{"comment":"The RMSE and Rel. L2 metrics are defined as the mean over samples of the per-sample values. This is an unconventional choice; readers may expect a pooled RMSE over all points. Please state explicitly that these are per-sample averages and justify why this is preferable for this benchmark.","section":"Appendix A.2"},{"comment":"UPT received a second round of hyperparameter optimization and a custom exponential-decay scheduler, while the other models did not receive comparable additional tuning. This is a reasonable response to the initially poor performance, but the asymmetry should be discussed as a possible source of bias, and the final ranking should be interpreted with this in mind.","section":"Section 4.1"},{"comment":"The caveat that the 3D experiment has no hyperparameter optimization and no repetition runs appears only in Section 5. Given that the abstract and conclusion emphasize the 3D results, this limitation should be stated in the abstract and revisited in the conclusion so that readers are not misled.","section":"Section 5"},{"comment":"Several typos and minor language issues: \"quadradic\" (Section 2), \"archtitecture\" (Section 1), \"tupel\" (Section 2), \"noticable\" (Section 5.1), \"opimization\" (Section 4.1). A careful proofread is needed.","section":"Throughout"},{"comment":"The appendix tables are detailed and helpful. However, the connection between the entries and the corresponding paragraphs in Section A.1 could be made clearer by referring to table numbers in the text (e.g., \"see Table 11 for BSMS-GNN\").","section":"Tables 11–18"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a useful benchmark, and the 2D experiment is methodologically strong. The 3D experiment, however, is not at the same standard, and the paper's headline conclusion leans heavily on it. I would encourage the editor to require either additional 3D repetitions (especially for BSMS-GNN and Transolver++) with equalized update counts, or a substantial rewriting that presents the 3D ranking as preliminary. I also note that BSMS-GNN is the authors' own architecture (Hines and Bekemeyer 2026) and that the underlying datasets are co-authored by a co-author of this manuscript; a clear conflict-of-interest statement and a description of the measures taken to ensure fairness would strengthen the credibility of the benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading. First, the 2D airfoil study is genuinely solid: they do per-model hyperparameter search on a validation split, ten retrained instantiations, 95% confidence intervals, and a careful look at shock handling. That result — Transolver, then BSMS-GNN, GAOT, UPT — is the paper's real contribution. Second, the 3D NASA CRM ranking, where BSMS-GNN jumps ahead of Transolver++ by 32–34%, is not established. It rests on a single run per model, no 3D hyperparameter tuning, and an uneven training budget: Table 10 shows BSMS-GNN, UPT, and GAOT got batch size 1 over 10,000 epochs while Transolver++ got batch size 4, so BSMS-GNN received roughly four times as many gradient updates. The paper itself says the results 'do not allow us to make any statistical statement,' and Section 6 admits the authors have far more experience with BSMS-GNN than with the others. The reversal may hold, but this evidence doesn't show it.\n\nWhat's new: this is the first head-to-head of BSMS-GNN, Transolver(++), UPT, and GAOT on the DLR 2D airfoil and NASA CRM datasets. That is useful. The local error analysis and compute table are also well done, and the subjective handling section is honest — they flag their own familiarity with BSMS-GNN. Reproducibility is a real gap: no code, data, or commit hashes, though hyperparameters are documented in detail.\n\nSoft spots in proportion. The 2D result is solid, but note the gaps between Transolver, BSMS-GNN, and GAOT are modest and some confidence intervals overlap; it's a clear ordering, not a blowout. UPT being worst on 2D is believable given their reported tuning difficulty, but the architecture is known to be finicky; a different tuner might get better results. The 3D experiment is the load-bearing weakness. The conclusion that BSMS-GNN and Transolver++ are promising for aerospace pressure prediction is plausible, but the specific claim that BSMS-GNN beats Transolver++ on 3D should not be used to discard Transolver++.\n\nFor a reading group, this is a good case study on benchmarking pitfalls and honesty about limitations. Practitioners should use the 2D ranking and treat the 3D results as preliminary. I would send this to serious peer review — a referee can push for repeated 3D runs and fair tuning — but I'd expect major revision.","headline":"The 2D airfoil benchmark is careful and useful, but the 3D ranking reversal is a single-run artifact with unequal training budgets — don't take BSMS-GNN's lead over Transolver++ seriously yet.","tokens_in":37348,"tokens_out":2726,"would_cite":true,"duration_ms":26102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four deep-learning operator models are benchmarked for predicting surface pressure in aerodynamics, and the ranking points to Transolver and BSMS-GNN as the most promising surrogates.","keywords":["neural operators","operator learning","surrogate models","aerodynamics","surface pressure prediction","graph neural networks","transformers","shock prediction"],"falsifier":"Retrain each model on the NASA CRM dataset with multiple seeds and a per-architecture hyperparameter optimization (e.g., systematically varying learning rate, batch size, and pooling radii). If, across repetitions, Transolver++ or another model achieves lower MAE and RMSE than BSMS-GNN with non-overlapping confidence intervals, the paper's 3D ranking is falsified.","tokens_in":36305,"feed_emoji":"✈️","tokens_out":2711,"duration_ms":28577,"temperature":0.7,"pith_summary":"This paper benchmarks four state-of-the-art operator-learning architectures—BSMS-GNN, Transolver(++), UPT, and GAOT—on aerodynamic surface pressure prediction, using a 2D airfoil dataset and an industrial-scale 3D NASA CRM aircraft dataset. On the 2D dataset, Transolver achieves the highest accuracy, closely followed by BSMS-GNN, then GAOT, with UPT clearly last. On the 3D dataset, the order reverses at the top: BSMS-GNN scores lowest errors, followed by Transolver++, then GAOT and UPT, with much larger gaps between models. The paper argues that BSMS-GNN and Transolver(++) are the most promising surrogate models for aerospace applications, while also noting that the best-performing models tend to be the most computationally lightweight.","feed_headline":"Transolver beats three rivals on airfoils; BSMS-GNN wins on full aircraft","feed_subtitle":"Head-to-head tests on airfoil and aircraft datasets identify which deep surrogate models handle shocks, geometry, and scarce data best.","key_machinery":"The comparison is carried by the architectural mechanisms underlying each model: BSMS-GNN's bi-stride multi-scale graph construction with a U-Net structure permits efficient long-range information exchange on user-provided graphs; Transolver's Physics attention groups mesh points into a small number of learned slices, making self-attention linear in the number of points; UPT uses a GNO encoder to pool input points into latent tokens processed by a transformer, with predictions via cross-attention; GAOT uses a multiscale attentional GNO (MAGNO) encoder to map input clouds to a regular latent grid. The benchmark protocol itself—hyperparameter optimization and ten repeated instantiations on 2D,","core_discovery":"On the paper's own terms, the central discovery is a performance ranking: on the 2D airfoil dataset, the highest accuracy is achieved by Transolver, followed tightly by BSMS-GNN, then GAOT, and somewhat behind, UPT; on the NASA CRM dataset, BSMS-GNN achieves the lowest errors across all metrics, followed by Transolver++, then GAOT, and UPT performs markedly worse. The authors further find that the best results in both settings are produced by the most lightweight models in terms of parameters and training time, and they highlight BSMS-GNN and Transolver(++) as promising surrogate models for aerodynamical applications.","pith_inferences":["The 3D ranking rests on a single training run per model with hyperparameters largely transferred from the 2D study or scaled manually; a fairer multi-seed comparison with per-model tuning on the CRM dataset could plausibly change the BSMS-GNN/Transolver++ gap.","The fact that the simpler, physics-attention-based Transolver matches or beats heavier hybrid architectures on 2D suggests that explicit multiscale graph construction is not always necessary for learning surface pressure on modest geometries.","The consistent misplacement of shocks across models points to a concrete research direction: incorporating shock-capturing or discontinuity-aware loss terms into operator learning could improve aerospace surrogate accuracy more than further architecture search.","If the reported 3D ordering holds under repeated runs, it implies that for high-resolution surface meshes with scarce training data, a deliberately coarsened graph with multi-scale message passing may generalize better than transformer-heavy latent-token designs."],"forward_implications":["Practitioners building surrogate models for similar aerospace pressure-prediction tasks can prioritize BSMS-GNN and Transolver(++) over UPT and GAOT, based on the reported error metrics.","The reversal of the top-two ranking between the 2D and 3D datasets indicates that architecture choice should depend on geometry complexity and dataset size, not just on general-purpose claims.","Since the most accurate models were also the most computationally efficient, large parameter counts are not necessary for good performance on these aerodynamic datasets.","All models degrade in the presence of strong shocks, especially near design-space boundaries, so shock prediction remains a shared weakness that future work would need to address.","The NASA CRM dataset's data-scarcity setting suggests that BSMS-GNN and Transolver++ can generalize from roughly 100 training samples to industrial-scale meshes, which is relevant to real engineering workflows."],"fun_headline_variants":["Transolver wins airfoils, BSMS-GNN tops aircraft","Lightweight models lead: Transolver and BSMS-GNN","Airfoil surrogates: Transolver, jet: BSMS-GNN","Benchmark crowns Transolver and BSMS-GNN"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 3D ranking assumes that a single training run per model, with hyperparameters chosen by transferring results from 2D or scaling them ad hoc, is enough to fairly establish the NASA CRM hierarchy, so an unlucky or favorable configuration for any architecture could change the reported order.","fun_headline_variants_meta":{"raw":{"variants":["Transolver wins airfoils, BSMS-GNN tops aircraft","Lightweight models lead: Transolver and BSMS-GNN","Airfoil surrogates: Transolver, jet: BSMS-GNN","Benchmark crowns Transolver and BSMS-GNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1082,"prompt_tokens":817,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":191}},"tokens_in":561,"tokens_out":265,"duration_ms":3022,"temperature":1.0,"reasoning_tokens":191,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:27:48.642742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain each model on the NASA CRM dataset with multiple seeds and a per-architecture hyperparameter optimization (e.g., systematically varying learning rate, batch size, and pooling radii). If, across repetitions, Transolver++ or another model achieves lower MAE and RMSE than BSMS-GNN with non-overlapping confidence intervals, the paper's 3D ranking is falsified.","supporting_citations":[],"review_version":1}