{"id":"eb73e69f-67c8-4d15-97cd-09ba0c3aae23","arxiv_id":"2501.01453","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"On the FlowBench lid-driven cavity benchmark, vision-transformer foundation models outperform neural operators in data-limited regimes, but all models generalize poorly to out-of-range Reynolds numbers and geometry generalization is not actually tested.","lead":"This paper benchmarks 11 machine-learning models that predict steady fluid flow around complex 2D shapes, testing two geometry encodings and several training set sizes. It finds that pretrained transformer models beat neural operators in low-data settings, while all models degrade sharply on Reynolds numbers outside the training range.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80/20 random sample split cannot support the paper's geometry-generalization claims, because each of the 300 geometries contributes 10 Reynolds samples that are shared between train and test; a geometry-stratified split is required.","rationale":"The paper has real strengths: it uses a public dataset, provides code, evaluates several model families, and reports both field-accuracy and physics-consistency metrics. However, the most load-bearing assumption for the central 'complex geometries' claim is that the evaluation protocol measures generalization to geometries not seen during training. Because each geometry contributes 10 Reynolds-number simulations and the random split is sample-wise, the test set is dominated by geometries whose other Reynolds samples appear in training. The extrapolatory split changes only the Reynolds range and similarly shares geometries. Therefore the headline comparisons and the OOD discussion do not establish geometry generalization. The reader's conditional verdict correctly identifies this as the key weakness, so no verdict adjustment is needed. The suggested geometry-stratified retest is the single check that would determine whether the comparative findings survive on truly unseen shapes.","tokens_in":19569,"tokens_out":3954,"duration_ms":41859,"concrete_test":"Re-run the main comparison with a geometry-stratified split: train on 240 geometries (2,400 samples) and test on the remaining 60 geometries (600 samples), ensuring the Reynolds ranges in train and test overlap; report M1, M2, M3 and the unified score for scOT-T, Poseidon-T, CNO, FNO, and Geo-DeepONet for both SDF and mask. If the foundation-model advantage and representation ordering reproduce on fully unseen geometries, the central claims survive; if the gap collapses or reverses, the reported scores conflate geometry interpolation with geometry generalization, and the generalization claims should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 states that the 3,000 samples are randomly divided 80/20, and Section 4.1 states that the dataset consists of 300 geometries each paired with 10 Reynolds numbers. A sample-wise random split therefore puts simulations of the same shape in both training and test sets: about 268 of the 300 geometries have at least one test sample, and only ~32 geometries are fully held out. This means the 'Random' scores in Tables 1 and 2 largely measure prediction at unseen Reynolds numbers for geometries already seen in training, not prediction around complex geometries in general. The 'extrapolatory' experiments in Section 2.3 restrict test/train by Reynolds number only, so they also share the same geometries; they test Reynolds extrapolation, not geometry extrapolation. The introduction explicitly lists 'extrapolatory regimes (of both geometry and Reynolds number)' as a research question, but no geometry-stratified experiment is reported. The central finding that foundation models outperform neural operators could still be true, but as implemented the benchmark does not test the geometry-generalization component of the claim. In addition, the abstract's statement that SDF yields superior results with sufficient training data is not consistently supported by Tables 1 and 2, where scOT/Poseidon/CNO score higher with masks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks 11 scientific-machine-learning models on the FlowBench 2D lid-driven cavity dataset (300 geometries, 10 Reynolds numbers each, 3000 samples). It compares SDF and binary-mask geometry encodings, studies training-set-size scaling, and evaluates random versus Reynolds-extreme 'extrapolatory' splits. A unified 0-100 score combines global MSE, boundary-layer MSE, and PDE momentum residual. The headline claims are that scOT/Poseidon foundation models outperform neural operators, especially in low-data regimes, and that SDF representations become superior with sufficient data, while all models struggle out of distribution.","tokens_in":19846,"tokens_out":5233,"duration_ms":52559,"significance":"The question is timely and the dataset/code release is a useful service. The paper is an honest attempt to compare many architectures under a common protocol, and the appendix provides substantial per-model detail. However, the current experimental design does not test geometry generalization because the sample-wise split leaks geometry identity; the SDF claim contradicts the main tables; and single-run comparisons lack uncertainty quantification. With a geometry-stratified split and revised claims, the benchmark could be a valuable reference for the SciML community.","major_comments":[{"comment":"The 80/20 random split is sample-wise, not geometry-stratified. Since each of the 300 geometries contributes 10 Reynolds-number simulations, only about 32 geometries are completely held out (0.8^10 per geometry); the remaining ~268 geometries appear in both train and test. Consequently, the 'Random' results in Tables 1 and 2 measure interpolation/extrapolation in Reynolds number for mostly seen geometries, not generalization to unseen complex geometries. The 'extrapolatory' split in Section 2.3 also varies only Reynolds number, so it does not test geometry extrapolation either. The geometry-generalization claims in the abstract and Section 3 are therefore not supported by the reported experiments. Please add a split that holds out all 10 samples for a disjoint set of geometries and report those scores separately.","section":"Section 2.1, Section 4.1"},{"comment":"The claim that 'SDF representations yield superior results with sufficient training data' is contradicted by the full-data random scores: poseidon-T scores 66.0 M1 with mask versus 64.9 with SDF, scOT-T 67.8 versus 64.6, and CNO 47.3 versus 44.8; SDF is better only for some neural operators (e.g., FNO, geometric-DeepONet). The conclusion that the best geometric representation is model-dependent is supported, but the global SDF-superiority claim should either be removed or replaced by a per-model, per-metric analysis.","section":"Abstract, Section 2.1, Tables 1 and 2"},{"comment":"All results appear to be single runs, with no random seeds, repetitions, or confidence intervals described in the experimental protocol. Many of the headline comparisons are small (e.g., scOT-T versus poseidon-T differ by 0.3-1.8 points in M1), so the statement that scOT/Poseidon 'consistently deliver the best performance, often by a significant margin' is not statistically supported. Please provide multiple seeds or error bars, or at least state explicitly that the margins are not significance-tested.","section":"Sections 2.2 and 2.4, Tables 1-2 and A.1-A.8"},{"comment":"The data-sufficiency experiments reduce the number of samples by drawing random subsets of the already-leaked training split, so the smaller configurations simultaneously reduce the number of geometries and the Reynolds coverage per geometry. The claim that 'neural operators reach an asymptotic error limit around 800 samples' conflates sample count with geometry diversity. The authors should report the number of distinct geometries and the Reynolds range covered at each training size, or stratify the training subsets by geometry.","section":"Section 2.2, Figure 4"}],"minor_comments":[{"comment":"Appendix A says the results are for 'one-third and one-tenth of the original data', but Table A.1 is labeled 'a subset of half of the dataset'; reconcile the labels and fix the 'exrapolatory' typo.","section":"Appendix A"},{"comment":"The score's endpoints MSEmax=1 and MSEmin=1e-6 are hand-chosen; because the score is a monotone transform of MSE, rank order per metric is unaffected, but absolute statements like 'score=50' should be presented as conventions rather than calibrated accuracy.","section":"Section 4.3"},{"comment":"Several tables contain repeated 0.0 entries for WNO's M3 metric; please clarify whether these are genuine residual values or placeholder scores from predictions that could not be evaluated.","section":"Tables 1, 2, A.1-A.8"},{"comment":"The caption says 'sample size of 300' in the third row while the text and tables refer to 240 samples; correct the inconsistency.","section":"Figure A.2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's main message is not novel in method, but the benchmark could be useful if the experimental claims are made trustworthy. Before acceptance, the authors should rerun with a geometry-stratified split and temper the SDF claim; the current version overclaims geometry generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful benchmark, but the paper's geometry-generalization framing goes beyond what the experimental setup can support. The 80/20 random sample split shares each geometry's 10 Reynolds simulations between train and test, so the \"random\" scores are mostly testing interpolation to new Reynolds numbers for geometries already seen in training. That is a real flaw in the headline claim. The benchmark itself—11 models, two representations, five training sizes, Reynolds extrapolation—is carefully executed and the results are informative.\n\nWhat is genuinely new is the systematic comparison on the FlowBench LDC subset. The finding that scOT and Poseidon consistently beat neural operators, especially in low-data regimes, is well supported by the tables. The data-sufficiency curves are a nice practical contribution, and the M3 residual metric adds a useful physical-consistency axis. The paper ships code and uses a public dataset, so the numbers are reproducible. The observation that representation choice is model-dependent, not universal, is honest and well documented.\n\nSoft spots, in order of severity. First, the missing geometry-stratified split is load-bearing. Section 2.1 says the 3,000 samples are randomly split 80/20, and Section 4.1 says each of 300 geometries has 10 Reynolds samples. So about 268 geometries appear in both train and test. The paper explicitly lists \"extrapolatory regimes (of both geometry and Reynolds number)\" as a research question, but no geometry-held-out experiment exists. This means the geometry-generalization claims, including the abstract's, are not directly supported. Second, the abstract says SDF representations yield superior results with sufficient training data, but Tables 1 and 2 show scOT/Poseidon/CNO score higher with masks; the body even says so. That is an abstract-overstatement issue. Third, there are no repeated runs or error bars, so some score gaps could be noise. Fourth, the scoring bounds (MSEmin=1e-6, MSEmax=1) are hand-chosen, but they are clearly documented and defensible.\n\nWho is this for? Practitioners choosing SciML models for flow prediction, and benchmark builders. It deserves serious refereeing because the comparative results are novel and relevant, but a revised version should add a geometry-stratified split, correct the SDF overstatement, and report variance across seeds. I would send it to review, not desk-reject it, and ask for those changes before acceptance.","headline":"The data-efficiency benchmark is real and useful, but the geometry-generalization claim is unsupported because the random split shares geometries between train and test.","tokens_in":20359,"tokens_out":1939,"would_cite":true,"duration_ms":20281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a 3,000-simulation complex-geometry flow benchmark, transformer-based operator models scOT and Poseidon beat classical neural operators, with the largest gap in low-data settings.","keywords":["Scientific machine learning","Neural operators","Vision transformers","Signed distance fields","Binary masks","Lid-driven cavity flow","Out-of-distribution generalization","Data efficiency"],"falsifier":"Retrain scOT-T, Poseidon-T, CNO, and DeepONet on a geometry-stratified split that holds out entire geometries or entire shape families instead of individual simulations, and compare held-out-shape MSE; if the transformer-based margin disappears or reverses, the central generalization claim is refuted.","tokens_in":19405,"feed_emoji":"🌊","tokens_out":9099,"duration_ms":80629,"temperature":0.7,"pith_summary":"The paper sets out to show that for predicting steady fluid flow around intricate two-dimensional geometries, the model class and the geometry encoding should be chosen together. Using the FlowBench 2D lid-driven cavity subset—3,000 simulations over 300 complex shapes—it benchmarks five classical neural operators against six variants of the transformer-based scOT/Poseidon family, under two geometry representations (signed distance fields and binary masks), several training-set sizes, and a Reynolds-number extrapolation split. Its central empirical claim is that scOT and Poseidon are the most accurate models, often by an order of magnitude in mean squared error, and that their advantage is most pronounced when training data is scarce, with pretrained Poseidon-T reaching roughly $10^{-4}$ MSE below 800 samples. It also claims that neither geometry representation is universally best: binary masks help scOT, Poseidon, and CNO, while SDF helps the other neural operators. If correct, the results give practitioners a concrete selection rule—transformer-based operators for data-limited complex-geometry flow prediction, with representation tuned per architecture—and a unified scoring scale for comparing future models.","feed_headline":"Transformer-based operators beat neural operators on complex flows","feed_subtitle":"A 3,000-simulation benchmark shows the win is largest with little data, and the best geometry encoding depends on the model.","key_machinery":"The argument is carried by a benchmark design and a scoring device. The design pairs each of 300 complex geometries with 10 Reynolds numbers, feeds the model a Reynolds scalar plus either a binary mask or a signed distance field at $512 \\times 512$ resolution, and asks it to output velocity components and pressure. The models are grouped into neural operators—FNO, CNO, WNO, DeepONet, and geometric-DeepONet—and six scOT/Poseidon variants; scOT is a hierarchical multiscale vision transformer built from Swin Transformer V2 and ConvNeXt residual blocks in a U-Net encoder-decoder, and Poseidon is the same architecture pretrained on PDE datasets. The scoring device is a logarithmic score mapping MSE $\\text{max}=1$ (a zero-prediction baseline) to 0 and MSE $\\text{min}=10^{-6}$ (CFD numerical accuracy) to 100, applied to global MSE (M1), boundary-layer MSE inside SDF $\\le 0.2$ (M2), and the $\\ell^2$ norm of Navier-Stokes momentum residuals (M3). This unified scale is what lets the paper compare models across accuracy, near-wall behavior, and physical consistency.","core_discovery":"On its own terms, the paper's discovery is a measured performance ranking with an architecture-dependent geometry-encoding result. Across random and extrapolatory test splits, scOT and Poseidon consistently deliver the lowest global and boundary-layer error, with Poseidon-T and scOT-T reaching scores around 65 on the random split (MSE near $10^{-4}$) while FNO, CNO, DeepONet, and WNO lag well behind; pretraining gives Poseidon its largest advantage when only a few hundred training samples are available, and by full data scOT matches it. The paper further finds that binary masks improve scOT, Poseidon, and CNO, whereas signed distance fields improve the other neural operators, and that neural operators saturate near 800 training samples with the mask representation while transformer-based models continue to improve with more data. On the physical-consistency metric, DeepONet records the lowest momentum residual despite worse pointwise accuracy. Finally, all models degrade sharply on the extrapolatory split, where test Reynolds numbers lie in the top or bottom 10 percent of the training range, showing that out-of-distribution generalization remains unsolved.","pith_inferences":["An editorial check: the paper's random 80/20 split is over individual simulations, and since each of the 300 geometries contributes 10 Reynolds numbers, the same shapes appear in both training and test sets; this means the reported accuracy may overstate generalization to unseen geometries, and a geometry-stratified holdout is needed to confirm the ranking.","If the data-efficiency result transfers beyond lid-driven cavities, pretrained operator transformers could serve as general-purpose initializations for steady-flow surrogates, cutting the cost of building new CFD surrogates.","The M3 residual metric appears to reward smoother fields, so combining it with gradient errors—as the paper's appendix does—may better capture the accuracy that lift and drag calculations require.","A direct testable extension would be to hold out entire shape families (NURBS, spherical harmonics, skeleton-derived shapes) and fine-tune Poseidon on two families while testing on the third; this isolates whether the advantage comes from geometric generalization or from memorizing seen shapes."],"forward_implications":["A practitioner with a small CFD dataset around complex 2D shapes can expect scOT or Poseidon to reach usable accuracy around $10^{-4}$ MSE with a few hundred samples, where classical neural operators are roughly an order of magnitude worse.","Geometry representation should be chosen per architecture: binary masks for scOT, Poseidon, and CNO; signed distance fields for FNO, DeepONet, geometric-DeepONet, and WNO.","Neural operators trained with binary masks stop improving near 800 samples, so extra data for them has little payoff; transformer-based models can still convert additional samples into lower error.","DeepONet remains the best choice when satisfying the momentum equation matters more than pointwise accuracy, and it is also the fastest at inference.","Out-of-distribution Reynolds numbers defeat all models regardless of training-set size, so deployment beyond the training range needs a different strategy."],"supporting_citations":[{"why":"Supplies the FlowBench 2D lid-driven cavity dataset of 3,000 simulations over 300 complex geometries that every model is trained and tested on.","marker":"[19]"},{"why":"Defines the Poseidon/scOT architecture and the pretrained weights used for the six transformer-based variants.","marker":"[43]"},{"why":"Provides the Fourier neural operator baseline whose performance is compared against the transformer-based models.","marker":"[47]"},{"why":"Provides the convolutional neural operator baseline, including the CNO results reported in the benchmark.","marker":"[48]"},{"why":"Provides the wavelet neural operator baseline used as one of the classical neural operators.","marker":"[49]"},{"why":"Provides the DeepONet branch-trunk architecture baseline, which also achieves the lowest physical-consistency residual.","marker":"[50]"},{"why":"Provides the geometric-DeepONet variant that incorporates geometry information into the trunk network.","marker":"[52]"},{"why":"Underpins the finite-element-style momentum residual computation used for the M3 physical-consistency metric.","marker":"[33]"},{"why":"Supplies the shifted boundary method used to generate the high-fidelity ground-truth flow solutions in the dataset.","marker":"[35]"}],"fun_headline_variants":["Transformers beat neural operators on complex flows","Best flow encoding depends on model type","Pretrained transformers win with few training samples","Out-of-distribution flows still defy SciML models","Benchmark: transformer-based models lead in accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the random 80/20 split of the 3,000 individual flow samples measures a model's ability to handle complex geometries, even though each of the 300 shapes contributes 10 Reynolds-number simulations, so the same shapes appear in both training and test sets; the extrapolatory split tests new Reynolds numbers, not new geometry.","fun_headline_variants_meta":{"raw":{"variants":["Transformers beat neural operators on complex flows","Best flow encoding depends on model type","Pretrained transformers win with few training samples","Out-of-distribution flows still defy SciML models","Benchmark: transformer-based models lead in accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001516,"raw_usage":{"total_tokens":6111,"prompt_tokens":1015,"completion_tokens":5096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":5028}},"tokens_in":631,"tokens_out":5096,"duration_ms":48406,"temperature":1.0,"reasoning_tokens":5028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:56:58.666314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain scOT-T, Poseidon-T, CNO, and DeepONet on a geometry-stratified split that holds out entire geometries or entire shape families instead of individual simulations, and compare held-out-shape MSE; if the transformer-based margin disappears or reverses, the central generalization claim is refuted.","supporting_citations":[{"cited_title":"Poseidon: Efficient foundation models for pdes, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the Poseidon/scOT architecture and the pretrained weights used for the six transformer-based variants."},{"cited_title":"Convolutional neural operators for robust and accurate learning of pdes, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the convolutional neural operator baseline, including the CNO results reported in the benchmark."},{"cited_title":"Learning nonlinear oper- ators via deeponet based on the universal approximation theorem of operators","cited_arxiv_id":null,"evidence_quote":"Provides the DeepONet branch-trunk architecture baseline, which also achieves the lowest physical-consistency residual."},{"cited_title":"Neural pde solvers for irregular domains","cited_arxiv_id":null,"evidence_quote":"Underpins the finite-element-style momentum residual computation used for the M3 physical-consistency metric."},{"cited_title":"The shifted boundary method for embedded domain computations","cited_arxiv_id":null,"evidence_quote":"Supplies the shifted boundary method used to generate the high-fidelity ground-truth flow solutions in the dataset."}],"review_version":1}