REVIEW 3 major objections 5 minor 51 references
This paper introduces a 2.4 TB benchmark for shock-driven compressible multiphase flows and shows that no single surrogate model wins across all datasets, while composite losses with interface- and structure-aware terms consistently improve
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 05:14 UTC pith:7QODXBRG
load-bearing objection The dataset and conservative downsampling are genuinely useful; the composite-loss comparisons are undermined by post-hoc selection of the training metrics. the 3 major comments →
Neptuna: A Comprehensive Machine Learning Framework for Benchmarking Complex Multiphase Flows
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the discovery is a negative result paired with a positive recipe: there is no architecture that dominates across shock–bubble and shock–droplet regimes, yet the training objective matters more than the architecture choice for the features that visual inspection cares about. Across six datasets, models trained with MSE alone reproduce the mean background flow (VRMSE below 1) but lose small-scale structure; augmenting with H1, SSIM and interface-weighted RMSE, with weights adjusted by SoftAdapt or GradNorm, cuts interface RMSE by 5–7% on bubble collapse and 40–50% on droplet breakup and improves wavelet-based spectral error and SSIM. SoftAdapt provides the most consis
What carries the argument
The central mechanism is the composite training objective built from four terms: MSE, H1-seminorm (first-derivative error), SSIM (local luminance/contrast/structure similarity), and IRMSE (error weighted by a soft mask around the density-derived interface), with per-term weights adapted during training by SoftAdapt (softmax-weighted recent loss slopes, no gradient storage). This objective counteracts the low-frequency bias of pure pointwise loss, and the benchmark measures its effects through autoregressive rollout evaluated by pointwise, wavelet-spectral, feature-focused, structure-aware, and physics-informed metrics.
Load-bearing premise
The gains claimed for composite losses depend on the loss terms and initial weights not being tuned on the same test distributions used to report the gains; the paper discloses in its limitations that the metrics were selected after observing which ones correlated positively, so a selection artifact could explain part of the improvement.
What would settle it
Run the same six datasets with the composite terms and initial weights fixed before any evaluation (no metric cherry-picking), compare MSE-only against SoftAdapt-trained models on a held-out set, and check whether IRMSE, MLW, and SSIM gains persist. If the improvements shrink or reverse, the benchmark's headline advantage is tied to the selection procedure, not to composite losses per se.
If this is right
- If composite losses generalize as claimed, practitioners should not treat MSE as sufficient for shock-driven multiphase surrogates; adding derivative, interface, and structure terms is a cheap way to recover fine-scale fidelity.
- Because no architecture wins everywhere, model selection must be regime- and metric-dependent: spectral operator models suit post-collapse dynamics where interfaces dissolve, while convolutional models hold a slight edge where coherent interfaces persist.
- SoftAdapt's consistency suggests adaptive balancing can be adopted almost for free, whereas GradNorm's roughly 5–10% memory overhead buys gains only on selected configurations.
- Conditioning matters: the pre-trained model with explicit conditioning/history windows leads; the one without underperforms despite its size, implying benchmark results depend on how simulation parameters are fed into the model.
- The release of 2.4 TB of high-fidelity trajectories with conservative downsampling supports reproducible comparisons and gives the community a testbed for next-generation surrogates.
Where Pith is reading between the lines
- (editorial inference) If the reported gains come from countering MSE's spectral bias, then losses acting directly on wavelet bands or vorticity may push further on the unresolved secondary droplets that the paper flags as a remaining gap.
- (editorial inference) The benchmark's design—mixing two breakup regimes in one dataset—invites a test of whether surrogates can infer regime switches from conditioning; inspecting predictions near the Weber-number transition would be a natural next step the paper does not take.
- (editorial inference) The conservative-downsampling result implies that earlier or future datasets that average primitive fields may embed O(10) momentum errors at interfaces; users of such datasets should verify conservation before attributing model errors to architecture.
- (editorial inference) SoftAdapt's near-zero overhead and consistency suggest it could transfer to other PDE surrogate tasks, but the evidence here only supports this within the six studied flow configurations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Neptuna, a benchmarking framework and dataset collection for shock-driven compressible multiphase flows, comprising six datasets (2D/3D bubble collapse and droplet breakup) with about 2.4 TB of high-fidelity simulation data generated with the ALPACA solver. It benchmarks six 2D baselines (ConvNeXt, CNO, FFNO, ScOT, DPOT-M, Poseidon-B) and two 3D baselines (ConvNeXt, FFNO), each trained with MSE alone and with a composite loss (MSE + H1 + SSIM + IRMSE) balanced by SoftAdapt or GradNorm. Evaluation covers pointwise, spectral, feature-focused, structure-aware, and physics-informed metrics. The main claims are that no single surrogate dominates across all datasets and metrics, that composite losses significantly improve interface preservation and spectral fidelity, and that SoftAdapt provides the most consistent improvements with negligible overhead. The paper also contributes a conservative downsampling method for multiphase data, detailed physics validation against external references, and a public benchmark repository and dataset.
Significance. If the benchmark is used as released, the datasets and conservative downsampling procedure are concrete, reusable contributions to the scientific-ML community; the physics validation against established references (Nourgaliev et al., Bempedelis and Ventikos, Meng and Colonius) strengthens trust in the data. The finding that no single architecture wins across all metrics is a useful calibration result. However, the headline loss-strategy claim currently rests on a post-hoc selection of the composite-loss terms that is disclosed but not quantified, and the aggregate tables lack uncertainty intervals and significance tests. These issues are fixable within the scope of the manuscript, but they must be addressed before the causal claim 'composite losses significantly improve...' can be accepted as stated.
major comments (3)
- [5.2 (Limitations)] The paper discloses that the composite loss terms H1, SSIM, and IRMSE "were obtained after analyzing results from a broad list of experiments and selectively picking the metric... when a positive correlation is obtained on the said metric or a diverse set of metrics during evaluation." It does not state whether this selection used the same test trajectories and metrics that appear in Tables 22-51, nor how many configurations were tried, nor whether any subset of the data was held out and untouched until final evaluation. Because the headline claim of improved interface preservation and spectral fidelity is based on exactly the metrics included in the training objective, the reported effect sizes may be selection artifacts. Please specify the selection protocol (e.g., a disjoint validation split) or re-run a nested evaluation that accounts for selection, or downgrade the causal claim acco
- [Tables 22-51 and Section 5.1] The aggregate tables report only scalar averages over trajectories and rollout steps, with no confidence intervals or significance tests. The papers' checklist points to Section C.2 for error bars, but those plots show trajectory variability bands, not uncertainty on the aggregated metrics used to support the "5-7%", "40-50%", and "consistent improvement" claims. The tables themselves contain counterexamples to "consistent": e.g., Table 24 shows ScOT IRMSE worsens from 0.9836 (MSE) to 1.0547 (SA), and Table 34 shows DPOT-M IRMSE worsens from 0.1522 (MSE) to 0.4103 (GN). A per-baseline win/loss/ite summary or a significance test is needed to support the consistency claim.
- [Section 5.1, Tables 25/30/35/40/45/50] SSIM is both a composite training term and a headline evaluation metric. The paper itself notes the SSIM improvement "may be partly attributed to the inclusion of SSIM as one of the terms in the composite training objective." This makes the SSIM result partially expected rather than independent evidence of better structure fidelity. Metrics not included in the training objective, such as SRMSE, wfRMSE, and iqRMSE, show mixed or negligible gains across the tables (e.g., Table 24 SRMSE is roughly flat or worse for several baselines). The paper should either weight non-training metrics in the headline comparison or explicitly frame the result as "training on these metrics improves the same metrics at inference" rather than as a general structure-fidelity improvement.
minor comments (5)
- [Section 3.3 / Figure 7] Typo: "Collaspe" should be "Collapse" in the section title and figure caption.
- [Section B.4.1] Typo: "Varience" should be "Variance" in the metric description.
- [Table 2] The end-time for 2D-SDBA is rendered as "91 - 1240" which is unclear; a unit or a range explanation would help.
- [Section B.2, Table 11] Typo in the row label: "Train time" appears as "Train timetime-restricted".
- [Section 5.1] Poseidon-B is excluded from the discussion with an explanation, but the radar charts still show it; this may confuse readers. Consider adding a footnote or explicit legend statement.
Circularity Check
Composite-loss improvement claims are partly selected on evaluation; the dataset and model-ranking results are self-contained.
specific steps
-
fitted input called prediction
[Section 5.2 (Limitations and Future Work); cf. Section 1 and Tables 12-13]
"The training metrics selected in this work for the composite losses were obtained after analyzing results from a broad list of experiments and selectively picking the metric to be included in the composite train loss only when a positive correlation is obtained on the said metric or a diverse set of metrics during evaluation."
The abstract's central loss claim ('composite losses significantly improve interface preservation and spectral fidelity') is certified using the same evaluation that was used to choose the composite terms (H1, SSIM, IRMSE, with MLW as a 'diverse' metric). The paper does not disclose a separate validation split, the number of configurations tried, or any hold-out that was untouched until final evaluation. Therefore the reported improvements on the selected metrics are not independent confirmations; they are the selection criterion itself. This affects only the loss-ablation claim: the benchmark data, model rankings, and the 'no single model best' conclusion remain externally supported by held-out test trajectories and references.
full rationale
The dataset generation is self-contained and externally validated: ALPACA/RDEMIC results are checked against Nourgaliev et al. (2006), Bempedelis & Ventikos (2020), and Meng & Colonius (2018), and the benchmark evaluations use held-out test trajectories with fixed metrics. No load-bearing self-citation chain or uniqueness import is present. The only circular component is the composite-loss ablation: H1, SSIM, and IRMSE appear both as training loss terms and as evaluation metrics, and Section 5.2 discloses that these terms were selected only after observing positive correlation during evaluation. Since no selection-on-validation protocol is reported, the headline improvements on those metrics are partly a post-hoc selection artifact. I therefore assign 4, not higher, because the dataset contribution, the per-metric tables, and the broader empirical comparisons retain independent content beyond the selected loss terms.
Axiom & Free-Parameter Ledger
free parameters (5)
- Initial composite loss weights (lambda_MSE, lambda_H1, lambda_SSIM, lambda_IRMSE) =
1.0, 2.0, 0.5, 1.0
- IRMSE density threshold ranges and softness =
SABW/SDBA: 350-550; SRBA: 4.5-8.5; softness 0.05; blur sigma 3.0
- SoftAdapt temperature (beta) =
1.0
- GradNorm asymmetry (alpha) and loss-weight learning rate =
1.0; 0.001
- wfRMSE level weights =
(1,0,0)/(0,1,0)/(0,0,1)
axioms (4)
- domain assumption ALPACA with RDEMIC accurately solves the compressible Euler equations with stiffened-gas EOS for the simulated configurations.
- ad hoc to paper The fixed density-threshold interface definition identifies physical phase boundaries in all trajectories.
- domain assumption Conservative downsampling preserves the information needed for surrogate training.
- domain assumption The chosen evaluation metrics sufficiently capture surrogate reliability.
read the original abstract
Compressible multiphase flows involving shocks and material interfaces arise in applications such as bubble collapse and droplet breakup, where strong nonlinear interactions produce complex interface deformation, mixing, and multiscale dynamics. Developing reliable machine learning surrogates for these flows remains challenging due to the simultaneous presence of compressibility, sharp discontinuities, and multiphase effects. In this work, we introduce the first large-scale benchmark specifically designed for shock-driven compressible multiphase flows, comprising 2.4 TB of high-fidelity 2D and 3D datasets featuring shock-induced bubble collapse and droplet breakup. We evaluate diverse surrogate model families on our benchmarking framework: Neptuna {https://github.com/tumaer/Neptuna}, including convolutional, spectral, transformer-based, and pre-trained PDE foundation models. Beyond standard MSE training, we investigate composite losses combining MSE with Sobolev, interface-aware, and structure-aware terms, together with adaptive loss balancing using SoftAdapt and GradNorm. Evaluation includes pointwise, spectral, feature-focused, structural, and physics-informed metrics. Results show that no single model performs best across all datasets and metrics, while composite losses significantly improve interface preservation and spectral fidelity. Among adaptive weighting strategies, SoftAdapt provides the most consistent improvements with almost no overhead compared to MSE-only training.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems , volume =
Pdebench: An extensive benchmark for scientific machine learning , author =. Advances in neural information processing systems , volume =
-
[2]
arXiv preprint arXiv:2209.15616 , year =
Towards multi-spatiotemporal-scale generalized pde modeling , author =. arXiv preprint arXiv:2209.15616 , year =
-
[3]
arXiv preprint arXiv:2307.14623 , year =
BubbleML: A multi-physics dataset and benchmarks for machine learning , author =. arXiv preprint arXiv:2307.14623 , year =
-
[4]
The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year =
Bubbleformer: Forecasting Boiling with Transformers , author =. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year =
-
[5]
Advances in Neural Information Processing Systems , volume =
The well: a large-scale collection of diverse physics simulations for machine learning , author =. Advances in Neural Information Processing Systems , volume =
-
[6]
Advances in Neural Information Processing Systems , volume =
Turbulence in focus: Benchmarking scaling behavior of 3D volumetric super-resolution with BLASTNet 2.0 data , author =. Advances in Neural Information Processing Systems , volume =
-
[7]
arXiv preprint arXiv:2601.01829 , year =
RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data , author =. arXiv preprint arXiv:2601.01829 , year =
-
[8]
arXiv preprint arXiv:2512.18595 , year =
Benchmarking neural surrogates on realistic spatiotemporal multiphysics flows , author =. arXiv preprint arXiv:2512.18595 , year =
-
[9]
Journal of Machine Learning Research , volume =
Climsim-online: A large multi-scale dataset and framework for hybrid physics-ml climate emulation , author =. Journal of Machine Learning Research , volume =
-
[10]
Advances in Neural Information Processing Systems , volume =
Apebench: A benchmark for autoregressive neural emulators of pdes , author =. Advances in Neural Information Processing Systems , volume =
-
[11]
International conference on machine learning , pages =
Learning to simulate complex physics with graph networks , author =. International conference on machine learning , pages =. 2020 , organization =
2020
-
[12]
arXiv preprint arXiv:2202.03376 , year =
Message passing neural PDE solvers , author =. arXiv preprint arXiv:2202.03376 , year =
-
[13]
International Conference on Machine Learning , pages =
Learning neural pde solvers with parameter-guided channel attention , author =. International Conference on Machine Learning , pages =. 2023 , organization =
2023
-
[14]
Advances in Neural Information Processing Systems , volume =
Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes , author =. Advances in Neural Information Processing Systems , volume =
-
[15]
ICLR 2023 workshop on physics for machine learning , year =
Convolutional neural operators , author =. ICLR 2023 workshop on physics for machine learning , year =
2023
-
[16]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages =
A convnet for the 2020s , author =. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages =
-
[17]
arXiv preprint arXiv:2111.13802 , year =
Factorized fourier neural operators , author =. arXiv preprint arXiv:2111.13802 , year =
-
[18]
Advances in Neural Information Processing Systems , volume =
Poseidon: Efficient foundation models for pdes , author =. Advances in Neural Information Processing Systems , volume =
-
[19]
2022 IEEE , author =
Swin transformer v2: Scaling up capacity and resolution. 2022 IEEE , author =. CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
2022
-
[20]
arXiv preprint arXiv:2403.03542 , year =
Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training , author =. arXiv preprint arXiv:2403.03542 , year =
-
[21]
2019 , url =
Sobolev Training With Higher Order Derivatives , author =. 2019 , url =
2019
-
[22]
2024 , eprint =
Sobolev Training for Operator Learning , author =. 2024 , eprint =
2024
-
[23]
2024 , eprint =
A generalised novel loss function for computational fluid dynamics , author =. 2024 , eprint =
2024
-
[24]
2022 , eprint =
Wavelet-based Loss for High-frequency Interface Dynamics , author =. 2022 , eprint =
2022
-
[25]
arXiv preprint arXiv:2006.13846 , year =
Understanding ssim , author =. arXiv preprint arXiv:2006.13846 , year =
Pith/arXiv arXiv 2006
-
[26]
IEEE Access , volume =
A hitchhiker’s guide to structural similarity , author =. IEEE Access , volume =. 2021 , publisher =
2021
-
[27]
arXiv preprint arXiv:1912.12355 , year =
Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions , author =. arXiv preprint arXiv:1912.12355 , year =
Pith/arXiv arXiv 1912
-
[28]
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2018) , author =. URL http://arxiv. org/abs/1711.02257. ArXiv , volume =
Pith/arXiv arXiv 2018
-
[29]
Computer Methods in Applied Mechanics and Engineering , volume =
Multi-objective loss balancing for physics-informed deep learning , author =. Computer Methods in Applied Mechanics and Engineering , volume =. 2025 , publisher =
2025
-
[30]
WaveLiT: A Parameter-Efficient Architecture for Neural PDE Solvers , author =
-
[31]
Physics-informed neural operator for learning partial differential equations. arXiv , author =. arXiv preprint arXiv:2111.03794 , year =
-
[32]
arXiv preprint arXiv:2402.09084 , year =
Sobolev Training for Operator Learning , author =. arXiv preprint arXiv:2402.09084 , year =
-
[33]
Manufacturing Letters , volume =
CRONet: A convolutional recurrent operator approximator network to accelerate topology optimization , author =. Manufacturing Letters , volume =. 2025 , publisher =
2025
-
[34]
Journal of Computational Physics , volume =
A robust high-resolution discrete-equations method for compressible multi-phase flow with accurate interface capturing , author =. Journal of Computational Physics , volume =. 2023 , publisher =
2023
-
[35]
Computer Physics Communications , volume =
ALPACA-a level-set based sharp-interface multiresolution solver for conservation laws , author =. Computer Physics Communications , volume =. 2022 , publisher =
2022
-
[36]
(No Title) , year =
Modern compressible flow: with historical perspective , author =. (No Title) , year =
-
[37]
Physics of Fluids , volume =
The Noble-Abel stiffened-gas equation of state , author =. Physics of Fluids , volume =. 2016 , publisher =
2016
-
[38]
Physics of fluids , volume =
On the physics of aerobreakup , author =. Physics of fluids , volume =. 2008 , publisher =
2008
-
[39]
11th International Symposium on Turbulence and Shear Flow Phenomena, TSFP 2019 , year =
Numerical investigation of 3d drop-breakup mechanisms using a sharp interface level-set method , author =. 11th International Symposium on Turbulence and Shear Flow Phenomena, TSFP 2019 , year =
2019
-
[40]
Journal of Fluid Mechanics , volume =
Numerical simulation of the aerobreakup of a water droplet , author =. Journal of Fluid Mechanics , volume =. 2018 , publisher =
2018
-
[41]
Journal of Computational Physics , volume =
Adaptive characteristics-based matching for compressible multifluid dynamics , author =. Journal of Computational Physics , volume =. 2006 , publisher =
2006
-
[42]
Journal of Fluid Mechanics , volume =
Energy focusing in shock-collapsed bubble arrays , author =. Journal of Fluid Mechanics , volume =. 2020 , publisher =
2020
-
[43]
2012 , publisher =
The acoustic bubble , author =. 2012 , publisher =
2012
-
[44]
Marine environmental research , volume =
Development of an air bubble curtain to reduce underwater noise of percussive piling , author =. Marine environmental research , volume =. 2000 , publisher =
2000
-
[45]
2017 , publisher =
Atomization and sprays , author =. 2017 , publisher =
2017
-
[46]
arXiv preprint arXiv:1607.06450 , year =
Layer normalization , author =. arXiv preprint arXiv:1607.06450 , year =
-
[47]
In Proceedings of the AAAI Conference on Artificial Intelligence , year =
Visual reasoning with a general conditioning layer, Courville , author =. In Proceedings of the AAAI Conference on Artificial Intelligence , year =
-
[48]
2013 , publisher =
Riemann solvers and numerical methods for fluid dynamics: a practical introduction , author =. 2013 , publisher =
2013
-
[49]
Journal of computational physics , volume =
A shock-stable modification of the HLLC Riemann solver with reduced numerical dissipation , author =. Journal of computational physics , volume =. 2020 , publisher =
2020
-
[50]
Journal of computational physics , volume =
Efficient implementation of weighted ENO schemes , author =. Journal of computational physics , volume =. 1996 , publisher =
1996
-
[51]
Mathematics of computation , volume =
Total variation diminishing Runge-Kutta schemes , author =. Mathematics of computation , volume =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.