REVIEW 3 major objections 5 minor 11 references
Trained on simple synthetic data, a billion-parameter transformer reconstructs unseen salt-body velocity models, narrowing the FWI generalization gap.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 19:55 UTC pith:5HP7DVQI
load-bearing objection A credible scaling recipe for data-driven FWI with an honest ablation ladder, but the zero-shot generalization claim is broader than the evaluation: the benchmark seismic data use the same forward operator and acquisition geometry as training. the 3 major comments →
Improving Full Waveform Inversion in Large Model Era
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a working large-model recipe can bridge the generalization gap in data-driven FWI. The authors show that a 1.03-billion-parameter non-causal transformer, paired with a high-fidelity ViT-VQGAN tokenizer trained on diffusion-augmented data, predicts velocity maps from seismic waveforms with MAE 0.0136 on the standard synthetic benchmark, outperforming CNN baselines by a wide margin. More strikingly, the same model, with no retraining on realistic data, reconstructs six unseen benchmark families—including salt bodies, overthrust structures, and subsalt targets—with substantially lower error and higher SSIM (overall 0.7669 versus 0.5844). The mechanism they credit is co
What carries the argument
The load-bearing mechanism is a tokenized sequence-modeling pipeline with three coordinated parts: (1) a non-causal 1B-parameter transformer that predicts all velocity tokens in parallel with full bidirectional attention, rather than autoregressively; (2) a ViT-VQGAN tokenizer with an expanded 25×25×196 latent grid and no compression bottleneck, so that thin layers and faults survive discretization; (3) a post-training stack of map-level reinforcement learning and gradient descent in the tokenizer's latent space. Each component contributes: data augmentation alone brings the causal baseline from MAE 0.0770 to 0.0368, non-causal decoding to 0.0330, the high-fidelity tokenizer to 0.0167, and R
Load-bearing premise
The claim rests on the assumption that seismic observations can be adequately represented by the same fixed acoustic forward model used in training—five evenly spaced 15 Hz Ricker sources, 70 receivers, one second of recording—so that any realistic dataset, if available, would still fall within this simulation's physics; if real acquisition geometry, elastic effects, or noise break that proxy, the zero-shot generalization may collapse.
What would settle it
Generate a velocity map using different physics or acquisition geometry—for example, an elastic wave equation, irregular receiver placement, or a 30 Hz source—run the trained model without retraining, and compare SSIM to the reported 0.77; if the score falls to the 0.58 range of the baselines, the claimed generalization is specific to the fixed acoustic simulator rather than to geological complexity.
If this is right
- Large learned inverse solvers can be trained on small, simple synthetic datasets and still transfer to much more complex structures, removing the need for large realistic training corpora.
- Non-causal parallel decoding, not merely parameter count, is what unlocks transformer-based FWI; a causal 1B model underperforms a much smaller CNN baseline until decoding is changed.
- A high-fidelity tokenizer with a wide latent grid matters more than more data: tenfold data with the old tokenizer underperforms twofold data with the new tokenizer.
- Physics-based refinement can be applied in the tokenizer's latent space rather than to the raw velocity field, correcting small inconsistencies without losing sharp details.
- Model capacity, data diversity, and training strategy interact: increasing only one axis yields limited gains, while increasing all three produces the reported jump.
Where Pith is reading between the lines
- If the recipe transfers, the same three-axis scaling stack—capacity, generative augmentation, and RL alignment—could apply to other ill-posed inverse problems with differentiable forward operators, such as electromagnetic tomography or medical ultrasound imaging; the paper does not test this.
- The authors' own ablations imply the diffusion-generated data need not be realistic to help; it is the diversity and volume that matter. A testable extension would be to vary the fidelity of the synthetic corpus and measure how performance scales with it.
- Because the paper uses ensemble averaging over five stochastic samples, part of the gain may come from variance reduction rather than from the learned prior; isolating single-sample performance would clarify where the improvement actually lives.
- The reported zero-shot numbers come from acoustic synthetic benchmarks; a stress test with added noise, elastic effects, or irregular acquisition geometry would directly probe whether the generalization claim survives real-world physics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a billion-parameter transformer-based approach to Full Waveform Inversion, combining a ViT-VQGAN tokenizer, diffusion-based synthetic data augmentation, non-causal parallel decoding, RL-based fine-tuning with a map-level reward, and latent-space gradient refinement. On the OpenFWI benchmark, the full pipeline reports MAE 0.0136, outperforming the BigFWI baselines. The authors further evaluate zero-shot performance on six geophysical benchmark velocity models (Marmousi, SEG/EAGE, BP, Sigsbee, SEAM) and report substantially improved SSIM over BigFWI, interpreting this as narrowing the generalization gap in data-driven FWI.
Significance. If the results are reproducible, the paper offers a valuable scaling recipe for large-model FWI and provides thorough component-wise ablations (data scale, model capacity, tokenizer, RL, latent GD). The central claim, however, is stronger than the evidence: the zero-shot benchmarks are generated with the same acoustic forward operator and acquisition geometry as the training data, so the experiments demonstrate generalization to unseen velocity structures under the same synthetic acquisition protocol, not the synthetic-to-real domain shift implied by 'generalization gap.' The paper would be significant after reframing this claim and adding comparisons with a broader set of baselines.
major comments (3)
- [§4.3, Table 2; Abstract] The central claim of 'substantially narrows the long-standing generalization gap' is not supported by the evaluation protocol. In §4.3, all realistic benchmarks (Marmousi, SEG/EAGE, 2004 BP, Sigsbee, SEAM) are evaluated using seismic data generated with 'the same acoustic forward-modeling operator as in OPENFWI' (same source count, Ricker frequency, receiver count, and timesteps), and all velocity maps are downsampled to 70×70 and rescaled to [1500,4500] m/s. Thus the only distribution shift is in the velocity structure; the physics, noise level, and acquisition geometry are identical to training. The results show generalization to unseen velocity model families within the same synthetic manifold, not a narrowing of the domain gap between synthetic training and real seismic observations. The abstract and conclusion should be revised to state precisely what is demonstrated, or new experim
- [§4.2, Table 1] The paper claims 'state-of-the-art results on the synthetic OPENFWI benchmark' but the only comparison is against BigFWI-M and BigFWI-L. Many other data-driven FWI methods (e.g., InversionNet, VelocityGAN, SimFWI, and recent transformer-based methods) have published results on OpenFWI subsets. Without comparing against these or at least citing and tabulating previously reported numbers under the same evaluation protocol, the SOTA claim is not established. Please either add broader baselines or temper the claim to 'outperforms the BigFWI baselines.'
- [§3.3, Eq. (9); Appendix A] The RL objective is described as encouraging 'geological continuity and physical plausibility,' but Eq. (9) defines the reward as the negative L1 distance between the generated velocity map and the ground-truth label. No explicit continuity or physical-plausibility term is defined. This is a mismatch between the stated mechanism and the actual reward. Either specify additional reward terms and their weights, or revise the description to state that RL is a supervised label-based reward at the map level rather than a physics-aware reward.
minor comments (5)
- [§4.2, after Table 1] The text states that '×10 data for 10 epochs achieved an MAE of 0.0287, while ×2 data for 50 epochs achieved 0.0247,' but these numbers do not appear in Table 1. Clarify which configuration each number corresponds to (e.g., with or without ViT-VQGAN) and reconcile with the reported 0.0167 for the final ViT-VQGAN model.
- [Appendix A, Table 3] The caption says 'Qualitative comparison' but the table is quantitative. Change to 'Quantitative comparison.'
- [§4.3] Typo in Table 2: 'Marmosi' should be 'Marmousi.' Also, 'as we use an acoustic forward modeling' is awkward; rephrase.
- [§4.1 and §3.2] The data augmentation claims expansion to 'over five million velocity–seismic pairs,' but the final backbone is trained on '×2 data' (about 816k pairs). Clarify whether the full 5M set is used only for tokenizer training and explain the final training set size precisely.
- [§3.3 and Appendix A] Hyperparameters for the GRPO-style RL (e.g., reward coefficient, group size, learning rate, number of steps) are not reported. Such details are necessary for reproducibility of the RL post-training stage.
Circularity Check
No significant circularity; generalization claim is empirical and tested on external velocity benchmarks, with a scope limitation rather than a circular derivation.
full rationale
The paper's derivation chain is not circular. The central claims—state-of-the-art OPENFWI results and strong zero-shot generalization—are empirical outcomes evaluated against external benchmark velocity models (Marmousi, SEG/EAGE, BP, Sigsbee, SEAM) that are not used in training. The model components (non-causal transformer, ViT-VQGAN, diffusion augmentation, RL, latent GD) are individually ablated and show independent performance gains; none is defined in terms of the benchmark target metric. The RL objective (Eq. 9) uses ground-truth velocity labels during training, which is standard supervised/RL fine-tuning, not a fitted parameter renamed as a prediction. Self-citations to OpenFWI (Deng et al. 2022) and BigFWI (Jin et al. 2024) are dataset/baseline references and are not load-bearing for the main argument. A scope limitation does exist: Section 4.3 states that benchmark seismic data are 'generated with the same acoustic forward-modeling operator as in OPENFWI: five evenly spaced sources with a 15 Hz Ricker wavelet, recorded by 70 receivers sampling 1,000 timesteps over one second,' so the test distribution shares forward physics and acquisition geometry with training. This narrows the meaning of 'realistic generalization' to unseen velocity structures rather than field-data domain shift, and the conclusion acknowledges 'simplified 2D acoustic settings.' That is an overclaim/evidence-scope issue, not circularity: the evaluation still tests generalization to unseen velocity models and compares against external baselines. No circular step is present.
Axiom & Free-Parameter Ledger
free parameters (5)
- Final data scaling factor (2x) =
2x (selected over 10x after ablations: 10x gave MAE 0.0287, 2x gave 0.0247 under same compute)
- ViT-VQGAN latent grid size =
25x25x196
- Ensemble sample count =
5 samples per velocity map
- Benchmark downsampling resolution =
70x70
- RL reward coefficient / GRPO hyperparameters =
Not fully specified; described only as 'GRPO-style optimization'
axioms (3)
- domain assumption Acoustic wave equation (Eq. 1) is an adequate forward model for seismic wave propagation.
- domain assumption OpenFWI synthetic dataset plus diffusion-augmented samples provide sufficient structural diversity for generalization.
- ad hoc to paper Diffusion-generated velocity maps with FID 260.33 are useful training data despite low fidelity.
read the original abstract
Full Waveform Inversion (FWI) is a highly nonlinear and ill-posed problem that aims to recover subsurface velocity maps from surface-recorded seismic waveforms data. Existing data-driven FWI typically uses small models, as available datasets have limited volume, geological diversity, and spatial extent, leading to substantial concerns about overfitting. Although they perform well on synthetic datasets, current methods fail to generalize to more realistic geological structures. In this work, we show that a model trained entirely on simulated and relatively simple data can generalize remarkably well to challenging and unseen geological benchmarks. We provide a working recipe that tames a billion-parameter model for FWI through coordinated scaling across three axes: model capacity, data diversity, and training strategy. Our model achieves state-of-the-art performance on OpenFWI and significantly narrows the generalization gap in data-driven FWI. Across six challenging geophysical benchmarks, including Marmousi, 2D SEG/EAGE Salt and Overthrust, 2004 BP, Sigsbee, and SEAM Phase I, it infers complex structures absent from the training set and delivers significant performance improvements (SSIM from 0.5844 to 0.7669). Overall, our results demonstrate that with an appropriate scaling strategy, large models trained on simple synthetic data can achieve substantial generalization to more complex and realistic geological structures.
Figures
Reference graph
Works this paper leans on
-
[5]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,
-
[7]
Subsalt multiple attenuation and imaging: Observations from the sigsbee2b synthetic dataset
Josef Paffenholz, Bill McLain, Joerg Zaske, and P Joseph Keliher. Subsalt multiple attenuation and imaging: Observations from the sigsbee2b synthetic dataset. InSEG international exposition and annual meeting, pages SEG–2002. SEG,
2002
-
[10]
Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627,
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627,
-
[11]
Group sequence policy optimization.arXiv preprint arXiv:2507.18071,
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,
-
[1990]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,
1901
-
[2018]
Hanchen Wang, Yinpeng Chen, Tariq Alkhalifah, Ting Chen, Youzuo Lin, and David Alumbaugh. Default: Deep-learning based fault delineation using the ibdp passive seismic data at the decatur co2 storage site.arXiv preprint arXiv:2311.04361, 2023b. Yue Wu and Youzuo Lin. Inversionnet: An efficient and accurate data-driven full waveform inversion.IEEE Transact...
-
[2019]
Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,
-
[2020]
Diffusion posterior sampling for general noisy inverse problems
Hyungjin Chung, Jeongsol Kim, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In11th International Conference on Learning Representations, ICLR 2023,
2023
-
[2022]
Auto-linear phenomenon in subsurface imaging
10 Yinan Feng, Yinpeng Chen, Peng Jin, Shihang Feng, and Youzuo Lin. Auto-linear phenomenon in subsurface imaging. InProceedings of the 41st International Conference on Machine Learning. JMLR.org, 2024a. Yinan Feng, Hanchen Wang, Yinpeng Chen, Luoyuan Zhang, Jeeun Kang, Yixuan Wu, Young Jin Kim, and Youzuo Lin. Wavediffusion: Joint latent diffusion for ph...
-
[2024]
The 2004 bp velocity benchmark
FJ Billette and Sverre Brandsberg-Dahl. The 2004 bp velocity benchmark. In67th EAGE Conference & Exhibition, pages cp–1. European Association of Geoscientists & Engineers,
2004
-
[2025]
Randy Harsuko, Shijun Cheng, and Tariq Alkhalifah. Propagating the prior from shallow to deep with a pre-trained velocity-model generative transformer network.Journal of Geophysical Research: Machine Learning and Computation, 2(1):e2024JH000408, 2025a. Randy Harsuko, Shijun Cheng, and Tariq Alkhalifah. Velocity model building with uncertainty quantificati...
Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.