Pith. sign in

REVIEW 3 major objections 5 minor 11 references

Trained on simple synthetic data, a billion-parameter transformer reconstructs unseen salt-body velocity models, narrowing the FWI generalization gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-02 19:55 UTC pith:5HP7DVQI

load-bearing objection A credible scaling recipe for data-driven FWI with an honest ablation ladder, but the zero-shot generalization claim is broader than the evaluation: the benchmark seismic data use the same forward operator and acquisition geometry as training. the 3 major comments →

arxiv 2603.00377 v2 pith:5HP7DVQI submitted 2026-02-27 cs.LG

Improving Full Waveform Inversion in Large Model Era

classification cs.LG
keywords full waveform inversionseismic velocity reconstructionlarge model scalingnon-causal transformerViT-VQGAN tokenizerdiffusion data augmentationzero-shot generalizationlatent gradient refinement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that data-driven full waveform inversion does not need realistic training data: a billion-parameter transformer, trained only on simple simulated velocity–seismogram pairs, can reconstruct complex, unseen geological structures. The authors identify the reason previous small models failed as a mismatch between model capacity, data diversity, and training objective, not as a fundamental limit of learned inversion. They build a recipe along three axes: scaling model size to 1B parameters, expanding the training corpus tenfold with diffusion-generated physically paired samples, and aligning the model with map-level rewards and physics-based latent refinement. On standard synthetic benchmarks this recipe sets a new state of the art, and in zero-shot tests on six realistic benchmark families it improves structural similarity from around 0.58 to 0.77, recovering salt bodies and sharp interfaces that prior methods smooth away. If true, the result reframes the scaling paradigm for scientific inverse problems: large models can transfer from simple physics to complex structures, provided the right tokenizer and training pipeline.

Core claim

The central claim is that a working large-model recipe can bridge the generalization gap in data-driven FWI. The authors show that a 1.03-billion-parameter non-causal transformer, paired with a high-fidelity ViT-VQGAN tokenizer trained on diffusion-augmented data, predicts velocity maps from seismic waveforms with MAE 0.0136 on the standard synthetic benchmark, outperforming CNN baselines by a wide margin. More strikingly, the same model, with no retraining on realistic data, reconstructs six unseen benchmark families—including salt bodies, overthrust structures, and subsalt targets—with substantially lower error and higher SSIM (overall 0.7669 versus 0.5844). The mechanism they credit is co

What carries the argument

The load-bearing mechanism is a tokenized sequence-modeling pipeline with three coordinated parts: (1) a non-causal 1B-parameter transformer that predicts all velocity tokens in parallel with full bidirectional attention, rather than autoregressively; (2) a ViT-VQGAN tokenizer with an expanded 25×25×196 latent grid and no compression bottleneck, so that thin layers and faults survive discretization; (3) a post-training stack of map-level reinforcement learning and gradient descent in the tokenizer's latent space. Each component contributes: data augmentation alone brings the causal baseline from MAE 0.0770 to 0.0368, non-causal decoding to 0.0330, the high-fidelity tokenizer to 0.0167, and R

Load-bearing premise

The claim rests on the assumption that seismic observations can be adequately represented by the same fixed acoustic forward model used in training—five evenly spaced 15 Hz Ricker sources, 70 receivers, one second of recording—so that any realistic dataset, if available, would still fall within this simulation's physics; if real acquisition geometry, elastic effects, or noise break that proxy, the zero-shot generalization may collapse.

What would settle it

Generate a velocity map using different physics or acquisition geometry—for example, an elastic wave equation, irregular receiver placement, or a 30 Hz source—run the trained model without retraining, and compare SSIM to the reported 0.77; if the score falls to the 0.58 range of the baselines, the claimed generalization is specific to the fixed acoustic simulator rather than to geological complexity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Large learned inverse solvers can be trained on small, simple synthetic datasets and still transfer to much more complex structures, removing the need for large realistic training corpora.
  • Non-causal parallel decoding, not merely parameter count, is what unlocks transformer-based FWI; a causal 1B model underperforms a much smaller CNN baseline until decoding is changed.
  • A high-fidelity tokenizer with a wide latent grid matters more than more data: tenfold data with the old tokenizer underperforms twofold data with the new tokenizer.
  • Physics-based refinement can be applied in the tokenizer's latent space rather than to the raw velocity field, correcting small inconsistencies without losing sharp details.
  • Model capacity, data diversity, and training strategy interact: increasing only one axis yields limited gains, while increasing all three produces the reported jump.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the recipe transfers, the same three-axis scaling stack—capacity, generative augmentation, and RL alignment—could apply to other ill-posed inverse problems with differentiable forward operators, such as electromagnetic tomography or medical ultrasound imaging; the paper does not test this.
  • The authors' own ablations imply the diffusion-generated data need not be realistic to help; it is the diversity and volume that matter. A testable extension would be to vary the fidelity of the synthetic corpus and measure how performance scales with it.
  • Because the paper uses ensemble averaging over five stochastic samples, part of the gain may come from variance reduction rather than from the learned prior; isolating single-sample performance would clarify where the improvement actually lives.
  • The reported zero-shot numbers come from acoustic synthetic benchmarks; a stress test with added noise, elastic effects, or irregular acquisition geometry would directly probe whether the generalization claim survives real-world physics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a billion-parameter transformer-based approach to Full Waveform Inversion, combining a ViT-VQGAN tokenizer, diffusion-based synthetic data augmentation, non-causal parallel decoding, RL-based fine-tuning with a map-level reward, and latent-space gradient refinement. On the OpenFWI benchmark, the full pipeline reports MAE 0.0136, outperforming the BigFWI baselines. The authors further evaluate zero-shot performance on six geophysical benchmark velocity models (Marmousi, SEG/EAGE, BP, Sigsbee, SEAM) and report substantially improved SSIM over BigFWI, interpreting this as narrowing the generalization gap in data-driven FWI.

Significance. If the results are reproducible, the paper offers a valuable scaling recipe for large-model FWI and provides thorough component-wise ablations (data scale, model capacity, tokenizer, RL, latent GD). The central claim, however, is stronger than the evidence: the zero-shot benchmarks are generated with the same acoustic forward operator and acquisition geometry as the training data, so the experiments demonstrate generalization to unseen velocity structures under the same synthetic acquisition protocol, not the synthetic-to-real domain shift implied by 'generalization gap.' The paper would be significant after reframing this claim and adding comparisons with a broader set of baselines.

major comments (3)
  1. [§4.3, Table 2; Abstract] The central claim of 'substantially narrows the long-standing generalization gap' is not supported by the evaluation protocol. In §4.3, all realistic benchmarks (Marmousi, SEG/EAGE, 2004 BP, Sigsbee, SEAM) are evaluated using seismic data generated with 'the same acoustic forward-modeling operator as in OPENFWI' (same source count, Ricker frequency, receiver count, and timesteps), and all velocity maps are downsampled to 70×70 and rescaled to [1500,4500] m/s. Thus the only distribution shift is in the velocity structure; the physics, noise level, and acquisition geometry are identical to training. The results show generalization to unseen velocity model families within the same synthetic manifold, not a narrowing of the domain gap between synthetic training and real seismic observations. The abstract and conclusion should be revised to state precisely what is demonstrated, or new experim
  2. [§4.2, Table 1] The paper claims 'state-of-the-art results on the synthetic OPENFWI benchmark' but the only comparison is against BigFWI-M and BigFWI-L. Many other data-driven FWI methods (e.g., InversionNet, VelocityGAN, SimFWI, and recent transformer-based methods) have published results on OpenFWI subsets. Without comparing against these or at least citing and tabulating previously reported numbers under the same evaluation protocol, the SOTA claim is not established. Please either add broader baselines or temper the claim to 'outperforms the BigFWI baselines.'
  3. [§3.3, Eq. (9); Appendix A] The RL objective is described as encouraging 'geological continuity and physical plausibility,' but Eq. (9) defines the reward as the negative L1 distance between the generated velocity map and the ground-truth label. No explicit continuity or physical-plausibility term is defined. This is a mismatch between the stated mechanism and the actual reward. Either specify additional reward terms and their weights, or revise the description to state that RL is a supervised label-based reward at the map level rather than a physics-aware reward.
minor comments (5)
  1. [§4.2, after Table 1] The text states that '×10 data for 10 epochs achieved an MAE of 0.0287, while ×2 data for 50 epochs achieved 0.0247,' but these numbers do not appear in Table 1. Clarify which configuration each number corresponds to (e.g., with or without ViT-VQGAN) and reconcile with the reported 0.0167 for the final ViT-VQGAN model.
  2. [Appendix A, Table 3] The caption says 'Qualitative comparison' but the table is quantitative. Change to 'Quantitative comparison.'
  3. [§4.3] Typo in Table 2: 'Marmosi' should be 'Marmousi.' Also, 'as we use an acoustic forward modeling' is awkward; rephrase.
  4. [§4.1 and §3.2] The data augmentation claims expansion to 'over five million velocity–seismic pairs,' but the final backbone is trained on '×2 data' (about 816k pairs). Clarify whether the full 5M set is used only for tokenizer training and explain the final training set size precisely.
  5. [§3.3 and Appendix A] Hyperparameters for the GRPO-style RL (e.g., reward coefficient, group size, learning rate, number of steps) are not reported. Such details are necessary for reproducibility of the RL post-training stage.

Circularity Check

0 steps flagged

No significant circularity; generalization claim is empirical and tested on external velocity benchmarks, with a scope limitation rather than a circular derivation.

full rationale

The paper's derivation chain is not circular. The central claims—state-of-the-art OPENFWI results and strong zero-shot generalization—are empirical outcomes evaluated against external benchmark velocity models (Marmousi, SEG/EAGE, BP, Sigsbee, SEAM) that are not used in training. The model components (non-causal transformer, ViT-VQGAN, diffusion augmentation, RL, latent GD) are individually ablated and show independent performance gains; none is defined in terms of the benchmark target metric. The RL objective (Eq. 9) uses ground-truth velocity labels during training, which is standard supervised/RL fine-tuning, not a fitted parameter renamed as a prediction. Self-citations to OpenFWI (Deng et al. 2022) and BigFWI (Jin et al. 2024) are dataset/baseline references and are not load-bearing for the main argument. A scope limitation does exist: Section 4.3 states that benchmark seismic data are 'generated with the same acoustic forward-modeling operator as in OPENFWI: five evenly spaced sources with a 15 Hz Ricker wavelet, recorded by 70 receivers sampling 1,000 timesteps over one second,' so the test distribution shares forward physics and acquisition geometry with training. This narrows the meaning of 'realistic generalization' to unseen velocity structures rather than field-data domain shift, and the conclusion acknowledges 'simplified 2D acoustic settings.' That is an overclaim/evidence-scope issue, not circularity: the evaluation still tests generalization to unseen velocity models and compares against external baselines. No circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

This is an empirical machine-learning recipe, not a theoretical derivation. The free parameters listed are the most load-bearing hand-chosen hyperparameters and evaluation choices that affect the reported numbers. The axioms are the domain assumptions about the physics and data distribution. No new physical entities (particles, forces, etc.) are introduced.

free parameters (5)
  • Final data scaling factor (2x) = 2x (selected over 10x after ablations: 10x gave MAE 0.0287, 2x gave 0.0247 under same compute)
    The paper chose to train the final transformer with 2x data instead of 10x, based on an efficiency-accuracy tradeoff. This is a design choice that affects the reported results.
  • ViT-VQGAN latent grid size = 25x25x196
    The expanded bottleneck preserves fine-scale geological details. The grid size is chosen by the authors to balance fidelity and compute, and it directly influences reconstruction accuracy.
  • Ensemble sample count = 5 samples per velocity map
    In zero-shot evaluation, the model predicts five samples per map and averages the refined outputs. This number is a hyperparameter that trades compute for reduced variance.
  • Benchmark downsampling resolution = 70x70
    All realistic benchmark velocity maps are downsampled to 70x70 and normalized to 1500-4500 m/s. This reduces the difficulty of inversion and is a non-learned preprocessing choice.
  • RL reward coefficient / GRPO hyperparameters = Not fully specified; described only as 'GRPO-style optimization'
    The RL stage uses a map-level reward -||v - v_hat(y)||, but the paper does not report the reward weight, KL penalty, or exact number of update steps. These are free parameters that affect the RL improvement magnitude.
axioms (3)
  • domain assumption Acoustic wave equation (Eq. 1) is an adequate forward model for seismic wave propagation.
    Used throughout for training-data simulation and for zero-shot benchmark data generation. Real seismic data includes elastic, anisotropic, and attenuation effects not captured by this equation.
  • domain assumption OpenFWI synthetic dataset plus diffusion-augmented samples provide sufficient structural diversity for generalization.
    The paper assumes that simple synthetic structures (curved layers, faults, styles) combined with generated hybrid maps are enough to learn features that transfer to complex salt bodies and overthrust structures.
  • ad hoc to paper Diffusion-generated velocity maps with FID 260.33 are useful training data despite low fidelity.
    The large FID indicates substantial distributional distance from OpenFWI, yet the paper assumes these synthetic maps still add beneficial diversity. This assumption is central to the data augmentation contribution.

pith-pipeline@v1.3.0-alltime-deepseek · 146 in / 10307 out tokens · 164100 ms · 2026-08-02T19:55:25.970145+00:00 · methodology

0 comments
read the original abstract

Full Waveform Inversion (FWI) is a highly nonlinear and ill-posed problem that aims to recover subsurface velocity maps from surface-recorded seismic waveforms data. Existing data-driven FWI typically uses small models, as available datasets have limited volume, geological diversity, and spatial extent, leading to substantial concerns about overfitting. Although they perform well on synthetic datasets, current methods fail to generalize to more realistic geological structures. In this work, we show that a model trained entirely on simulated and relatively simple data can generalize remarkably well to challenging and unseen geological benchmarks. We provide a working recipe that tames a billion-parameter model for FWI through coordinated scaling across three axes: model capacity, data diversity, and training strategy. Our model achieves state-of-the-art performance on OpenFWI and significantly narrows the generalization gap in data-driven FWI. Across six challenging geophysical benchmarks, including Marmousi, 2D SEG/EAGE Salt and Overthrust, 2004 BP, Sigsbee, and SEAM Phase I, it infers complex structures absent from the training set and delivers significant performance improvements (SSIM from 0.5844 to 0.7669). Overall, our results demonstrate that with an appropriate scaling strategy, large models trained on simple synthetic data can achieve substantial generalization to more complex and realistic geological structures.

Figures

Figures reproduced from arXiv: 2603.00377 by Peng Jin, Yinan Feng, Yinpeng Chen, Youzuo Lin, Yuzhe Guo.

Figure 1
Figure 1. Figure 1: Reconstruction results on challenging realistic benchmarks. BigFWI tends to collapse [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of progressive model enhancements and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of recon￾structed velocity maps. Remarkably, our model reaches state-of-the-art end-to-end performance for data-driven FWI, even before any post-processing refinement is applied. Overall, these results demonstrate that data scaling, non-causal modeling, improved to￾kenization, RL alignment, and physics-based re￾finement collectively define a powerful and ex￾tensible recipe for large-… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of recon￾structed velocity maps. Our method produces sharper and finer structures. For evaluation, our model predicts five samples per velocity map via stochastic sampling. Each sample is refined through latent space GD, and the final prediction is obtained by averaging the refined outputs. We evaluate predicted velocity maps using MAE, RMSE, and Structural Similar￾ity (SSIM). MAE an… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 6 linked inside Pith

  1. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

  2. [7]

    Subsalt multiple attenuation and imaging: Observations from the sigsbee2b synthetic dataset

    Josef Paffenholz, Bill McLain, Joerg Zaske, and P Joseph Keliher. Subsalt multiple attenuation and imaging: Observations from the sigsbee2b synthetic dataset. InSEG international exposition and annual meeting, pages SEG–2002. SEG,

  3. [10]

    Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627,

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627,

  4. [11]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

  5. [1990]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

  6. [2018]

    Default: Deep-learning based fault delineation using the ibdp passive seismic data at the decatur co2 storage site.arXiv preprint arXiv:2311.04361, 2023b

    Hanchen Wang, Yinpeng Chen, Tariq Alkhalifah, Ting Chen, Youzuo Lin, and David Alumbaugh. Default: Deep-learning based fault delineation using the ibdp passive seismic data at the decatur co2 storage site.arXiv preprint arXiv:2311.04361, 2023b. Yue Wu and Youzuo Lin. Inversionnet: An efficient and accurate data-driven full waveform inversion.IEEE Transact...

  7. [2019]

    Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,

  8. [2020]

    Diffusion posterior sampling for general noisy inverse problems

    Hyungjin Chung, Jeongsol Kim, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In11th International Conference on Learning Representations, ICLR 2023,

  9. [2022]

    Auto-linear phenomenon in subsurface imaging

    10 Yinan Feng, Yinpeng Chen, Peng Jin, Shihang Feng, and Youzuo Lin. Auto-linear phenomenon in subsurface imaging. InProceedings of the 41st International Conference on Machine Learning. JMLR.org, 2024a. Yinan Feng, Hanchen Wang, Yinpeng Chen, Luoyuan Zhang, Jeeun Kang, Yixuan Wu, Young Jin Kim, and Youzuo Lin. Wavediffusion: Joint latent diffusion for ph...

  10. [2024]

    The 2004 bp velocity benchmark

    FJ Billette and Sverre Brandsberg-Dahl. The 2004 bp velocity benchmark. In67th EAGE Conference & Exhibition, pages cp–1. European Association of Geoscientists & Engineers,

  11. [2025]

    Randy Harsuko, Shijun Cheng, and Tariq Alkhalifah. Propagating the prior from shallow to deep with a pre-trained velocity-model generative transformer network.Journal of Geophysical Research: Machine Learning and Computation, 2(1):e2024JH000408, 2025a. Randy Harsuko, Shijun Cheng, and Tariq Alkhalifah. Velocity model building with uncertainty quantificati...