REVIEW 4 major objections 5 minor 12 references
Explicit pairwise feature interactions lift neural PDE solvers
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 00:26 UTC pith:7BADYX6M
load-bearing objection A useful, honestly reported extension of NFM interactions to PDE solving; the operator results are solid enough to take seriously, but the FM-Operator causal claim needs one more control and the FM-PINN half is thinner than the abstract suggests. the 4 major comments →
Feature Interaction Modeling for Physics-Informed Neural Networks and Neural Operators
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the solution maps of many PDEs contain substantial cross-field dependence — between spatial coordinates, between space and time, and between the input function and the query location — and that a network that constructs these pairwise products before nonlinear processing exploits that structure better than a plain concatenation MLP. The paper's bi-interaction pooling computes, for a set of M input fields, the vector sum of all pairwise Hadamard products of their embeddings; this is motivated by a blockwise second-order Taylor expansion, where the cross-field Hessian terms δFiᵀ Hij δFj correspond exactly to the pooled products. On smooth high-dimensional PINN problem
What carries the argument
The central object is the bi-interaction pooling layer, inherited from neural factorization machines. Each input field Fi (a coordinate, sensor vector, query vector, or parameter) is mapped through its own affine embedding Wi Fi + βi to a shared dimension r, and the layer returns z_BI = Σ_{i<j} e_i ⊙ e_j, which can be computed as 1/2[(Σ e_i)^⊙2 − Σ e_i^⊙2]. This layer makes all pairwise cross-field products explicit before an MLP decoder, at rank at most r per pair. The paper's Taylor-expansion analysis shows these products mirror the mixed second-derivative terms of the solution map, which explains why the module helps when mixed derivatives are large — as in high-dimensional coordinates or
Load-bearing premise
The load-bearing premise is that the chosen benchmark PDEs and manufactured solutions fairly represent the families where pairwise feature interactions matter; the paper's own Appendix D shows that switching a manufactured solution from separable sines to quadratic phases reverses the ranking between FM-PINN and SPINN, so conclusions drawn from one solution family may not generalize.
What would settle it
Run the same FM-PINN versus SPINN protocol on a family of manufactured solutions with smoothly varying CP rank and interaction strength (e.g., interpolation between Setting S and Setting Q). If, on non-axis-aligned solutions, rank-128 SPINN still beats FM-PINN in two dimensions, or if FM-Operator's advantage over DeepONet disappears when sensor statistics are appended to DeepONet under matched seeds, the central claims would be weakened. More directly: a single benchmark family with large mixed derivatives but where a plain MLP already matches FM-Operator would falsify the claim that explicit
If this is right
- If correct, a single architectural module provides a parameter-efficient way to improve PINN accuracy on high-dimensional smooth PDEs, with the gain growing with the number of coordinate fields (from one pair in 2D to ten in 5D).
- The FM-Operator result implies that explicit sensor–query interactions are a strong inductive bias for shock-dominated problems, achieving lower error than a much larger DeepONet; the improvement is not reproduced by just adding sensor statistics to DeepONet's input.
- The benefit is conditional: on Helmholtz equations, low-dimensional smooth problems, and Kuramoto–Sivashinsky, the interaction module offers no consistent advantage, so the paper implies that interaction terms should be added selectively rather than universally.
- The comparison with Shift-DeepONet suggests that explicit cross-field products can be more effective for moving-front tracking than input-dependent coordinate transformations of trunk basis functions, at roughly a third of the parameters.
- The paper's own Appendix D cautions that architecture rankings can reverse when the manufactured solution changes structure, so the gains are solution-structure-dependent rather than universal — a direct corollary, not an external critique.
Where Pith is reading between the lines
- A testable extension: an adaptive gating mechanism that adds interaction terms only for field pairs with large estimated mixed derivatives could turn the conditional benefit from a fixed module into a learnable one, possibly recovering the negative cases (2D Darcy, reaction–diffusion, Helmholtz).
- The strong performance on conservation laws suggests that higher-order (third-degree) interaction features, or interactions between summary statistics and local query neighborhoods, might further improve shock tracking; the paper's Taylor motivation caps at second order.
- If the benchmark-dependence worry is taken seriously, a productive next experiment is to sample manufactured solutions with varying CP rank and measure when FM-PINN overtakes rank-128 SPINN; the paper's Appendix D provides the protocol for exactly such a sweep.
- The 'no consistent advantage on smooth operator learning' caveat implies that the module's value lies in problems where sensor–query coupling is strong; this could be used as a selection criterion when deploying the method to new PDE families.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes embedding factorization-machine-style bi-interaction layers into PINNs and neural operators. FM-PINN treats each coordinate as a separate field and computes pairwise Hadamard products of field embeddings before an MLP decoder; FM-Operator applies the same interaction pooling to sensor values, query coordinates, sensor statistics, and optional physical parameters, combined with a linear first-order path; FM-DeepONet injects an interaction block into the branch–trunk features of DeepONet. The design is motivated by a second-order Taylor expansion of the target function/operator, with the authors explicitly stating that this is an inductive-bias argument, not a universal-approximation proof. Experiments report that FM-PINN improves relative L2 error on 16 of 18 selected multidimensional smooth PDE problems, and that FM-Operator achieves substantial error reductions on several nonlinear conservation-law and shock benchmarks while using roughly 43K parameters. The paper also reports five-seed operator results, parameter-matched ablations, a DeepONet+statistics control, and an appendix exploring sensitivity of the FM-PINN/SPINN comparison to the manufactured-solution family.
Significance. If the central mechanism claim holds, the paper offers a simple, parameter-efficient architectural module that could improve neural PDE solvers on problems with strong cross-field dependencies. The manuscript has real strengths: it is candid that the Taylor expansion is only a motivation, it reports multi-seed statistics for the main operator experiments, it includes several thoughtful ablations (short DeepONet, compact FM-DeepONet, DeepONet with appended statistics), and its Appendix D explicitly tests how the FM-PINN/SPINN ranking changes with solution structure. These features raise the evidentiary bar relative to many architecture-proposal papers. However, the causal attribution for FM-Operator's gains is not fully supported: no experiment removes the bi-interaction pooling while keeping the rest of the FM-Operator architecture. In addition, the FM-PINN evidence is based on single runs and on a narrow family of manufactured solutions, so the broad wording of the abstract overstates what the current experiments establish.
major comments (4)
- [§5, “Parameter-Controlled Ablations”; Eq. (6) and Appendix A] The central causal claim—that explicit pairwise cross-field interactions drive FM-Operator's accuracy gains—is not tested. Tables 15–17 compare FM-Operator against reduced-size DeepONet, compact FM-DeepONet, and DeepONet with appended statistics, but no ablation removes the bi-interaction pooling z_BI from FM-Operator while retaining the per-field affine embeddings, MLP decoder, and linear first-order path. A control that replaces z_BI with, e.g., elementwise sum or concatenation of the same embeddings is needed. Without it, the gains could come from the embedding-plus-MLP architecture itself rather than from the interaction products. This is load-bearing for the paper's title and for the attribution in §6 to “how the models construct cross-field interactions.”
- [§5, Table 2; §6 Limitation] The FM-PINN comparisons are single-run. Many reported differences are small (1D Burgers 0.96%, 2D Poisson 3.64%, 4D Poisson 3.99%) and could easily be within run-to-run stochastic variation, especially for PINN training with AdamW and randomized LHS sampling. The paper itself states in §6 that the PINN comparisons are based on single runs. Given that the abstract and introduction claim that FM-PINN “improves the approximation accuracy for smooth high-order PDEs,” the central PINN claim needs either multi-seed statistics with confidence intervals or a substantially softened conclusion.
- [Appendix D, Table 19 and Eq. (14)] The broad FM-PINN claim is fragile with respect to the benchmark design. Appendix D shows that the relative ranking of FM-PINN and SPINN reverses when the manufactured solution changes from a sum of separable sines (Setting S) to a quadratic-phase solution (Setting Q), even though the PDE operator, optimizer, network scale, and evaluation grid are unchanged. The main FM-PINN evaluation uses only the separable-sine family, which is a narrow and structurally favorable test set. The manuscript's own appendix concludes that the evidence points to a “solution-structure-dependent regime rather than a universal ranking.” The abstract and Section 1 should be scoped accordingly, or additional manufactured-solution families should be included in the main evaluation.
- [Appendix C, Table 8 vs. main Table 2] The main text's “16 of 18” FM-PINN summary omits the full set of results reported in Appendix C. The appendix shows substantial negative results in 1D: convection–diffusion 1D is 48.62% worse and reaction–diffusion 1D is 295.37% worse than PINN, while Table 2 restricts attention mainly to equations with two or more spatial coordinates. This selective presentation inflates the apparent generality of the FM-PINN improvement. The main text should either report the complete set of benchmarks or explicitly state the exclusion criterion and acknowledge the 1D failures.
minor comments (5)
- [§5 / Table 3 caption] The caption says the best mean is “shown in blue,” but the table appears in plain text. If color is used in the final PDF, ensure it is also indicated by boldface or an asterisk for accessibility.
- [Appendix C, Figure 13 caption] Typo: “FM-PINN CLinear Advection 4D” should be “FM-PINN Linear Advection 4D.”
- [§5, Kuramoto–Sivashinsky setup] The KS benchmark is described only briefly. The domain length, initial-condition normalization, and sensor/output grid should be stated precisely, since the KS results are an important exception to the FM-Operator advantage.
- [Appendix C, Table 12] The “2D diagonal lift” benchmarks appear in the results table but are not defined in the experimental setup. Please provide the construction of these two-dimensional operator inputs.
- [General] No code or data release is mentioned. Given the number of bespoke numerical reference solvers and benchmark configurations, a public implementation would substantially aid reproducibility.
Circularity Check
No significant circularity: the interaction mechanism is an empirical, falsifiable inductive bias with external benchmarks and explicit caveats.
full rationale
The paper does not derive its accuracy claims from its own construction. The Taylor expansion in Section 4 is explicitly presented only as motivation, not as a proof: 'A sufficiently large concatenation MLP can approximate multiplication and therefore represent the same cross-field products. The proposed block does not prove superior expressiveness; it makes these products explicit before deep nonlinear processing.' The central evidence is experimental and falsifiable: FM-PINN is compared with PINN, and FM-Operator/FM-DeepONet are compared with DeepONet and Shift-DeepONet on externally generated reference solutions (finite-volume, WENO5, pseudospectral, and analytic benchmarks). No parameter is fitted to the reported test errors and then renamed a prediction; checkpoints are selected by training loss, and the ablations (short DeepONet, compact FM-DeepONet, DeepONet+stats, split vs non-split, xDeepFM) serve as genuine controls rather than circular reductions. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggling: the NFM bi-interaction mechanism is attributed to He and Chua (2017), and the paper explicitly notes where the Taylor interpretation does not apply, e.g., discontinuous transport. Appendix D strengthens the non-circularity verdict by showing that the SPINN comparison reverses under a different manufactured solution, demonstrating that the claims are not established by definition and can be contradicted by evidence. The absence of a no-interaction FM-Operator control is a legitimate attribution concern but concerns experimental validity, not circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- field embedding dimension r =
64 (FM-PINN), 128 (FM-Operator/FM-DeepONet)
- MLP decoder width/depth =
128/4 (PINN); 128/3 (operators)
- training budget and optimizer =
30,000 AdamW steps, lr=1e-3, wd=1e-6
- sensor count and output grid =
64 sensors, 128x80 grid
- loss weights (PDE/boundary/initial) =
1/1/1
axioms (4)
- domain assumption The target PDE solution maps are locally C^3 with bounded third derivative, so the second-order Taylor expansion applies.
- domain assumption The chosen benchmarks and manufactured solutions are representative of the claimed PDE families (smooth high-order PDEs; shock-dominated conservation laws).
- domain assumption The numerical reference solvers (WENO5, finite-volume Rusanov, pseudospectral ETDRK4) are accurate enough for the reported relative L2 errors to be meaningful.
- ad hoc to paper The NFM bi-interaction layer is a beneficial inductive bias for cross-field dependencies.
Cite this review
Pith. "Pith review of Feature Interaction Modeling for Physics-Informed Neural Networks and Neural Operators." pith.science (2026). https://pith.science/paper/7BADYX6M
@misc{pith2026260728762,
author = {Pith},
title = {Pith review of: Feature Interaction Modeling for Physics-Informed Neural Networks and Neural Operators},
year = {2026},
howpublished = {\url{https://pith.science/paper/7BADYX6M}},
note = {Machine review of arXiv:2607.28762}
}
read the original abstract
This work embeds feature interaction modules derived from factorization machines (FMs) into physics-informed neural networks (PINNs) and neural operator learning, to enhance model expressiveness for solution manifolds of parameterized partial differential equations (PDEs). Motivated by the second-order Taylor expansion of multivariate functions to characterize variable couplings, we first propose FM-PINN. It explicitly captures spatio-temporal variable interactions and improves the approximation accuracy for smooth high-order PDEs. We further group spatial coordinates, time, physical parameters, and initial and boundary conditions into independent feature sets and model their cross-group interactions. Based on this strategy, we develop FM-Operator and FM-DeepONet, which are particularly effective for nonlinear conservation laws and problems with sharp gradients or discontinuities, while offering no consistent advantage on smooth operator learning benchmarks. Numerical tests demonstrate that the proposed mechanism delivers substantial accuracy gains on challenging shock-dominated equations, indicating a promising direction for physics-consistent modeling of parameterized PDEs with strong cross-field dependencies.
Figures
Reference graph
Works this paper leans on
-
[1]
Journal of Computational Physics, 533: 113996
Chen,Y.;Lin,Y.;Sun,X.;Yuan,C.;andGao,Z.2025.Tensor decomposition-based neural operator with dynamic mode decomposition for parameterized time-dependent problems. Journal of Computational Physics, 533: 113996. Cho, J.; Nam, S.; Yang, H.; Yun, S.-B.; Hong, Y.; and Park, E
2025
-
[5]
Multi-Grid Tensorized Fourier Neural Op- erator for High-Resolution PDEs. arXiv:2310.00120. Lanthaler,S.;Molinaro,R.;Hadorn,P.;andMishra,S.2023. Nonlinear Reconstruction for Operator Learning of PDEs withDiscontinuities. InInternationalConferenceonLearn- ing Representations. Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhat- tacharya, K.; Stuart, A...
Pith/arXiv arXiv 2023
-
[6]
Physics- Informed Neural Operator for Learning Partial Differential Equations. arXiv:2111.03794. Lian,J.;Zhou,X.;Zhang,F.;Chen,Z.;Xie,X.;andSun,G
-
[8]
U-NO: U-shaped Neural Operators. arXiv:2204.11127. Raissi, M.; Perdikaris, P.; and Karniadakis, G. E
-
[10]
Fac- torized Fourier Neural Operators. arXiv:2111.13802. Wang, S.; Wang, H.; and Perdikaris, P
-
[11]
PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations. arXiv:2512.23056. A Methodology Inthissection,weintroducethethreenetworkarchitecturesproposedinthispaper:FM-PINN,FM-DeepONet,andFM-Operator, which are illustrated in Figures 5, 6, and 7, respectively. FM-PINN Figure 5: FM-PINN FM-PINN is a physics-informed neu...
-
[2010]
In2010 IEEE International Conference on Data Mining, 995–1000
Factorization Machines. In2010 IEEE International Conference on Data Mining, 995–1000. Tancik,M.;Srinivasan,P.P.;Mildenhall,B.;Fridovich-Keil, S.; Raghavan, N.; Singhal, U.; Ramamoorthi, R.; Barron, J.T.;andNg,R.2020. FourierFeaturesLetNetworksLearn High Frequency Functions in Low Dimensional Domains. InAdvances in Neural Information Processing Systems, v...
2020
-
[2018]
xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. InProceedings of the24thACMSIGKDDInternationalConferenceonKnowl- edge Discovery and Data Mining, 1754–1763. Liu,Y.;Zhang,Z.;andSchaeffer,H.2023. PROSE:Predict- ing Operators and Symbolic Expressions using Multimodal Transformers. arXiv:2309.16816. Lu, L.; Jin, P.; Pang,...
Pith/arXiv arXiv 2023
-
[2020]
Communications in Computational Physics, 28(5): 2002–
Extended Physics-Informed Neural Networks (XPINNs): A General- izedSpace-TimeDomainDecompositionBasedDeepLearn- ing Framework for Nonlinear Partial Differential Equations. Communications in Computational Physics, 28(5): 2002–
2002
-
[2022]
extends the standard DeepONet by allowing its trunk basis functions to adapt to the input function. A conventional DeepONet represents the learned operator as bG(s)(y) = pX k=1 bk(s)t k(y),(12) Table14:FM-Operator(NFM)andShift-DeepONetontheviscousBurgersequationwithν= 0.001.Resultsarereportedover three seeds as mean±sample standard deviation. Model Parame...
2038
-
[2023]
Separable PINN: Mitigating the Curse of Dimensionality in Physics-Informed Neural Networks. arXiv:2211.08761. Fazliani, S.; Frangella, Z.; Chu, Y.-C.; and Udell, M
-
[2025]
OpenReview: tS4vGObuCA
Enhancing Physics-Informed Neural Networks through Fea- ture Engineering.Transactions on Machine Learning Re- search. OpenReview: tS4vGObuCA. Hadorn,P.2022.Shift-DeepONet:ExtendingDeepOperator Networks for Discontinuous Output Functions. He, X.; and Chua, T.-S
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.