{"id":"db4fa0d1-e86a-41b6-958f-a8148b6e8460","arxiv_id":"2505.23106","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NIPS is a neural operator that uses linear attention and Fourier kernels to simultaneously predict PDE solutions and recover hidden material properties from limited data.","lead":"This paper introduces NIPS, a neural network architecture that learns to solve PDEs while also discovering hidden physical parameters from data. It combines linear attention with Fourier-based convolution to make the learning faster and more scalable than prior attention-based operator models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Darcy 'zero-shot' evaluation likely uses the same microstructures in train and test: the per-task permutation sampling followed by a random 90/10 sample split does not hold out tasks, so the main generalization claim is not yet demonstrated.","rationale":"The paper's efficiency architecture is internally consistent: the linear-attention factorization in Section 3.3 correctly reduces the per-layer cost from O(N^2 d) to O(N d^2 + N d log N), and Table 6 confirms the memory and runtime advantage over NAO. The AFNO and NAO comparisons, while imperfect, are secondary to the core contribution. The load-bearing weakness is the experimental support for 'zero-shot generalization to unseen physical systems.' Section A.1 describes a sample-wise split of 10,000 permuted contexts drawn from 100 microstructures, not a task-wise split, so the ID Darcy results in Tables 1 and 3 cannot establish generalization to new permeability fields; the model has very likely seen all 100 microstructures during training. The OOD rows in Table 3 do hold out tasks via a different covariance for b, but they stay within the Darcy family and are not the basis for the headline 'diverse benchmarks' claim. The reader's verdict is CONDITIONAL; this concern does not change that verdict but makes the required condition concrete: re-analyze with a task-wise split, and if the performance gap is large, temper the zero-shot language. Credit is due for the honest Limitations paragraph, which admits the method may fail on advection-dominated problems; that admission is consistent with narrowing the generalizability claim.","tokens_in":18008,"tokens_out":19120,"duration_ms":204366,"concrete_test":"Re-run the Darcy ID experiment with a task-wise split: train on 90 of the 100 microstructures (all 100 permutations per task) and test on the remaining 10 microstructures (all 100 permutations), with no task overlap. Report NIPS and NAO test errors, kernel errors, and microstructure errors for this split, and compare with the current sample-wise split. If NIPS's test error rises substantially (e.g., from roughly 2% to above 10%) or the NIPS-versus-NAO ordering changes, the zero-shot generalization claim is not supported by the current protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section A.1, the Darcy dataset is built from 100 microstructures (tasks), each with 100 loading/solution pairs; for each task, 100 random permutations of the pair indices are taken, yielding 10,000 samples, of which 9,000 are used for training and 1,000 for testing. The split is described as a split of these 10,000 samples, not a split of the 100 tasks. Under a random 90/10 sample split, each microstructure is expected to appear in both training and test (about 90 training and 10 test permutations per task). Consequently, the ID results in Tables 1 and 3 measure the model's ability to re-learn kernels for already-seen permeability fields under different contexts, not zero-shot generalization to unseen physical systems. The abstract and Section 4.1 claim 'unseen physical systems' and 'unseen governing equations'; the only results that truly hold out tasks are the OOD scenarios in Table 3, which use a different covariance for b and still do not cross PDE families. This concern does not invalidate the efficiency analysis or the OOD evidence, but it removes the main experimental support for the zero-shot generalization component of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Neural Interpretable PDEs (NIPS), an attention-based neural operator that extends the Nonlocal Attention Operator (NAO) by combining a linear-attention mechanism with learnable Fourier-domain kernels. The central architectural claims are that factoring the NAO kernel-weighted integral reduces the computational cost from quadratic-in-tokens to O(N d^2 + N d log N), and that learning the projection kernel directly in Fourier space reduces the parameter footprint. The paper presents the derivation in Section 3.3, Algorithm 1, and experiments on three benchmarks (Darcy flow, Mechanical MNIST, synthetic tissue learning), reporting that NIPS consistently outperforms NAO and its variants in accuracy and runtime. Additional results cover scalability to 14,641 tokens on a single A100, robustness to noise, an ablation of the permutation augmentation parameter nrand and projection dimension dk, and interpretability experiments that recover a hidden permeability field from the learned kernel. The paper further claims zero-shot generalization to unseen physical systems and the amortization of pairwise interaction costs via Fourier convolution.","tokens_in":18233,"tokens_out":17312,"duration_ms":156012,"significance":"If the zero-shot and accuracy claims are substantiated, NIPS is a useful architectural step for multi-system operator learning: the linear-attention reformulation of NAO is conceptually clean, the O(N d^2 + N d log N) complexity derivation in Section 3.3 is transparent and plausible, and Table 6 gives concrete evidence that the method operates well beyond the memory limits of quadratic NAO (NAO exceeds memory at 3,721 tokens, while NIPS reaches 14,641 tokens on one A100 with 14.4 GB peak memory). The interpretability validation against an externally computed stiffness matrix is a good-faith check of the kernel interpretation, and Table 2's parametric study of nrand and dk is a useful ablation. The explicit Limitations statement about the structural-similarity assumption is a strength. However, the headline generalization claim currently rests on an experimental design that, for the flagship Darcy experiment, does not hold out tasks; the reported numbers therefore support in-context kernel retrieval for seen tasks more strongly than zero-shot transfer to unseen systems.","major_comments":[{"comment":"The Darcy-flow evaluation, which is the paper's primary evidence for generalization, does not hold out physical systems. Appendix A.1 describes generating 100 microstructures (tasks), each with 100 loading/solution pairs, then taking 100 random permutations of the pair indices per task, yielding 10,000 samples of which 9,000 are used for training and 1,000 for testing. Because this is a random sample split over the 10,000 permuted samples, not a split over the 100 tasks, every microstructure appears in both training and test (approximately 90 training and 10 test permutations per task). The test tasks in Tables 1 and 3 are therefore not unseen physical systems as claimed in the abstract and in Section 4.1: the model has been trained on other permutations of the same loading/solution pairs for the same permeability field, so the reported ID errors measure in-context retrieval and the permutation-invariance property in Eq. (12), not zero-shot generalization. I ask the authors to re-run the Darcy experiments with a task-level holdout (e.g., train on 80 or 90 microstructures and test on the remaining 20 or 10, reporting the number of held-out tasks), and to state explicitly whether the OOD rows of Table 3 use a task-level split or the same random sample split. This is not a request for a new algorithm; it is a correction of the experimental protocol that supports the paper's central generalization claim.","section":"Appendix A.1, Tables 1 and 3"},{"comment":"The claim that the experiments demonstrate generalization to unseen governing equations is not supported by the reported benchmarks. Each experiment stays within a single PDE family (elliptic Darcy flow; Neo-Hookean hyperelasticity in MMNIST; the synthetic tissue constitutive model), and the test-time variation is in parameter fields or loadings, not in the governing law. The evidence that is consistent with a zero-shot claim is the synthetic tissue experiment (Appendix A.3 uses a 450/50 split over material sets) and possibly the MMNIST experiment, whose split is not described; the OOD scenarios in Table 3 change the covariance of the loading and the permeability field but do not cross PDE families. Given that the paper's own Limitations section concedes that the approach assumes structural similarity between the target and training kernels, the abstract and Section 4 phrasing should be revised to describe what is actually demonstrated (transfer across parameter-field distributions within a family), or supplemented with a cross-family experiment.","section":"Abstract, Sections 4 and 5"},{"comment":"Errors of 52.09%, 52.84%, and 98.70% for the AFNO baseline are far outside the range typical of Fourier token-mixing architectures on Darcy-type operator problems, and this casts doubt on whether the comparison is a fair one. Since the paper explicitly states that it follows the fairness protocol of McGreivy and Hakim (2024), the authors should describe how the d context function pairs are encoded for AFNO, whether AFNO is being asked to solve the same transductive in-context task as NIPS (construct a kernel from the context and predict the remaining pairs), and what hyperparameter search was performed. If AFNO is evaluated under a different task formulation (e.g., no context or per-sample regression), the comparison should be reframed or replaced with a correctly configured baseline; as presented, the claim that NIPS consistently surpasses this baseline rests on a configuration that may be mis-specified.","section":"Section 4.1, Tables 1 and 7, Appendix B"},{"comment":"Most accuracy results are reported for a single run: only Appendix Table 7 provides mean and standard deviation, and only for the first Darcy configuration. The claim that NIPS consistently surpasses NAO would be stronger with repeated-seed statistics for all tables, particularly for the Table 3 ID configuration where the NIPS/NAO test-error difference is small (4.09% versus 4.15%) and NAO actually reports a slightly better microstructure error (7.09% versus 7.92%). In addition, the configuration behind Table 3 is not reconciled with Table 1: Table 3 lists 327,744 parameters for NIPS, while Table 1 lists 108,692 parameters for the 4-layer, 441-token case and 366,932 for the 1,681-token case; the grid resolution, context size d, and layer count used for Table 3 should be stated so the discrepancy can be understood.","section":"Tables 1, 3, 4, 5 and Appendix Table 7"}],"minor_comments":[{"comment":"The abstract's code URL (https://github.com/fishmoon1234/Nonlocal-Attention-Operator) points to a repository named after the NAO baseline rather than NIPS; please link the actual NIPS implementation and indicate which files correspond to NIPS versus the baselines.","section":"Abstract"},{"comment":"Appendix A.1 states that p(x) is the source term and g(x) is the hydraulic height (the solution), which reverses the roles in Eq. (11), where g is the source and p is the solution; please correct.","section":"Appendix A.1"},{"comment":"In the ID no-noise row of Table 3, NIPS's microstructure error (7.92%) is larger than NAO's (7.09%), so the sentence that NIPS achieves the lowest errors across nearly all scenarios should name this exception explicitly.","section":"Table 3, Section 4.1"},{"comment":"The phrase channel-independent convolution in Fourier space is not evidently consistent with Eq. (10), in which the learnable R multiplies F(g W^Q) after a channel-mixing linear map; please clarify what is channel-independent in the implementation.","section":"Abstract and Section 3.3"},{"comment":"In Table 3, the OOD Scenario 1 with sigma = 0.1 reports a lower test error (3.84%) than the ID case with sigma = 0.01 (4.40%) and the ID case with sigma = 0.1 (9.98%); a brief explanation of this nonmonotonicity would prevent the reader from doubting the noise-robustness claim.","section":"Table 3"},{"comment":"Equation (12) uses sigma(1:d) without definition; please define sigma as a permutation of the index set {1, ..., d}.","section":"Eq. (12)"},{"comment":"Algorithm 1 and the text should state explicitly that the test-time kernel is constructed from the same d pairs whose prediction error is being computed (a transductive, in-context protocol); this affects the interpretation of all reported errors, and it is relevant to the task-split issue raised in the major comments.","section":"Algorithm 1, Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's baselines are almost entirely the authors' own prior work (NAO and its two variants), with AFNO as the only independent architecture; given the paper's explicit appeal to McGreivy and Hakim (2024) on reporting bias, the editor may want the authors to add at least one standard independent operator baseline (e.g., FNO or DeepONet) in the revision and to report multi-seed statistics for all main tables. The flagship zero-shot claim needs the task-level re-split described in the first major comment; the tissue experiment already appears to use a proper task holdout, so the fix is feasible within scope. The suspiciously high AFNO errors should be checked for a task-formulation mismatch before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the NIPS paper. The architectural core is legitimate and the complexity analysis is clean. Replacing NAO's quadratic attention with linear attention and moving the kernel projection into Fourier space is a sensible, incremental improvement, and the reported speedups on larger grids are credible. The ablation against NAO variants is useful: it shows the design choices matter and that the extra parameters in NAO-Wp hurt. The appendix gives three-seed error bars for the main Darcy case, which is more than many operator-learning papers do.\n\nThe soft spot is the zero-shot claim. Section A.1 describes a Darcy dataset built from 100 microstructures, with 100 loading/solution pairs per microstructure. For each microstructure the authors take 100 random permutations of the pair indices, giving 10,000 samples, and split 90/10. The test set is therefore permutations of the same microstructures seen in training. The model is re-learning kernels for already-seen permeability fields, not generalizing to unseen physical systems. The abstract says 'unseen physical systems' and 'unseen governing equations'; that is an overstatement. The OOD scenarios in Table 3 do shift the covariance of b or g, but they still stay within the Darcy family, and the same task-leakage applies there because the permutations are per-task. So the main experimental support for the headline generalization claim is missing. This does not kill the architecture; it means the paper should be reframed as an efficiency and accuracy improvement over NAO on fixed task sets, with zero-shot generalization left as future work with a proper task-level split.\n\nOther issues are minor relative to that. Most tables are single runs, though the appendix partially addresses that. The AFNO baseline is weak; a 50% error on Darcy suggests it was not tuned properly for this task, and the comparison would be more convincing with a couple of strong baselines like FNO or a transformer. The interpretability experiment is a reasonable external check via stiffness-matrix fitting, but the recovery of b is smoothed and the boundary errors are acknowledged.\n\nThe math is sound as far as I checked. The O(N d^2 + N d log N) derivation is plausible. Code is available, which helps.\n\nWho gets value from this: people building efficient attention-based neural operators, especially those wanting to scale NAO-like models. The paper deserves a serious referee, but it needs a revised evaluation before acceptance. Recommend major revision with a corrected task split and reframed claims.","headline":"A solid incremental architecture with clean complexity analysis, but the zero-shot generalization claim is undermined by a train/test split that leaks tasks.","tokens_in":18763,"tokens_out":3293,"would_cite":false,"duration_ms":30574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","35R30","65N21"],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural operator that couples linear attention with Fourier-domain convolution can simultaneously predict PDE solutions and recover hidden parameter fields, and it does so with far lower cost than its quadratic-attention predecessor.","keywords":["neural operator","linear attention","Fourier neural operator","inverse PDE","zero-shot generalization","interpretable kernel","Darcy flow","Mechanical MNIST"],"falsifier":"Train NIPS only on diffusion problems with symmetric kernels and then evaluate it zero-shot on an advection-dominated system whose stiffness matrix is nonsymmetric; the paper's own limitation predicts a large accuracy drop and a wrongly symmetric recovered kernel, which would directly bound the zero-shot claim.","tokens_in":17777,"feed_emoji":"🧮","tokens_out":6582,"duration_ms":68074,"temperature":0.7,"pith_summary":"This paper claims that a neural operator can simultaneously solve forward PDE problems (predict the solution field from the loading) and inverse PDE problems (recover the hidden parameter field or kernel) across many physical systems at once, with no retraining for a new system. The proposed architecture, NIPS, replaces the quadratic-complexity attention and explicit pairwise interaction matrix of Nonlocal Attention Operators with linear attention and a learnable Fourier-domain kernel, turning spatial interactions into FFT operations. On Darcy flow, Mechanical MNIST, and synthetic tissue benchmarks, NIPS reports lower test error than NAO and AFNO while using comparable or fewer parameters, and it scales to 14,641 tokens on one GPU where the quadratic NAO runs out of memory. If correct, this is a concrete step toward interpretable, resolution-invariant models that infer material properties from deformation or flow data.","feed_headline":"Fourier attention solves PDEs and reveals hidden physics","feed_subtitle":"Linear attention plus Fourier kernels cuts cost and recovers hidden material fields.","key_machinery":"The load-bearing object is the attention-based kernel map $K[u_{1:d}, f_{1:d}](x,y)$ built from $L$ iterative blocks that alternately apply channel-independent convolution in Fourier space, via a learnable spectral kernel $R_l$, and linear attention over the $d$ context pairs. The key algebraic step rewrites the nonlocal kernel integral as the product of a Fourier-filtered query term and a key-value inner product, which removes the explicit $N \\times N$ pairwise matrix; normalization and random permutation of the $d$ pairs enforce the permutation symmetry $K[u_{1:d}, f_{1:d}] = K[u_{\\sigma(1:d)}, f_{\\sigma(1:d)}]$. This kernel map carries both tasks: applied to a loading field it solves the forward problem, and read off as a matrix it is interpreted as an inverse stiffness operator whose row sums reveal the hidden permeability or fiber microstructure.","core_discovery":"The paper's central claim is that a data-dependent kernel operator built from iterative Fourier-convolution and linear-attention blocks can learn the mapping from context input-output pairs $(u,f)$ to the PDE kernel $K_b$, so one trained model serves as both a forward solver and an inverse solver. By writing the attention integral as $\\mathcal{F}^{-1}(R \\cdot \\mathcal{F}(pW^Q))(x)$ times an inner product over the key-value pairs, NIPS avoids forming the $N \\times N$ attention and projection matrix: the cost per block drops from $O(N^2 d)$ to $O(N d^2 + N d \\log N)$, and memory scales with the number of retained Fourier modes $m \\ll N$. Experiments show consistent accuracy gains over NAO variants (for example, 1.03% versus 1.45% test error at four layers on Darcy flow, and 1.11% versus 5.27% at four layers on Mechanical MNIST), generalization to out-of-distribution loadings and microstructures and to noisy data, zero-shot transfer to unseen systems, and recovery of two-phase permeability microstructures from the learned kernel, with interpretability assessed against the inverse stiffness matrix $K^{-1}$.","pith_inferences":["Editorial inference: because the kernel is learned as a continuous function of $x$ and $y$, the same architecture could be probed for closed-form or symbolic PDE discovery, such as reading off Green's functions or constitutive tensors; the paper only demonstrates this numerically for two-phase fields.","A testable consequence the paper does not run: train only on symmetric diffusion problems and test on an advection-dominated problem with a nonsymmetric stiffness matrix; the stated limitation predicts failure, and the error gap would quantify how much of the zero-shot claim rests on kernel-structure priors.","The Fourier-mode parameterization suggests a natural resolution-invariance check at much finer grids: if the number of retained modes $m$ stays fixed while $N$ grows, the test error should stay roughly flat if the learned kernel is truly continuous.","Connecting to neighboring problems, the same mechanism could serve as a differentiable surrogate for per-specimen material identification in digital twins, where an inverse solve currently requires re-optimizing the model for each new specimen."],"forward_implications":["Zero-shot transfer: a model trained on a family of systems can predict solutions and recover kernels for a new unseen parameter field without retraining, as long as the new kernel is structurally similar to the training family.","Scalability on a single GPU: NIPS runs at 14,641 tokens where the quadratic NAO exceeds 40 GB of memory, and at 1,681 tokens it reduces per-epoch time from 58.8 seconds to 8.4 seconds.","Interpretable recovery: the learned kernel approximates $K^{-1}$, and row-summing followed by thresholding reconstructs two-phase microstructures with roughly 8 to 16 percent error, degrading only gradually when observational noise is added.","Random permutation of context pairs is a necessary ingredient: on Darcy flow the test error falls from 42.04% with one permutation to 2.28% with 100 permutations, confirming that the kernel must be invariant to the ordering of the input-output pairs.","The comparison protocol used in the paper holds parameter counts roughly equal across models, so the reported accuracy gains are not attributable simply to larger models."],"supporting_citations":[{"why":"Defines the Nonlocal Attention Operator that NIPS extends; supplies the attention-based kernel map, the cross-system prior, and the zero-shot evaluation setup.","marker":"Yu et al., 2024"},{"why":"Introduces the Fourier Neural Operator and the shift-equivariant kernel convolution whose $O(N \\log N)$ spectral evaluation NIPS adopts.","marker":"Li et al., 2020c"},{"why":"Frames inverse PDE solving as a non-identifiable problem and provides neural inverse operators as a supervised baseline that NIPS improves upon in an unsupervised setting.","marker":"Molinaro et al., 2023"},{"why":"Supplies the Mechanical MNIST benchmark of heterogeneous Neo-Hookean materials used as one of the three test problems.","marker":"Lejeune, 2020"},{"why":"Provides the AFNO convolution-based attention baseline that NIPS compares against and outperforms in zero-shot transfer.","marker":"Guibas et al., 2021"},{"why":"Origin of the attention mechanism whose quadratic complexity NIPS removes through its Fourier-domain linear reformulation.","marker":"Vaswani et al., 2017"},{"why":"Shows how linear attention can act as a learnable kernel in neural operators, the reformulation NIPS builds on.","marker":"Cao, 2021"},{"why":"Supplies the fairness principles for baseline comparison, equal accuracy or equal runtime, that the paper follows when tuning models to comparable parameter counts.","marker":"McGreivy & Hakim, 2024"}],"fun_headline_variants":["Fourier attention cuts PDE cost and uncovers hidden physics","NIPS: Fourier attention for scalable, interpretable PDE discovery","Fourier attention makes PDE learning fast and interpretable","Efficient Fourier-attention neural operator for PDE discovery","Fourier attention slashes PDE cost, reveals hidden fields"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-shot claim stands or falls on whether a new system's governing law resembles the family of equations used in training; the authors state that a diffusion-trained model that expects symmetric interactions will likely fail on advection-dominated systems with non-symmetric interactions.","fun_headline_variants_meta":{"raw":{"variants":["Fourier attention cuts PDE cost and uncovers hidden physics","NIPS: Fourier attention for scalable, interpretable PDE discovery","Fourier attention makes PDE learning fast and interpretable","Efficient Fourier-attention neural operator for PDE discovery","Fourier attention slashes PDE cost, reveals hidden fields"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3087,"prompt_tokens":1004,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2000}},"tokens_in":620,"tokens_out":2083,"duration_ms":16886,"temperature":1.0,"reasoning_tokens":2000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:52:43.135970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train NIPS only on diffusion problems with symmetric kernels and then evaluate it zero-shot on an advection-dominated system whose stiffness matrix is nonsymmetric; the paper's own limitation predicts a large accuracy drop and a wrongly symmetric recovered kernel, which would directly bound the zero-shot claim.","supporting_citations":[],"review_version":1}