Pith.

Artificial Intelligence · arXiv:2611.14949 v1 · submitted 11 Nov 2026 · 13 references

RSI-PDEbench: A Benchmark for Self-Improving Agents in PDE Solving

Binghang Lu, Lang Cao, Changhong Mou, Yige Sun, Cindy Xiangrui Kong, Zheyuan Deng, Tian Yuan, Bing Hu

show 3 more authors

Runyu Zhang, Na Lu, Guang Lin

The paper says

“Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts.”

more of the abstract

Can large language model (LLM) agents automate this workflow accurately and efficiently? To investigate this question, we introduce \bench, a benchmark of 1,001 cases spanning 42 equation templates, including both classical and fractional PDEs. It provides three evaluation settings: Full evaluates performance on noise-free solution data; Noisy tests robustness to Gaussian noise with an RMS equal to 50% of the clean-signal RMS; and Hard-OOD tests challenging out-of-distribution generalization to unseen equation templates and shifted coefficient ranges. We further introduce RSI-PDE, a recursively self-improving agent harness that shortlists candidate equations by residual loss, uses relative L2 feedback to refine coefficients and numerical solvers, and applies a validation gate to policy and memory updates. We compare RSI-PDE with single-run baselines and native agent harnesses (Claude Code or Codex) across four frontier models. Across the three settings, RSI-PDE improves the pass rate averaged across the four models by 106.5–181.9% relative to single-run baselines and by 3.0–19.6% relative to native harnesses, while reducing total recorded model calls by 46.9% across the 12 model–setting pairs. These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

A well-constructed benchmark for PDE agent tasks, but the marginal gains over native harnesses are thinner than the headlines suggest and need replication. This line is wrong

the referee pass did not run

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 13 references say.

What would settle it

Run the Full setting for Opus 5.5 with at least 5 independent seeds (varying the random seed for model-generated responses, not benchmark case seeds) for both the native harness and RSI-PDE. Compute the mean pass rate and standard deviation for each. If the mean RSI-PDE gain is less than one standard deviation of the native harness, then the improvement is not statistically significant, which would weaken the central claim.

from the skeptic's reading

How the review went

3 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 25 s · 24,578 tokens read, 2,906 written
  3. Editor wrote the verdict above 12 s · 25,470 tokens read, 1,237 written

Not run on this paper: the referee, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hey,

This paper brings a real contribution in the benchmark itself: 1,001 PDE/FPDE cases across 42 templates, with cleanly separated public/hidden splits, a thoughtful residual-guided harness, and transparent reporting of limitations. The iterative improvement over single-shot baselines (106–182%) is clear and practically useful. The benchmark fills a genuine gap for agent-driven equation identification plus solver construction.

The soft spot is the central comparison against native harnesses (Claude Code / Codex). Most comparisons are single-run, and the only setting with replication (Opus 5.5 Full native, three runs) shows an 11-case range that swallows the 3-case RSI-PDE gain. The paper acknowledges this (b293) but still presents the improvement descriptively. The Noisy settings show larger margins (44 cases on Opus) but also single-run. The 46.9% call reduction is real but uneven: RSI-PDE uses more calls in 4 of 12 model-setting pairs, and the dollar-cost claims mix retrospective stop-rule replays.

The authors are honest about these limits – they call the record ‘developmental’ and call for repeated runs. So the paper doesn't oversell, but the headline numbers should be taken with a grain of salt.

Who gets value: Anyone working on LLM agents for scientific computing, especially those building benchmarks or iterative refinement systems. The harness design and residual guidance are worth studying even if the exact margins aren't pinned down.

Verdict: Send to peer review. The benchmark is solid, the open-sourced code and data are a serious resource, and the limitations are stated. Conditional acceptance that asks for multi-run replication and uncertainty quantification on the central comparison would be appropriate.

The strongest objection

The paper claims that RSI-PDE improves pass rates over native harnesses by 3.0–19.6% across settings. However, the vast majority of comparisons are single-run. The one setting with replication (Opus 5.5 Full native, three runs) shows an 11-case range (187–198 passes), which is larger than the 3-case gain RSI-PDE achieves over the native mean (193→196). The paper acknowledges this (b293) but still presents the improvement descriptively without uncertainty quantification. For GPT-5.5 Full, the 9-case gain (95→104) is comparable to the observed native variance. For Noisy Opus, the 44-case gain is larger but still single-run. Without multiple independent runs or error bars, it is impossible to determine whether the observed margins are real or due to random variation. The central claim is therefore less robust than the headline numbers suggest. The paper's own limitation statement (b293) confirms that 'the three-case Opus difference is smaller than the native harness’s observed 11-case range.' This is the most load-bearing concern: the evidence for improvement over native harnesses is thin where it is claimed to be present.

The paper is transparent about the single-run limitation and presents results descriptively. The reader's weakest assumption identified the same concern: the three-run Opus range is large relative to margins. My stress-test confirms this is the weakest point, but the paper does not overclaim statistical significance; it states results are descriptive. Therefore the conditional verdict remains appropriate. No change to verdict needed, but the concern is real and load-bearing.

Pith's reading of the paper (a machine's summary, not the authors')

RSI-PDEbench provides 1,001 test cases across 42 equation types (classical PDEs and fractional PDEs) for an agent that must identify the equation family, estimate its parameters, and produce an accurate numerical solver, all from 256 noisy or clean samples and known boundary/initial conditions. The agent uses a large language model (LLM) as its core, writing code to experiment. The harness structure adds three innovations. First, a residual screen quickly ranks candidate equation families by how well they fit a reconstructed field, keeping the top five. Second, for each plausible family the agent builds a solver, runs it, and gets the relative L2 error on public training cases as feedback. Third, a validation gate accepts solver updates only if they improve public-case accuracy without breaking existing passes. This policy and memory also evolve. Across four frontier LLMs (Opus 5.5, GPT-5.5, GPT-6.1, Sonnet 5.5), the RSI-PDE harness averaged 81% pass rate on full noise-free data versus 79% for native harnesses (Claude Code/Codex) and 29% for single-run outputs. On 50%-noise cases it reached 64% versus 53% and 27%. On out-of-distribution cases (unseen templates and shifted coefficients) it reached 77% versus 73% and 37%. Total model calls dropped by 47% across all comparisons. The paper notes that most comparisons are single runs, the native-harness margin is small, and the benchmark is developmental rather than a pristine holdout.

Core claim. The RSI-PDE harness improves mean pass rate across four frontier LLMs by 106-182% over single-run baselines and by 3.0-19.6% over native harnesses (Claude Code/Codex) on three evaluation settings (noise-free, 50% Gaussian noise, and out-of-distribution), while reducing total recorded model calls by 46.9% across 12 model-setting pairs.

The premise it rests on. Most model-setting comparisons rely on a single run; the three-run native Opus 5.5 Full setting showed an 11-case range (187-198 passes), which is larger than several RSI-PDE gains (e.g., 3-9 cases). The paper acknowledges this but still presents the mean improvements as descriptive. If the observed margins are within natural variance, the central claim would weaken.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts. Can large language model (LLM) agents automate this workflow accurately and efficiently? To investigate this question, we introduce \bench, a benchmark of 1,001 cases spanning 42 equation templates, including both classical and fractional PDEs. It provides three evaluation settings: Full evaluates performance on noise-free solution data; Noisy tests robustness to Gaussian noise with an RMS equal to 50% of the clean-signal RMS; and Hard-OOD tests challenging out-of-distribution generalization to unseen equation templates and shifted coefficient ranges. We further introduce RSI-PDE, a recursively self-improving agent harness that shortlists candidate equations by residual loss, uses relative L2 feedback to refine coefficients and numerical solvers, and applies a validation gate to policy and memory updates. We compare RSI-PDE with single-run baselines and native agent harnesses (Claude Code or Codex) across four frontier models. Across the three settings, RSI-PDE improves the pass rate averaged across the four models by 106.5–181.9% relative to single-run baselines and by 3.0–19.6% relative to native harnesses, while reducing total recorded model calls by 46.9% across the 12 model–setting pairs. These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.94

“These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing.”

How the main claim is stated: hedged possibility 0.80

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.09 · human participants 0.03 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts. background 1.00 0.06 0.02 asserted 0.98
Can large language model (LLM) agents automate this workflow accurately and efficiently? gap 0.85 0.04 0.02 unclear 0.90
To investigate this question, we introduce \bench{}, a benchmark of 1,001 cases spanning 42 equation templates, including both classical and fractional PDEs. contribution 0.95 0.95 0.02 asserted 1.00
It provides three evaluation settings: Full evaluates performance on noise-free solution data; Noisy tests robustness to Gaussian noise with an RMS equal to 50\% of the clean-signal RMS; and Hard-OOD tests challenging out-of-distribution generalization to unseen equation templates and shifted coefficient ranges. method 0.63 0.85 0.02 asserted 0.99
We further introduce RSI-PDE, a recursively self-improving agent harness that shortlists candidate equations by residual loss, uses relative $L^2$ feedback to refine coefficients and numerical solvers, and applies a validation gate to policy and memory updates. method 0.87 0.93 0.04 asserted 0.99
We compare RSI-PDE with single-run baselines and native agent harnesses (Claude Code or Codex) across four frontier models. method 0.87 0.92 0.02 asserted 0.84
Across the three settings, RSI-PDE improves the pass rate averaged across the four models by 106.5--181.9\% relative to single-run baselines and by 3.0--19.6\% relative to native harnesses, while reducing total recorded model calls by 46.9\% across the 12 model--setting pairs. result 1.00 0.97 0.44 asserted 1.00
These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing. interpretation 1.00 0.43 0.91 hedged possibility 0.80

What kind of paper the classifier reports this is: dataset or benchmark 0.94 · evidence from observation or experiment 0.55

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 6,553 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (13)
  1. [1] Bai, Y., Lu, B., Liu, Y., Akbari, E., Kolouri, S., Wang, L., He, P., Wang, S., Zhang, R., and Lin, G. SE(3)- MeanFlow: Few-step protein backbone generation on Lie groups.arXiv preprint arXiv:2607.27431, on PithPith's reviewarXiv
  2. [2] Templates marked† were hardened after calibration (Section C); templates marked‡ take their input fields from the DiffusionPDE test set. Table 5 | The 42 templates. Dim.: spatial dimension (0 for the fractional ODE). Steady: no time dependence. ID Template Class Dim. Steady Cl... on Pith
  3. [8] Lu, B., Deng, Z., Zhang, R., Hu, B., Zhao, Y., Tian, Y., Mou, C., Lin, G., and Li, X. Muon-OGD: Muon-based spectral orthogonal gradient projection for LLM continual learning.arXiv preprint arXiv:2605.08949, 2026a. Lu, B., Mou, C., and Lin, G. iPINNER: An iterative physics-info... on PithPith's reviewarXiv
  4. [10] Novikov, A., V˜u, N., Eisenberger, M., Dupont, E., Huang, P.-S., et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery .arXiv preprint arXiv:2506.13131, on PithPith's reviewarXiv
  5. [11] Tian, Y., Hu, B., Wang, H., Lu, B., and Wu, F. Harness evolution hits a ceiling: When weight training should begin.arXiv preprint arXiv:2610.11655, 2026a. Tian, Y., Hu, B., Wu, F., Li, X., Lu, B., and Gong, N. Z. When think-with-image meets safety: What determines multimodal j... on PithPith's reviewarXiv
  6. [12] 16 RSI-PDEbenchA PREPRINT A. Related Work PDE learning and equation discovery.PDEBench, PDEArena, PINNacle and The Well provide important test beds for learned numerical models (Takamoto et al., 2022; Gupta & Brandstetter, 2022; Hao et al., 2024; Ohana et al., 2024). Physics-i... on Pith
  7. [13] and MedRouter for routing- based reasoning across medical language models (Cao et al., 2026). They illustrate structured generation and task-specific routing in domains beyond PDE solver construction. Agent reliability also has dimensions beyond numerical accuracy: Chain of Ri... on Pith
  8. [2015] Gupta, J. K. and Brandstetter, J. Towards multi-spatiotemporal-scale generalized PDE modeling.arXiv preprint arXiv:2209.15616, on PithPith's reviewarXiv
  9. [2016] Cao, L., Lu, B., Shen, Y., and Guo, Y. MedRouter: Demystifying knowledge differences across medical LLMs for routing-based reasoning.arXiv preprint arXiv:2609.33119, on PithPith's reviewarXiv
  10. [2023] Mou, C., Lu, B., and Lin, G. Neural-POD: A plug-and-play neural operator framework for infinite-dimensional functional nonlinear proper orthogonal decomposition.arXiv preprint arXiv:2602.15632, on PitharXiv
  11. [2024] Li, S., Marwah, T., Shen, J., Sun, W., Risteski, A., Yang, Y., and Talwalkar, A. CodePDE: An inference framework for LLM-driven PDE solver generation.Transactions on Machine Learning Research, 2026a. Li, X., Hou, J., Deng, Z., Zhang, Z., Li, T., Lu, B., Hu, B., Zhao, Y., and H... on PithPith's reviewarXiv
  12. [2025] Deng, Z., Lu, B., Feng, H., Huang, S., Wang, D., Xu, Y., Zhang, Z., Sun, Y., Mou, C., Zhang, R., et al. CONTRAMEM: Learning self-evolving procedural memory from contrasting multi-model trajectories.arXiv preprint arXiv:2608.22533, on PithPith's reviewarXiv
  13. [2026] URLhttps://arxiv.org/abs/2605.09636. 14 RSI-PDEbenchA PREPRINT Hao, Z., Yao, J., Su, C., Su, H., Wang, Z., Lu, F., Xia, Z., Zhang, Y., Liu, S., Lu, L., and Zhu, J. PINNacle: A comprehensive benchmark of physics-informed neural networks for solving PDEs. InAdvances in Neural In... on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment