{"id":"0c591cf7-96ba-443d-96b3-d985eedefba8","arxiv_id":"2608.04384","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ADSL-PDE represents neural PDE solver designs as structured DSL programs with a deterministic compiler, and evolves them with LLM agents, reporting large early search improvements.","lead":"This paper introduces ADSL-PDE, a domain-specific language that lets AI agents design neural PDE solvers by editing structured search states instead of raw Python code. It reports that this representation improves search efficiency and achieves over 52% performance gains in early evolution iterations across PDE benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim is internally inconsistent: ADSL-PDE's 1D Burgers error is 2.2e-3 in Table 1 but 5.52e-5 in Tables 2/4/5, and 'best on all' is contradicted by 1D Advection.","rationale":"The paper's central claim is an empirical one: the DSL representation yields >52% improvement in early evolution. The most load-bearing assumption is therefore that the reported benchmark numbers are trustworthy and comparable. On inspection, they are not: the same 1D Burgers task has ADSL-PDE at 2.2e-3 in Table 1 and at 5.52e-5 in Tables 2, 4, and 5, with no explanation of differing protocols. In addition, the text's claim that Table 1 shows ADSL-PDE best on every listed entry is factually wrong: U-NO's 1D Advection error (6.9e-3) is lower than ADSL-PDE's (9.4e-3). Without code, seeds, or the evaluation harness, these cannot be resolved by the reader. I therefore treat the empirical core as unverified. The backend-attribution concern raised by the reviewer is important but secondary, since unstable numbers cannot support any causal attribution. The reviewer's verdict (REJECT) remains appropriate, and a conditional accept would require reconciled tables and released artifacts. I partially agree with the reader: the rationale anticipated these inconsistencies, but the stated weakest assumption was about the backend rather than about the internal coherence of the headline numbers.","tokens_in":11840,"tokens_out":5866,"duration_ms":53622,"concrete_test":"Independently reproduce the 1D Burgers experiment under the protocols implied by Table 1 and Table 2 using the same PDE definition, training budget, and error metric. If the ADSL-PDE error is 5.52e-5 under the Table 1 protocol, then Table 1 is wrong; if it is 2.2e-3 under the Table 2 protocol, then Tables 2/4/5 are wrong. In the same run, re-scan all 11 Table 1 rows and report every baseline cell whose error is below the ADSL-PDE entry, to test the 'best on all listed benchmark entries' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ADSL-PDE improves search efficiency by >52% is an empirical claim, and the reported benchmark numbers do not cohere. Table 1 lists ADSL-PDE on 1D Burgers at 2.2e-3, while Tables 2, 4, and 5 all list 5.52e-5 for the same task with no stated change of protocol; a 40x discrepancy in the headline task makes the empirical base unreliable. The text also claims ADSL-PDE 'achieves the best result on all listed benchmark entries' for Table 1, but in the 1D Advection row U-NO's 6.9e-3 is lower than ADSL-PDE's 9.4e-3, so the claim is false as stated. Since no code, data, seeds, or evaluation harness are released, the reader cannot determine which number is correct, and the '52% within the first ten iterations' result cannot be independently checked. The representation-attribution argument depends on these numbers; an internally inconsistent table set is a necessary blocker before any causal conclusion can be drawn.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADSL-PDE, an agent-oriented domain-specific language and typed intermediate representation for the auto-design of neural PDE solvers. Solver designs are expressed as structured schema blocks (architecture, physical constraints, sampling, training, evaluation), statically verified, and compiled by a deterministic backend into executable solvers. An LLM-based evolutionary framework edits these DSL programs, with a quick-training stage that feeds performance and diagnostic feedback back to the agent, plus a controlled cross-island mechanism that can register new solver implementations. The paper claims that this representation reshapes the search space, improving candidate validity, token efficiency, search stability, and final solver accuracy, with a headline result of more than 52% improvement within the first ten evolution iterations. The evaluation compares ADSL-PDE against manually designed solvers, search-based and LLM-agent baselines, and includes ablations of the language components.","tokens_in":12032,"tokens_out":5785,"duration_ms":54885,"significance":"If the central claim is correct, the paper addresses a genuine bottleneck in LLM-driven scientific auto-design: representing the solver search space at a level that removes invalid programs while preserving solver-level design freedom. The idea of combining a typed IR, static validation, deterministic compilation, and field-aligned feedback is well motivated and potentially useful beyond PDE solvers. The ablations introduce appropriate metrics such as valid candidate rate, search efficiency, and token-normalized efficiency, and the framing of representation as a search-space problem is a positive conceptual contribution. However, the significance cannot be assessed as submitted because the empirical evidence is internally inconsistent, key baseline results are missing, and no code, data, seeds, or evaluation harness are provided. The conclusions therefore rest on numbers that cannot currently be verified.","major_comments":[{"comment":"The text states that ADSL-PDE 'achieves the best result on all listed benchmark entries,' but the 1D Advection row of Table 1 contradicts this: U-NO's 6.9e-3 is lower than ADSL-PDE's 9.4e-3. This is a direct counterexample to the headline claim and should be corrected or the claim qualified.","section":"Main Results against Manually Designed Solvers (Table 1)"},{"comment":"The same task, 1D Burgers, is reported as 2.2e-3 in Table 1 and as 5.52e-5 in Tables 2, 4, and 5, with no stated change of protocol. This 40x discrepancy in a headline task means the reported numbers are internally inconsistent; without code or experimental logs, neither value can be trusted.","section":"Table 1 vs Tables 2, 4, and 5"},{"comment":"RandomAgent and PINNsAgent are named as baselines in the experimental setup and in the discussion of Table 2, but neither appears in Table 2 or anywhere else in the results. The comparison against RandomAgent is specifically invoked to support the claim that the improvement is not merely due to exploring more candidates; with those results missing, that argument cannot be evaluated.","section":"Experimental Setup / Table 2"},{"comment":"The attribution claim that the DSL representation causes the observed gains is not established. The method allows the LLM agent to register new solver implementations in the backend library through cross-island innovation, and the backend already contains human-designed solver-family implementations. Since the backend code is not released, the assertion that 'the compiler does not perform hidden search, tune hyperparameters, or introduce task-specific heuristics beyond the DSL specification' is unverifiable, and improvements could originate from newly registered or embedded implementations rather than from the language action space.","section":"ADSL Language Design / Language-Guided Evolution Framework"},{"comment":"The abstract's claim of 'more than 52% improvement within the first ten evolution iterations' is not directly supported by any reported number. Figure 4 shows final improvement rates of approximately 52% for DeepSeek-V4-Pro, but it does not report first-ten-iteration values, and the text describes 52% as a final rate. The headline claim should either be backed by a table of ten-iteration results or removed.","section":"Abstract / Sensitivity to LLM Backbone (Figure 4)"},{"comment":"No seeds, repeated runs, or error bars are reported anywhere in the paper. Given the stochasticity of LLM sampling and evolutionary search, the claims of 'optimization stability' and the exact improvement curves in Figures 2 and 4 cannot be assessed from single-point numbers.","section":"Experimental Setup / all reported results"}],"minor_comments":[{"comment":"The sentence 'Its main benefit come from reshaping the search space' contains a subject-verb agreement error and should read 'benefit comes'.","section":"Discussion"},{"comment":"The caption says the reported metric is MSE or relative L2 error depending on the benchmark, but the table does not indicate which metric applies to each row; the geometric mean over mixed metrics should be justified.","section":"Table 1 caption"},{"comment":"The caption mentions shaded bands and dashed curves, but the text does not explain what the shaded bands represent or how they were computed.","section":"Figure 2 caption"},{"comment":"The entries 'Sharma 2025a' and 'Sharma 2025b' appear to reference the same technical report and should be consolidated.","section":"References"},{"comment":"The paper does not include a data and code availability statement; given the backend attribution issue, releasing the compiler, evaluation harness, and seeds is essential for any verification of the claims.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The empirical base is not internally coherent: the false 'best on all entries' claim and the 40x discrepancy for the headline Burgers task are, in my view, sufficient to reject without requesting a revision. If the authors can resolve the inconsistencies, release the backend and evaluation code, report seeds and repetitions, and include the declared baselines, a future submission on the same idea could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the ADSL-PDE paper. The core idea is genuinely useful: represent neural PDE solver designs in a typed DSL, validate statically, compile deterministically, and let an LLM evolve the DSL rather than raw Python. That is a sensible response to a real problem, and the design choices (verifier, backend attribution, island evolution) are thoughtful. The ablation structure in Tables 3 and 4 is also well conceived: comparing direct Python, plain config, low-level DSL, and full ADSL isolates the contribution nicely.\n\nBut the empirical core, as submitted, does not hold up. Table 1 reports ADSL-PDE's 1D Burgers error as 2.2e-3, while Tables 2, 4, and 5 all report 5.52e-5 for the same task with no stated protocol change. That is a 40x discrepancy. The text also claims ADSL-PDE \"achieves the best result on all listed benchmark entries\" in Table 1, but U-NO's 0.0069 on 1D Advection beats ADSL's 0.0094. These two inconsistencies alone break the central empirical claim as stated. The \"52% improvement within the first ten iterations\" comes from Figure 4, but no seeds, error bars, code, data, DSL grammar, or backend code are released; RandomAgent is mentioned as a baseline but does not appear in Table 2. The attribution claim depends on the backend doing no hidden search or tuning, and that cannot be checked without artifacts.\n\nI want to be fair: none of this makes the idea wrong. The representation-level argument is plausible, and the ablations point in the right direction. But the paper as submitted is not publishable until the numbers are reconciled and the artifacts are released. The internal table contradiction is a necessary blocker; a referee cannot tell which number is correct.\n\nWho is this for? Researchers working on LLM-driven scientific discovery and automated solver design. It deserves a serious referee, conditional on revision, because the method is concrete and the problem is real. Recommendation: send it to peer review with a clear request to reconcile the numbers, correct the Table 1 claim, and release the artifacts.","headline":"The DSL design is a genuinely good idea worth engaging, but the headline numbers are internally inconsistent and the artifacts are missing, so the empirical core is not currently defensible.","tokens_in":12630,"tokens_out":1693,"would_cite":false,"duration_ms":15189,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that representing neural PDE solver designs as structured DSL schemas instead of Python programs improves auto-design, achieving more than 52% performance gains within the first ten evolution iterations.","keywords":["domain-specific language","neural PDE solver auto-design","search-space representation","evolutionary search","LLM agent","physics-informed neural networks","operator learning","structured schema"],"falsifier":"Compare ADSL-PDE against a control that uses the same backend and verifier but lets the agent edit a plain configuration file instead of the DSL; if the improvement disappears, the DSL representation is the cause. Also inspect the backend library for hand-written solver code inserted by cross-island innovation, since hidden implementations would break the no-hidden-search premise.","tokens_in":11556,"feed_emoji":"🧮","tokens_out":6707,"duration_ms":60004,"temperature":0.7,"pith_summary":"Neural PDE solver auto-design has a sparse valid program problem: most Python programs an LLM writes are invalid, so search effort is wasted on implementation failures. The paper's central claim is that representing solver designs in a domain-specific language, with typed blocks for architecture, physical constraints, objectives, sampling, and training, concentrates search on valid, consequential decisions. A deterministic compiler turns each valid design into an executable solver, and an evolutionary agent edits these designs under empirical feedback. On a suite of PDE benchmarks, the system finds solvers that beat both manually designed baselines and direct code-generation agents, with more than 52% performance improvement within the first ten evolution iterations.","feed_headline":"Structured DSL boosts neural PDE solver auto-design by 52%","feed_subtitle":"Agents edit design decisions instead of Python code, raising valid candidates from 34% to 91%.","key_machinery":"The central object is the ADSL language front-end and its typed intermediate representation. Each solver is a schema z=(T,F,A,C,S,R,E) — task, solver family, architecture, physical constraints, sampling, training recipe, evaluation protocol — expressed in typed blocks. The parser converts a DSL program into an IR, the verifier rejects schemas that are incomplete, type-incompatible, or unsupported by the backend before any training, and the deterministic backend compiler instantiates the executable solver. Evolution runs on method islands, one per solver family, with occasional cross-island innovation that registers new implementations in the backend library. The mechanism that carries the argument is the coupling of a small, validated, solver-level action space with a compiler that guarantees executability, so that every token the agent spends edits a design decision that can actually be evaluated.","core_discovery":"The paper claims that effective auto-design is not mainly a matter of stronger agent reasoning but of the search representation. It introduces ADSL-PDE, where a solver is a structured schema (task, solver family, architecture, physical constraints, sampling, training recipe, evaluation protocol) rather than a Python program. The schema is parsed into a typed intermediate representation, statically verified, and compiled deterministically to executable code; the agent only edits schema-level fields. The paper asserts this reshapes the search space by eliminating invalid regions and raising the density of meaningful candidates, and reports that full ADSL-PDE raises the valid candidate rate from 0.34 to 0.91 and lowers geometric mean error across heterogeneous PDE tasks below all compared baselines.","pith_inferences":["Editorial extension: the search-space-reshaping principle should transfer to other scientific auto-design domains where valid programs are sparse, such as surrogate model construction or experimental protocol design; the same schema-plus-compiler architecture could be tested there.","A direct test of the mechanism would measure how often an agent's edits change validation loss; the paper's VCR and SE metrics are consistent with the reshaping story, but the causal link between field-level edits and performance changes is not directly quantified.","The ten-iteration improvement ceiling leaves open whether the DSL's advantage persists at longer horizons, where direct code generation might escape invalid regions and catch up.","Cross-island innovation is the least controlled part of the pipeline: the registered backend implementations are new code inserted into the deterministic compiler's library, so future work could isolate how much of the gain comes from that growing library versus from the language itself."],"forward_implications":["Solver auto-design can be measured by token-normalized search efficiency, not just final error, and the DSL's gain comes from not wasting tokens on invalid candidates.","The representation layer is portable across evolution backends: ADSL-PDE improved 10 of 12 task-backend comparisons, so the benefit is not tied to one search algorithm.","The LLM backbone becomes a moderate factor when the DSL constrains edits; cheaper models may suffice when final performance differences of roughly ten percent are acceptable.","Static verification before training can reject many invalid candidates inexpensively, shifting compute budgets from failed training runs to full training of promising designs.","The observed stability improvement suggests the DSL reduces the rate of backsliding during evolution, not only the probability of finding good candidates."],"supporting_citations":[{"why":"Provides the physics-informed neural network baseline and the equation-loss formulation that the ADSL physics block generalizes.","marker":"Raissi, Perdikaris, and Karniadakis 2019"},{"why":"Supplies the Fourier neural operator baseline and operator-learning benchmark tasks the DSL must beat.","marker":"Li et al. 2021"},{"why":"Supplies the U-NO baseline used in the manual-solver comparison.","marker":"Rahman, Ross, and Azizzadenesheli 2022"},{"why":"Represents the direct Python-generating LLM agent baseline that ADSL-PDE compares against.","marker":"Wuwu et al. 2025"},{"why":"Represents the prompt-driven multi-agent baseline in the auto-design comparisons.","marker":"He et al. 2025"},{"why":"Provides the AlphaEvolve mutation-and-evaluation baseline used to test the DSL as a representation layer.","marker":"Novikov et al. 2025"},{"why":"Provides the OpenEvolve evolution backend that ADSL-PDE wraps in the generality experiments.","marker":"Sharma 2025b"},{"why":"Supplies the PDEBench benchmark suite that defines the evaluation tasks and metrics.","marker":"Takamoto et al. 2022"},{"why":"Supplies the domain-specific-language motivation for compact structured abstractions.","marker":"Fowler 2010"},{"why":"Supplies the LMQL approach to constrained LLM generation, the closest related structured-language baseline.","marker":"Beurer-Kellner, Fischer, and Vechev 2023"}],"fun_headline_variants":["Editing decisions, not code, lifts PDE solver auto-design by 52%","ADSL-PDE: DSL turns solver design into schema edits, boosts search 52%","Structured search state lifts valid PDE solvers from 34% to 91%","Representation over reasoning: ADSL-PDE improves PDE solver search 52%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the deterministic backend compiler performs no hidden search, tuning, or task-specific heuristics, so every gain is attributable to the DSL's search space rather than to prebuilt solver implementations.","fun_headline_variants_meta":{"raw":{"variants":["Editing decisions, not code, lifts PDE solver auto-design by 52%","ADSL-PDE: DSL turns solver design into schema edits, boosts search 52%","Structured search state lifts valid PDE solvers from 34% to 91%","Representation over reasoning: ADSL-PDE improves PDE solver search 52%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1681,"prompt_tokens":947,"completion_tokens":734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":645}},"tokens_in":563,"tokens_out":734,"duration_ms":8017,"temperature":1.0,"reasoning_tokens":645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:32:39.762327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare ADSL-PDE against a control that uses the same backend and verifier but lets the agent edit a plain configuration file instead of the DSL; if the improvement disappears, the DSL representation is the cause. Also inspect the backend library for hand-written solver code inserted by cross-island innovation, since hidden implementations would break the no-hidden-search premise.","supporting_citations":[{"cited_title":"2025 , eprint =","cited_arxiv_id":null,"evidence_quote":"Provides the AlphaEvolve mutation-and-evaluation baseline used to test the DSL as a representation layer."}],"review_version":1}