{"id":"44af9d3b-3483-4ae9-998e-56489c330b20","arxiv_id":"2412.06686","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Across six operator-learning tasks, GELU was the best activation, dropout always hurt, stochastic weight averaging helped only for small learning rates, and a learning rate finder was reported but not shown.","lead":"This paper reports small benchmark experiments testing activation functions, dropout, stochastic weight averaging, and learning rate finders for three neural operator architectures. A generalist might read it to see which training defaults save time in operator learning, though the results need verification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own SWA tables contradict the Section 6 claim that SWA improves at LR <= original, and Section 7's no-LR-finder recommendation is unquantified; the empirical basis for the headline recommendations is internally unsupported.","rationale":"I agree with the reader's REJECT verdict, but not with the weakest_assumption they name. The generalization/representativeness concern is real but secondary: one cannot generalize from six benchmarks if the data do not even support the claims within those six benchmarks. The load-bearing issue is internal validity. Section 6's summary sentence is contradicted by Tables 7 and 9, and Section 7's recommendation is unsupported by any reported numbers. Since the paper's stated purpose is to provide robust defaults, a conclusion that is contradicted or unmeasured on two of four recommendations cannot pass. The reader's rationale already notes these issues, which is why I mark partial agreement: their explicit weakest_assumption field points to representativeness, while my load-bearing concern is the internal evidence. No further experiments are needed to establish the contradiction; the published numbers suffice. The verdict remains REJECT, so no verdict adjustment is needed.","tokens_in":9542,"tokens_out":6027,"duration_ms":54739,"concrete_test":"Recompute from Tables 7 and 9 the comparison between 'none' and each SWA learning rate in the recommended range (1e-4 and 1e-3). If any in-range SWA error exceeds the corresponding 'none' error, the Section 6 statement that SWA 'improved the accuracy ... when the learning rate was the same or one tenth of the original learning rate' is false. This check is purely arithmetic and settles whether the internal contradiction exists; if run, it will show two counterexamples: DeepONet-KdV at 1e-4 and Koopman-pendulum at 1e-3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 8's four recommendations are empirical claims, but the evidence in Sections 6 and 7 is internally inconsistent or absent. Section 6 states: 'In each experiment, stochastic weight averaging improved the accuracy of the model when the learning rate was the same or one tenth of the original learning rate.' The published tables contradict this. Table 7 (DeepONet, KdV): no-SWA error is 1.261e-2; SWA with LR=1e-4 gives 4.700e-2, a 3.7x degradation. Table 9 (Koopman, pendulum): no-SWA is 4.370e-4; SWA with LR=1e-3 gives 4.919e-4, a degradation. Thus within the recommended range 'at most the original learning rate', 2 of 12 within-range rows are worse than baseline, so the claim 'in each experiment' is false and the Section 8 recommendation is stronger than the data allow. This correctness risk is independent of whether the six benchmarks generalize. Separately, Section 7 recommends not using a learning rate finder but provides no table, no error numbers, and no protocol, only 'The authors did experiments ... was worse.' A recommendation with zero reported measurements cannot be evaluated or replicated. Both flaws are load-bearing because the paper's entire contribution is the set of recommended defaults.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a series of hyperparameter experiments for three neural-operator architectures (DeepONet, Fourier neural operator, and Koopman autoencoder) on five differential equations. It examines four choices: activation function, dropout, stochastic weight averaging, and learning-rate finder. The paper concludes with four recommended defaults: use gelu, do not use dropout, use stochastic weight averaging with a learning rate no larger than the base learning rate, and do not use a learning-rate finder. The reported evidence consists mainly of error tables for activations, dropout, and SWA, plus a qualitative account of learning-rate-finder experiments.","tokens_in":9968,"tokens_out":4628,"duration_ms":47696,"significance":"If the claims held, these would be practical and inexpensive defaults for practitioners, and the public code repository and the explicit non-comparative framing across architectures are strengths. The activation and dropout findings are consistently supported by all six reported benchmark pairs. The self-citation to Ref. [43] is not circular: that reference supplies only the Koopman training loss, while the activation, dropout, SWA, and learning-rate-finder conclusions are independent empirical findings. However, the SWA recommendation is contradicted by two entries in the paper's own tables, and the learning-rate-finder recommendation is presented without any quantitative evidence. As stated, the Section 8 recommendations therefore overstate what the reported data can support.","major_comments":[{"comment":"The sentence in Section 6, 'In each experiment, stochastic weight averaging improved the accuracy of the model when the learning rate was the same or one tenth of the original learning rate,' is contradicted by the paper's own tables. For the DeepONet KdV experiment, the no-SWA error is 1.261e-2, while SWA with LR=1e-4 gives 4.700e-2, a degradation of roughly 3.7x. For the Koopman pendulum experiment, the no-SWA error is 4.370e-4, while SWA with LR=1e-3 gives 4.919e-4. Both LR values lie inside the range 'at most the original learning rate' that Section 8 recommends. Counting all within-range rows (LR=1e-3 and LR=1e-4), 2 of 12 rows are worse than no-SWA, so the claim 'in each experiment' is false and the Section 8 recommendation is stronger than the data allow.","section":"6, Tables 7 and 9"},{"comment":"Section 7 recommends not using a learning-rate finder, but it reports no measurements: there is no table, no error value, no experiment count, no description of the swept learning-rate range, no definition of the optimization protocol, and no comparison with a tuned baseline. A recommendation with zero reported quantitative evidence cannot be evaluated or replicated. Since this is one of the four headline conclusions in Section 8, the empirical basis for it is missing.","section":"7"},{"comment":"All activation and dropout tables report a single error value per configuration, with no number of random seeds, no standard deviation, and no statement about whether the reported comparisons were repeated. This matters for the strength of the claimed defaults: for example, in Table 3 the pendulum elu error (1.360e-4) is lower than the gelu error (1.752e-4), and in Table 2 the KdV gelu error (3.731e-3) is only slightly below the elu error (3.909e-3). Without run-to-run variability, the universal statement that gelu is best in operator learning is not yet supported with statistical assurance, even though the six-table trend is suggestive.","section":"4 and 5"}],"minor_comments":[{"comment":"There are several typos: 'preformed' should be 'performed', 'is is recommend' should be 'it is recommended', and 'The dropout ;is' should be 'The dropout is'.","section":"Sections 4 and 6"},{"comment":"The differential equation notation 'd2t', 'd2x', and 'd3x' should be rendered as d²t, d²x, and d³x for readability.","section":"Section 3"},{"comment":"In the Koopman loss description, 'where E is the encoder, R is the encoder' should read 'where E is the encoder and R is the decoder'.","section":"Section 2.3"},{"comment":"The left caption says 'for the equation for burger's equation'; it should say 'for Burger's equation'. Also, 'Burger' should be capitalized consistently throughout the tables.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful scope and the activation and dropout results are consistent, but the central recommendation list is only partially supported by the reported evidence. The SWA claim can be corrected by either restricting the recommendation to the cases where the data support it or by explaining the anomalous rows; the learning-rate-finder claim needs actual data or should be removed. If the authors cannot supply the missing LR-finder evidence, I would move to reject rather than accept after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper reports empirical comparisons of four training choices (activation, dropout, stochastic weight averaging, learning rate finder) for DeepONets, FNOs, and Koopman autoencoders on a small set of PDE/ODE benchmarks. The activation and dropout sections are clean and genuinely useful: GELU wins in all six activation runs, and dropout hurts in all six dropout runs. Those two findings alone are worth having, and the code is public, which is a real plus.\n\nThe soft spot is the SWA section. Section 6 claims that \"in each experiment\" SWA improved accuracy when the SWA learning rate was the same or one tenth of the original. That is false. Table 7 (DeepONet, KdV) shows SWA at 1e-4 giving 4.700e-2 versus 1.261e-2 without; Table 9 (Koopman pendulum) shows 4.919e-4 with SWA at 1e-3 versus 4.370e-4 without. So the headline recommendation \"use SWA with a learning rate at most the original\" is stronger than the data allow. The stress-test note is correct here.\n\nSection 7 is worse: the learning-rate-finder recommendation is stated with zero reported measurements. No table, no error values, no protocol. That is not an empirical claim; it is an anecdote. The conclusion then packages all four recommendations as equal-strength best practices, which overstates the support.\n\nThere are also smaller issues: no error bars or multiple-seed variation, and the six equation-architecture pairs are treated as if they represent all operator learning. But the core problem is internal: two of the four recommendations are not backed by the paper's own evidence.\n\nThat said, the paper is not a waste of time. The activation and dropout results are consistent and credible, and the SWA data, once properly qualified, still suggest that small SWA learning rates can help in some settings. The idea of documenting robust defaults for these architectures is reasonable.\n\nI would send this to peer review with a clear instruction that the authors must fix the SWA claim, add numbers for the learning-rate finder, and include variance estimates. A good referee could get a solid, narrow paper out of this. But accepting it in its current form would be wrong.\n\nBest,\n\n[Your name]","headline":"Useful activation/dropout tables, but the SWA and learning-rate-finder recommendations are contradicted or unsupported by the paper's own data.","tokens_in":10381,"tokens_out":1767,"would_cite":false,"duration_ms":18983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes four robust defaults for operator learning: GELU activations, no dropout, stochastic weight averaging at the original learning rate, and no learning-rate finder.","keywords":["operator learning","best practices","hyperparameters","activation function","dropout","stochastic weight averaging","learning rate finder","neural operators"],"falsifier":"Train the same three architectures on a new operator-learning task outside the tested set, such as a two-dimensional PDE with noisy initial conditions, comparing GELU against tanh/ReLU, dropout against none, and SWA at one-tenth the base rate; any result where GELU is not best, dropout helps, or lower-rate SWA hurts would falsify the recommendations.","tokens_in":9314,"feed_emoji":"⚙️","tokens_out":6990,"duration_ms":65405,"temperature":0.7,"pith_summary":"This paper tries to turn hyperparameter choice in operator learning from an expensive search into a small set of defaults. Training DeepONets, Fourier neural operators, and Koopman autoencoders on five differential equations, it compares activation functions, dropout, stochastic weight averaging, and learning-rate finders one at a time. The consistent result is that GELU always beat tanh, ReLU, and ELU; dropout always made accuracy worse; stochastic weight averaging helped when its learning rate was the original or one tenth of it; and a learning-rate finder did not find good rates. If these trends hold beyond the six tested cases, practitioners can adopt these defaults and skip much of the hyperparameter search.","feed_headline":"GELU best, dropout worst in operator learning tests","feed_subtitle":"Across DeepONets, FNOs, and Koopman autoencoders, the same four defaults win on five equations.","key_machinery":"The machinery is a controlled hyperparameter sweep: each architecture is trained on two benchmark equations with a base learning rate of 0.001, one choice is varied at a time, and the mean squared error is compared. The test beds are the pendulum, Lorenz system, and fluid-attractor ODEs plus Burger's and KdV PDEs. For Koopman autoencoders, the loss includes accuracy, encoder–decoder consistency, and a unitarity penalty on the Koopman operator, which defines the training regime in which the comparisons happen.","core_discovery":"The central claim is a set of four recommendations for operator learning: use GELU as the activation function; do not use dropout; use stochastic weight averaging with a learning rate at most the original learning rate; and do not use a learning-rate finder. The evidence is empirical: six experiments, one per architecture–equation pair, with errors reported in tables. The paper also notes that tanh, despite being common in practice, was worst in three of the six activation experiments, while ReLU was worst in the other three. The claims are presented as robust trends rather than as a theoretical guarantee.","pith_inferences":["The paper does not test noisy data or small training sets, so its no-dropout recommendation may not extend to regimes where dropout's regularization is most useful.","All benchmark equations are low-dimensional, so whether GELU remains best on high-dimensional PDEs is an open question the paper leaves unanswered.","The learning-rate finder result is tied to the specific finder implementation used, and other finder schemes might behave differently.","If the SWA trend reflects flat-minima selection, a natural extension would be to test the same default in transfer learning or fine-tuning settings."],"forward_implications":["A full hyperparameter search over activation functions and dropout can be skipped, saving a large share of the compute.","GELU can replace tanh as the default smooth activation, even though tanh is the more common choice in practice.","Stochastic weight averaging at the original or one-tenth learning rate becomes a cheap accuracy improvement with no extra search.","Learning-rate finders should be avoided for these architectures; a fixed standard rate with SWA is the safer route."],"supporting_citations":[{"why":"Defines the DeepONet architecture used as one of the three test beds for the hyperparameter comparisons.","marker":"[4–9]"},{"why":"Defines the Fourier neural operator and its spectral convolution layers, the second test bed.","marker":"[9–16]"},{"why":"Defines Koopman autoencoders, the third test bed.","marker":"[19–23]"},{"why":"Supplies the Koopman autoencoder loss terms used in the experiments.","marker":"[43]"},{"why":"Introduces dropout, the training technique the paper tests and ultimately recommends against.","marker":"[48, 49]"},{"why":"Introduces stochastic weight averaging, the method the paper recommends using at or below the original learning rate.","marker":"[50–53]"},{"why":"Introduces learning-rate finders, the hyperparameter-search shortcut the paper finds ineffective.","marker":"[54–56]"}],"fun_headline_variants":["GELU best, dropout worst in operator learning","Operator learning: GELU wins, dropout loses","Four defaults for operator learning: GELU, no dropout, SWA, no LR finder","GELU beats ReLU and tanh in operator learning","Operator learning: use GELU and SWA, avoid dropout and LR finder"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recommendations assume the six tested benchmark combinations—three architectures matched with two equations each—are representative enough to support defaults for all operator learning tasks.","fun_headline_variants_meta":{"raw":{"variants":["GELU best, dropout worst in operator learning","Operator learning: GELU wins, dropout loses","Four defaults for operator learning: GELU, no dropout, SWA, no LR finder","GELU beats ReLU and tanh in operator learning","Operator learning: use GELU and SWA, avoid dropout and LR finder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000516,"raw_usage":{"total_tokens":2377,"prompt_tokens":692,"completion_tokens":1685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":308,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":308,"tokens_out":1685,"duration_ms":15373,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:20:01.276325+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three architectures on a new operator-learning task outside the tested set, such as a two-dimensional PDE with noisy initial conditions, comparing GELU against tanh/ReLU, dropout against none, and SWA at one-tenth the base rate; any result where GELU is not best, dropout helps, or lower-rate SWA hurts would falsify the recommendations.","supporting_citations":[],"review_version":1}