{"id":"7ce3ac6b-ff9c-4a6d-9cae-bab99fad2e06","arxiv_id":"2412.04578","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A benchmark study recommends full accuracy loss, reconstruction loss, unitary loss, and tridiagonal operator form for Koopman autoencoders.","lead":"This paper compares loss functions and operator forms for Koopman autoencoders, neural networks that learn time evolution of differential equations, and introduces four new loss terms. Its main recommendation, based on grid-search experiments on eight dynamic systems, is to use full accuracy loss, reconstruction loss, unitary loss, and a tridiagonal operator.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run grid searches do not support the 'most robust' recommendations; repeated-seed variance is needed.","rationale":"The reader's weakest assumption correctly identifies the lack of repeated runs and error bars as the load-bearing weakness. I agree with that assessment and with the CONDITIONAL verdict: the recommendations are plausible and the experiments are broad, but the evidence does not yet support the 'most robust' label. My additional observations -- the contradictory mask finding in Section 5.4 and the mislabeled tables in Section 6 -- reinforce rather than replace the reader's concern. I did not find a deeper logical inconsistency in the loss-term definitions or the operator-form restrictions; the central issue is purely statistical robustness. Therefore no verdict change is needed beyond the existing CONDITIONAL, provided the authors add repeated-seed experiments, quantify uncertainty, and fix the table errors.","tokens_in":16912,"tokens_out":2249,"duration_ms":25194,"concrete_test":"Rerun the Section 6 comparison (Tables 13-17) for each dynamical system with 10 random seeds, keeping architecture, optimizer, and epoch count identical, and record per-configuration mean and standard deviation of the test error. Then check whether the reported best configuration (e.g., tridiagonal with unitary loss) remains best in mean error and is separated by more than one standard deviation from each alternative. If for any system a non-recommended configuration (e.g., Jordan with no operator loss for Lorenz, or dense with no operator loss for pendulum) falls within one standard deviation, or if the best configuration changes across seeds, the claim that tridiagonal + unitary is 'most robust' is not supported by the data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 7 -- that the full accuracy loss, reconstruction loss, unitary loss, and tridiagonal operator form are 'most robust' -- depends entirely on the aggregation of short, single-run experiments in Sections 5 and 6. The paper states in Section 5 that the objective is to 'find robust trends' from grid searches with 'only a small number of epochs,' but no experiment is repeated with different random seeds, and no standard deviations or error bars are reported. Robustness is instead inferred from recurring combinations in top-ten lists and from qualitative mean-effect plots. This inference is not statistically supported. A concrete warning sign appears in Section 5.4: the text says that 'not having a mask performed much better' on average, yet 'seven of the best ten combinations used a mask.' Mean effects and top-k selections can disagree due to strong interactions, but without variance or interaction analysis the 'most robust' language is ambiguous. Section 6 selects the best combination from single runs per cell in Tables 13-17, making the tridiagonal/unitary recommendation vulnerable to a single-run artifact. Quality-control errors -- e.g., Table 14 is introduced as pendulum results but contains Lorenz data, and Table 16 is introduced as fluid attractor but contains Burger's data -- further reduce confidence in the data pipeline. Consequently, the empirical foundation for the headline recommendation is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative empirical study of loss terms and operator parameterizations for Koopman autoencoders. It defines several accuracy, encoding, and operator loss terms (full, max, and discounted accuracy; reconstruction, consistency, and metric; isometry, norm, unitary, and determinant; plus auxiliary absolute-max and energy-conservation losses) and three operator forms (dense, tridiagonal, and a block-diagonal \"Jordan\" form), claims novelty for the discounted accuracy, unitary, determinant, and absolute-max losses, and runs grid searches on eight ODE/PDE benchmarks. Based on short grid searches and longer per-equation comparisons, it recommends the full accuracy loss, the reconstruction loss, and the unitary loss with a tridiagonal operator.","tokens_in":17319,"tokens_out":7273,"duration_ms":63108,"significance":"If the conclusions were supported, the paper would provide useful practitioner guidance and fill a gap in the Koopman-autoencoder literature, where loss terms and operator forms are often chosen ad hoc. The systematic enumeration of choices across several equations, the public code, and the clearly stated final recommendation are strengths. However, the empirical basis is currently too fragile: the headline recommendations rest on single-run, short-horizon experiments with no variance estimates, and several table/caption inconsistencies make it difficult to verify the reported comparisons. The central claims are therefore not yet established.","major_comments":[{"comment":"The headline recommendations in Section 7 rest on grid searches whose \"robust trends\" are inferred from single runs at 20 to 50 epochs with no repeated seeds, no standard deviations, and no statistical tests. Section 5 itself states that the experiments are \"grid searches that cover a large number of combinations for only a small number of epochs\" and that the objective is to \"find robust trends,\" but robustness cannot be established from one realization per configuration. Concretely, Section 5.1 reports that full and discounted accuracy losses \"performed about the same,\" yet no measure of variance is given, and Section 5.4 says that \"not having as mask performed much better\" while \"seven of the best ten combinations used a mask,\" showing that mean-effect and top-k readings can conflict. Without repeated seeds, error bars, or an interaction analysis, the \"most robust\" conclusions in Sections 5.7 and 7 are not supported.","section":"Section 5"},{"comment":"The table captions and the surrounding text do not match. Table 14 is introduced in the text as \"the results for the equation for the pendulum\" but is labeled \"Results for the Lorenz system\" and contains Lorenz data; Table 16 is introduced as \"the results for the equation for the fluid attractor equation\" but is labeled \"Results for Burger's equation.\" These mismatches are not cosmetic: the Section 6.2 conclusion that the tridiagonal form and unitary loss are \"most robust\" is reached by reading these tables across equations, so the reader cannot tell which equation produced which rank. The tables must be relabeled and every row verified against the experimental log.","section":"Section 6, Tables 14 and 16"},{"comment":"The conclusion that the unitary loss term is the most robust operator loss term is not supported by the five tables it is based on. In Tables 13-17, the unitary loss gives the best error in only two cases (Tables 13 and 17); in Table 14 the best entry uses no operator loss, in Table 15 it uses the determinant loss, and in Table 16 it uses no operator loss. The text acknowledges that \"the best combination varied from equation to equation\" but then concludes robustness without defining any quantitative aggregation rule, such as mean rank, median error, or win rate. A qualitative reading of five single-run tables does not establish that the unitary loss is the most robust option.","section":"Section 6.2"},{"comment":"Several reported grid sizes and dimensions are inconsistent with the stated option sets, which prevents reproduction of the search. Section 5.5 says \"the embedding dimensions 256 was considered,\" but Table 9 reports entries at dimension 128 and Figure 9's caption says the embedding dimension is 64. In addition, the stated numbers of combinations do not match the stated restriction that the determinant loss is only considered with a mask: for example, Section 5.1 reports 216 combinations, but 3 accuracy terms x 3 embedding terms x 4 operator terms x 2 mask choices x 3 dimensions gives 189 after excluding determinant-without-mask; Section 5.2 reports 288, but the analogous calculation gives 252. Section 5.3 contains two competing descriptions of the grid. These inconsistencies make it impossible to know what was actually run and should be corrected or explained.","section":"Sections 5.1-5.6"}],"minor_comments":[{"comment":"The sentence \"the loss functions and operator forms are introduction in Section 3\" is repeated, and \"introduction\" should be \"introduced.\"","section":"Sections 1 and 3"},{"comment":"The text contains \"to to learn\" and the Introduction says \"This chapter\" even though the manuscript is an arXiv paper; both should be corrected.","section":"Section 2"},{"comment":"In the definition of the reconstruction loss, the text says \"where E is the encoder, R is the encoder\"; the second identifier should be \"decoder.\"","section":"Section 3.2"},{"comment":"What is called the \"Jordan form\" is a real block-diagonal canonical form with 2x2 rotation-scaling blocks, not the usual Jordan form; if this nonstandard terminology is intended, it should be defined and justified explicitly.","section":"Section 3.3"},{"comment":"The experimental description is internally contradictory: the first paragraph lists embedding dimensions 8, 16, 32, 64 and discount factors 1.0, 0.975, 0.95, while the next paragraph restates the setup with only dimensions 32 and 64 and no discount factors; this looks like leftover text from an earlier draft and should be removed.","section":"Section 5.3"},{"comment":"The heat-equation top-ten table reports errors rounded to one significant digit (all rows are 1e-6), which obscures the ranking; more digits are needed.","section":"Table 7"},{"comment":"The sentence \"The most accurate combination was the tridiagonal form without an unitary loss\" should be reworded, since the table's best entry is \"tridiagonal none\" and the intended meaning appears to be \"without an operator loss term.\"","section":"Section 6.1, Table 16"},{"comment":"The abstract says \"systemic study\"; this should be \"systematic study.\"","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an unrevised dissertation chapter and needs substantial editorial cleanup. The technical revision should focus on repeated-seed experiments, variance reporting, and correction of the table/caption mismatches; without those changes, the central recommendations cannot be considered established. If the authors are unable to provide repeated-seed results, I would not be able to support acceptance even after cosmetic revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful survey of a messy design space, but the headline 'most robust' recommendations are not yet supported by the evidence. The paper deserves peer review with major revision, not desk rejection.\n\nWhat is actually new: as far as the cited literature goes, this is the first systematic comparison of Koopman autoencoder loss terms and operator forms across eight ODE/PDE systems. The discounted accuracy, unitary, determinant, and absolute-max losses are each small variations on existing terms, but the authors are honest about that. Code is public, which earns real credit. The recommended configuration—full accuracy + reconstruction + unitary, tridiagonal operator—is plausible and gives practitioners a concrete default.\n\nSoft spots: the central issue is that the experiments are single-run grid searches at 20–50 epochs with no standard deviations or repeated seeds. The paper itself says the goal is to find 'robust trends,' but robustness cannot be inferred from one run per configuration. Mean-effect plots and top-ten lists can conflict even when data are clean; here they do, e.g., the heat equation where the mean says 'no mask' is better but seven of the top ten use a mask. That is not damning by itself, but it shows the need for variance estimates or an interaction analysis. The Section 6 operator-form comparison is thinner: each cell is a single number, and the recommendations ride on which table is which. That brings me to the real red flag: Tables 13–17 have caption/content mismatches. Table 14 is introduced as pendulum results but contains Lorenz data; Table 16 is introduced as fluid attractor but contains Burger's. These are exactly the errors that make a reader question the data pipeline. Minor issues: duplicated sentences in the introduction and in Section 5.3, 'this chapter' language from a dissertation, and novelty claims resting on 'the authors are unaware'—fine as a claim, not a verified negative.\n\nBottom line: as a practical starting point the recommended configuration is reasonable, but the empirical foundation is not yet established. A revision with repeated seeds, distributions, corrected tables, and less categorical language would turn this into a reference worth citing. As it stands, it is a useful draft that deserves referee time, and a serious editor should send it out.","headline":"Broad empirical comparison of Koopman autoencoder design choices, but single-run experiments and table mismatches undercut the 'most robust' claims.","tokens_in":17644,"tokens_out":3829,"would_cite":false,"duration_ms":36675,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that Koopman autoencoder training has robust default choices: full accuracy loss, reconstruction or consistency loss, tridiagonal operator form, and unitary operator loss.","keywords":["Koopman autoencoder","operator learning","loss function","operator form","tridiagonal operator","unitary loss","differential equations","neural network"],"falsifier":"Re-run the recommended configuration (full accuracy + reconstruction + unitary loss + tridiagonal operator) against the main alternatives on the pendulum and Lorenz benchmarks with many random seeds and full training; if the recommended configuration is not consistently among the best, or if another configuration consistently beats it, the robustness claim is unsupported.","tokens_in":16727,"feed_emoji":"🧭","tokens_out":6798,"duration_ms":71460,"temperature":0.7,"pith_summary":"Koopman autoencoders are a popular way to learn the time evolution of differential equations, but the literature disagrees on how to build the loss function and what form the learned linear operator should take. This paper runs a systematic comparison of those choices across eight ODE and PDE benchmarks and tries to identify which options are robustly best. It introduces several new loss terms—discounted accuracy, unitary, determinant, and absolute max—and also compares dense, tridiagonal, and Jordan operator forms. The paper's recommendation is a concrete default configuration: full accuracy loss, reconstruction loss (with consistency loss as a safe alternative), the tridiagonal operator form, and the unitary loss. If these defaults survive full-length training, practitioners gain a strong baseline that removes much of the guesswork in Koopman autoencoder design.","feed_headline":"Best Koopman defaults found: tridiagonal operator, unitary loss","feed_subtitle":"Across eight ODE/PDE benchmarks, these choices beat alternatives and make safe starting points for training.","key_machinery":"The load-bearing objects are the loss terms themselves and the structural form of the Koopman operator. The accuracy terms compare predicted and target states; the encoding terms constrain the encoder–decoder pair; the operator terms push the learned matrix toward unitarity. The operator forms are the dense matrix, the tridiagonal matrix (motivated by the fact that a unitary operator is tridiagonal in some basis), and the Jordan form built from $2\\times 2$ blocks $\\left(\\begin{smallmatrix}a & b \\\\ -b & a\\end{smallmatrix}\\right)$. The recommended unitary loss $\\lVert K K^{T} - I \\rVert^{2}$ directly enforces that the operator be orthogonal, and the tridiagonal mask cuts parameters while preserving expressiveness. These choices together determine the inductive bias of the whole architecture.","core_discovery":"On the paper's own terms, the central claim is that training choices for Koopman autoencoders have robust winners. Across the mean effects, best-combination tables, and direct comparisons, the full accuracy loss—the plain mean squared error between predictions and targets—is the most dependable accuracy term; the reconstruction loss and the consistency loss are the most dependable encoding terms; the tridiagonal form of the operator is the most dependable operator form; and the unitary loss, which penalizes $\\lVert K K^{T} - I \\rVert^{2}$, is the most dependable operator loss. The paper also reports that the norm loss is consistently weak, that the metric loss is usually weak, and that auxiliary losses such as the absolute max loss or energy conservation do not consistently help. Novel loss terms introduced here—discounted accuracy loss, unitary loss, determinant loss, and absolute max loss—are documented so that later work can test them independently.","pith_inferences":["An implication not tested here: the recommended configuration is a natural default for long-horizon rollout, since a nearly orthogonal tridiagonal operator should limit error growth, though the paper's experiments are short.","Because the grid searches used single runs, the defaults are probably conservative: a practitioner with a specific equation and enough compute may find that the Jordan form or a different operator loss wins for that equation after full convergence.","The same comparison could be extended to Koopman-based control, where the operator's spectrum and boundedness matter for controller design; a unitary-plus-tridiagonal inductive bias may interact favorably with stability constraints.","The novel determinant loss could become practical at scale if combined with the tridiagonal form, since the determinant of a tridiagonal matrix is computable in linear time; the paper notes this but does not scale the test."],"forward_implications":["A default recipe now exists: full accuracy loss, reconstruction loss, unitary loss, and a tridiagonal operator, so new Koopman autoencoder projects can start there instead of grid-searching.","The metric loss and norm loss can be deprioritized in most settings; the paper finds them consistently less robust than their alternatives.","The consistency loss can substitute for reconstruction loss when a problem makes reconstruction expensive, since the two perform similarly overall.","The unitary and determinant losses, both introduced here, are the strongest operator regularizers in the comparisons."],"supporting_citations":[{"why":"Establishes the Koopman formulation of classical mechanics that the architecture discretizes.","marker":"[33]"},{"why":"Supplies the max loss, reconstruction loss, and the 2x2-block Jordan operator form used in the comparisons.","marker":"[39]"},{"why":"Introduces the metric loss, one of the encoding loss terms compared.","marker":"[13]"},{"why":"Introduces the consistency loss, the main alternative to reconstruction loss.","marker":"[43]"},{"why":"Supplies the norm loss, an operator loss baseline that the experiments find weak.","marker":"[45]"},{"why":"Provides the unitary-operator properties that motivate the isometry, unitary, and determinant loss terms.","marker":"[46]"},{"why":"Supplies the real-matrix block decomposition behind the Jordan operator form.","marker":"[44]"},{"why":"Provides the Runge-Kutta solver used to generate training data for all ODE benchmarks.","marker":"[50]"}],"fun_headline_variants":["Best Koopman defaults: tridiagonal operator plus unitary loss","Koopman autoencoder study: tridiagonal operator and unitary loss top","Tridiagonal operator, unitary loss win in Koopman tests","Study ranks Koopman losses: unitary loss, tridiagonal operator","Top Koopman choices: full accuracy loss, tridiagonal operator"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions rest on short grid searches with a single run per configuration; if those runs are noisy or do not extrapolate to full training, the 'most robust' rankings could change.","fun_headline_variants_meta":{"raw":{"variants":["Best Koopman defaults: tridiagonal operator plus unitary loss","Koopman autoencoder study: tridiagonal operator and unitary loss top","Tridiagonal operator, unitary loss win in Koopman tests","Study ranks Koopman losses: unitary loss, tridiagonal operator","Top Koopman choices: full accuracy loss, tridiagonal operator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1574,"prompt_tokens":754,"completion_tokens":820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":370,"completion_tokens_details":{"reasoning_tokens":724}},"tokens_in":370,"tokens_out":820,"duration_ms":7979,"temperature":1.0,"reasoning_tokens":724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:01.216425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the recommended configuration (full accuracy + reconstruction + unitary loss + tridiagonal operator) against the main alternatives on the pendulum and Lorenz benchmarks with many random seeds and full training; if the recommended configuration is not consistently among the best, or if another configuration consistently beats it, the robustness claim is unsupported.","supporting_citations":[{"cited_title":"Matrix analysis.CambridgeuniversitypressCambridge, 1985","cited_arxiv_id":null,"evidence_quote":"Supplies the real-matrix block decomposition behind the Jordan operator form."},{"cited_title":"Astudyondata-driveniden- tification and representation of nonlinear dynamical systems with a physics-integrated deep learning approach: Koopman operators and nonlinear normal modes","cited_arxiv_id":null,"evidence_quote":"Introduces the consistency loss, the main alternative to reconstruction loss."},{"cited_title":"Representer theorem for learning Koopman operators","cited_arxiv_id":null,"evidence_quote":"Supplies the norm loss, an operator loss baseline that the experiments find weak."}],"review_version":1}