Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts

Amarpal Sahota; David A.W. Barton; Grant Stevens; Roussel Desmond Nzoyem; Tom Deakin

REVIEW 4 major objections 5 minor 73 references

Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MixER, a mixture-of-experts layer with a K-means and least-squares router, lets one model reconstruct several loosely related dynamical-system families at once, but trails single models when data is too similar.

desk verdict MixER is a genuinely new routing scheme for hierarchical DSR, but its headline experiment omits the one baseline that would prove the mechanism. read the letter →

arxiv 2502.05335 v2 pith:AFZ2RXNM submitted 2025-02-07 cs.LG

Roussel Desmond Nzoyem , Grant Stevens , Amarpal Sahota , David A.W. Barton , Tom Deakin This is my paper · ORCID

classification cs.LG

keywords dynamicalsystemreconstructionhierarchicalmeta-learningmixtureofexpertssparseroutingK-meansgatingleastsquarescontextualneuralODE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a sparse mixture-of-experts layer can promote a single-family meta-learner into a hierarchical meta-learner that reconstructs several loosely related families of dynamical systems at once from few trajectories. The proposed layer, MixER, sends each environment to one expert using a learned context vector, and it updates the router by clustering contexts with $K$-means and fitting the gating matrix by least squares instead of by gradient descent. On benchmarks built from two and ten parametric ODE families, this removes the failure in which gradient-based gating sends every environment to the same expert, and it supports efficient training and one-shot adaptation to new environments. The authors also mark the boundary of the idea: when environments are abundant and highly related, or the context representations are ambiguous, MixER underperforms a single meta-learner and its routing does not separate classes. The wider point matters because scientific data often arrives as small trajectory sets from many different systems, so a model that learns across families without manual partitioning is a concrete step towards a foundation model for dynamical systems.

What carries the argument

The load-bearing object is the MixER layer itself, with its non-gradient gating update. On each router refresh, the algorithm clusters the current context vectors with Lloyd's $K$-means, computes every expert's reconstruction loss on every environment, assigns each cluster a distinct expert by greedily taking the best available expert per cluster, converts the assignment into one-hot labels, and solves a least-squares problem for the linear gate $W$ mapping $\xi^e$ to those labels. Because the router never receives gradient signal through the expert outputs, the experts function independently outside the layer, the softmax-weighted output aggregation is removed, and no load-balancing loss is needed. Context splitting (each expert sees its own slice of the context) and context shifting (each expert applies a learned scalar offset) are auxiliary mechanisms that support clustering in low-relatedness regimes.

What would settle it

Train MixER on ODEBench-10B with the number of experts deliberately misspecified (for example $M=7$ or $M=13$ instead of the ten true families) and measure one-shot adaptation error against the $M=10$ case; if performance degrades sharply or routing no longer tracks families, the method's guarantees depend on knowing the family count in advance, while if it holds up the clustering-based gating is robust to misspecification.

Watch

Extended reading notes

Core claim

MixER is a sparse top-1 Mixture of Expert Reconstructors layer whose gating network consumes environment-specific context vectors $\xi^e$ rather than raw state vectors, and whose gating weights $W$ are refreshed by a four-stage procedure: Lloyd's $K$-means clusters the contexts; per-expert, per-environment reconstruction losses are computed; each cluster is paired with its best-scoring expert without reuse; and $W$ is fit by least squares to the one-hot labels of those pairings, with small noise added for stability. The authors report that once family-expert pairings are established, reconstruction quality improves sharply, whereas a naive MoE with gradient-updated gating routes all contexts to a single expert. Across ODEBench datasets with two and ten families of parametric ODEs, MixER trains efficiently and scales, but its advantage is not uniform: on the ten-family, sixteen-environment-per-family setting, the ten-expert MixER trails a gradient-gated MoE and single-expert models on training error, while a twenty-expert version yields the best adaptation error for two of the three backbones. On classical closely related DSR benchmarks and on epileptic EEG data, MixER partitions data sensibly but loses to its own backbone, and the context clusters it learns do not cleanly separate epileptic from healthy subjects. The paper's central claim is therefore conditional: a non-gradient, clustering-based router lets MoE handle hierarchical DSR on sparse, loosely related data, but the same mechanism hurts when data is dense and homogeneous.

Load-bearing premise

The method assumes that the context vectors learned by the shared meta-learner are comparable across environments and that $K$-means clusters of those contexts match the true families, with one exclusive expert able to serve each family; the authors show this premise can break on ambiguous EEG data, and the number of experts is always set to the known family count rather than learned.

Editorial extensions

If this is right

A single MixER model can be trained on loosely related ODE families together and then adapted to a new environment by optimizing only its context vector, so the user no longer needs to know or supply the family structure before training.
Replacing gradient-based gating with $K$-means-plus-least-squares updates prevents the one-expert collapse and supports sparse top-1 reconstruction of up to ten parametric ODE families with efficient training.
When environments are closely related and data is abundant, partitioning them among experts is counterproductive: MixER underperforms its own single-expert backbone, so the hierarchical advantage is specific to heterogeneous, low-data settings.
The routing assignments produced by MixER form an unsupervised partition of environments that is informative on structured synthetic data (grouping the pairs of similar control-chart classes) but fails on noisy EEG data, degrading downstream classification.
The cluster-expert associations shift over training, a behavior the paper points to as a route toward continual learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The success of a router trained without gradients suggests a general recipe for sparse conditioning in other meta-learning problems: whenever gradient signals to a router are weak or delayed, periodic clustering of latent codes followed by a least-squares refit may be cheaper and more stable than end-to-end gating.
Because the paper always sets the number of experts equal to the true family count, a natural extension is to let the clustering determine $M$ or add a rejection option; the mixed results at $M=10$ versus $M=20$ on ODEBench-10B hint that the optimal $M$ for training error and for one-shot adaptation error may differ.
The connection the authors observe between context-cluster quality and genuine hierarchy implies MixER could double as a diagnostic: unstable or meaningless routing clusters on a new dataset would be evidence that the data lacks usable family structure, a use not developed in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: MixER is a real new combination — context-based top-1 MoE routing via K-means plus least squares — and the paper is honest about where it fails. But the central attribution is untested: the gains in the motivating ODEBench-2 experiment could come from partitioning the data into the true families, not from the K-means/least-squares gating protocol itself.

What is actually new: the gating update in Algorithm 1 is a genuine departure from gradient-based MoE gates. It treats context vectors as the routing signal, clusters them with K-means, and fits a linear gate by least squares. Running this with CoDA, NCF, and GEPS as experts is sensible, and the ODEBench-2 result (relative L2 drops from roughly 1.0–1.9 to 0.05–0.19) looks like a real effect. The paper also deserves credit for reporting its failures: MixER degrades on the high-data classical benchmarks, and the Epilepsy2 experiment shows the routing does not cleanly separate classes. That honesty is rare.

The soft spots are real. The stress-test note is right: there is no cluster-then-train baseline, so we cannot distinguish "the router helps" from "having one expert per family helps." The problem is compounded by setting M equal to the true family count in every family-structured experiment, and by initializing the vanilla K-means baseline at ground-truth means in Figure 6. The motivating Figure 2 has no error bars. On ODEBench-10B, MixER-10's adaptation is worse than MixER-1 for both NCF (1.43 vs 3.20) and CoDA (14.8 vs 0.34). The abstract concedes the high-data underperformance, but the positive case rests on a narrow set of results.

The circularity concern is milder. Routing to your own learned contexts is somewhat self-confirming, but the backbones are external and the SCCTS/Epilepsy2 evaluations at least use held-out contexts. Not disqualifying.

Who is this for: anyone working on meta-learning for scientific time series, and MoE routing generally. It is a workshop-grade contribution with a testable idea, not a settled result.

My recommendation: send it to review, but require the cluster-then-train baseline, error bars on the main experiment, and an explicit statement about how M is chosen. The paper survives that scrutiny only if the baseline comparison is favorable.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces MixER, a sparse top-1 mixture-of-experts layer for hierarchical meta-learning of dynamical system reconstruction. The router is trained by a non-gradient update: environment-specific context vectors (learned by the backbone meta-learner) are clustered with K-means, each cluster is paired with a distinct expert via a median-loss assignment, and a linear gating map is fitted by least squares to one-hot labels. Experiments use NCF, CoDA, and GEPS backbones on ODEBench-2/10A/10B, classical DSR benchmarks (Lotka-Volterra, Glycolytic Oscillator, Sel'kov), and two clustering datasets (SCCTS, Epilepsy2). The paper reports that MixER substantially improves reconstruction on the two-family ODEBench-2 task, improves training performance on ODEBench-10A, but yields mixed or degraded adaptation performance on ODEBench-10B and on the classical benchmarks, which the authors honestly acknowledge.

Significance. If the central claims are established, MixER would be a useful contribution: it demonstrates a non-gradient, context-conditioned routing alternative to standard MoE gating for DSR, and it provides evidence that naively increasing expert count is not sufficient. The paper is honest about failure regimes and provides code and detailed appendices. However, the current experimental evidence does not isolate the routing protocol as the source of the gains, because the cluster-then-train baseline is missing and the headline result has no error bars. Given the mixed results in larger benchmarks, the significance of the contribution is proportional to whether the missing attribution test can be supplied.

major comments (4)

[Section 4.1, Figure 2] The central positive result on ODEBench-2 is presented without error bars or seed variation. The top panel of Figure 2 reports a single relative L2 value per backbone (NCF: 0.186, CoDA: 0.045, GEPS: 0.142) with no variance; given that the same method on ODEBench-10A reports CoDA adaptation as 15.9 ± 4.2 (Table 2), a single run is not sufficient to support the claim of dramatic improvement. Please provide means and standard deviations over at least three seeds for the ODEBench-2 experiment, and state the number of seeds used in Figure 2.
[Section 4.1, Tables 2-3 and Section 1] The paper never compares against the cluster-then-train baseline that the Introduction claims MixER obviates. With M set equal to the true number of families in every structured experiment, the gains could arise from the a priori provision of the correct number of clusters or from added expert capacity, rather than from the K-means/least-squares gating update. Add a baseline that partitions the training environments into the true families (or via K-means on the same contexts) and trains one expert per partition with matched total parameter count, and report it on ODEBench-2 and ODEBench-10A/10B. Without this comparison, the core attribution claim of the paper is untested.
[Section 3.1 and all experiments] M is a free parameter that is always set to the true family count, so the method relies on an oracle quantity despite the claim of unsupervised family discovery. Add sensitivity experiments with M different from the true family count (e.g., M=3 or 5 on ODEBench-2, and M=5 or 20 on ODEBench-10B), or provide a selection procedure for M that does not use family labels. This also affects the SCCTS evaluation, where three experts are chosen to match the known pairs of classes (Section 4.3).
[Section 4.3, Figure 6] The SCCTS comparison with vanilla K-means is qualitative and the baseline is initialized with ground-truth means; the claim that MixER 'unambiguously outperforms' is not supported by any quantitative clustering metric. Report adjusted Rand index or normalized mutual information for both methods on the test contexts, and specify the K-means initialization and number of clusters. Since SCCTS is one of the few tasks where MixER shows a clear advantage, this evidence needs to be properly quantified.

minor comments (5)

[Section 3.2, Algorithm 1] If K-Means returns a null centroid for an empty cluster (Algorithm 2, line 9), Algorithm 1 does not specify a fallback; please state the handling of empty clusters (e.g., reinitialize the centroid) or guarantee that they cannot occur.
[Section 4.1, Table 3] The TPRMSE metric is defined in Eq. (10) but the threshold ε=0.1 is only used in Table 3; please define the abbreviation explicitly in the main text and consider reporting the threshold in the metric name.
[Appendix C.1] Context splitting and context shifting are introduced as mechanisms that improve clustering, but no ablation quantifies their contribution; please add an ablation or at least describe their effect on the routing heatmaps.
[Section 6] The computational cost comparison is only qualitative ('typically exceed'); since the appendix provides per-experiment training times, a direct memory and wall-clock comparison against the single meta-learner would strengthen the stated limitation.
[Appendix D, Figures 10-11] Figures 10 and 11 in the appendix are not referenced in the main text; please cite them or remove them.

Circularity Check

2 steps flagged · score 3.0 of 10

Minor circularity in SCCTS evaluation (ground-truth K-means initialization) and an untested cluster-then-train alternative, but the central MixER routing derivation is not circular.

renaming known result [Section 4.3, Figure 6 caption and surrounding text]
"Vanilla K-Means MixER - 1Expert MixER - 3 Experts. Figure 6. PCA clusters formed when training a MixER on the SCCTS dataset. (Top) Coloring using the ground truth labels; (Bottom) Coloring using labels from a K-means algorithm, with its means initialized at the ground truth means."

The SCCTS evaluation gives the K-means baseline (and, through Algorithm 1's K-means stage, the MixER routing) centroid initialization at the ground-truth class means. With 600 time series each forming its own environment and 6 ground-truth classes collapsed into 3 families, initializing cluster means at the true class means largely fixes the cluster assignment before any learning happens. The reported 'improved class separation' then partly reflects the injected ground truth rather than an emergent property of the learned contexts. This is a fitted-input-called-prediction pattern for the clustering claim, though it does not affect the ODEBench reconstruction results.
other [Section 2.2 and Tables 2-3; contrast with Section 5 statement 'Clustering before training... would constrain the adaptability of our models']
"We claim that strategic combination of contextual meta-learners enables simultaneous reconstruction across all families while preserving rapid adaptation capabilities, obviating the need for manual dataset partitioning prior to meta-learning on each subset."

The central attribution — that the K-means/least-squares gating protocol, rather than merely partitioning the data into the correct number of families, drives the gains — is never tested against the obvious 'cluster-then-train' baseline that the paper itself says MixER obviates. In every family-structured benchmark M is set equal to the true family count (M=2 for ODEBench-2, M=10 for ODEBench-10A/B, M=2 for Epilepsy2), so MixER is given the number of clusters in advance. This is a missing control rather than a demonstrated circular reduction; however, the absence of that control weakens the load-bearing comparison and pushes the score above the minimum. It is not a definitional equivalence because the routing still solves a nontrivial assignment problem on learned contexts.

full rationale

The paper's core derivation is not circular by construction. MixER's gating update (Algorithm 1) takes learned contexts Xi, clusters them with K-means, computes per-expert losses, creates one-hot proxy labels, and fits W by least squares; none of these steps re-inserts the final test metric as a fitted constant, and the backbones (NCF, CoDA, GEPS, hier-shPLRNN) are externally published methods. The ODEBench results are genuine empirical comparisons against MixER-1 and a gradient-gated MoE, and the paper honestly reports that MixER-10 underperforms in several ODEBench-10B columns, which is evidence against a forced favorable outcome. Two concerns raise the score to 3 rather than 0. First, the SCCTS Figure 6 evaluation leaks ground truth through K-means initialization at the true class means, making the 'improved clustering' claim partially self-fulfilling; the text explicitly describes initializing the K-means baseline 'with its means initialized at the ground truth means,' and the same K-means primitive drives MixER's own routing, so the clustering visualization is partly contaminated. Second, the central claim that MixER 'obviates the need for manual dataset partitioning' is not tested against a cluster-then-train baseline, and M is always set to the true family count, so part of the observed improvement could be attributed to supplying the correct number of clusters rather than to the K-means/least-squares routing protocol. These are experimental-validity gaps, not definitional circularity: no equation reduces to its own input, and no load-bearing result is imported solely from the authors' prior work. The paper is self-contained against external benchmarks and includes honest limitation statements, so a score of 3 — rather than 0 — reflects the ground-truth leakage and the missing control while acknowledging that the central reconstruction derivation is independent.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or ontological entities. Its free parameters are standard architectural and algorithmic choices, but the most consequential is the expert count M, which is chosen to match the true number of families in every experiment. The core assumptions concern the geometry of learned contexts and the suitability of one-expert-per-family routing.

free parameters (4)

Number of experts M = 2, 10, 20, 3, 2 depending on dataset, usually equal to the true family count
Chosen by hand to match ground truth family count; routing success depends on this choice and M is not learned or evaluated for misspecification.
Context dimension dξ = 40 (ODEBench), 512 (NCF), 2 (CoDA/GEPS), 10 (hier-shPLRNN)
Hyperparameter per backbone, chosen for the experiments and listed in Section 4 and Appendix C.
Noise standard deviation σ for least-squares gating fit = 1e-4
Injected into contexts before the least squares solve in Algorithm 1; tuned by hand.
Context splitting and shifting parameters = m = M equal-length splits, one offset per expert
Ad hoc mechanisms introduced in Appendix C.1 to encourage clustering when families are loosely related.

assumptions (4)

domain assumption Context vectors learned by the backbone meta-learner are comparable across environments and cluster by family under L1 distance.
K-means gating in Algorithm 1 clusters contexts directly; if contexts do not reflect family structure, routing fails, as seen in the Epilepsy2 experiment.
domain assumption Each environment is best served by exactly one exclusive expert, and each expert is assigned to exactly one cluster of environments.
Algorithm 1 enforces one-to-one expert-cluster pairing and disallows soft mixing or multi-expert routing.
domain assumption The number of experts M equals the number of underlying families and is known a priori.
All experiments set M to the true family count; the method does not estimate M and the paper does not study what happens when M is wrong.
standard math Proximal alternating minimization converges to a useful second-order critical point under the stated assumptions.
Invoked in Section 3.1 based on Li et al. (2019); the assumptions are asserted to be easily met without verification in this setting.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts." pith.science (2026). https://pith.science/paper/AFZ2RXNM

@misc{pith2026250205335,
  author       = {Pith},
  title        = {Pith review of: Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFZ2RXNM}},
  note         = {Machine review of arXiv:2502.05335}
}

abstract

As foundational models reshape scientific discovery, a bottleneck persists in dynamical system reconstruction (DSR): the ability to learn across system hierarchies. Many meta-learning approaches have been applied successfully to single systems, but falter when confronted with sparse, loosely related datasets requiring multiple hierarchies to be learned. Mixture of Experts (MoE) offers a natural paradigm to address these challenges. Despite their potential, we demonstrate that naive MoEs are inadequate for the nuanced demands of hierarchical DSR, largely due to their gradient descent-based gating update mechanism which leads to slow updates and conflicted routing during training. To overcome this limitation, we introduce MixER: Mixture of Expert Reconstructors, a novel sparse top-1 MoE layer employing a custom gating update algorithm based on $K$-means and least squares. Extensive experiments validate MixER's capabilities, demonstrating efficient training and scalability to systems of up to ten parametric ordinary differential equations. However, our layer underperforms state-of-the-art meta-learners in high-data regimes, particularly when each expert is constrained to process only a fraction of a dataset composed of highly related data points. Further analysis with synthetic and neuroscientific time series suggests that the quality of the contextual representations generated by MixER is closely linked to the presence of hierarchical structure in the data.

Figures

Figures reproduced from arXiv: 2502.05335 by the authors.

**Figure 1.** ). Training Samples Adaptation Sample Task-Specific Meta-Learning Task-Specific Meta-Learning Hierarchical Meta-Learning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Limitation of task-specific meta-learning and vanilla MoE on the two families of ODEs from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Illustration of vanilla MoE and our proposed MixER layer. (Left) Vanilla MoE setting where a single input x is passed through a gating network whose outputs enable the router to assign computation to a specific expert (Chen et al., 2022). (Right) Our sparse MixER layer requires a context vector ξ alongside the input x. The gating network computes expert affinities based on this context vector. Contrary to MoE, the M… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Algorithm 1 Gating Network Update 1: Require: Θ := {θm}m∈[M] mixture of M experts 2: Ξ := {ξ e} e∈[E×F ] context vectors 3: Ξ := ¯ { ¯ξm}m∈[M] centroid initialization 4: Dtr ≜ {De tr} e∈[E] training data 5: σ > 0 noise standard deviation 6: C, Ξ¯ ← K-Means(Ξ, Ξ) ¯ ▷ se…

**Figure 5.** Figure 5: Gating weights on ODEBench-10B, at the end of training with MixER-10. (Top) Gating heatmap. (Bottom) Histogram across all 160 environments. 2022); and (iii) Sel’kov Model (SM), a more complex 2- dimensional ODE for glycolysis that exhibits a Hopf bifurcation (Nzoyem e…

**Figure 7.** Figure 7: PCA clusters on the Epilepsy2 datasets, using the hiershPLRNN meta-learning backbone. Accuracy scores are obtained on the testing contexts upon training a logistic regression classifier. Conversely, Epilepsy2 results ( [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 6.** Figure 6: PCA clusters formed when training a MixER on the SCCTS dataset. (Top) Coloring using the ground truth labels; (Bottom) Coloring using labels from a K-means algorithm, with its means initialized at the ground truth means. SCCTS results ( [PITH_FULL_IMAGE:figures/full_f…

**Figure 8.** Figure 8: Visualisation of the clustering heatmap as the training progresses on ODEBench-2. The four columns correspond to outer training steps 0, 25, 125, and 250 respectively (from left to right). (Top) Naive mixture of two GEPS models with gating updates via vanilla gradient …

**Figure 9.** Figure 9: Heatmaps of the gating values of MixER with 3 experts on three classical meta-learning datasets: LV, GO, and SM. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

**Figure 10.** Figure 10: Visualization of a single testing trajectory and the phase space within the first 5 families with 10 expert GEPS meta-learners on the large ODEBench-10B dataset. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

**Figure 11.** Figure 11: Visualization of a single testing trajectory and the phase space within the last 5 families with 10 expert GEPS meta-learners on the large ODEBench-10B dataset. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 39 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501.12370, 2025

arXiv 2025
[3]

Synthetic Control Chart Time Series

Alcock, R. Synthetic Control Chart Time Series . UCI Machine Learning Repository, 1999. DOI : https://doi.org/10.24432/C59G75

work page doi:10.24432/c59g75 1999
[4]

G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C

Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C. E. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64 0 (6): 0 061907, 2001

work page 2001
[5]

Invariant risk minimization

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

arXiv 1907
[6]

Robust solutions of optimization problems affected by uncertain probabilities

Ben-Tal, A., Den Hertog, D., De Waegenaere, A., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59 0 (2): 0 341--357, 2013

work page 2013
[7]

and Lelarge, M

Blanke, M. and Lelarge, M. Interpretable meta-learning of physical systems. In ICLR 2024-The Twelfth International Conference on Learning Representations, 2024

work page 2024
[8]

P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al

Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 2024

arXiv 2024

Show all 73 references

[9]

J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/google/jax

2018
[10]

Message passing neural pde solvers

Brandstetter, J., Worrall, D., and Welling, M. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022

2022 arXiv
[11]

Learning interpretable hierarchical dynamical systems models from time series data

Brenner, M., Weber, E., Koppe, G., and Durstewitz, D. Learning interpretable hierarchical dynamical systems models from time series data. arXiv preprint arXiv:2410.04814, 2024

2024 arXiv
[12]

Multitask learning

Caruana, R. Multitask learning. Machine learning, 28: 0 41--75, 1997

1997
[13]

T., Rubanova, Y., Bettencourt, J., and Duvenaud, D

Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

2018
[14]

Towards understanding the mixture-of-experts layer in deep learning

Chen, Z., Deng, Y., Wu, Y., Gu, Q., and Li, Y. Towards understanding the mixture-of-experts layer in deep learning. Advances in neural information processing systems, 35: 0 23049--23062, 2022

2022
[15]

Cox, D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20 0 (2): 0 215--232, 1958

1958
[16]

S., Giampaolo, F., Rozza, G., Raissi, M., and Piccialli, F

Cuomo, S., Di Cola, V. S., Giampaolo, F., Rozza, G., Raissi, M., and Piccialli, F. Scientific machine learning through physics--informed neural networks: Where we are and what’s next. Journal of Scientific Computing, 92 0 (3): 0 88, 2022

2022
[17]

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

2024 arXiv
[18]

ODEF ormer: Symbolic regression of dynamical systems with transformers

d'Ascoli, S., Becker, S., Schwaller, P., Mathis, A., and Kilbertus, N. ODEF ormer: Symbolic regression of dynamical systems with transformers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TzoHLiGVMo

2024
[19]

and Giltinan, D

Davidian, M. and Giltinan, D. M. Nonlinear models for repeated measurement data: an overview and update. Journal of agricultural, biological, and environmental statistics, 8: 0 387--419, 2003

2003
[20]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

2022
[21]

Model-agnostic meta-learning for fast adaptation of deep networks

Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.\ 1126--1135. PMLR, 2017

2017
[22]

W., Rezende, D., and Eslami, S

Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. Conditional neural processes. In International conference on machine learning, pp.\ 1704--1713. PMLR, 2018

2018
[23]

and Bengio, Y

Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

2010
[24]

Out-of-domain generalization in dynamical systems reconstruction

G \"o ring, N., Hess, F., Brenner, M., Monfared, Z., and Durstewitz, D. Out-of-domain generalization in dynamical systems reconstruction. arXiv preprint arXiv:2402.18377, 2024

2024 arXiv
[25]

and Dao, T

Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

2023 arXiv
[26]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[27]

and Ruthotto, L

Haber, E. and Ruthotto, L. Stable architectures for deep neural networks. Inverse problems, 34 0 (1): 0 014004, 2017

2017
[28]

Neural networks: a comprehensive foundation

Haykin, S. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994

1994
[29]

He, X. O. Mixture of a million experts. arXiv preprint arXiv:2407.04153, 2024

2024 arXiv
[30]

Poseidon: Efficient foundation models for pdes

Herde, M., Raoni \'c , B., Rohner, T., K \"a ppeli, R., Molinaro, R., de B \'e zenac, E., and Mishra, S. Poseidon: Efficient foundation models for pdes. arXiv preprint arXiv:2405.19101, 2024

2024 arXiv
[31]

Generalized teacher forcing for learning chaotic dynamics

Hess, F., Monfared, Z., Brenner, M., and Durstewitz, D. Generalized teacher forcing for learning chaotic dynamics. arXiv preprint arXiv:2306.04406, 2023

2023 arXiv
[32]

Meta-learning in neural networks: A survey

Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 5149--5169, 2021

2021
[33]

J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[34]

A., Jordan, M

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3 0 (1): 0 79--87, 1991

1991
[35]

Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

2024 arXiv
[36]

Keogh, E. J. and Pazzani, M. J. Derivative dynamic time warping. In SDM, 2001. URL https://api.semanticscholar.org/CorpusID:6611383

2001
[37]

On neural differential equations

Kidger, P. On neural differential equations. arXiv preprint arXiv:2202.02435, 2022

2022 arXiv
[38]

and Garcia, C

Kidger, P. and Garcia, C. E quinox: neural networks in JAX via callable P y T rees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021

2021
[39]

Neural controlled differential equations for irregular time series

Kidger, P., Morrill, J., Foster, J., and Lyons, T. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33: 0 6696--6707, 2020

2020
[40]

Generalizing to new physical systems via context-informed dynamics model

Kirchmeyer, M., Yin, Y., Don \`a , J., Baskiotis, N., Rakotomamonjy, A., and Gallinari, P. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pp.\ 11283--11301. PMLR, 2022

2022
[41]

o wer, M., Lottes, J., Rasp, S., D \

Kochkov, D., Yuval, J., Langmore, I., Norgaard, P., Smith, J., Mooers, G., Kl \"o wer, M., Lottes, J., Rasp, S., D \"u ben, P., et al. Neural general circulation models for weather and climate. Nature, 632 0 (8027): 0 1060--1066, 2024

2024
[42]

K., Benet, J

Koupa \" , A. K., Benet, J. M., Yin, Y., Vittaut, J.-N., and Gallinari, P. Boosting generalization in parametric pde neural solvers through adaptive conditioning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[43]

L., Tombolini, C., Koppe, G., and Durstewitz, D

Kramer, D., Bommer, P. L., Tombolini, C., Koppe, G., and Durstewitz, D. Reconstructing nonlinear dynamical systems from multi-modal time series. arXiv preprint arXiv:2111.02922, 2021

2021 arXiv
[44]

Out-of-distribution generalization via risk extrapolation (rex)

Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. Out-of-distribution generalization via risk extrapolation (rex). In International conference on machine learning, pp.\ 5815--5826. PMLR, 2021

2021
[45]

Alternating minimizations converge to second-order optimal solutions

Li, Q., Zhu, Z., and Tang, G. Alternating minimizations converge to second-order optimal solutions. In International Conference on Machine Learning, pp.\ 3935--3943. PMLR, 2019

2019
[46]

Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models

Liang, W., Yu, L., Luo, L., Iyer, S., Dong, N., Zhou, C., Ghosh, G., Lewis, M., Yih, W.-t., Zettlemoyer, L., et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models. arXiv e-prints, pp.\ arXiv--2411, 2024

2024
[47]

T., Ben-Hamu, H., Nickel, M., and Le, M

Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

2022 arXiv
[48]

Flow straight and fast: Learning to generate and transfer data with rectified flow

Liu, X., Gong, C., and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=XVjTT1nw5z

2023
[49]

Least squares quantization in pcm

Lloyd, S. Least squares quantization in pcm. IEEE transactions on information theory, 28 0 (2): 0 129--137, 1982

1982
[50]

K., and Grover, A

Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., and Grover, A. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343, 2023

2023 arXiv
[51]

D., Barton, D

Nzoyem, R. D., Barton, D. A., and Deakin, T. A comparison of mesh-free differentiable programming and data-driven strategies for optimal control under pde constraints. In Proceedings of the SC'23 Workshops of The International Conference on High Performance Computing, Network,...

2023
[52]

D., Barton, D

Nzoyem, R. D., Barton, D. A., and Deakin, T. Extending contextual self-modulation: Meta-learning across modalities, task dimensionalities, and data regimes. arXiv preprint arXiv:2410.01655, 2024

2024 arXiv
[53]

D., Barton, D

Nzoyem, R. D., Barton, D. A., and Deakin, T. Neural context flows for meta-learning of dynamical systems. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=8vzMLo8LDN

2025
[54]

and Chan, A

Pham, D. and Chan, A. Control chart pattern recognition using a new type of self-organizing neural network. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 212 0 (2): 0 115--127, 1998

1998
[55]

Universal differential equations for scientific machine learning

Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., Ramadhan, A., and Edelman, A. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020

2001 arXiv
[56]

Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017

2017 arXiv
[57]

Efficient amortised bayesian inference for hierarchical and nonlinear dynamical systems

Roeder, G., Grant, P., Phillips, A., Dalchau, N., and Meeds, E. Efficient amortised bayesian inference for hierarchical and nonlinear dynamical systems. In International Conference on Machine Learning, pp.\ 4445--4455. PMLR, 2019

2019
[58]

W., Hashimoto, T

Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apri...

2020
[59]

K., Wang, T

Serrano, L., Koupa \" , A. K., Wang, T. X., Erbacher, P., and Gallinari, P. Zebra: In-context and generative pretraining for solving parametric pdes. arXiv preprint arXiv:2410.03437, 2024

2024 arXiv
[60]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017 a . URL https://openreview.net/forum?id=B1ckMDqlg

2017
[61]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017 b

2017 arXiv
[62]

P., Gentine, P., Bandai, T., Gupta, H., Tartakovsky, A., Baity-Jesi, M., Fenicia, F., Kifer, D., Li, L., et al

Shen, C., Appling, A. P., Gentine, P., Bandai, T., Gupta, H., Tartakovsky, A., Baity-Jesi, M., Fenicia, F., Kifer, D., Li, L., et al. Differentiable modelling to unify machine learning and physical models for geosciences. Nature Reviews Earth & Environment, pp.\ 1--16, 2023

2023
[63]

Differentiable clustering with perturbed spanning forests

Stewart, L., Bach, F., Llinares-L \'o pez, F., and Berthet, Q. Differentiable clustering with perturbed spanning forests. Advances in Neural Information Processing Systems, 36, 2024

2024
[64]

Strogatz, S. H. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018

2018
[65]

W., and Gholami, A

Subramanian, S., Harrington, P., Keutzer, K., Bhimji, W., Morozov, D., Mahoney, M. W., and Gholami, A. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36, 2024

2024
[66]

Learning neural pde solvers with parameter-guided channel attention

Takamoto, M., Alesiani, F., and Niepert, M. Learning neural pde solvers with parameter-guided channel attention. In International Conference on Machine Learning, pp.\ 33448--33467. PMLR, 2023

2023
[67]

Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation

Wang, H., Zhao, H., and Li, B. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. In International conference on machine learning, pp.\ 10991--11002. PMLR, 2021

2021
[68]

Meta-learning dynamics forecasting using task inference

Wang, R., Walters, R., and Yu, R. Meta-learning dynamics forecasting using task inference. Advances in Neural Information Processing Systems, 35: 0 21640--21653, 2022

2022
[69]

A proposal on machine learning via dynamical systems

Weinan, E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 1 0 (5): 0 1--11, 2017

2017
[70]

Leads: Learning dynamical systems that generalize across environments

Yin, Y., Ayed, I., de B \'e zenac, E., Baskiotis, N., and Gallinari, P. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Information Processing Systems, 34: 0 7561--7573, 2021

2021
[71]

Self-supervised contrastive pre-training for time series via time-frequency consistency

Zhang, X., Zhao, Z., Tsiligkaridis, T., and Zitnik, M. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35: 0 3988--4003, 2022

2022
[72]

C., Dvornek, N., Papademetris, X., and Duncan, J

Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvornek, N., Papademetris, X., and Duncan, J. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33: 0 18795--18806, 2020

2020
[73]

Fast context adaptation via meta-learning

Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and Whiteson, S. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp.\ 7693--7702. PMLR, 2019

2019

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501.12370, 2025

arXiv 2025

[3] [3]

Synthetic Control Chart Time Series

Alcock, R. Synthetic Control Chart Time Series . UCI Machine Learning Repository, 1999. DOI : https://doi.org/10.24432/C59G75

work page doi:10.24432/c59g75 1999

[4] [4]

G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C

Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C. E. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64 0 (6): 0 061907, 2001

work page 2001

[5] [5]

Invariant risk minimization

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

arXiv 1907

[6] [6]

Robust solutions of optimization problems affected by uncertain probabilities

Ben-Tal, A., Den Hertog, D., De Waegenaere, A., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59 0 (2): 0 341--357, 2013

work page 2013

[7] [7]

and Lelarge, M

Blanke, M. and Lelarge, M. Interpretable meta-learning of physical systems. In ICLR 2024-The Twelfth International Conference on Learning Representations, 2024

work page 2024

[8] [8]

P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al

Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 2024

arXiv 2024