{"id":"1467018e-c427-44e2-89ea-60d83f2abedd","arxiv_id":"2502.04131","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mapping MAP estimates of unidentifiable dynamical models to structurally identifiable parameter combinations improves time series classification generalization, especially with few training samples.","lead":"This paper introduces a method for improving machine learning classifiers on time series data when the underlying biological model has several parameter settings that produce identical output. The method maps each data sample to the model's identifiable parameter combinations, which improves generalization especially when training data is scarce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SIM's central claim rests on the untested assumption that class labels depend only on observable behavior; all experiments generate labels aligned with identifiable combinations, so the method's failure mode is never examined.","rationale":"The reader identified the same load-bearing concern: the classification task is assumed to assign the same label to observationally equivalent parameter configurations, and the synthetic experiments never violate this. I agree this is the most fundamental risk to the central claim. If the assumption fails, SIM could remove information that is relevant for the label, directly undermining the claimed improvement. The paper states the assumption explicitly (Section II.B) but does not analyze its scope, provide a robustness test, or discuss when it might be violated in practice. The statistical-significance issue flagged by the reader is secondary: even if formal tests were added, the qualitative conclusion would stand under the assumption; the label assumption, by contrast, determines whether the method is even appropriate. My recommendation is unchanged (CONDITIONAL) because the paper is otherwise well-structured and the experiments are reproducible, but it should be revised to address this assumption, either by additional experiments along the lines of my concrete test or by an explicit theoretical discussion of when SIM is beneficial.","tokens_in":20318,"tokens_out":7841,"duration_ms":90834,"concrete_test":"Use the released MATLAB code and the BR model from Section III-A.4. Generate synthetic data exactly as in Experiment 1 (with y(t)=x(t) observed), but define the class label directly as c=1 iff the hidden parameter b2 exceeds its class-0 mean (30) and c=0 otherwise, while keeping all other parameter draws unchanged. Train the PO and PO+SIM classifiers at Ntrain=10 and Ntest=400 with the same SVM settings and 20 subsamples; compare mean generalization error and its standard deviation. If PO+SIM does not match or beat PO (i.e., the method loses its advantage or hurts), the label assumption is load-bearing. Repeat the analogous experiment for CCM2 with labels depending on k12 alone while observing only x1, and for CML with labels depending on the unobserved state x2 at a fixed time.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core justification for SIM appears in Section II.B: observationally equivalent parameters (those with the same identifiable combinations Φ) 'should therefore be associated with the same class.' This is a substantive assumption, not a consequence of the setup. In the synthetic experiments (Section III.B.1), class labels are generated by drawing θ from class-conditional distributions, and the class means are chosen so that classes differ in identifiable combinations: in BR, the classes differ in Y, which enters Φ4=b2Y; in CCM2 and CCM4, the classes differ in k12, k21 and related rates, which enter the identifiable products; in CML the same holds. Thus, the experiments only exercise the regime where the label is a function of Φ, and never the regime where the label depends on unidentifiable directions (e.g., on b2 separately from Y, or on the unobserved state x2 in the compartmental models). If real-world labels depend on such hidden quantities, SIM's projection onto Φ discards label-relevant information and could systematically degrade classification accuracy. The discussion (Section V) acknowledges limitations regarding optimization and MAP uncertainty but does not analyze or test this label assumption. Consequently, the central claim that SIM 'significantly improves' generalization is conditional on an assumption that is plausible for well-posed problems but is not guaranteed in the partially observed biomedical contexts the paper targets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Structural Identifiability Mapping (SIM), a preprocessing step for model-based time series classification with partially observed dynamical systems. When the underlying ODE model is structurally unidentifiable, each time series is first represented by a MAP estimate of the model parameters, and that estimate is then mapped through the structurally identifiable parameter combinations Φ = g(θ). A vectorial classifier (an SVM in the experiments) is trained in this Φ-space rather than in the original parameter space. The method is tested on four synthetic binary classification tasks built from biomedical example models (toy model, two- and four-compartment catenary models, a compartment model with a loop, and a batch reactor model), with experiments varying the number of training samples, the observational noise level, and the density and regularity of the time grid. The central claim is that SIM improves generalization, especially in the small-training-sample regime, and that the learned decision boundary remains interpretable in the original parameter space via the pre-image g^{-1}(Φ).","tokens_in":20574,"tokens_out":3757,"duration_ms":39222,"significance":"The paper addresses a genuinely underappreciated issue: structural unidentifiability of a mechanistic model can mislead classifiers trained on MAP estimates of its parameters. The proposed SIM is simple, computationally cheap after the one-time identifiability analysis, and builds on established structural identifiability results from the systems-biology literature. The empirical study is broad in the sense that it covers linear and nonlinear ODE models, several noise levels, and three types of time grids, and the authors provide public code for reproducibility. The regularizing interpretation of SIM (it enforces that observationally equivalent parameters receive the same label) is plausible and coherent. However, the strength of the central claim is not matched by the statistical evidence: no significance tests are reported, and some of the headline comparisons are within one standard deviation. In addition, the method's key assumption—that class labels should be constant on equivalence classes of observationally indistinguishable parameters—is never tested, and the synthetic setup is constructed so that labels are functions of the identifiable combinations by design.","major_comments":[{"comment":"The abstract and conclusion claim that SIM \"significantly improves\" generalization, but no statistical significance tests are reported, and several of the reported differences are within one standard deviation. In Table III, the CCM2 row at Nmin shows a mean generalization error of .07 (.04) for PO versus .06 (.05) for PO + SIM, a difference of 0.01 with overlapping standard deviations. In Table V, the CCM2 irregular-grid row shows identical means of .2 (.09) vs .2 (.04). The claim of a consistent, significant improvement is therefore not supported by the presented statistics. The authors should either add paired significance tests across the 20 random subsamples, provide confidence intervals, or substantially temper the wording of the central claim.","section":"Section IV.A, Tables III, IV, V"},{"comment":"The method rests on the assumption, stated in Section II.B, that any two parameter values yielding identical system output \"should therefore be associated with the same class.\" This is a substantive modeling assumption, not a mathematical consequence. The synthetic experiments do not test it: in every case the class-conditional parameter means differ only in identifiable combinations—for BR, the classes differ in Y, which enters Φ4 = b2Y; for CCM2 and CCM4, the classes differ in conversion rates k12, k21, which enter the identifiable products Φ2_j; for CML the same pattern holds. If a real-world classification label depended on an unidentifiable direction (e.g., on b2 separately from Y in the batch reactor, or on the unobserved substrate concentration), SIM would discard label-relevant information and could systematically degrade accuracy. The paper needs to either add an experiment in which the class label depends on a non-identifiable parameter or an unobserved state, or provide a clear argument why such dependence cannot occur in the intended applications.","section":"Section II.B and Section III.B.1"},{"comment":"The MAP problem in Eq. (24) is non-unique under structural unidentifiability, as the paper acknowledges, and the actual solution found by simulannealbnd will be an arbitrary point on the equivalence manifold, subject to optimizer convergence and noise. The invariance of Φ under exact equivalence is mathematical, but with finite noisy data and numerical optimization the MAP estimate may not lie exactly on the true manifold, and the paper does not analyze how optimization error propagates through g. This is a secondary concern compared to the label-assumption issue, but a short analysis of the sensitivity of Φ-representations to MAP estimation error would strengthen the method's practical claims.","section":"Section III.B.1, Eq. (24)"}],"minor_comments":[{"comment":"The text states that \"the choices for tdense for the different example models are reported in Table I,\" but the dense grids are listed in Table II; the cross-reference should be corrected.","section":"Section III.B.3"},{"comment":"The description of the clinical model underlying CCM2 is slightly confusing: \"Their 3-compartment model is equivalent to a CCM with 2 compartments with non-zero input.\" Clarifying whether the equivalence is to a two-compartment catenary model with an input term, and how the third compartment is absorbed, would help the reader.","section":"Section III.A.2"},{"comment":"In the AutoRepar reparametrisation of CCM2, the second equation reads \"˙˜x2 = ˜k21x1 + 2˜x2,\" which appears to use the original state x1 instead of the transformed state ˜x1; this is likely a typographical error and should be checked.","section":"Appendix A"},{"comment":"The learning curves in Figure 5 show error bars but no indication of the number of repetitions or the method used to compute the standard deviations; the caption should state that these are over the 20 random subsamples mentioned in Section III.B.1.","section":"Section IV.A, Figure 5"},{"comment":"The term \"significantly\" is used repeatedly in Sections I, IV, and VI without a statistical test backing it; the authors should either perform tests and report p-values or replace this wording with \"consistently lower\" or \"noticeably lower\" where appropriate.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the presentation is generally clear, but the experimental support for the headline claim needs to be tightened. The lack of significance testing and the untested label-assumption are load-bearing: the reader cannot currently know whether SIM helps only in the favorable regime where labels align with identifiable combinations. Both issues are addressable within the paper's scope, so I see no reason to reject, but the manuscript should not be accepted in its present form. The reviewer should also check whether the authors' code, if run, reproduces the tables; the GitHub link is given but the paper does not state a version or checksum."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful paper with a modest but real idea. SIM—mapping MAP estimates to structurally identifiable parameter combinations before training a classifier—is a straightforward, low-cost trick that recovers much of the gap between partially and fully observed model-based classifiers in the low-data regime. The experiments are reproducible (code on GitHub), the models are standard from the systems-biology literature, and the identifiable combinations are taken from published SI results rather than fit to the data. That last point matters: the circularity burden is low.\n\nWhat's genuinely new is treating identifiability as a representation choice for the classifier rather than a modeling problem to be solved by reparametrisation. The paper shows that on four of five benchmarks, PO+SIM is close to FO at small Ntrain, and it stays ahead of PO across noise levels and sampling grids. The mechanism—regularizing the decision boundary along unidentifiable fibers—is clearly explained.\n\nThe soft spots are statistical. The abstract and conclusion say 'significantly improves,' but there are no significance tests anywhere. At Nmin, CCM2's gain is .07 vs .06 with overlapping standard deviations; that is not a significant improvement. The larger gains on CCM4, CML, and BR are more convincing, but without tests or effect sizes we cannot call them significant. This is an easy fix: report paired tests (e.g., Wilcoxon over the 20 subsamples) or at least confidence intervals on the differences.\n\nOn the stress-test: I think the 'label assumption' concern is mostly a misfire. If two parameter vectors produce identical observable trajectories, then no classifier looking only at those trajectories can assign them different labels with any reliability—the problem would be statistically unidentifiable in the classification sense. So SIM is not discarding label-relevant information; it is removing an ambiguity that no classifier could exploit in a generalizable way. The synthetic experiments do only exercise the regime where the label is a function of Φ, but that is the only regime in which perfect classification from observations is possible. The paper could have said this explicitly, and I would encourage a revision to do so, but I do not see it as a load-bearing flaw.\n\nVerdict: worth a serious referee. It is not a major theoretical advance, but it is a practical method the model-based ML community should know about. The writing is clear, the code is public, and the scope is honestly stated. With the statistical claims tightened, this would be a solid contribution.","headline":"A sensible, simple representation trick for model-based time series classification, with real gains at low sample sizes—but the 'significant' claims need proper tests before they earn that word.","tokens_in":21074,"tokens_out":3722,"would_cite":true,"duration_ms":41531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","93B30","34A55"],"pacs":[],"model":"deepseek-v4-flash","headline":"Mapping time-series data to structurally identifiable parameter combinations, rather than raw parameter estimates, improves classifier generalization for partially observed dynamical models—most when training data are scarce.","keywords":["structural identifiability","time series classification","partially observed dynamical systems","maximum a posteriori estimation","identifiable parameter combinations","compartmental models","model-based classification"],"falsifier":"Construct the toy-model classification with labels drawn from the parameter $a$ alone while holding $ab$ fixed, so that two parameter vectors with identical output $x(t) = e^{-abt}$ carry different labels; if the SIM classifier trained on $\\Phi = ab$ then shows higher test error than a classifier trained on raw $(a,b)$ MAP estimates, the equivalence-class assumption is violated and SIM has discarded label-relevant information.","tokens_in":20141,"feed_emoji":"🧬","tokens_out":16065,"duration_ms":135053,"temperature":0.7,"pith_summary":"Partially observed dynamical models used to represent time series are often structurally unidentifiable: many different parameter settings produce exactly the same observable output, so point estimates of the parameters are ambiguous. The paper claims that training a classifier on such ambiguous estimates wastes capacity on separating models the data cannot distinguish, and proposes a cheap fix called Structural Identifiability Mapping (SIM). SIM computes a maximum-a-posteriori estimate for each time series, then projects it through the map $g$ onto the structurally identifiable parameter combinations $\\Phi = g(\\theta)$ before training any vectorial classifier. Across four example systems, including compartmental pharmacokinetic models and a microbial batch-reactor model, SIM reduces generalization error and classifier complexity relative to raw-parameter training, often bringing the partially observed classifier close to the fully observed ideal in the small-data regime. The paper's experiments show the gain persists under observational noise and under sparse and irregular sampling, which is exactly the data regime where model-based classification is most attractive.","feed_headline":"Project parameter fits onto identifiable combos to cut test error","feed_subtitle":"A structural-identifiability preprocessing step improves scarce-data time-series classification on four biomedical models.","key_machinery":"The central object is the identifiable parameter combination map $g : \\mathbb{R}^n \\to \\mathbb{R}^m$ (with $m < n$) obtained from structural identifiability analysis, together with the equivalence classes $M_\\Phi = \\{\\theta \\in \\mathbb{R}^n \\mid \\Phi = g(\\theta)\\}$ it defines; any two parameters in one class produce identical system trajectories. The pipeline computes a MAP estimate $\\theta^k_{\\mathrm{MAP}}$ for each time series under a uniform prior on a region of interest, applies $g$ to obtain $\\Phi^k$, and trains a Gaussian-kernel support vector machine on the transformed data. The map $g$ does the work: it collapses the directions in parameter space along which the input-output behavior is invariant, guarantees that the classifier's decision boundary is constant on each equivalence class, and simplifies the geometry of the task (the toy model with $\\Phi = ab$ becomes linearly separable in $\\Phi$-space). For the linear compartmental models the combinations come from Laplace-transform output-equality analysis (adjacent rate products such as $k_{j,j-1}k_{j-1,j}$), and for the batch reactor from earlier identifiability studies giving $\\Phi = (b_1, \\mu_m, K_d, b_2 Y, b_2/K_s)$.","core_discovery":"The paper's central claim is that structural identifiability—a property long studied in systems biology but largely ignored in machine learning—materially affects whether a model-based time-series classifier generalizes, and that accounting for it is nearly free. The mechanism is that two parameter vectors producing identical input-output behavior are observationally equivalent, so a classifier trained on raw MAP estimates can waste capacity separating parameter points that no data could distinguish. SIM removes this redundancy before learning: each estimate $\\theta^k_{\\mathrm{MAP}}$ is mapped to $\\Phi^k = g(\\theta^k_{\\mathrm{MAP}})$, so the learned decision boundary in parameter space becomes a union of full equivalence classes $M_\\Phi = \\{\\theta \\mid g(\\theta) = \\Phi\\}$, guaranteeing that output-equivalent parameters receive the same label. In experiments on four example systems—two catenary compartmental models, a loop compartmental model, and a nonlinear Michaelis-Menten batch reactor—SIM reduces the generalization error of the partially observed classifier in the low-training-data regime, often approaching the fully observed ideal, and it outperforms raw-parameter training across all tested noise levels and time grids. The authors position SIM as an alternative to reparametrising the model: it applies whenever identifiable combinations can be derived, even when a full reparametrisation is not feasible.","pith_inferences":["My inference: the equivalence-class argument should transfer to fully Bayesian pipelines, where the posterior is ridged along output-invariant directions, so projecting posterior draws through $g$ would regularise Bayesian classification the same way; the authors flag posterior sampling as future work but do not test this projection.","My inference: any parameter-estimation front end could be swapped in—amortized inference, recurrent encoders, or physics-informed networks that output parameter fits for unidentifiable ODEs should gain the same benefit from projecting through $g$, an experiment the paper recommends but does not run.","My inference: the untested risk is labels that depend on hidden internal states; if the class of a time series is set by an unobserved variable, the map $g$ discards exactly the information the label needs, and synthetic labels drawn from such a hidden state would likely make SIM underperform raw-parameter training.","My inference: SIM can be viewed as a principled, model-derived data-augmentation scheme for deep time-series classifiers in the biomedical small-data regime, since the equivalence structure $g$ supplies an invariance that a network would otherwise have to learn from very few examples."],"forward_implications":["At the smallest training sizes tested (10 examples per class), SIM cuts mean generalization error on every example system—for the batch reactor from 0.3 to 0.1—and several SIM classifiers approach the performance of a fully observed model.","As training data grow, raw-parameter and SIM classifiers converge, so SIM acts as a regularizer for the scarce-data regime rather than imposing a ceiling on performance.","The improvement persists across the tested range of observational noise and across dense, sparse, and irregular time grids, meaning the method does not require high-quality observations to help.","Structural identifiability is a property of the model, not of the data, so computing the map $g$ is a one-time cost; after that, SIM adds no extra computation to classifier training.","Decision boundaries learned in the identifiable-combination space can be pulled back to the original parameter space through the pre-image $g^{-1}(\\Phi)$, preserving interpretability for domain experts."],"supporting_citations":[{"why":"Supplies the model-based Bayesian MAP representation of partially observed dynamical systems on which SIM is built, along with the caveat that point estimates ignore posterior uncertainty.","marker":"[7]"},{"why":"Derives the 2n−1 structurally identifiable parameter combinations (diagonal terms and adjacent rate products) that define the SIM map for the catenary compartmental models.","marker":"[33]"},{"why":"Provides the clinical pharmacokinetic study whose cluster parametrisations define the CCM2 binary classification task used in the experiments.","marker":"[34]"},{"why":"Shows no scaling reparametrisation exists for the loop compartmental model, which motivates applying SIM where reparametrisation fails.","marker":"[35]"},{"why":"Supplies the batch-reactor model's realistic parameter configurations and the identifiability result for the fully observed case.","marker":"[37]"},{"why":"Demonstrates the batch reactor becomes structurally unidentifiable when only the microorganism concentration is observed, yielding the identifiable combinations in Eq. (21).","marker":"[38]"},{"why":"Extends the procedure for generating locally identifiable reparametrisations, supporting the batch-reactor identifiability analysis used to define the SIM map.","marker":"[39]"},{"why":"Describes the automated reparametrisation tool the authors try on the compartmental models; its inability to handle the loop model frames SIM as the practical alternative.","marker":"[21]"}],"fun_headline_variants":["Identifiability-aware training slashes test error on sparse time series","Fixing unidentifiable models boosts classifier generalization","Structural identifiability: key to ML on partially observed systems","Observational equivalence pre-processing improves time-series ML","Teaching classifiers to ignore unidentifiable parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that any two parameter settings producing identical observed time series should receive the same class label, an assumption the synthetic experiments build in by construction but that real classification problems could violate.","fun_headline_variants_meta":{"raw":{"variants":["Identifiability-aware training slashes test error on sparse time series","Fixing unidentifiable models boosts classifier generalization","Structural identifiability: key to ML on partially observed systems","Observational equivalence pre-processing improves time-series ML","Teaching classifiers to ignore unidentifiable parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000801,"raw_usage":{"total_tokens":3580,"prompt_tokens":1060,"completion_tokens":2520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":2443}},"tokens_in":676,"tokens_out":2520,"duration_ms":18873,"temperature":1.0,"reasoning_tokens":2443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:23:29.777791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct the toy-model classification with labels drawn from the parameter $a$ alone while holding $ab$ fixed, so that two parameter vectors with identical output $x(t) = e^{-abt}$ carry different labels; if the SIM classifier trained on $\\Phi = ab$ then shows higher test error than a classifier trained on raw $(a,b)$ MAP estimates, the equivalence-class assumption is violated and SIM has discarded label-relevant information.","supporting_citations":[{"cited_title":"Classification framework for partially observed dynamical systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the model-based Bayesian MAP representation of partially observed dynamical systems on which SIM is built, along with the caveat that point estimates ignore posterior uncertainty."},{"cited_title":"Algorithms for the identifiable parameter combinations and parameter bounds of uniden- tifiable catenary compartmental models,","cited_arxiv_id":null,"evidence_quote":"Derives the 2n−1 structurally identifiable parameter combinations (diagonal terms and adjacent rate products) that define the SIM map for the catenary compartmental models."},{"cited_title":"Learning pharmacokinetic models for in vivo glucocorticoid activation,","cited_arxiv_id":null,"evidence_quote":"Provides the clinical pharmacokinetic study whose cluster parametrisations define the CCM2 binary classification task used in the experiments."},{"cited_title":"Identifiable reparametrizations of lin- ear compartment models,","cited_arxiv_id":null,"evidence_quote":"Shows no scaling reparametrisation exists for the loop compartmental model, which motivates applying SIM where reparametrisation fails."},{"cited_title":"On the practical identifiability of microbial growth mod- els incorporating michaelis-menten type nonlinearities,","cited_arxiv_id":null,"evidence_quote":"Supplies the batch-reactor model's realistic parameter configurations and the identifiability result for the fully observed case."},{"cited_title":"Structural identifiability of the pa- rameters of a nonlinear batch reactor model,","cited_arxiv_id":null,"evidence_quote":"Demonstrates the batch reactor becomes structurally unidentifiable when only the microorganism concentration is observed, yielding the identifiable combinations in Eq. (21)."},{"cited_title":"Extensions to a procedure for gener- ating locally identifiable reparameterisations of unidentifiable systems,","cited_arxiv_id":null,"evidence_quote":"Extends the procedure for generating locally identifiable reparametrisations, supporting the batch-reactor identifiability analysis used to define the SIM map."},{"cited_title":"Autorepar: A method to obtain identifiable and observable reparameterizations of dynamic models with mechanistic insights,","cited_arxiv_id":null,"evidence_quote":"Describes the automated reparametrisation tool the authors try on the compartmental models; its inability to handle the loop model frames SIM as the practical alternative."}],"review_version":1}