Interpretable Neural System Dynamics: Combining Deep Learning with System Dynamics Modeling to Support Critical Applications

Riccardo D'Elia

REVIEW 4 major objections 5 minor 16 references

Interpretable Neural System Dynamics: Combining Deep Learning with System Dynamics Modeling to Support Critical Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An integrated pipeline that combines concept learning, causal learning, and equation learning can make deep-learning system dynamics models interpretable and causally reliable.

desk verdict A clear, well-motivated PhD proposal that describes a plausible integration of interpretability methods, but offers no evidence and under-specifies the causal discovery step on which its central promise rests. read the letter →

arxiv 2505.14428 v1 pith:V3YG222S submitted 2025-05-20 cs.LG cs.AI

Riccardo D'Elia This is my paper · ORCID

classification cs.LGcs.AI

keywords ExplainableArtificialIntelligenceConcept-basedinterpretabilityMechanisticCausalmachinelearningSystemdynamicsreliabilityAutonomousmultimodaltransportNeuro-symbolicAI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This proposal argues that the opacity and causal unreliability of deep learning models can be overcome, at least for system dynamics problems, by replacing black-box predictors with a pipeline that first learns human-meaningful concepts, then learns the causal graph among those concepts, and finally learns interpretable structural equations. The author's claim is that combining concept-based interpretability, causal machine learning, and mechanistic interpretability in one framework yields models that are both scalable, like deep learning, and transparent and causally reliable, like traditional system dynamics. If correct, this would make deep-learning forecasts actionable in safety-critical settings such as autonomous multimodal freight terminals, where operators need to know why a model predicts a delay and what happens under interventions. The paper is a research blueprint rather than a completed system; the validation is planned through the AutoMoTIF project.

What carries the argument

The load-bearing construction is the three-stage pipeline. Stage one, concept learning, uses concept-based interpretability to map raw sensor or operational data onto high-level human-meaningful variables. Stage two, causal learning, applies causal machine learning and causal discovery, including the neural causal models framework, to infer a directed causal graph over those concepts, optionally informed by background knowledge. Stage three, equation learning, uses mechanistic interpretability, with graph-based architectures such as graph neural networks, to derive explicit structural equations describing how the concepts evolve over time. The claim is that the causal graph is the piece that anchors the neural model to real mechanisms; the equations inherit their causal reliability from that graph.

What would settle it

Run the pipeline on a synthetic dynamical system whose true causal graph and structural equations are known, such as a benchmark epidemic or multimodal-terminal model, and compare the learned causal graph and equations against that ground truth; if the recovered graph omits a known edge, adds a spurious one, or yields equations that fail to reproduce the system under an intervention not seen in training, the central claim of causal reliability is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that the perceived trade-off between deep learning's predictive power and system dynamics' interpretability is not necessary: a three-step Interpretable Neural System Dynamics pipeline can deliver both. The paper's way of stating this is that semantic opacity, mechanistic opacity, and causal unreliability should be addressed jointly, not separately as they usually are in XAI research. The pipeline's output is a neural model that operates on semantically meaningful variables, embeds a learned causal directed graph, and is governed by explicit structural equations, so predictions can be traced back to concepts and causes, and interventions can be simulated. The author proposes to validate this on intermodal terminal operations in the AutoMoTIF project, where understanding and causal reliability are required for risk assessment and compliance.

Load-bearing premise

The framework stands or falls on the assumption that step two, causal learning, can recover the true causal dependencies among the learned concepts from data and background knowledge; if the learned graph does not match the real mechanisms, the structural equations derived from it will not be causally reliable even if they are interpretable.

Editorial extensions

If this is right

If the pipeline works as proposed, neural system dynamics models could be audited by tracing any prediction to the defining concepts and to the causal influences among them.
Operators in high-risk logistics settings could simulate interventions such as increasing crane availability and see predicted long-term effects instead of receiving an unexplained forecast.
The same architecture could be carried to other critical domains where transparency is mandatory, including healthcare, environmental monitoring, and finance.
Because the equations are explicit, the resulting models may support the kind of documentation and risk assessment that high-risk AI regulation asks for, such as under the EU AI Act.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the author leaves implicit: the causal-learning step may need interventional or counterfactual data, or much stronger background knowledge, because passive observation alone cannot generally disambiguate which of several graphs consistent with the data is the true one.
The proposal could be made falsifiable earlier by first testing the pipeline on a synthetic benchmark with a known causal graph, and only then moving to the AutoMoTIF case; real-world evaluation alone cannot show whether failures come from causal discovery or equation learning.
The temporal aspect of concept learning is flagged as underdeveloped; an implicit consequence is that the pipeline's success may hinge on new methods that make concepts evolve over time, rather than applying static concept classifiers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a doctoral research proposal, not a research paper with results. It describes a three-step pipeline that would combine concept-based interpretability, causal learning, and mechanistic interpretability to build system dynamics models that are both interpretable and causally reliable. There is no implementation, derivation, or data; the central claim is explicitly a working hypothesis. You should read it as a roadmap, not as a contribution to be accepted or rejected.

What the paper does well: it motivates the integration of these three threads more clearly than most XAI position pieces I've seen. The observation that semantic and mechanistic opacity have been treated separately, and that causal reliability is a third orthogonal problem, is fair and worth saying. The AutoMoTIF running example keeps the proposal grounded, and the milestones are realistic for a four-year PhD. The author also manages to avoid overselling existing methods; the limitations of concept-based models are acknowledged.

The soft spots are predictable for a proposal. The load-bearing step is the causal discovery in Section 3.2, where the pipeline promises to 'uncover the underlying causal structure' from data and background knowledge. No identifiability conditions are stated, no intervention plan is described, and there is no criterion for validating the recovered graph against ground truth. This is a serious omission because the equation learning step and the claimed 'causal reliability' both depend on that graph being correct. Observational causal discovery is underdetermined; learned concepts from Step 1 are estimate, so measurement error can confound discovery further. The paper would be stronger if it at least listed the assumptions needed (e.g., causal sufficiency, faithfulness, no hidden confounders) and sketched how they would be tested. This is not a fatal flaw in a proposal, but it is the difference between a research question and a hand-wave. Also, the term 'causal reliability' is borrowed from the literature but never defined formally; a precise definition would help.

On balance, this is an honest, coherent research plan. It does not deserve a full peer review as a research paper because there is no result to verify. If there is a doctoral consortium or a workshop for position papers, it could be reviewed there. I would not cite it in my own work, but I would recommend it to a student looking for a model of how to structure a PhD proposal in the XAI/system dynamics space.

Referee Report

4 major / 5 minor

Summary. The paper proposes a research plan for an 'Interpretable Neural System Dynamics' (INSD) pipeline that combines concept-based interpretability, causal machine learning, and mechanistic interpretability to build neural models of dynamical systems that are interpretable and causally reliable. It motivates the work by contrasting deep learning's predictive power with its opacity and lack of causal reliability, and positions System Dynamics as a source of interpretable, causal structure. The pipeline is to be validated on the AutoMoTIF freight-transportation scenario. The manuscript explicitly frames the project as a doctoral research proposal, stating working hypotheses and listing milestones, with no implementation, experiments, or formal derivations.

Significance. The paper identifies a real and timely research gap: deep learning models for dynamical systems lack semantic and mechanistic interpretability as well as causal reliability, and existing XAI methods address these aspects separately. The proposed integration of concept-based interpretability, causal learning, and mechanistic interpretability is a plausible and potentially valuable research direction, and the AutoMoTIF application provides a concrete and relevant testbed. The strengths of the manuscript are its clear problem statement, structured research questions, and explicit milestones. However, the central claim that the INSD pipeline will deliver causally reliable and interpretable models is only a working hypothesis; the paper contains no derivations, no algorithms, no empirical evidence, and no falsifiable predictions. Any assessment of the pipeline's feasibility, soundness, or expected impact therefore rests entirely on plausibility rather than demonstrated results.

major comments (4)

[§3.2] The central claim that the pipeline will achieve 'causal reliability' depends on the causal discovery step recovering the true causal dependencies among the concepts from data and background knowledge. The paper states that CML and Neural Causal Models will 'uncover the underlying causal structure of a system by learning a graph,' but it provides no identifiability conditions, no discussion of Markov equivalence classes, and no explanation of how background knowledge would resolve graph underdetermination. Without such conditions, the learned graph may be one of many compatible with the data, and the structural equations learned in §3.3 would not be causally reliable.
[§3.1–§3.3] The three pipeline steps are described only at the level of research themes; no concrete algorithms, formal problem statements, or computational procedures are given for concept learning, causal discovery, or equation learning. For instance, §3.3 mentions graph neural networks and 'mechanistic interpretability' but does not specify how structural dynamic equations are represented, how their parameters are estimated, or how the inferred causal graph constrains them. As presented, the pipeline is not implementable or testable, making it impossible to scrutinize the claimed behavior.
[§4] The milestones (M1–M6) list planned deliverables (D1–D6) but no quantitative success criteria, evaluation metrics, baselines, or validation protocols. The paper says the pipeline will be 'validated' on the AutoMoTIF scenario, but it never defines what would count as a successful validation, either for interpretability or for causal reliability. This absence prevents any empirical assessment of the central claim, even in principle.
[§3.2] Causal discovery is performed on concepts that are themselves learned from raw data in Step 1, meaning the discovery algorithm operates on estimated variables rather than ground-truth quantities. The paper does not address how concept-estimation error, measurement noise, or potential information loss in the concept bottleneck could introduce confounding or spurious dependency edges. This is a load-bearing issue because any error in the concept layer propagates into the learned causal graph and the subsequent equations.

minor comments (5)

[Section 1] In the sentence 'the proposed approach focuses on the construction of a interpretable by design neural systems dynamic framework,' 'a interpretable' should be 'an interpretable.'
[Figure 1] The figure caption contains 'Casual Reliability,' which should be 'Causal Reliability.'
[Abstract and Full Text] The phrase 'operating onsemantically meaningful variables' appears without a space between 'on' and 'semantically'; fix the typo.
[Throughout] Section headings are inconsistent in capitalization (e.g., 'Causal Learning' vs 'causal learning'); use a consistent heading style.
[References] Reference [7] is a preprint with a version number; if a peer-reviewed version exists, it should be cited instead of or in addition to the arXiv version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a research proposal whose central claims are asserted rather than derived from fitted inputs or self-citations.

full rationale

The paper presents a doctoral research proposal and does not contain a substantive derivation chain, empirical predictions, or fitted parameters. The INSD pipeline is described as three planned learning steps — concept learning, causal learning, and equation learning — but none of these steps is executed, and no result is claimed to be predicted from a fitted input. The term 'causal reliability' is introduced via citation [2], but only as a definitional label for a known problem, not as a conclusion derived from that citation. Similarly, the reliance on concept-based interpretability, causal machine learning, and mechanistic interpretability is supported by external survey and methodology references that are independent of the proposal's own claims. There is no self-citation chain that forces the conclusion, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The main vulnerability noted in Section 3.2 — that causal discovery must recover true dependencies from data and background knowledge without stated identifiability conditions — is a correctness or feasibility risk, not a circularity. Because the proposal makes no concrete predictive claim that reduces by construction to its own assumptions or inputs, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three untested working hypotheses about the feasibility of concept learning, causal discovery, and equation learning. The paper provides no free parameters or invented entities because it contains no fitted models or new postulates.

assumptions (3)

domain assumption Concepts can be learned from raw data in a semantically meaningful way
Section 3.1 relies on concept-based interpretability to extract high-level variables, but no evidence is given that this is feasible for the intended system dynamics tasks.
domain assumption Causal discovery from data and background knowledge can recover the true causal graph
Section 3.2 assumes that causal learning techniques can identify true causal dependencies, which is a known hard problem with no general solution.
domain assumption Mechanistic interpretability can derive explicit structural equations from neural networks
Section 3.3 assumes that mechanistic interpretability can yield interpretable dynamic equations, but no technique is specified that guarantees this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Interpretable Neural System Dynamics: Combining Deep Learning with System Dynamics Modeling to Support Critical Applications." pith.science (2026). https://pith.science/paper/V3YG222S

@misc{pith2026250514428,
  author       = {Pith},
  title        = {Pith review of: Interpretable Neural System Dynamics: Combining Deep Learning with System Dynamics Modeling to Support Critical Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3YG222S}},
  note         = {Machine review of arXiv:2505.14428}
}

read the original abstract

The objective of this proposal is to bridge the gap between Deep Learning (DL) and System Dynamics (SD) by developing an interpretable neural system dynamics framework. While DL excels at learning complex models and making accurate predictions, it lacks interpretability and causal reliability. Traditional SD approaches, on the other hand, provide transparency and causal insights but are limited in scalability and require extensive domain knowledge. To overcome these limitations, this project introduces a Neural System Dynamics pipeline, integrating Concept-Based Interpretability, Mechanistic Interpretability, and Causal Machine Learning. This framework combines the predictive power of DL with the interpretability of traditional SD models, resulting in both causal reliability and scalability. The efficacy of the proposed pipeline will be validated through real-world applications of the EU-funded AutoMoTIF project, which is focused on autonomous multimodal transportation systems. The long-term goal is to collect actionable insights that support the integration of explainability and safety in autonomous systems.

Figures

Figures reproduced from arXiv: 2505.14428 by the authors.

**Figure 1.** This project offers a new approach to jointly address the challenges of interpretability and causal reliability posed by the widespread use of DL methods in the modeling of dynamical systems. Rather than focusing on the development and implementation of post-hoc interpretability techniques, the proposed approach focuses on the construction of a interpretable by design neural systems dynamic framework. A plethora of … view at source ↗

**Figure 2.** Overview of the INSD pipeline, from concept learning to causal and equation learning, ensuring interpretability and causal reliability in the resulting model. epidemiological modeling, deep learning algorithms have the potential to discern underlying patterns of disease transmission; however, they have difficulty formulating these patterns using conventional epidemiological factors, such as contact rate or incubatio… view at source ↗

**Figure 3.** Gantt diagram of the doctoral project. Expected Contribution and Impact. Through the development of a unified interpretability framework for DL-based System Dynamics models, this research aspires to bridge the current divide between theoretical advancements in eXplainable AI and their application in high-stakes, real-world environments. By bringing together causal, mechanistic, and concept-based perspectives withi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

[1]

Quaranta, W

G. Quaranta, W. Lacarbonara, S. F. Masri, A review on computational intelligence for identification of nonlinear dynamical systems, Nonlinear Dynamics 99 (2020) 1709–1761. URL: http://link. springer.com/10.1007/s11071-019-05430-7. doi:10.1007/s11071-019-05430-7

doi:10.1007/s11071-019-05430-7 2020
[2]

Termine, G

A. Termine, G. Primiero, Causality Problems in Machine Learning Systems, in: The Routledge Handbook of Causality and Causal Methods, 1 ed., Routledge, New York, 2024, pp. 325–341. URL: https://www.taylorfrancis.com/books/9781003528937/chapters/10.4324/9781003528937-37. doi:10.4324/9781003528937-37

arXiv 2024
[3]

Flammini, C

F. Flammini, C. Alcaraz, E. Bellini, S. Marrone, J. Lopez, A. Bondavalli, Towards Trustworthy Autonomous Systems: Taxonomies and Future Perspectives, IEEE Transactions on Emerging Topics in Computing 12 (2024) 601–614. URL: https://ieeexplore.ieee.org/document/9979717/. doi:10.1109/TETC.2022.3227113

work page arXiv 2024
[4]

B. P. Bhuyan, A. Ramdane-Cherif, R. Tomar, T. P. Singh, Neuro-symbolic artificial intelligence: a survey, Neural Computing and Applications 36 (2024) 12809–12844. URL: https://link.springer. com/10.1007/s00521-024-09960-z. doi: 10.1007/s00521-024-09960-z

doi:10.1007/s00521-024-09960-z 2024
[5]

Poeta, G

E. Poeta, G. Ciravegna, E. Pastor, T. Cerquitelli, E. Baralis, Concept-based Explainable Artificial Intelligence: A Survey, 2023. URL: http://arxiv.org/abs/2312.12936. doi:10.48550/arXiv.2312. 12936, arXiv:2312.12936 [cs]

doi:10.48550/arxiv.2312.12936 2023
[6]

Bereska, E

L. Bereska, E. Gavves, Mechanistic Interpretability for AI Safety – A Review, 2024. URL: http: //arxiv.org/abs/2404.14082. doi:10.48550/arXiv.2404.14082, arXiv:2404.14082 [cs]

arXiv doi:10.48550/arxiv.2404.14082 2024
[7]

Kaddour, A

J. Kaddour, A. Lynch, Q. Liu, M. J. Kusner, R. Silva, Causal Machine Learning: A Survey and Open Problems, 2022. URL: https://arxiv.org/abs/2206.15475. doi:10.48550/ARXIV.2206.15475, version Number: 2

arXiv doi:10.48550/arxiv.2206.15475 2022
[8]

Bellogín, O

A. Bellogín, O. Grau, S. Larsson, G. Schimpf, B. Sengupta, G. Solmaz, The EU AI Act and the Wager on Trustworthy AI, Communications of the ACM 67 (2024) 58–65. URL: https://dl.acm.org/doi/10. 1145/3665322. doi:10.1145/3665322

doi:10.1145/3665322 2024

Show all 16 references

[9]

Borrelli, S

M. Borrelli, S. Musch, S. Khan, Applying the EU AI Act to the Automotive Industry: Ensuring Explainability and Transparency, SSRN Electronic Journal (2024). URL: https://www.ssrn.com/ abstract=4819226. doi:10.2139/ssrn.4819226

2024 doi
[10]

Sokol, P

K. Sokol, P. Flach, Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence, 2022. URL: http://arxiv.org/abs/2112.14466. doi:10.48550/ arXiv.2112.14466, arXiv:2112.14466 [cs]

2022 doi arXiv
[11]

Facchini, A

A. Facchini, A. Termine, Towards a Taxonomy for the Opacity of AI Systems, in: V. C. Müller (Ed.), Philosophy and Theory of Artificial Intelligence 2021, volume 63, Springer International Publishing, Cham, 2022, pp. 73–89. URL: https://link.springer.com/10.1007/978-3-031-09153...

2021 doi
[12]

Kästner, B

L. Kästner, B. Crook, Explaining AI through mechanistic interpretability, European Journal for Philosophy of Science 14 (2024) 52. URL: https://link.springer.com/10.1007/s13194-024-00614-4. doi:10.1007/s13194-024-00614-4

2024 doi
[13]

Scholkopf, F

B. Scholkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, Y. Bengio, Toward Causal Representation Learning, Proceedings of the IEEE 109 (2021) 612–634. URL: https://ieeexplore. ieee.org/document/9363924/. doi:10.1109/JPROC.2021.3058954

2021
[14]

Pearl, Causality: Models, Reasoning, and Inference, 2 ed., Cambridge University Press,

J. Pearl, Causality: Models, Reasoning, and Inference, 2 ed., Cambridge University Press,
[15]

Carloni, A

G. Carloni, A. Berti, S. Colantonio, The role of causality in explainable artificial intelligence, 2023. URL: http://arxiv.org/abs/2309.09901. doi:10.48550/arXiv.2309.09901, arXiv:2309.09901 [cs]

2023 doi arXiv
[2009]

URL: https://www.cambridge.org/core/product/identifier/9780511803161/type/book. doi:10. 1017/CBO9780511803161

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Quaranta, W

G. Quaranta, W. Lacarbonara, S. F. Masri, A review on computational intelligence for identification of nonlinear dynamical systems, Nonlinear Dynamics 99 (2020) 1709–1761. URL: http://link. springer.com/10.1007/s11071-019-05430-7. doi:10.1007/s11071-019-05430-7

doi:10.1007/s11071-019-05430-7 2020

[2] [2]

Termine, G

A. Termine, G. Primiero, Causality Problems in Machine Learning Systems, in: The Routledge Handbook of Causality and Causal Methods, 1 ed., Routledge, New York, 2024, pp. 325–341. URL: https://www.taylorfrancis.com/books/9781003528937/chapters/10.4324/9781003528937-37. doi:10.4324/9781003528937-37

arXiv 2024

[3] [3]

Flammini, C

F. Flammini, C. Alcaraz, E. Bellini, S. Marrone, J. Lopez, A. Bondavalli, Towards Trustworthy Autonomous Systems: Taxonomies and Future Perspectives, IEEE Transactions on Emerging Topics in Computing 12 (2024) 601–614. URL: https://ieeexplore.ieee.org/document/9979717/. doi:10.1109/TETC.2022.3227113

work page arXiv 2024

[4] [4]

B. P. Bhuyan, A. Ramdane-Cherif, R. Tomar, T. P. Singh, Neuro-symbolic artificial intelligence: a survey, Neural Computing and Applications 36 (2024) 12809–12844. URL: https://link.springer. com/10.1007/s00521-024-09960-z. doi: 10.1007/s00521-024-09960-z

doi:10.1007/s00521-024-09960-z 2024

[5] [5]

Poeta, G

E. Poeta, G. Ciravegna, E. Pastor, T. Cerquitelli, E. Baralis, Concept-based Explainable Artificial Intelligence: A Survey, 2023. URL: http://arxiv.org/abs/2312.12936. doi:10.48550/arXiv.2312. 12936, arXiv:2312.12936 [cs]

doi:10.48550/arxiv.2312.12936 2023

[6] [6]

Bereska, E

L. Bereska, E. Gavves, Mechanistic Interpretability for AI Safety – A Review, 2024. URL: http: //arxiv.org/abs/2404.14082. doi:10.48550/arXiv.2404.14082, arXiv:2404.14082 [cs]

arXiv doi:10.48550/arxiv.2404.14082 2024

[7] [7]

Kaddour, A

J. Kaddour, A. Lynch, Q. Liu, M. J. Kusner, R. Silva, Causal Machine Learning: A Survey and Open Problems, 2022. URL: https://arxiv.org/abs/2206.15475. doi:10.48550/ARXIV.2206.15475, version Number: 2

arXiv doi:10.48550/arxiv.2206.15475 2022

[8] [8]

Bellogín, O

A. Bellogín, O. Grau, S. Larsson, G. Schimpf, B. Sengupta, G. Solmaz, The EU AI Act and the Wager on Trustworthy AI, Communications of the ACM 67 (2024) 58–65. URL: https://dl.acm.org/doi/10. 1145/3665322. doi:10.1145/3665322

doi:10.1145/3665322 2024