pith. sign in

arxiv: 1907.04708 · v1 · pith:3273XN7Lnew · submitted 2019-07-10 · 💻 cs.LG · stat.ML

Learning a Behavior Model of Hybrid Systems Through Combining Model-Based Testing and Machine Learning (Full Version)

Bernhard K. Aichernig , Roderick Bloem , Masoud Ebrahimi , Martin Horn , Franz Pernkopf , Wolfgang Roth , Astrid Rupp , Martin Tappler
show 1 more author
Markus Tranninger
This is my paper

Pith reviewed 2026-05-24 23:52 UTC · model grok-4.3

classification 💻 cs.LG stat.ML
keywords hybrid systemsmodel-based testingautomata learningrecurrent neural networksplatooningcrash detectioncyber-physical systemsmachine learning
0
0 comments X

The pith

Recurrent neural networks trained on data from model-based testing and automata learning detect crashes in hybrid platooning systems with five times lower error using up to a thousand times fewer samples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to generate training data automatically for machine learning models of hybrid systems by combining automata learning with model-based testing. This solves the problem of needing enough representative data to capture both physical and digital behaviors in cyber-physical systems. In platooning experiments, recurrent neural networks trained on the generated data reduce crash detection classification error by a factor of five compared to random data. They also reach similar F1-scores with up to three orders of magnitude fewer training samples. A sympathetic reader would care because manual model building for such systems is hard and this automation could speed up early design checks.

Core claim

Combining automata learning and model-based testing produces training data that is sufficient and representative of hybrid system behavior, allowing recurrent neural networks to learn accurate models as evidenced by fivefold lower classification error and comparable F1-scores with far fewer samples than random data in a platooning scenario.

What carries the argument

The combination of automata learning and model-based testing that automatically generates representative training data for recurrent neural network models of hybrid systems.

If this is right

  • Behavior models for hybrid systems can be learned automatically without manual behavior specification.
  • Machine learning for cyber-physical systems can use structured test data to achieve lower classification error.
  • Comparable F1-scores for crash detection are possible with up to three orders of magnitude fewer training samples.
  • The method provides an automated way to support simulation and analysis in the design of cyber-physical systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The data generation approach could extend to other hybrid systems involving vehicle coordination or control loops.
  • Learned models might be combined with existing verification methods to check safety properties of the original system.
  • This technique may lower the overall cost of training machine learning components for physical-digital systems by reducing data requirements.
  • Application to additional scenarios could reveal whether the performance gains hold when the underlying hybrid dynamics change.

Load-bearing premise

The model-based testing procedure combined with automata learning produces training data that is both sufficient and representative of the hybrid system's full behavior.

What would settle it

A recurrent neural network trained on the generated data would misclassify crash events in the actual platooning system under input conditions absent from the test data.

Figures

Figures reproduced from arXiv: 1907.04708 by Astrid Rupp, Bernhard K. Aichernig, Franz Pernkopf, Markus Tranninger, Martin Horn, Martin Tappler, Masoud Ebrahimi, Roderick Bloem, Wolfgang Roth.

Figure 1
Figure 1. Figure 1: Learning a behavior model of a black-box hybrid system. In this paper, we combine automata learning and Model-Based Testing (MBT) to derive an adequate training set, and then use machine learning to learn a be￾havior model from a black-box hybrid sys￾tem. We can use the learned behavior model for multiple purposes such as mon￾itoring runtime behavior. Furthermore, it could be used as a surrogate of a compl… view at source ↗
Figure 2
Figure 2. Figure 2: Platooning as distributed control scenario. Adapted from a figure in [9]. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Abstract automata learning through a mapper [36]. Learning and Model-Based Testing. Teach￾ers are usually implemented via testing to learn models of black-box systems, The teacher in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Components involved in the testing process [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Recurrent neural network. The output yi of the RNN at time step i does not only depend on the input xi at time step i, but also on the accumulated knowledge in the hidden state vector hi−1 at the previous time step i − 1. from Section 3.1.1, i.e., the acceleration value acc and the orientation ∆ of the leader car in radians. We preprocess the orientation ∆ to contain the angular difference of orientation i… view at source ↗
Figure 6
Figure 6. Figure 6: Performance measures for all testing strategies over changing [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CDF plots for the difference between true crash time and predicted crash [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
read the original abstract

Models play an essential role in the design process of cyber-physical systems. They form the basis for simulation and analysis and help in identifying design problems as early as possible. However, the construction of models that comprise physical and digital behavior is challenging. Therefore, there is considerable interest in learning such hybrid behavior by means of machine learning which requires sufficient and representative training data covering the behavior of the physical system adequately. In this work, we exploit a combination of automata learning and model-based testing to generate sufficient training data fully automatically. Experimental results on a platooning scenario show that recurrent neural networks learned with this data achieved significantly better results compared to models learned from randomly generated data. In particular, the classification error for crash detection is reduced by a factor of five and a similar F1-score is obtained with up to three orders of magnitude fewer training samples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript proposes combining automata learning with model-based testing to automatically generate training data for recurrent neural networks that model the hybrid behavior of cyber-physical systems. Experiments on a platooning scenario demonstrate that RNNs trained on data from this pipeline achieve a five-fold reduction in crash-detection classification error and comparable F1-scores using up to three orders of magnitude fewer samples than models trained on randomly generated data.

Significance. If the reported performance delta holds under the described protocol, the work supplies a practical, automated solution to the data-sufficiency problem for ML on hybrid systems. The concrete quantitative comparison (error reduction and sample efficiency) in a safety-relevant scenario constitutes a clear empirical contribution that could be directly useful for model-based design of autonomous systems.

minor comments (3)
  1. [§4] §4 (Evaluation): the description of the random-generation baseline could be expanded with the exact sampling distribution and any rejection criteria used, to support exact reproduction of the reported factor-of-five gap.
  2. [Abstract and §4] The abstract states the key quantitative claims; the body supplies the supporting protocol, but a short table summarizing the RNN hyperparameters and data-set sizes across both conditions would improve readability.
  3. [Figures in §4] Figure captions and axis labels in the experimental plots should explicitly state the number of independent runs and whether error bars represent standard deviation or standard error.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive review and the recommendation to accept the manuscript.

Circularity Check

0 steps flagged

No circularity: empirical comparison of data-generation strategies

full rationale

The paper describes an empirical pipeline that combines automata learning with model-based testing to produce training data for recurrent neural networks, then evaluates performance on a platooning scenario against a random-generation baseline. All central claims (5× lower crash-detection error, comparable F1-score with far fewer samples) are instantiated by concrete experimental metrics that lie outside the generation method itself. No equations, fitted parameters, or predictions reduce to their inputs by construction; no self-citation chain supplies a load-bearing uniqueness theorem or ansatz; the work contains no derivations at all. The representativeness assumption is therefore falsifiable by the reported performance delta rather than presupposed.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the unstated assumption that the generated dataset adequately covers hybrid dynamics.

pith-pipeline@v0.9.0 · 5708 in / 1113 out tokens · 15938 ms · 2026-05-24T23:52:16.404955+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 1 internal anchor

  1. [1]

    In: FM (2012)

    Aarts, F., Heidarian, F., Kuppens, H., Olsen, P., Vaandrager, F.W.: Automata learning through counterexample guided abstraction refinement. In: FM (2012)

    work page 2012

  2. [2]

    In: Bennaceur, A., H¨ ahnle, R., Meinke, K

    Aichernig, B.K., Mostowski, W., Mousavi, M.R., Tappler, M., Taromirad, M.: Model learning and model-based testing. In: Bennaceur, A., H¨ ahnle, R., Meinke, K. (eds.) Machine Learning for Dynamic Software Analysis: Potentials and Limits - International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers. Lecture Notes in Com...

    work page doi:10.1007/978-3-319-96562-8_3 2016

  3. [3]

    Journal of Automated Reasoning (Oct 2018)

    Aichernig, B.K., Tappler, M.: Efficient active automata learning via mutation test- ing. Journal of Automated Reasoning (Oct 2018)

    work page 2018

  4. [4]

    Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. (1987)

    work page 1987

  5. [5]

    BMC Bioinformatics 15, S4 (2014)

    Cangelosi, D., Muselli, M., Parodi, S., Blengio, F., Becherini, P., Versteeg, R., Conte, M., Varesio, L.: Use of attribute driven incremental discretization and logic learning machine to build a prognostic classifier for neuroblastoma patients. BMC Bioinformatics 15, S4 (2014)

    work page 2014

  6. [6]

    https://keras.io (2015) Learning a Behavior Model of Hybrid Systems 19

    Chollet, F., et al.: Keras. https://keras.io (2015) Learning a Behavior Model of Hybrid Systems 19

    work page 2015

  7. [7]

    IEEE Trans- actions on Software Engineering 4(3), 178–187 (May 1978)

    Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans- actions on Software Engineering 4(3), 178–187 (May 1978)

    work page 1978

  8. [8]

    Proceedings of the IEEE 100(1), 13–28 (2012)

    Derler, P., Lee, E.A., Sangiovanni-Vincentelli, A.L.: Modeling cyber-physical sys- tems. Proceedings of the IEEE 100(1), 13–28 (2012). , https://doi.org/10.1109/ JPROC.2011.2160929

    work page arXiv 2012

  9. [9]

    IEEE Transactions on Intelligent Transportation Sys- tems 18(12), 3486–3500 (Dec 2017)

    Dolk, V.S., Ploeg, J., Heemels, W.P.M.H.: Event-triggered control for string- stable vehicle platooning. IEEE Transactions on Intelligent Transportation Sys- tems 18(12), 3486–3500 (Dec 2017)

    work page 2017

  10. [10]

    In: Balcan, M., Weinberger, K.Q

    Duan, Y., Chen, X., Houthooft, R., Schulman, J., Abbeel, P.: Benchmarking deep reinforcement learning for continuous control. In: Balcan, M., Weinberger, K.Q. (eds.) ICML 2016. JMLR Workshop and Conference Proceedings, vol. 48, pp. 1329–1338. JMLR.org (2016), http://jmlr.org/proceedings/papers/v48/ duan16.html

    work page 2016

  11. [11]

    In: WFCS (2018)

    Fermi, A., Mongelli, M., Muselli, M., Ferrari, E.: Identification of safety regions in vehicle platooning via machine learning. In: WFCS (2018)

    work page 2018

  12. [12]

    IEEE Transactions on Software Engineering 17(6), 591–603 (1991)

    Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Transactions on Software Engineering 17(6), 591–603 (1991)

    work page 1991

  13. [13]

    In: LICS (1996)

    Henzinger, T.A.: The theory of hybrid automata. In: LICS (1996)

    work page 1996

  14. [14]

    Neural Computation 9(8), 1735–1780 (1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)

    work page 1997

  15. [15]

    In: Machine Learning for Dynamic Software Analysis: Potentials and Limits - International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers

    Howar, F., Steffen, B.: Active automata learning in practice - an annotated bib- liography of the years 2011 to 2016. In: Machine Learning for Dynamic Software Analysis: Potentials and Limits - International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers. pp. 123–148 (2018)

    work page 2011

  16. [16]

    In: ISoLA

    Howar, F., Steffen, B., Merten, M.: From ZULU to RERS - lessons learned in the ZULU challenge. In: ISoLA. pp. 687–704 (2010)

    work page 2010

  17. [17]

    Isberner, M., Howar, F., Steffen, B.: The open-source LearnLib - A framework for active automata learning. In: CAV. pp. 487–495 (2015)

    work page 2015

  18. [18]

    MIT Press, Cambridge, MA, USA (1994)

    Kearns, M.J., Vazirani, U.V.: An Introduction to Computational Learning Theory. MIT Press, Cambridge, MA, USA (1994)

    work page 1994

  19. [19]

    Adam: A Method for Stochastic Optimization

    Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015), arXiv: 1412.6980

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  20. [20]

    Prentice Hall, New Jersey (1999)

    Ljung, L.: System Identification: Theory for the User, PTR Prentice Hall Informa- tion and System Sciences Series. Prentice Hall, New Jersey (1999)

    work page 1999

  21. [21]

    IEEE Transactions on Cybernetics 48(8), 2357–2367 (Aug 2018)

    Lv, C., Liu, Y., Hu, X., Guo, H., Cao, D., Wang, F.: Simultaneous observation of hybrid states for cyber-physical systems: A case study of electric vehicle powertrain. IEEE Transactions on Cybernetics 48(8), 2357–2367 (Aug 2018)

    work page 2018

  22. [22]

    In: Hybrid Systems (1992)

    Manna, Z., Pnueli, A.: Verifying hybrid systems. In: Hybrid Systems (1992)

    work page 1992

  23. [23]

    In: EPEW (2017)

    Meinke, K.: Learning-based testing of cyber-physical systems-of-systems: A pla- tooning study. In: EPEW (2017)

    work page 2017

  24. [24]

    In: Ben- naceur, A., H¨ ahnle, R., Meinke, K

    Meinke, K.: Learning-based testing: Recent progress and future prospects. In: Ben- naceur, A., H¨ ahnle, R., Meinke, K. (eds.) Machine Learning for Dynamic Soft- ware Analysis: Potentials and Limits - International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers. Lecture Notes in Computer Science, vol. 11026, pp. 53–73....

    work page 2016

  25. [25]

    IEEE Trans

    O’Shea, T.J., Hoydis, J.: An introduction to deep learning for the physical layer. IEEE Trans. Cogn. Comm. & Networking 3(4), 563–575 (2017). , https://doi. org/10.1109/TCCN.2017.2758370 20 B. K. Aichernig et al

    work page doi:10.1109/tccn.2017.2758370 2017

  26. [26]

    Journal of Au- tomata, Languages and Combinatorics 7(2), 225–246 (2002)

    Peled, D.A., Vardi, M.Y., Yannakakis, M.: Black box checking. Journal of Au- tomata, Languages and Combinatorics 7(2), 225–246 (2002). , https://doi.org/ 10.25596/jalc-2002-225

    work page doi:10.25596/jalc-2002-225 2002

  27. [27]

    IEEE Transactions on Intelligent Transporta- tion Systems 15(2), 854–865 (April 2014)

    Ploeg, J., Shukla, D.P., van de Wouw, N., Nijmeijer, H.: Controller synthesis for string stability of vehicle platoons. IEEE Transactions on Intelligent Transporta- tion Systems 15(2), 854–865 (April 2014)

    work page 2014

  28. [28]

    In: IEEE Inter- national Conference on Robotics and Automation, ICRA 2015, Seattle, WA, USA, 26-30 May, 2015

    Punjani, A., Abbeel, P.: Deep learning helicopter dynamics models. In: IEEE Inter- national Conference on Robotics and Automation, ICRA 2015, Seattle, WA, USA, 26-30 May, 2015. pp. 3223–3230. IEEE (2015). , https://doi.org/10.1109/ICRA. 2015.7139643

    work page doi:10.1109/icra 2015

  29. [29]

    In: SEFM (2018)

    Rashid, A., Siddique, U., Hasan, O.: Formal verification of platoon control strate- gies. In: SEFM (2018)

    work page 2018

  30. [30]

    IEEE Control Systems Letters 1(2), 274–279 (Oct 2017)

    Rupp, A., Steinberger, M., Horn, M.: Sliding mode based platooning with non-zero initial spacing errors. IEEE Control Systems Letters 1(2), 274–279 (Oct 2017)

    work page 2017

  31. [31]

    In: FM (2009)

    Shahbaz, M., Groz, R.: Inferring Mealy machines. In: FM (2009)

    work page 2009

  32. [32]

    Structural and Multidisciplinary Optimization 27(5), 302–313 (2004)

    Simpson, T., Booker, A., Ghosh, D., Giunta, A., Koch, P., Yang, R.J.: Ap- proximation methods in multidisciplinary analysis and optimization: a panel dis- cussion. Structural and Multidisciplinary Optimization 27(5), 302–313 (2004). , https://doi.org/10.1007/s00158-004-0389-9

    work page doi:10.1007/s00158-004-0389-9 2004

  33. [33]

    In: ICFEM (2015)

    Smeenk, W., Moerman, J., Vaandrager, F.W., Jansen, D.N.: Applying automata learning to embedded control software. In: ICFEM (2015)

    work page 2015

  34. [34]

    2017 6th International Symposium on Advanced Con- trol of Industrial Processes (AdCONIP) pp

    Spielberg, S., Gopaluni, R.B., Loewen, P.D.: Deep reinforcement learning ap- proaches for process control. 2017 6th International Symposium on Advanced Con- trol of Industrial Processes (AdCONIP) pp. 201–206 (2017)

    work page 2017

  35. [35]

    IEEE Transactions on Automatic Control 58(4), 891–904 (apr 2013)

    Tanwani, A., Shim, H., Liberzon, D.: Observability for switched linear systems: Characterization and observer design. IEEE Transactions on Automatic Control 58(4), 891–904 (apr 2013)

    work page 2013

  36. [36]

    Vaandrager, F.W.: Model learning. Commun. ACM (2017)

    work page 2017

  37. [37]

    Cybernetics 9(4), 653–665 (1973)

    Vasilevskii, M.P.: Failure diagnosis of automata. Cybernetics 9(4), 653–665 (1973)

    work page 1973

  38. [38]

    In: Interdisciplinary Applied Mathematics, pp

    Vidal, R., Ma, Y., Sastry, S.S.: Hybrid system identification. In: Interdisciplinary Applied Mathematics, pp. 431–451. Springer New York (2016)

    work page 2016