pith. sign in

arxiv: 1906.09179 · v1 · pith:2O623JKBnew · submitted 2019-06-21 · 💻 cs.SE

Challenges for Verifying and Validating Scientific Software in Computational Materials Science

Thomas Vogel , Stephan Druskat , Markus Scheidgen , Claudia Draxl , Lars Grunske This is my paper

Pith reviewed 2026-05-25 18:35 UTC · model grok-4.3

classification 💻 cs.SE
keywords scientific softwareverification and validationcomputational materials sciencequality assuranceresearch software engineeringsoftware challengessimulation reliability
0
0 comments X

The pith

Researchers must address specific verification and validation challenges to trust results from computational materials science software.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that many scientific fields depend on software to produce reliable answers, so quality assurance through verification and validation is essential to avoid invalid research outcomes. Drawing on their experience in computational materials science, the authors identify concrete challenges in checking both the software itself and the results it generates. A sympathetic reader would care because flawed software could undermine conclusions in any domain that relies on simulations or computations. The work also sketches directions for future research aimed at overcoming those challenges. This is a review-style contribution that organizes practical obstacles rather than proving a new theorem or method.

Core claim

For valid results researchers need to trust the results scientific software produces, and consequently quality assurance is of utmost importance. The authors formulate challenges for validation and verification of scientific software and their results in the domain of computational materials science, based on their experience, and describe directions for future research that can potentially help dealing with these challenges.

What carries the argument

The set of challenges for validation and verification of scientific software and their results, derived from domain experience in computational materials science.

If this is right

  • Quality assurance practices must be treated as a core requirement when developing or using scientific software.
  • Specific obstacles in verifying both code and outputs need targeted solutions to maintain trust in computed results.
  • Future research should prioritize methods that address the identified validation and verification challenges.
  • Without progress on these challenges, results from computational studies risk remaining difficult to validate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same verification obstacles could appear in other simulation-heavy fields such as climate modeling or bioinformatics.
  • Standard software engineering techniques might need adaptation to handle the unique numerical and physical constraints of materials simulations.
  • Improved documentation and testing protocols could reduce the impact of these challenges on everyday research workflows.

Load-bearing premise

The challenges observed in computational materials science are representative and generalizable to scientific software in other fields.

What would settle it

A broad survey of verification practices across multiple scientific domains that finds no shared challenges would show the listed issues are not general.

Figures

Figures reproduced from arXiv: 1906.09179 by Claudia Draxl, Lars Grunske, Markus Scheidgen, Stephan Druskat, Thomas Vogel.

Figure 1
Figure 1. Figure 1: Architectural view of NOMAD. individual limitations and trade-offs. Second, codes focus on different aspects and produce different physical properties of a simulated material. For instance, a code may specialize in electrical, optical, or thermal properties. Third, data is provided in different unit systems (e.g., International System Units (SI) or atomic units). Fourth, although most codes use a text form… view at source ↗
read the original abstract

Many fields of science rely on software systems to answer different research questions. For valid results researchers need to trust the results scientific software produces, and consequently quality assurance is of utmost importance. In this paper we are investigating the impact of quality assurance in the domain of computational materials science (CMS). Based on our experience in this domain we formulate challenges for validation and verification of scientific software and their results. Furthermore, we describe directions for future research that can potentially help dealing with these challenges.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that quality assurance is of utmost importance for scientific software in computational materials science (CMS) because researchers need to trust the results it produces. Based solely on the authors' experience in the domain, it formulates challenges for the validation and verification of such software and results, and outlines directions for future research to address them.

Significance. If the formulated challenges accurately reflect common issues in CMS, the paper could usefully direct attention and research effort toward improving V&V practices in a domain where computational results underpin materials discoveries. Its contribution is primarily as an experience report that raises domain-specific awareness rather than through new empirical data, formal analysis, or validated prevalence claims.

major comments (2)
  1. [Abstract and challenges formulation section] The central claim rests on the authors' experience in CMS, yet the manuscript provides no details on the scope, number of projects, or specific software systems involved (e.g., in the section describing the challenges). This makes it difficult to assess whether the listed challenges are representative or generalizable, which is load-bearing for the paper's contribution as a domain-specific formulation.
  2. [Challenges section] No empirical data, case studies, or references to documented V&V failures in CMS are supplied to support the prevalence or importance of the identified challenges. The claims therefore rest entirely on unverified author experience, weakening the ability to evaluate their significance relative to other potential issues.
minor comments (2)
  1. [Abstract] The abstract and introduction could more explicitly distinguish between verification (of the software) and validation (of the results) with brief examples from CMS to aid readers unfamiliar with the distinction.
  2. [Introduction or related work] Related work on V&V in scientific computing (e.g., from other domains such as climate modeling or astrophysics) is not referenced, which would help situate the CMS-specific challenges.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful review and the recommendation of minor revision. The manuscript is framed as an experience report drawing on the authors' domain expertise in computational materials science. We address each major comment below and indicate where revisions will be made.

read point-by-point responses

  1. Referee: [Abstract and challenges formulation section] The central claim rests on the authors' experience in CMS, yet the manuscript provides no details on the scope, number of projects, or specific software systems involved (e.g., in the section describing the challenges). This makes it difficult to assess whether the listed challenges are representative or generalizable, which is load-bearing for the paper's contribution as a domain-specific formulation.

    Authors: We agree that providing additional context on the authors' experience would assist readers in evaluating the basis for the formulated challenges. In the revised manuscript we will add a concise description of the authors' collective backgrounds and the classes of CMS projects and software systems (e.g., density-functional theory codes, molecular-dynamics packages, and workflow tools) that informed the work. Because the paper is explicitly an experience report rather than a survey or empirical study, we will not claim statistical representativeness or broad generalizability. revision: partial

  2. Referee: [Challenges section] No empirical data, case studies, or references to documented V&V failures in CMS are supplied to support the prevalence or importance of the identified challenges. The claims therefore rest entirely on unverified author experience, weakening the ability to evaluate their significance relative to other potential issues.

    Authors: The abstract and introduction already state that the challenges are derived from the authors' experience rather than from new empirical data or case studies; this is the intended scope of the contribution. We can incorporate additional references to published discussions of V&V issues in scientific computing more generally. However, the relative scarcity of publicly documented, CMS-specific failure cases is itself a characteristic of the domain and not something the current manuscript can remedy. We therefore maintain that the experience-based formulation remains a legitimate way to surface issues for future research attention. revision: partial

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a descriptive experience report that formulates V&V challenges in computational materials science based on the authors' domain work. It contains no equations, derivations, parameter fitting, predictions, or uniqueness theorems. The central claim—that the authors formulate challenges and suggest future directions—rests solely on stated experience and does not reduce to any self-referential construction, self-citation chain, or renamed input. No load-bearing steps match the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper does not rely on fitted parameters, mathematical axioms, or new postulated entities; it is an experience-based discussion of software engineering challenges.

pith-pipeline@v0.9.0 · 5608 in / 873 out tokens · 27097 ms · 2026-05-25T18:35:04.893594+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 63 canonical work pages

  1. [1]

    J. C. Carver, N. P. C. Hong, and G. K. Thiruvathukal, Software Engineering for Science . CRC Press, 2016

    work page 2016

  2. [2]

    Dealing with risk in scientific software development,

    R. Sanders and D. Kelly, “Dealing with risk in scientific software development,” IEEE Software, vol. 25, no. 4, pp. 21–28, 2008

    work page 2008

  3. [3]

    A scientist’s nightmare: Software problem leads to five retractions,

    G. Miller, “A scientist’s nightmare: Software problem leads to five retractions,” Science, vol. 314, no. 5807, pp. 1856–1857, 2006

    work page 2006

  4. [4]

    How do scientists develop and use scientific software?

    J. E. Hannay, C. MacLeod, J. Singer, H. P. Langtangen, D. Pfahl, and G. Wilson, “How do scientists develop and use scientific software?” in ICSE Workshop on Software Engineering for Computational Science and Engineering, ser. SECSE. IEEE, 2009, pp. 1–8

    work page 2009

  5. [5]

    A survey of scien- tific software development,

    L. Nguyen-Hoan, S. Flint, and R. Sankaranarayana, “A survey of scien- tific software development,” in International Symposium on Empirical Software Engineering and Measurement . ACM, 2010, pp. 12:1–12:10

    work page 2010

  6. [6]

    Claims about the use of software engineer- ing practices in science: A systematic literature review,

    D. Heaton and J. C. Carver, “Claims about the use of software engineer- ing practices in science: A systematic literature review,” Information & Software Technology, vol. 67, pp. 207–219, 2015

    work page 2015

  7. [7]

    Bridging the chasm: A survey of software engineering practice in scientific programming,

    T. Storer, “Bridging the chasm: A survey of software engineering practice in scientific programming,” ACM Comput. Surv., vol. 50, no. 4, pp. 47:1–47:32, 2017

    work page 2017

  8. [8]

    Software engineering for com- putational science: Past, present, future,

    A. N. Johanson and W. Hasselbring, “Software engineering for com- putational science: Past, present, future,” Computing in Science and Engineering, vol. 20, no. 2, pp. 90–109, 2018

    work page 2018

  9. [9]

    Five recommended practices for computational scientists who write software,

    D. Kelly, D. Hook, and R. Sanders, “Five recommended practices for computational scientists who write software,” Computing in Science and Engineering, vol. 11, no. 5, pp. 48–53, 2009

    work page 2009

  10. [10]

    Best practices for scientific computing,

    G. Wilson, D. A. Aruliah, C. T. Brown, N. P. Chue Hong, M. Davis, R. T. Guy, S. H. D. Haddock, K. D. Huff, I. M. Mitchell, M. D. Plumbley, B. Waugh, E. P. White, and P. Wilson, “Best practices for scientific computing,” PLOS Biology, vol. 12, no. 1, pp. 1–7, 01 2014

    work page 2014

  11. [11]

    When software engineers met research scientists: A case study,

    J. Segal, “When software engineers met research scientists: A case study,” Empirical Software Eng. , vol. 10, no. 4, pp. 517–536, 2005

    work page 2005

  12. [12]

    Scientists and software engineers: A tale of two cultures,

    ——, “Scientists and software engineers: A tale of two cultures,” in Workshop of the Psychology of Programming Interest Group, 2008, p. 6

    work page 2008

  13. [13]

    How accurate is scientific software?

    L. Hatton and A. Roberts, “How accurate is scientific software?” IEEE Trans. Software Eng., vol. 20, no. 10, pp. 785–797, 1994

    work page 1994

  14. [14]

    Testing as an essential process for developing and maintaining scientific software,

    T. Clune, M. Rilee, and D. Rouson, “Testing as an essential process for developing and maintaining scientific software,” in The 2nd Workshop on Sustainable Software for Science: Practices and Experiences , 2014

    work page 2014

  15. [15]

    Examining random and designed tests to detect code mistakes in scientific software,

    D. Kelly, R. Gray, and Y . Shao, “Examining random and designed tests to detect code mistakes in scientific software,”J. Comput. Science, vol. 2, no. 1, pp. 47–56, 2011

    work page 2011

  16. [16]

    Scientific software testing: Analysis with four dimensions,

    D. Kelly, S. Thorsteinson, and D. Hook, “Scientific software testing: Analysis with four dimensions,” Softw., vol. 28, no. 3, pp. 84–90, 2011

    work page 2011

  17. [17]

    A theory-driven testing methodology for developing scientific software,

    P. C. R. Lane and F. Gobet, “A theory-driven testing methodology for developing scientific software,”J. Exp. Theor. Artif. Intell., vol. 24, no. 4, pp. 421–456, 2012

    work page 2012

  18. [18]

    Scientific software development viewed as knowledge acqui- sition: Towards understanding the development of risk-averse scientific software,

    D. Kelly, “Scientific software development viewed as knowledge acqui- sition: Towards understanding the development of risk-averse scientific software,” Journal of Systems and Software , vol. 109, pp. 50–61, 2015

    work page 2015

  19. [19]

    Testing scientific software: A system- atic literature review,

    U. Kanewala and J. M. Bieman, “Testing scientific software: A system- atic literature review,” Information and Software Technology , vol. 56, no. 10, pp. 1219–1232, 2014

    work page 2014

  20. [20]

    Creating a flexible environment for testing scientific software,

    M. C. Smith, R. L. Kelsey, J. M. Riese, and G. A. Young, “Creating a flexible environment for testing scientific software,” in Enabling Technologies for Simulation Science VIII, SPIE 5423, 2004, pp. 288–296

    work page 2004

  21. [21]

    Testing scientific programs,

    P. F. Dubois, “Testing scientific programs,” Computing in Science and Engineering, vol. 14, no. 4, pp. 69–73, 2012

    work page 2012

  22. [22]

    Design and use of reference data sets for testing scientific software,

    M. Cox and P. Harris, “Design and use of reference data sets for testing scientific software,” Analytica Chimica Acta , vol. 380, no. 2, pp. 339– 351, 1999

    work page 1999

  23. [23]

    Test-driven development in scientific software: a survey,

    A. Nanthaamornphong and J. C. Carver, “Test-driven development in scientific software: a survey,” Software Quality Journal , vol. 25, no. 2, pp. 343–372, Jun 2017

    work page 2017

  24. [24]

    Software testing and verification in model development,

    T. L. Clune and R. Rood, “Software testing and verification in model development,” IEEE Software, vol. 28, no. 6, pp. 49–55, 2011

    work page 2011

  25. [25]

    Software development environments for scientific and engineering software: A series of case studies,

    J. C. Carver, R. P. Kendall, S. E. Squires, and D. E. Post, “Software development environments for scientific and engineering software: A series of case studies,” in 29th International Conference on Software Engineering (ICSE). IEEE, 2007, pp. 550–559

    work page 2007

  26. [26]

    Testing for trustworthiness in scientific soft- ware,

    D. Hook and D. Kelly, “Testing for trustworthiness in scientific soft- ware,” in ICSE Workshop on Software Engineering for Computational Science and Engineering, SE-CSE . IEEE, 2009, pp. 59–64. 31

    work page 2009

  27. [27]

    Nomad: The fair concept for big data-driven materials science,

    C. Draxl and M. Scheffler, “Nomad: The fair concept for big data-driven materials science,” MRS Bulletin, vol. 43, no. 9, pp. 676–682, 2018

    work page 2018

  28. [28]

    exciting: a full- potential all-electron package implementing density-functional theory and many-body perturbation theory,

    A. Gulans, S. Kontur, C. Meisenbichler, D. Nabok, P. Pavone, S. Rig- amonti, S. Sagmeister, U. Werner, and C. Draxl, “exciting: a full- potential all-electron package implementing density-functional theory and many-body perturbation theory,” Journal of Physics: Condensed Matter, vol. 26, no. 36, p. 363202, 2014

    work page 2014

  29. [29]

    Recent developments in the ABINIT software package,

    X. Gonze, F. Jollet, F. Abreu Araujo, D. Adams, B. Amadon, T. Ap- plencourt, C. Audouze, J. M. Beuken, J. Bieder, A. Bokhanchuk, E. Bousquet, F. Bruneval, D. Caliste, M. C ˆot´e, F. Dahm, F. Da Pieve, M. Delaveau, M. Di Gennaro, B. Dorado, C. Espejo, G. Geneste, L. Gen- ovese, A. Gerossier, M. Giantomassi, Y . Gillet, D. R. Hamann, L. He, G. Jomard, J. La...

    work page 2016

  30. [30]

    The FAIR Guiding Principles for scientific data management and stewardship,

    M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, A. Gonzalez-Beltran, A. J. G. Gray, P. Groth, C. Goble, J. S. Grethe, J. Heringa, P. A. C. ’t Hoen, R. Hoo...

    work page 2016

  31. [31]

    The lda-1/2 method applied to atoms and molecules,

    R. Rodrigues Pela, A. Gulans, and C. Draxl, “The lda-1/2 method applied to atoms and molecules,” Journal of Chemical Theory and Computation, vol. 14, no. 9, pp. 4678–4686, 2018, pMID: 30119607

    work page 2018

  32. [32]

    Navigat- ing the maze: The impact of configurability in bioinformatics software,

    M. Cashman, M. B. Cohen, P. Ranjan, and R. W. Cottingham, “Navigat- ing the maze: The impact of configurability in bioinformatics software,” in 33rd International Conference on Automated Software Engineering , ser. ASE. ACM, 2018, pp. 757–767

    work page 2018

  33. [33]

    Verifying and validating software requirements and design specifications,

    B. W. Boehm, “Verifying and validating software requirements and design specifications,” IEEE Software, vol. 1, no. 1, pp. 75–88, 1984

    work page 1984

  34. [34]

    Software engineering for scientists,

    D. Kelly, S. Smith, and N. Meng, “Software engineering for scientists,” Computing in Science Engineering , vol. 13, no. 5, pp. 7–11, 2011

    work page 2011

  35. [35]

    System testing a scientific framework using a regression-test environment,

    H. Remmel, B. Paech, P. Bastian, and C. Engwer, “System testing a scientific framework using a regression-test environment,” Computing in Science and Engineering , vol. 14, no. 2, pp. 38–45, 2012

    work page 2012

  36. [36]

    Modeling input space for testing scientific computational software: A case study,

    S. A. Vilkomir, W. T. Swain, J. H. Poore, and K. T. Clarno, “Modeling input space for testing scientific computational software: A case study,” in 8th International Conference on Computational Science (ICCS) , ser. LNCS, vol. 5103. Springer, 2008, pp. 291–300

    work page 2008

  37. [37]

    Moving forward with combinatorial interaction testing,

    C. Yilmaz, S. Fouch, M. B. Cohen, A. Porter, G. Demiroz, and U. Koc, “Moving forward with combinatorial interaction testing,” Computer, vol. 47, no. 2, pp. 37–45, 2014

    work page 2014

  38. [38]

    Climate change: A grand software challenge,

    S. M. Easterbrook, “Climate change: A grand software challenge,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineer- ing Research, ser. FoSER ’10. ACM, 2010, pp. 99–104

    work page 2010

  39. [39]

    Common Workflow Language, v1.0,

    B. Chapman, J. Chilton, M. Heuer, A. Kartashov, D. Leehr, H. M ´enager, M. Nedeljkovich, M. Scales, S. Soiland-Reyes, and L. Stojanovic, “Common Workflow Language, v1.0,” Common Workflow Language working group, Specification, Jul. 2016. [Online]. Available: https://w3id.org/cwl/v1.0/

    work page 2016

  40. [40]

    Elastic: A tool for calculating second-order elastic constants from first principles,

    R. Golesorkhtabar, P. Pavone, J. Spitaler, P. Puschnig, and C. Draxl, “Elastic: A tool for calculating second-order elastic constants from first principles,” Computer Physics Communications , vol. 184, no. 8, pp. 1861–1873, 2013

    work page 2013

  41. [41]

    Global software development,

    J. D. Herbsleb and D. Moitra, “Global software development,” IEEE Software, vol. 18, no. 2, pp. 16–20, March 2001

    work page 2001

  42. [42]

    Reproducibility in density functional theory calculations of solids,

    K. Lejaeghere, G. Bihlmayer, T. Bj ¨orkman, P. Blaha, S. Bl¨ugel, V . Blum, D. Caliste, I. E. Castelli, S. J. Clark, A. Dal Corso, S. de Giron- coli, T. Deutsch, J. K. Dewhurst, I. Di Marco, C. Draxl, M. Dułak, O. Eriksson, J. A. Flores-Livas, K. F. Garrity, L. Genovese, P. Giannozzi, M. Giantomassi, S. Goedecker, X. Gonze, O. Gr ˚an¨as, E. K. U. Gross, A...

    work page 2016

  43. [43]

    Quality Control of Numerical Settings for DFT Calculations and Materials Databases,

    C. Carbogno, K. Thygesen, B. Bieniek, C. Draxl, L. Ghiringhelli, A. Gu- lans, O. Hofmann, K. Jacobsen, S. Lubeck, J. Mortensen, M. Strange, E. Wruss, and M. Scheffler, “Quality Control of Numerical Settings for DFT Calculations and Materials Databases,” in APS Meeting Abstracts , 2018, p. P12.003

    work page 2018

  44. [44]

    Techniques for testing scientific programs without an oracle,

    U. Kanewala and J. M. Bieman, “Techniques for testing scientific programs without an oracle,” in Intl. Workshop on Software Engineering for Computational Science and Engineering . IEEE, 2013, pp. 48–57

    work page 2013

  45. [45]

    PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs,

    W. K. Chan, S. Cheung, J. C. F. Ho, and T. H. Tse, “PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs,”Journal of Systems and Software, vol. 82, no. 3, pp. 422–434, 2009

    work page 2009

  46. [46]

    Mutation sensitivity testing,

    D. Hook and D. Kelly, “Mutation sensitivity testing,” Computing in Science and Engineering , vol. 11, no. 6, pp. 40–47, 2009

    work page 2009

  47. [47]

    Metamorphic testing: a new approach for generating next test cases,

    T. Y . Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: a new approach for generating next test cases,” HKUST-CS98-01, Department of Computer Science, Hong Kong , Tech. Rep., 1998

    work page 1998

  48. [48]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Trans. Software Eng., vol. 41, no. 5, pp. 507–525, 2015

    work page 2015

  49. [49]

    A survey on metamorphic testing,

    S. Segura, G. Fraser, A. B. S ´anchez, and A. R. Cort ´es, “A survey on metamorphic testing,” IEEE Trans. Software Eng. , vol. 42, no. 9, pp. 805–824, 2016

    work page 2016

  50. [50]

    Automated metamorphic testing,

    A. Gotlieb and B. Botella, “Automated metamorphic testing,” in 27th International Computer Software and Applications Conference (COMP- SAC). IEEE, 2003, pp. 34–40

    work page 2003

  51. [51]

    Properties of machine learning applications for use in metamorphic testing,

    C. Murphy, G. E. Kaiser, L. Hu, and L. Wu, “Properties of machine learning applications for use in metamorphic testing,” in Intl. Conf. on Softw. Engineering & Knowledge Engineering. KSI, 2008, pp. 867–872

    work page 2008

  52. [52]

    Testing and validating machine learning classifiers by metamorphic testing,

    X. Xie, J. W. K. Ho, C. Murphy, G. E. Kaiser, B. Xu, and T. Y . Chen, “Testing and validating machine learning classifiers by metamorphic testing,” Journal of Sys. and Softw. , vol. 84, no. 4, pp. 544–558, 2011

    work page 2011

  53. [53]

    An innovative approach for testing bioinformatics programs using metamorphic testing,

    T. Y . Chen, J. W. Ho, H. Liu, and X. Xie, “An innovative approach for testing bioinformatics programs using metamorphic testing,” BMC bioinformatics, vol. 10, no. 1, p. 24, 2009

    work page 2009

  54. [54]

    A machine learning approach for developing test oracles for testing scientific software,

    J. Ding and D. Zhang, “A machine learning approach for developing test oracles for testing scientific software,” in Intl. Conference on Software Engineering and Knowledge Engineering . KSI, 2016, pp. 390–395

    work page 2016

  55. [55]

    Cost- efficient sampling for performance prediction of configurable systems (T),

    A. Sarkar, J. Guo, N. Siegmund, S. Apel, and K. Czarnecki, “Cost- efficient sampling for performance prediction of configurable systems (T),” in 30th International Conference on Automated Software Engi- neering, ASE. IEEE, 2015, pp. 342–352

    work page 2015

  56. [56]

    A classification and survey of analysis strategies for software product lines,

    T. Th ¨um, S. Apel, C. K¨astner, I. Schaefer, and G. Saake, “A classification and survey of analysis strategies for software product lines,” ACM Comput. Surv., vol. 47, no. 1, pp. 6:1–6:45, 2014

    work page 2014

  57. [57]

    Technological spaces: An initial appraisal,

    I. Ivanov, J. B ´ezivin, and M. Aksit, “Technological spaces: An initial appraisal,” 10 2002, pp. 1–6

    work page 2002

  58. [58]

    Towards a common format for computational material science data,

    L. M. Ghiringhelli, C. Carbogno, S. Levchenko, F. Mohamed, G. Huhs, M. Lueders, M. Oliveira, and M. Scheffler, “Towards a common format for computational material science data,” 2016

    work page 2016

  59. [59]

    Ten recommendations for software engineering in research,

    J. Hastings, K. Haug, and C. Steinbeck, “Ten recommendations for software engineering in research,” GigaScience, vol. 3, Dec. 2014

    work page 2014

  60. [60]

    Supervised intentional process models discovery using Hidden Markov models,

    G. Khodabandelou, C. Hug, R. Deneck `ere, and C. Salinesi, “Supervised intentional process models discovery using Hidden Markov models,” in 7th International Conference on Research Challenges in Information Science (RCIS), 2013, pp. 1–11

    work page 2013

  61. [61]

    Unsupervised Discovery of Intentional Process Models from Event Logs,

    ——, “Unsupervised Discovery of Intentional Process Models from Event Logs,” in Proceedings of the 11th Working Conference on Mining Software Repositories, ser. MSR 2014. ACM, 2014, pp. 282–291

    work page 2014

  62. [62]

    Some challenges facing scientific software developers: The case of molecular biology,

    C. Morris and J. Segal, “Some challenges facing scientific software developers: The case of molecular biology,” in Fifth International Conference on e-Science . IEEE, 2009, pp. 216–222

    work page 2009

  63. [63]

    The need for a rigorous development and testing methodology for medical software,

    W. A. Reupke, E. Srinivasan, P. V . Rigterink, and D. N. Card, “The need for a rigorous development and testing methodology for medical software,” in First Annual IEEE Symposium on Computer-Based Medical Systems (CBMS’88). IEEE, 1988, pp. 15–20. 32

    work page 1988