Guidelines for benchmarking of optimization approaches for fitting mathematical models

Clemens Kreutz

arxiv: 1907.03427 · v1 · pith:JSHPMJICnew · submitted 2019-07-08 · 💻 cs.PF · stat.CO

Guidelines for benchmarking of optimization approaches for fitting mathematical models

Clemens Kreutz This is my paper

Pith reviewed 2026-05-25 00:56 UTC · model grok-4.3

classification 💻 cs.PF stat.CO

keywords benchmarkingoptimizationmodel fittingsystems biologyparameter estimationcomputational biologyguidelines

0 comments

The pith

The paper presents tailored guidelines for performing informative and unbiased benchmarking of optimization approaches for fitting mathematical models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies insufficient performance of optimization methods as a major bottleneck in systems biology due to biased or uninformative benchmark studies. It summarizes the reasons and methodological challenges behind this issue and their effects on benchmarks. Drawing from general guidelines in computational biology, it offers a specific set of recommendations to improve the quality of evidence from such studies. A reader would care because these guidelines aim to establish more reliable methodologies for comparing fitting approaches, which could accelerate progress in modeling biological systems.

Core claim

Based on general guidelines for benchmarking in computational biology, a collection of tailored guidelines is presented for performing informative and unbiased benchmarking of optimization-based fitting approaches, with a call for comprehensive studies to establish robust methodology.

What carries the argument

The tailored guidelines derived from general benchmarking principles to address specific challenges in optimization fitting for mathematical models.

Load-bearing premise

The summarized reasons and methodological challenges capture the main sources of bias and that the tailored guidelines will demonstrably increase the evidence quality of benchmark outcomes.

What would settle it

A study that applies the guidelines to an existing benchmark and shows that the conclusions about which optimization approaches perform best change substantially compared to the original biased study.

Figures

Figures reproduced from arXiv: 1907.03427 by Clemens Kreutz.

**Figure 1.** Figure 1: Tasks to be accomplished for fitting ODE models. Fitting of ODE models requires several tasks. Most approaches combine both, global and local search strategies. A prominent global search strategy is random drawing of multiple initial guesses and performing local optimization for each starting point. In each optimization step, the ODEs have to be solved for evaluation of the objective function χ 2 (θ). More… view at source ↗

**Figure 2.** Figure 2: Impact of configuration parameters. Multiple configurations can have an impact on the performance and performance benefits. In this illustration example, two optimization approaches have different sensitivity with respect to the choice of tolerances controlling the numerical error of ODE integration. Moreover, both approaches have different optimal choices for this tuning parameter. For most integration to… view at source ↗

**Figure 3.** Figure 3: Ambiguous interpretation of optimization outcomes. For non-trivial optimization problems, the results of independent optimization runs are typically not the same. The upper left panel indicates an outcome for three optimization runs, e.g. generated with different starting points. If the objective function values after optimization are different (“scenario A”), such an outcome could be explained by local op… view at source ↗

read the original abstract

Insufficient performance of optimization approaches for fitting of mathematical models is still a major bottleneck in systems biology. In this manuscript, the reasons and methodological challenges are summarized as well as their impact in benchmark studies. Important aspects for increasing evidence of outcomes of benchmark analyses are discussed. Based on general guidelines for benchmarking in computational biology, a collection of tailored guidelines is presented for performing informative and unbiased benchmarking of optimization-based fitting approaches. Comprehensive benchmark studies based on these recommendations are urgently required for establishing of a robust and reliable methodology for the systems biology community.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

A practical checklist of tailored guidelines for benchmarking optimizer performance in systems biology model fitting, with no demonstration that the rules change outcomes.

read the letter

The paper pulls together existing general benchmarking advice and adapts it to the specific setting of comparing optimization routines for fitting ODE or other mathematical models. It lists common issues such as non-representative test problems, cherry-picked performance metrics, and weak statistical comparisons, then offers concrete recommendations on problem selection, measure choice, and reporting. That tailoring is the main contribution; nothing new is derived or tested. The text itself notes that full studies applying the guidelines are still needed, which is accurate. The absence of even one worked example or re-analysis of a published benchmark means we cannot tell whether the suggested rules would have altered any prior conclusions or reduced bias. The writing is clear and the challenges it flags are real in the field, but the piece stays at the level of methodological opinion. Readers who run or referee fitting benchmarks in systems biology will find it a useful reminder list. It is not a research result and does not claim to be. A serious editor could send it to review as a methods or guidelines contribution, provided the authors frame it explicitly as advice rather than validated procedure.

Referee Report

1 major / 1 minor

Summary. The paper summarizes the reasons and methodological challenges impacting benchmark studies of optimization approaches for fitting mathematical models in systems biology. It discusses aspects for increasing the evidence quality of such analyses and, based on general guidelines for benchmarking in computational biology, presents a collection of tailored guidelines for performing informative and unbiased benchmarking. The manuscript concludes by noting that comprehensive benchmark studies based on these recommendations are urgently required.

Significance. If the proposed guidelines prove effective, they could substantially improve the reliability and comparability of benchmark studies in the field by addressing common sources of bias and methodological shortcomings. The paper provides a useful synthesis of challenges and adapts broader best practices to this specific domain, which may encourage more rigorous benchmarking practices among researchers.

major comments (1)

[Abstract] The central claim that the tailored guidelines will enable 'informative and unbiased benchmarking of optimization-based fitting approaches' is not supported by any empirical validation, case study, or re-analysis of existing benchmarks in the manuscript. This is particularly relevant as the abstract itself states that 'Comprehensive benchmark studies based on these recommendations are urgently required for establishing of a robust and reliable methodology', highlighting that the guidelines' benefits remain unproven.

minor comments (1)

[Abstract] The phrasing 'establishing of a robust and reliable methodology' is grammatically awkward and should be corrected to 'establishing a robust and reliable methodology'.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review. We address the single major comment below.

read point-by-point responses

Referee: [Abstract] The central claim that the tailored guidelines will enable 'informative and unbiased benchmarking of optimization-based fitting approaches' is not supported by any empirical validation, case study, or re-analysis of existing benchmarks in the manuscript. This is particularly relevant as the abstract itself states that 'Comprehensive benchmark studies based on these recommendations are urgently required for establishing of a robust and reliable methodology', highlighting that the guidelines' benefits remain unproven.

Authors: We agree that the manuscript contains no empirical validation, case study, or re-analysis of the tailored guidelines. The paper's contribution is a synthesis of known challenges and an adaptation of existing computational-biology benchmarking practices to the specific setting of optimization-based model fitting; it does not claim to have tested the guidelines' effectiveness. The abstract already states that comprehensive studies are required. We will revise the abstract (and any similar phrasing in the introduction) to present the guidelines as recommendations intended to improve benchmarking rather than as already-demonstrated means of achieving informative and unbiased results. This revision will be incorporated in the next version. revision: yes

Circularity Check

0 steps flagged

No circularity: advisory guidelines with no derivation chain or fitted predictions

full rationale

The paper is a set of recommendations for benchmarking optimization-based model fitting. It summarizes domain challenges from the literature and tailors general computational-biology benchmarking principles into specific advice on problem selection, performance measures, and statistical analysis. No equations, parameters, or predictions appear; the central claim is simply that following these guidelines would be informative. The text itself states that comprehensive studies applying the guidelines remain to be done. No self-citation is load-bearing for any derivation, and no step reduces by construction to its own inputs. This is a standard non-circular advisory document.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that general benchmarking guidelines exist and can be usefully specialized, plus the premise that current benchmark practices suffer from identifiable methodological flaws that the new guidelines will mitigate.

axioms (1)

domain assumption General guidelines for benchmarking in computational biology exist and form a suitable base for specialization.
Invoked when the paper states it builds tailored guidelines on these general ones (abstract).

pith-pipeline@v0.9.0 · 5598 in / 1052 out tokens · 26731 ms · 2026-05-25T00:56:48.960599+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

[1]

M. R. Aniba, O. Poch, and J. D. Thompson. Issues in bioinformatics benchmarking: the case study of multiple sequence alignment. Nucleic Acids Research, 38(21):7353–7363, 2010

work page 2010
[2]

Ashyraliyev, Y

M. Ashyraliyev, Y. Fomekong-Nanfack, J. A. Kaandorp, and J. G. Blom. Systems biology: parameter estimation for biochemical models. The FEBS journal , 276(4):886–902, 2009

work page 2009
[3]

F. T. Bergmann, R. Adams, S. Moodie, J. Cooper, M. Glont, M. Golebiewski, M. Hucka, C. Laibe, A. K. Miller, D. P. Nickerson, et al. Combine archive and omex format: one file to share all information to reproduce a modeling project. BMC Bioinformatics , 15(1):369, 2014

work page 2014
[4]

Boulesteix

A.-L. Boulesteix. Ten simple rules for reducing overoptimistic reporting in methodological computational research, 2015

work page 2015
[5]

Boulesteix, H

A.-L. Boulesteix, H. Binder, M. Abrahamowicz, W. Sauerbrei, and S. P. of the STRATOS Initiative. On the necessity and design of studies comparing statistical methods. Biometrical Journal, 60(1):216–218, 2018

work page 2018
[6]

Boulesteix, S

A.-L. Boulesteix, S. Lauer, and M. J. Eugster. A plea for neutral comparison studies in computational sciences. PloS one , 8(4):e61562, 2013

work page 2013
[7]

Capella-Gutierrez, D

S. Capella-Gutierrez, D. De La Iglesia, J. Haas, A. Lourenco, J. M. F. Gonzalez, D. Repchevsky, C. Dessimoz, T. Schwede, C. Notredame, J. L. Gelpi, et al. Lessons learned: Recommendations for establishing critical periodic scientific benchmarking. BioRxiv, page 181677, 2017

work page 2017
[8]

T. F. Coleman and Y. Li. An interior trust region approach for nonlinear minimization subject to bounds. SIAM Journal on optimization , 6(2):418–445, 1996

work page 1996
[9]

Degasperi, D

A. Degasperi, D. Fey, and B. N. Kholodenko. Performance of objective functions and optimisation procedures for parameter estimation in system biology models. NPJ systems biology and applications , 3(1):20, 2017

work page 2017
[10]

J. A. Egea, E. Balsa-Canto, M.-S. G. Garcia, and J. R. Banga. Dynamic optimization of nonlinear processes with an enhanced scatter search method. Industrial & Engineering Chemistry Research , 48(9):4388–4401, 2009

work page 2009
[11]

J. A. Egea, R. Marti, and J. R. Banga. An evolutionary method for complex-process optimization. Computers & Operations Research , 37(2):315–324, 2010. 11/13

work page 2010
[12]

Fiedler, S

A. Fiedler, S. Raeth, F. J. Theis, A. Hausser, and J. Hasenauer. Tailored parameter optimization methods for ordinary differential equation models with steady-state constraints. BMC systems biology , 10(1):80, 2016

work page 2016
[13]

Fr¨ ohlich, B

F. Fr¨ ohlich, B. Kaltenbacher, F. J. Theis, and J. Hasenauer. Scalable parameter estimation for genome-scale biochemical reaction networks. PLoS computational biology, 13(1):e1005331, 2017

work page 2017
[14]

Fr¨ ohlich, C

F. Fr¨ ohlich, C. Loos, and J. Hasenauer. Scalable inference of ordinary differential equation models of biochemical processes. In Gene Regulatory Networks , pages 385–422. Springer, New York, 2019

work page 2019
[15]

Fr¨ ohlich, F

F. Fr¨ ohlich, F. J. Theis, J. O. R¨ adler, and J. Hasenauer. Parameter estimation for dynamical systems with discrete events and logical operations. Bioinformatics, 33(7):1049–1056, 2016

work page 2016
[16]

G´ abor and J

A. G´ abor and J. R. Banga. Robust and efficient parameter estimation in dynamic models of biological systems. BMC systems biology , 9(1):74, 2015

work page 2015
[17]

H. Hass, C. Loos, E. Raim´ undez-´Alvarez, J. Timmer, J. Hasenauer, and C. Kreutz. Benchmark problems for dynamic modeling of intracellular processes. Bioinformatics, jan 2019

work page 2019
[18]

Hengl, C

S. Hengl, C. Kreutz, J. Timmer, and T. Maiwald. Data-based identifiability analysis of non-linear dynamical models. Bioinformatics, 23(19):2612–2618, 2007

work page 2007
[19]

J. P. Ioannidis. Meta-research: Why research on research matters. PLoS biology, 16(3):e2005468, 2018

work page 2018
[20]

C. Kreutz. New concepts for evaluating the performance of computational methods. IF AC-PapersOnLine, 49(26):63–70, 2016

work page 2016
[21]

C. Kreutz. An easy and efficient approach for testing identifiability. Bioinformatics, 34(11):1913–1921, 2018

work page 1913
[22]

C. Li, M. Donizelli, N. Rodriguez, H. Dharuri, L. Endler, V. Chelliah, L. Li, E. He, A. Henry, M. I. Stefan, et al. Biomodels database: An enhanced, curated and annotated resource for published quantitative kinetic models. BMC systems biology, 4(1):92, 2010

work page 2010
[23]

Mangul, L

S. Mangul, L. S. Martin, B. L. Hill, A. K.-M. Lam, M. G. Distler, A. Zelikovsky, E. Eskin, and J. Flint. Systematic benchmarking of omics computational tools. Nature communications, 10, 2019

work page 2019
[24]

Meyer, T

P. Meyer, T. Cokelaer, D. Chandran, K. H. Kim, P.-R. Loh, G. Tucker, M. Lipson, B. Berger, C. Kreutz, A. Raue, et al. Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach. BMC systems biology , 8(1):13, 2014

work page 2014
[25]

C. G. Moles, P. Mendes, and J. R. Banga. Parameter estimation in biochemical pathways: a comparison of global optimization methods. Genome research, 13(11):2467–2474, 2003

work page 2003
[26]

Peifer and J

M. Peifer and J. Timmer. Parameter estimation in ordinary differential equations for biochemical processes using the method of multiple shooting. IET Systems Biology, 1(2):78–88, 2007. 12/13

work page 2007
[27]

Peters, S

B. Peters, S. E. Brenner, E. Wang, D. Slonim, and M. G. Kann. Putting benchmarks in their rightful place: The heart of computational biology, 2018

work page 2018
[28]

A. Raue, C. Kreutz, T. Maiwald, J. Bachmann, M. Schilling, U. Klingm¨ uller, and J. Timmer. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics, 25(15):1923–1929, 2009

work page 1923
[29]

A. Raue, M. Schilling, J. Bachmann, A. Matteson, M. Schelke, D. Kaschek, S. Hug, C. Kreutz, B. D. Harms, F. J. Theis, et al. Lessons learned from quantitative dynamical modeling in systems biology. PloS one , 8(9):e74335, 2013

work page 2013
[30]

A. Raue, B. Steiert, M. Schelker, C. Kreutz, T. Maiwald, H. Hass, J. Vanlier, C. Toensing, L. Adlung, R. Engesser, W. Mader, T. Heinemann, J. Hasenauer, M. Schilling, T. Hoefer, E. Klipp, F. Theis, U. Klingmueller, B. Schoeberl, and J. Timmer. Data2Dynamics: a modeling environment tailored to parameter estimation in dynamical systems. Bioinformatics, 31:3...

work page 2015
[31]

Schweiger

T. Schweiger. The performance of constrained optimization for ordinary differential equation models. Bachelor thesis, University of Freiburg, 2017

work page 2017
[32]

Steiert, A

B. Steiert, A. Raue, J. Timmer, and C. Kreutz. Experimental design for parameter estimation of gene regulatory networks. PloS one , 7(7):e40052, 2012

work page 2012
[33]

J. Sun, J. M. Garibaldi, and C. Hodgman. Parameter estimation using metaheuristics in systems biology: a comprehensive review. IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB) , 9(1):185–202, 2012

work page 2012
[34]

A. F. Villaverde, F. Fr¨ ohlich, D. Weindl, J. Hasenauer, and J. R. Banga. Benchmarking optimization methods for parameter estimation in large kinetic models. Bioinformatics, 35(5):830–838, 2018

work page 2018
[35]

A. F. Villaverde, D. Henriques, K. Smallbone, S. Bongard, J. Schmid, D. Cicin-Sain, A. Crombach, J. Saez-Rodriguez, K. Mauch, E. Balsa-Canto, P. Mendes, J. Jaeger, and J. R. Banga. BioPreDyn-bench: a suite of benchmark problems for dynamic modelling in systems biology. BMC Syst. Biol. , 9:8, 2015

work page 2015
[36]

L. M. Weber, W. Saelens, R. Cannoodt, C. Soneson, A. Hapfelmeier, P. P. Gardner, A.-L. Boulesteix, Y. Saeys, and M. D. Robinson. Essential guidelines for computational method benchmarking. Genome Biology, 20(1):125, 2019

work page 2019
[37]

F. Wieland. Implementation and assessment of optimization approaches for parameter estimation in systems biology. Masters thesis, University of Freiburg, 2016

work page 2016
[38]

Winkelmann

B. Winkelmann. Implementatoin and evaluation of optimization algorithms for parameter estimation in systems biology. Bachelor thesis, University of Freiburg, 2016. 13/13

work page 2016

[1] [1]

M. R. Aniba, O. Poch, and J. D. Thompson. Issues in bioinformatics benchmarking: the case study of multiple sequence alignment. Nucleic Acids Research, 38(21):7353–7363, 2010

work page 2010

[2] [2]

Ashyraliyev, Y

M. Ashyraliyev, Y. Fomekong-Nanfack, J. A. Kaandorp, and J. G. Blom. Systems biology: parameter estimation for biochemical models. The FEBS journal , 276(4):886–902, 2009

work page 2009

[3] [3]

F. T. Bergmann, R. Adams, S. Moodie, J. Cooper, M. Glont, M. Golebiewski, M. Hucka, C. Laibe, A. K. Miller, D. P. Nickerson, et al. Combine archive and omex format: one file to share all information to reproduce a modeling project. BMC Bioinformatics , 15(1):369, 2014

work page 2014

[4] [4]

Boulesteix

A.-L. Boulesteix. Ten simple rules for reducing overoptimistic reporting in methodological computational research, 2015

work page 2015

[5] [5]

Boulesteix, H

A.-L. Boulesteix, H. Binder, M. Abrahamowicz, W. Sauerbrei, and S. P. of the STRATOS Initiative. On the necessity and design of studies comparing statistical methods. Biometrical Journal, 60(1):216–218, 2018

work page 2018

[6] [6]

Boulesteix, S

A.-L. Boulesteix, S. Lauer, and M. J. Eugster. A plea for neutral comparison studies in computational sciences. PloS one , 8(4):e61562, 2013

work page 2013

[7] [7]

Capella-Gutierrez, D

S. Capella-Gutierrez, D. De La Iglesia, J. Haas, A. Lourenco, J. M. F. Gonzalez, D. Repchevsky, C. Dessimoz, T. Schwede, C. Notredame, J. L. Gelpi, et al. Lessons learned: Recommendations for establishing critical periodic scientific benchmarking. BioRxiv, page 181677, 2017

work page 2017

[8] [8]

T. F. Coleman and Y. Li. An interior trust region approach for nonlinear minimization subject to bounds. SIAM Journal on optimization , 6(2):418–445, 1996

work page 1996

[9] [9]

Degasperi, D

A. Degasperi, D. Fey, and B. N. Kholodenko. Performance of objective functions and optimisation procedures for parameter estimation in system biology models. NPJ systems biology and applications , 3(1):20, 2017

work page 2017

[10] [10]

J. A. Egea, E. Balsa-Canto, M.-S. G. Garcia, and J. R. Banga. Dynamic optimization of nonlinear processes with an enhanced scatter search method. Industrial & Engineering Chemistry Research , 48(9):4388–4401, 2009

work page 2009

[11] [11]

J. A. Egea, R. Marti, and J. R. Banga. An evolutionary method for complex-process optimization. Computers & Operations Research , 37(2):315–324, 2010. 11/13

work page 2010

[12] [12]

Fiedler, S

A. Fiedler, S. Raeth, F. J. Theis, A. Hausser, and J. Hasenauer. Tailored parameter optimization methods for ordinary differential equation models with steady-state constraints. BMC systems biology , 10(1):80, 2016

work page 2016

[13] [13]

Fr¨ ohlich, B

F. Fr¨ ohlich, B. Kaltenbacher, F. J. Theis, and J. Hasenauer. Scalable parameter estimation for genome-scale biochemical reaction networks. PLoS computational biology, 13(1):e1005331, 2017

work page 2017

[14] [14]

Fr¨ ohlich, C

F. Fr¨ ohlich, C. Loos, and J. Hasenauer. Scalable inference of ordinary differential equation models of biochemical processes. In Gene Regulatory Networks , pages 385–422. Springer, New York, 2019

work page 2019

[15] [15]

Fr¨ ohlich, F

F. Fr¨ ohlich, F. J. Theis, J. O. R¨ adler, and J. Hasenauer. Parameter estimation for dynamical systems with discrete events and logical operations. Bioinformatics, 33(7):1049–1056, 2016

work page 2016

[16] [16]

G´ abor and J

A. G´ abor and J. R. Banga. Robust and efficient parameter estimation in dynamic models of biological systems. BMC systems biology , 9(1):74, 2015

work page 2015

[17] [17]

H. Hass, C. Loos, E. Raim´ undez-´Alvarez, J. Timmer, J. Hasenauer, and C. Kreutz. Benchmark problems for dynamic modeling of intracellular processes. Bioinformatics, jan 2019

work page 2019

[18] [18]

Hengl, C

S. Hengl, C. Kreutz, J. Timmer, and T. Maiwald. Data-based identifiability analysis of non-linear dynamical models. Bioinformatics, 23(19):2612–2618, 2007

work page 2007

[19] [19]

J. P. Ioannidis. Meta-research: Why research on research matters. PLoS biology, 16(3):e2005468, 2018

work page 2018

[20] [20]

C. Kreutz. New concepts for evaluating the performance of computational methods. IF AC-PapersOnLine, 49(26):63–70, 2016

work page 2016

[21] [21]

C. Kreutz. An easy and efficient approach for testing identifiability. Bioinformatics, 34(11):1913–1921, 2018

work page 1913

[22] [22]

C. Li, M. Donizelli, N. Rodriguez, H. Dharuri, L. Endler, V. Chelliah, L. Li, E. He, A. Henry, M. I. Stefan, et al. Biomodels database: An enhanced, curated and annotated resource for published quantitative kinetic models. BMC systems biology, 4(1):92, 2010

work page 2010

[23] [23]

Mangul, L

S. Mangul, L. S. Martin, B. L. Hill, A. K.-M. Lam, M. G. Distler, A. Zelikovsky, E. Eskin, and J. Flint. Systematic benchmarking of omics computational tools. Nature communications, 10, 2019

work page 2019

[24] [24]

Meyer, T

P. Meyer, T. Cokelaer, D. Chandran, K. H. Kim, P.-R. Loh, G. Tucker, M. Lipson, B. Berger, C. Kreutz, A. Raue, et al. Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach. BMC systems biology , 8(1):13, 2014

work page 2014

[25] [25]

C. G. Moles, P. Mendes, and J. R. Banga. Parameter estimation in biochemical pathways: a comparison of global optimization methods. Genome research, 13(11):2467–2474, 2003

work page 2003

[26] [26]

Peifer and J

M. Peifer and J. Timmer. Parameter estimation in ordinary differential equations for biochemical processes using the method of multiple shooting. IET Systems Biology, 1(2):78–88, 2007. 12/13

work page 2007

[27] [27]

Peters, S

B. Peters, S. E. Brenner, E. Wang, D. Slonim, and M. G. Kann. Putting benchmarks in their rightful place: The heart of computational biology, 2018

work page 2018

[28] [28]

A. Raue, C. Kreutz, T. Maiwald, J. Bachmann, M. Schilling, U. Klingm¨ uller, and J. Timmer. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics, 25(15):1923–1929, 2009

work page 1923

[29] [29]

A. Raue, M. Schilling, J. Bachmann, A. Matteson, M. Schelke, D. Kaschek, S. Hug, C. Kreutz, B. D. Harms, F. J. Theis, et al. Lessons learned from quantitative dynamical modeling in systems biology. PloS one , 8(9):e74335, 2013

work page 2013

[30] [30]

A. Raue, B. Steiert, M. Schelker, C. Kreutz, T. Maiwald, H. Hass, J. Vanlier, C. Toensing, L. Adlung, R. Engesser, W. Mader, T. Heinemann, J. Hasenauer, M. Schilling, T. Hoefer, E. Klipp, F. Theis, U. Klingmueller, B. Schoeberl, and J. Timmer. Data2Dynamics: a modeling environment tailored to parameter estimation in dynamical systems. Bioinformatics, 31:3...

work page 2015

[31] [31]

Schweiger

T. Schweiger. The performance of constrained optimization for ordinary differential equation models. Bachelor thesis, University of Freiburg, 2017

work page 2017

[32] [32]

Steiert, A

B. Steiert, A. Raue, J. Timmer, and C. Kreutz. Experimental design for parameter estimation of gene regulatory networks. PloS one , 7(7):e40052, 2012

work page 2012

[33] [33]

J. Sun, J. M. Garibaldi, and C. Hodgman. Parameter estimation using metaheuristics in systems biology: a comprehensive review. IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB) , 9(1):185–202, 2012

work page 2012

[34] [34]

A. F. Villaverde, F. Fr¨ ohlich, D. Weindl, J. Hasenauer, and J. R. Banga. Benchmarking optimization methods for parameter estimation in large kinetic models. Bioinformatics, 35(5):830–838, 2018

work page 2018

[35] [35]

A. F. Villaverde, D. Henriques, K. Smallbone, S. Bongard, J. Schmid, D. Cicin-Sain, A. Crombach, J. Saez-Rodriguez, K. Mauch, E. Balsa-Canto, P. Mendes, J. Jaeger, and J. R. Banga. BioPreDyn-bench: a suite of benchmark problems for dynamic modelling in systems biology. BMC Syst. Biol. , 9:8, 2015

work page 2015

[36] [36]

L. M. Weber, W. Saelens, R. Cannoodt, C. Soneson, A. Hapfelmeier, P. P. Gardner, A.-L. Boulesteix, Y. Saeys, and M. D. Robinson. Essential guidelines for computational method benchmarking. Genome Biology, 20(1):125, 2019

work page 2019

[37] [37]

F. Wieland. Implementation and assessment of optimization approaches for parameter estimation in systems biology. Masters thesis, University of Freiburg, 2016

work page 2016

[38] [38]

Winkelmann

B. Winkelmann. Implementatoin and evaluation of optimization algorithms for parameter estimation in systems biology. Bachelor thesis, University of Freiburg, 2016. 13/13

work page 2016