Pith. sign in

REVIEW 4 major objections 5 minor 65 references

An Autonomous Performance Testing Framework using Self-Adaptive Fuzzy Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that SaFReL, a fuzzy reinforcement learning tester, can learn how to steer software to its performance breaking point and reuse that policy on similar programs, reaching the target in fewer trials than typical exploratory…

desk verdict SaFReL is a plausible fuzzy-RL framework for performance test generation, but its 'model-free' claim is undermined by the fact that the reward, the similarity signal, and the simulator all share the same sensitivity parameters. read the letter →

arxiv 1908.06900 v2 pith:CYTVV7QZ submitted 2019-08-19 cs.SE cs.AIcs.PF

Mahshid Helali Moghadam , Mehrdad Saadatmand , Markus Borg , Markus Bohlin , Björn Lisper This is my paper · ORCID
classification cs.SEcs.AIcs.PF
keywords performancetestingbreakingpointreinforcementlearningfuzzylogicstatedetectiontransferadaptiveepsilon-greedyQ-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the optimal policy for performance testing—finding the point where a program's response time breaks a requirement—can be learned rather than derived from source code or models. It proposes SaFReL, a fuzzy reinforcement learning tester that first learns a policy on one software system under test and then reuses it on similar systems while keeping the learning active. On a simulated performance environment, the paper reports that SaFReL reaches the target breaking point in fewer trials than a typical random stepwise stress-testing baseline: about 42 percent fewer trials on a homogeneous set of CPU-intensive programs, and about 31 percent fewer on a heterogeneous set when the exploration rate is adapted to the detected similarity between programs. If these results hold, automated performance testing can be model-free, knowledge-reusing, and adaptive to new programs with different resource sensitivities.

What carries the argument

The central machinery is SaFReL itself: a single Q-learning agent with fuzzy state detection and an adaptive epsilon-greedy action selection strategy. Fuzzy membership functions over the normalized metrics produce fuzzy states through a 24-rule inference base, and the membership degree of the detected state weights the Q-value update, so uncertain states contribute less to learning. During transfer learning, the cosine similarity of performance sensitivity vectors tunes epsilon between 0.2 and 0.5, letting the agent exploit the stored policy on similar SUTs and explore more on dissimilar ones. This adaptive exploration is the load-bearing element: it is the reason the learned policy can be reused efficiently across programs with different sensitivities to CPU, memory, and disk resources.

What would settle it

Run SaFReL against real programs on a controlled platform, measuring actual response times as resources are reduced, with sensitivity vectors estimated from those real measurements; if the average trials to reach a breaking point do not fall below a random stepwise stress-testing baseline, the central efficiency claim fails. A sharper check is to give the reward function sensitivity values that differ from the environment's true sensitivities: if efficiency does not drop, the reward signal is not doing the work the paper attributes to it.

Watch

Extended reading notes

Core claim

SaFReL operates in two learning phases. In the initial learning phase, a Q-learning agent interacts with the first SUT, observing normalized response time and CPU, memory, and disk utilization improvements; fuzzy inference turns these continuous measurements into fuzzy states using a rule base of 24 rules, and Q-values are updated with a membership-weighted rule. After convergence, the transfer learning phase reuses the stored Q-table on new SUT instances while a strategy adaptation module computes the cosine similarity between the current SUT's performance sensitivity vector and previously seen vectors, raising epsilon to encourage exploration when similarity is low and lowering it to encourage exploitation when similarity is high. The paper's central empirical claim is that this reuse-plus-adaptation finds the performance breaking point (defined in the experiments as response time exceeding 1.5 times the requirement) in fewer learning trials than a typical exploratory stress testing procedure: about 42 percent fewer trials on a homogeneous set of 50 CPU-intensive SUTs with fixed epsilon 0.2, and about 31 percent fewer on a heterogeneous set of 50 SUTs with adaptive epsilon. The same experiments show that fixed exploration schemes do not reliably beat the baseline on heterogeneous SUTs, which supports the paper's conclusion that adaptive exploration is the mechanism making policy transfer work across programs with different resource sensitivities.

Load-bearing premise

The evaluation assumes that the simulation equation using the resource sensitivity values from Table 2 faithfully represents how real programs respond to reductions in CPU, memory, and disk capacity, and that those sensitivity values can be obtained or estimated in practice.

Editorial extensions

If this is right

  • Performance test case generation can be automated for systems where source code and performance models are unavailable, provided resource sensitivity indicators can be measured or estimated.
  • A policy learned on one SUT can be transferred to other SUTs with similar resource sensitivity, reducing the computation time needed to find a performance breaking point.
  • On heterogeneous sets of SUTs, fixed exploration-exploitation settings are not enough; the tester must detect when a new SUT is dissimilar and increase exploration.
  • The efficiency gain depends on reusing the policy rather than merely exploring, since the homogeneous-set improvement grows when epsilon is lower and more policy reuse is allowed.
  • The reported sensitivity analysis indicates that learning parameters affect efficiency more on homogeneous sets than on heterogeneous sets, where adaptive epsilon keeps average trial counts roughly stable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The simulation, reward function, and similarity detection all use the same sensitivity vector, so part of the measured gain may come from a self-consistent loop; a real deployment would need independently measured sensitivities to confirm that transfer still pays off.
  • The same reuse-and-adapt mechanism could be extended to workload-based test conditions, such as request rate or concurrency, if a workload-sensitivity descriptor analogous to the resource sensitivity vector can be defined.
  • The similarity threshold of 0.8 and the epsilon values 0.2 and 0.5 are manually chosen; a fully autonomous tester would need to learn or tune these meta-parameters as well.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SaFReL, a self-adaptive fuzzy reinforcement learning framework that learns a policy for generating platform-based performance test cases (reductions of CPU, memory, and disk capacity) with the goal of reaching a performance breaking point. The framework uses fuzzy state detection over response time and resource utilization metrics, Q-learning with a modified update rule, and an adaptive epsilon-greedy strategy that adjusts exploration based on cosine similarity between sensitivity vectors of SUT instances. The approach is evaluated entirely on a self-built simulator: 50 homogeneous CPU-intensive SUT instances and 50 heterogeneous SUT instances, with a random 'typical stress testing' procedure as baseline. The paper reports an average computation-time improvement of about 42% on the homogeneous set with epsilon=0.2 and about 31% on the heterogeneous set with adaptive epsilon, and concludes that SaFReL generates performance test cases efficiently and adaptively without access to source code or performance models.

Significance. If the claimed results transfer to real systems, SaFReL would offer a genuinely useful capability: learning and reusing a test-generation policy for performance breaking points under varying resource configurations, without requiring source code or explicit system models. The paper contributes a fairly detailed algorithmic description of the framework, a reproducible simulation-based setup, and a sensitivity analysis of two learning parameters. The central claim, however, is currently supported only by experiments in which the agent's reward function, similarity detection, and simulator dynamics all consume the same sensitivity parameters. The significance of the claimed 42% and 31% efficiency gains therefore depends on breaking that circularity and on validating the simulator against real software behavior.

major comments (4)
  1. [Section 5, Eqs. 7 and 10; Section 7.1, Eq. 15; Section 8.3] The agent's learning signal and the environment model are constructed from the same sensitivity values: the reward function in Eq. (10) uses SenC, SenM, SenD, the similarity detection in Eq. (7) uses the same sensitivity vector, and the simulator in Eq. (15) uses the same values to determine throughput and response time. Consequently, the experiments hand the agent the key parameters of the environment model, so the reported gains may be an artifact of this coupling rather than evidence that the policy is learned without access to a performance model. Section 8.3 acknowledges that the dependency on sensitivity values 'is still considered as a source of threat' and states that 'their exact values are not necessary' in real deployment, but no experiment perturbs these values or estimates them from observations. The authors should either evaluate with inexact or estimated sensitivity values, vary them independently in training versus simulation, or test on real SUTs; without such evidence, the central claim of being model-free is not supported.
  2. [Section 7.2, Tables 4 and 6; Section 7.2.1] The efficiency comparison reports only point averages of learning trials, with no variance, confidence intervals, or number of repeated runs for either SaFReL or the random 'typical stress testing' baseline. Since the baseline is random, its mean is itself a random variable; a single realization comparing averages is not statistically meaningful. The authors should report distributions across repeated runs and ideally a paired statistical test over the same SUT instances.
  3. [Section 6, Eq. (13); Algorithm 2] The Q-learning update in Eq. (13) multiplies the entire new Q-value by the fuzzy membership degree mu. This is not the standard fuzzy Q-learning update, which applies the membership degree as a learning-rate modifier rather than as a multiplicative factor on the whole Q-value. With repeated updates this scaling can systematically shrink Q-values and may prevent the learned policy from converging to the optimal Q-function. The authors should justify this update rule or replace it with a standard fuzzy Q-learning formulation.
  4. [Section 7.2.1, Figure 6 and Table 3] The choice of epsilon=0.2 as the best action-selection strategy is made from the initial convergence results on the same type of SUT instances (CPU-intensive) that are then used to report the 42% improvement on the homogeneous set. This introduces a selection bias: the hyperparameter is tuned and evaluated on the same data. The evaluation should use held-out SUT instances or a separate tuning set to demonstrate that the reported improvement is not an artifact of in-sample selection.
minor comments (5)
  1. [Section 7.2.1, Figure 8 and Figure 12] There are typos in figure labels: 'Learning Trias' in Figure 8 and 'Learninhg' in Figure 12 should be corrected.
  2. [Section 7.2.1, paragraph II] The text contains a duplicated phrase 'to answer RQ1 and and also answer RQ2'; this should be edited to 'to answer RQ1 and also RQ2'.
  3. [Section 4.1, Fuzzy Rules] The paper states that 24 fuzzy rules are defined and that the remaining rules are defined 'similarly' to Rule 1, but the complete rule base is not provided. For reproducibility, the full set of rules and their consequent fuzzy states should be listed or included in an appendix.
  4. [Section 4.1, Fuzzification] The membership function ranges are said to be 'selected empirically,' but the manuscript does not specify their exact values or how they were chosen. Providing the actual functions (e.g., the parameters of the trapezoidal and triangular membership functions) would improve replicability.
  5. [Section 8.2] The spelling 'SaFRel' appears in the lessons-learned paragraph; the name should be consistently written as 'SaFReL'.

Circularity Check

2 steps flagged · score 5.0 of 10

Agent is handed the simulator's sensitivity parameters in both the reward and the similarity-based adaptation, so the 'model-free' efficiency gains are partly constructed.

  1. other [Section 5, Eq. 10; Section 7.1, Eq. 15]
    "UE_n = SenC CUI′_n + SenM MUI′_n + SenD DUI′_n (Eq. 10). ... The simulation module receives the resource sensitivity values ... estimates the program throughput using the following equation proposed by [42]: (Eq. 15)."

    The same sensitivity triple (SenC, SenM, SenD) appears both in the agent's reward (Eq. 10), which drives Q-learning, and in the environment's throughput equation (Eq. 15), which determines whether a breaking point is reached. The agent is therefore not learning 'without access to performance models': it is given the model's key parameter vector as a reward weight. Reducing the resource with the largest Sen value is rewarded and also produces the largest simulated throughput drop, so the policy that minimizes trials is encoded in the reward by construction. The comparison to random stress testing then measures an agent that is handed the simulator's answer key against one that is not.

  2. other [Section 5, Eq. 7; Section 7.2.1, Algorithm 3; Section 8.3]
    "The sensitivity vector contains the values of the sensitivity indicators of the SUT instance, SenC, SenM and SenD. ... SaFReL uses the approximate estimated similarity to tune the ε value adaptively. ... the performance simulation module requires the performance sensitivity values for the SUTs as we described in our experiments. ... it is still considered as a source of threat."

    The adaptive-epsilon results (31% improvement, Figs. 9-10) rely on Algorithm 3, which sets ε from cosine similarity between sensitivity vectors. Those vectors are the same values that parameterize the simulator (Table 2 from [42]). Thus the paper's adaptivity result, knowing when to explore vs. exploit, is computed from ground-truth knowledge of what makes the SUTs similar or different in the simulator, rather than discovered from interaction. The paper's own Section 8.3 concedes the dependency on sensitivity values is a validity threat. This is the same parameter-sharing reduction as the reward/simulator overlap, applied to the meta-learning layer.

full rationale

SaFReL's Q-learning machinery, fuzzy state modeling, and two-phase policy reuse are genuine algorithmic content, and the paper's formal derivation of the learning update is self-contained. The circularity burden is concentrated in the evaluation and in the central 'without access to performance models' claim. The agent's reward (Eq. 10) and its similarity-based adaptation (Eq. 7) are both built from the same sensitivity vector (SenC, SenM, SenD) that defines the simulated environment's throughput (Eq. 15, from external reference [42]). Consequently, the measured 42% and 31% improvements over random stress testing are not clean evidence for model-free operation: the agent is given the simulator's key parameters as reward weights and as the basis for deciding when to reuse a policy. The paper itself acknowledges in Section 8.3 that the dependency on performance sensitivity values is a threat to validity and that exact values would not be needed in a real deployment, but no experiment removes or perturbs them. Self-citations [34] and [35] are developmental and not load-bearing, and Eq. 15 is externally sourced, so the issue is not a self-citation chain. Because the method still has independent algorithmic content and the paper honestly discloses the dependency, the appropriate score is 5 rather than higher: the 'model-free' demonstration is partially constructed by the evaluation setup, but not every derived claim reduces to the inputs.

Assumptions & free parameters 7 free parameters · 5 assumptions · 2 invented entities

The ledger contains 7 free parameters, 5 axioms, and 2 invented entities. The most significant honesty issue is that the performance sensitivity values are shared between the reward function, the simulator, and the similarity check, so the agent's learning signal is aligned with the simulated environment by construction. The fuzzy membership ranges and rule base are also underspecified.

free parameters (7)
  • learning rate alpha = 0.1
    Set based on sensitivity analysis on the same simulated data (Section 7.2.2); no held-out validation.
  • discount factor gamma = 0.5
    Chosen in Section 7.2.1 and analyzed in Section 7.2.2; affects Q-value updates.
  • epsilon-greedy exploration parameters = epsilon=0.2, epsilon=0.5, decaying; adaptive thresholds 0.8/0.5/0.2
    Variant values selected based on fastest initial convergence in Figure 6; adaptive thresholds in Algorithm 3 set by hand.
  • reward weighting beta = not reported
    In Eq. 8 beta trades off response-time deviation vs. resource usage; no value is given in the experiments.
  • fuzzy membership function ranges = empirical, unspecified
    Section 4.1 states the ranges were selected empirically and could be updated; exact values not given.
  • performance sensitivity values (SenC, SenM, SenD) = 12 programs from [42] Table 2
    These values drive both the simulator (Eq. 15) and the reward (Eq. 10); they are taken from prior work, not measured in this paper.
  • similarity thresholds = 0.8, 0.8, 0.2, 0.5
    Algorithm 3 uses hard-coded thresholds for epsilon adjustment; no sensitivity analysis of these thresholds is reported.
assumptions (5)
  • standard math Q-learning converges to an optimal policy for a Markov decision process (Sutton and Barto [39])
    Invoked in Sections 2.1 and 6; the paper assumes the environment state is Markov and exploration converges.
  • domain assumption The throughput prediction model in Eq. 15 (from Taheri et al. [42]) accurately represents real program performance under resource changes
    Section 7.1; the entire evaluation is generated by this equation, so its validity determines whether the reported efficiency transfers to real SUTs.
  • domain assumption The performance sensitivity values in Table 2 are correct for the named programs
    Section 7.1; these values set both the simulated response and the reward signal, and are taken from [42] without verification.
  • ad hoc to paper Fuzzy membership function ranges chosen empirically are appropriate for the normalized metrics
    Section 4.1; no procedure or data is given for selecting the ranges, and they affect state detection and Q-updates.
  • ad hoc to paper The 24 fuzzy rules defined 'similarly' to Rule 1 capture the relevant state space
    Section 4.1; the rule base is not fully enumerated, and its completeness is assumed.
invented entities (2)
  • Fuzzy state representation (e.g., HHLN)
    purpose: Soft state labels for the RL environment
    Introduced in Section 4.1; internal to the framework with no external falsifiable handle.
  • Performance sensitivity vector SV
    purpose: Input to similarity detection for adaptive epsilon selection
    Introduced in Section 5; values are taken from prior work and not independently validated, and both the reward and simulator depend on them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Autonomous Performance Testing Framework using Self-Adaptive Fuzzy Reinforcement Learning." pith.science (2026). https://pith.science/paper/CYTVV7QZ

@misc{pith2026190806900,
  author       = {Pith},
  title        = {Pith review of: An Autonomous Performance Testing Framework using Self-Adaptive Fuzzy Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYTVV7QZ}},
  note         = {Machine review of arXiv:1908.06900}
}
read the original abstract

Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments on a simulated environment, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process, and performs adaptively without access to source code and performance models.

Figures

Figures reproduced from arXiv: 1908.06900 by the authors.

Figure 1
Figure 1. Interaction between agent and SUT in RL 3 Architecture This section provides an overview of the architecture of the proposed smart performance testing framework, SaFReL (see [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. SaFReL architecture parts together with the fuzzy rules, fuzzy operators, and the implication method that are used, are described in Section 4.1. 4.1 State Modeling and Fuzzy Inference Normalization. As described in the previous section, a set of quality measurements, CPU, memory, and disk utilization improvements and response time of the SUT, represent the state of the environ￾ment. The values of these measurements… view at source ↗
Figure 3
Figure 3. Fuzzy representation of quality measurements [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Fuzzy states of the environment They are intended to provide a right trade-off between exploration of the state action space and exploitation of the learned policy. In SaFReL, we use ε-greedy as the action selection strategy and the proposed strategy adaptation feature…
Figure 5
Figure 5. Figure 5: Implementation structure [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Initial convergence of SaFReL in 100 learning episodes during the initial learning [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Efficiency of SaFReL on a homogeneous set of SUTs in the transfer learning [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Efficiency of SaFReL on a heterogeneous set of SUTs regarding the use of typical config [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Efficiency of SaFReL on a heterogeneous set of SUTs regarding the use of adaptive [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Adaptivity of SaFReL on a heterogeneous set of SUTs regarding the use of different [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Sensitivity of SaFReL to learning rate and discount factor on the homogeneous set of [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Sensitivity of SaFReL to learning rate and discount factor on the heterogeneous set of [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 63 canonical work pages

  1. [1]

    Did You Know A Slow Webpage Can Cost You 7% of Your Sales,

    NS8. Did You Know A Slow Webpage Can Cost You 7% of Your Sales,

    work page

  2. [2]

    Experience with performance testing of software systems: issues, an approach, and case study

    Elaine J Weyuker and Filippos I V okolos. Experience with performance testing of software systems: issues, an approach, and case study. IEEE transactions on software engineering , 26(12):1147–1156, 2000

    work page 2000

  3. [3]

    Performance-oriented DevOps: A Research Agenda

    Andreas Brunnert, Andr ´e van Hoorn, Felix Willnecker, Alexandru Danciu, Wilhelm Hassel- bring, Christoph Heger, Nikolas Herbst, Pooyan Jamshidi, Reiner Jung, Joakim von Kistowski, et al. Performance-oriented devops: A research agenda. arXiv preprint arXiv:1508.04752 , 2015

    work page Pith review arXiv 2015

  4. [4]

    Solving enterprise applications performance puzzles: queuing models to the rescue

    Leonid Grinshpan. Solving enterprise applications performance puzzles: queuing models to the rescue. John Wiley & Sons, 2012

    work page 2012

  5. [5]

    ISO/IEC 25010 - System and software quality models, 2019

    ISO 25000. ISO/IEC 25010 - System and software quality models, 2019. Available athttps: //iso25000.com/index.php/en/iso-25000-standards/iso-25010 , Re- trieved July, 2019

    work page 2019

  6. [6]

    On non-functional requirements

    Martin Glinz. On non-functional requirements. In 15th IEEE International Requirements Engineering Conference (RE 2007), pages 21–26. IEEE, 2007

    work page 2007

  7. [7]

    Springer Science & Business Media, 2012

    Lawrence Chung, Brian A Nixon, Eric Yu, and John Mylopoulos.Non-functional requirements in software engineering, volume 5. Springer Science & Business Media, 2012

    work page 2012

  8. [8]

    A survey on load testing of large-scale software systems

    Zhen Ming Jiang and Ahmed E Hassan. A survey on load testing of large-scale software systems. IEEE Transactions on Software Engineering, 41(11):1091–1118, 2015

    work page 2015

Show all 65 references
  1. [9]

    Model-based software perfor- mance analysis

    Vittorio Cortellessa, Antinisca Di Marco, and Paola Inverardi. Model-based software perfor- mance analysis. Springer Science & Business Media, 2011

    2011

  2. [10]

    Performance modeling and design of computer systems: queueing theory in action

    Mor Harchol-Balter. Performance modeling and design of computer systems: queueing theory in action. Cambridge University Press, 2013

    2013

  3. [11]

    Introduction to computer system performance evaluation

    Krishna Kant and MM Srinivasan. Introduction to computer system performance evaluation . McGraw-Hill College, 1992

    1992

  4. [12]

    Early performance testing of dis- tributed software applications

    Giovanni Denaro, Andrea Polini, and Wolfgang Emmerich. Early performance testing of dis- tributed software applications. In ACM SIGSOFT Software Engineering Notes , volume 29, pages 94–103. ACM, 2004. 28

    2004

  5. [13]

    Compositional load test genera- tion for software pipelines

    Pingyu Zhang, Sebastian Elbaum, and Matthew B Dwyer. Compositional load test genera- tion for software pipelines. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, pages 89–99. ACM, 2012

    2012

  6. [14]

    Automated test case generation for the stress testing of multimedia systems

    Jian Zhang and Shing Chi Cheung. Automated test case generation for the stress testing of multimedia systems. Software: Practice and Experience, 32(15):1411–1435, 2002

    2002

  7. [15]

    Search-based performance testing of applications with composite services

    Yuanyan Gu and Yujia Ge. Search-based performance testing of applications with composite services. In 2009 International Conference on Web Information Systems and Mining , pages 320–324. IEEE, 2009

    2009

  8. [16]

    Search-based testing of service level agreements

    Massimiliano Di Penta, Gerardo Canfora, Gianpiero Esposito, Valentina Mazza, and Marcello Bruno. Search-based testing of service level agreements. In Proceedings of the 9th annual conference on Genetic and evolutionary computation, pages 1090–1097. ACM, 2007

    2007

  9. [17]

    A genetic algorithm-based stress test requirements generator tool and its em- pirical evaluation

    Vahid Garousi. A genetic algorithm-based stress test requirements generator tool and its em- pirical evaluation. IEEE Transactions on Software Engineering, 36(6):778–797, 2010

    2010

  10. [18]

    Empirical analysis of a genetic algorithm-based stress test technique

    Vahid Garousi. Empirical analysis of a genetic algorithm-based stress test technique. In Pro- ceedings of the 10th annual conference on Genetic and evolutionary computation, pages 1743–

  11. [19]

    Traffic-aware stress testing of distributed real-time systems based on uml models using genetic algorithms

    Vahid Garousi, Lionel C Briand, and Yvan Labiche. Traffic-aware stress testing of distributed real-time systems based on uml models using genetic algorithms. Journal of Systems and Software, 81(2):161–185, 2008

    2008

  12. [20]

    Generating performance test scripts and scenarios based on abstract intermediate models

    Leandro T Costa, Ricardo M Czekster, Fl ´avio Moreira de Oliveira, Elder de M Rodrigues, Maicon Bernardino da Silveira, and Avelino F Zorzo. Generating performance test scripts and scenarios based on abstract intermediate models. In SEKE, pages 112–117, 2012

    2012

  13. [21]

    Generation of scripts for performance testing based on uml models

    Maicon Bernardino da Silveira, Elder de M Rodrigues, Avelino F Zorzo, Leandro T Costa, Hugo V Vieira, and Fl´avio Moreira de Oliveira. Generation of scripts for performance testing based on uml models. In SEKE, pages 258–263, 2011

    2011

  14. [22]

    Realistic load testing of web applications

    Dirk Draheim, John Grundy, John Hosking, Christof Lutteroth, and Gerald Weber. Realistic load testing of web applications. In Conference on Software Maintenance and Reengineering (CSMR’06), pages 11–pp. IEEE, 2006

    2006

  15. [23]

    Modeling a realistic workload for performance testing

    Christof Lutteroth and Gerald Weber. Modeling a realistic workload for performance testing. In 2008 12th International IEEE Enterprise Distributed Object Computing Conference, pages 149–158. IEEE, 2008

    2008

  16. [24]

    Behavior-driven load testing using contextual knowledge-approach and experiences

    Henning Schulz, Du ˇsan Okanovi ´c, Andr ´e van Hoorn, Vincenzo Ferme, and Cesare Pau- tasso. Behavior-driven load testing using contextual knowledge-approach and experiences. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering , pages 265–2...

    2019

  17. [25]

    A declarative approach for performance tests execution in continuous software development environments

    Vincenzo Ferme and Cesare Pautasso. A declarative approach for performance tests execution in continuous software development environments. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering, pages 261–272. ACM, 2018. 29

    2018

  18. [26]

    Towards holistic continuous software performance as- sessment

    Vincenzo Ferme and Cesare Pautasso. Towards holistic continuous software performance as- sessment. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion, pages 159–164. ACM, 2017

    2017

  19. [27]

    Ask- ing what?, automating the how?: The vision of declarative performance engineering

    J ¨urgen Walter, Andre van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. Ask- ing what?, automating the how?: The vision of declarative performance engineering. In Pro- ceedings of the 7th ACM/SPEC on International Conference on Performance Engineering , pages 91–9...

    2016

  20. [28]

    Mission-critical and safety-critical systems handbook: Design and development for embedded applications

    Kim Fowler. Mission-critical and safety-critical systems handbook: Design and development for embedded applications. Newnes, 2009

    2009

  21. [29]

    Resource management in clouds: Survey and research challenges

    Brendan Jennings and Rolf Stadler. Resource management in clouds: Survey and research challenges. Journal of Network and Systems Management, 23(3):567–619, 2015

    2015

  22. [30]

    Ernest: efficient performance prediction for large-scale advanced analytics

    Shivaram Venkataraman, Zongheng Yang, Michael Franklin, Benjamin Recht, and Ion Stoica. Ernest: efficient performance prediction for large-scale advanced analytics. In13th{USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 16), pages 363–378, 2016

    2016

  23. [31]

    Tradeoffs in modeling performance of highly configurable software systems

    Sergiy Kolesnikov, Norbert Siegmund, Christian K ¨astner, Alexander Grebhahn, and Sven Apel. Tradeoffs in modeling performance of highly configurable software systems. Software & Systems Modeling, 18(3):2265–2283, 2019

    2019

  24. [32]

    Virtualization on internet of things edge devices with container technolo- gies: a performance evaluation

    Roberto Morabito. Virtualization on internet of things edge devices with container technolo- gies: a performance evaluation. IEEE Access, 5:8835–8850, 2017

    2017

  25. [33]

    Web performance evaluation for internet of things applications

    Zoran B Babovic, Jelica Protic, and Veljko Milutinovic. Web performance evaluation for internet of things applications. IEEE Access, 4:6974–6992, 2016

    2016

  26. [34]

    Machine learning to guide performance testing: An autonomous test framework

    Mahshid Helali Moghadam, Mehrdad Saadatmand, Markus Borg, Markus Bohlin, and Bj ¨orn Lisper. Machine learning to guide performance testing: An autonomous test framework. In 2019 IEEE International Conference on Software Testing, Verification and Validation Work- shops (ICSTW), ...

    2019

  27. [35]

    Adaptive service performance control using cooperative fuzzy reinforcement learning in virtualized environments

    Olumuyiwa Ibidunmoye, Mahshid Helali Moghadam, Ewnetu Bayuh Lakew, and Erik Elm- roth. Adaptive service performance control using cooperative fuzzy reinforcement learning in virtualized environments. In Proceedings of the10th International Conference on Utility and Cloud Compu...

    2017

  28. [36]

    Performance anomaly detection and bottleneck identification

    Olumuyiwa Ibidunmoye, Francisco Hern ´andez-Rodriguez, and Erik Elmroth. Performance anomaly detection and bottleneck identification. ACM Computing Surveys (CSUR) , 48(1):4, 2015

    2015

  29. [37]

    Anomaly detection: A survey

    Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):15, 2009

    2009

  30. [38]

    Systems performance: enterprise and the cloud

    Brendan Gregg. Systems performance: enterprise and the cloud. Pearson Education, 2013

    2013

  31. [39]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  32. [40]

    Fuzzy classifiers

    Ludmila I Kuncheva. Fuzzy classifiers. Scholarpedia, 3(1):2925, 2008. 30

    2008

  33. [41]

    Fuzzy Inference Process, 2019

    MathWorks. Fuzzy Inference Process, 2019. Available at https://www.mathworks. com/help/fuzzy/fuzzy-inference-process.html, Retrieved July, 2019

    2019

  34. [42]

    vmbbthrpred: A black-box throughput predictor for virtual machines in cloud environments

    Javid Taheri, Albert Y Zomaya, and Andreas Kassler. vmbbthrpred: A black-box throughput predictor for virtual machines in cloud environments. In European Conference on Service- Oriented and Cloud Computing, pages 18–33. Springer, 2016

    2016

  35. [43]

    Load testing, benchmarking, and application performance management for the web

    Daniel A Menasc ´e. Load testing, benchmarking, and application performance management for the web. In Int. CMG Conference, pages 271–282, 2002

    2002

  36. [44]

    Tools for continuously evaluating distributed system qualities

    James Hill, D Schmidt, James Edmondson, and Aniruddha Gokhale. Tools for continuously evaluating distributed system qualities. IEEE software, 27(4):65–71, 2009

    2009

  37. [45]

    Autoperf: Automated load testing and resource usage profiling of multi-tier internet appli- cations

    Varsha Apte, TVS Viswanath, Devidas Gawali, Akhilesh Kommireddy, and Anshul Gupta. Autoperf: Automated load testing and resource usage profiling of multi-tier internet appli- cations. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, pag...

    2017

  38. [46]

    Cloud- perf: A performance test framework for distributed and dynamic multi-tenant environments

    Nicolas Michael, Nitin Ramannavar, Yixiao Shen, Sheetal Patil, and Jan-Lung Sung. Cloud- perf: A performance test framework for distributed and dynamic multi-tenant environments. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, pages 1...

    2017

  39. [47]

    Performance modeling for cloud microservice applications

    Anshul Jindal, Vladimir Podolskiy, and Michael Gerndt. Performance modeling for cloud microservice applications. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, pages 25–32. ACM, 2019

    2019

  40. [48]

    Stress testing real-time systems with genetic algorithms

    Lionel C Briand, Yvan Labiche, and Marwa Shousha. Stress testing real-time systems with genetic algorithms. In Proceedings of the 7th annual conference on Genetic and evolutionary computation, pages 1021–1028. ACM, 2005

    2005

  41. [49]

    Automatic generation of load tests

    Pingyu Zhang, Sebastian Elbaum, and Matthew B Dwyer. Automatic generation of load tests. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pages 43–52. IEEE Computer Society, 2011

    2011

  42. [50]

    One size does not fit all: In-test workload adaptation for performance testing of enterprise applications

    Vanessa Ayala-Rivera, Maciej Kaczmarski, John Murphy, Amarendra Darisa, and A Omar Portillo-Dominguez. One size does not fit all: In-test workload adaptation for performance testing of enterprise applications. In Proceedings of the 2018 ACM/SPEC International Con- ference on Pe...

    2018

  43. [51]

    Towards a structural load testing tool

    Cheer-Sun D Yang and Lori L Pollock. Towards a structural load testing tool. In ACM SIG- SOFT Software Engineering Notes, volume 21, pages 201–208. ACM, 1996

    1996

  44. [52]

    A model-based approach for test- ing the performance of web applications

    Mahnaz Shams, Diwakar Krishnamurthy, and Behrouz Far. A model-based approach for test- ing the performance of web applications. In Proceedings of the 3rd international workshop on Software quality assurance, pages 54–61. ACM, 2006

    2006

  45. [53]

    Christian V ¨ogele, Andr´e van Hoorn, Eike Schulz, Wilhelm Hasselbring, and Helmut Krcmar. Wessbas: extraction of probabilistic workload specifications for load testing and performance predictiona model-driven approach for session-based application systems.Software & Systems Mo...

    2018

  46. [54]

    Generating workload for erp applica- tions through end-user organization categorization using high level business operation data

    Gururaj Maddodi, Slinger Jansen, and Rolf de Jong. Generating workload for erp applica- tions through end-user organization categorization using high level business operation data. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering , pages ...

    2018

  47. [55]

    Identifying performance deviations in thread pools

    Mark D Syer, Bram Adams, and Ahmed E Hassan. Identifying performance deviations in thread pools. In 2011 27th IEEE International Conference on Software Maintenance (ICSM), pages 83–92. IEEE, 2011

    2011

  48. [56]

    Reliability estimation for large distributed software systems

    Alberto Avritzer, Fl ´avio P Duarte, Rosa Maria Meri Leao, Edmundo de Souza e Silva, Michal Cohen, and David Costello. Reliability estimation for large distributed software systems. In Cascon, page 12. Citeseer, 2008

    2008

  49. [57]

    Automatic detection of performance deviations in the load testing of large scale systems

    Haroon Malik, Hadi Hemmati, and Ahmed E Hassan. Automatic detection of performance deviations in the load testing of large scale systems. In Proceedings of the 2013 International Conference on Software Engineering, pages 1012–1021. IEEE Press, 2013

    2013

  50. [58]

    Automatic comparison of load tests to support the performance analysis of large enterprise systems

    Haroon Malik, Zhen Ming Jiang, Bram Adams, Ahmed E Hassan, Parminder Flora, and Gilbert Hamann. Automatic comparison of load tests to support the performance analysis of large enterprise systems. In 2010 14th European conference on software maintenance and reengi- neering, pag...

    2010

  51. [59]

    Auto-scale: automatic scaling of virtualised resources using neuro-fuzzy reinforcement learning approach

    T Veni and S Mary Saira Bhanu. Auto-scale: automatic scaling of virtualised resources using neuro-fuzzy reinforcement learning approach. International Journal of Big Data Intelligence, 3(3):145–153, 2016

    2016

  52. [60]

    Fuzzy self-learning controllers for elasticity management in dynamic cloud ar- chitectures

    Pooyan Jamshidi, Amir Sharifloo, Claus Pahl, Hamid Arabnejad, Andreas Metzger, and Gio- vani Estrada. Fuzzy self-learning controllers for elasticity management in dynamic cloud ar- chitectures. In 2016 12th International ACM SIGSOFT Conference on Quality of Software Architectur...

    2016

  53. [61]

    Adaptive runtime response time control in plc-based real-time systems using reinforce- ment learning

    Mahshid Helali Moghadam, Mehrdad Saadatmand, Markus Borg, Markus Bohlin, and Bj ¨orn Lisper. Adaptive runtime response time control in plc-based real-time systems using reinforce- ment learning. In 2018 IEEE/ACM 13th International Symposium on Software Engineering for Adaptive...

    2018

  54. [62]

    Pyse: Automatic worst- case test generation by reinforcement learning

    Jinkyu Koo, Charitha Saumya, Milind Kulkarni, and Saurabh Bagchi. Pyse: Automatic worst- case test generation by reinforcement learning. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pages 136–147. IEEE, 2019

    2019

  55. [63]

    Exploratory performance testing using reinforcement learning

    Tanwir Ahmad, Adnan Ashraf, Dragos Truscan, and Ivan Porres. Exploratory performance testing using reinforcement learning. In 2019 45th Euromicro Conference on Software Engi- neering and Advanced Applications (SEAA), pages 156–163. IEEE, 2019

    2019

  56. [64]

    Automatically finding performance problems with feedback-directed learning software testing

    Mark Grechanik, Chen Fu, and Qing Xie. Automatically finding performance problems with feedback-directed learning software testing. In 2012 34th International Conference on Soft- ware Engineering (ICSE), pages 156–166. IEEE, 2012. 32

    2012

  57. [2018]

    Available at https://www.ns8.com/en/ns8u/did-you-know/ a-slow-webpage-can-cost-you-7-percent-of-your-sales , Retrieved July, 2019

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.