Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Black-Box Bug-Amplification for Multithreaded Software

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Black-box bug amplification: a learned ensemble of classifiers substantially raises how often rare concurrency bugs surface under a fixed execution budget.

desk verdict Framing is nice, but the evaluation confounds screening density with learning and the numbers don't reconcile; worth a reviewer's time with a demand for controls. read the letter →

arxiv 2507.21318 v1 pith:G2AP6RZD submitted 2025-07-28 cs.SE

Yeshayahu Weiss , Gal Amram , Achiya Elyasaf , Eitan Farchi , Oded Margalit , Gera Weiss This is my paper · ORCID
classification cs.SE
keywords concurrencybugsbugamplificationHeisenbugsblack-boxtestingensemblelearningrare-eventregressionmodel-guidedtestgenerationmultithreadedsoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a purely black-box testing strategy can substantially amplify the occurrence of rare concurrency bugs within a fixed execution budget. Instead of reading source code or controlling the thread scheduler, the method varies input parameters such as delays and workload settings, runs the system repeatedly, and trains a predictive model on the observed failures. The model then ranks future inputs so testing effort concentrates where failures are most likely. On 17 benchmark concurrency bugs, an ensemble of classifiers reportedly outperforms random sampling, simulated annealing, and a genetic algorithm across nearly all problems, often by an order of magnitude. If the claim holds, elusive Heisenbugs become much cheaper to expose without instrumentation or architectural changes.

What carries the argument

The central object is the ensemble stacking classifier over the space of delay parameters: four base learners (logistic regression, decision tree, random forest, and a multilayer perceptron) each return a failure probability, and a logistic-regression meta-learner combines those probabilities with the raw features using out-of-fold predictions and an oversampling step to handle extreme class imbalance. It is paired with a generator-based simulator in which each thread yields virtual delays and the scheduler picks the thread with the earliest wake-up time, so interleavings are determined by the delay parameters. Those delay parameters are the inputs the model learns on, and repeated runs with different random seeds supply the noisy failure-rate labels that drive the search.

What would settle it

Take a benchmark bug whose failure probability is essentially flat across all allowed delay configurations, with the trigger controlled only by the random noise in the delay draws, and run the ensemble-guided search against random sampling; if the learned model cannot beat random on that flat landscape, the parameter-sensitivity premise fails. A less extreme check is to measure whether the ensemble's advantage shrinks as the input variables are made progressively less correlated with failure.

Watch

Extended reading notes

Core claim

The central claim is that bug amplification can be reformulated as a rare-event regression problem solvable in a black-box manner: every input configuration is a point whose failure probability is estimated from repeated trial executions, and a stacked ensemble of classifiers trained on those noisy labels identifies high-probability regions of the input space. On a curated benchmark of 17 concurrency bugs spanning deadlocks, unexpected data, and mutual-exclusion violations, the ensemble method is the strongest overall, reaching average success probabilities above 0.5 after only 500 test-cases and near-perfect detection on more than half of the problems by 3,900 test-cases, while the brute-force baseline averages below 0.15. The authors present this as evidence that learning-guided, feedback-driven search is more sample-efficient and more reliable than uninformed sampling.

Load-bearing premise

The method assumes that a bug's chance of showing up is meaningfully controlled by the input parameters exposed to the test generator, so bugs that depend only on internal scheduler decisions or deep state interactions give it little or no leverage.

Editorial extensions

If this is right

  • With a fixed execution budget, model-guided search can expose substantially more bug instances than brute-force random testing, often by an order of magnitude.
  • Useful inputs emerge within the first few hundred executions, so the approach suits settings where each test run is expensive.
  • Because the method needs no source access or instrumentation, it can be applied to systems where white-box analysis is infeasible.
  • Ranked top-k inputs are also better than random top-k inputs, giving debugging and fault-localization pipelines a more useful set of candidates.
  • The approach complements schedule-level techniques such as systematic concurrency testing and randomized schedulers rather than replacing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to combine input-level amplification with schedule perturbation, since the two levers are largely orthogonal; the paper's own simulator could test whether the gains compound.
  • The evidence is gathered entirely inside a generator-based simulator, so the strongest next test is to run the same black-box pipeline against real multithreaded programs by varying externally visible parameters such as thread counts, environment variables, or timing knobs.
  • The retraining overhead the paper flags suggests an active-learning or Bayesian-optimization variant that spends fewer executions per model update could make the method viable at industrial scale.
  • The ranking the model produces could double as a crude fault-localization report, showing which timing or workload parameters most strongly drive a bug's appearance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a black-box method for amplifying the occurrence of rare concurrency bugs. The system under test (SUT) is treated as a black box; test inputs are vectors of delay/timing parameters, and repeated trial executions produce noisy binary labels. An ensemble stacking classifier is trained on these executions and used to rank and select test inputs within a fixed execution budget. The method is evaluated on 17 Python-generator-based concurrency bug simulations, comparing brute-force random search (BF), simulated annealing (SA), a genetic algorithm (GA), and the ensemble classifier (Ens). The central claim, stated in the abstract and Section 7.6, is that the ensemble model significantly increases bug-triggering probability compared with uninformed random sampling, often by an order of magnitude.

Significance. If the central claim were established, the approach would be a practical, non-invasive aid for exposing concurrency faults, and the released benchmark and framework would be a useful community resource. The paper has several strengths: it reports 50 independent trials per method--problem pair, uses one-sided Wilcoxon tests, provides code and data in a public repository, and includes a candid limitations section. However, the evaluation as presented does not isolate the effect of learning from the number of distinct candidates screened under the fixed budget, and the reported numerical results are internally inconsistent across sections. These issues currently prevent the significance of the claimed result from being assessed reliably.

major comments (4)
  1. [Sections 6.1, 6.4, 7.1, Table 2] The headline comparison between Ens and BF is confounded by candidate-count asymmetry. Under a budget B, BF (Section 6.1) fixes k=30 and therefore screens only B/k distinct candidates (about 130 at B=3900), whereas Ens (Section 6.4 and Table 2) evaluates roughly one execution per candidate over about B distinct inputs (100 random plus 100 model-ranked per iteration). The best-of-n advantage alone can make Ens look superior even without any learning. The paper does not include an uninformed control that screens the same number of distinct candidates (for example, B one-shot random evaluations followed by re-execution of the top candidates). Without such a control, the claimed learning-based amplification in Section 7.6 is not isolated from the effect of screening more candidates. Please add a same-candidate-count random baseline and compare against it.
  2. [Sections 7.1, 7.5, and 7.6] The paper reports mutually inconsistent success probabilities for what appear to be the same quantities. At 500 test cases, Section 7.1 gives mean success probabilities of 0.68 for Ens, 0.17 for BF, 0.24 for GA, and 0.04 for SA, while Section 7.5 gives 51.8%, 3.1%, 8.1%, and 1.5%, and Section 7.6 states that Ens exceeds 0.53 while the other three remain below 0.13. At 3900 test cases the discrepancies are similarly large (0.87/0.46/0.39/0.11 in Section 7.1 versus 59.8%/13.6%/17.3%/3.9% in Section 7.5). These numbers cannot all describe the same metric. The authors must clarify the exact definition of the reported probability in each section and correct the inconsistencies.
  3. [Section 7.4, Table 3] The text states that the table comprises 68 directional pairwise comparisons (17 problems × 4 method pairs) and then says that only 14 of the 102 comparisons are inconclusive. The table actually has six method-pair columns, giving 17 × 6 = 102 comparisons. The relationship between 68 and 102 is unclear and must be corrected, along with the statement about the number of method pairs used.
  4. [Sections 4, 9, and 10] The benchmark is entirely synthetic: all 17 problems are Python generator simulations based on The Deadlock Empire, and the only exposed inputs are delay parameters. Section 1 states that these parameters were tuned so that failures occur with low probability under default settings, and Section 10 concedes that the method assumes bug manifestation is meaningfully influenced by the exposed input parameters. Because the benchmark is constructed to satisfy that assumption, the experiments do not test the premise on real or naturally occurring multithreaded systems. The paper should either include at least one real SUT for which parameter sensitivity is not hand-tuned, or substantially soften the general practitioner-facing claims in the abstract and Section 7.6.
minor comments (5)
  1. [Abstract and Section 6.4] The abstract refers to 'an ensemble of regression models,' while Section 6.4 describes a stacking classifier; the terminology should be aligned throughout.
  2. [Section 7.4, Table 3 caption] The caption contains the typo 'aech' where 'each' is intended.
  3. [Section 6.4] The paragraph 'Observations and Rationale' asserts that the ensemble classifier 'consistently demonstrated reliable and accurate predictions' without presenting quantitative support in that section; this claim should either be moved to the Results section or supported with evidence.
  4. [References] Several references have incomplete venue or publication information, including [1], [5], [14], and [23]; these should be completed.
  5. [Figures 2 and 5] The captions of Figures 2 and 5 refer to '4 (out of 20) test-cases' and '3 (out of 17) problems' in a way that is confusing; please clarify what is displayed (budget levels versus problems).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the core comparison is a standard black-box ML pipeline with held-out evaluation; the only self-citation is minor and non-load-bearing.

full rationale

The paper's central claim is empirical: the ensemble is trained on black-box SUT executions and its selected test-cases are scored by massive independent re-execution, not by the model's own predictions (Sections 5.1 and 6.4). There is no fitted constant that is later reported as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The load-bearing assumption is stated explicitly in Section 10 ('Our method assumes that the probability of bug manifestation is meaningfully influenced by the input parameters exposed to the test generation engine'), which is an honest limitation rather than a circular definition. The only self-citation is reference [13], used in Section 6.3 to motivate a genetic-programming variant; that variant failed to converge and was abandoned, so the citation is not load-bearing for the paper's conclusions. A methodological concern exists but is not circularity: BF evaluates only B/k distinct candidates while Ens screens roughly B distinct candidates (Table 2), so part of Ens's advantage may reflect screening density rather than learning; this is a validity/fairness caveat, not a reduction of the prediction to its inputs. Reported numerical inconsistencies between Sections 7.1, 7.5 and 7.6 are correctness risks, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a handful of hand-chosen parameters and domain assumptions. The most consequential free parameters are the per-benchmark choice of which delay parameters to expose (Section 4) and the ensemble configuration selected after experimentation (Section 6.4). The key assumptions are that the simulator faithfully models concurrency and that bug probability is learnable from the exposed inputs.

free parameters (4)
  • k (fitness repetitions) = 30
    Number of SUT executions per candidate for BF and SA; chosen by authors citing LLN/CLT. Affects the comparison: BF spends budget on precise estimates of few candidates, Ens uses single runs of many candidates.
  • SA neighborhood radius epsilon = 0.1 (initial)
    Initial radius of sampling ball for simulated annealing; annealing schedule described only qualitatively in Section 6.2.
  • GA hyperparameters = pop=50, crossover=0.5, mutation=0.15, tournament=4
    Chosen based on standard practice (Goldberg 1989, Karafotias et al. 2015), not from the data. Tuning of these could affect relative performance.
  • Ensemble model configuration = Layer-1 LR/DT/RF/MLP, meta-learner LR with passthrough, SMOTE, cv=5
    The authors state the configuration was found 'after experimentation' (Section 6.4), i.e., selected by performance on the benchmark, a form of fitting to the evaluation set.
assumptions (4)
  • domain assumption The generator-based delay simulation (Listing 1, 2) faithfully represents multithreaded execution interleavings.
    Section 5: thread steps are atomic operations separated by yields; scheduling is by lowest wake-up time. Real systems have preemption, memory models, and OS scheduling that this may not capture.
  • domain assumption Bug manifestation probability is a learnable function of the exposed input delay parameters.
    Section 10: 'Our method assumes that the probability of bug manifestation is meaningfully influenced by the input parameters exposed to the test generation engine.' If false, the entire approach fails.
  • domain assumption The 17 benchmark programs are representative of real-world concurrency bugs.
    Section 4: benchmark drawn from Deadlock Empire puzzles plus custom variants; the paper asserts representativeness but all are toy simulations.
  • standard math Central Limit Theorem heuristic justifies k=30 repetitions for stable estimates.
    Section 6.1 invokes LLN and CLT; this is standard statistical reasoning, though it understates the variance for skewed Bernoulli failure indicators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Black-Box Bug-Amplification for Multithreaded Software." pith.science (2026). https://pith.science/paper/G2AP6RZD

@misc{pith2026250721318,
  author       = {Pith},
  title        = {Pith review of: Black-Box Bug-Amplification for Multithreaded Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2AP6RZD}},
  note         = {Machine review of arXiv:2507.21318}
}
read the original abstract

Bugs, especially those in concurrent systems, are often hard to reproduce because they manifest only under rare conditions. Testers frequently encounter failures that occur only under specific inputs, even when occurring with low probability. We propose an approach to systematically amplify the occurrence of such elusive bugs. We treat the system under test as a black-box and use repeated trial executions to train a predictive model that estimates the probability of a given input configuration triggering a bug. We evaluate this approach on a dataset of 17 representative concurrency bugs spanning diverse categories. Several model-based search techniques are compared against a brute-force random sampling baseline. Our results show that an ensemble of regression models can significantly increase bug occurrence rates across nearly all scenarios, often achieving an order-of-magnitude improvement over random sampling. The contributions of this work include: (i) a novel formulation of bug-amplification as a rare-event regression problem; (ii) an empirical evaluation of multiple techniques for amplifying bug occurrence, demonstrating the effectiveness of model-guided search; and (iii) a practical, non-invasive testing framework that helps practitioners expose hidden concurrency faults without altering the internal system architecture.

Figures

Figures reproduced from arXiv: 2507.21318 by the authors.

Figure 1
Figure 1. Geometric intuition in 2D of the update step. The k candidates are sampled inside the dotted ball B(u,ε) centered at u. We move from u to unext = t [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Bird’s-eye view for all problems, probability of triggering bug after 500, 1100, 2100, and 3900 test-cases. Each bar is one experiment and based on 50 independent runs. The X axis is all 17 problems, and for each problem, 4 methods and 4 (out of 20) test-cases are shown. The y-axis is the maximum probability for the best test-case [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Aggregated performance comparison of four methods across all 17 benchmark concurrency problems. The x-axis shows the number of test-cases used in each evaluation, and the y-axis shows the average fault￾triggering probability. For each method, the curve represents the mean of the mean of the best test-case’s fault-triggering probability across all problems. 0 1000 2000 3000 4000 0 0.1 0.2 Probability Shared Flag 0 10… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Bug-detection rates across three benchmark problems with different detectability levels. Based on 50 runs; error bars = SD. 7.3. Top-k Case Effectiveness This section compares the performance of Ens compare to the BF method when selecting the top-5 and top-10 best test…
Figure 5
Figure 5. Figure 5: The 5 th and 10th best test-cases probability. In three detectability levels of problems that cover three ranges of probability: Shared Flag (low); Atomicity Bypass (medium), and Race-To-Wait (high). Each bar is one experiment and based on 50 independent runs. The X ax…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 54 canonical work pages

  1. [1]

    Why Do Computers Stop and What Can Be Done About It?

    Jim Gray. Why Do Computers Stop and What Can Be Done About It?. , 1985

    work page 1985

  2. [2]

    Intermittent Failures in Hardware and Software

    Bakhshi, Roozbeh and Kunche, Surya and Pecht, Michael. Intermittent Failures in Hardware and Software. Journal of Electronic Packaging, 2014. 33 of 35

    work page 2014

  3. [3]

    Finding and Reproducing Heisenbugs in Concurrent Programs

    Madanlal Musuvathi and Shaz Qadeer and Thomas Ball and Gerard Basler and Piramanayagam Arumuga Nainar and Iulian Neamtiu. Finding and Reproducing Heisenbugs in Concurrent Programs. 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2008), 2008

    work page 2008

  4. [4]

    A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs

    Sebastian Burckhardt and Pravesh Kothari and Madanlal Musuvathi and Santosh Nagarakatte. A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs. 15th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’10), 2010

    work page 2010

  5. [5]

    Levin and David A

    Patrice Godefroid and Michael Y. Levin and David A. Molnar. Effective Testing for Concurrency Bugs. , 2015

    work page 2015

  6. [6]

    Necula and Koushik Sen

    Tayfun Elmas and Jacob Burnim and George C. Necula and Koushik Sen. CONCURRIT: A Domain Specific Language for Reproducing Concurrency Bugs. 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13), 2013

    work page 2013

  7. [7]

    DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization

    Xiaowen Li and Weihai Li and Yingjun Zhang and Lijie Zhang. DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization. 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’19), 2019

    work page 2019

  8. [8]

    Learn&Fuzz: Machine Learning for Input Fuzzing

    Konstantin Böttinger and Patrice Godefroid and Rishabh Singh. Learn&Fuzz: Machine Learning for Input Fuzzing. arXiv, 2018

    work page 2018

Show all 55 references
  1. [9]

    Lukman and Shan Lu and Haryadi S

    Tanakorn Leesatapornwongsa and Jeffrey F. Lukman and Shan Lu and Haryadi S. Gunawi. TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems. 51st ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16), 2016

    2016

  2. [10]

    Goldberg

    David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. , 1989

    1989

  3. [11]

    Sipper, Moshe and Green, Brian and Ronen, Yakir and Gat, Tomer and Hoffman, Shaked and Zohar, Noam. EC-KitY. SoftwareX, 2023

    2023

  4. [12]

    Giorgos Karafotias and Mark Hoogendoorn and A. E. Eiben. Parameter Control in Evolutionary Algorithms: Trends and Challenges. IEEE Transactions on Evolutionary Computation, 2015

    2015

  5. [13]

    Generalized Coverage Criteria for Combinatorial Sequence Testing

    Elyasaf, Achiya and Farchi, Eitan and Margalit, Oded and Weiss, Gera and Weiss, Yeshayahu. Generalized Coverage Criteria for Combinatorial Sequence Testing. IEEE Transactions on Software Engineering, 2023

    2023

  6. [14]

    Wasserstein and Nicole A

    Ronald L. Wasserstein and Nicole A. Lazar. The ASA’s Statement on p-Values: Context, Process, and Purpose. The American Statistician, 2016

    2016

  7. [15]

    Zhang and Mark Harman and Yudong Han and Yun Ma and Yihong Dong and Ge Li and Gang Huang

    Kaibo Liu and Zhenpeng Chen and Yiyang Liu and Jie M. Zhang and Mark Harman and Yudong Han and Yun Ma and Yihong Dong and Ge Li and Gang Huang. LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs. arXiv preprint arXiv:2404.10304, 2024

    2024 arXiv

  8. [16]

    Ouédraogo and Laura Plein and Kader Kaboré and Andrew Habib and Jacques Klein and David Lo and Tegawendé F

    Wendkûuni C. Ouédraogo and Laura Plein and Kader Kaboré and Andrew Habib and Jacques Klein and David Lo and Tegawendé F. Bissyandé. Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports. Empirical Software Engineering, 2025

    2025

  9. [17]

    Should we really use post-hoc tests based on mean-ranks?

    Alessio Benavoli and Giorgio Corani and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks?. CoRR, 2015

    2015

  10. [18]

    A Family of Abstract Interpretations for Static Analysis of Concurrent Higher-Order Programs

    Matthew Might and David Van Horn. A Family of Abstract Interpretations for Static Analysis of Concurrent Higher-Order Programs. Static Analysis (SAS 2011), 2011

    2011

  11. [19]

    OpenMP Aware MHP Analysis for Improved Static Data -Race Detection

    Utpal Bora and Shraiysh Vaishay and Saurabh Joshi and Ramakrishna Upadrasta. OpenMP Aware MHP Analysis for Improved Static Data -Race Detection. 7th IEEE/ACM Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC ’21), 2021

    2021

  12. [20]

    Matsakis and Felix S

    Nicholas D. Matsakis and Felix S. Klock II. The Rust Language. Ada Letters, 2014

    2014

  13. [21]

    DeepRace: Finding Data Race Bugs via Deep Learning

    Ali Tehrani and Mohammed Khaleel and Reza Akbari and Ali Jannesari. DeepRace: Finding Data Race Bugs via Deep Learning. arXiv preprint arXiv:1907.07110, 2019

    1907 arXiv

  14. [22]

    Hongxu Chen and Shengjian Guo and Yinxing Xue and Yulei Sui and Cen Zhang and Yuekang Li and Haijun Wang and Yang Liu. MUZZ. 29th USENIX Security Symposium (USENIX Security ’20), 2020

    2020

  15. [23]

    Jake Roemer and Kaan Genç and Michael D. Bond. SmartTrack: Efficient Predictive Race Detection. 41st ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’20), 2020

    2020

  16. [24]

    Engineering Record And Replay For Deployability

    Robert O’Callahan and Chris Jones and Nathan Froyd and Kyle Huey and Albert Noll and Nimrod Partush. Engineering Record And Replay For Deployability. 2017 USENIX Annual Technical Conference (USENIX ATC ’17), 2017

    2017

  17. [25]

    Holzmann

    Gerard J. Holzmann. The Model Checker SPIN. IEEE Transactions on Software Engineering, 1997

    1997

  18. [26]

    Clarke and Armin Biere and Richard Raimi and Yunshan Zhu

    Edmund M. Clarke and Armin Biere and Richard Raimi and Yunshan Zhu. Bounded Model Checking Using Satisfiability Solving. Formal Methods in System Design, 2001

    2001

  19. [27]

    Namjoshi and Richard J

    Kedar S. Namjoshi and Richard J. Trefler. Parameterized Compositional Model Checking. Tools and Algo- rithms for the Construction and Analysis of Systems (TACAS 2016), 2016. 34 of 35

    2016

  20. [28]

    Smolka and Radu Grosu

    Axel Legay and Anna Lukina and Louis-Marie Traonouez and Junxing Yang and Scott A. Smolka and Radu Grosu. Statistical Model Checking. Computing and Software Science, 2019

    2019

  21. [29]

    KRACE: Data Race Fuzzing for Kernel File Systems

    Meng Xu and Sanidhya Kashyap and Hanqing Zhao and Taesoo Kim. KRACE: Data Race Fuzzing for Kernel File Systems. 2020 IEEE Symposium on Security and Privacy (SP), 2020

    2020

  22. [30]

    Learning from Mistakes: A Comprehensive Study on Real World Concurrency Bug Characteristics

    Shan Lu and Soyeon Park and Eunsoo Seo and Yuanyuan Zhou. Learning from Mistakes: A Comprehensive Study on Real World Concurrency Bug Characteristics. 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’08), 2008

    2008

  23. [31]

    Engler and Joseph C

    Madanlal Musuvathi and Shaz Qadeer and Thomas Ball and Gerard Basler and Dirk R. Engler and Joseph C. Foster and Amit K. Ghosh. Finding and Reproducing Heisenbugs in Concurrent Programs. 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2008

    2008

  24. [32]

    RACEBENCH: A Benchmark Suite for Data Race Detection Tools

    Yongjun Tian and Yongfei Yu and Peng Wang and Ruihong Zhou and Hui Jin and Tao Xie. RACEBENCH: A Benchmark Suite for Data Race Detection Tools. 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11), 2011

    2011

  25. [33]

    ConSeq: Detecting Concurrency Bugs Through Sequential Errors

    Wei Zhang and Chen Yao and Shan Lu and Jeff Huang and Tian Tan and Xu Liu. ConSeq: Detecting Concurrency Bugs Through Sequential Errors. 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’11), 2011

    2011

  26. [34]

    JaConTeBe: A Benchmark Suite of Real-World Java Concurrency Bugs

    Ziyi Lin and Darko Marinov and Hao Zhong and Yuting Chen and Jianjun Zhao. JaConTeBe: A Benchmark Suite of Real-World Java Concurrency Bugs. 30th IEEE/ACM International Conference on Automated Software Engineering (ASE ’15), 2015

    2015

  27. [35]

    Defects4J

    René. Defects4J. 2014 International Symposium on Software Testing and Analysis (ISSTA ’14), 2014

    2014

  28. [36]

    BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies

    Fernanda Madeiral and Simon Urli and Marcelo de Almeida Maia and Martin Monperrus. BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies. 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’19), 2019

    2019

  29. [37]

    How Often Do Single -Statement Bugs Occur?: The ManySStuBs4J Dataset

    Rafael-Michael Karampatsis and Charles Sutton. How Often Do Single -Statement Bugs Occur?: The ManySStuBs4J Dataset. 17th International Conference on Mining Software Repositories (MSR ’20), 2020

    2020

  30. [38]

    Understanding Real-World Concurrency Bugs in Go

    Tengfei Tu and Xiaoyu Liu and Linhai Song and Yiying Zhang. Understanding Real-World Concurrency Bugs in Go. 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19), 2019

    2019

  31. [39]

    GoBench: A Benchmark Suite of Real-World Go Concurrency Bugs

    Ting Yuan and Guangwei Li and Jie Lu and Chen Liu and Lian Li and Jingling Xue. GoBench: A Benchmark Suite of Real-World Go Concurrency Bugs. 18th Annual IEEE/ACM International Symposium on Code Generation and Optimization (CGO ’21), 2021

    2021

  32. [40]

    ConFuzzius: A Data Dependency-Aware Hybrid Fuzzer for Smart Contracts

    Christof Ferreira Torres and Antonio Ken Iannillo and Arthur Gervais and Radu State. ConFuzzius: A Data Dependency-Aware Hybrid Fuzzer for Smart Contracts. 2021 IEEE European Symposium on Security and Privacy (EuroS&P ’21), 2021

    2021

  33. [41]

    DART: Directed Automated Random Testing

    Godefroid, Patrice and Klarlund, Nils and Sen, Koushik. DART: Directed Automated Random Testing. 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2005

    2005

  34. [42]

    and Grumberg, Orna and Jha, Somesh and Lu, Yuan and Veith, Helmut

    Clarke, Edmund M. and Grumberg, Orna and Jha, Somesh and Lu, Yuan and Veith, Helmut. Counterexample- Guided Abstraction Refinement. 12th International Conference on Computer Aided Verification (CAV), 2000

    2000

  35. [43]

    Bianchi and Mauro Pezzè

    Francesco A. Bianchi and Mauro Pezzè. A Search-Based Approach to Reproduce Crashes in Concurrent Programs. 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), 2017

    2017

  36. [44]

    Artificial Intelligence Applied to Software Testing: A Tertiary Study

    Amalfitano, Domenico and Faralli, Stefano and Hauck, Jean Carlo Rossa and Matalonga, Santiago and Distante, Damiano. Artificial Intelligence Applied to Software Testing: A Tertiary Study. ACM Computing Surveys, 2023

    2023

  37. [45]

    Probabilistic Concurrency Testing for Weak Memory Programs

    Lee, Sanghoon and Zhang, Hui and Viswanathan, Mahesh. Probabilistic Concurrency Testing for Weak Memory Programs. 28th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), 2023

    2023

  38. [46]

    Effective Concurrency Testing for Go via Directional Primitive Scheduling

    Chen, Yuzhe and Liu, Shuhan and Gan, Qiao. Effective Concurrency Testing for Go via Directional Primitive Scheduling. 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023

    2023

  39. [47]

    Fray: An Efficient General-Purpose Concurrency Testing Platform for JVM

    Kumar, Ravi and Lee, Jungho and Padhye, Rohan. Fray: An Efficient General-Purpose Concurrency Testing Platform for JVM. arXiv, 2025

    2025

  40. [48]

    Concurrency Testing in the Linux Kernel via eBPF

    Xu, Jinyang and Wolff, David and Han, Xinyu and Li, Jie and Roychoudhury, Abhik. Concurrency Testing in the Linux Kernel via eBPF. arXiv, 2025

    2025

  41. [49]

    CARDSHARK: Understanding and Stabilizing Linux Kernel Concurrency Bugs Against the Odds

    Han, Tianshuo and Gong, Xiangyu and Liu, Jie. CARDSHARK: Understanding and Stabilizing Linux Kernel Concurrency Bugs Against the Odds. 33rd USENIX Security Symposium (USENIX Security 24), 2024. 35 of 35

    2024

  42. [50]

    and Rowe, Anthony

    Ramesh, Arjun and Huang, Tianshu and Riar, Jaspreet and Titzer, Ben L. and Rowe, Anthony. Unveiling Heisenbugs with Diversified Execution. ACM on Programming Languages, 2025

    2025

  43. [51]

    On the Effect of Instrumentation on Test Flakiness

    Rasheed, Shawn and Dietrich, Jens and Tahir, Amjed. On the Effect of Instrumentation on Test Flakiness. 2023 IEEE/ACM International Conference on Automation of Software Test (AST), 2023

    2023

  44. [52]

    Nekara: A Generalized Concurrency Testing Library

    Shashank, Sai Shashidhar and Sachdeva, Jatin and Mukherjee, Suvam and Deligiannis, Pantazis. Nekara: A Generalized Concurrency Testing Library. 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021

    2021

  45. [53]

    Fast simulation of rare events in queueing and reliability models

    Heidelberger, Philip. Fast simulation of rare events in queueing and reliability models. ACM Trans. Model. Comput. Simul., 1995

    1995

  46. [54]

    Younes and Reid G

    Håkan L.S. Younes and Reid G. Simmons. Statistical probabilistic model checking with a focus on time- bounded properties. Information and Computation, 2006

    2006

  47. [55]

    Selectively Uniform Concur- rency Testing

    Zhao, Huan and Wolff, Dylan and Mathur, Umang and Roychoudhury, Abhik. Selectively Uniform Concur- rency Testing. Proceedings of the ACM on Programming Languages (ASPLOS), 2025. Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are so...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.