On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories

Ali Tizghadam; Francis Bordeleau; Henri A\"idasso

REVIEW 4 major objections 4 minor 29 references

On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Flaky job failures fall into 46 identifiable categories, with 14 priority targets for automated diagnosis and repair.

desk verdict Useful first taxonomy of flaky job failure causes, but the priority list is built on non-random missing logs and an optimistic tool evaluation. read the letter →

arxiv 2501.04976 v1 pith:BZTCTATB submitted 2025-01-09 cs.SE

Henri Aïdasso , Francis Bordeleau , Ali Tizghadam This is my paper · ORCID

classification cs.SE

keywords flakyjobsjobfailuresfailurecategoriesRFManalysisprioritizationcontinuousintegrationdeploymentmachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flaky job failures—non-deterministic failures that pass or fail on rerun without code changes—waste time and machine resources in continuous-deployment pipelines. The paper analyzes 4,511 labeled flaky job failures from the CI/CD system of a large telecommunications company and claims these failures fall into 46 identifiable categories, with the five most frequent covering more than 40% of all cases. It then ranks the categories by recency, frequency, and estimated monetary cost, and claims that 14 of them are priority targets for automated diagnosis and repair, led by misconfigured environment variables and job execution timeouts. The point of the ranking is to focus future tooling and repair efforts on the failures that are most recent, most recurrent, and most expensive, rather than trying to handle every possible cause.

What carries the argument

The machinery has three parts. First, a semi-automated labeling tool built from 51 regex search rules assigns each failure log to one of 46 category labels, with categories consolidated by engineers and validated on held-out samples. Second, an RFM model computes, for each category, recency (average days since the last three occurrences), frequency (total occurrence count), and monetary value (estimated machine run cost plus engineer diagnosis delay cost derived from a cost model). Third, these three measures are normalized into RFM scores and clustered with K-means, after five outlier categories are removed with an isolation-forest detector, producing eight clusters that are then interpreted as priority levels from high priority to irrelevant.

What would settle it

Re-run the labeling and RFM ranking after restoring the missing logs from the platform-migration period, or apply the same pipeline to a second organization's CI logs with complete rerun histories; if misconfigured environment variables and job execution timeouts no longer lead the recency-frequency-cost ranking, the priority list depends on the missing data or on this company's specific environment rather than on the categories themselves.

Watch

Extended reading notes

Core claim

The paper's central claim is that flaky job failures are not an unmanageable mix of one-off causes: they can be labeled into a stable taxonomy of 46 categories, and the categories can be prioritized for automated diagnosis and repair using recency-frequency-monetary (RFM) analysis. The most frequent category, misconfigured environment variables, accounts for about 15% of the labeled failures; the five most frequent categories together account for more than 40%. The five most costly categories overlap only partially with the five most frequent ones, showing that frequency alone is not a sufficient guide. When recency, frequency, and cost are considered together, 14 categories emerge as priorities, with misconfigured environment variables and job execution timeouts at the top. The paper also claims that the RFM-based prioritization approach itself is transferable to other contexts where failure categories must be ranked.

Load-bearing premise

The load-bearing premise is that a job failure counts as flaky only when re-running the same commit changes the outcome, so failures that are never rerun or whose reruns never alternate success and failure are invisible to the dataset and to every category and priority derived from it.

Editorial extensions

If this is right

Automated diagnosis and repair work should be aimed first at the 14 priority categories, with misconfigured environment variables and job execution timeouts as the top two targets.
Recency, frequency, and monetary cost should be treated as independent signals: the paper shows that the most frequent categories are not the most costly, so prioritization needs all three measures.
Fourteen additional categories, including idle and emerging issues, should be monitored because they are either recently active but cheap or historically costly and possibly resurgent.
The remaining categories, about 39% of the taxonomy, can be deprioritized as relic, negligible, or irrelevant, freeing effort for the priority set.
A regex-based labeling tool built from manually labeled samples labels 85.83% of the original 5,256 flaky failures automatically, which makes the taxonomy usable beyond the initial study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One consequence the authors leave implicit is that the same RFM pipeline could be applied to another organization's CI logs; if the leading categories differ sharply, the priority list is likely specific to this company's toolchain rather than a general property of CI flakiness.
The top category also exposes a blind spot the paper notes but does not pursue: environment-variable changes are not version-controlled, so standard 'no code changes' flakiness definitions can miss an entire cause class; versioning or auditing CI environment variables would be a concrete next repair strategy.
Because roughly 30% of flaky failures had missing logs around a major CI-platform migration, the recency and frequency estimates may under-represent categories active during that window; recovering those logs could move some categories in or out of the priority 14.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is the first empirical taxonomy of flaky job failure causes (not just flaky tests) built from a large industrial dataset, and the RFM prioritization is a simple but sensible way to turn that taxonomy into an action list. I think the core contribution—46 categories with frequencies and project spread—is real and will be useful to CI researchers and practitioners. The paper is worth serious referee time.

Where it gets soft: the missing-log problem is not a side worry, it's load-bearing. 33% of flaky failures have no logs, and the blank window is exactly the period around the 2022 GitLab migration. RQ3's claims that docker daemon connection failure has vanished and api gateway deployment error only recently appeared are indistinguishable from 'logs for those failures were lost.' R, F, and M are all computed from the labeled 4,511 failures, so Table VI inherits that non-random missingness. The paper acknowledges the limitation in Threats to Validity but offers no sensitivity analysis. That needs to be addressed before the priority list can be taken as stable.

Second soft spot: the labeling tool's reported recall is not an independent estimate. Sample 2 was used to refine the tool and then used to evaluate it; the 91.34% is therefore optimistic. A truly held-out sample is needed. Also, the numbers in Section V appear scrambled: it says precision ~82% on an unseen sample, but the reported precision on Sample 2 is 99.31% and the recall is 81.56%. Those should be reconciled.

Third, the cost model is coarse—lead time as a proxy for diagnosis time, flat salary rate, no sensitivity analysis—but this is less damaging because the prioritization is mostly driven by frequency and recency, and the authors say cost is for comparative purposes. Still, a simple one-way sensitivity on S and M would help.

The rerun heuristic the reviewer worried about is actually fine: defining flaky as outcome changes on same-commit reruns is standard and acknowledged. I don't see a circularity problem in the taxonomy itself.

Bottom line: the taxonomy and the general approach are solid and new. The current version overstates confidence in the specific priority rankings because of missing data and the evaluation design. This deserves peer review, but it needs revision: held-out evaluation, sensitivity analysis for the missing window, and cleaned-up accuracy numbers.

I'd bring it to reading group and would cite it for the taxonomy. Recommendation: send to review, with a request for major revision.

Referee Report

4 major / 4 minor

Summary. The paper presents an empirical study of 4,511 flaky job failures from 80 TELUS projects over a 6.5-year period. The authors develop a semi-automated regex-based labeling tool, classify failures into 46 categories, estimate each category's recency, frequency, and monetary cost, and then apply k-means clustering on RFM scores to identify 14 priority categories for automated diagnosis and repair. The paper's contributions are a taxonomy of flaky job failure categories and an RFM-based prioritization framework that the authors propose as transferable to other contexts.

Significance. If the taxonomy and priority list are reliable, the study fills a real gap between flaky test classification and flaky job detection by providing a diagnosis-oriented taxonomy and a concrete prioritization method. The scale of the industrial dataset, the involvement of TELUS engineers in validating labels, and the promise of a replication package are notable strengths. However, the validity of the central claims depends on two currently weak points: the non-random exclusion of missing logs, which overlaps with the temporal window used for recency claims, and the non-independent evaluation of the labeling tool. The RFM prioritization also relies on several unvalidated parameter choices. These issues do not invalidate the taxonomy as a qualitative contribution but they do undermine the specific priority ranking unless addressed with sensitivity analyses.

major comments (4)

[Section III-B, RQ3/RQ4 (Fig. 4, Tables IV and VI)] The exclusion of 2,507 flaky failures with missing logs (33.29%) is not random: the missing period coincides with the 'big blank period' between early 2022 and Q1 2023, as shown in Figure 4. Because Recency and Frequency in RQ3 and RQ4 are computed from the observed labels only, the statement that 'docker daemon connection failure ... have not occurred for over a year' and the claim that 'api gateway deployment error' appeared only recently are indistinguishable from log loss during this period. The paper acknowledges this limitation in Section V but provides no sensitivity analysis. I request a robustness analysis that shows whether the 14 priority categories in Table VI survive alternative treatments of the missing window, for example by restricting the analysis to the post-migration period, by imputing missing occurrences, or by recomputing RFM scores after removing the entire blank period.
[Section III-C, Table II] The reported recall of 91.34% on Sample 2 is not an independent estimate of labeling quality because the tool was iteratively refined using the unlabeled portion of Sample 2 until recall exceeded 90%; the final value is an in-sample measure. In addition, the text states 'Our labeling tool achieves a precision score of 91% on an unseen representative sample,' but Table II lists precision 99.31% and recall 91.34% at the final iteration, and Section V refers to '≈82% precision' where Table II shows 81.56% recall. Since the correctness of the taxonomy depends on label quality, the authors should either evaluate on a fresh held-out sample that is entirely uninvolved in tool development, or report the iteration-0 recall as the honest out-of-sample estimate and clearly distinguish it from the final recall.
[Section IV-D, Tables V and VI] The prioritization depends on several user-chosen parameters whose influence is not examined: the number of clusters k=8, the IsolationForest contamination ratio of 10%, the choice to average recency over the last three occurrences, and the quintile-based RFM scoring. Because the paper's central contribution is the priority list, I ask for a stability analysis over a plausible grid of these parameters, or a bootstrap resampling across categories, to show that the set of 14 priority categories is not an artifact of one particular parameterization.
[Section IV-B, Eq. (2) and Fig. 3] The diagnosis cost is estimated as the time from the initial failure to the last rerun in the same rerun sequence, which may include waiting periods, weekends, and unrelated delays; the paper acknowledges this in Section V. However, Figure 3 shows that diagnosis cost constitutes roughly 99% or more of every category's total cost, so the monetary (M) component in the RFM model is effectively determined by this proxy. Any systematic bias in the proxy, such as categories whose rerun sequences are naturally longer (e.g., job execution timeout), will propagate directly into the priority rankings. I recommend a sensitivity analysis that caps the delay, excludes non-working hours, or uses an alternative cost measure, to confirm that the top priority categories remain unchanged.

minor comments (4)

[Section V, Internal Validity] The sentence 'we developed a regex-based labeling tool that achieved ≈82% precision on an unseen sample' should refer to recall, not precision; Table II reports 99.31% precision and 81.56% recall at iteration 0.
[Table V] The meaning of the 'Std' column is unclear; the header appears to combine the standard deviation of RFM scores with the average values in an ambiguous way, and the text should explain what this number represents.
[Section III-B] The flakiness heuristic requires a rerun on the same commit with an outcome change; this systematically excludes flaky failures that are never rerun or whose reruns occur on different commits. The paper notes this aligns with TELUS practices, but the possible impact on the category distribution should be explicitly acknowledged in the limitations.
[Section III-C] The phrase 'a precision of 100% and a recall of 99.44%' for Sample 1 is clear, but the later sentence 'Our labeling tool achieves a precision score of 91% on an unseen representative sample' is a misstatement of the reported metrics and should be corrected.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor non-independence in labeling-tool recall estimate; central taxonomy and RFM prioritization are otherwise self-contained empirical analyses.

fitted input called prediction [Section III-C, 'Tool Evaluation and Improvement' (Table II)]
"Although we obtained a satisfying recall, we sought to improve it using the remaining unlabeled data from Sample 2. So, we followed the manual labeling process again, updating the regex patterns and search rules after each iteration. In the end, no new categories were discovered. We stopped after achieving precision and recall scores of over 90%. As a result, we increased the tool’s recall to 91.34% on Sample 2."

The reported recall is computed on Sample 2, but Sample 2 was used during the refinement loop to update regex patterns and search rules. Thus the 91.34% figure is a fit-to-evaluation-sample statistic, not an independent accuracy estimate. The paper later presents this as validation ('Our labeling tool achieves a precision score of 91% on an unseen representative sample'), which overstates independence. This does not make the taxonomy itself circular, but the performance claim is not an external check of the labeling tool.

full rationale

The paper's main outputs are an empirically derived taxonomy of 46 flaky-failure categories and an RFM-based prioritization. No step derives a result from an input that already contains it: categories are induced from logs via open coding and regexes; frequencies, costs, recency, and clusters are all computed from the labeled dataset rather than predicted from external assumptions. The only circularity-adjacent step is the labeling-tool evaluation: the tool's regexes were refined on Sample 2 and then recall was reported on Sample 2 as if it were an unseen test set. This inflates the reported accuracy but is not load-bearing for RQ1-RQ4 conclusions, which are descriptive statistics over the full labeled set. The missing-log window ('big blank period' in Fig. 4) is a serious validity threat to recency claims, but it is a data-availability bias, not a circular derivation. No self-citation chain or uniqueness theorem carries the argument.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. The main assumptions are the flaky-job identification heuristic and the log-based labeling process. The free parameters are the cost constants and clustering hyperparameters, all chosen by hand rather than fitted. The RFM analysis itself is a standard method, not a new entity.

free parameters (5)

Machine cost rate M = $0.14/min
Chosen as average rate for a high-memory VM (n2-highmem-128) on Google Cloud; used in Eq 1 to compute all machine costs.
Diagnosis cost rate S = $0.6/min
Derived from average engineer salary of $36/hour; used in Eq 2. Salary is an assumption and affects all monetary values, dominating total costs.
K-means cluster count k = 8
Set to 8 'in line with previous work' [15], not selected by data; cluster partition determines priority groups.
IsolationForest contamination = 10%
Used to identify outliers in the RFM dataset; affects which categories are treated as outliers and excluded from clustering.
Recency window = last 3 occurrences
Recency is computed as average days since the last three occurrences, rather than the last occurrence, to reduce the impact of sporadic categories.

assumptions (5)

domain assumption A job is flaky if, on the same commit, it has both successful and failed reruns
Used in Section III-B to define the dataset of 7,763 flaky failures; excludes flaky failures that were never rerun or whose reruns all failed.
domain assumption Root cause can be inferred from job logs alone
The labeling process (Section III-C) assigns categories based on log messages and regex patterns; failures where logs are missing or unclear are excluded.
domain assumption Time delay from initial failure to last rerun approximates diagnosis time
Used in Eq 2 for diagnosis cost; the paper acknowledges this may include idle waiting and weekends.
domain assumption The 80 selected projects and 14 GitLab groups represent TELUS CI/CD
Data collection in Section III-B relies on this representativeness to generalize findings.
domain assumption Regex rules and manual open coding produce valid category labels
The validity of the taxonomy depends on the correctness of the labeling tool; precision is estimated at 99% but based on a partially non-independent sample.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories." pith.science (2026). https://pith.science/paper/BZTCTATB

@misc{pith2026250104976,
  author       = {Pith},
  title        = {Pith review of: On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZTCTATB}},
  note         = {Machine review of arXiv:2501.04976}
}

read the original abstract

The continuous delivery of modern software requires the execution of many automated pipeline jobs. These jobs ensure the frequent release of new software versions while detecting code problems at an early stage. For TELUS, our industrial partner in the telecommunications field, reliable job execution is crucial to minimize wasted time and streamline Continuous Deployment (CD). In this context, flaky job failures are one of the main issues hindering CD. Prior studies proposed techniques based on machine learning to automate the detection of flaky jobs. While valuable, these solutions are insufficient to address the waste associated with the diagnosis of flaky failures, which remain largely unexplored due to the wide range of underlying causes. This study examines 4,511 flaky job failures at TELUS to identify the different categories of flaky failures that we prioritize based on Recency, Frequency, and Monetary (RFM) measures. We identified 46 flaky failure categories that we analyzed using clustering and RFM measures to determine 14 priority categories for future automated diagnosis and repair research. Our findings also provide valuable insights into the evolution and impact of these categories. The identification and prioritization of flaky failure categories using RFM analysis introduce a novel approach that can be used in other contexts.

Figures

Figures reproduced from arXiv: 2501.04976 by the authors.

**Figure 2.** Overview of the approach for building an automated flaky failure labeling tool. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Ranking of the 20 most costly flaky failures categories. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Evolution over time of the 20 most frequent flaky failure categories, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 18 canonical work pages

[1]

Humble and D

J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation , 1st ed. Addison-Wesley Professional, Jul. 2010

work page 2010
[2]

Usage, costs, and benefits of continuous integration in open-source projects,

M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’16. New York, NY , USA: Association for Computing Machinery, Aug. 2016, pp. 426–437. [Online]. Available: https:/...

arXiv 2016
[3]

DeFlaker: automatically detecting flaky tests,

J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “DeFlaker: automatically detecting flaky tests,” in Proceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, May 2018, pp. 433–444. [Online]. Available: https://dl.acm.org/doi/10.1145/3180155.3180164

arXiv 2018
[4]

An empirical analysis of flaky tests,

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, Nov. 2014, pp. 643–653. [Online]. Available: https: //doi.org/10.1145/2635868.2635920

arXiv 2014
[5]

Do Not Trust Build Results at Face Value - An Empirical Study of 30 Million CPAN Builds,

M. Zolfagharinia, B. Adams, and Y .-G. Gu ´eh´enuc, “Do Not Trust Build Results at Face Value - An Empirical Study of 30 Million CPAN Builds,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) , May 2017, pp. 312–322. [Online]. Available: https://ieeexplore.ieee.org/document/7962381

work page arXiv 2017
[6]

When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla,

J. Lampel, S. Just, S. Apel, and A. Zeller, “When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Athens Greece: ACM, Aug. 2021, pp. 1381–1392. [Online]. Available: https...

arXiv 2021
[7]

Towards language- independent brown build detection,

D. Olewicki, M. Nayrolles, and B. Adams, “Towards language- independent brown build detection,” in Proceedings of the 44th International Conference on Software Engineering . Pittsburgh Pennsylvania: ACM, May 2022, pp. 2177–2188. [Online]. Available: https://dl.acm.org/doi/10.1145/3510003.3510122

arXiv 2022
[8]

Automated Identification of Flaky Builds using Knowledge Graphs

F. Moriconi, R. Troncy, A. Francillon, and J. Zouaoui, “Automated Identification of Flaky Builds using Knowledge Graphs.”

work page

Show all 29 references

[9]

FlakeSync: Automatically Repairing Async Flaky Tests,

S. Rahman and A. Shi, “FlakeSync: Automatically Repairing Async Flaky Tests,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery, Apr. 2024, pp. 1–12. [Online]. Available...

2024
[10]

FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair,

S. Fatima, H. Hemmati, and L. Briand, “FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair,” Aug. 2024, arXiv:2307.00012 [cs]. [Online]. Available: http://arxiv.org/abs/2307.00012

2024 arXiv
[11]

On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories,

H. A ¨ıdasso, “On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories,” 2025, artwork Size: 7060489 Bytes Pages: 7060489 Bytes. [Online]. Available: https://figshare.com/ articles/online resource/On the Diagnosis of Flaky Job Failures Underst...

2025
[12]

Secured Communication Using Virtual Private Net- work (VPN),

P. J. Ezra, S. Misra, A. Agrawal, J. Oluranti, R. Maskeliunas, and R. Damasevicius, “Secured Communication Using Virtual Private Net- work (VPN),” in Cyber Security and Digital Forensics, K. Khanna, V . V . Estrela, and J. J. P. C. Rodrigues, Eds. Singapore: Springer, 2022, pp...

2022
[13]

Available: https://kubernetes.io/docs/concepts/ workloads/pods/

“Pods.” [Online]. Available: https://kubernetes.io/docs/concepts/ workloads/pods/
[14]

Empirical Study of Restarted and Flaky Builds on Travis CI,

T. Durieux, C. Le Goues, M. Hilton, and R. Abreu, “Empirical Study of Restarted and Flaky Builds on Travis CI,” in Proceedings of the 17th International Conference on Mining Software Repositories . Seoul Republic of Korea: ACM, Jun. 2020, pp. 254–264. [Online]. Available: http...

2020
[15]

Data Mining Using RFM Analysis,

D. Birant, “Data Mining Using RFM Analysis,” in Knowledge- Oriented Applications in Data Mining , K. Funatsu, Ed. InTech, Jan. 2011. [Online]. Available: http://www.intechopen. com/books/knowledge-oriented-applications-in-data-mining/ data-mining-using-rfm-analysis

2011
[16]

Answer to

4ces, “Answer to ”GitLab CI/CD: This job does not have a trace. & No job log”,” Sep. 2021. [Online]. Available: https: //stackoverflow.com/a/69378998/26508024

2021
[17]

Modern code reviews in open-source projects: which problems do they fix?

M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, “Modern code reviews in open-source projects: which problems do they fix?” in Proceedings of the 11th Working Conference on Mining Software Repositories, ser. MSR 2014. New York, NY , USA: Association for Computing Machiner...

2014
[18]

Studying the Impact of Noises in Build Breakage Data,

T. A. Ghaleb, D. Alencar Da Costa, Y . Zou, and A. E. Hassan, “Studying the Impact of Noises in Build Breakage Data,” IEEE Transactions on Software Engineering , pp. 1–1, 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8839858/

2019
[19]

Answer to

A. Marshall, “Answer to ”GitLab pipeline exiting with error code 137 when running Cypress”,” Oct. 2021. [Online]. Available: https://stackoverflow.com/a/69676383/26508024

2021
[20]

Open coding,

S. H. Khandkar, “Open coding,” University of Calgary , vol. 23,
[21]

Spencer, Card Sorting: Designing Usable Categories

D. Spencer, Card Sorting: Designing Usable Categories . Rosenfeld Media, 2009, google-Books-ID: h4D9gqi5tsC

2009
[22]

The Art of Testing Less without Sacrificing Quality,

K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy, “The Art of Testing Less without Sacrificing Quality,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1, May 2015, pp. 483–493, iSSN: 1558-1225. [Online]. Available: https://ieeexplore.ieee...

2015
[23]

Available: https://scikit-learn/stable/modules/ generated/sklearn.cluster.KMeans.html

“KMeans.” [Online]. Available: https://scikit-learn/stable/modules/ generated/sklearn.cluster.KMeans.html
[24]

A review on customer segmentation methods for personalized customer targeting in e-commerce use cases,

M. Alves Gomes and T. Meisen, “A review on customer segmentation methods for personalized customer targeting in e-commerce use cases,” Information Systems and e-Business Management , vol. 21, no. 3, pp. 527–570, Sep. 2023. [Online]. Available: https://doi.org/10.1007/ s10257-0...

2023
[25]

IsolationForest example

“IsolationForest example.” [Online]. Available: https://scikit-learn/ stable/auto examples/ensemble/plot isolation forest.html
[26]

Error Sum of Squares

“Error Sum of Squares.” [Online]. Available: https://hlab.stanford.edu/ brian/error sum of squares.html
[27]

Understanding flaky tests: the developer’s perspective,

M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: the developer’s perspective,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/F...

2019
[28]

Predicting Flaky Tests Categories using Few-Shot Learning,

A. Akli, G. Haben, S. Habchi, M. Papadakis, and Y . L. Traon, “Predicting Flaky Tests Categories using Few-Shot Learning,” Aug. 2022, arXiv:2208.14799 [cs]. [Online]. Available: http://arxiv.org/abs/ 2208.14799

2022 arXiv
[2009]

Available: https://cspages.ucalgary.ca/ ∼saul/wiki/ uploads/CPSC681/open-coding.pdf

[Online]. Available: https://cspages.ucalgary.ca/ ∼saul/wiki/ uploads/CPSC681/open-coding.pdf

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Humble and D

J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation , 1st ed. Addison-Wesley Professional, Jul. 2010

work page 2010

[2] [2]

Usage, costs, and benefits of continuous integration in open-source projects,

M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’16. New York, NY , USA: Association for Computing Machinery, Aug. 2016, pp. 426–437. [Online]. Available: https:/...

arXiv 2016

[3] [3]

DeFlaker: automatically detecting flaky tests,

J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “DeFlaker: automatically detecting flaky tests,” in Proceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, May 2018, pp. 433–444. [Online]. Available: https://dl.acm.org/doi/10.1145/3180155.3180164

arXiv 2018

[4] [4]

An empirical analysis of flaky tests,

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, Nov. 2014, pp. 643–653. [Online]. Available: https: //doi.org/10.1145/2635868.2635920

arXiv 2014

[5] [5]

Do Not Trust Build Results at Face Value - An Empirical Study of 30 Million CPAN Builds,

M. Zolfagharinia, B. Adams, and Y .-G. Gu ´eh´enuc, “Do Not Trust Build Results at Face Value - An Empirical Study of 30 Million CPAN Builds,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) , May 2017, pp. 312–322. [Online]. Available: https://ieeexplore.ieee.org/document/7962381

work page arXiv 2017

[6] [6]

When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla,

J. Lampel, S. Just, S. Apel, and A. Zeller, “When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Athens Greece: ACM, Aug. 2021, pp. 1381–1392. [Online]. Available: https...

arXiv 2021

[7] [7]

Towards language- independent brown build detection,

D. Olewicki, M. Nayrolles, and B. Adams, “Towards language- independent brown build detection,” in Proceedings of the 44th International Conference on Software Engineering . Pittsburgh Pennsylvania: ACM, May 2022, pp. 2177–2188. [Online]. Available: https://dl.acm.org/doi/10.1145/3510003.3510122

arXiv 2022

[8] [8]

Automated Identification of Flaky Builds using Knowledge Graphs

F. Moriconi, R. Troncy, A. Francillon, and J. Zouaoui, “Automated Identification of Flaky Builds using Knowledge Graphs.”

work page