What Makes a Fairness Tool Project Sustainable in Open Source?

Andrew Meenly; Brittany Johnson; Fatemeh Vares; Sadia Afrin Mim

REVIEW 4 major objections 6 minor 89 references

What Makes a Fairness Tool Project Sustainable in Open Source?

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read More than half of open-source fairness tools go inactive within three years, while only a third remain active at the time of the study.

desk verdict Useful new dataset and plausible descriptive findings, but the headline '53% inactive in three years' claim is not backed by the methods as written. read the letter →

arxiv 2505.09802 v1 pith:YPMWKUC4 submitted 2025-05-14 cs.SE cs.HC

Sadia Afrin Mim , Fatemeh Vares , Andrew Meenly , Brittany Johnson This is my paper · ORCID

classification cs.SEcs.HC

keywords fairnesstoolsopensourceGitHubsoftwaresustainabilitymaintenancecommunityengagementmachinelearningbias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that open-source fairness tools, meaning software for detecting and mitigating bias in machine learning, have a sustainability problem. Using a curated sample of 61 GitHub projects, it reports that 53 percent become inactive within three years and that only 32.2 percent are active at the time of study. This matters because fairness is an ongoing requirement: a bias-mitigation tool that stops being maintained can quietly become outdated or unreliable. The paper also finds that industry-backed tools receive the most consistent engagement and maintenance, while academic and individual projects are more fragile, with domain-specific tools as a notable exception. If the finding holds, the fairness tooling landscape is less healthy than the number of available tools suggests.

What carries the argument

The carrying mechanism is a two-part empirical apparatus. The first part is a curated dataset of 61 projects built by extracting fairness-related natural-language and tool-specific keywords from an earlier taxonomy of fairness tools and querying the GitHub API, followed by manual validation of the results. The second is a maintenance classifier, adopted from prior work and trained on over a thousand labeled repositories, that labels each project Active, Inactive, or Archived using features such as forks, issues, pull requests, commits, and contributor counts over a 24-month window. Around this classifier, the paper layers engagement metrics, including stars, watches, forks, and pull requests, and a lifespan measurement that compares the time since the last commit with the project's age.

What would settle it

Run a wider replication that searches GitHub without the top-ten cutoff, includes non-Python fairness tools, and tracks every qualifying repository for three years from creation; if the three-year inactivity rate in that larger sample is substantially below 53 percent, the paper's central claim would be refuted.

Watch

Extended reading notes

Core claim

The paper establishes that sustainability is the exception rather than the rule in open-source fairness tooling. Of the 61 projects analyzed, 53 percent become inactive within three years, and at the time of the study only 32.2 percent are classified as active, with 62.7 percent inactive and 5.1 percent archived. The most stable and consistently engaged repositories are affiliated with industry or non-academic organizations, while academic and individual projects tend to show lower, more sporadic pull-request and commit activity. The paper also observes that well-maintained tools can still lack engagement, so maturity and visibility matter as much as upkeep, and that a small set of domain-specific tools, for areas such as healthcare, language models, energy, and recommender systems, sustain steady if modest engagement. The central conclusion is that the fairness tool ecosystem is fragile, and that survival depends on factors beyond code quality, including organizational backing and niche relevance.

Load-bearing premise

The central percentages rest on the assumption that 61 repositories found by searching the top ten GitHub results for fairness-related keywords, mostly Python tools, represent the whole population of open-source fairness tools.

Editorial extensions

If this is right

Practitioners choosing a fairness tool should treat maintenance status as a first-class criterion, since the majority of projects stop receiving commits within three years.
Maintainers and funders should plan for multi-year stewardship rather than one-time releases, because the data show that only industry-backed projects tend to sustain consistent engagement and maintenance.
Academic and individual fairness-tool projects may need institutional support or community scaffolding, since they show lower and more sporadic pull-request and commit activity.
Domain-specific fairness tools, such as those for healthcare or natural language processing, can sustain engagement even without a large organization behind them, suggesting that niche relevance is a resilience factor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 53 percent inactivity rate generalizes beyond Python and GitHub, it implies that funders and institutions should plan multi-year stewardship for fairness tools rather than one-off releases.
The paper does not distinguish abandoned projects from feature-complete ones; an inactive repository that stopped changing because it works would soften the bleak reading, so a future analysis separating 'done' from 'dead' would sharpen the result.
The late-2023 engagement surge in language-model-related fairness tools suggests that sustainability may track funding and hype cycles; one could test whether tools tied to currently funded subfields survive longer than tools tied to older problem framings.
Higher owner-commit counts in inactive projects hint at bus-factor risk, meaning projects maintained by one person may die when that person steps away; testing contributor concentration as a predictor of inactivity would be a direct follow-up.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on the fairness tools sustainability paper. It's a genuinely useful empirical contribution—the first dataset I know of that tracks maintenance and engagement across a broad set of open source fairness tools, with a public replication artifact. The descriptive patterns (industry-backed projects more active, engagement concentrated in a few large repos) are plausible and consistent with what I'd expect. I'd happily point students to the dataset.

The problem is the headline number. The 53% 'inactive within three years' claim in the abstract is not supported by the methods as written. RQ3's method is just 'timespan of last commit since initial commit'—that's not a survival analysis, no three-year window defined, no handling of right-censoring for young repos. The number never appears in the findings section. The stress-test note is right: the statistic is not derivable from the paper. That's a load-bearing flaw because it's the abstract's marquee finding.

Also, the maintenance classifier features (total issues, commits, etc.) are the same ones compared in Table II, so the significant differences are partly circular. The feature overlap doesn't invalidate the descriptive comparisons, but it does mean those p-values aren't independent evidence of a real distinction. The cohort sizes also wobble (50, 61, 62), which is sloppy. The search methodology (top-10 per keyword, Python-only imports, seeded from the authors' own taxonomy) is pragmatically reasonable but makes population-level percentages shaky.

That said, the paper is honest about some limitations in Section VI, and the core observation—that many fairness tools go quiet—is almost certainly correct in direction. What's missing is the analytical scaffolding to put a number on it.

For peer review: yes, this deserves serious refereeing. The dataset and descriptive analysis are worth publishing, but the paper needs major revision: either add a proper survival analysis or drop the three-year claim, address the circularity, and clean up the inconsistencies. If the authors do that, it's a solid contribution. I'd bring it to a reading group as a case study in why descriptive repo-mining papers need to match their claims to their methods.

Referee Report

4 major / 6 minor

Summary. The paper presents a GitHub mining study of open-source machine learning fairness tools. The authors extend a previously curated dataset with GitHub keyword search, classify repositories into Active, Inactive, and Archived states using a Random Forest model adopted from prior work, and compare community engagement and maintenance metrics between these groups. The central claimed result is that 53 percent of fairness projects become inactive within their first three years, with only 32.2 percent of the studied projects active at the time of data collection. The paper also analyzes stars, forks, pull requests, issue resolution times, and maintenance-related keywords, and discusses implications for sustainability of fairness tooling.

Significance. If the headline statistic were supported, the paper would supply an important empirical baseline for the responsible-AI and software-engineering communities: it would quantify the fragility of the open-source fairness tool ecosystem and would motivate targeted sustainability efforts. The paper has genuine strengths: it ships a public replication artifact, it makes a systematic attempt to extend an existing dataset, and its descriptive analyses of star, fork, and pull-request activity provide useful qualitative insight into engagement patterns. However, the current manuscript does not support the headline three-year inactivity rate, and the maintenance-classification analysis contains a circularity that undermines the statistical comparisons in Table II. The significance of the contribution therefore depends on substantial reanalysis and clarification.

major comments (4)

[Abstract; Section III-G; Section IV-D] The claim that "53 percent of fairness projects become inactive within the first three years" is not derivable from the methods described. Section III-G states that the authors measured "the timespan of the last commit since their initial commit," which measures recency of the last activity relative to the data-collection date, not the elapsed time from a project's first commit to its becoming inactive. The manuscript provides no definition of a three-year window, no survival analysis, no treatment of right-censoring for projects younger than three years, and no threshold for "inactive" tied to the claim. The 53 percent figure does not appear in Section IV-D, Figure 7, or any table, so it cannot be traced to a reported calculation. The authors must either perform a proper time-to-event analysis and report the exact computation, or remove the claim from the abstract and framing.
[Section III-F; Table II] The Mann-Whitney U tests in Table II are partly circular. The Active/Inactive labels used as the grouping variable come from a Random Forest classifier trained on features that include total issues, closed issues, open pull requests, closed pull requests, merged pull requests, total commits, most active developer commits, and contributors. Table II then tests for differences between the Active and Inactive groups on those same features. Significant differences on classifier training features are expected by construction and do not provide independent evidence about what distinguishes maintained from unmaintained fairness tools. This affects the interpretation of the starred rows in Table II and the related discussion in Section IV-C. Please re-run the analysis using labels obtained independently of the tested features, or restrict the significance tests to features that were not used in classification.
[Section III-A to III-D; Section IV-A; Abstract; Conclusion] The reported cohort size is inconsistent across the manuscript: the Abstract says 50 projects, the Introduction and Conclusion say 61, Section IV-A says 62, and Section III-F states that the test set comprised 61 fairness-related repositories. Because the paper's percentages (53 percent, 32.2 percent, 62.7 percent, 5.1 percent) all depend on the denominator, this inconsistency must be resolved. In addition, the sampling procedure uses a top-10 cutoff per keyword and only Python import statements, so the resulting set is likely a convenience subset rather than a complete census of fairness tools. The paper acknowledges this threat in Section VI-A, but still presents population-level rates; please report exact denominators and discuss how selection bias could change the headline percentages.
[Section III-F] The transfer of the maintenance classifier from the 1,003 general repositories used in Coelho et al. to fairness-tool repositories is not validated. The manuscript reports no accuracy, precision, recall, or cross-validation results for the Random Forest, and it provides no manual validation of the predicted labels on fairness repositories. The 32.2 percent active estimate therefore has unknown error, and this error propagates to every comparison that uses the Active/Inactive grouping. Please report the classifier's performance on its original dataset and, ideally, validate a sample of fairness-repository labels manually or through a held-out fairness-specific test set.

minor comments (6)

[Section I] There is a missing citation placeholder in the first paragraph of the Introduction: the text contains "[ ? ]".
[Section IV-A] The text says "a total of 62 open source fairness tools in Figure ??", but no figure number is provided; the reference should be corrected.
[Section IV; Section IV-C] The repository name "pymetrics/audit-ai" is sometimes written as "pymetrics/auditai" and "pymetrics/auditai"; please standardize the spelling.
[Throughout] There are several typographical errors, including "predicticing" (Section II-A), "T echnology" (the author affiliation line), "langauge" (Section I), "V ares" (author list), and "Maintability" (the subsection heading in Section IV-C).
[Section III-F] The description of the Random Forest classifier omits the model hyperparameters, the exact feature-selection outcome after the correlation-based clustering, and the version of the classifier implementation; adding these details would improve replicability.
[Section IV] The anonymous artifact link should be replaced with a permanent public link in the camera-ready version, since the current link may not be accessible after the review process.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: maintenance-status labels are produced by a classifier trained on the same repository features that Table II later tests, and the RQ3 lifespan finding restates the Section III-F label definitions; the 53% three-year claim is separately unsupported but not circular.

fitted input called prediction [Section III-F (Evaluating Tool Maintenance); Table II in Section IV-C]
"The features we included for training are forks, total issues, closed issues, open pull requests, closed pull requests, merged pull requests, total commits, max days without a commit, most active developer commits, contributors, owner projects, and owner commits."

The Active/Inactive labels for the 61 fairness repositories are outputs of a Random Forest classifier trained on exactly these repository metrics. Section IV-C then runs Mann-Whitney U tests between the Active and Inactive groups on total issues, open PRs, closed PRs, merged PRs, total commits, most active dev commits, and contributors (Table II), i.e., the same variables that entered the classifier. Because the group labels are a function of those inputs, the reported significant differences are at least partly an artifact of the classifier's decision rule rather than an independent empirical discovery about fairness-tool maintenance. The test thus recapitulates the classifier's inputs instead of validating an external hypothesis.
self definitional [Section IV-D (Project Lifespan, RQ3); compare Section III-F Active/Inactive definitions]
"Active projects generally exhibit recent commit activity, with Last Commit Since values frequently below one year, suggesting ongoing maintenance. In contrast, Inactive repositories tend to have longer periods since their last commit, often 2-4 years, reflecting reduced or paused development."

Section III-F defines the maintenance labels using commit recency: Active repositories 'had at least one commit in the last 6 months' and Inactive ones have 'little activity throughout the repository in the last 2 years.' Section IV-D's lifespan finding is the same commit-recency distinction restated: Active projects have recent last commits, Inactive projects have older ones. Since the labeled training data used for the classifier were themselves created from those criteria, the RQ3 result is largely a paraphrase of the label definitions rather than an independent measurement of how long fairness projects remain active.

full rationale

The paper's clear circularity is confined to the maintenance-status analyses. The Active/Inactive labels are produced by a Random Forest trained on repository metrics (forks, issues, PRs, commits, contributors), and Table II then reports Mann-Whitney tests on those same metrics, so the significant differences are partly a restatement of the classifier's inputs. Similarly, RQ3's lifespan finding that Active projects have recent last commits while Inactive projects have older ones essentially re-describes the Section III-F label definitions, which are themselves based on commit recency. The central abstract claim that '53 percent of fairness projects become inactive within the first three years' is not circular, but it is also not derivable from the described method: Section III-G only measures the timespan since the last commit, with no survival analysis, no three-year window, and no reported calculation for the 53% figure, and the paper gives inconsistent cohort sizes (50, 61, 62). The use of the authors' prior dataset [63] as a seed is transparent and is augmented by the GitHub keyword search, so it does not by itself make the dataset circular; the paper even concedes in Section VI-A that the sample is only a subset. Overall, the partial circularity in the maintenance and lifespan comparisons warrants a score of 6; the headline inactivity rate is a separate validity gap rather than an equivalence-to-inputs problem.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a convenience dataset assembled from the authors' own earlier taxonomy, a keyword search with an ad hoc top-10 cutoff, and a transferred maintenance classifier. These choices should be treated as assumptions rather than external benchmarks; the lifespan statistic is the most independent result.

free parameters (4)

GitHub keyword search top-k cutoff = 10
The search keeps the top 10 results per keyword (Sec III-C), directly determining which emergent tools enter the dataset and all downstream percentages.
Active window = 6 months
Repositories with at least one commit in the last 6 months are labeled Active, following Coelho et al. [26]; the threshold determines the 32.2% active figure.
Inactive window = 2 years
Inactive status follows the unmaintained criteria from [25] with little activity over the last 2 years; this builds the inactive category.
Random Forest classifier parameters = trained on 1003 repos in Coelho et al. [26]
The classifier used to label fairness repositories has fixed weights from prior work, not re-fit or validated on fairness tools; these weights materially affect the maintenance distribution.

assumptions (5)

domain assumption The Original Toolset from Mim et al. [63] is a valid and complete-enough seed for keyword extraction.
All keywords and the initial tool list derive from the authors' own prior taxonomy; if this seed is unrepresentative, the emergent search inherits the bias.
domain assumption GitHub repository metrics (stars, watchers, forks, PRs, commits) are meaningful proxies for community engagement and maintenance.
The paper treats these signals as engagement; prior work supports this, but it is an interpretive assumption, and the paper does not triangulate with user interviews.
ad hoc to paper A maintenance classifier trained on 1,003 general repositories transfers to fairness tool repositories.
The Random Forest from Coelho et al. [26] was not evaluated on fairness repositories; its transferability is assumed and it drives the active/inactive split.
standard math Mann-Whitney U test assumptions are satisfied (independent samples, ordinal data).
The test is appropriate for non-normal data, but multiple comparisons are not corrected, so p-values below 0.05 may include false positives.
domain assumption Analyzing only Python-based keyword imports captures the fairness tool ecosystem.
Sec III-A restricts keyword analysis to Python tools; non-Python fairness tooling such as R or JavaScript is underrepresented.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Makes a Fairness Tool Project Sustainable in Open Source?." pith.science (2026). https://pith.science/paper/YPMWKUC4

@misc{pith2026250509802,
  author       = {Pith},
  title        = {Pith review of: What Makes a Fairness Tool Project Sustainable in Open Source?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPMWKUC4}},
  note         = {Machine review of arXiv:2505.09802}
}

read the original abstract

As society becomes increasingly reliant on artificial intelligence, the need to mitigate risk and harm is paramount. In response, researchers and practitioners have developed tools to detect and reduce undesired bias, commonly referred to as fairness tools. Many of these tools are publicly available for free use and adaptation. While the growing availability of such tools is promising, little is known about the broader landscape beyond well-known examples like AI Fairness 360 and Fairlearn. Because fairness is an ongoing concern, these tools must be built for long-term sustainability. Using an existing set of fairness tools as a reference, we systematically searched GitHub and identified 50 related projects. We then analyzed various aspects of their repositories to assess community engagement and the extent of ongoing maintenance. Our findings show diverse forms of engagement with these tools, suggesting strong support for open-source development. However, we also found significant variation in how well these tools are maintained. Notably, 53 percent of fairness projects become inactive within the first three years. By examining sustainability in fairness tooling, we aim to promote more stability and growth in this critical area.

Figures

Figures reproduced from arXiv: 2505.09802 by the authors.

**Figure 3.** Distribution of Watches 2) Project Engagement: To characterize deeper engagement with fairness tool projects, we analyzed the annual counts of open, closed, and merged pull requests (PRs) to observe distinct patterns of development and maintenance activity ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Evolution of Open/Closed/Merged PR for the projects [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Distribution of Forks that Active repositories exhibit more balance across open PRs, closedPRs, and merged PRs. For Inactive projects, we observed fewer contributions (lower owner commits, contributors, and most active dev commits). We also observed longer periods of …

**Figure 6.** Figure 6: Distribution of Mean Issue Resolution Time [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Distribution of Lifespan of the Projects [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 70 canonical work pages

[1]

https://deon.dri vendata.org/

An ethics checklist data scientists. https://deon.dri vendata.org/
[2]

https://ai.facebook.com/blog/how-we re-using-fairness-ﬂow-to-help-build-ai-that-works-b etter-for-everyone/

Fairness Flow. https://ai.facebook.com/blog/how-we re-using-fairness-ﬂow-to-help-build-ai-that-works-b etter-for-everyone/
[3]

https://github.com/pymetrics/a udit-ai./

pymetric/audit-ai. https://github.com/pymetrics/a udit-ai./
[4]

https://github.com/linkedin/LiFT/, 2021

2021
[5]

https://github.com/Tizpaz/Parfait-ML/, 2021

2021
[6]

https://github.com/dchen236/FairFace/ , 2021

2021
[7]

An empirical study on the survival rate of github projects

Adem Ait, Javier Luis C´ anovas Izquierdo, and Jordi Cabo t. An empirical study on the survival rate of github projects. In Proceedings of the 19th International Conference on Mining Software Repositories , pages 365– 375, 2022

2022
[8]

A taxo nomy and mapping of computer-based critiquing tools

Norhayati Mohd Ali, John Hosking, and John Grundy. A taxo nomy and mapping of computer-based critiquing tools. IEEE Transactions on Software Engineering, 39(11):1494–1520, 2013

2013

Show all 89 references

[9]

Uncovering and mitigating algorit hmic bias through learned latent structure

Alexander Amini, Ava P Soleimany, Wilko Schwarting, San geeta N Bhatia, and Daniela Rus. Uncovering and mitigating algorit hmic bias through learned latent structure. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 289–295, 2019

2019
[10]

Fairness tool evaluation submission 0b7e

Anonymous. Fairness tool evaluation submission 0b7e. https://anonymous.4open.science/r/Fairness-tool-evaluation-submission-0B7E/, 2024

2024
[11]

Gpt-4: A review o n advancements and opportunities in natural language proces sing

Jawid Ahmad Baktash and Mursal Dawodi. Gpt-4: A review o n advancements and opportunities in natural language proces sing. arXiv preprint arXiv:2305.03195, 2023

2023 arXiv
[12]

Themis-ml: A fairness-aware machine l earning interface for end-to-end discrimination discovery and mitigation

Niels Bantilan. Themis-ml: A fairness-aware machine l earning interface for end-to-end discrimination discovery and mitigation. Journal of Technology in Human Services , 36(1):15–30, 2018

2018
[13]

Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report

Lingfeng Bao, Zhenchang Xing, Xin Xia, David Lo, and Sha nping Li. Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pages 170–181...

2017
[14]

Soc ial networking meets software development: Perspectives from github, msd n, stack exchange, and topcoder

Andrew Begel, Jan Bosch, and Margaret-Anne Storey. Soc ial networking meets software development: Perspectives from github, msd n, stack exchange, and topcoder. IEEE software, 30(1):52–66, 2013

2013
[15]

Ai fairness 36 0: An extensible toolkit for detecting and mitigating algorithm ic bias

Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Ho ffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquely n Martino, Sameep Mehta, Aleksandra Mojsilovi´ c, et al. Ai fairness 36 0: An extensible toolkit for detecting and mitigating algorithm ic bias. IBM Journa...

2019
[16]

Marcus Vinicius Bertoncello, Gustavo Pinto, Igor Scal iante Wiese, and Igor Steinmacher. Pull requests or commits? which method sh ould we use to study contributors’ behavior? In 2020 IEEE 27th Interna- tional Conference on Software Analysis, Evolution and Reen gineering (SANE...

2020
[17]

Stefano A Bini. Artiﬁcial intelligence, machine learn ing, deep learning, and cognitive computing: what do these terms mean and how wil l they impact health care? The Journal of arthroplasty, 33(8):2358–2361, 2018

2018
[18]

Fairlearn: A toolkit for assessing and improving fa irness in ai

Sarah Bird, Miro Dud´ ık, Richard Edgar, Brandon Horn, R oman Lutz, V anessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathle en Walker. Fairlearn: A toolkit for assessing and improving fa irness in ai. Microsoft, Tech. Rep. MSR-TR-2020-32 , 2020

2020
[19]

Do the machine learning models on a crowd sourced platform exhibit bias? an empirical study on model fairness

Sumon Biswas and Hridesh Rajan. Do the machine learning models on a crowd sourced platform exhibit bias? an empirical study on model fairness. In Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the found ations of software...

2020
[20]

What’s in a githu b star? understanding repository starring practices in a social co ding platform

Hudson Borges and Marco Tulio V alente. What’s in a githu b star? understanding repository starring practices in a social co ding platform. Journal of Systems and Software , 146:112–129, 2018

2018
[21]

Software fairness

Y uriy Brun and Alexandra Meliou. Software fairness. In Proceedings of the 2018 26th ACM joint meeting on european software engin eering conference and symposium on the foundations of software eng ineering, pages 754–759, 2018

2018
[22]

GraphQL in action

Samer Buna. GraphQL in action . Simon and Schuster, 2021

2021
[23]

Reuse and maintenance practices among divergent forks in three so ftware ecosystems

John Businge, Moses Openja, Sarah Nadi, and Thorsten Be rger. Reuse and maintenance practices among divergent forks in three so ftware ecosystems. Empirical Software Engineering , 27(2):54, 2022

2022
[24]

Fairness in machine lea rning: A survey

Simon Caton and Christian Haas. Fairness in machine lea rning: A survey. arXiv preprint arXiv:2010.04053 , 2020

2010 arXiv
[25]

Why modern open source projects fail

Jailton Coelho and Marco Tulio V alente. Why modern open source projects fail. In Proceedings of the 2017 11th Joint meeting on foundations of software engineering , pages 186–196, 2017

2017
[26]

Is this github project maintained? measuring the lev el of maintenance activity of open-source projects

Jailton Coelho, Marco Tulio V alente, Luciano Milen, an d Luciana L Silva. Is this github project maintained? measuring the lev el of maintenance activity of open-source projects. Information and Software Technology, 122:106274, 2020

2020
[27]

Social coding in github: transparency and collaboration in an open software repository

Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herb sleb. Social coding in github: transparency and collaboration in an open software repository. In Proceedings of the ACM 2012 conference on computer supported cooperative work , pages 1277–1286, 2012

2012
[28]

Sampl ing projects in github for msr studies

Ozren Dabic, Emad Aghajani, and Gabriele Bavota. Sampl ing projects in github for msr studies. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) , pages 560–564. IEEE, 2021

2021
[29]

Identifying and ch aracterizing unmaintained projects in github

Jailton Junior de Sousa Coelho et al. Identifying and ch aracterizing unmaintained projects in github. 2019

2019
[30]

Syntax is all you need: A universal-language app roach to mutant generation

Sourav Deb, Kush Jain, Rijnard V an Tonder, Claire Le Gou es, and Alex Groce. Syntax is all you need: A universal-language app roach to mutant generation. Proceedings of the ACM on Software Engineering , 1(FSE):654–674, 2024

2024
[31]

A taxonomy a nd catalog of runtime software-fault monitoring tools

Nelly Delgado, Ann Q Gates, and Steve Roach. A taxonomy a nd catalog of runtime software-fault monitoring tools. IEEE Transactions on software Engineering , 30(12):859–872, 2004

2004
[32]

Exploring how machine learning practitioners (try to) use f airness toolkits

Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, and Hai yi Zhu. Exploring how machine learning practitioners (try to) use f airness toolkits. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, ...

2022
[33]

Predicting long-time contributors for github projects usi ng machine learning

Vijaya Kumar Eluri, Thomas A Mazzuchi, and Shahram Sark ani. Predicting long-time contributors for github projects usi ng machine learning. Information and Software Technology , 138:106616, 2021

2021
[34]

Gender bias in translation using googl e translate: Problems and solution

Tira Nur Fitria. Gender bias in translation using googl e translate: Problems and solution. Language Circle: Journal of Language and Literature, 15(2), 2021

2021
[35]

2020 s urvey of artiﬁcial general intelligence projects for ethics, risk, and policy

McKenna Fitzgerald, Aaron Boddy, and Seth D Baum. 2020 s urvey of artiﬁcial general intelligence projects for ethics, risk, and policy. Global Catastrophic Risk Institute Technical Report , pages 20–1, 2020

2020
[36]

Prac tical and open source best practices for ethical machine learning

Jeroen Franse, Violeta Misheva, and Daniel S V ale. Prac tical and open source best practices for ethical machine learning. In Towards Trustworthy Artiﬁcial Intelligent Systems , pages 77–84. Springer, 2022

2022
[37]

Justicia: A stochastic sat approach to formally verify fairness

Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S Meel . Justicia: A stochastic sat approach to formally verify fairness. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence , volume 35, pages 7554– 7563, 2021

2021
[38]

The quest for open source projects that u se uml

Mining GitHub. The quest for open source projects that u se uml
[39]

Olf-ml : An offen- sive language framework for detection, categorization, an d offense target identiﬁcation using text processing and machine learning a lgorithms

MD Nahid Hasan, Kazi Shadman Sakib, Taghrid Tahani Pree ti, Jeza Allohibi, Abdulmajeed Atiah Alharbi, and Jia Uddin. Olf-ml : An offen- sive language framework for detection, categorization, an d offense target identiﬁcation using text processing and machine learning a lgorith...

2024
[40]

Same ﬁle, different changes: the potential of meta- maintenance on github

Hideaki Hata, Raula Gaikovina Kula, Takashi Ishio, and Christoph Treude. Same ﬁle, different changes: the potential of meta- maintenance on github. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages 773–784. IEEE, 2021. 10

2021
[41]

F airea: A model behaviour mutation approach to benchmarking bias mi tigation methods

Max Hort, Jie M Zhang, Federica Sarro, and Mark Harman. F airea: A model behaviour mutation approach to benchmarking bias mi tigation methods. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the F ound ations of Softwar...

2021
[42]

Fairness-repository-mining-

INSPIRED-GMU. Fairness-repository-mining-. Availa ble at GitHub: Applicationsfoundagainstethicalkeywords.xlsx , 2023. https://github.com/INSPIRED-GMU/Fairness-repository-mining-

2023
[43]

Fairness-repository-mining-

INSPIRED-GMU. Fairness-repository-mining-. Availa ble at GitHub: Applicationsfoundagainsttool-speciﬁckey-word s.xlsx, 2023. https://github.com/INSPIRED-GMU/Fairness-repository-mining-

2023
[44]

Assurance of machine learning/tinyml in sa fety-critical domains

Zain Iqbal. Assurance of machine learning/tinyml in sa fety-critical domains. In 2022 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC) , pages 1–2. IEEE, 2022

2022
[45]

Github projects

Oskar Jarczyk, Bła˙ zej Gruszka, Szymon Jaroszewicz, L eszek Bukowski, and Adam Wierzbicki. Github projects. quality analysis of o pen-source software. In Social Informatics: 6th International Conference, SocInf o 2014, Barcelona, Spain, November 11-13, 2014. Proceedings 6, pa...

2014
[46]

Availability and usage of pl atform-speciﬁc apis: A ﬁrst empirical study

Ricardo Job and Andre Hora. Availability and usage of pl atform-speciﬁc apis: A ﬁrst empirical study. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) , pages 27–31. IEEE, 2024

2024
[47]

Fairkit, fairkit, on the wall, who’s the fairest of them all? supporting data scientists in training fair models

Brittany Johnson, Jesse Bartola, Rico Angell, Katheri ne Keith, Sam Witty, Stephen J Giguere, and Y uriy Brun. Fairkit, fairkit, on the wall, who’s the fairest of them all? supporting data scientists in training fair models. arXiv preprint arXiv:2012.09951 , 2020

2012 arXiv
[48]

Make your tools sparkle with trust: The p icse framework for trust in software tools

Brittany Johnson, Christian Bird, Denae Ford, Nicole F orsgren, and Thomas Zimmermann. Make your tools sparkle with trust: The p icse framework for trust in software tools. In 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering: Software Eng ineering in Pract...

2023
[49]

Fairkit-learn: a fair ness evaluation and comparison toolkit

Brittany Johnson and Y uriy Brun. Fairkit-learn: a fair ness evaluation and comparison toolkit. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceeding s, pages 70–74, 2022

2022
[50]

Towards ethical dat a-driven software: ﬁlling the gaps in ethics research & practice

Brittany Johnson and Justin Smith. Towards ethical dat a-driven software: ﬁlling the gaps in ethics research & practice. In 2021 IEEE/ACM 2nd International W orkshop on Ethics in Software Engineering Research and Practice (SEthics) , pages 18–25. IEEE, 2021

2021
[51]

An overview of ethical issues in u sing ai systems in hiring with a case study of amazon’s ai based hirin g tool

Akhil Alfons Kodiyan. An overview of ethical issues in u sing ai systems in hiring with a case study of amazon’s ai based hirin g tool. Researchgate Preprint, pages 1–19, 2019

2019
[52]

A survey on datasets for fairness-aware machine le arning

Tai Le Quy, Arjun Roy, V asileios Iosiﬁdis, Wenbin Zhang , and Eirini Ntoutsi. A survey on datasets for fairness-aware machine le arning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Disc overy, 12(3):e1452, 2022

2022
[53]

The landscape and gap s in open source fairness toolkits

Michelle Seng Ah Lee and Jat Singh. The landscape and gap s in open source fairness toolkits. In Proceedings of the 2021 CHI conference on human factors in computing systems , pages 1–13, 2021

2021
[54]

David J. Lilja. Measuring computer performance : A prac titioner’s guide. 2000

2000
[55]

The possessive investment in whiteness: How white people proﬁt from identity politics

George Lipsitz. The possessive investment in whiteness: How white people proﬁt from identity politics . Temple University Press, 2006

2006
[56]

Bias mit igation post-processing for individual and group fairness

Pranay K Lohia, Karthikeyan Natesan Ramamurthy, Manis h Bhide, Diptikalyan Saha, Kush R V arshney, and Ruchir Puri. Bias mit igation post-processing for individual and group fairness. In Icassp 2019-2019 ieee international conference on acoustics, speech and sig nal processing...

2019
[57]

Survey on causal-based machine learning fairness notions

Karima Makhlouf, Sami Zhioua, and Catuscia Palamidess i. Survey on causal-based machine learning fairness notions. arXiv preprint arXiv:2010.09553, 2020

2010 arXiv
[58]

On the applicability of machine learning fairness notions

Karima Makhlouf, Sami Zhioua, and Catuscia Palamidess i. On the applicability of machine learning fairness notions. ACM SIGKDD Explorations Newsletter, 23(1):14–23, 2021

2021
[59]

A tax- onomy of tools and approaches for fairiﬁcation

Dario Mangione, Leonardo Candela, and Donatella Caste lli. A tax- onomy of tools and approaches for fairiﬁcation. In CEUR W orkshop Proceedings, Padova, Italy , pages 24–25, 2022

2022
[60]

Mining co-change information to understand when bu ild changes are necessary

Shane McIntosh, Bram Adams, Meiyappan Nagappan, and Ah med E Hassan. Mining co-change information to understand when bu ild changes are necessary. In 2014 IEEE International Conference on Software Maintenance and Evolution , pages 241–250. IEEE, 2014

2014
[61]

Statistical methods for reliability data

William Q Meeker, Luis A Escobar, and Francis G Pascual. Statistical methods for reliability data . John Wiley & Sons, 2022

2022
[62]

A survey on bias and fairness in machine le arning

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kr istina Lerman, and Aram Galstyan. A survey on bias and fairness in machine le arning. ACM Computing Surveys (CSUR) , 54(6):1–35, 2021

2021
[63]

A t axonomy of machine learning fairness tool speciﬁcations, features an d workﬂows

Sadia Afrin Mim, Justin Smith, and Brittany Johnson. A t axonomy of machine learning fairness tool speciﬁcations, features an d workﬂows. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) , pages 222–225, 2023

2023
[64]

Peer interactio n effectively, yet infrequently, enables programmers to discover new tools

Emerson Murphy-Hill and Gail C Murphy. Peer interactio n effectively, yet infrequently, enables programmers to discover new tools. In Proceed- ings of the ACM 2011 conference on Computer supported cooper ative work, pages 405–414, 2011

2011
[65]

An automated approach to assess the similarity of githu b reposito- ries

Phuong T Nguyen, Juri Di Rocco, Riccardo Rubei, and Davi de Di Rus- cio. An automated approach to assess the similarity of githu b reposito- ries. Software Quality Journal , 28:595–631, 2020

2020
[66]

Assessing and mitigating bias in medical artiﬁcia l intelligence: the effects of race and ethnicity on a deep learning model for ecg anal- ysis

Peter A Noseworthy, Zachi I Attia, LaPrincess C Brewer, Sharonne N Hayes, Xiaoxi Y ao, Suraj Kapa, Paul A Friedman, and Francisc o Lopez- Jimenez. Assessing and mitigating bias in medical artiﬁcia l intelligence: the effects of race and ethnicity on a deep learning model for e...

2020
[67]

Scikit-learn: Mac hine learning in python

Fabian Pedregosa, Ga¨ el V aroquaux, Alexandre Gramfor t, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel , Peter Pretten- hofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Mac hine learning in python. the Journal of machine Learning research , 12:2825–...

2011
[68]

A review on fairness in ma chine learning

Dana Pessach and Erez Shmueli. A review on fairness in ma chine learning. ACM Computing Surveys (CSUR) , 55(3):1–44, 2022

2022
[69]

Towards fairness in practice: A practitioner-oriented rubric for evaluating fair ml toolk its

Brianna Richardson, Jean Garcia-Gathright, Samuel F W ay, Jennifer Thom, and Henriette Cramer. Towards fairness in practice: A practitioner-oriented rubric for evaluating fair ml toolk its. In Proceed- ings of the 2021 CHI Conference on Human Factors in Computing Systems, page...

2021
[70]

Estimating development effort in free/open source software projects by mining software re positories: a case study of openstack

Gregorio Robles, Jes´ us M Gonz´ alez-Barahona, CarlosCervig´ on, Andrea Capiluppi, and Daniel Izquierdo-Cort´ azar. Estimating development effort in free/open source software projects by mining software re positories: a case study of openstack. In Proceedings of the 11th W or...

2014
[71]

{SourceFinder}: Finding malware {Source-Code} from publicly available repositories in {GitHub}

Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelo s E Papalexakis, and Michalis Faloutsos. {SourceFinder}: Finding malware {Source-Code} from publicly available repositories in {GitHub}. In 23rd International Symposium on Research in Attacks, Intru sions and Defenses (RA...

2020
[72]

Aequi tas: A bias and fairness audit toolkit

Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. Aequi tas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577 , 2018

2018 arXiv
[73]

Towards m ining norms in open source software repositories

Bastin Tony Roy Savarimuthu and Hoa Khanh Dam. Towards m ining norms in open source software repositories. In Agents and Data Mining Interaction: 9th International W orkshop, ADMI 2013, Saint Paul, MN, USA, May 6-7, 2013, Revised Selected Papers 9 , pages 26–39. Springer, 2014

2013
[74]

Towards efﬁcient software engineering in t he era of ai and ml: Best practices and challenges

V arun Shah. Towards efﬁcient software engineering in t he era of ai and ml: Best practices and challenges. International Journal of Computer Science and Technology , 3(3):63–78, 2019

2019
[75]

Taxonomy of automated software testing tools

Kamran Shaukat, Usman Shaukat, Faran Feroz, Shahraiz K ayani, and Ali Akbar. Taxonomy of automated software testing tools. International Journal of Computer Science and Innovation , 1:7–18, 2015

2015
[76]

Fat forensics: a python toolbox for algorithmic fairness, accountability a nd transparency

Kacper Sokol, Raul Santos-Rodriguez, and Peter Flach. Fat forensics: a python toolbox for algorithmic fairness, accountability a nd transparency. arXiv preprint arXiv:1909.05167 , 2019

1909 arXiv
[77]

Fat forensics: A python toolbox for algorithmic fairness, accountability a nd transparency

Kacper Sokol, Raul Santos-Rodriguez, and Peter Flach. Fat forensics: A python toolbox for algorithmic fairness, accountability a nd transparency. Software Impacts, 14:100406, 2022

2022
[78]

The language interpretabilit y tool: Extensible, interactive visualizations and analysis for n lp models

Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolu kbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna , Carey Radebaugh, Emily Reif, et al. The language interpretabilit y tool: Extensible, interactive visualizations and analysis for n lp models. arXiv pr...

2008 arXiv
[79]

Fairness-aware conﬁguration of machine learning librarie s

Saeid Tizpaz-Niari, Ashish Kumar, Gang Tan, and Ashuto sh Trivedi. Fairness-aware conﬁguration of machine learning librarie s. In Pro- ceedings of the 44th International Conference on Software E ngineering, pages 909–920, 2022

2022
[80]

Sveva fair: A framew ork for eval- uating fairness in speaker veriﬁcation

Wiebke Toussaint and Aaron Yi Ding. Sveva fair: A framew ork for eval- uating fairness in speaker veriﬁcation. arXiv preprint arXiv:2107.12049, 2021. 11

2021 arXiv
[81]

In-p rocessing modeling techniques for machine learning fairness: A surve y

Mingyang Wan, Daochen Zha, Ninghao Liu, and Na Zou. In-p rocessing modeling techniques for machine learning fairness: A surve y. ACM Transactions on Knowledge Discovery from Data , 17(3):1–27, 2023

2023
[82]

Mitigating bias in face recog nition using skewness-aware reinforcement learning

Mei Wang and Weihong Deng. Mitigating bias in face recog nition using skewness-aware reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni tion, pages 9322–9331, 2020

2020
[83]

The what-if tool: Inter active probing of machine learning models

James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Mart in Wattenberg, Fernanda Vi´ egas, and Jimbo Wilson. The what-if tool: Inter active probing of machine learning models. IEEE transactions on visualization and computer graphics , 26(1):56–65, 2019

2019
[84]

ethical ai

David Gray Widder, Dawn Nafus, Laura Dabbish, and James Herbsleb. Limits and possibilities for “ethical ai” in open source: A s tudy of deepfakes. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 2035–2046, 2022

2022
[85]

T he rise and potential of large language model based agents: A survey

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, B oyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. T he rise and potential of large language model based agents: A survey . arXiv preprint arXiv:2309.07864, 2023

2023 arXiv
[86]

What do users ask in open-source ai repositories? an empirical study of github i ssues

Zhou Y ang, Chenyu Wang, Jieke Shi, Thong Hoang, Pavneet Kochhar, Qinghua Lu, Zhenchang Xing, and David Lo. What do users ask in open-source ai repositories? an empirical study of github i ssues. In 2023 IEEE/ACM 20th International Conference on Mining Soft ware Repositories (...

2023
[87]

Measuring the m aintain- ability of open-source software

Liguo Y u, Stephen R Schach, and Kai Chen. Measuring the m aintain- ability of open-source software. In 2005 International Symposium on Empirical Software Engineering, 2005. , pages 7–pp. IEEE, 2005

2005
[88]

” ignorance and prejudice” i n software fairness

Jie M Zhang and Mark Harman. ” ignorance and prejudice” i n software fairness. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages 1436–1447. IEEE, 2021

2021
[89]

What the fork: a study of inefﬁcient and efﬁcient forking practices in soci al coding

Shurui Zhou, Bogdan V asilescu, and Christian K¨ astner . What the fork: a study of inefﬁcient and efﬁcient forking practices in soci al coding. In Proceedings of the 2019 27th ACM joint meeting on european so ftware engineering conference and symposium on the foundations of s...

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

https://deon.dri vendata.org/

An ethics checklist data scientists. https://deon.dri vendata.org/

[2] [2]

https://ai.facebook.com/blog/how-we re-using-fairness-ﬂow-to-help-build-ai-that-works-b etter-for-everyone/

Fairness Flow. https://ai.facebook.com/blog/how-we re-using-fairness-ﬂow-to-help-build-ai-that-works-b etter-for-everyone/

[3] [3]

https://github.com/pymetrics/a udit-ai./

pymetric/audit-ai. https://github.com/pymetrics/a udit-ai./

[4] [4]

https://github.com/linkedin/LiFT/, 2021

2021

[5] [5]

https://github.com/Tizpaz/Parfait-ML/, 2021

2021

[6] [6]

https://github.com/dchen236/FairFace/ , 2021

2021

[7] [7]

An empirical study on the survival rate of github projects

Adem Ait, Javier Luis C´ anovas Izquierdo, and Jordi Cabo t. An empirical study on the survival rate of github projects. In Proceedings of the 19th International Conference on Mining Software Repositories , pages 365– 375, 2022

2022

[8] [8]

A taxo nomy and mapping of computer-based critiquing tools

Norhayati Mohd Ali, John Hosking, and John Grundy. A taxo nomy and mapping of computer-based critiquing tools. IEEE Transactions on Software Engineering, 39(11):1494–1520, 2013

2013