{"id":"b1509c5a-caea-41ea-a8f1-97eb4addae9a","arxiv_id":"2505.09802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A GitHub mining study of 61 fairness tools finds that only about a third are actively maintained and industry-backed projects are the most stable.","lead":"This paper studied 61 open source fairness tools on GitHub and found that most stop being maintained within a few years. It shows industry-backed tools are the most stable, but the headline 53 percent statistic is not fully backed by the paper's own figures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 53% three-year inactivity rate appears only in the abstract and is not derivable from the described method; the paper's own cohort sizes also oscillate between 50, 61, and 62 projects.","rationale":"I agree with the reader that the representativeness of the 61-tool dataset is a real threat: the search is seeded by the authors' own Original Toolset, restricted to Python import keywords, and truncated to the top 10 results per keyword, so population-level percentages are fragile. However, the more load-bearing problem is that the central statistic itself is not traceable. Even a perfectly representative sample cannot support a percentage that the paper never shows how to compute. The reader flagged the 53% statistic as not traceable in their rationale, but their formal weakest_assumption was dataset representativeness; my emphasis is different, hence partial agreement. I keep the reader's CONDITIONAL verdict rather than moving to REJECT because the concern is addressable: the authors can report the exact calculation, define the inactivity window, handle censoring, and reconcile the 50/61/62 cohort sizes. The paper's public artifact and the plausibility of the broader finding that many fairness tools lose maintenance activity are independent points in its favor, but the headline claim should not be accepted at face value until the computation is reproduced.","tokens_in":16256,"tokens_out":4013,"duration_ms":44325,"concrete_test":"Obtain the public artifact (anonymous.4open.science/r/Fairness-Tool-Maintenace--BEB1), reconstruct the repository list, and compute for every repository the time from first commit to the last commit before a gap of at least 12 months (or the paper's stated 'inactive' threshold). Count how many repositories have that duration at most three years. The 53% claim lands only if this count is approximately 32 of 61 and if the same count holds when using the trained classifier's Active/Inactive labels rather than raw commit gaps. If the number is not reproducible, or if it merely reflects current inactive status rather than inactivity within the first three years, the abstract overstates the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is '53 percent of fairness projects become inactive within the first three years' (Abstract). Section III-G, which addresses project lifespan, says only that the authors 'measured the timespan of the last commit since their initial commit.' That measures how recently a repository last changed, not how long after its first commit it became inactive. There is no survival analysis, no definition of the three-year window, no treatment of right-censoring for young projects, and no threshold for 'inactive' tied to the claim. The 53% figure is never mentioned again in Section IV-D or any table, so it cannot be traced to a reported calculation. Compounding this, the full text gives inconsistent cohort sizes: 50 in the Abstract, 61 in the Introduction, 62 in Section IV-A, and 61 in the Conclusion. A public replication artifact is a point in the paper's favor, but it does not validate a statistic whose derivation is absent from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a GitHub mining study of open-source machine learning fairness tools. The authors extend a previously curated dataset with GitHub keyword search, classify repositories into Active, Inactive, and Archived states using a Random Forest model adopted from prior work, and compare community engagement and maintenance metrics between these groups. The central claimed result is that 53 percent of fairness projects become inactive within their first three years, with only 32.2 percent of the studied projects active at the time of data collection. The paper also analyzes stars, forks, pull requests, issue resolution times, and maintenance-related keywords, and discusses implications for sustainability of fairness tooling.","tokens_in":16435,"tokens_out":3838,"duration_ms":40554,"significance":"If the headline statistic were supported, the paper would supply an important empirical baseline for the responsible-AI and software-engineering communities: it would quantify the fragility of the open-source fairness tool ecosystem and would motivate targeted sustainability efforts. The paper has genuine strengths: it ships a public replication artifact, it makes a systematic attempt to extend an existing dataset, and its descriptive analyses of star, fork, and pull-request activity provide useful qualitative insight into engagement patterns. However, the current manuscript does not support the headline three-year inactivity rate, and the maintenance-classification analysis contains a circularity that undermines the statistical comparisons in Table II. The significance of the contribution therefore depends on substantial reanalysis and clarification.","major_comments":[{"comment":"The claim that \"53 percent of fairness projects become inactive within the first three years\" is not derivable from the methods described. Section III-G states that the authors measured \"the timespan of the last commit since their initial commit,\" which measures recency of the last activity relative to the data-collection date, not the elapsed time from a project's first commit to its becoming inactive. The manuscript provides no definition of a three-year window, no survival analysis, no treatment of right-censoring for projects younger than three years, and no threshold for \"inactive\" tied to the claim. The 53 percent figure does not appear in Section IV-D, Figure 7, or any table, so it cannot be traced to a reported calculation. The authors must either perform a proper time-to-event analysis and report the exact computation, or remove the claim from the abstract and framing.","section":"Abstract; Section III-G; Section IV-D"},{"comment":"The Mann-Whitney U tests in Table II are partly circular. The Active/Inactive labels used as the grouping variable come from a Random Forest classifier trained on features that include total issues, closed issues, open pull requests, closed pull requests, merged pull requests, total commits, most active developer commits, and contributors. Table II then tests for differences between the Active and Inactive groups on those same features. Significant differences on classifier training features are expected by construction and do not provide independent evidence about what distinguishes maintained from unmaintained fairness tools. This affects the interpretation of the starred rows in Table II and the related discussion in Section IV-C. Please re-run the analysis using labels obtained independently of the tested features, or restrict the significance tests to features that were not used in classification.","section":"Section III-F; Table II"},{"comment":"The reported cohort size is inconsistent across the manuscript: the Abstract says 50 projects, the Introduction and Conclusion say 61, Section IV-A says 62, and Section III-F states that the test set comprised 61 fairness-related repositories. Because the paper's percentages (53 percent, 32.2 percent, 62.7 percent, 5.1 percent) all depend on the denominator, this inconsistency must be resolved. In addition, the sampling procedure uses a top-10 cutoff per keyword and only Python import statements, so the resulting set is likely a convenience subset rather than a complete census of fairness tools. The paper acknowledges this threat in Section VI-A, but still presents population-level rates; please report exact denominators and discuss how selection bias could change the headline percentages.","section":"Section III-A to III-D; Section IV-A; Abstract; Conclusion"},{"comment":"The transfer of the maintenance classifier from the 1,003 general repositories used in Coelho et al. to fairness-tool repositories is not validated. The manuscript reports no accuracy, precision, recall, or cross-validation results for the Random Forest, and it provides no manual validation of the predicted labels on fairness repositories. The 32.2 percent active estimate therefore has unknown error, and this error propagates to every comparison that uses the Active/Inactive grouping. Please report the classifier's performance on its original dataset and, ideally, validate a sample of fairness-repository labels manually or through a held-out fairness-specific test set.","section":"Section III-F"}],"minor_comments":[{"comment":"There is a missing citation placeholder in the first paragraph of the Introduction: the text contains \"[ ? ]\".","section":"Section I"},{"comment":"The text says \"a total of 62 open source fairness tools in Figure ??\", but no figure number is provided; the reference should be corrected.","section":"Section IV-A"},{"comment":"The repository name \"pymetrics/audit-ai\" is sometimes written as \"pymetrics/auditai\" and \"pymetrics/auditai\"; please standardize the spelling.","section":"Section IV; Section IV-C"},{"comment":"There are several typographical errors, including \"predicticing\" (Section II-A), \"T echnology\" (the author affiliation line), \"langauge\" (Section I), \"V ares\" (author list), and \"Maintability\" (the subsection heading in Section IV-C).","section":"Throughout"},{"comment":"The description of the Random Forest classifier omits the model hyperparameters, the exact feature-selection outcome after the correlation-based clustering, and the version of the classifier implementation; adding these details would improve replicability.","section":"Section III-F"},{"comment":"The anonymous artifact link should be replaced with a permanent public link in the camera-ready version, since the current link may not be accessible after the review process.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper's biggest risk is that the headline result is currently unsupported by the described methodology. I recommend the editor ask the authors for a full reanalysis of RQ3 using survival-analysis methods and for an independent validation of the maintenance labels before considering the paper for publication. The public artifact is a positive signal, but it does not substitute for a traceable derivation of the 53 percent claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the fairness tools sustainability paper. It's a genuinely useful empirical contribution—the first dataset I know of that tracks maintenance and engagement across a broad set of open source fairness tools, with a public replication artifact. The descriptive patterns (industry-backed projects more active, engagement concentrated in a few large repos) are plausible and consistent with what I'd expect. I'd happily point students to the dataset.\n\nThe problem is the headline number. The 53% 'inactive within three years' claim in the abstract is not supported by the methods as written. RQ3's method is just 'timespan of last commit since initial commit'—that's not a survival analysis, no three-year window defined, no handling of right-censoring for young repos. The number never appears in the findings section. The stress-test note is right: the statistic is not derivable from the paper. That's a load-bearing flaw because it's the abstract's marquee finding.\n\nAlso, the maintenance classifier features (total issues, commits, etc.) are the same ones compared in Table II, so the significant differences are partly circular. The feature overlap doesn't invalidate the descriptive comparisons, but it does mean those p-values aren't independent evidence of a real distinction. The cohort sizes also wobble (50, 61, 62), which is sloppy. The search methodology (top-10 per keyword, Python-only imports, seeded from the authors' own taxonomy) is pragmatically reasonable but makes population-level percentages shaky.\n\nThat said, the paper is honest about some limitations in Section VI, and the core observation—that many fairness tools go quiet—is almost certainly correct in direction. What's missing is the analytical scaffolding to put a number on it.\n\nFor peer review: yes, this deserves serious refereeing. The dataset and descriptive analysis are worth publishing, but the paper needs major revision: either add a proper survival analysis or drop the three-year claim, address the circularity, and clean up the inconsistencies. If the authors do that, it's a solid contribution. I'd bring it to a reading group as a case study in why descriptive repo-mining papers need to match their claims to their methods.","headline":"Useful new dataset and plausible descriptive findings, but the headline '53% inactive in three years' claim is not backed by the methods as written.","tokens_in":16948,"tokens_out":2128,"would_cite":true,"duration_ms":22462,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"More than half of open-source fairness tools go inactive within three years, while only a third remain active at the time of the study.","keywords":["fairness tools","open source","GitHub","software sustainability","maintenance","community engagement","machine learning bias"],"falsifier":"Run a wider replication that searches GitHub without the top-ten cutoff, includes non-Python fairness tools, and tracks every qualifying repository for three years from creation; if the three-year inactivity rate in that larger sample is substantially below 53 percent, the paper's central claim would be refuted.","tokens_in":16048,"feed_emoji":"⚖️","tokens_out":6993,"duration_ms":68156,"temperature":0.7,"pith_summary":"The paper tries to establish that open-source fairness tools, meaning software for detecting and mitigating bias in machine learning, have a sustainability problem. Using a curated sample of 61 GitHub projects, it reports that 53 percent become inactive within three years and that only 32.2 percent are active at the time of study. This matters because fairness is an ongoing requirement: a bias-mitigation tool that stops being maintained can quietly become outdated or unreliable. The paper also finds that industry-backed tools receive the most consistent engagement and maintenance, while academic and individual projects are more fragile, with domain-specific tools as a notable exception. If the finding holds, the fairness tooling landscape is less healthy than the number of available tools suggests.","feed_headline":"53% of open-source fairness tools go inactive within three years","feed_subtitle":"A GitHub mining study of 61 bias-mitigation projects finds only a third still active; industry backs the stable ones.","key_machinery":"The carrying mechanism is a two-part empirical apparatus. The first part is a curated dataset of 61 projects built by extracting fairness-related natural-language and tool-specific keywords from an earlier taxonomy of fairness tools and querying the GitHub API, followed by manual validation of the results. The second is a maintenance classifier, adopted from prior work and trained on over a thousand labeled repositories, that labels each project Active, Inactive, or Archived using features such as forks, issues, pull requests, commits, and contributor counts over a 24-month window. Around this classifier, the paper layers engagement metrics, including stars, watches, forks, and pull requests, and a lifespan measurement that compares the time since the last commit with the project's age.","core_discovery":"The paper establishes that sustainability is the exception rather than the rule in open-source fairness tooling. Of the 61 projects analyzed, 53 percent become inactive within three years, and at the time of the study only 32.2 percent are classified as active, with 62.7 percent inactive and 5.1 percent archived. The most stable and consistently engaged repositories are affiliated with industry or non-academic organizations, while academic and individual projects tend to show lower, more sporadic pull-request and commit activity. The paper also observes that well-maintained tools can still lack engagement, so maturity and visibility matter as much as upkeep, and that a small set of domain-specific tools, for areas such as healthcare, language models, energy, and recommender systems, sustain steady if modest engagement. The central conclusion is that the fairness tool ecosystem is fragile, and that survival depends on factors beyond code quality, including organizational backing and niche relevance.","pith_inferences":["If the 53 percent inactivity rate generalizes beyond Python and GitHub, it implies that funders and institutions should plan multi-year stewardship for fairness tools rather than one-off releases.","The paper does not distinguish abandoned projects from feature-complete ones; an inactive repository that stopped changing because it works would soften the bleak reading, so a future analysis separating 'done' from 'dead' would sharpen the result.","The late-2023 engagement surge in language-model-related fairness tools suggests that sustainability may track funding and hype cycles; one could test whether tools tied to currently funded subfields survive longer than tools tied to older problem framings.","Higher owner-commit counts in inactive projects hint at bus-factor risk, meaning projects maintained by one person may die when that person steps away; testing contributor concentration as a predictor of inactivity would be a direct follow-up."],"forward_implications":["Practitioners choosing a fairness tool should treat maintenance status as a first-class criterion, since the majority of projects stop receiving commits within three years.","Maintainers and funders should plan for multi-year stewardship rather than one-time releases, because the data show that only industry-backed projects tend to sustain consistent engagement and maintenance.","Academic and individual fairness-tool projects may need institutional support or community scaffolding, since they show lower and more sporadic pull-request and commit activity.","Domain-specific fairness tools, such as those for healthcare or natural language processing, can sustain engagement even without a large organization behind them, suggesting that niche relevance is a resilience factor."],"supporting_citations":[{"why":"Supplies the Original Toolset and the fairness-related keywords that seed the GitHub search and define the sample.","marker":"[63]"},{"why":"Provides the three-way maintenance classification model, Active, Inactive, Archived, and the Random Forest approach used to label the 61 repositories.","marker":"[26]"},{"why":"Defines the criteria for an unmaintained repository that underpin the Inactive label.","marker":"[25]"},{"why":"Supports the choice of pull requests over raw commits as the more representative measure of contributor engagement.","marker":"[16]"},{"why":"Supports counting forks as a form of engagement that signals intent to build on or contribute to a project.","marker":"[89]"}],"fun_headline_variants":["53% of fairness tools go dark within 3 years","Only a third of open-source fairness tools stay active","Sustainability is rare in open-source fairness tools","Fairness tool projects often fizzle out in 3 years","Industry backing keeps fairness tools alive longer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central percentages rest on the assumption that 61 repositories found by searching the top ten GitHub results for fairness-related keywords, mostly Python tools, represent the whole population of open-source fairness tools.","fun_headline_variants_meta":{"raw":{"variants":["53% of fairness tools go dark within 3 years","Only a third of open-source fairness tools stay active","Sustainability is rare in open-source fairness tools","Fairness tool projects often fizzle out in 3 years","Industry backing keeps fairness tools alive longer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1326,"prompt_tokens":917,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":533,"tokens_out":409,"duration_ms":4175,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:23:43.578321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a wider replication that searches GitHub without the top-ten cutoff, includes non-Python fairness tools, and tracks every qualifying repository for three years from creation; if the three-year inactivity rate in that larger sample is substantially below 53 percent, the paper's central claim would be refuted.","supporting_citations":[{"cited_title":"A t axonomy of machine learning fairness tool specifications, features an d workflows","cited_arxiv_id":null,"evidence_quote":"Supplies the Original Toolset and the fairness-related keywords that seed the GitHub search and define the sample."},{"cited_title":"Is this github project maintained? measuring the lev el of maintenance activity of open-source projects","cited_arxiv_id":null,"evidence_quote":"Provides the three-way maintenance classification model, Active, Inactive, Archived, and the Random Forest approach used to label the 61 repositories."},{"cited_title":"Why modern open source projects fail","cited_arxiv_id":null,"evidence_quote":"Defines the criteria for an unmaintained repository that underpin the Inactive label."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the choice of pull requests over raw commits as the more representative measure of contributor engagement."},{"cited_title":"What the fork: a study of inefficient and efficient forking practices in soci al coding","cited_arxiv_id":null,"evidence_quote":"Supports counting forks as a form of engagement that signals intent to build on or contribute to a project."}],"review_version":1}