{"id":"c0498154-2d7c-43ae-a1d0-5656bc225827","arxiv_id":"1908.05986","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Research software in computer science stays active much longer (median 5 years) than in computational science (median 15 days), revealing different publishing and maintenance cultures.","lead":"This paper reviews how research software is published and maintained in different scientific fields, using over 5,000 GitHub repositories linked to papers. It finds that computer science software tends to stay active for years while computational science code is often abandoned within days, and recommends FAIR and open publication practices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ACM-vs-DOI lifespan gap likely conflates software-as-output with mature third-party tools cited by CS papers, so the 'reuse vs reproducibility' field difference is not established.","rationale":"The reader's weakest assumption was coverage bias from GitHub-only sampling. That is a real limitation, but it is not the load-bearing concern. Even with perfect hosting coverage, the comparison is invalid because the ACM and DOI samples are selected by different link directions and contain different repository types. The paper itself provides evidence for this: its ACM examples include Caffe, Spark, three.js, and Guava, which are not software outputs of the citing papers. Thus the median lifespan gap could be an artifact of mixing 'software as output' with 'software as tool/object of study'. This concern is directly testable with the paper's own taxonomy. I recommend keeping the conditional verdict, but the condition should explicitly require re-analysis restricted to comparable software-as-output repositories and release of the underlying data for that re-analysis.","tokens_in":11363,"tokens_out":6642,"duration_ms":68155,"concrete_test":"Manually classify a random sample of ACM-cited repositories using the paper's own five relationship categories, then recompute the median lifespan for only 'software as output of research' (the first two categories), excluding tools, objects of study, forks, and related-work citations. If the ACM median drops toward the DOI/arXiv medians, the headline gap is an artifact of comparing one-off output code with mature third-party code; if it remains near five years, the field-difference claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim compares Figure 3(a) ACM-cited repositories with Figure 3(b) DOI-citing repositories, but these two samples measure different constructs. The ACM set is built from publications that cite GitHub repositories, and it includes mature third-party tools such as Caffe, Apache Spark, three.js, and Google Guava, which are not software outputs of the citing papers. The DOI set is built from repositories that cite a publication, i.e. software self-identified as accompanying a paper. The paper's own taxonomy in 'Relationships and categories' lists 'Software used as a tool or framework', 'Software itself as an object of study or analysis', and 'related work, background, or example' as categories present in the ACM repositories. Including such repositories in the lifespan distribution inflates the ACM median: mature frameworks maintained by companies have lifespans of years, while one-off paper code has a burst of commits around publication. The observed 5-year vs 15-day gap may therefore reflect different selection criteria and link directions, not different 'publishing practices' or 'reuse vs reproducibility' cultures. No analysis controls for publication year or for relationship type between the paper and the repository, so the central field-difference claim is not supported by the presented evidence.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that research software should be FAIR and open, reviews current obstacles and practices (software credit, artifact evaluation, archival vs. development platforms), and presents an initial empirical study that relates research publications to GitHub repositories via three routes: publications in the ACM Digital Library, repositories containing DOIs, and e-prints on arXiv. The study reports lifespan distributions for the three sets, with a headline observation that ACM-linked computer science repositories have a median lifespan of about five years, DOI-linked computational science repositories have a median lifespan of about 15 days, and arXiv-linked repositories fall in between (about eight months). On this basis, the paper claims that computational science emphasizes reproducibility while computer science emphasizes reuse, and it proposes Research Software Observatories and recommendations for making computer science research software FAIR and open. The paper is framed as an initial study and explicitly discloses its sampling assumptions in 'Threats to Validity'.","tokens_in":11497,"tokens_out":5737,"duration_ms":55411,"significance":"If its empirical claims were established, this paper would provide a useful quantitative picture of field differences in research software sustainability and publishing practices, which is timely for the FAIR research software agenda. The authors deserve credit for clearly stating their sampling assumptions, for labeling the analysis as an initial study, and for pairing the empirical exploration with concrete recommendations (the GitHub/Zenodo archival-versus-development split, artifact evaluation, software citation metadata, and Research Software Observatories) that have value independent of the specific median values. The proposal of Research Software Observatories is a constructive step toward more rigorous comparative studies. However, the headline empirical claim is not currently supported by the presented evidence because of the confounds and missing statistical analysis detailed below.","major_comments":[{"comment":"The central empirical comparison in Figure 3 is confounded: the ACM-linked set is constructed from publications that cite GitHub repositories, and the paper's own examples (Caffe, Apache Spark, three.js, Google Guava, and the Hough et al. repository) show that it includes mature third-party tools and software studied as an object of analysis, which are not outputs of the citing papers. In contrast, the DOI-linked set is constructed from repositories that cite a publication, i.e., software self-identifying as accompanying a paper. The reported median lifespan gap (5 years versus 15 days) may therefore reflect different inclusion criteria and link directions rather than different field-specific publishing cultures. To support the 'reuse vs. reproducibility' claim, the analysis should either restrict the ACM set to repositories that are outputs of the citing paper or systematically code and control for the relationship type (output, tool, object of study, fork, background) across all three datasets.","section":"Relationships and categories / Figure 3"},{"comment":"The claim of 'significant differences' in the Key Insights is not supported by any statistical testing: no significance tests, confidence intervals, or effect sizes are reported, and there are no controls for publication year, repository age, or activity level. In addition, the lifespan defined as the interval between first and last commit is right-censored for repositories that are still live, which can bias the medians when comparing sets with different fractions of active repositories (e.g., 75% live for arXiv versus lower fractions elsewhere). The authors should provide inferential statistics and a censoring-aware (e.g., survival) analysis, and they should release the data and analysis scripts, especially since the paper advocates FAIR and open practices that would demand such transparency.","section":"Sustainability of research software"},{"comment":"The paper's qualitative interpretation that computer science emphasizes reuse while computational science emphasizes reproducibility is not directly measured. The taxonomy of relationships is derived from manual inspection of only the most active ACM repositories, and no corresponding coding is reported for the DOI or arXiv sets; the sentences 'Our hypothesis is...' are presented as explanations without testing. Either the manuscript should be reframed as proposing hypotheses for future work, or the relationship categories should be coded systematically across the full dataset to test whether the field difference in lifespans is associated with the repository role in the publication.","section":"Sustainability of research software / Relationships and categories"}],"minor_comments":[{"comment":"There are typos in this section: 'An first interesting observation' should be 'A first interesting observation', and 'Remind that we only collect...' should be 'Note that we only collect...'.","section":"Covered research areas"},{"comment":"The y-axis label 'Respoitories' in Figure 3a is misspelled; it should be 'Repositories'. Also, 'Github' should be 'GitHub' in the figure captions for consistency.","section":"Figure 3"},{"comment":"The abstract expands FAIR as 'findable, accessible, interoperable and repeatable', but the R in FAIR stands for 'Reusable'; the paper itself uses 'Reusable' in the main text. This inconsistency should be corrected.","section":"Abstract"},{"comment":"The sentence 'A third of these repositories are live for less than 1 day' is confusing because 'live' is defined by any activity during the last year; the intended meaning is likely that a third of the repositories have a lifespan of less than one day. Please rephrase to avoid ambiguity.","section":"Sustainability of research software"}],"recommendation":"major_revision","confidential_remarks":"The paper sits between a position piece and an empirical study. If the journal views the empirical comparison as a key contribution, the revision must address the confound between sample construction and field, the lack of inferential statistics, and the unavailability of data/scripts. If the paper is intended as a vision/position paper with an illustrative pilot, the authors should soften the 'significant differences' claim and clearly label the empirical results as exploratory. I would not reject outright because the recommendations and the observatory concept are of value, but the current form is not acceptable without substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a position paper with a small empirical study, not a breakthrough. The new thing is a cross-domain comparison of research software repository lifespans: ACM-linked GitHub repos (median ~5 years), DOI-linked GitHub repos (median ~15 days), and arXiv-linked repos (median ~8 months). That comparison is genuinely new and could be useful for science policy discussions. The paper also gives sensible recommendations: GitHub for maintenance, Zenodo for archival, artifact evaluation, software citation, and research software observatories. The authors are transparent about their sampling assumptions and include a Threats to Validity section. That is real credit. The soft spot is exactly what the stress-test note says. The ACM set consists of repositories cited by papers, and the examples include mature third-party tools like Caffe, Apache Spark, three.js, and Google Guava. The DOI set consists of repositories that cite papers, meaning software that self-identifies as accompanying a publication. Those are different constructs. The observed lifespan gap likely reflects selection criteria and link direction as much as any cultural difference between computer science and computational science. The authors do present their interpretations as hypotheses, but the abstract and Key Insights state the field difference as an observed fact. That overreach is the main load-bearing flaw. Two smaller issues: no data or scripts are released, which undercuts the claim that the analysis is repeatable; and no statistical tests accompany the descriptive statistics. Both are minor in an initial study but should be addressed if the empirical claim is to be taken as solid. Overall, this paper deserves a serious referee. The recommendations are sound and the topic is important. The referee should require either controlling for the repository-paper relationship type or softening the field-difference claim. I would bring it to a reading group only if the discussion focuses on methodology, not on the headline gap as established fact. I would not cite it in my own work within a year, but I would encourage engagement with the ideas.","headline":"A useful and honest position paper whose headline empirical gap is real but likely inflated by comparing software-as-output with mature third-party tools in the ACM sample.","tokens_in":691,"tokens_out":2375,"would_cite":false,"duration_ms":35654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that research software in computer science is maintained for years, while software tied to computational science papers is typically abandoned within days, reflecting different publishing cultures.","keywords":["research software","FAIR principles","open science","reproducibility","reusability","GitHub","artifact evaluation","software citation"],"falsifier":"Re-run the analysis with the same definitions but include Bitbucket, GitLab, and links through software manuals and papers, and also track whether DOI-linked repositories continue to receive commits on other hosts after their GitHub snapshot; if the median lifespan gap shrinks to months or disappears, the paper's central difference is not about sustainability but about where software is hosted.","tokens_in":11102,"feed_emoji":"💻","tokens_out":5703,"duration_ms":51611,"temperature":0.7,"pith_summary":"This paper argues that 'research software' is not one thing: computer science and computational science publish and sustain software in fundamentally different ways. Analyzing over 5,000 GitHub repositories linked to publications, it finds that software cited by ACM computer science papers has a median lifespan of about five years, software linked to DOIs in computational science papers lives a median of about fifteen days, and arXiv-linked software sits in between at about eight months. The paper reads this as computational science treating software as a reproducibility artifact, released with the paper and then abandoned, while computer science treats software as reusable infrastructure maintained for years. From this it recommends pairing GitHub for maintenance and reuse with Zenodo for archival snapshots, plus artifact evaluation and software citation metadata, to make research software both FAIR and open. A sympathetic reader should care because the numbers expose a sustainability gap hidden by the single label 'research software'.","feed_headline":"CS code lives 5 years; computational science code, 15 days","feed_subtitle":"A 5,000-repository study links the gap to reproducibility-first vs reuse-first publishing cultures.","key_machinery":"The mechanism is a three-way corpus linking publications to software. Research software is identified either by a publication containing a GitHub link or by a GitHub repository containing a DOI. Three sets are formed: 1,204 GitHub repositories containing a DOI, mostly from computational science; 1,091 GitHub repositories cited in ACM digital library papers, mostly from computer science; and 2,872 repositories cited by arXiv e-prints, mostly from AI-related computer science. Lifespan is defined as the time between first and last commit, and a repository is 'live' if it had any activity in the past year. These choices produce the headline medians and the characterization of different publishing cultures.","core_discovery":"The paper's central empirical claim is that research software publishing practices differ sharply across disciplines: computational science emphasizes reproducibility, releasing software to accompany a paper and then leaving it unmaintained, while computer science emphasizes reuse, keeping software alive as infrastructure. The supporting observation is the lifespan distribution of GitHub repositories connected to publications: median 5 years for ACM-linked computer science software, median 15 days for DOI-linked computational science software, and median 8 months for arXiv-linked software, with 75% of arXiv repositories active. The paper also proposes that the arXiv/AI community's 'publish early' attitude correlates with developing software openly from the start of a project. Based on this, it argues that research software should be open source, archived for reproducibility and actively maintained for reuse, with artifact evaluation and software citation standards in place.","pith_inferences":["As an inference beyond the paper: the 15-day median may overstate abandonment in computational science, because researchers there often keep code in lab servers or institutional repositories and publish only a GitHub snapshot at paper time; a study that tracks private maintenance would separate 'not shared' from 'not maintained'.","The paper's longevity hypothesis for computer science—that long-lived repositories are often commercial open-source frameworks rather than researcher-built code—implies that the 5-year median partly measures industry infrastructure, so the gap may be about who funds maintenance, not which field values reuse.","A testable extension: if the arXiv/AI 'publish early' culture drives open development, commit activity for arXiv-linked repositories should cluster before or at paper submission rather than after acceptance; the paper does not run this check.","The observatory proposal implies a citation-graph standard; one could extend the analysis to count software mentions in full text rather than URLs, which would capture papers and manuals as citation routes and likely shrink the apparent gap."],"forward_implications":["If the median-lifespan gap is real, computational science software is currently a reproducibility artifact rather than a reusable asset, so the FAIR goal of reuse requires new publishing incentives there.","A practical workflow that follows from the analysis is to use GitHub for living development and Zenodo for archived, citable snapshots, so the same software serves reuse and reproducibility.","Artifact evaluation, already correlated with higher citation counts in the cited work, becomes a concrete lever for improving reusability if adopted more widely in computer science subdisciplines.","Software citation metadata standards such as CFF and CodeMeta only help if widely adopted; the paper implies that search engines exploiting them are the missing piece.","Research software observatories that catalogue software-to-publication links would make analyses like this repeatable and extendable to GitLab, Bitbucket, and non-GitHub citations."],"supporting_citations":[{"why":"Supplies the precedent of a large-scale GitHub code-activity analysis in bioinformatics, which the paper extends to computer science versus computational science.","marker":"[35]"},{"why":"Provides the measured correlation between artifact evaluation and higher citation counts, supporting the paper's recommendation for artifact evaluation.","marker":"[7]"},{"why":"Documents the ACM artifact evaluation process, which the paper proposes as a mechanism for improving reusability.","marker":"[28]"},{"why":"Establishes that software engineering practices are rarely adopted in computational science, a premise for the observed publishing differences.","marker":"[24]"},{"why":"Shows that the computational science community now sees software engineering as central to research productivity, motivating the paper's recommendations.","marker":"[34]"},{"why":"Defines the FAIR data principles that the paper extends to research software.","marker":"[40]"},{"why":"Provides the software citation principles that underpin the paper's recommendations for findability and credit.","marker":"[36]"},{"why":"Supplies existing metadata standards and citation practices for software, which the paper argues need wider adoption.","marker":"[26]"}],"fun_headline_variants":["CS code: 5-year lifespan; computational science: 15 days","Why computational science code is dead in 15 days","Reuse extends code life: CS vs computational science gap","Research software split: 15 days vs 5 years by discipline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on identifying research software through a GitHub link in a publication or a DOI in a GitHub repository; if computational science software is systematically less likely to be on GitHub or cited through those routes, the five-year versus fifteen-day gap could be an artifact of citation and hosting habits rather than a real difference in maintenance.","fun_headline_variants_meta":{"raw":{"variants":["CS code: 5-year lifespan; computational science: 15 days","Why computational science code is dead in 15 days","Reuse extends code life: CS vs computational science gap","Research software split: 15 days vs 5 years by discipline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2145,"prompt_tokens":949,"completion_tokens":1196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1137}},"tokens_in":565,"tokens_out":1196,"duration_ms":12191,"temperature":1.0,"reasoning_tokens":1137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:58:07.670819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the analysis with the same definitions but include Bitbucket, GitLab, and links through software manuals and papers, and also track whether DOI-linked repositories continue to receive commits on other hosts after their GitHub snapshot; if the median lifespan gap shrinks to months or disappears, the paper's central difference is not about sustainability but about where software is hosted.","supporting_citations":[{"cited_title":"Russell, Rachel L","cited_arxiv_id":null,"evidence_quote":"Supplies the precedent of a large-scale GitHub code-activity analysis in bioinformatics, which the paper extends to computer science versus computational science."},{"cited_title":"Childers and Panos K","cited_arxiv_id":null,"evidence_quote":"Provides the measured correlation between artifact evaluation and higher citation counts, supporting the paper's recommendation for artifact evaluation."},{"cited_title":"Software engineering for computational sci- ence: Past, present, future","cited_arxiv_id":null,"evidence_quote":"Establishes that software engineering practices are rarely adopted in computational science, a premise for the observed publishing differences."},{"cited_title":"Katz and Neil P","cited_arxiv_id":null,"evidence_quote":"Supplies existing metadata standards and citation practices for software, which the paper argues need wider adoption."}],"review_version":1}