{"id":"bf2f462c-8085-4ae5-8a4b-c5a71d2df947","arxiv_id":"2507.16166","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper repackages established software engineering best practices as ten high-level guidelines for research software, with added notes for legacy codebases.","lead":"This preprint lists ten common-sense software engineering practices, from planning to maintenance, aimed at researchers who write code for science. It is a practical guide for scientists and research software engineers who want their tools to be reliable and reusable.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'ten essential guidelines' claim is universal and prescriptive, but the selection of the ten, their transferability across research-software contexts, and the outcome 'impactful' are untested; the paper's own caveats implicitly qualify the claim.","rationale":"Read in good faith, this is an expository guidelines paper in a well-established genre; its advice is largely consistent with prior best-practice lists and with the authors' collective RSE experience. I therefore do not regard the lack of a controlled experiment as a fatal flaw, and I agree with the reader's UNVERDICTED verdict. The load-bearing weakness is that the strongest phrasing ('ten essential guidelines', 'researchers can create... impactful software') is a universal prescriptive claim, while the evidence is anecdotal and the selection of the ten is unexplained. My concern is not that the guidelines are wrong; it is that the paper's central claim is unverified in a way that its own caveats implicitly acknowledge. A structured retrospective evaluation would either support the selection and transferability or reveal that the guidelines should be reframed as context-dependent heuristics rather than essential requirements. This is essentially the same weakness the reader identified, hence agreement.","tokens_in":10185,"tokens_out":7199,"duration_ms":77272,"concrete_test":"Run a retrospective study on a stratified sample of 50-100 research software projects (single-investigator scripts, group-developed HPC codes, ML pipelines). For each project, compute an adherence score for the ten guidelines via repository inspection and maintainer interviews, and measure maintainability (release cadence, issue-resolution time, bus factor) and impact (downloads, citations, active user community). Regress these outcomes on the adherence score, controlling for team size and discipline. If adherence shows no positive association, or if practices omitted from the list (packaging, usability testing, security hardening) predict outcomes more strongly, then the claim that these ten are the 'essential' guidelines is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To make the central claim true - 'by following the ten guidelines... researchers can create software that is maintainable, extensible, and impactful' - two things must hold: the chosen ten are the right/essential set, and the practices transfer across the heterogeneous research-software landscape. Neither is demonstrated. The selection is asserted: the paper cites two prior experience-based lists ([10], [11]) and says it adapts principles, but no systematic review, expert-elicitation method, or empirical criterion distinguishes 'essential' from merely helpful practices. Transferability is assumed rather than shown; research software ranges from single-investigator analysis scripts to multi-institution HPC codes, and the same ten practices are unlikely to be equally essential in every context. The paper itself signals this in Guideline 5 ('right-size your practices'), Guideline 9 (titled 'as Necessary'), and the legacy-project variants, which qualify or exempt projects from parts of the list. Finally, 'impactful' is never operationally defined, so the advertised outcome is not measurable even in principle. These are not internal contradictions - the guidance is sensible - but the universal, prescriptive framing is substantially stronger than the evidence supplied.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ten guidelines for developing high-quality research software: planning, modular design, clean code, version control, regular testing, peer code review, documentation, reproducibility, performance/scalability, and long-term maintenance. Each guideline is presented with concrete recommendations, tool suggestions, and a paragraph on how it can be adapted for legacy projects. The authors position the work as a practical resource for researchers and research software engineers, drawing on established software-engineering practices and two prior best-practice papers ([10], [11]). The central claim, stated in the Introduction and Conclusion, is that following these ten guidelines enables researchers to create software that is functional, maintainable, extensible, and impactful.","tokens_in":10449,"tokens_out":3630,"duration_ms":40688,"significance":"If taken as experience-based practitioner guidance, the paper is a useful and sensible synthesis. Its strengths are that it is grounded in established software-engineering literature, includes concrete tool recommendations, supplies legacy-project adaptations for every guideline, and contains useful caveats such as 'right-size your practices' in Guideline 5 and 'as Necessary' in Guideline 9. However, the paper provides no empirical evaluation, no systematic literature review, and no operational definition of the advertised outcome 'impactful.' The significance is therefore conditional: the guidelines are plausible and likely helpful, but the universal, prescriptive framing of the title and abstract is not supported by the evidence presented. The paper can serve as an accessible introduction for researchers new to software quality, but not as a validated standard or a measured claim about outcomes.","major_comments":[{"comment":"The central claim, 'By following the ten guidelines outlined here, researchers can create software that is not only functional but also maintainable, extensible, and impactful,' is load-bearing but unsupported. 'Impactful' is never operationally defined, and no evidence is offered that adoption of these guidelines produces maintainable, extensible, or impactful software. Since this sentence is the advertised contribution, please either soften the claim (e.g., 'these guidelines reflect widely used best practices that are likely to improve quality') or provide case-study, survey, or other empirical evidence supporting the causal claim. A short limitations paragraph stating that the guidelines are experience-based rather than empirically validated would address this concern.","section":"Introduction; Conclusion"},{"comment":"The selection of exactly these ten guidelines, and the word 'essential,' is asserted rather than justified. The paper cites two prior best-practice lists ([10], [11]) and says it adapts principles, but no method—systematic literature review, expert elicitation, or evaluative criterion—distinguishes 'essential' practices from merely helpful ones. Moreover, the paper's own caveats (Guideline 5's 'right-size your practices,' Guideline 9's 'as Necessary,' and the legacy-project variants) indicate that the applicability of each guideline is context-dependent. Please justify the selection and the term 'essential,' or reframe the contribution as 'ten widely recommended practices' rather than a necessary-and-sufficient set.","section":"Ten Essential Guidelines (before Guideline 1)"},{"comment":"There is a tension between the universal framing of the abstract ('ten guidelines for producing high-quality research software') and the paper's repeated qualifications that practices should be right-sized to project maturity and that performance work should be done 'as Necessary.' If practices are context-dependent, then the claim that following the ten guidelines will produce high-quality software cannot hold without specifying scope conditions. Please resolve this tension by stating the intended scope (e.g., newly formed, medium-to-large research projects) and by indicating which guidelines are conditional and under what conditions they are essential.","section":"Abstract; Guideline 5; Guideline 9"}],"minor_comments":[{"comment":"There are missing spaces in 'canactasfacilitators,translatinghigh-levelvaluesintoactionablestrategies'; please correct the typographical errors.","section":"Introduction (RSE paragraph)"},{"comment":"The sentence 'This “Boy Scout Rule” approachalways leaves the code better than you found it helps modernize the codebase' is missing a dash or quotation mark and a word boundary; it should read 'This “Boy Scout Rule” approach—always leaves the code better than you found it—helps modernize the codebase.'","section":"Guideline 6, legacy paragraph"},{"comment":"There is an extra space before the comma in 'Pay Attention to Performance and Scalability , as Necessary'; please remove it.","section":"Guideline 9 title"},{"comment":"The paper promises a distinction from [10] and [11] ('core values that can be translated as high-level practices, as opposed to finer delineated lists of individual actions'), but this distinction is not explicitly developed anywhere in the body; a short paragraph stating the specific added value relative to those two papers would help readers understand the novel contribution.","section":"Introduction"},{"comment":"The sentence 'While legacy code often lacks an adequate test suite, it is rare for it not to have at least some examples' is grammatically awkward; consider 'Even legacy code that lacks an adequate test suite usually has at least some example inputs or outputs.'","section":"Guideline 5"},{"comment":"The reproducibility discussion refers to 'concurrency' and nondeterministic order of operations, which is a useful and often-overlooked point, but the paragraph would benefit from a concrete example (e.g., floating-point reduction order in MPI) to make the concern actionable for readers without HPC experience.","section":"Guideline 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is sensible and likely useful for a practitioner-oriented venue such as Computing in Science and Engineering, but for a research-oriented journal the lack of any empirical or systematic evidence behind the 'essential' claim is a scope concern. The authors are credible and the content is not problematic in itself; a major revision focused on reframing the claims, defining scope, and explicitly acknowledging the evidence base would make the paper honest and publishable. I do not see grounds for rejection, but the current universal prescriptive framing is stronger than the evidence supplied."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2507.16166. It's a solid practitioner paper, but don't expect new research. The ten guidelines are standard fare—Wilson et al. 2014 and 2017 plus common software engineering advice—repackaged at a higher level of abstraction. What is actually new and useful: the legacy-project paragraphs under each guideline. The advice there is concrete and experience-based, e.g., using examples as integration tests, the Boy Scout Rule, and the git-svn migration notes. If I were an RSE inheriting a messy codebase, I'd find these genuinely helpful.\n\nThe paper does well in tone and organization. It's clearly written, with sensible caveats like 'right-size your practices' and Guideline 9's 'as Necessary' caveat. The references to prior work are appropriate, and the self-citations are background only.\n\nThe soft spots are the title and framing. Calling these ten 'essential' is an overclaim. There's no systematic method for selecting these ten over others; the paper simply adapts two prior lists and adds maintenance and performance. 'Impactful' is never defined, so the advertised outcome is not measurable. And the universal prescriptive framing sits uneasily with the paper's own allowances for right-sizing and cutting practices. These aren't internal contradictions—the guidance is sound—but the abstract and conclusion promise more than the paper supports.\n\nFor a venue like CiSE, I'd send it to peer review. It deserves a serious referee, mainly to pressure the authors into tempering the claims and perhaps positioning the paper as an experience-based synthesis rather than an evidence-based standard. For a research-oriented venue, it would be a desk reject, but that's not the right audience.\n\nI wouldn't cite it in my own work beyond maybe a nod to the legacy-project advice. It's not a reading-group paper, but if someone is writing guidelines for RSEs, it's worth a skim.\n\nHonest verdict: take it as a well-written best-practices summary with good legacy-project supplements. Not more.","headline":"A well-organized best-practices summary with genuinely useful legacy-project advice; the 'ten essential' framing overpromises, but the practical content is sound.","tokens_in":10873,"tokens_out":2246,"would_cite":false,"duration_ms":22961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that ten practical guidelines, applied across the software lifecycle, can make research software maintainable, extensible, and reproducible.","keywords":["research software","software quality","best practices","software engineering","reproducibility","maintainability","scientific computing","peer code review"],"falsifier":"A multi-year controlled comparison in which matched research teams either adopt all ten guidelines or follow their usual practices, measuring defect rates, onboarding time, external reproducibility of results, and software survival, would settle the claim: if the adopting teams do not clearly outperform on these measures, the paper's central claim is falsified.","tokens_in":10059,"feed_emoji":"✅","tokens_out":6798,"duration_ms":61065,"temperature":0.7,"pith_summary":"The paper argues that researchers can raise the quality of their software by following ten prescribed practices that span the whole development lifecycle: planning, modular design, clean code, version control, regular testing, peer code review, documentation, reproducibility, performance awareness, and long-term maintenance. The authors present these as high-level core values rather than a fine-grained checklist, and each guideline includes advice for applying it to existing legacy code. The intended payoff is software that is not only functional for a single study but also maintainable, extensible, and impactful for the broader research ecosystem.","feed_headline":"Ten guidelines claim to make research software maintainable","feed_subtitle":"A practical checklist covering planning, testing, documentation, and long-term maintenance for scientists who code.","key_machinery":"The central object is the set of ten guidelines itself, each stated as a high-level practice with a concrete example and a note on how to adapt it to legacy projects. The guidelines are mutually reinforcing: version control enables peer review and continuous integration; testing protects refactoring; documentation supports reproducibility; modularity eases maintenance. The paper treats this interdependence as the mechanism by which adopting all ten, rather than any single practice, produces high-quality software.","core_discovery":"In the paper's own terms, high-quality research software is produced by adhering to ten essential guidelines, and these guidelines together cover every stage of the development lifecycle. The paper claims that by following them, researchers can overcome common failures such as lack of usability, poor documentation, difficulty in reproduction, and inadequate long-term support. The intended outcome is software that advances the researcher's scientific objectives and contributes to a broader ecosystem of reliable, reusable research tools.","pith_inferences":["The ten guidelines could be operationalized into a maturity rubric or self-assessment, scoring a project's adoption of each practice; such a rubric would let the central claim be tested empirically against project outcomes like defect density, onboarding time, and replication success.","Different research fields likely need different weightings: an experimental physics code may prioritize performance and reproducibility, while a data-analysis pipeline may prioritize documentation and environment pinning; a one-size-fits-all 'essential' set is a plausible baseline, not a final answer.","If the guidelines are right, funding agencies and journals that require code availability could reasonably require evidence of practices like testing, version control, and environment specification, shifting incentives for research software quality.","The guidelines' advice to survey existing tools and reuse well-tested components implicitly predicts that teams who do so will produce higher-quality software than teams who build from scratch; this is a testable claim the paper itself does not test."],"forward_implications":["If the claim holds, research teams can treat the ten guidelines as a shared checklist for new projects and as a diagnostic for existing ones.","Adopting the guidelines should reduce the common failure modes the paper lists: unclear usability, missing documentation, irreproducible results, and unsupported code.","The legacy-project adaptations imply that even long-standing research code can be improved incrementally without a rewrite, using the 'Boy Scout Rule' of leaving code better than found.","The emphasis on reproducibility ties software quality directly to scientific credibility, giving research software engineers a concrete mandate within research teams.","The paper's framing of guidelines as high-level core values, rather than a fine-grained action list, makes the set more adaptable but also harder to audit mechanically."],"supporting_citations":[{"why":"Prior best-practice list for scientific computing that this paper reworks into high-level core values.","marker":"[10]"},{"why":"Companion prior list of 'good enough practices' that this paper positions itself against and extends.","marker":"[11]"},{"why":"Defines clean-code principles that Guideline 3 draws on for readability and maintainability.","marker":"[5]"},{"why":"Supplies the tiered approach to right-sizing software quality practices that Guideline 5 references.","marker":"[6]"},{"why":"Establishes reproducibility as a scientific requirement that Guideline 8 builds upon.","marker":"[7]"},{"why":"Argues that irreproducible computational research undermines trust, motivating the reproducibility guideline.","marker":"[8]"},{"why":"Provides survey evidence of how scientists actually develop software, motivating the need for guidelines.","marker":"[2]"}],"fun_headline_variants":["Ten guidelines for research software that lasts","A practical checklist for robust research software","Ten steps to reproducible research software","Build better research code with these ten rules","Your research software deserves these ten guidelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that software-engineering practices proven in commercial settings and earlier guideline lists transfer to the highly varied environments of research software, and that the ten selected here are the essential ones; if that premise fails, the guidelines are reasonable opinions rather than evidence-based standards.","fun_headline_variants_meta":{"raw":{"variants":["Ten guidelines for research software that lasts","A practical checklist for robust research software","Ten steps to reproducible research software","Build better research code with these ten rules","Your research software deserves these ten guidelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1176,"prompt_tokens":770,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":386,"tokens_out":406,"duration_ms":4793,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:15:36.671793+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A multi-year controlled comparison in which matched research teams either adopt all ten guidelines or follow their usual practices, measuring defect rates, onboarding time, external reproducibility of results, and software survival, would settle the claim: if the adopting teams do not clearly outperform on these measures, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Jablin, Arun Raman, Yun Zhang, Jialu Huang, Hanjun Kim, Nick P","cited_arxiv_id":null,"evidence_quote":"Prior best-practice list for scientific computing that this paper reworks into high-level core values."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Companion prior list of 'good enough practices' that this paper positions itself against and extends."},{"cited_title":"The emperor's old clothes","cited_arxiv_id":null,"evidence_quote":"Defines clean-code principles that Guideline 3 draws on for readability and maintainability."},{"cited_title":"Clean code: a handbook of agile software craftsmanship","cited_arxiv_id":null,"evidence_quote":"Supplies the tiered approach to right-sizing software quality practices that Guideline 5 references."},{"cited_title":"A tiered approach to scientific software quality practices","cited_arxiv_id":null,"evidence_quote":"Establishes reproducibility as a scientific requirement that Guideline 8 builds upon."},{"cited_title":"Reproducibility and Replicability in Science","cited_arxiv_id":null,"evidence_quote":"Argues that irreproducible computational research undermines trust, motivating the reproducibility guideline."},{"cited_title":"Eisty and Jeffrey C","cited_arxiv_id":null,"evidence_quote":"Provides survey evidence of how scientists actually develop software, motivating the need for guidelines."}],"review_version":1}