REVIEW 3 major objections 6 minor 14 references
Ten Essential Guidelines for Building High-Quality Research Software
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper argues that ten practical guidelines, applied across the software lifecycle, can make research software maintainable, extensible, and reproducible.
desk verdict A well-organized best-practices summary with genuinely useful legacy-project advice; the 'ten essential' framing overpromises, but the practical content is sound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the set of ten guidelines itself, each stated as a high-level practice with a concrete example and a note on how to adapt it to legacy projects. The guidelines are mutually reinforcing: version control enables peer review and continuous integration; testing protects refactoring; documentation supports reproducibility; modularity eases maintenance. The paper treats this interdependence as the mechanism by which adopting all ten, rather than any single practice, produces high-quality software.
What would settle it
A multi-year controlled comparison in which matched research teams either adopt all ten guidelines or follow their usual practices, measuring defect rates, onboarding time, external reproducibility of results, and software survival, would settle the claim: if the adopting teams do not clearly outperform on these measures, the paper's central claim is falsified.
Extended reading notes
Core claim
In the paper's own terms, high-quality research software is produced by adhering to ten essential guidelines, and these guidelines together cover every stage of the development lifecycle. The paper claims that by following them, researchers can overcome common failures such as lack of usability, poor documentation, difficulty in reproduction, and inadequate long-term support. The intended outcome is software that advances the researcher's scientific objectives and contributes to a broader ecosystem of reliable, reusable research tools.
Load-bearing premise
The load-bearing premise is that software-engineering practices proven in commercial settings and earlier guideline lists transfer to the highly varied environments of research software, and that the ten selected here are the essential ones; if that premise fails, the guidelines are reasonable opinions rather than evidence-based standards.
Editorial extensions
If this is right
- If the claim holds, research teams can treat the ten guidelines as a shared checklist for new projects and as a diagnostic for existing ones.
- Adopting the guidelines should reduce the common failure modes the paper lists: unclear usability, missing documentation, irreproducible results, and unsupported code.
- The legacy-project adaptations imply that even long-standing research code can be improved incrementally without a rewrite, using the 'Boy Scout Rule' of leaving code better than found.
- The emphasis on reproducibility ties software quality directly to scientific credibility, giving research software engineers a concrete mandate within research teams.
- The paper's framing of guidelines as high-level core values, rather than a fine-grained action list, makes the set more adaptable but also harder to audit mechanically.
Reading between the lines
- The ten guidelines could be operationalized into a maturity rubric or self-assessment, scoring a project's adoption of each practice; such a rubric would let the central claim be tested empirically against project outcomes like defect density, onboarding time, and replication success.
- Different research fields likely need different weightings: an experimental physics code may prioritize performance and reproducibility, while a data-analysis pipeline may prioritize documentation and environment pinning; a one-size-fits-all 'essential' set is a plausible baseline, not a final answer.
- If the guidelines are right, funding agencies and journals that require code availability could reasonably require evidence of practices like testing, version control, and environment specification, shifting incentives for research software quality.
- The guidelines' advice to survey existing tools and reuse well-tested components implicitly predicts that teams who do so will produce higher-quality software than teams who build from scratch; this is a testable claim the paper itself does not test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ten guidelines for developing high-quality research software: planning, modular design, clean code, version control, regular testing, peer code review, documentation, reproducibility, performance/scalability, and long-term maintenance. Each guideline is presented with concrete recommendations, tool suggestions, and a paragraph on how it can be adapted for legacy projects. The authors position the work as a practical resource for researchers and research software engineers, drawing on established software-engineering practices and two prior best-practice papers ([10], [11]). The central claim, stated in the Introduction and Conclusion, is that following these ten guidelines enables researchers to create software that is functional, maintainable, extensible, and impactful.
Significance. If taken as experience-based practitioner guidance, the paper is a useful and sensible synthesis. Its strengths are that it is grounded in established software-engineering literature, includes concrete tool recommendations, supplies legacy-project adaptations for every guideline, and contains useful caveats such as 'right-size your practices' in Guideline 5 and 'as Necessary' in Guideline 9. However, the paper provides no empirical evaluation, no systematic literature review, and no operational definition of the advertised outcome 'impactful.' The significance is therefore conditional: the guidelines are plausible and likely helpful, but the universal, prescriptive framing of the title and abstract is not supported by the evidence presented. The paper can serve as an accessible introduction for researchers new to software quality, but not as a validated standard or a measured claim about outcomes.
major comments (3)
- [Introduction; Conclusion] The central claim, 'By following the ten guidelines outlined here, researchers can create software that is not only functional but also maintainable, extensible, and impactful,' is load-bearing but unsupported. 'Impactful' is never operationally defined, and no evidence is offered that adoption of these guidelines produces maintainable, extensible, or impactful software. Since this sentence is the advertised contribution, please either soften the claim (e.g., 'these guidelines reflect widely used best practices that are likely to improve quality') or provide case-study, survey, or other empirical evidence supporting the causal claim. A short limitations paragraph stating that the guidelines are experience-based rather than empirically validated would address this concern.
- [Ten Essential Guidelines (before Guideline 1)] The selection of exactly these ten guidelines, and the word 'essential,' is asserted rather than justified. The paper cites two prior best-practice lists ([10], [11]) and says it adapts principles, but no method—systematic literature review, expert elicitation, or evaluative criterion—distinguishes 'essential' practices from merely helpful ones. Moreover, the paper's own caveats (Guideline 5's 'right-size your practices,' Guideline 9's 'as Necessary,' and the legacy-project variants) indicate that the applicability of each guideline is context-dependent. Please justify the selection and the term 'essential,' or reframe the contribution as 'ten widely recommended practices' rather than a necessary-and-sufficient set.
- [Abstract; Guideline 5; Guideline 9] There is a tension between the universal framing of the abstract ('ten guidelines for producing high-quality research software') and the paper's repeated qualifications that practices should be right-sized to project maturity and that performance work should be done 'as Necessary.' If practices are context-dependent, then the claim that following the ten guidelines will produce high-quality software cannot hold without specifying scope conditions. Please resolve this tension by stating the intended scope (e.g., newly formed, medium-to-large research projects) and by indicating which guidelines are conditional and under what conditions they are essential.
minor comments (6)
- [Introduction (RSE paragraph)] There are missing spaces in 'canactasfacilitators,translatinghigh-levelvaluesintoactionablestrategies'; please correct the typographical errors.
- [Guideline 6, legacy paragraph] The sentence 'This “Boy Scout Rule” approachalways leaves the code better than you found it helps modernize the codebase' is missing a dash or quotation mark and a word boundary; it should read 'This “Boy Scout Rule” approach—always leaves the code better than you found it—helps modernize the codebase.'
- [Guideline 9 title] There is an extra space before the comma in 'Pay Attention to Performance and Scalability , as Necessary'; please remove it.
- [Introduction] The paper promises a distinction from [10] and [11] ('core values that can be translated as high-level practices, as opposed to finer delineated lists of individual actions'), but this distinction is not explicitly developed anywhere in the body; a short paragraph stating the specific added value relative to those two papers would help readers understand the novel contribution.
- [Guideline 5] The sentence 'While legacy code often lacks an adequate test suite, it is rare for it not to have at least some examples' is grammatically awkward; consider 'Even legacy code that lacks an adequate test suite usually has at least some example inputs or outputs.'
- [Guideline 8] The reproducibility discussion refers to 'concurrency' and nondeterministic order of operations, which is a useful and often-overlooked point, but the paragraph would benefit from a concrete example (e.g., floating-point reduction order in MPI) to make the concern actionable for readers without HPC experience.
Circularity Check
No circularity: the paper makes no quantitative derivation, and its recommendations are transparently adapted from external best-practice literature.
full rationale
This is a prescriptive guidelines paper, not a derivation or prediction exercise. There are no equations, no fitted parameters, no quantitative claims that could reduce by construction, and no target result that is defined in terms of its own inputs. The central assertion—'By following the ten guidelines outlined here, researchers can create software that is not only functional but also maintainable, extensible, and impactful'—is a practical recommendation, not a derived theorem. The selection and framing of the ten guidelines are explicitly anchored to external prior work: the paper says, 'While there have been previous papers of this kind ([10], [11]), this paper is intended to present core values that can be translated as high-level practices.' The self-citations that appear ([1] by Eisty and Carver, [6] by Mundt et al.) are contextual or ancillary: [1] supports the general observation that software is integral to scientific disciplines, and [6] is referenced once in Guideline 5 for the idea of right-sizing testing practices to project maturity. Neither citation is load-bearing for the paper's central claim, which stands as an experience-based synthesis rather than a result derived from those citations. Concerns that the 'essential' set is untested or that the guidelines may not transfer across all research-software contexts are about evidential strength and framing, not circularity. Therefore, the paper is self-contained in the sense relevant to circularity analysis, and the appropriate score is 0.
Assumptions & free parameters
assumptions (2)
- domain assumption General software engineering best practices (agile planning, code review, testing, etc.) transfer to research software development and lead to higher quality
- ad hoc to paper The ten listed practices are 'essential', i.e., the set is both necessary and sufficient for high-quality research software
Cite this review
Pith. "Pith review of Ten Essential Guidelines for Building High-Quality Research Software." pith.science (2026). https://pith.science/paper/2RQ5VUOP
@misc{pith2026250716166,
author = {Pith},
title = {Pith review of: Ten Essential Guidelines for Building High-Quality Research Software},
year = {2026},
howpublished = {\url{https://pith.science/paper/2RQ5VUOP}},
note = {Machine review of arXiv:2507.16166}
}
read the original abstract
High-quality research software is a cornerstone of modern scientific progress, enabling researchers to analyze complex data, simulate phenomena, and share reproducible results. However, creating such software requires adherence to best practices that ensure robustness, usability, and sustainability. This paper presents ten guidelines for producing high-quality research software, covering every stage of the development lifecycle. These guidelines emphasize the importance of planning, writing clean and readable code, using version control, and implementing thorough testing strategies. Additionally, they address key principles such as modular design, reproducibility, performance optimization, and long-term maintenance. The paper also highlights the role of documentation and community engagement in enhancing software usability and impact. By following these guidelines, researchers can create software that advances their scientific objectives and contributes to a broader ecosystem of reliable and reusable research tools. This work serves as a practical resource for researchers and developers aiming to elevate the quality and impact of their research software.
Reference graph
Works this paper leans on
-
[10]
Jablin, Arun Raman, Yun Zhang, Jialu Huang, Hanjun Kim, Nick P
Prakash Prabhu, Thomas B. Jablin, Arun Raman, Yun Zhang, Jialu Huang, Hanjun Kim, Nick P. Johnson, Feng Liu, Soumyadeep Ghosh, Stephen Beard, Taewook Oh, Matthew Zoufaly, David Walker, and David I. August. A survey of the practice of computational science. In State of the Practice Reports , SC '11, New York, NY, USA, 2011. Association for Computing Machinery
work page 2011
-
[11]
Greg Wilson, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White, and Paul Wilson. Best practices for scientific computing. PLOS Biology , 12(1):1--7, 01 2014
work page 2014
-
[1]
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 9 12 #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockconfadjsp...
-
[2]
Nasir U. Eisty and Jeffrey C. Carver. Testing research software: a survey. Empirical Software Engineering , 27(138):1--38, 2022
work page 2022
-
[3]
Jo Erskine Hannay, Carolyn MacLeod, Janice Singer, Hans Petter Langtangen, Dietmar Pfahl, and Greg Wilson. How do scientists develop and use scientific software? In 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering , pages 1--8, 2009
work page 2009
-
[4]
Dustin Heaton and Jeffrey C. Carver. Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology , 67:207--219, 2015
work page 2015
-
[5]
Charles Antony Richard Hoare. The emperor's old clothes. Commun. ACM , 24(2):75–83, February 1981
work page 1981
-
[6]
Clean code: a handbook of agile software craftsmanship
Robert C Martin. Clean code: a handbook of agile software craftsmanship . Pearson Education, 2009
work page 2009
Show all 14 references
-
[7]
A tiered approach to scientific software quality practices
Miranda Mundt, Wade Burgess, and Dena Vigil. A tiered approach to scientific software quality practices. In Proceedings of the 2022 Improving Scientific Software Conference , 2022
2022
-
[8]
Reproducibility and Replicability in Science
Committee on R eproducibility and R eplicability in S cience. Reproducibility and Replicability in Science . National Academies Press, 2019
2019
-
[9]
Reproducible research in computational science
Roger D Peng. Reproducible research in computational science. Science , 334(6060):1226--1227, 2011
2011
-
[12]
Greg Wilson, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, and Tracy K. Teal. Good enough practices in scientific computing. PLoS Comput Biol , 13(6):e1005510, 2017
2017
-
[13]
write newline
" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write ...
-
[14]
write newline
" write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTI...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.