REVIEW 1 major objections 5 minor 21 references
Ten simple rules for PIs to integrate Research Software Engineering into their research group
T0 review · 1 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Ten simple rules put research software engineering within a PI's reach.
desk verdict A well-written, PI-facing synthesis of known RSE practices; the advice is sound but the abstract overpromises a causal payoff the paper can't back. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the set of ten rules itself, framed as actionable guidance rather than a technical manual. The paper groups the work of RSEng into three activity categories—technical (code, architecture, testing), management (team, project, product), and knowledge transfer (documentation, training, community)—and each rule is a concrete practice a PI can mandate or resource. The devices that carry the argument are analogies between computational work and experimental science: version control and documentation are the computational scientist's lab notebook, architecture is the plan for a house, and untested software is uncalibrated lab equipment.
What would settle it
A controlled comparison of matched research groups—half adopting the ten rules, half continuing usual practice—measuring defect density, successful reproduction of results, or software citation rates over 12–24 months would settle the claim; if adopters show no improvement, the causal link fails. Alternatively, a survey-based study showing that groups reporting high rule adherence have no better reproducibility scores than low-adherence groups would falsify it.
Extended reading notes
Core claim
The central claim is that Research Software Engineering is much more than writing code—it is a process built from technical, management, and knowledge-transfer activities—and that the barriers preventing PIs from engaging with it can be removed by following ten simple rules. The discovery, in effect, is an accessible framing of RSEng for group leaders: start the engineering practices at the beginning of a project, translate research needs into software requirements, think about architecture before and during coding, choose tools in context, keep everything under version control, develop in the open, document for real audiences, apply automated quality assurance with continuous integration, and treat software as a citable scholarly object. If the paper is right, a group that follows these rules produces software that is maintainable, reusable, and trustworthy, and the research built on that software inherits those properties.
Load-bearing premise
The paper assumes that a group that follows these ten rules will end up with higher-quality, more reproducible, more trustworthy software—and ultimately better research—but it presents no empirical evidence comparing adopters with non-adopters.
Editorial extensions
If this is right
- Groups that adopt the rules will allocate time for requirements, architecture, version control, documentation, and testing from the first day of a project rather than as a cleanup phase.
- Research software developed this way becomes easier to reuse, extend, and maintain, reducing waste and enabling collaborations that would otherwise stall.
- Open-source development and published, citable software give the group scholarly credit and make results reproducible by third parties.
- Automated quality assurance and code review reduce the chance of undetected errors, protecting the group from embarrassing retractions.
- PIs can engage with RSEng at the level of practices and expectations, even when they do not personally write code.
Reading between the lines
- If the rules work as claimed, they could serve as a lightweight audit rubric: funders and institutions could check a project's plan for version control, testing, and documentation before release, turning an advice document into an assessment checklist.
- The causal claim invites a direct empirical test: compare matched research groups that follow the rules against those that do not on defect density, reproducibility success rates, or software citation counts after a fixed period.
- The paper stops short of specifying how much effort each rule costs; an implicit testable extension is measuring time-to-first-release or maintenance burden under partial versus full rule adoption.
- The 'lab notebook' framing implies that institutional research-integrity policies could recognise version control history and test logs as legitimate research records, a step that would give PIs a concrete incentive to adopt the practices.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript presents ten rules aimed at principal investigators for incorporating research software engineering (RSEng) practices into their research groups. The rules span early adoption of RSEng, requirements elicitation, software architecture, tool selection, version control, open-source development, documentation, quality assurance, and software publication and citation. The paper is a position piece in the 'Ten Simple Rules' genre; it contains no new empirical data and instead synthesizes existing literature and the authors' expertise. The abstract and introduction assert that following the rules will improve software quality, reproducibility, and trustworthiness, and ultimately lead to better research outcomes.
Significance. The paper addresses a real gap: most RSEng guidance is technical and aimed at developers, while PIs make the resource decisions. The rules are concrete, actionable, and largely consistent with established software engineering practice and prior 'Ten Simple Rules' literature. The authors correctly emphasize that RSEng involves management and knowledge transfer, not just coding. The principal weakness is that the paper's central promise—that adopting these rules causes better research outcomes—is asserted rather than demonstrated; the cited support is mostly editorial or position literature, and the surveys mentioned in the Acknowledgments are not provided. As a practical checklist, the paper is a useful contribution; as an empirical claim, it is unsubstantiated.
major comments (1)
- [Abstract and Introduction] The abstract ('By following these rules, readers can improve ... ultimately leading to better, more reproducible and more trustworthy research outcomes') and the introduction ('better research software leads to better research' [2]) make a causal claim that is not supported by the evidence cited. References [1] and [2] are a position paper and an editorial, respectively, and the surveys mentioned in the Acknowledgments are not included in the manuscript. Because this causal promise is the paper's primary value proposition to PIs, please either (a) supply or cite empirical evidence (e.g., the survey instruments and results, or a synthesis of quantitative studies linking RSEng practices to research outcomes), or (b) temper the claims to state that these are expert recommendations whose benefits are widely acknowledged but not yet rigorously measured.
minor comments (5)
- [Rule 3] The heading contains a typo: 'W e' should be 'We'.
- [Rule 7] The sentence 'Software isopen-source when its source code is publicly available' is missing a space; it should read 'Software is open-source'.
- [Rule 5] The phrase 'it also not to be disregarded lightly' is grammatically incomplete; it should read 'it also is not to be disregarded lightly' or 'it should not be disregarded lightly'.
- [References] Reference 14 has a typo ('Foundaton' should be 'Foundation') and Reference 20 has a typo ('Allaince' should be 'Alliance').
- [Rule 9] The informal aside 'oops, now I have made this a scary story you might tell PIs around a campfire' is out of place in a journal article and should be removed or rewritten.
Circularity Check
No significant circularity: the paper is an advisory list with no derived quantitative claims; its self-citations are supporting references, not load-bearing reductions.
full rationale
This is a Ten Simple Rules advisory paper. It contains no equations, no fitted parameters, no predictions from a model, and no formal derivation whose conclusions are equivalent to its inputs by construction. The opening causal claim that RSEng improves research outcomes is supported by citations [1,2], and reference [1] is co-authored by some of the present authors. However, the rules themselves are stated as independent practical heuristics; the self-citations are used to point at prior work and definitions, not to force a derivation. The acknowledgment that surveys informed the content does not make the content circular, because the rules are not presented as empirical predictions and no survey-based quantity is reused as an output. The abstract's promise of improved outcomes is an unsupported overclaim, but unsupported causal strength is an evidence and risk issue, not a circularity issue under the specified criteria. Accordingly no circular steps can be quoted and the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Better research software leads to better and more reproducible research outcomes.
- ad hoc to paper The ten selected rules are the appropriate and sufficient set of practices for PIs to integrate RSEng.
- domain assumption PIs can realistically implement these rules within their groups' constraints of time, funding, and expertise.
Cite this review
Pith. "Pith review of Ten simple rules for PIs to integrate Research Software Engineering into their research group." pith.science (2026). https://pith.science/paper/WKQRPEMT
@misc{pith2026250620217,
author = {Pith},
title = {Pith review of: Ten simple rules for PIs to integrate Research Software Engineering into their research group},
year = {2026},
howpublished = {\url{https://pith.science/paper/WKQRPEMT}},
note = {Machine review of arXiv:2506.20217}
}
read the original abstract
Research Software Engineering (RSEng) is a key success factor in producing high-quality research software, which in turn enables and improves research outcomes. However, as a principal investigator or leader of a research group you may not know what RSEng is, where to get started with it, or how to use it to maximize its benefit for your research. RSEng also often comes with technical complexity, and therefore reduced accessibility to some researchers. The ten simple rules presented in this paper aim to improve the accessibility of RSEng, and provide practical and actionable advice to PIs and leaders for integrating RSEng into their research group. By following these rules, readers can improve the quality, reproducibility, and trustworthiness of their research software, ultimately leading to better, more reproducible and more trustworthy research outcomes.
Reference graph
Works this paper leans on
-
[2]
Better Software, Better Research
Goble C. Better Software, Better Research. IEEE Internet Computing. 2014;18(5):4–8. doi:10.1109/MIC.2014.88
-
[1]
The Four Pillars of Research Software Engineering
Cohen J, Katz DS, Barker M, Chue Hong N, Haines R, Jay C. The Four Pillars of Research Software Engineering. IEEE Software. 2021;38(1):97–105. doi:10.1109/MS.2020.2973362
arXiv 2021
-
[3]
H Washizaki, eds . Guide to the Software Engineering Body of Knowledge (SWEBOK Guide)Version 4.0, IEEE Computer Society,; 2024.www.swebok.org. September 23, 2025 8/10
work page 2024
-
[4]
Ten Simple Rules for Making Research Software More Robust
Taschuk M, Wilson G. Ten Simple Rules for Making Research Software More Robust. PLOS Computational Biology. 2017;13(4):e1005412. doi:10.1371/journal.pcbi.1005412
-
[5]
Ten Simple Rules on Writing Clean and Reliable Open-Source Scientific Software
Hunter-Zinck H, de Siqueira AF, V´ asquez VN, Barnes R, Martinez CC. Ten Simple Rules on Writing Clean and Reliable Open-Source Scientific Software. PLOS Computational Biology. 2021;17(11):e1009481. doi:10.1371/journal.pcbi.1009481
-
[6]
Ten Simple Rules for Reproducible Computational Research
Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology. 2013;9(10):e1003285. doi:10.1371/journal.pcbi.1003285
-
[7]
Ten Simple Rules for Quick and Dirty Scientific Programming
Balaban G, Grytten I, Rand KD, Scheffer L, Sandve GK. Ten Simple Rules for Quick and Dirty Scientific Programming. PLOS Computational Biology. 2021;17(3):e1008549. doi:10.1371/journal.pcbi.1008549
-
[8]
Requirements engineering as a success factor in software projects
Hofmann HF, Lehner F. Requirements engineering as a success factor in software projects. IEEE Software. 2001;18(4):58–66. doi:10.1109/ms.2001.936219
arXiv 2001
Show all 21 references
-
[9]
Claims about the use of software engineering practices in science: A systematic literature review
Heaton D, Carver JC. Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology. 2015;67:207–219. doi:10.1016/j.infsof.2015.07.011
2015 doi
-
[10]
Requirements Engineering for Research Software: A Vision
Bajraktari A, Binder M, Vogelsang A. Requirements Engineering for Research Software: A Vision. In: 2024 IEEE 32nd International Requirements Engineering Conference (RE); 2024. p. 423–431. Available from: https://doi.org/10.1109/RE59067.2024.00050
2024
-
[11]
Manifesto for Agile Software Development; 2001
Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al.. Manifesto for Agile Software Development; 2001. https://agilemanifesto.org
2001
-
[12]
Better Architecture, Better Software, Better Research
Druskat S, Eisty NU, Chisholm R, Chue Hong N, Cocking RC, Cohen MB, et al. Better Architecture, Better Software, Better Research. Computing in Science & Engineering. 2025; p. 1–11. doi:10.1109/MCSE.2025.3573887
2025
-
[13]
The Open Source Definition; 2024
Open Source Initiative. The Open Source Definition; 2024. https://opensource.org/osd
2024
-
[14]
How to Choose a License for Your Own Work; 2024
Free Software Foundaton. How to Choose a License for Your Own Work; 2024. https://www.gnu.org/licenses/license-recommendations.html
2024
-
[15]
How open science helps researchers succeed
McKiernan EC, Bourne PE, Brown CT, Buck S, Kenall A, Lin J, et al. How open science helps researchers succeed. eLife. 2016;5. doi:10.7554/eLife.16800
2016 doi
-
[16]
10 quick tips for making your software outlive your job
Littauer R, Wilson G, Ainali J, AlOmar EA, Arabas S, Bellini Saibene Y, et al. 10 quick tips for making your software outlive your job. arXiv. 2025;(2505.06484). doi:10.48550/arXiv.2505.06484
2025 doi
-
[17]
Software Development Environments for Scientific and Engineering Software: A Series of Case Studies
Carver JC, Kendall RP, Squires SE, Post DE. Software Development Environments for Scientific and Engineering Software: A Series of Case Studies. In: 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society;...
2007 doi
-
[18]
Testing scientific software: A systematic literature review
Kanewala U, Bieman JM. Testing scientific software: A systematic literature review. Inf Softw Technol. 2014;56(10):1219–1232. doi:10.1016/J.INFSOF.2014.05.006
2014 doi
-
[19]
Challenges for verifying and validating scientific software in computational materials science
Vogel T, Druskat S, Scheidgen M, Draxl C, Grunske L. Challenges for verifying and validating scientific software in computational materials science. In: Proceedings of the 14th International Workshop on Software Engineering for Science, SE4Science@ICSE 2019, Montreal, QC, Cana...
2019
-
[20]
F AIR Principles for Research Software (F AIR4RS Principles)
Chue Hong NP, Katz DS, Barker M, Lamprecht AL, Martinez C, Psomopoulos FE, et al. F AIR Principles for Research Software (F AIR4RS Principles). Research Data Allaince, Research Software Alliance, and FORCE11; 2022. Available from: https://doi.org/10.15497/RDA00068
2022 doi
-
[21]
Software citation principles
Smith AM, Katz DS, Niemeyer KE. Software citation principles. PeerJ Computer Science. 2016;2:e86. doi:10.7717/peerj-cs.86. September 23, 2025 10/10
2016 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.