REVIEW 3 major objections 6 minor 34 references
Rookie Mistakes: Measuring Software Quality in Student Projects to Guide Educational Enhancement
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Student Java projects repeat the same quality flaws, study finds
desk verdict Useful descriptive data on where intermediate OOP students struggle, but the architectural frequencies need validation before being cited as facts. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-part static analysis pipeline: ArchUnit, a Java library that encodes architectural rules as JUnit tests, checks that no dependency crosses the prescribed package boundaries (for example, GUI must not touch Model); SonarQube, configured with its 677 default Java rules, classifies findings into Bugs, Vulnerabilities, and Code Smells. The ArchUnit rule suite was written by two of the authors specifically for the taught architecture, while SonarQube supplies the code-quality categories and severities that drive the second research question. What carries the argument is the coupling of these two detections with per-project violation counts, which converts individual student code into frequencies over 83 projects.
What would settle it
Manually inspect a random sample of, say, 20 of the 83 projects, labelling actual GUI-to-Model dependencies and unclosed resources by hand, then compare with the ArchUnit and SonarQube output; if the automated rates deviate substantially from the manual rates, the reported frequencies are test artifacts rather than student behaviour.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that intermediate OOP students, given a prescribed four-package architecture (GUI, Controller, Model, DAO) and a substantial database-backed GUI assignment, systematically produce a recognizable set of quality defects. The headline numbers: GUI-to-Model dependencies in 75% of projects, unclosed resources in 76%, duplicated string literals in 96%, naming-convention violations in up to 94%, and hard-coded database credentials in 39%. The authors interpret these as symptoms of two underlying causes: students take shortcuts to avoid serialization between layers, and they misunderstand responsibility assignment between packages. They also note that many resource-handling mistakes may come from students carrying over C habits to Java, wrongly trusting garbage collection to manage streams and connections.
Load-bearing premise
The reported frequencies rest on the accuracy of the custom ArchUnit rules written by two of the authors and on SonarQube's default 677-rule set as a fair proxy for code quality; neither is independently validated against a manually labelled sample, so the percentages could be artifacts of the rule definitions.
Editorial extensions
If this is right
- If the frequencies hold, OOP courses should explicitly teach why GUI-to-Model shortcuts are harmful, not just how to structure packages; exercises that force serialization through controllers would address the most common architectural violation.
- The 76% unclosed-resources rate suggests teaching try-with-resources and contrasting Java's garbage collection with C's manual management early, before students build database-backed apps.
- Naming conventions and duplicated-string problems dominate (84–96%), so a first pass with an automated formatter or linter in CS1 could remove most of the maintainability debt before advanced software-engineering courses.
- Educators can use the same static-analysis pipeline as an assessment artifact: running SonarQube and ArchUnit on submissions gives objective, reproducible quality baselines for grading and feedback.
Reading between the lines
- The high prevalence of GUI-to-Model links suggests a testable hypothesis: students who receive worked examples of serialization refactorings will show lower violation rates in subsequent projects; the paper does not test this.
- Because the data comes from a single university with a specific taught architecture, the exact percentages likely shift in other settings, but the ordering (naming > resource handling > architectural shortcuts) may be a stable pattern worth treating as a prior for course design elsewhere.
- The finding that 39% of projects hard-code database credentials, though not a majority issue, implies that security basics could be taught at the same stage; the paper flags it as Blocker severity but does not fold it into the main teaching recommendations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an empirical study of software quality in 83 Java projects developed by 172 undergraduate students across four editions of an OOP course. Using a static-analysis pipeline that combines SonarQube and a custom ArchUnit rule suite, the authors answer two research questions: RQ1 measures the prevalence of architectural violations (e.g., GUI-to-Model dependencies in 75% of projects) and RQ2 measures the prevalence of code smells and bugs (e.g., unclosed resources in 76% and duplicated string literals in 96% of projects). The authors argue that these frequencies identify where educators should focus quality-related teaching, and they provide course details plus a discussion of threats to validity.
Significance. If the reported frequencies are accurate, the paper fills a genuine gap: prior work mostly targets novice programmers and small, self-contained tasks, whereas this corpus consists of multi-component GUI/database projects developed by intermediate-level students. The use of two complementary tools, the relatively large corpus, and the explicit description of the course context are strengths, as is the promise of a replication package. However, the central RQ1 claim rests on a custom ArchUnit rule suite that is neither specified nor independently validated, and the RQ2 headline percentages in Table 4 contain arithmetic inconsistencies with Table 1. These issues must be resolved before the empirical frequencies can be regarded as reliable evidence for the paper's educational recommendations.
major comments (3)
- [Section 3.3, Table 2 (RQ1)] The ArchUnit rule suite is the measurement instrument for RQ1, but the paper does not specify the rules, how student packages were mapped to the four taught layers, or how the suite was validated. Section 3.3 only says the suite was 'created by two of the authors' and Section 6 states that it was 'tuned and validated' without reporting any procedure. Because the RQ1 metric is presence/absence of a violation per project, a systematic false-positive or false-negative rate can move the reported percentages (e.g., the 75% G-to-M figure) across the majority threshold. Please provide the full rule definitions, the package-mapping strategy (including handling of non-canonical names and nested or transitive references), and a validation against manually annotated projects, such as precision/recall or inter-rater agreement.
- [Table 4 versus Table 1 (RQ2)] The counts in Table 4 are internally inconsistent with Table 1. Table 1 lists 20 projects for Group 1 in 2022/23, but Table 4 reports 17 (77%) for 'Resources should be closed' in that cohort, which would require 17/22, not 17/20; the row 'String literals' (21 (95%)) also implies a denominator of 22. If the true denominator is 22, the total number of projects becomes 85, not 83. Moreover, the 'Overall' percentages do not match the sums of the per-group counts divided by 83: 'Resources' sums to 65 (78%), not 76%, and 'Unnecessary imports' sums to 64 (77%), not 75%. Since these percentages are the headline results of RQ2, the table must be reconciled with the stated corpus size.
- [Section 4, RQ2 filtering] The selection of the 17 highlighted issue types is not fully reproducible. The paper states that issues 'explicitly related to topics outside the scope of the course' were filtered out, but it gives only one example (lambda expressions) and does not list the excluded rules or the complete set of rules with prevalence above 50%. Because the central RQ2 list is defined by this post-hoc filter, please provide the full per-rule prevalence data, the exclusion list with reasons, and the per-project raw data in the replication package so that readers can verify which rules were considered and why.
minor comments (6)
- [Section 4, RQ2, Table 3] The sentence 'A possible cause for these slight reductions... might be due to the increased effectiveness of teaching in person' is speculative; Table 1 shows that mean NCLOC also decreased in the 22/23 editions (e.g., Group 1 from 5797 to 3887), so project size is a confound. Please soften the claim or control for size.
- [Section 3.3] Please specify the SonarQube version and the exact rule-set version, since the number of default Java rules varies across releases and would affect the reproducibility of the RQ2 counts.
- [Table 2] The statement that disallowed dependencies are 'highlighted in red' is not accessible in monochrome print; use an explicit marker such as an asterisk or a 'Disallowed?' column.
- [Section 2.1] The term 'social debt' is introduced without definition; if it refers to the social dimension of software quality, please define it explicitly or rephrase.
- [Section 4, RQ1] There is a typo: 'betweeen' should be 'between'.
- [Reference [1]] The replication package is described as anonymized, but the URL (https://github.com/SergioDME/SEAA2025) appears to contain an author name; please verify that the link is correct and that the package includes the ArchUnit rules and raw per-project data.
Circularity Check
No significant circularity: the reported violation frequencies are empirical measurements, the custom ArchUnit rules operationalize the taught architecture, and the self-citation to prior work [9] is not load-bearing.
full rationale
The paper's core outputs are empirical frequency counts obtained by running SonarQube's 677 default Java rules and a custom ArchUnit suite over 83 student projects. There is no fitted parameter later renamed as a prediction, no equation whose output equals its input, and no uniqueness theorem or prior result that forces the reported percentages. The custom ArchUnit rules operationalize the four-package architecture taught in the course; the G-to-M and other coupling percentages are then measured by checking actual package dependencies, so the central claims are externally checkable against the replication package and could have come out differently. The only self-referential element is citation [9], the same authors' earlier exploratory study, cited as support for the analysis pipeline, together with Section 6's unelaborated statement that ArchUnit was 'specifically tuned and validated.' This raises a measurement-validity concern, but it is not a circular derivation, because the current frequencies do not depend logically on [9] for their truth; the results stand or fall on the actual projects analyzed and the reproducibility of the tool configuration. SonarQube's rule set is an independent, widely used external tool, and the custom ArchUnit rules are a stated operationalization rather than a concealed redefinition of the outcome. Accordingly, no circularity step is identified; the score reflects only the minor self-citation in the instrument's provenance, not a load-bearing circularity.
Assumptions & free parameters
free parameters (1)
- majority prevalence threshold =
>50% of projects
assumptions (4)
- domain assumption SonarQube's default Java rules are a valid operationalization of code quality and code smells.
- ad hoc to paper The custom ArchUnit test suite correctly encodes the taught architecture and detects violations.
- domain assumption Students' projects are representative of intermediate-level OOP students beyond this institution.
- domain assumption Observed issues are attributable to student inexperience or lack of quality education rather than to tool artifacts or project requirements.
Cite this review
Pith. "Pith review of Rookie Mistakes: Measuring Software Quality in Student Projects to Guide Educational Enhancement." pith.science (2026). https://pith.science/paper/TOSEYKOF
@misc{pith2026250712488,
author = {Pith},
title = {Pith review of: Rookie Mistakes: Measuring Software Quality in Student Projects to Guide Educational Enhancement},
year = {2026},
howpublished = {\url{https://pith.science/paper/TOSEYKOF}},
note = {Machine review of arXiv:2507.12488}
}
read the original abstract
When teaching Programming and Software Engineering in Bachelor's Degree programs, the emphasis on creating functional software projects often overshadows the focus on software quality, a trend that aligns with ACM curricula recommendations. Software Engineering courses are typically introduced later in the curriculum, and can generally allocate only limited time to quality-related topics, leaving educators with the challenge of deciding which quality aspects to prioritize. In this decision, the literature offers limited guidance, as most existing studies focus on code written by novice students and small code units, making it unclear whether those findings extend to intermediate-level students with foundational object-oriented programming skills working on more complex software projects. To address this gap, we analyze 83 object-oriented team projects developed by 172 university students across 4 different editions of the Object-Oriented Programming course. We apply a static analysis pipeline used in prior research to assess software quality, combining SonarQube and ArchUnit to detect code smells and architectural anti-patterns. Our findings highlight recurring quality issues and offer concrete evidence of the challenges students face at this stage, providing valuable guidance for educators aiming to continuously improve Software Engineering curricula and promote quality-oriented development practices.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/SergioDME/SEAA2025 16 De Luca et al
Github repo. https://github.com/SergioDME/SEAA2025 16 De Luca et al
-
[2]
In: Proceedings of the 24th International Conference on Soft- ware Engineering
Aldrich, J., Chambers, C., Notkin, D.: Archjava: Connecting software architecture to implementation. In: Proceedings of the 24th International Conference on Soft- ware Engineering. p. 187–197. ICSE ’02, Association for Computing Machinery, New York, NY, USA (2002). https://doi.org/10.1145/581339.581365
-
[3]
In: 2017 IEEE Frontiers in Education Conference (FIE)
de Andrade Gomes, P.H., Garcia, R.E., Spadon, G., Eler, D.M., Olivete, C., Mes- sias Correia, R.C.: Teaching software quality via source code inspection tool. In: 2017 IEEE Frontiers in Education Conference (FIE). pp. 1–8 (2017). https: //doi.org/10.1109/FIE.2017.8190658
-
[4]
Cassee, N., Vasilescu, B., Serebrenik, A.: The silent helper: The impact of con- tinuous integration on code reviews. In: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). pp. 423–434 (2020). https://doi.org/10.1109/SANER48275.2020.9054818
arXiv 2020
- [5]
-
[6]
Chren, S., Macák, M., Rossi, B., Buhnova, B.: Evaluating code improvements in software quality course projects. In: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering. p. 160–169. EASE ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3530019.3530036
-
[7]
In: Proceedings of the 50th ACM Tech- nical Symposium on Computer Science Education
Clear, A., Parrish, A.S., Impagliazzo, J., Zhang, M.: Computing curricula 2020: introduction and community engagement. In: Proceedings of the 50th ACM Tech- nical Symposium on Computer Science Education. pp. 653–654 (2019)
work page 2019
-
[8]
Clements, P., Garlan, D., Little, R., Nord, R., Stafford, J.: Documenting software architectures: views and beyond. In: 25th International Conference on Software En- gineering, 2003. Proceedings. pp. 740–741 (2003). https://doi.org/10.1109/ICSE. 2003.1201264
work page Pith review arXiv 2003
Show all 34 references
-
[9]
In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering
De Luca, M., Di Meglio, S., Fasolino, A.R., Starace, L.L.L., Tramontana, P.: Auto- matic assessment of architectural anti-patterns and code smells in student software projects. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engine...
2024
-
[10]
Expert Systems with Applications231, 120597 (06 2023)
De Luca, M., Fasolino, A., Ferraro, A., Moscato, V., Sperlì, G., Tramontana, P.: A community detection approach based on network representation learning for repository mining. Expert Systems with Applications231, 120597 (06 2023). https: //doi.org/10.1016/j.eswa.2023.120597
2023
-
[11]
Pearson, 11th edn
Deitel,P.J.,Deitel,H.:JavaHowtoProgram,EarlyObjects,StudentValueEdition (11th Edition). Pearson, 11th edn. (2017)
2017
-
[12]
Denning, P.J.: What is software quality? Communications of the ACM35(1), 13– 15 (1992)
1992
-
[13]
In: 2025 IEEE Confer- ence on Software Testing, Verification and Validation (ICST)
Di Meglio, S., Libero Lucio Starace, L., Di Martino, S.: E2e-loader: A tool to generate performance tests from end-to-end gui-level tests. In: 2025 IEEE Confer- ence on Software Testing, Verification and Validation (ICST). pp. 747–751 (2025). https://doi.org/10.1109/ICST62969....
2025
-
[14]
In: IWSM- Mensura (2023)
Di Meglio, S., Starace, L.L.L., Di Martino, S.: Starting a new rest api project? a performance benchmark of frameworks and execution environments. In: IWSM- Mensura (2023)
2023
-
[15]
In: Proceedings of the 53rd ACM Technical Symposium on Computer Sci- ence Education - Volume 1
Effenberger,T.,Pelánek,R.:Codequalitydefectsacrossintroductoryprogramming topics. In: Proceedings of the 53rd ACM Technical Symposium on Computer Sci- ence Education - Volume 1. p. 941–947. SIGCSE 2022, Association for Computing Machinery, New York, NY, USA (2022). https://doi...
2022
-
[16]
In: Systems, Software and Services Process Improvement: 25th European Conference, EuroSPI 2018, Bilbao, Spain, September 5-7, 2018, Proceedings 25
Estdale, J., Georgiadou, E.: Applying the iso/iec 25010 quality models to software product. In: Systems, Software and Services Process Improvement: 25th European Conference, EuroSPI 2018, Bilbao, Spain, September 5-7, 2018, Proceedings 25. pp. 492–503. Springer (2018)
2018
-
[17]
Addison-Wesley Longman Publishing Co., Inc., USA, 3 edn
Fowler, M.: UML Distilled: A Brief Guide to the Standard Object Modeling Lan- guage. Addison-Wesley Longman Publishing Co., Inc., USA, 3 edn. (2003)
2003
-
[18]
In: 2010 Fifth International Conference on Software Engineering Advances
Grambow, G., Oberhauser, R.: Towards automated context-aware software qual- ity management. In: 2010 Fifth International Conference on Software Engineering Advances. pp. 347–352 (2010). https://doi.org/10.1109/ICSEA.2010.59
2010 doi
-
[19]
In: Proceedings of the Software Engineering Education and Training (SEET) Workshop at ICSE
Jansen, J., Oprescu, A., Bruntink, M.: The impact of automated code quality feedback in programming education. In: Proceedings of the Software Engineering Education and Training (SEET) Workshop at ICSE. vol. 2070 (2017)
2017
-
[20]
In: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education
Keuning, H., Heeren, B., Jeuring, J.: Code quality issues in student programs. In: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education. p. 110–115. ITiCSE ’17, Association for Computing Machinery, New York, NY, USA (2017). https://...
2017
-
[21]
Journal of Systems and Software198, 111575 (2023)
Lenarduzzi, V., Pecorelli, F., Saarimaki, N., Lujan, S., Palomba, F.: A crit- ical comparison on six static analysis tools: Detection, agreement, and preci- sion. Journal of Systems and Software198, 111575 (2023). https://doi.org/https: //doi.org/10.1016/j.jss.2022.111575
2023
-
[22]
Frontiers of Computer Science14(5) (2019)
Lu, Y., Mao, X., Wang, T., Yin, G., Li, Z.: Improving students’ programming qual- ity with the continuous inspection process: a social coding perspective. Frontiers of Computer Science14(5) (2019). https://doi.org/10.1007/s11704-019-9023-2
2019 doi
-
[23]
Pearson Education (2009)
Martin, R.C.: Clean code: a handbook of agile software craftsmanship. Pearson Education (2009)
2009
-
[24]
IEEE Access 12, 161649–161669 (2024)
Meglio, S.D., Libero Lucio Starace, L.: Evaluating performance and resource con- sumption of rest frameworks and execution environments: Insights and guide- lines for developers and companies. IEEE Access 12, 161649–161669 (2024). https://doi.org/10.1109/ACCESS.2024.3489892
2024
-
[25]
In: 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR)
Meglio, S.D., Starace, L.L.L., Pontillo, V., Opdebeeck, R., Roover, C.D., Mar- tino, S.D.: E2egit: A dataset of end-to-end web tests in open source projects. In: 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). pp. 836–840 (2025). https://doi....
2025
-
[26]
IEEE Transactions on Software Engineering 47(5), 1008–1028 (2021)
Mo, R., Cai, Y., Kazman, R., Xiao, L., Feng, Q.: Architecture anti-patterns: Auto- matically detectable violations of design principles. IEEE Transactions on Software Engineering 47(5), 1008–1028 (2021). https://doi.org/10.1109/TSE.2019.2910856
2021
-
[27]
Software Quality Journal 31(1), 211–241 (Sep 2022)
Oliveira, G.P., Moura, A.F.C., Batista, N.A., Brandão, M.A., Hora, A., Moro, M.M.: How do developers collaborate? investigating github heterogeneous net- works. Software Quality Journal 31(1), 211–241 (Sep 2022). https://doi.org/10. 1007/s11219-022-09598-x
2022
-
[28]
International Journal of Automation and Com- puting 10(2), 157–166 (Apr 2013)
Pan, W.F., Jiang, B., Li, B.: Refactoring software packages via community detec- tion in complex software networks. International Journal of Automation and Com- puting 10(2), 157–166 (Apr 2013). https://doi.org/10.1007/s11633-013-0708-y
2013 doi
-
[29]
437–442 (01 2018)
Poth, A., Heimann, C.: How to Innovate Software Quality Assurance and Testing in Large Enterprises?: 25th European Conference, EuroSPI 2018, Bilbao, Spain, September 5-7, 2018, Proceedings, pp. 437–442 (01 2018). https://doi.org/10.1007/ 978-3-319-97925-0_37
2018
-
[30]
Prentice-Hall, Inc
Shaw, M., Garlan, D.: Software architecture: perspectives on an emerging disci- pline. Prentice-Hall, Inc. (1996) 18 De Luca et al
1996
-
[31]
IEEE Access 8, 37505–37517 (2020)
Sun, Q., Wu, J., Liu, K.: Toward understanding students’ learning performance in an object-oriented programming course: The perspective of program quality. IEEE Access 8, 37505–37517 (2020)
2020
-
[32]
John Wiley & Sons (2010)
Taylor, R.N., Medvidovic, N., Dashofy, E.M.: Software Architecture: Foundations, Theory, and Practice. John Wiley & Sons (2010)
2010
-
[33]
Review of Research in Education44(1), 403–433 (2020)
Yurkofsky, M.M., Peterson, A.J., Mehta, J.D., Horwitz-Willis, R., Frumin, K.M.: Research on continuous improvement: Exploring the complexities of managing ed- ucational change. Review of Research in Education44(1), 403–433 (2020)
2020
-
[34]
In: Proceedings of the 35th Annual ACM Sympo- sium on Applied Computing
Zakurdaeva, A., Weiss, M., Muegge, S.: Detecting architectural integrity violation patterns using machine learning. In: Proceedings of the 35th Annual ACM Sympo- sium on Applied Computing. p. 1480–1487. SAC ’20, Association for Computing Machinery, New York, NY, USA (2020). ht...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.