Pith. sign in

REVIEW 2 major objections 2 minor 26 references

Gemini 3 Flash scores highest on 993 Scrum certification questions, followed by GPT-5 mini and DeepSeek Chat 3.2 with low variability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-02 20:02 UTC pith:WUMFYHCK

load-bearing objection Gemini 3 Flash leads on the PSM I questions with low variance, but the 993-question set's origin is too vague to trust the ranking without more detail. the 2 major comments →

arxiv 2607.00048 v1 pith:WUMFYHCK submitted 2026-06-29 cs.SE cs.AI

Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

Robson Alves Vilar , Emanuel Dantas Filho , Ademar Fran\c{c}a de Sousa Neto , Mirko Perkusich , Danyllo Wagner Albuquerque , Jo\~ao Paiva , Kyller Gorg\^onio , Angelo Perkusich This is my paper
classification cs.SE cs.AI
keywords Large Language ModelsScrum CertificationQuestion AnsweringPerformance ComparisonPrompt EngineeringSoftware EngineeringError Analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests three large language models on 993 questions formatted like the Professional Scrum Master I certification exam. It applies three different prompting approaches and repeats the tests to measure consistency. The evaluation covers accuracy by model, question type, and Scrum topic, plus a review of why models get answers wrong. A reader would care because these tasks require precise application of defined rules rather than general knowledge, revealing how ready LLMs are for professional certification support in software engineering.

Core claim

Gemini 3 Flash achieved the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2. Intra-model variability was low across all prompting strategies. Models performed best on single-answer multiple-choice questions and on topics with clear normative definitions like Artifacts and Empiricism, while struggling more with multi-select questions, True/False items, and topics like Scrum Values and Self-Managing Teams. Errors followed patterns such as overgeneralization and misalignment with strict Scrum definitions rather than random mistakes.

What carries the argument

Systematic comparison of three LLMs under zero-shot, chain-of-thought, and source-grounded prompting on a fixed set of 993 PSM I-aligned questions, with analysis of accuracy, stability, topic/format breakdowns, and qualitative error patterns.

Load-bearing premise

The selected 993 questions represent the PSM I assessment without bias and the three prompting strategies test model capabilities fairly.

What would settle it

Running the same models on an official set of PSM I practice questions released by Scrum.org and finding reversed accuracy rankings or high variability across runs would falsify the reported differences and stability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Model selection can meaningfully affect performance in Scrum certification preparation tasks.
  • Low intra-model variability suggests consistent outputs suitable for repeated testing scenarios.
  • Performance gaps by topic indicate that some Scrum areas are more aligned with current LLM training data than others.
  • Systematic error patterns point to specific challenges in handling restrictive wording and market vs. official definition conflicts.
  • Highest accuracy on single-answer questions suggests simpler formats are more reliable for current models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • These results could guide the development of domain-specific fine-tuning for agile and Scrum knowledge.
  • Similar evaluations on other certification exams might show whether the model ranking holds across software engineering domains.
  • The topic-specific weaknesses could inform targeted improvements in how LLMs are trained on normative frameworks.
  • Low variability supports using these models for generating practice exams with predictable behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper empirically compares three LLMs (GPT-5 mini, Gemini 3 Flash, DeepSeek Chat 3.2) on 993 Scrum certification-style questions aligned with the PSM I format. It evaluates performance under zero-shot, chain-of-thought, and source-grounded prompting, with repeated runs to measure intra-model stability. Results are broken down by question format (single-answer MCQ highest accuracy; multi-select and T/F more error-prone) and topic (stronger on Artifacts/Empiricism/Product Value; weaker on Scrum Values/Self-Managing Teams/Stakeholders), plus a qualitative review of systematic error types (overgeneralization, restrictive wording, compound distractors, market vs. strict Scrum conflicts). The headline finding is that Gemini 3 Flash leads in accuracy, followed by GPT-5 mini then DeepSeek, with low variability across runs.

Significance. If the central claims hold after addressing dataset details, the work supplies a useful benchmark for LLM behavior on normatively strict, domain-specific certification tasks in software engineering. Strengths include the repeated-execution design for stability assessment and the topic/format/error-pattern breakdowns, which move beyond aggregate accuracy to identify where models are fragile. These elements could inform both LLM-assisted Scrum training tools and broader questions about LLM reliability on rule-based professional knowledge.

major comments (2)
  1. [Methods] Methods / dataset construction: the 993 questions are described only as 'aligned with' PSM I format, with no details on sourcing (official Scrum.org materials vs. author-generated), selection criteria, post-hoc filtering, or balance against official topic weights. This is load-bearing for the model ranking (Gemini 3 Flash highest) because curation bias could systematically favor topics or phrasings where one model already excels.
  2. [Results] Results: accuracy differences and the claim of 'clear differences among models' are reported without statistical tests (e.g., proportion tests or paired comparisons across the repeated executions) or confidence intervals that account for both stochasticity and question sampling. This leaves the ordering vulnerable to sampling variation in the 993-item set.
minor comments (2)
  1. [Prompting Strategies] The description of the 'source-grounded' prompting strategy would benefit from an explicit example showing what source material is supplied and how it is formatted, to support reproducibility.
  2. [Results] Tables reporting per-topic or per-format accuracies should include the number of questions per cell and, ideally, variability measures from the repeated runs.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript accordingly to improve transparency and statistical rigor.

read point-by-point responses

  1. Referee: [Methods] Methods / dataset construction: the 993 questions are described only as 'aligned with' PSM I format, with no details on sourcing (official Scrum.org materials vs. author-generated), selection criteria, post-hoc filtering, or balance against official topic weights. This is load-bearing for the model ranking (Gemini 3 Flash highest) because curation bias could systematically favor topics or phrasings where one model already excels.

    Authors: We agree that the Methods section requires greater detail on dataset construction to permit evaluation of potential curation effects. The 993 questions were drawn from publicly available PSM I practice examinations and study resources that follow the official Scrum.org PSM I format and topic distribution. Questions were selected to achieve approximate proportionality with the official PSM I topic weights, with deduplication as the only post-selection step. We will add a dedicated Methods subsection that specifies sourcing, selection criteria, and a comparison table of topic coverage versus official weights. This addition will allow readers to assess whether any observed model ordering could be attributable to dataset composition. revision: yes

  2. Referee: [Results] Results: accuracy differences and the claim of 'clear differences among models' are reported without statistical tests (e.g., proportion tests or paired comparisons across the repeated executions) or confidence intervals that account for both stochasticity and question sampling. This leaves the ordering vulnerable to sampling variation in the 993-item set.

    Authors: We accept that formal statistical support is needed to substantiate the reported model ordering. Although the five repeated executions per condition already demonstrate low intra-model variance and consistent rank order, we will incorporate McNemar's tests for paired accuracy comparisons across models on identical question sets and bootstrap confidence intervals that reflect both model stochasticity and the finite sample of 993 questions. These results, together with updated tables, will be added to the Results section. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical comparison on fixed questions

full rationale

The paper conducts a direct empirical evaluation of three LLMs on a fixed set of 993 Scrum certification-style questions under three prompting strategies, measuring accuracy, stability, topic/format performance, and error patterns. No derivations, equations, fitted parameters, predictions, uniqueness theorems, or ansatzes are present. Results are raw measurements against an external question set; no step reduces to its own inputs by construction or self-citation. The central claims rest on observable outputs rather than any load-bearing self-referential structure.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical benchmarking study; no mathematical derivations, free parameters, axioms, or invented entities are present in the central claim.

pith-pipeline@v0.9.1-grok · 5888 in / 1106 out tokens · 43384 ms · 2026-07-02T20:02:13.444218+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns." pith.science (2026). https://pith.science/paper/WUMFYHCK

@misc{pith2026260700048,
  author       = {Pith},
  title        = {Pith review of: Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUMFYHCK}},
  note         = {Machine review of arXiv:2607.00048}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, \textit{GPT-5 mini}, \textit{Gemini 3 Flash}, and \textit{DeepSeek Chat 3.2}, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (\textit{zero-shot}, \textit{chain-of-thought}, and \textit{source-grounded}), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders \& Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.

Figures

Figures reproduced from arXiv: 2607.00048 by Ademar Fran\c{c}a de Sousa Neto, Angelo Perkusich, Danyllo Wagner Albuquerque, Emanuel Dantas Filho, Jo\~ao Paiva, Kyller Gorg\^onio, Mirko Perkusich, Robson Alves Vilar.

Figure 1
Figure 1. Figure 1: Overview of the experimental design. 3.2.2 Prompting Strategies. We evaluated three prompting strate￾gies that reflect different levels of instruction and source grounding: • Zero-shot: the question and its alternatives are presented without additional guidance, serving as a baseline for direct answer generation. • Chain-of-Thought: the model is instructed to reason briefly before stating the final answer,… view at source ↗
Figure 2
Figure 2. Figure 2: Average accuracy (%) by model and prompting strat [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy (%) by question format and prompting strategy across the three evaluated models. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Vahid Ashrafimoghari et al. 2024. Evaluating large language models on the GMAT: Implications for the future of business education.arXiv preprint arXiv:2401.02985(2024)

    work page arXiv 2024

  2. [2]

    Beatriz Cabrero-Daniel, Tomas Herda, Victoria Pichler, and Martin Eder. 2024. Exploring human-ai collaboration in agile: Customised llm meeting assistants. InInternational Conference on Agile Software Development. Springer Nature Switzerland Cham, 163–178

    work page 2024

  3. [3]

    Jarosław A Chudziak and Konrad Cinkusz. 2024. Towards LLM-augmented multiagent systems for agile software engineering. In2024 39th IEEE/ACM Inter- national Conference on Automated Software Engineering (ASE). IEEE, 2476–2477

    work page 2024

  4. [4]

    Konrad Cinkusz et al. 2024. Cognitive agents powered by large language models for agile software project management.Electronics14, 1 (2024), 87

    work page 2024

  5. [5]

    Rodrigo A Crisostomo, Nicolas A Nunez, and Giuliano Lopez-Burga. 2024. En- hancing Professional Employability: The Impact of Agile Methodology Training. International Journal of Engineering Pedagogy14, 8 (2024)

    work page 2024

  6. [6]

    Cruzes and Tore Dybå

    Daniela S. Cruzes and Tore Dybå. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In2011 International Symposium on Empirical Software Engineering and Measurement. IEEE, 275–284. doi:10.1109/ESEM.2011.36

    work page doi:10.1109/esem.2011.36 2011

  7. [7]

    G Dhruva, Ishaan Shettigar, Srikrshna Parthasarthy, and VM Sapna. 2024. Agile Project Management Using Large Language Models. In2024 5th International Conference on Innovative Trends in Information Technology (ICITIIT). IEEE, 1–6

    work page 2024

  8. [8]

    Alexandre O Gérard et al. 2025. Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers.British Journal of Clinical Pharmacology(2025)

    work page 2025

  9. [9]

    Keito Inoshita. 2024. Assessing GPT’s Legal Knowledge in Japanese Real Estate Transactions Exam. In2024 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT). IEEE, 149–155

    work page 2024

  10. [10]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Informa- tion Processing Systems, Vol. 33. 9459–9474

    work page 2020

  11. [11]

    N. C. Mendonça. 2024. Evaluating ChatGPT-4 Vision on Brazil’s National Under- graduate Computer Science Exam.ACM Transactions on Computing Education 24, 3 (2024), 1–56

    work page 2024

  12. [12]

    Rahul Modak et al. 2023. Integrating LLMs into Agile Software Development: A 2023 Perspective on Productivity and Code Quality.Well Testing Journal32, 2 (2023), 130–146

    work page 2023

  13. [13]

    Alessandra Montenegro. 2019. Competences for the Future: A Comparative Analysis of Agile Certifications.European Project Management Journal9, 2 (2019), 46–54

    work page 2019

  14. [14]

    Andrea S Patrucco, Filomena Canterino, and Inga Minelgaite. 2022. How do scrum methodologies influence the team’s cultural values? A multiple case study on agile teams in Nonsoftware industries.IEEE Transactions on Engineering Management69, 6 (2022), 3503–3513

    work page 2022

  15. [15]

    Pires, T

    R. Pires, T. S. Almeida, H. Q. Abonizio, and R. F. Nogueira. 2023. Evaluating GPT-4’s Vision Capabilities on Brazilian University Admission Exams.arXiv preprint arXiv:2311.14169(2023)

    work page arXiv 2023

  16. [16]

    Paul Ralph, Nauman Bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, et al

    work page

  17. [17]

    Empirical Standards for Software Engineering Research.arXiv preprint arXiv:2010.03525(2020)

    work page arXiv 2010

  18. [18]

    Leila Samimi and Shekoufeh Kolahdouz Rahimi. 2025. Bridging Agility and Automation: Enhancing Model-Driven Engineering with LLMs in Scrum. In Agile Model-driven Engineering Workshop. CEUR Workshop Proceedings

    work page 2025

  19. [19]

    Which prompting technique should i use? an empirical investigation of prompting techniques for software engineering tasks,

    Edson G. Santana Jr. et al . 2025. Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks.arXiv preprint arXiv:2506.05614(2025). Preprint; under review

    work page arXiv 2025

  20. [20]

    Scrum.org. 2020. Scrum Guide and Professional Scrum Certifications. https: //www.scrum.org/. Accessed: 2026-04-13

    work page 2020

  21. [21]

    Jiho Shin, , et al. 2023. Prompt Engineering or Fine-Tuning: An Empirical As- sessment of LLMs for Code.arXiv preprint arXiv:2310.10508(2023)

    work page arXiv 2023

  22. [22]

    Hanna Soroka-Potrzebna. 2021. The importance of certification in project man- agement in the labor market.Procedia Computer Science192 (2021), 1934–1943

    work page 2021

  23. [23]

    Cayo Viegas, Rohit Gheyi, and Márcio Ribeiro. 2025. Assessing the Capability of LLMs in Solving POSCOMP Questions.Journal of the Brazilian Computer Society 31, 1 (Oct. 2025), 991–1004

    work page 2025

  24. [24]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837

    work page 2022

  25. [25]

    Ohlsson, Björn Regnell, and Anders Wesslén

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012.Experimentation in Software Engineering(2 ed.). Springer. doi:10.1007/978-3-642-29044-2

    work page doi:10.1007/978-3-642-29044-2 2012

  26. [26]

    Zheying Zhang et al. 2024. Llm-based agents for automating the enhancement of user story quality: An early report. InInternational Conference on Agile Software Development. Springer Nature Switzerland Cham, 117–126

    work page 2024