Pith. sign in

REVIEW 2 major objections 2 minor 21 references

GPT-5 reaches certification-level accuracy above 85 percent on Professional Scrum Master questions with citation prompts performing best at 89.1 percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-02 19:59 UTC pith:2PJRXHQJ

load-bearing objection This paper gives concrete accuracy numbers for three prompting methods on 993 PSM questions but the error categorization has no described method. the 2 major comments →

arxiv 2607.00049 v1 pith:2PJRXHQJ submitted 2026-06-29 cs.SE cs.AI

Prompting GPT-5 on Scrum Certification Questions: An Empirical Accuracy Study

Mirko Perkusich , Danyllo Albuquerque , Jo\~ao Paiva , Robson Vilar , Emanuel Dantas , Ademar Fran\c{c}a de Sousa Neto , Rohit Gheyi , Kyller Gorg\^onio
show 1 more author
Angelo Perkusich
This is my paper
classification cs.SE cs.AI
keywords Large Language ModelsScrumPrompt EngineeringProfessional Scrum MasterCertification AccuracyAgile Software DevelopmentGPT-5Empirical Study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests GPT-5 on 993 validated questions drawn from Professional Scrum Master exam content using three prompt styles: zero-shot, chain-of-thought, and prompts that require citing the Scrum Guide. All three styles cleared 85 percent accuracy, though the citation style posted the highest score and fewest errors. Performance held up best on topics with explicit rules such as the Definition of Done and on single-answer questions, while multi-select items and interpretive topics like Scrum Team roles showed more instability. The study breaks down the 16.2 percent of questions where at least one prompt failed into misalignment with the guide, material outside its scope, and outdated or biased readings.

Core claim

Prompt techniques enable GPT-5 to answer Scrum certification questions at passing levels, with the citation-based variant delivering the strongest result of 89.1 percent accuracy and the lowest error rate across the 993-question set.

What carries the argument

Comparison of zero-shot, chain-of-thought, and citation-based prompting on the 993-question PSM-aligned dataset.

Load-bearing premise

The 993-question dataset correctly reflects the rules and scope of the Scrum Guide without external bias or outdated content.

What would settle it

Independent scoring of the same 993 questions by certified PSM holders or direct line-by-line comparison of model answers against the Scrum Guide text.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • LLMs become viable tools for Agile certification preparation when citation prompts are used.
  • Citation prompts reduce version drift and misinterpretation compared with simpler prompting.
  • Accuracy is higher on single-answer questions and well-defined topics than on multi-select or interpretive ones.
  • Error patterns point to specific topics where additional guardrails would help LLM responses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same prompting comparison could be run on other Agile certifications or frameworks to test transferability.
  • Training platforms might embed citation-style prompts to improve reliability for users studying Scrum.
  • Future model updates trained on the latest Scrum Guide could shrink the performance gap between prompt styles.
  • The three error categories could guide targeted fine-tuning or retrieval-augmented generation setups.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper reports an empirical evaluation of GPT-5 accuracy on a dataset of 993 validated PSM certification questions using zero-shot, chain-of-thought, and citation-based prompting. All techniques exceed 85% accuracy, with citation prompting reaching 89.1% and the lowest error rate; among the 16.2% of questions where at least one prompt failed, errors are partitioned into 28% misalignment with the Scrum Guide, 34% out-of-scope content, and 38% outdated or biased interpretations. Correct answers are concentrated in well-defined topics and single-answer items.

Significance. If the dataset validation and error labeling are reliable, the work supplies concrete evidence on the modest but consistent gains from citation prompting for normative domain knowledge and identifies topic-level stability differences that could guide LLM deployment in Agile training and certification preparation.

major comments (2)
  1. [Methods] Methods section: the claim that the 993-question dataset is 'validated' and 'PSM-aligned' is asserted without any description of question sourcing, alignment verification against the Scrum Guide (2020), or validation criteria; this directly underpins the headline accuracy figures and the representativeness of the reported topic-level patterns.
  2. [Error analysis] Error analysis section: the partitioning of the 16.2% failed questions into misalignment (28%), out-of-scope (34%), and outdated/biased (38%) is presented without any account of the categorization procedure, number of raters, or inter-rater reliability metric; because the paper attributes the citation variant's advantage to reduced misinterpretation, the absence of these details makes it impossible to isolate prompt effects from annotation artifacts.
minor comments (2)
  1. [Abstract] Abstract: the statement that 'correct answers concentrated in well-defined topics' would be clearer if accompanied by the per-topic accuracy numbers or a supporting table reference.
  2. [Results] Results: no mention of whether the 89.1% figure is from a single run or averaged across multiple generations, nor any statistical test for the differences between prompting conditions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on methodological transparency. We address each major comment below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

  1. Referee: [Methods] Methods section: the claim that the 993-question dataset is 'validated' and 'PSM-aligned' is asserted without any description of question sourcing, alignment verification against the Scrum Guide (2020), or validation criteria; this directly underpins the headline accuracy figures and the representativeness of the reported topic-level patterns.

    Authors: We accept the point that the Methods section lacks sufficient detail on dataset construction. In the revised version we will add a dedicated subsection describing: sourcing from publicly available PSM practice exams and study materials aligned with the 2020 Scrum Guide; the verification process in which each question was manually mapped to specific Guide sections by the authors; and the validation criteria requiring that answers be directly derivable from the Guide without external assumptions. These additions will substantiate the 'validated' and 'PSM-aligned' claims and allow readers to assess representativeness. revision: yes

  2. Referee: [Error analysis] Error analysis section: the partitioning of the 16.2% failed questions into misalignment (28%), out-of-scope (34%), and outdated/biased (38%) is presented without any account of the categorization procedure, number of raters, or inter-rater reliability metric; because the paper attributes the citation variant's advantage to reduced misinterpretation, the absence of these details makes it impossible to isolate prompt effects from annotation artifacts.

    Authors: We agree that the error categorization procedure must be documented. We will revise the Error analysis section to describe the process: the approximately 161 questions were reviewed using explicit decision rules for each category (misalignment = direct contradiction with the Guide; out-of-scope = topic absent from the Guide; outdated/biased = reference to prior versions or subjective framing), with all assignments reached through team discussion and consensus. We will also note the absence of a formal inter-rater reliability statistic as a limitation while explaining how the rule-based approach reduces the risk of annotation artifacts affecting the comparison of prompting techniques. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical measurement on fixed dataset

full rationale

The paper reports direct accuracy measurements (zero-shot 85+% , citation variant 89.1%) on a pre-existing 993-question dataset against ground-truth labels. No equations, fitted parameters, predictions derived from the same data, or self-citation chains appear in the abstract or described methodology. Error clustering (28/34/38%) is a post-hoc summary of observed failures, not a definitional or fitted step. The study is self-contained against external benchmarks (the Scrum Guide and the fixed question set).

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical measurement study; central claim rests on unstated assumptions about dataset quality and error labeling rather than mathematical axioms or free parameters.

pith-pipeline@v0.9.1-grok · 5823 in / 1058 out tokens · 29593 ms · 2026-07-02T19:59:33.937609+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Prompting GPT-5 on Scrum Certification Questions: An Empirical Accuracy Study." pith.science (2026). https://pith.science/paper/2PJRXHQJ

@misc{pith2026260700049,
  author       = {Pith},
  title        = {Pith review of: Prompting GPT-5 on Scrum Certification Questions: An Empirical Accuracy Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2PJRXHQJ}},
  note         = {Machine review of arXiv:2607.00049}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) are increasingly used in Agile Software Development for documentation, coaching, and training. As practitioners adopt these tools to prepare for certifications such as Professional Scrum Master (PSM), a key question is whether LLMs can reliably reason about Scrum, a framework with normative, well-defined rules described in the Scrum Guide (2020). This paper examines how different prompt techniques affect the factual accuracy of LLM responses to Scrum certification-style questions. A dataset of 993 validated PSM-aligned questions was answered by GPT-5 using three techniques: zero-shot, chain-of-thought, and with-source citation. All prompts achieved certification-level accuracy above 85\%, with the citation-based variant performing best (89.1\%) and yielding the lowest error rate. Correct answers concentrated in well-defined topics, such as \emph{Definition of Done}, Events, and Product Backlog Management, and in single-answer multiple-choice items, while multi-select questions and more interpretive areas, such as Scrum Team and Product Value, were less stable. Among questions where at least one prompt failed (16.2\%), errors clustered into misalignment with the Scrum Guide (28\%), content outside its scope (34\%), and outdated or biased interpretations (38\%). Overall, prompt techniques produced modest but consistent improvements, particularly in reducing misinterpretation and version drift, supporting more reliable use of LLMs in Agile learning and certification preparation.

Figures

Figures reproduced from arXiv: 2607.00049 by Ademar Fran\c{c}a de Sousa Neto, Angelo Perkusich, Danyllo Albuquerque, Emanuel Dantas, Jo\~ao Paiva, Kyller Gorg\^onio, Mirko Perkusich, Robson Vilar, Rohit Gheyi.

Figure 1
Figure 1. Figure 1: Overview of the empirical process. Research Questions RQ1. What is GPT-5 accuracy under different prompt techniques? RQ2. What patterns of correct responses emerge across prompts, subject areas, and question formats? RQ3. What error patterns emerge across prompts, subject areas, and question formats? Together, these RQs examine overall factual correctness, variation across topics and question formats, and … view at source ↗
Figure 2
Figure 2. Figure 2: Prompt accuracy and error rate by subject (Top 8). Bars represent mean accuracy per prompt (zero-shot [P1], chain-of-thought [P2], and with-source [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

  1. [1]

    How do scrum methodologies influence the team’s cultural values? a multiple case study on agile teams in nonsoftware industries,

    A. S. Patrucco, F. Canterino, and I. Minelgaite, “How do scrum methodologies influence the team’s cultural values? a multiple case study on agile teams in nonsoftware industries,”IEEE Transactions on Engineering Management, vol. 69, no. 6, pp. 3503–3513, 2022

    work page 2022

  2. [2]

    Sutherland,Scrum

    J. Sutherland,Scrum. Bentang Pustaka, 2018

    work page 2018

  3. [3]

    Enhancing professional employability: The impact of agile methodology training

    R. A. Crisostomo, N. A. Nunez, and G. Lopez-Burga, “Enhancing professional employability: The impact of agile methodology training.” International Journal of Engineering Pedagogy, vol. 14, no. 8, 2024

    work page 2024

  4. [4]

    Competences for the future: A comparative analysis of agile certifications,

    A. Montenegro, “Competences for the future: A comparative analysis of agile certifications,”European Project Management Journal, vol. 9, no. 2, pp. 46–54, 2019

    work page 2019

  5. [5]

    Assessing the capability of llms in solving poscomp questions,

    C. Viegas, R. Gheyi, and M. Ribeiro, “Assessing the capability of llms in solving poscomp questions,”Journal of the Brazilian Computer Society, vol. 31, no. 1, p. 991–1004, Oct. 2025

    work page 2025

  6. [6]

    Assessing gpt’s legal knowledge in japanese real estate transactions exam,

    K. Inoshita, “Assessing gpt’s legal knowledge in japanese real estate transactions exam,” in2024 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT). IEEE, 2024, pp. 149–155

    work page 2024

  7. [7]

    Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code,

    J. Shin,et al., “Prompt engineering or fine-tuning? an empirical as- sessment of large language models in automated software engineering tasks,”arXiv preprint arXiv:2310.10508, 2023

    work page arXiv 2023

  8. [8]

    Which prompting technique should i use? an empirical investigation of prompting techniques for software engineering tasks,

    G. e. a. Santana Jr., E. “Which prompting technique should i use? an empirical investigation of prompting techniques for software engineering tasks,”arXiv preprint arXiv:2506.05614, 2025, preprint; under review

    work page arXiv 2025

  9. [9]

    Scrum Guide and Professional Scrum Certifications,

    Scrum.org, “Scrum Guide and Professional Scrum Certifications,” https: //www.scrum.org/, 2020, accessed: 2026-04-13

    work page 2020

  10. [10]

    The importance of certification in project man- agement in the labor market,

    H. Soroka-Potrzebna, “The importance of certification in project man- agement in the labor market,”Procedia Computer Science, vol. 192, pp. 1934–1943, 2021

    work page 1934

  11. [11]

    Llm-based agents for automating the enhancement of user story quality: An early report,

    Z. Zhanget al., “Llm-based agents for automating the enhancement of user story quality: An early report,” inInternational Conference on Agile Software Development. Springer Nature Switzerland Cham, 2024, pp. 117–126

    work page 2024

  12. [12]

    Exploring human-ai collaboration in agile: Customised llm meeting assistants,

    B. Cabrero-Daniel, T. Herda, V . Pichler, and M. Eder, “Exploring human-ai collaboration in agile: Customised llm meeting assistants,” inInternational Conference on Agile Software Development. Springer Nature Switzerland Cham, 2024, pp. 163–178

    work page 2024

  13. [13]

    Towards llm-augmented multiagent systems for agile software engineering,

    J. A. Chudziak and K. Cinkusz, “Towards llm-augmented multiagent systems for agile software engineering,” in2024 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2024, pp. 2476–2477

    work page 2024

  14. [14]

    Cognitive agents powered by large language models for agile software project management,

    K. Cinkuszet al., “Cognitive agents powered by large language models for agile software project management,”Electronics, vol. 14, no. 1, p. 87, 2024

    work page 2024

  15. [15]

    Integrating llms into agile software development: A 2023 perspective on productivity and code quality,

    R. Modaket al., “Integrating llms into agile software development: A 2023 perspective on productivity and code quality,”Well Testing Journal, vol. 32, no. 2, pp. 130–146, 2023

    work page 2023

  16. [16]

    Agile project management using large language models,

    G. Dhruva, I. Shettigar, S. Parthasarthy, and V . Sapna, “Agile project management using large language models,” in2024 5th International Conference on Innovative Trends in Information Technology (ICITIIT). IEEE, 2024, pp. 1–6

    work page 2024

  17. [17]

    Bridging agility and automation: Enhanc- ing model-driven engineering with llms in scrum,

    L. Samimi and S. K. Rahimi, “Bridging agility and automation: Enhanc- ing model-driven engineering with llms in scrum,” inAgile Model-driven Engineering Workshop. CEUR Workshop Proceedings, 2025

    work page 2025

  18. [18]

    Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers,

    A. O. G ´erardet al., “Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers,”British Journal of Clinical Pharmacology, 2025

    work page 2025

  19. [19]

    Evaluating large language models on the gmat: Implications for the future of business education,

    V . Ashrafimoghariet al., “Evaluating large language models on the gmat: Implications for the future of business education,”arXiv preprint arXiv:2401.02985, 2024

    work page arXiv 2024

  20. [20]

    Machine vs machine: Large language models (llms) in applied machine learning high-stakes open-book exams,

    K. Quilleet al., “Machine vs machine: Large language models (llms) in applied machine learning high-stakes open-book exams,”Revista de Educaci´on a Distancia (RED), vol. 24, no. 78, 2024

    work page 2024

  21. [21]

    Breaking the silence: the threats of using llms in software engineering,

    J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” inProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, 2024, pp. 102–106

    work page 2024