{"id":"038e7ff7-3bca-48cb-962e-026314cf4f97","arxiv_id":"2607.00048","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Gemini 3 Flash achieved the highest accuracy on PSM I-style questions among three tested LLMs, with low intra-model variability and systematic error patterns by question format and topic.","lead":"This paper evaluates three large language models on 993 Scrum certification-style questions using multiple prompting strategies and repeated runs. A smart generalist might read it to understand current AI reliability for domain-specific professional exams in software engineering.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Question sourcing/selection process is underspecified and could introduce systematic bias in the reported model ranking.","rationale":"The reader's weakest_assumption is exactly the load-bearing point. Because the abstract alone cannot confirm unbiased question selection, the UNVERDICTED verdict is appropriate; the full text would need to supply the missing sourcing details or a topic-balance check before the ranking claim can be treated as robust.","tokens_in":1830,"tokens_out":360,"duration_ms":20192,"concrete_test":"From the full paper, extract the per-topic question counts (or the table/appendix that lists them) and compare their proportions to the official PSM I topic weights published by Scrum.org; if the paper's distribution deviates by more than 15 percentage points on any major topic, recompute the model accuracies after re-weighting or subsampling to match the official distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim (Gemini 3 Flash highest accuracy, followed by GPT-5 mini then DeepSeek, with low intra-model variance) depends on the 993 questions constituting an unbiased probe of PSM I capabilities. The abstract states only that the questions are 'aligned with' the PSM I format and that performance varies by topic and format; it supplies no information on whether the set was drawn from official Scrum.org materials, generated by the authors, filtered post-hoc, or balanced against the official topic weights. If the authors' curation favored topics where one model already performs well (e.g., Artifacts, Empiricism) or used prompts that interact with model-specific training data, the observed ordering is not guaranteed to reflect genuine capability differences. The reported stability across repeated runs addresses only stochasticity, not selection bias.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper empirically compares three LLMs (GPT-5 mini, Gemini 3 Flash, DeepSeek Chat 3.2) on 993 Scrum certification-style questions aligned with the PSM I format. It evaluates performance under zero-shot, chain-of-thought, and source-grounded prompting, with repeated runs to measure intra-model stability. Results are broken down by question format (single-answer MCQ highest accuracy; multi-select and T/F more error-prone) and topic (stronger on Artifacts/Empiricism/Product Value; weaker on Scrum Values/Self-Managing Teams/Stakeholders), plus a qualitative review of systematic error types (overgeneralization, restrictive wording, compound distractors, market vs. strict Scrum conflicts). The headline finding is that Gemini 3 Flash leads in accuracy, followed by GPT-5 mini then DeepSeek, with low variability across runs.","tokens_in":1984,"tokens_out":617,"duration_ms":26689,"significance":"If the central claims hold after addressing dataset details, the work supplies a useful benchmark for LLM behavior on normatively strict, domain-specific certification tasks in software engineering. Strengths include the repeated-execution design for stability assessment and the topic/format/error-pattern breakdowns, which move beyond aggregate accuracy to identify where models are fragile. These elements could inform both LLM-assisted Scrum training tools and broader questions about LLM reliability on rule-based professional knowledge.","major_comments":[{"comment":"Methods / dataset construction: the 993 questions are described only as 'aligned with' PSM I format, with no details on sourcing (official Scrum.org materials vs. author-generated), selection criteria, post-hoc filtering, or balance against official topic weights. This is load-bearing for the model ranking (Gemini 3 Flash highest) because curation bias could systematically favor topics or phrasings where one model already excels.","section":"Methods"},{"comment":"Results: accuracy differences and the claim of 'clear differences among models' are reported without statistical tests (e.g., proportion tests or paired comparisons across the repeated executions) or confidence intervals that account for both stochasticity and question sampling. This leaves the ordering vulnerable to sampling variation in the 993-item set.","section":"Results"}],"minor_comments":[{"comment":"The description of the 'source-grounded' prompting strategy would benefit from an explicit example showing what source material is supplied and how it is formatted, to support reproducibility.","section":"Prompting Strategies"},{"comment":"Tables reporting per-topic or per-format accuracies should include the number of questions per cell and, ideally, variability measures from the repeated runs.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim rests on an underspecified dataset whose construction details are not visible even in the full manuscript; this is a scope issue that can be fixed by expansion but currently prevents verification of the ranking."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript accordingly to improve transparency and statistical rigor.","responses":[{"response":"We agree that the Methods section requires greater detail on dataset construction to permit evaluation of potential curation effects. The 993 questions were drawn from publicly available PSM I practice examinations and study resources that follow the official Scrum.org PSM I format and topic distribution. Questions were selected to achieve approximate proportionality with the official PSM I topic weights, with deduplication as the only post-selection step. We will add a dedicated Methods subsection that specifies sourcing, selection criteria, and a comparison table of topic coverage versus official weights. This addition will allow readers to assess whether any observed model ordering could be attributable to dataset composition.","revision_made":"yes","referee_comment":"[Methods] Methods / dataset construction: the 993 questions are described only as 'aligned with' PSM I format, with no details on sourcing (official Scrum.org materials vs. author-generated), selection criteria, post-hoc filtering, or balance against official topic weights. This is load-bearing for the model ranking (Gemini 3 Flash highest) because curation bias could systematically favor topics or phrasings where one model already excels."},{"response":"We accept that formal statistical support is needed to substantiate the reported model ordering. Although the five repeated executions per condition already demonstrate low intra-model variance and consistent rank order, we will incorporate McNemar's tests for paired accuracy comparisons across models on identical question sets and bootstrap confidence intervals that reflect both model stochasticity and the finite sample of 993 questions. These results, together with updated tables, will be added to the Results section.","revision_made":"yes","referee_comment":"[Results] Results: accuracy differences and the claim of 'clear differences among models' are reported without statistical tests (e.g., proportion tests or paired comparisons across the repeated executions) or confidence intervals that account for both stochasticity and question sampling. This leaves the ordering vulnerable to sampling variation in the 993-item set."}],"tokens_in":1573,"tokens_out":451,"duration_ms":25037,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Gemini 3 Flash came out ahead of GPT-5 mini and DeepSeek Chat 3.2 on these 993 Scrum-style questions, with all three models showing low run-to-run variation. The paper also breaks results down by topic and question format and catalogs recurring error types such as overgeneralization and clashes with strict Scrum definitions.\n\nThe useful parts are the repeated executions for stability, the topic-level splits (stronger on Artifacts and Empiricism, weaker on Scrum Values), and the format comparison (single-answer multiple choice easier than multi-select or true/false). Those pieces give concrete data on where the models struggle in a certification-style setting. The qualitative error notes add a layer beyond raw accuracy.\n\nThe soft spot is exactly the one flagged in the stress test: the abstract gives no information on how the 993 questions were assembled, whether they follow official PSM I topic weights, or if they were filtered after the fact. Without that, the model ordering could reflect selection choices rather than capability. The stability checks only address randomness, not curation bias. No statistical tests are mentioned either.\n\nThis is narrow-scope empirical work aimed at people who evaluate LLMs for professional training or software engineering education tasks. A reader looking for topic-specific LLM behavior on Scrum material could extract value from the breakdowns. It has enough structure and data to go to peer review, but the methods section on question sourcing and any significance testing will need to be filled in.","headline":"Gemini 3 Flash leads on the PSM I questions with low variance, but the 993-question set's origin is too vague to trust the ranking without more detail.","tokens_in":2485,"tokens_out":375,"would_cite":false,"duration_ms":20995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Gemini 3 Flash scores highest on 993 Scrum certification questions, followed by GPT-5 mini and DeepSeek Chat 3.2 with low variability.","keywords":["Large Language Models","Scrum Certification","Question Answering","Performance Comparison","Prompt Engineering","Software Engineering","Error Analysis"],"falsifier":"Running the same models on an official set of PSM I practice questions released by Scrum.org and finding reversed accuracy rankings or high variability across runs would falsify the reported differences and stability.","tokens_in":2756,"feed_emoji":"📊","tokens_out":677,"duration_ms":29039,"temperature":0.7,"pith_summary":"This paper tests three large language models on 993 questions formatted like the Professional Scrum Master I certification exam. It applies three different prompting approaches and repeats the tests to measure consistency. The evaluation covers accuracy by model, question type, and Scrum topic, plus a review of why models get answers wrong. A reader would care because these tasks require precise application of defined rules rather than general knowledge, revealing how ready LLMs are for professional certification support in software engineering.","feed_headline":"Gemini 3 Flash scores highest on 993 Scrum certification questions","feed_subtitle":"The model edges out GPT-5 mini and DeepSeek with low variability across prompting methods and question types.","key_machinery":"Systematic comparison of three LLMs under zero-shot, chain-of-thought, and source-grounded prompting on a fixed set of 993 PSM I-aligned questions, with analysis of accuracy, stability, topic/format breakdowns, and qualitative error patterns.","core_discovery":"Gemini 3 Flash achieved the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2. Intra-model variability was low across all prompting strategies. Models performed best on single-answer multiple-choice questions and on topics with clear normative definitions like Artifacts and Empiricism, while struggling more with multi-select questions, True/False items, and topics like Scrum Values and Self-Managing Teams. Errors followed patterns such as overgeneralization and misalignment with strict Scrum definitions rather than random mistakes.","pith_inferences":["These results could guide the development of domain-specific fine-tuning for agile and Scrum knowledge.","Similar evaluations on other certification exams might show whether the model ranking holds across software engineering domains.","The topic-specific weaknesses could inform targeted improvements in how LLMs are trained on normative frameworks.","Low variability supports using these models for generating practice exams with predictable behavior."],"forward_implications":["Model selection can meaningfully affect performance in Scrum certification preparation tasks.","Low intra-model variability suggests consistent outputs suitable for repeated testing scenarios.","Performance gaps by topic indicate that some Scrum areas are more aligned with current LLM training data than others.","Systematic error patterns point to specific challenges in handling restrictive wording and market vs. official definition conflicts.","Highest accuracy on single-answer questions suggests simpler formats are more reliable for current models."],"fun_headline_variants":["Gemini 3 Flash Achieves Highest Accuracy on Scrum Questions","Low Variability Across All LLM Scrum Certification Tests","Models Stronger on Artifacts Than Scrum Values Topics","Single-Answer Questions Outperform Multi-Select for LLMs","Systematic Errors Mark LLM Responses to Strict Scrum Items"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The selected 993 questions represent the PSM I assessment without bias and the three prompting strategies test model capabilities fairly.","fun_headline_variants_meta":{"raw":{"variants":["Gemini 3 Flash Achieves Highest Accuracy on Scrum Questions","Low Variability Across All LLM Scrum Certification Tests","Models Stronger on Artifacts Than Scrum Values Topics","Single-Answer Questions Outperform Multi-Select for LLMs","Systematic Errors Mark LLM Responses to Strict Scrum Items"]},"model":"grok-4.3","cost_usd":0.004602,"raw_usage":{"total_tokens":2347,"prompt_tokens":797,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":46024500,"prompt_tokens_details":{"text_tokens":797,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1474,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":797,"tokens_out":76,"duration_ms":11463,"temperature":1.0,"reasoning_tokens":1474,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T20:02:13.444218+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the same models on an official set of PSM I practice questions released by Scrum.org and finding reversed accuracy rankings or high variability across runs would falsify the reported differences and stability.","supporting_citations":[],"review_version":1}