{"id":"28bdc6f2-dfd1-4481-ae67-be987f7a8626","arxiv_id":"1908.07783","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic review of 17 primary studies finds 13 distinct approaches that apply creativity to requirements elicitation in agile development, with moderate and mostly case-based evidence.","lead":"This paper searched 1451 academic articles and identified 17 studies covering 13 approaches that use creative techniques, like brainstorming and design thinking, to collect software requirements in agile projects. It maps what these approaches promise, what evidence supports them, and what limitations teams should expect.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'moderate evidence' conclusion conflicts with the paper's own Kitchenham-based scoring: all primary studies would score poor (0), yet RQ2 aggregates them to 'moderate'.","rationale":"The reader's weakest assumption concerns the search and selection process: the simplified query in Section 3.3, the subjective inclusion criterion in Section 3.4, and the lack of an excluded-study list. That is a legitimate concern about coverage. My concern is different and, I think, more directly load-bearing: the paper's own evidence-grading framework, which it explicitly adopts at Table 23, classifies every primary study as 'poor', yet the review then converts a cross-study project count into an overall 'moderate' rating. This is an internal inconsistency in the argument for the headline claim, not merely a disagreement about search coverage. Even if the search were perfect, the abstract's 'moderate evidence' and 'successfully implemented in real software projects' wording would still need justification against the paper's stated per-study scores. The review has genuine strengths: it follows a recognized protocol, documents its quality assessment in detail, and is honest about per-study limitations. The concern is not that the authors are hiding the poor per-study evidence; they explicitly acknowledge it. The problem is that the synthesis then discards that acknowledged evidence class without a defended aggregation rule. This reinforces the CONDITIONAL verdict rather than overturning it: the review can be accepted if the authors either provide a defensible aggregation method or soften the evidence claims. I therefore keep the reader's conditional recommendation, with the condition sharpened to include an evidence-scoring reconciliation.","tokens_in":32640,"tokens_out":4176,"duration_ms":40740,"concrete_test":"Re-compute RQ2 at the level of the 13 unique proposals using the paper's own Kitchenham criteria. For each proposal (S1, S2, S3, the S4 group, the S10 group, the S13 group, etc.), count the number of distinct projects or within-company data sets reported, assign poor/fair/good/excellent according to Table 23, and report the distribution. If, as the paper itself implies, all 13 proposals score poor (0), then the 'moderate empirical evidence' conclusion in Section 5.1 and the abstract is unsupported and must be downgraded. Separately, verify whether any single proposal reaches the 10-project threshold needed for 'fair'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 (RQ2) is the load-bearing point for the abstract's claim that enhanced creativity in requirements elicitation 'can be successfully implemented in real software projects' and that the empirical evidence is 'moderate'. The paper cites Kitchenham et al.'s evidence criteria (Table 23), under which fewer than 10 projects per data set scores 0 ('poor'), 10-20 scores 0.33 ('fair'), and so on. The authors then state that, if applied per study, 'all the selected studies would have received a score of 0' and the evidence would be 'poor', because most approaches were tested in only one or two projects. Instead of reporting that result, they replace per-study grading with a cross-study aggregate count: 'between 10 and 20 projects have provided some sort of evidence', and from this they conclude that the overall evidence is 'moderate'. This is an unstated change in the unit of analysis. Counting distinct projects across 13 heterogeneous proposals cannot establish that any single approach has moderate evidence; it only indicates that the field has some initial trials. The central claim therefore rests on an aggregation rule that is not defended and that contradicts the paper's own cited criteria. Either the abstract should be softened to 'promising but mostly weak/early evidence', or the review needs an explicit aggregation rationale with per-approach evidence counts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a systematic literature review, following the Kitchenham/Biolchini protocol, of approaches that leverage creativity in requirements elicitation (RE) within agile software development (ASD). From 1451 initial search results, the authors selected 17 primary studies corresponding to 13 unique proposals. The review classifies the proposals by type (technique, process, framework, tool, model), assesses their quality using rigor/credibility/relevance criteria, and answers five research questions covering existing techniques, empirical evidence, benefits and limitations, suitable project contexts, and agile methodologies used. The authors conclude that enhanced creativity in RE can be successfully implemented in real software projects, that the empirical evidence is moderate, that mobile and web user-interface projects are good candidates, and that creativity alone is insufficient without user engagement and a supportive context.","tokens_in":32918,"tokens_out":7449,"duration_ms":64525,"significance":"If the evidence-related claims are repaired, this is a useful contribution to the software engineering literature. The paper appears to be the first systematic review combining creativity, requirements elicitation, and agile development, and it makes its process transparent: the search query, data sources, quality rubric, per-study summaries, and a full list of primary studies are all included. The map of 13 proposals and the observation that design-thinking-derived methods dominate the area are valuable for researchers and practitioners. The main scientific conclusion, however, the strength of empirical evidence, is currently overstated because of an unsupported aggregation in the answer to RQ2; this needs to be fixed before the paper can be accepted.","major_comments":[{"comment":"The answer to RQ2 is internally inconsistent. The authors state that if each selected study were graded with Kitchenham et al.'s criteria (Table 23), 'all the selected studies would have received a score of 0' and the evidence would be 'poor', because most approaches were tested in only one or two projects. They then replace this per-study grading with a cross-study count of projects, 'between 10 and 20 projects', and conclude that the overall evidence is 'moderate'. This is an unstated change in the unit of analysis: counting distinct projects across 13 heterogeneous proposals does not establish that any single approach has moderate evidence. Since the abstract's central claim ('these approaches provide evidence that enhanced creativity ... can be successfully implemented in real software projects') and the 'moderate evidence' statement rest on this aggregation, the authors should either soften the conclusion to early, promising, or weak evidence, or provide an explicit aggregation rationale with per-approach evidence counts.","section":"Section 5.1, RQ2; Table 23"},{"comment":"The two RQ2 evidence tables contradict each other, and this matters because RQ2 is load-bearing for the abstract claim. Table 13 codes S12 as not an experiment, but Section 4.1.10 describes S12 as providing two controlled experiments and Table 22 lists S12 as a laboratory experiment. Table 13 codes S3 as not an experience report, but Table 22 lists S3 as an experience report; Table 13 codes S9 as an experience report, but Table 22 omits S9. The percentages in Table 22 are also not consistent with the listed study counts (e.g., 3 of 17 is 18%, not 23%). The evidence synthesis should be recomputed from a single reconciled coding of the 17 primary studies.","section":"Section 4.2, Table 13 vs Section 5.1, Table 22"},{"comment":"The completeness and reproducibility of the review are not fully established. Section 3.3 explains that the query was deliberately simplified by dropping a fourth keyword block and agile synonyms such as 'agility', feature-driven development, and DSDM 'to keep the query as simple as possible'; the authors do not quantify how many additional relevant results these omitted terms would have returned. Section 3.4 uses a subjective inclusion criterion ('it is obvious that the study presents some kind of creative approach') and does not report how many reviewers applied it or whether inter-rater agreement was checked. The authors should state which search engines imposed term limits, report the screening procedure in more detail, and perform or justify a sensitivity check on the omitted synonyms; otherwise the claim that the 17 primary studies define the state of the art is not fully supported.","section":"Sections 3.3 and 3.4"},{"comment":"The abstract and Section 5.2 say that agile methodologies such as Scrum, Extreme Programming, or rapid-modelling-based methodologies are 'preferred' for introducing creativity into requirements elicitation, but Table 16 does not support a preference: only S1 and S8 mention Scrum, only S5 mentions XP, and four studies mention agile model-driven development, while the majority of the 17 primary studies specify no methodology. The wording should be changed to reflect that a minority of proposals name a specific methodology, and that among those, agile model-driven development is the most frequent.","section":"Section 5.2 and Table 16"}],"minor_comments":[{"comment":"The caption 'Procedure for execution os the systematic review' contains a typo: 'os' should be 'of'.","section":"Section 3.4, Figure 6"},{"comment":"The creative technique is spelled 'PICL' in Table 21 but 'PCIL' in the text; please standardize the acronym.","section":"Section 5.1, RQ1, Table 21"},{"comment":"The heading for S6 names the authors as 'Percival, Braz and Adilson', but Appendix I lists the authors as Lucena, Braz, Chicoria, and Tizzei; the heading should be corrected.","section":"Section 4.1.6 and Appendix I"},{"comment":"The percentages in Tables 24 and 25 are not always derivable from the cited study lists (for example, 'Requirements obtained are seen as less useful' cites only S1, which is 1/17, not 8%); please reconcile the percentages and the study lists.","section":"Tables 24 and 25"},{"comment":"Figure 11 is captioned 'ADDAM Process detailed description [S3]', but ADDAM is the process described in S2, not S3; the caption appears to be mislabeled.","section":"Section 4.1.3, Figure 11"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the review effort is genuine, but the RQ2 evidence claim needs to be reworked before publication. The main worry is not that the conclusion is wrong, but that the aggregation from per-study 'poor' to overall 'moderate' is not defensible as written. If the authors soften the claim or add a transparent aggregation rationale, and reconcile the evidence tables, I would be willing to see a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, it appears to be the first systematic review that specifically combines creativity in requirements elicitation with agile development, and as a map of the area it is genuinely useful: 17 primary studies, 13 distinct proposals, clean synthesis tables, and a reasonably careful quality assessment. Second, the headline conclusion—that the empirical evidence is 'moderate'—does not survive contact with the paper's own evidence criteria. They cite Kitchenham's scoring (Table 23), note that every single study would score 0 ('poor') because most were tested in only one or two projects, and then, instead of reporting that, they count distinct projects across all 13 heterogeneous approaches and call the aggregate 'moderate.' That is an unstated change in the unit of analysis. It cannot establish that any individual approach has moderate evidence; it only shows the field has had some initial trials. The abstract goes further and says these approaches 'provide evidence that enhanced creativity in requirements elicitation can be successfully implemented in real software projects'—which is plausible as a qualitative statement but not supported by the Kitchenham-based grade they claim.\n\nWhat the paper does well: the protocol is recognizable (Kitchenham/Biolchini), the search is documented with numbers at each stage, the per-study summaries are readable, and the authors are transparent about several limitations—small corpus, mostly single-case evidence, simplified query. The discussion of benefits and limitations is honest, and the distinction between the 13 proposals and their underlying creative techniques is a reasonable organizing principle.\n\nSoft spots, in proportion. The RQ2 aggregation flaw is the main one and is load-bearing for the abstract's evidence claim. The inclusion criteria are also vague: 'it is obvious that the study presents some kind of creative approach' invites subjectivity, and there is no inter-rater reliability check or list of excluded studies. The search was deliberately simplified (agile synonyms dropped), which is handled openly but still means the 17 primary studies may not be complete. These are not fatal for a niche SLR, but they should be fixed in revision.\n\nWho this is for: researchers entering this intersection and practitioners who want a quick catalogue of techniques like creativity workshops, mind mapping, and design thinking variants applied to agile elicitation. It deserves a serious referee—the synthesis itself is new and the flaws are repairable. My recommendation: send it to peer review with a request for major revision focused on RQ2, and ask the authors to either soften the evidence claim to 'promising but mostly early/weak' or explicitly justify a cross-study aggregation rule with per-approach evidence counts.","headline":"Useful niche SLR that overclaims its evidence grade: the 'moderate' RQ2 conclusion contradicts the paper's own Kitchenham scoring, but the map of 13 approaches is worth having.","tokens_in":33345,"tokens_out":1206,"would_cite":true,"duration_ms":14845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A systematic review of 1,451 studies identifies 13 distinct proposals for using creativity in agile requirements elicitation and concludes they can be implemented successfully in real software projects.","keywords":["requirements elicitation","agile software development","creative thinking","systematic literature review","design thinking","gamification","mind mapping","empirical evidence"],"falsifier":"Run the same review with an expanded query that adds back the dropped terms—agility, feature-driven development, DSDM, and the information-systems keyword block—and count how many additional unique approaches emerge; if the set grows materially or changes the synthesis, the claim that 13 unique proposals represent the state of the art would need revision.","tokens_in":32388,"feed_emoji":"💡","tokens_out":6040,"duration_ms":58263,"temperature":0.7,"pith_summary":"The paper asks whether creativity techniques can be systematically brought into requirements elicitation inside agile software development. By searching six digital libraries with a three-part query covering creativity, requirements, and agile terms, it identified 1,451 candidate studies, 17 primary studies, and 13 distinct proposals. The review's central claim is that these proposals are not merely theoretical: they have been applied in industry and laboratory settings and can improve problem understanding, stakeholder engagement, and how well software meets business needs. It also finds that agile methods such as Scrum, Extreme Programming, and rapid modeling are the preferred context, and that user-interface-heavy projects such as mobile and web applications are especially good candidates. The paper is careful to say the empirical evidence is moderate rather than strong, because most approaches have been tested in only one or two projects.","feed_headline":"13 creative approaches bring real results in agile requirements","feed_subtitle":"Review of 1,451 studies maps where creativity boosts elicitation and where the evidence stays thin.","key_machinery":"The machinery of the paper is the systematic review protocol itself. It starts from a Boolean query built from three keyword domains—creativity terms, requirements terms, and agile terms—run across six digital libraries. After title and abstract screening, duplicates were removed, leaving 17 primary studies; quality was then scored on rigor, credibility, and relevance using a structured checklist; and the extracted data were synthesized into tables that map each proposal's type, research method, reported benefits and limitations, suitable contexts, and agile methodology. These synthesis tables, rather than any single creativity technique, carry the argument: they turn 17 heterogeneous papers into a countable set of 13 unique proposals and a set of cross-study findings.","core_discovery":"The central discovery is a mapped and synthesized landscape: 13 unique proposals exist that leverage creativity in requirements elicitation within agile software development. These proposals span several types: processes, frameworks, techniques, tools, and one model. The evidence drawn from the primary studies indicates that creativity can be implemented successfully in real software projects, mostly in large organizations and in user-interface-oriented development, and that reported benefits such as facilitated problem understanding and better meeting business needs recur across studies. The paper also concludes that creativity alone is not sufficient: generating new requirements also requires user engagement and a context with flexibility, time, and resources.","pith_inferences":["A direct implication the review leaves implicit: creative elicitation is most valuable in early discovery or sprint-zero phases, where openness to initially 'less useful' requirements can be tolerated before the team converges on a solution.","If engagement and context are the limiting factors the review names, the next testable extension is pairing creativity techniques with engagement mechanics such as gamification or participatory co-design in the same project and measuring requirement novelty against a control group.","The absence of any explicit creativity-level classification suggests a comparative study could classify each proposal by the four-C model and test whether approaches targeting everyday creativity behave differently from those seeking professional-level creative output.","Because the evidence base is mostly single-case action research and case studies, a multi-project replication trial—same technique, several organizations, identical outcome measures—would be a more decisive test than another systematic review."],"forward_implications":["The 13 proposals give practitioners a concrete menu of creativity techniques—workshops, mind maps, prototyping, goal sketching, gamification—that can be inserted into agile requirements work rather than treated as abstract theory.","Teams building mobile or web applications are the most promising early adopters, since the reviewed evidence clusters around user-interface-intensive development.","Agile contexts built on Scrum, Extreme Programming, or rapid modeling appear to be the natural home for creative elicitation, though most proposals avoid binding themselves to one method.","Reported benefits concentrate on problem understanding and meeting business needs, while the main costs are training, open-minded users, and possible team reorganization.","Because most approaches have only one or two project evaluations, the strength of evidence should be read as moderate: enough to justify adoption experiments, not enough for blanket prescriptions."],"supporting_citations":[{"why":"Supplies the systematic-review methodology that the paper follows for planning, execution, and result analysis.","marker":"[42]"},{"why":"Provides the four-stage review process and template used to structure planning, execution, result analysis, and packaging.","marker":"[8]"},{"why":"Supplies the study-selection procedure used to reduce 1,451 search results to the relevant set of studies.","marker":"[67]"},{"why":"Provides the empirical-evidence scoring criteria used to judge the strength of evidence in the primary studies.","marker":"[43]"},{"why":"Supplies the quality-assessment questions the review adapts for rigor, credibility, and relevance.","marker":"[44]"},{"why":"Prior mapping study on creativity in requirements engineering that motivates focusing on the elicitation phase.","marker":"[49]"},{"why":"Prior work on creativity in agile systems development that the review positions itself against.","marker":"[17]"},{"why":"Establishes the recognized benefits of agile methods that the review's motivation questions regarding innovation.","marker":"[25]"}],"fun_headline_variants":["Agile requirements get creative: 13 proven approaches","Creativity boosts agile requirements: 13 methods, real results","13 creative methods for agile requirements: benefits and limits","In agile requirements, creativity helps but needs more than that"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole synthesis rests on the assumption that the search and selection process captured the relevant literature; the authors deliberately kept the query simple by dropping a fourth keyword block and agile synonyms such as agility, feature-driven development, and DSDM, and the inclusion criterion was the subjective judgment that a study obviously presented some kind of creative approach.","fun_headline_variants_meta":{"raw":{"variants":["Agile requirements get creative: 13 proven approaches","Creativity boosts agile requirements: 13 methods, real results","13 creative methods for agile requirements: benefits and limits","In agile requirements, creativity helps but needs more than that"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1358,"prompt_tokens":955,"completion_tokens":403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":571,"tokens_out":403,"duration_ms":4808,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:55:43.513470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same review with an expanded query that adds back the dropped terms—agility, feature-driven development, DSDM, and the information-systems keyword block—and count how many additional unique approaches emerge; if the set grows materially or changes the synthesis, the claim that 13 unique proposals represent the state of the art would need revision.","supporting_citations":[{"cited_title":"Keele, UK, Keele University 33(2004), 1–26 (2004)","cited_arxiv_id":null,"evidence_quote":"Supplies the systematic-review methodology that the paper follows for planning, execution, and result analysis."},{"cited_title":"Software Quality Journal 16(2), 237–261 (2008)","cited_arxiv_id":null,"evidence_quote":"Supplies the study-selection procedure used to reduce 1,451 search results to the relevant set of studies."},{"cited_title":"Empirical Software Engineering 13(1), 97–121 (2008)","cited_arxiv_id":null,"evidence_quote":"Provides the empirical-evidence scoring criteria used to judge the strength of evidence in the primary studies."},{"cited_title":"In: Guide to advanced empirical software engineering, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the quality-assessment questions the review adapts for rigor, credibility, and relevance."},{"cited_title":"In: Pro- ceedings of the 27th Annual ACM Symposium on Applied Com- puting, pp","cited_arxiv_id":null,"evidence_quote":"Prior mapping study on creativity in requirements engineering that motivates focusing on the elicitation phase."},{"cited_title":"In: Information Systems– Creativity and Innovation in Small and Medium-Sized Enter- prises, pp","cited_arxiv_id":null,"evidence_quote":"Prior work on creativity in agile systems development that the review positions itself against."},{"cited_title":"Information and software technol- ogy 50(9-10), 833–859 (2008)","cited_arxiv_id":null,"evidence_quote":"Establishes the recognized benefits of agile methods that the review's motivation questions regarding innovation."}],"review_version":1}