Pith. sign in

REVIEW 4 major objections 5 minor 18 references

GenAI-Enabled Backlog Grooming in Agile Software Projects: An Empirical Study

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A GenAI assistant can automate Agile backlog grooming at 100% precision while cutting time-to-completion by about 45 percent.

desk verdict Plausible prototype and useful qualitative data, but the quantitative claims are undermined by an arithmetic inconsistency (Table 1 implies 51 issues, text says 30) and by overstating human-filtered precision as AI performance. read the letter →

arxiv 2507.10753 v1 pith:FPK53U2N submitted 2025-07-14 cs.SE

Kasper Lien Oftebro , Anh Nguyen-Duc , Kai-Kristian Kemell This is my paper
classification cs.SE
keywords automatedbackloggroominggenerativeAIduplicatedetectionvectorembeddingscosinesimilaritylargelanguagemodelsAgilesoftwaredevelopmenthuman-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a generative-AI assistant can automate a core piece of Agile backlog maintenance—finding and resolving duplicate issues—with perfect precision and at roughly half the manual time cost. The authors built a Jira plug-in that embeds backlog items as vectors, flags near-duplicates by cosine similarity, and asks GPT-4o to propose merges, deletions, or new issues, with every suggestion requiring human confirmation. In a controlled study with five senior practitioners and a 30-issue backlog containing 41 true duplicate pairs, tool-assisted groomers found more true duplicates (12–21) than manual-only groomers (7–12), all rejected the tool's false positives, and time per detected duplicate fell by about 45%. The result matters because cluttered backlogs are a known drag on Agile delivery, and a controllable assistant that widens the net for human review points to a concrete path for reducing that drag.

What carries the argument

The load-bearing mechanism is the Jira plug-in's three-stage pipeline: text-embedding-3-large converts each backlog issue into a dense vector; cosine similarity between issue vectors flags candidate duplicates that exceed a similarity threshold; and GPT-4o generates contextual recommendations to merge, delete, or create issues. A tabular user interface presents these suggestions so users can inspect, accept, modify, or reject them, and only accepted actions are written back to Jira. The evaluation machinery is a ground-truth set of 41 duplicate pairs produced by the authors' thorough manual review, against which confusion-matrix metrics (precision, recall, accuracy, F1) and time-to-completion are computed.

What would settle it

Run the same tool on a backlog whose true duplicate pairs are labeled independently by at least two raters with reported inter-rater agreement. If tool-assisted sessions then produce any accepted false positive, or if the time-per-duplicate advantage disappears when evaluators are blind to the ground-truth list, the paper's central claim would be undercut. A quick numerical check already available in the paper: the auto-mode confusion matrix (35 true positives, 8 false positives) gives about 81% precision, so the 100% figure is entirely a property of participants rejecting every false positive—an experiment where some false positives are accepted would falsify the claim that the interactive workflow guarantees perfect precision.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that combining semantic embeddings with a large language model in a human-supervised loop makes backlog grooming both faster and more accurate than doing it by hand. In the experiment, the tool's auto-mode—accepting every model suggestion—found 35 of 41 true duplicate pairs while making 8 false positives; in the interactive sessions, participants rejected all 8 false positives, giving every tool-using tester a precision of 100%. The same practitioners, when working manually, detected only 7 to 12 duplicates in 24–25 minutes, whereas tool-assisted sessions detected 12 to 21 duplicates in 11–20 minutes, reducing the average time per duplicate from 2:54 to 1:35 (a 45.38% decrease). The authors read this as evidence that GenAI and vector embeddings form a viable path for practical backlog management, provided that human judgment remains the final arbiter.

Load-bearing premise

The entire comparison rests on the authors' private ground-truth list of 41 duplicate pairs in a 30-item backlog; if that manual labeling is incomplete, biased, or inconsistent with what another expert would produce, every reported precision, recall, accuracy, and F1 number shifts, and the 100-percent-precision result may be an artifact of the small, self-defined test set.

Editorial extensions

If this is right

  • In interactive use, teams can expect zero false-positive duplicate suggestions: all eight false positives generated by the auto-mode engine were rejected by every tester.
  • Tool-assisted grooming roughly doubles true-duplicate detection over manual grooming in the same or less wall-clock time; for example, tester #11 rose from 12 to 20 true positives while time halved from 24 to 12 minutes.
  • Time per detected duplicate drops about 45%, from an average of 2:54 to 1:35, implying that backlog grooming can be done in less than half the effort.
  • User experience and adoption-intent ratings were consistently high (7–9/10) for the tool versus low (2–4/10) for manual grooming, suggesting the human-in-the-loop design is acceptable to senior practitioners.
  • Because auto-mode recall is modest (about 29–51% on the 41-pair ground truth), the tool is best positioned as a triage assistant that widens the net for human review, not as a standalone replacement for grooming.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 100% precision headline is best understood as a property of the human-in-the-loop workflow, not of the underlying model: the raw auto-mode precision on the same data is about 81% (35 true positives out of 43 positives).
  • The same embedding-plus-LLM pipeline could plausibly be extended to other backlog hygiene tasks—flagging outdated items, splitting oversized stories, or detecting missing requirements—and the accept/reject interface would give the same audit trail for trust.
  • Because manual scanning effort grows with backlog size while embedding search scales nearly linearly with the number of stored vectors, the 45% time saving observed on a 30-item backlog may become larger on real-world backlogs with hundreds or thousands of items; a scale-up experiment would test this.
  • The absence of inter-rater agreement on the 41-pair ground truth means the absolute recall numbers are not yet comparable across studies; publishing the labeled backlog as a benchmark would let other teams calibrate their own grooming tools against the same yardstick.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a Design Science study of a GenAI-based Jira plug-in that embeds backlog issues with text-embedding-3-large, detects duplicates via cosine similarity, and uses GPT-4o to propose merges, deletions, or new issues. Five experienced practitioners performed manual, tool-assisted, or tool-only grooming sessions on a labeled Jira backlog, and the authors report precision, recall, F1, accuracy, time-to-completion, and subjective UX ratings. The headline claims are that AI-assisted backlog grooming achieved 100% precision and reduced time-to-completion by about 45% relative to manual grooming. The manuscript also discusses limitations related to small sample size, domain transfer, and labeled-data scope.

Significance. If its claims were fully supported, the paper would be a useful empirical demonstration of LLM-assisted backlog grooming for agile project management, an area with relatively little prior work. The concrete Jira plug-in artifact, the inclusion of experienced practitioners, the use of multiple quantitative metrics, and the explicit discussion of user experience are genuine strengths. However, the current evidence is primarily exploratory: the ground-truth label set is private and unaudited, the sample is very small, and two headline numbers (100% precision and the 45% time reduction) are partly artifacts of the experimental design and normalization choices. The underlying tool and study may be salvageable, but the claims as written are not yet supported.

major comments (4)
  1. [§4.1 and Table 1] The data in Table 1 are internally inconsistent with the stated backlog size. Section 4.1 describes a Jira project of 30 known issues, but every row of Table 1 sums to 1275 pairwise comparisons (e.g., #8 Manual: 8+1+33+1233=1275; #9 Auto: 21+0+20+1234=1275), which is C(51,2). With 30 issues there are only 435 possible pairs, so the TN and FN columns cannot be correct; with 51 issues, the §4.1 description and all derived rates are wrong. Because every precision, recall, accuracy, and F1 value depends on these totals, the headline comparisons are not reproducible until the authors reconcile the issue count and recompute the metrics consistently.
  2. [§5, Table 1, and §6.1] The 100% precision attributed to 'AI-assisted backlog grooming' is a property of the human-in-the-loop protocol, not of the AI component. Section 5 reports that the tool in auto mode obtained 35 true positives, 6 false negatives, and 8 false positives, which is an AI-only precision of 35/43 ≈ 81.4%; the participants rejected all eight false positives. The Table 1 tool rows then show FP=0 by construction. The abstract and Section 6.1 do not state this distinction and imply the AI system itself achieved 100% precision. Please report the standalone auto-mode precision explicitly and rephrase the claim as the precision of the combined human-AI decision process, or remove the unqualified claim.
  3. [§5, Efficiency paragraph] The efficiency comparison is not presented as a valid time-to-completion reduction. The text states that manual grooming was capped at 24–25 minutes by the test leader, and the 45.38% decrease is computed as time per detected duplicate (2 min 54 s vs. 1 min 35 s), not as total task completion time. An externally imposed cap on the manual condition, combined with participants stopping before completing all 41 ground-truth duplicates, makes the per-duplicate normalization incomparable to an uncapped tool condition. Please report raw completion times, state explicitly whether each session was user-terminated or externally capped, and justify the chosen normalization before using the 45% figure, which the abstract calls a time-to-completion reduction.
  4. [§4.2 and Table 1] All performance metrics are computed against a private ground-truth label set of 41 duplicate pairs established by the authors' 'thorough manual review.' No inter-rater reliability, external audit, or release of the label set is provided. Since every precision, recall, accuracy, and F1 value in Table 1 is conditional on this label set, the absence of any reproducibility measure or agreement statistic is a load-bearing gap. Please provide the label set as supplementary material and report at least one inter-rater or audit statistic, or explicitly frame all results as relative to the authors' own labels rather than as objective ground truth.
minor comments (5)
  1. [Throughout] Typographical and grammatical errors should be corrected, including 'cicles' in the abstract, 'Reseach Question' in Section 1, 'gived' in Section 4.1, and 'consolidaton' in Section 3.
  2. [Table 1 and §5] The table label 'Auto' is ambiguous because Section 5 defines auto mode as operation without human oversight; the tool-assisted rows should be renamed 'Tool-assisted' or 'Interactive' to avoid confusion with the auto-mode confusion matrix in Figure 3.
  3. [§5, Results text] The text says 'Tester #8’s TPs rose from 8 to 12,' but Table 1 lists #8 Auto TP=15; this discrepancy should be corrected and explained in the verification of the data.
  4. [§4.2, Time metric] The phrase 'an expert-reviewed, data-sensitive notion of task completeness' is undefined; please provide a concrete operationalization of the reference point used to determine completeness.
  5. [§7, Conclusions] The statement that the research question is 'conclusively addressed' overstates the evidence given five participants, one project, and a single private label set; soften the conclusion to match the exploratory scope acknowledged in Section 6.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the empirical metrics are computed against an independently defined ground truth; the 100% precision is transparently explained by human rejection of false positives, and no load-bearing step reduces to its own input.

full rationale

The paper's headline results come from a user experiment, not from a derivation chain that feeds its target back in. The 41 duplicate pairs used as ground truth are defined in Section 4.2 as "a ground truth established through a thorough manual review of the backlog to validate all true duplicate pairs." This label set is a measurement, independent of the tool's embedding threshold and GPT-4o prompting; an unaudited or internally inconsistent label set would threaten validity, but it does not make the evaluation circular. The 100% precision figure is explicitly tied to human oversight: Section 5 says "the FPs suggested by the tool were in every case rejected by the participants using the tool in interactive sessions," so the reported precision concerns the final human-accepted set, not the raw AI detector. This is an empirical outcome of a small sample, not a quantity forced by construction or by fitting a parameter. The 45% time reduction is an arithmetic normalization of measured completion times and duplicate counts, again not a fitted prediction. Self-citations ([10], [11]) are background motivation, not load-bearing justification for the empirical claim. The only serious issues are data-consistency problems—Section 4.1 says "30 known issues" while Table 1's 1275 pairwise decisions imply C(51,2)=51 issues, and no inter-rater agreement is reported for the 41 manual labels. Those are correctness/reproducibility risks, not circularity. Accordingly, no circular step can be quoted and exhibited, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The main unstated assumptions are the quality of the private ground-truth labels, the validity of embedding-based similarity as a duplicate proxy, and the representativeness of the 30-issue test backlog. No new theoretical entities are introduced.

free parameters (1)
  • cosine similarity threshold
    Section 3 states that entries exceeding a defined similarity threshold are flagged, but the value is never disclosed; this threshold directly controls the TP/FP trade-off.
assumptions (3)
  • domain assumption Cosine similarity on text embeddings is a valid proxy for duplicate backlog issues.
    Invoked in Sections 2.2 and 3; no validation that this embedding space separates duplicates from non-duplicates in the test domain.
  • domain assumption The authors' manual review correctly identified all 41 true duplicate pairs.
    Section 4.2 establishes ground truth via 'thorough manual review' with no inter-rater agreement or external audit.
  • domain assumption The 30-issue test backlog resembles a real-world project backlog.
    The authors acknowledge in Section 6.2 that domain-specific applicability is unexplored.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GenAI-Enabled Backlog Grooming in Agile Software Projects: An Empirical Study." pith.science (2026). https://pith.science/paper/FPK53U2N

@misc{pith2026250710753,
  author       = {Pith},
  title        = {Pith review of: GenAI-Enabled Backlog Grooming in Agile Software Projects: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FPK53U2N}},
  note         = {Machine review of arXiv:2507.10753}
}
read the original abstract

Effective backlog management is critical for ensuring that development teams remain aligned with evolving requirements and stakeholder expectations. However, as product backlogs consistently grow in scale and complexity, they tend to become cluttered with redundant, outdated, or poorly defined tasks, complicating prioritization and decision making processes. This study investigates whether a generative-AI (GenAI) assistant can automate backlog grooming in Agile software projects without sacrificing accuracy or transparency. Through Design Science cycles, we developed a Jira plug-in that embeds backlog issues with the vector database, detects duplicates via cosine similarity, and leverage the GPT-4o model to propose merges, deletions, or new issues. We found that AI-assisted backlog grooming achieved 100 percent precision while reducing the time-to-completion by 45 percent. The findings demonstrated the tool's potential to streamline backlog refinement processes while improving user experiences.

Figures

Figures reproduced from arXiv: 2507.10753 by the authors.

Figure 1
Figure 1. Workflow of the Generative AI Tool for Backlog Management [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Final version of the Backlog Grooming Tool - decision-making interface 5 Results Performance: The results focus on the detection of duplicate issues in a labeled Jira backlog and show key metrics such as true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). The labeled backlog contained a total of 41 true positives (issues that refer to the same functionality). The tool, when opera… view at source ↗
Figure 3
Figure 3. Confusion matrix showing the performance of the tool operating without human intervention, highlighting true and false predictions for duplicate and non-duplicate labels. testers, #9 and #12, who only performed grooming using the tool, correctly found 21 and 12 TPs in 15 and 20 minutes respectively [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Time-to-completion versus the number of true positives detected across manual and automated grooming. User Experience: All testers were asked to rate their experience, boasting only scores ranging from 7-9 out of 10. By comparison, the user experience ratings of perfor…
Figure 5
Figure 5. Figure 5: User experienced rated by Testers The findings demonstrate how GenAI and vector embeddings have the ability to identify redundant backlog items and suggest actionable resolutions, under￾scoring its ability to streamline certain agile workflows. Through these promising …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

  1. [1]

    In: Proceedings of the 1st Workshop on Agile Require- ments Engineering

    Bjarnason, E., Wnuk, K., Regnell, B.: A case study on benefits and side-effects of agile practices in large-scale requirements engi- neering. In: Proceedings of the 1st Workshop on Agile Require- ments Engineering. AREW ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2068783.2068786, https://doi.org/10.1145/2068...

    work page arXiv 2011

  2. [2]

    Calegario, F., Burégio, V., Erivaldo, F., Andrade, D.M.C., Felix, K., Barbosa, N., da Silva Lucena, P.L., França, C.: Exploring the intersection of generative ai and software development (2023), https://arxiv.org/abs/2312.14262

    work page Pith review arXiv 2023

  3. [3]

    Crawford, T., Duong, S., Fueston, R., Lawani, A., Owoade, S., Uzoka, A., Parizi, R.M., Yazdinejad, A.: Ai in software engineering: A survey on project management applications (2023), https://arxiv.org/abs/2307.15224

    arXiv 2023

  4. [4]

    Journal of Systems and Software 85(6), 1213–1221 (Jun 2012)

    Dingsøyr, T., Nerur, S., Balijepally, V., Moe, N.B.: A decade of agile method- ologies: Towards explaining agile software development. Journal of Systems and Software 85(6), 1213–1221 (Jun 2012). https://doi.org/10.1016/j.jss.2012.02.033, https://www.sciencedirect.com/science/article/pii/S0164121212000532, number: 6

    work page doi:10.1016/j.jss.2012.02.033 2012

  5. [5]

    Information and Software Technology 50(9), 833–859 (Aug 2008)

    Dybå, T., Dingsøyr, T.: Empirical studies of agile software devel- opment: A systematic review. Information and Software Technology 50(9), 833–859 (Aug 2008). https://doi.org/10.1016/j.infsof.2008.01.006, https://www.sciencedirect.com/science/article/pii/S0950584908000256

    work page doi:10.1016/j.infsof.2008.01.006 2008

  6. [6]

    https://doi.org/10.48550/arXiv.2404.04834, http://arxiv.org/abs/2404.04834, arXiv:2404.04834 [cs] 12 Oftebro et al

    He, J., Treude, C., Lo, D.: LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead (Dec 2024). https://doi.org/10.48550/arXiv.2404.04834, http://arxiv.org/abs/2404.04834, arXiv:2404.04834 [cs] 12 Oftebro et al

    arXiv doi:10.48550/arxiv.2404.04834 2024

  7. [7]

    MIS Quarterly 28(1), 75–105 (2004)

    Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in In- formation Systems Research. MIS Quarterly 28(1), 75–105 (2004). https://doi.org/10.2307/25148625, https://www.jstor.org/stable/25148625, publisher: Management Information Systems Research Center, University of Minnesota

    work page arXiv 2004

  8. [8]

    ACM Trans

    Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H.: Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol.33(8), 220:1–220:79 (Dec 2024). https://doi.org/10.1145/3695988, https://dl.acm.org/doi/10.1145/3695988

    doi:10.1145/3695988 2024

Show all 18 references
  1. [9]

    In: 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI)

    Liu, K., Reddivari, S., Reddivari, K.: Artificial intelligence in software require- ments engineering: State-of-the-art. In: 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI). pp. 106–111 (2022). https://doi.org/10.1109/IRI5479...

    2022

  2. [10]

    International Journal of Advanced Computer Science and Ap- plications 15 (01 2024)

    Nasiri, S., Lahmer, M.: A smart ai framework for backlog refinement and uml di- agram generation. International Journal of Advanced Computer Science and Ap- plications 15 (01 2024). https://doi.org/10.14569/IJACSA.2024.0150474

    2024

  3. [11]

    257–273 (06 2024)

    NguyenDuc,A.,Khanna,D.:Value-BasedAdoptionofChatGPTinAgileSoftware Development: A Survey Study of Nordic Software Experts, pp. 257–273 (06 2024). https://doi.org/10.1007/978-3-031-55642-5_12

    2024 doi

  4. [12]

    Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M.: The impact of ai on developer productivity: Evidence from github copilot (2023), https://arxiv.org/abs/2302.06590

    2023 arXiv

  5. [13]

    Ramesh, B., Cao, L., Baskerville, R.: Agile requirements engineering prac- tices and challenges: An empirical study. Inf. Syst. J. 20, 449–480 (09 2010). https://doi.org/10.1111/j.1365-2575.2007.00259.x

    2010

  6. [14]

    International Journal of Data Science and Analytics (06 2024)

    dos Santos, C., Bouchard, K., Napoleão, B.: Automatic user story generation: a comprehensive systematic literature review. International Journal of Data Science and Analytics (06 2024). https://doi.org/10.1007/s41060-024-00567-0

    2024 doi

  7. [15]

    Shahmirzadi, O., Lugowski, A., Younge, K.: Text similarity in vector space models: A comparative study (2018), https://arxiv.org/abs/1810.00664

    2018 arXiv

  8. [16]

    European Journal of Advances in Engineering and Technology9(9), 67–72 (2022)

    Siddique, I.M.: Harnessing artificial intelligence for systems engineering: Promises and pitfalls. European Journal of Advances in Engineering and Technology9(9), 67–72 (2022). https://doi.org/https://doi.org/10.5281/zenodo.11545453

    2022 doi

  9. [17]

    International Journal of Research (IJR) 11(08), 386–391 (August 2024)

    Vayyavur, R.: Why ai projects fail: The importance of strategic align- ment and systematic prioritization. International Journal of Research (IJR) 11(08), 386–391 (August 2024). https://doi.org/10.5281/zenodo.13370566, https://doi.org/10.5281/zenodo.13370566, received: 2 Augus...

    2024 doi

  10. [18]

    In: 2023 IEEE International Con- ference on Medical Artificial Intelligence (MedAI)

    Wang, J., Chen, Y.: A Review on Code Generation with LLMs: Application and Evaluation. In: 2023 IEEE International Con- ference on Medical Artificial Intelligence (MedAI). pp. 284– 289 (Nov 2023). https://doi.org/10.1109/MedAI59581.2023.00044, https://ieeexplore.ieee.org/abstr...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.