Agile Story-Point Estimation: Is RAG a Better Way to Go?

Chanchal Roy; Lamyea Maha; Tajmilur Rahman

arxiv: 2604.03443 · v1 · submitted 2026-04-03 · cs.SE · cs.AI· cs.LG

Agile Story-Point Estimation: Is RAG a Better Way to Go?

Lamyea Maha , Tajmilur Rahman , Chanchal Roy This is my paper

Reviewed by Pith2026-05-13 18:17 UTCgrok-4.3open to challenge →

classification cs.SE cs.AIcs.LG

keywords agilestory pointsRAGretrieval augmented generationeffort estimationsprint planningsoftware engineering

0 comments

The pith

RAG-based automation of agile story point estimation shows no statistically significant improvement over baselines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether retrieval-augmented generation can automate the manual consensus process of assigning story points to user stories during agile sprint planning. Researchers built a RAG system with a retriever and generator, applied it to story data from 23 open-source projects using two embedding models, and measured how retrieval settings, project size, and model choice affected results. They then compared the outputs against existing baseline methods for automated estimation. The RAG approach beat baselines in some cases, but the differences lacked statistical significance across projects and models. The work concludes that further refinement of RAG techniques is required before the method can reliably replace human effort estimation.

Core claim

We applied a RAG framework consisting of a retriever and generator to estimate story points across 23 open-source projects. Using embedding models bge-large-en-v1.5 and all-mpnet-base-v2, we analyzed the impact of retrieval hyperparameters, project sizes, and model selection. Although the RAG method outperformed baselines on several occasions, no statistically significant differences emerged across projects or embedding models.

What carries the argument

RAG pipeline with a retriever that fetches similar past stories and a generator that produces the estimate, powered by sentence embedding models.

If this is right

Further studies are needed to refine RAG and model adaptation strategies for improved accuracy.
If accuracy improves, automation could shorten time spent in sprint planning sessions.
Estimation performance appears consistent across projects of different sizes.
Choice of embedding model does not drive meaningful differences in result quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If story points contain substantial team-specific noise, any automated estimator will face the same ceiling regardless of technique.
Combining RAG retrieval with additional signals such as code metrics could raise performance above current levels.
The absence of significant gains may mean that simpler retrieval without generation already captures most available signal.

Load-bearing premise

The story points already assigned in the 23 projects accurately reflect true development effort rather than varying team biases or inconsistent definitions of complexity.

What would settle it

A follow-up study that collects fresh story points from multiple independent teams on the same tasks and checks whether RAG predictions align more closely with averaged human estimates than the baselines do.

Figures

Figures reproduced from arXiv: 2604.03443 by Chanchal Roy, Lamyea Maha, Tajmilur Rahman.

**Figure 1.** Figure 1: Methodology tokens, SBERT produces embeddings for the entire sentence, making it more efficient for directly computing semantic similarity between sentences [15]. While sentence-level embeddings can also be derived by standard ‘BERT’ model aggregating token embeddings [15], we opted for ‘SBERT’ due to its simplicity and suitability for this task. We used one of the sentence transformers all-mpnetbase-v2… view at source ↗

read the original abstract

The sprint-based iterative approach in the Agile software development method allows continuous feedback and adaptation. One of the crucial Agile software development activities is the sprint planning session where developers estimate the effort required to complete tasks through a consensus-based estimation technique such as Planning Poker. In the Agile software development method, a common unit of measuring development effort is Story Point (SP) which is assigned to tasks to understand the complexity and development time needed to complete them. Despite the benefits of this process, it is an extremely time-consuming manual process. To mitigate this issue, in this study, we investigated if this manual process can be automated using Retrieval Augmented Generation (RAG) which comprises a "Retriever" and a "Generator". We applied two embedding models - bge-large-en-v1.5, and Sentence-Transformers' all-mpnet-base-v2 on 23 open-source software projects of varying sizes and examined four key aspects: 1) how retrieval hyper-parameters influence the performance, 2) whether estimation accuracy differs across different sizes of the projects, 3) whether embedding model choice affects accuracy, and 4) how the RAG-based approach compares to the existing baselines. Although the RAG-based approach outperformed the baseline models in several occasions, our results did not exhibit statistically significant differences in performance across the projects or across the embedding models. This highlights the need for further studies and refinement of the RAG, and model adaptation strategies for better accuracy in automatically estimating user stories.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

RAG for story point estimation beats baselines on some runs but shows no statistical significance on 23 projects, with the main open question being whether the labels are noisy enough to mask real differences.

read the letter

The paper applies an off-the-shelf RAG pipeline to story point estimation and reports that it edges out published baselines in a few cases but never reaches statistical significance across projects or embedding models. That negative result is the core finding and the main thing a reader should take away first. They ran the experiment on 23 open-source projects, varied retrieval k and similarity thresholds, compared two embeddings, and broke results down by project size. The setup is straightforward and the code and data appear to be public, which is useful for anyone who wants to re-run or extend it. The work is honest about the lack of significance and calls for more refinement rather than claiming a breakthrough. That restraint is a plus. The soft spot is the ground truth. Story points come from independent Planning Poker sessions across different teams, so scale, definition of complexity, and team velocity can all differ. If that label variance is large, any accuracy metric will be dominated by noise rather than model quality, which makes the no-significance outcome unsurprising rather than informative. The paper does not appear to test inter-rater reliability or normalize scales across projects, so that assumption sits on the central claim. The citation pattern is fine; it references standard RAG and effort estimation baselines without over-claiming novelty. This is the kind of solid negative result that belongs in a workshop or a short journal note rather than a flagship venue. A reader working on automated effort estimation would get value from the dataset and the hyper-parameter sweep. A serious editor should send it to review so referees can check the statistical tests and push on the label reliability question. It is not ready as-is for a top-tier venue, but it is worth the referee time.

Referee Report

3 major / 2 minor

Summary. The manuscript investigates automating story-point estimation in Agile development via Retrieval-Augmented Generation (RAG), employing two embedding models (bge-large-en-v1.5 and all-mpnet-base-v2) on data from 23 open-source projects. It examines the effects of retrieval hyperparameters, project size, embedding choice, and compares RAG performance against published baselines. The central claim is that RAG occasionally outperforms baselines but yields no statistically significant differences across projects or embeddings, underscoring the need for further refinement.

Significance. If the non-significance result is robust, the work demonstrates that current RAG pipelines do not deliver a reliable improvement for story-point estimation over existing methods. This highlights persistent challenges in automating subjective effort estimation and the value of reproducible evaluations on multi-project data for guiding future adaptations in software engineering.

major comments (3)

[Abstract] Abstract: The headline finding of 'no statistically significant differences' is load-bearing for the conclusion yet provides no details on the exact test (e.g., Wilcoxon, t-test, or ANOVA), p-value threshold, sample sizes per comparison, or correction for multiple tests across projects and embeddings. Without these, it is impossible to distinguish true equivalence from low statistical power.
[Evaluation] Evaluation (assumed §4–5): Story-point labels from independent Planning-Poker sessions across 23 projects are treated as fixed, comparable ground truth. Different teams may employ distinct Fibonacci scales, complexity definitions, and velocity baselines; this unaddressed label noise can dominate MAE or accuracy metrics and mechanically produce non-significant results. A cross-project normalization or inter-rater reliability check is required to support the equivalence claim.
[Results] Results: The paper states RAG 'outperformed the baseline models in several occasions' but reports no per-project or per-metric breakdown (e.g., Table X showing MAE values). If the outperformance is small or inconsistent, the absence of significance is expected and does not yet justify the call for 'further studies and refinement' without quantifying effect sizes.

minor comments (2)

[Abstract] Abstract: The phrase 'on several occasions' is vague; replace with concrete counts or percentages of runs where RAG was superior.
[Related Work] The manuscript should explicitly list the baseline methods and their original references in the comparison section to allow direct replication.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We have addressed each major comment below with specific plans for revision where appropriate, while providing honest clarifications on the manuscript's current content and limitations.

read point-by-point responses

Referee: [Abstract] Abstract: The headline finding of 'no statistically significant differences' is load-bearing for the conclusion yet provides no details on the exact test (e.g., Wilcoxon, t-test, or ANOVA), p-value threshold, sample sizes per comparison, or correction for multiple tests across projects and embeddings. Without these, it is impossible to distinguish true equivalence from low statistical power.

Authors: We agree that these statistical details are necessary for proper interpretation. The manuscript currently states the non-significance result without full specification. In the revision, we will update the abstract and add a dedicated subsection in the evaluation to explicitly describe the Wilcoxon signed-rank test (chosen for its suitability with small, non-normal samples common in effort estimation), p < 0.05 threshold, n=23 projects per comparison, and note that no multiple-testing correction was applied due to the limited, pre-planned nature of the comparisons. We will also report exact p-values and include a post-hoc power analysis to address concerns about statistical power. revision: yes
Referee: [Evaluation] Evaluation (assumed §4–5): Story-point labels from independent Planning-Poker sessions across 23 projects are treated as fixed, comparable ground truth. Different teams may employ distinct Fibonacci scales, complexity definitions, and velocity baselines; this unaddressed label noise can dominate MAE or accuracy metrics and mechanically produce non-significant results. A cross-project normalization or inter-rater reliability check is required to support the equivalence claim.

Authors: We acknowledge the inherent subjectivity and potential label noise in story points across teams, which is a known challenge in the domain. Our study follows the established practice in prior story-point estimation papers by using the raw assigned points from the public datasets as ground truth without additional normalization. To address the concern, we will add a new paragraph in the threats-to-validity and discussion sections explicitly discussing this limitation, reporting observed variance in story-point distributions across projects, and noting that inter-rater reliability data is unavailable in the source datasets. We cannot retroactively perform such checks but will recommend them for future work. revision: partial
Referee: [Results] Results: The paper states RAG 'outperformed the baseline models in several occasions' but reports no per-project or per-metric breakdown (e.g., Table X showing MAE values). If the outperformance is small or inconsistent, the absence of significance is expected and does not yet justify the call for 'further studies and refinement' without quantifying effect sizes.

Authors: We agree that greater transparency on per-project performance is needed to contextualize the 'several occasions' of outperformance. In the revised manuscript, we will add a new table (and accompanying text) in the results section providing MAE, RMSE, and accuracy values for each of the 23 projects across RAG configurations and baselines. We will also compute and report effect sizes (e.g., Cohen's d) for the comparisons to better justify the call for further refinement by highlighting where gains are modest or inconsistent. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical evaluation

full rationale

The paper reports an experimental comparison of a RAG-based estimator against published baselines on story-point labels from 23 external open-source projects. All performance numbers (MAE, accuracy, statistical tests) are computed directly from those fixed labels; no parameter is fitted to a subset and then re-used as a 'prediction,' no self-citation supplies a uniqueness theorem, and no ansatz is smuggled in. The absence of statistical significance is simply the outcome of the reported tests on the observed data. This is a standard empirical study whose central claims rest on external benchmarks rather than internal re-definition.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Abstract-only review limits visibility into exact parameters. The approach implicitly assumes story points are a stable target variable and that embedding similarity captures the factors teams use when assigning points.

free parameters (1)

retrieval hyper-parameters (k, similarity threshold)
The abstract states these were varied to study influence on performance, implying they were tuned or selected rather than derived.

axioms (1)

domain assumption Story points assigned in open-source projects constitute an objective ground truth for model training and evaluation.
Invoked when treating historical story points as labels for the RAG generator.

pith-pipeline@v0.9.0 · 5575 in / 1444 out tokens · 29945 ms · 2026-05-13T18:17:32.176226+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 1 internal anchor

[1]

Pekka Abrahamsson, Ilenia Fronza, Raimund Moser, Jelena Vlasenko, and Witold Pedrycz. 2011. Predicting development effort from user stories. In2011 Inter- national Symposium on Empirical Software Engineering and Measurement. IEEE, 400–403

work page 2011
[2]

Mohammed Alhamed and Tim Storer. 2021. Playing planning poker in crowds: human computation of software effort estimates. In2021 IEEE/ACM 43rd Interna- tional Conference on Software Engineering (ICSE). IEEE, 1–12

work page 2021
[3]

Frank KY Chan and James YL Thong. 2009. Acceptance of agile methodologies: A critical review and conceptual framework.Decision support systems46, 4 (2009), 803–814. Agile Story-Point Estimation: Is RAG a Better Way to Go? ICPC ’26, April 12–13, 2026, Rio de Janeiro, Brazil

work page 2009
[4]

Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Pham, Aditya Ghose, and Tim Menzies. 2018. A deep learning model for estimating story points. IEEE Transactions on Software Engineering45, 7 (2018), 637–656

work page 2018
[5]

2005.Agile estimating and planning

Mike Cohn. 2005.Agile estimating and planning. Pearson Education

work page 2005
[6]

Tore Dybå and Torgeir Dingsøyr. 2008. Empirical studies of agile software development: A systematic review.Information and software technology50, 9-10 (2008), 833–859

work page 2008
[7]

Anirban Basu Evita Coelho. 2012. Effort Estimation in Agile Software Develop- ment using Story Points.International Journal of Applied Information Systems3, 7 (August 2012), 7–10. doi:10.5120/ijais12-450574

work page doi:10.5120/ijais12-450574 2012
[8]

Michael Fu and Chakkrit Tantithamthavorn. 2022. GPT2SP: A transformer-based agile story point estimation approach.IEEE Transactions on Software Engineering 49, 2 (2022), 611–625

work page 2022
[9]

Magne Jørgensen. 2004. A review of studies on expert estimation of software development effort.Journal of Systems and Software70, 1-2 (2004), 37–60

work page 2004
[10]

Gaurav Kumar and Pradeep Kumar Bhatia. 2012. Impact of agile methodology on software development process.International Journal of Computer Technology and Electronics Engineering (IJCTEE)2, 4 (2012), 46–50

work page 2012
[11]

Viljan Mahnič and Tomaž Hovelja. 2012. On using planning poker for estimating user stories.Journal of Systems and Software85, 9 (2012), 2086–2095

work page 2012
[12]

Kjetil Moløkken-Østvold and Magne Jørgensen. 2004. Group processes in soft- ware effort estimation.Empirical Software Engineering9, 4 (2004), 315–334

work page 2004
[13]

Hung Phan and Ali Jannesari. 2022. Heterogeneous graph neural networks for software effort estimation. InProceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 103–113

work page 2022
[14]

Simone Porru, Alessandro Murgia, Serge Demeyer, Michele Marchesi, and Roberto Tonelli. 2016. Estimating story points from issue reports. InProceedings of the the 12th international conference on predictive models and data analytics in software engineering. 1–10

work page 2016
[15]

N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks.arXiv preprint arXiv:1908.10084(2019)

work page internal anchor Pith review Pith/arXiv arXiv 2019
[16]

Ritu and Pankaj Bhambri. 2023. Software effort estimation with machine learning– A systematic literature review.Agile software development: Trends, challenges and applications(2023), 291–308

work page 2023
[17]

SBERT. [n. d.]. Pretrained Models — Sentence Transformers documentation. https://www.sbert.net/docs/sentence_transformer/pretrained_models.html [On- line; accessed 2025-01-31]

work page 2025
[18]

Vali Tawosi, Afnan Al-Subaihin, and Federica Sarro. 2022. Investigating the effectiveness of clustering for story point estimation. In2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 827–838

work page 2022
[19]

Vali Tawosi, Salwa Alamir, and Xiaomo Liu. 2023. Search-based optimisation of LLM learning shots for story point estimation. InInternational Symposium on Search Based Software Engineering. Springer, 123–129

work page 2023
[20]

Vali Tawosi, Rebecca Moussa, and Federica Sarro. 2022. Deep Learning for Agile Effort Estimation Have We Solved the Problem Yet.arXiv preprint arXiv:2201.05401 (2022)

work page arXiv 2022
[21]

Vali Tawosi, Rebecca Moussa, and Federica Sarro. 2024. Agile Effort Estimation: Have We Solved the Problem Yet? Insights From the Replication of the GPT2SP Study. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 1034–1041

work page 2024
[22]

Adam Trendowicz and Ross Jeffery. 2014. Software project effort estimation. Foundations and Best Practice Guidelines for Success, Constructive Cost Model– COCOMO pags12 (2014), 277–293

work page 2014
[23]

Muhammad Usman, Emilia Mendes, Francila Weidt, and Ricardo Britto. 2014. Effort estimation in agile software development: a systematic literature review. InProceedings of the 10th international conference on predictive models in software engineering. 82–91

work page 2014

[1] [1]

Pekka Abrahamsson, Ilenia Fronza, Raimund Moser, Jelena Vlasenko, and Witold Pedrycz. 2011. Predicting development effort from user stories. In2011 Inter- national Symposium on Empirical Software Engineering and Measurement. IEEE, 400–403

work page 2011

[2] [2]

Mohammed Alhamed and Tim Storer. 2021. Playing planning poker in crowds: human computation of software effort estimates. In2021 IEEE/ACM 43rd Interna- tional Conference on Software Engineering (ICSE). IEEE, 1–12

work page 2021

[3] [3]

Frank KY Chan and James YL Thong. 2009. Acceptance of agile methodologies: A critical review and conceptual framework.Decision support systems46, 4 (2009), 803–814. Agile Story-Point Estimation: Is RAG a Better Way to Go? ICPC ’26, April 12–13, 2026, Rio de Janeiro, Brazil

work page 2009

[4] [4]

Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Pham, Aditya Ghose, and Tim Menzies. 2018. A deep learning model for estimating story points. IEEE Transactions on Software Engineering45, 7 (2018), 637–656

work page 2018

[5] [5]

2005.Agile estimating and planning

Mike Cohn. 2005.Agile estimating and planning. Pearson Education

work page 2005

[6] [6]

Tore Dybå and Torgeir Dingsøyr. 2008. Empirical studies of agile software development: A systematic review.Information and software technology50, 9-10 (2008), 833–859

work page 2008

[7] [7]

Anirban Basu Evita Coelho. 2012. Effort Estimation in Agile Software Develop- ment using Story Points.International Journal of Applied Information Systems3, 7 (August 2012), 7–10. doi:10.5120/ijais12-450574

work page doi:10.5120/ijais12-450574 2012

[8] [8]

Michael Fu and Chakkrit Tantithamthavorn. 2022. GPT2SP: A transformer-based agile story point estimation approach.IEEE Transactions on Software Engineering 49, 2 (2022), 611–625

work page 2022

[9] [9]

Magne Jørgensen. 2004. A review of studies on expert estimation of software development effort.Journal of Systems and Software70, 1-2 (2004), 37–60

work page 2004

[10] [10]

Gaurav Kumar and Pradeep Kumar Bhatia. 2012. Impact of agile methodology on software development process.International Journal of Computer Technology and Electronics Engineering (IJCTEE)2, 4 (2012), 46–50

work page 2012

[11] [11]

Viljan Mahnič and Tomaž Hovelja. 2012. On using planning poker for estimating user stories.Journal of Systems and Software85, 9 (2012), 2086–2095

work page 2012

[12] [12]

Kjetil Moløkken-Østvold and Magne Jørgensen. 2004. Group processes in soft- ware effort estimation.Empirical Software Engineering9, 4 (2004), 315–334

work page 2004

[13] [13]

Hung Phan and Ali Jannesari. 2022. Heterogeneous graph neural networks for software effort estimation. InProceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 103–113

work page 2022

[14] [14]

Simone Porru, Alessandro Murgia, Serge Demeyer, Michele Marchesi, and Roberto Tonelli. 2016. Estimating story points from issue reports. InProceedings of the the 12th international conference on predictive models and data analytics in software engineering. 1–10

work page 2016

[15] [15]

N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks.arXiv preprint arXiv:1908.10084(2019)

work page internal anchor Pith review Pith/arXiv arXiv 2019

[16] [16]

Ritu and Pankaj Bhambri. 2023. Software effort estimation with machine learning– A systematic literature review.Agile software development: Trends, challenges and applications(2023), 291–308

work page 2023

[17] [17]

SBERT. [n. d.]. Pretrained Models — Sentence Transformers documentation. https://www.sbert.net/docs/sentence_transformer/pretrained_models.html [On- line; accessed 2025-01-31]

work page 2025

[18] [18]

Vali Tawosi, Afnan Al-Subaihin, and Federica Sarro. 2022. Investigating the effectiveness of clustering for story point estimation. In2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 827–838

work page 2022

[19] [19]

Vali Tawosi, Salwa Alamir, and Xiaomo Liu. 2023. Search-based optimisation of LLM learning shots for story point estimation. InInternational Symposium on Search Based Software Engineering. Springer, 123–129

work page 2023

[20] [20]

Vali Tawosi, Rebecca Moussa, and Federica Sarro. 2022. Deep Learning for Agile Effort Estimation Have We Solved the Problem Yet.arXiv preprint arXiv:2201.05401 (2022)

work page arXiv 2022

[21] [21]

Vali Tawosi, Rebecca Moussa, and Federica Sarro. 2024. Agile Effort Estimation: Have We Solved the Problem Yet? Insights From the Replication of the GPT2SP Study. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 1034–1041

work page 2024

[22] [22]

Adam Trendowicz and Ross Jeffery. 2014. Software project effort estimation. Foundations and Best Practice Guidelines for Success, Constructive Cost Model– COCOMO pags12 (2014), 277–293

work page 2014

[23] [23]

Muhammad Usman, Emilia Mendes, Francila Weidt, and Ricardo Britto. 2014. Effort estimation in agile software development: a systematic literature review. InProceedings of the 10th international conference on predictive models in software engineering. 82–91

work page 2014