Pith. sign in

REVIEW 3 major objections 4 minor 60 references

Extracting Knowledge Graphs from User Stories using LangChain

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fully automated LangChain-based module, USGT, turns user stories into queryable Neo4j knowledge graphs and improves entity extraction by 16 percent on average over the prior LLM method, while remaining model-agnostic.

desk verdict A useful, reproducible incremental extension of Arulmohan et al., but the headline 16% improvement over the repository baseline is not secured because evaluation equivalence is never demonstrated. read the letter →

arxiv 2506.11020 v1 pith:IK25RLK7 submitted 2025-05-14 cs.SE cs.AI

Thayná Camargo da Silva This is my paper · ORCID
classification cs.SEcs.AI
keywords knowledgegraphextractionuserstoriesLangChainlargelanguagemodelsrequirementsengineeringNeo4jLLMGraphTransformerautomatedsoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis tries to establish that a fully automated, model-agnostic pipeline built on LangChain can convert user stories into queryable knowledge graphs more accurately than the previous LLM-based method. The proposed UserStoryGraphTransformer (USGT) module extracts persona, action, entity, and benefit nodes along with trigger and target relationships, then stores the result in Neo4j. On the same annotated backlog dataset, its entity extraction improved by 16% on average in F-measure compared with the published GPT-4-turbo benchmark, while persona and action scores also came out at or above that baseline. The practical significance is that a product backlog can become a structured, queryable requirements model without manual modeling, NLP expertise, or dependence on a single LLM provider.

What carries the argument

The load-bearing mechanism is the UserStoryGraphTransformer (USGT) module, a customized version of LangChain's LLMGraphTransformer. It splits extraction into two prompts: the main prompt asks the LLM to name one persona, all actions and entities, and the TRIGGERS and TARGETS relationships among them, while a separate benefit prompt extracts only the optional "so that" benefit sentence, avoiding a failure mode where the model emitted a benefit but dropped the associated action and entity. A Graph Transformer component then adds the user story itself as a node and derives the HAS_* relationships from the node types already present, so the LLM only does the semantically hard relationship work. LangChain's LCEL chains, output parsers, and Neo4jGraph integration supply the automation and provider abstraction, letting the same module call function-capable models like GPT-4o mini and non-function-capable models like Llama 3.

What would settle it

Run the benchmark's original prompt and the USGT module with the exact same LLM, model version, date, and temperature on the same cleaned dataset; if the original prompt matches or beats USGT's F-measures, the claimed improvement is not attributable to the USGT design.

Watch

Extended reading notes

Core claim

The paper's central claim is that the USGT module outperforms the earlier LLM-based extraction method on the same cleaned annotated dataset. In the strict comparison, GPT-4-turbo run through USGT achieves average F-measures of 0.99 for persona, 0.80 for entity, and 0.79 for action, against the benchmark's 0.98, 0.69, and 0.73; entity extraction therefore improves by 16% on average. The module asks the LLM for only the ontology's core components: a main prompt extracts persona, action, and entity nodes plus TRIGGERS and TARGETS relationships, while a separate prompt extracts the optional benefit node. The Graph Transformer component then adds the user story itself as a node and derives the HAS_PERSONA, HAS_ACTION, HAS_ENTITY, and HAS_BENEFIT links by logical inference, yielding a complete Graph Document written directly into Neo4j. The thesis also claims the solution is model-agnostic, having run with GPT-4o mini, which supports function calls, and Llama 3, which does not, and it still trails the trained CRF baseline, though by a narrower margin than the earlier LLM method.

Load-bearing premise

The load-bearing premise is that the earlier published benchmark scores are a fair baseline for the new module's results, even though the winning numbers come from a different model generation tested later; if the model's progress, not the module's design, explains the gain, the central comparison collapses.

Editorial extensions

If this is right

  • A product backlog can be converted into a queryable Neo4j knowledge graph end to end, with no manual modeling step and no NLP tooling beyond the LLM call.
  • Teams are not locked into a single LLM provider; swapping GPT-4o mini for Llama 3 is a configuration change, at a measurable accuracy cost in exact-match extraction.
  • The accuracy gap between LLM-based extraction and a trained CRF model narrows, making LLM-based requirements modeling competitive for projects that cannot afford a training phase.
  • Reusable evaluation machinery, including comparison modes, precision, recall, F-measure, and BERTScore, lets future extraction methods be measured against the same ground truth and compared directly.
  • Including benefit nodes in evaluation extends coverage of the ontology beyond the earlier benchmark, so reported scores reflect a fuller user-story structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the 16% entity gain could be driven by prompt decomposition rather than by the module's graph plumbing; a direct test would be to run the benchmark's original prompt on GPT-4o mini and Llama 3 against the same dataset.
  • The dual-prompt split, hard nodes first and optional benefit later, is a general recipe: any ontology-constrained extraction task with an optional component could benefit from isolating that component, a hypothesis the thesis does not itself test.
  • Shared persona and entity nodes across stories in the resulting graph could be used to detect dependencies, overlaps, and conflicts in the backlog; the thesis visualizes this potential but does not quantify it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript (a Master's thesis posted to arXiv) proposes the UserStoryGraphTransformer (USGT), a LangChain-based module that uses an LLM to extract node and relationship components from user stories according to the ontology of Arulmohan et al. [1]. It splits extraction into a main prompt (persona, actions, entities, TRIGGERS, TARGETS) and a benefit prompt, enriches outputs with a user-story node and logically inferred HAS_* relationships, and stores the resulting graph in Neo4j. The evaluation compares USGT with Llama 3 and GPT-4o mini on a cleaned 1,459-story annotated dataset under strict, inclusive, relaxed, and BERTScore matching, and compares GPT-4-turbo USGT results against benchmark repository scores for GPT-4-turbo and CRF. The central claim is that USGT is fully automated, model-agnostic, and improves entity extraction by 16% over the earlier LLM-based method.

Significance. The paper's practical contributions are real: a reusable implementation, an automated evaluation script, use of standard precision/recall/F-measure plus BERTScore, and reuse of an externally annotated dataset with high inter-annotator agreement, which keeps circularity low. If the comparative results are confirmed under a single evaluation pipeline, the USGT would be a useful, low-cost alternative to NLP-based user-story modeling and would meaningfully close the gap with CRF. However, the headline claim currently rests on an unverified comparison with repository scores, and no relationship-level metrics are reported even though the first research question concerns relationship extraction.

major comments (3)
  1. [Section 5.4, Table 5.5] Section 5.2 documents exactly two USGT experiments (Llama 3 and GPT-4o mini) and does not describe a GPT-4-turbo run, yet Table 5.5 reports a 'GPT-4-turbo using USGT' column and Section 5.4 uses it to claim a 16% entity-extraction improvement. The text says the earlier results were 'found' in repository [39], but it never states that the repository outputs and the USGT outputs were scored with the same evaluation.py, the same cleaned backlog subset, and the same strict-mode matching rules. Unless this evaluation-equivalence is demonstrated, ideally by running both output sets through one script and publishing the outputs, the reported improvement is not secured. This is load-bearing for RQ1 and for the headline comparison.
  2. [Section 5.3, Tables 5.1-5.4, and RQ1 in Section 1.3] RQ1 asks about the accuracy of both nodes and relationships, and Section 5.1 describes TP/FP/FN counting for KG components, but Section 5.3 and Tables 5.1-5.4 report F-measures only for Persona, Entity, Action, and Benefit. No precision/recall/F-measure for TRIGGERS or TARGETS is given anywhere, and Table 5.5 also omits relationships. The claim in Section 5.4 that the solution 'successfully extracts nodes and relationships' is therefore unsupported by the reported evidence. The authors should either add relationship-level metrics or explicitly limit all claims to node extraction and revise the RQ1 answer accordingly.
  3. [Section 7.3 and Table 5.5] Section 7.3 acknowledges that LLM outputs are non-deterministic even with temperature set to zero, but Table 5.5 reports single numbers per backlog with no repeated runs or variance measures. Since the headline comparison is an average over 21 backlogs, run-to-run variability could affect the size of the reported gap. The paper should at least report the number of runs behind each score and, if possible, provide multiple runs or confidence intervals before presenting the 16% improvement as a stable result.
minor comments (4)
  1. [Section 5.4] The phrase 'entity extraction improved by 16% on average' is ambiguous: the table shows 0.69 to 0.80, which is an increase of 11 percentage points, or approximately 16% relative. Please state both values explicitly.
  2. [Table 5.5 caption] The caption uses 'GPT-4-turbo from Benchmark' while the text refers to 'GPT-4-0125-preview (also referred to as GPT-4-turbo)'. Please use one consistent model name throughout the table and text.
  3. [Section 5.3] The sentence 'Moving to the inclusive and (Table 5.2) relaxed (Table 5.3) modes' is grammatically incomplete; it should read 'Moving to the inclusive (Table 5.2) and relaxed (Table 5.3) modes'.
  4. [Section 7.3] The threats-to-validity discussion is helpful, but it does not mention the comparison-equivalence threat that affects Table 5.5; adding it would improve transparency about the central comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central evaluation is measured against an external annotated dataset, and the derived HAS_* edges are a design convenience rather than a claimed predictive result.

full rationale

The paper's core empirical claim, the 16% improvement in entity F-measure in Section 5.4, is a comparison between the USGT module's outputs and scores taken from an external repository [39] for the earlier LLM-based method, both measured against the same externally annotated dataset [49]. The benchmark numbers are not fitted parameters of this thesis, nor are they derived from the USGT outputs, so the comparison cannot be circular by construction. The HAS_* relationships generated by the Graph Transformer (Section 4.2.1, Listing 4.10) are logically inferred from the presence of extracted nodes, so they are correct by construction; however, the evaluation tables report node-level F-measures and do not use these inferred edges as evidence of predictive performance, so this design choice does not support a circularity finding. The thesis also adopts the ontology and comparison modes from Arulmohan et al. [1], but sharing the benchmark's ontology and metrics is what makes the comparison meaningful rather than circular. The remaining concern, that repository baseline scores and USGT scores may not have been produced by a single identically configured evaluation pipeline, is a validity and fairness threat, not a self-definitional or fitted-input circularity. The thesis's own limitations section similarly flags prompt subjectivity and LLM nondeterminism, which are external-validity issues, not circular reasoning. No load-bearing step reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

USGT reuses the existing ontology from [1], including Userstory, Persona, Action, Entity, and Benefit nodes, and introduces no new physical or conceptual entities with independent falsifiable evidence. The only hand-chosen numerical setting is the LLM temperature. The four axioms above are domain assumptions about ontology fit, ground-truth reliability, dataset representativeness, and output stability.

free parameters (1)
  • LLM temperature = 0.0
    Hand-chosen in Section 5.2 to reduce output stochasticity. It is a configuration setting rather than a fitted constant, but it affects every reported F-measure.
assumptions (4)
  • domain assumption Arulmohan et al. ontology is the correct target structure for representing user stories.
    Adopted as the knowledge graph blueprint in Section 2.2.1; all prompts and the Graph Transformer enforce it, so the evaluation measures fit to this ontology, not general KG quality.
  • domain assumption The annotated Ace-design/qualified-user-stories dataset is a trustworthy ground truth.
    Used for all precision, recall, and F-measure calculations (Section 5.1). The 94% inter-annotator agreement is cited from [1], but Section 7.3 admits that annotation remains subjective.
  • domain assumption The cleaned 87% dataset subset is representative and suitable for comparison.
    The experiments reuse the cleaned subset from [1] (Section 5.2), which contains only stories all tools could process. This selection may bias results, though it enables direct comparison.
  • domain assumption Temperature-zero LLM outputs are stable enough for single-run evaluation.
    Section 5.2 sets temperature to 0, but Section 7.3 states outputs can still vary across executions. No repeated runs, standard deviations, or confidence intervals are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extracting Knowledge Graphs from User Stories using LangChain." pith.science (2026). https://pith.science/paper/IK25RLK7

@misc{pith2026250611020,
  author       = {Pith},
  title        = {Pith review of: Extracting Knowledge Graphs from User Stories using LangChain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IK25RLK7}},
  note         = {Machine review of arXiv:2506.11020}
}
read the original abstract

This thesis introduces a novel methodology for the automated generation of knowledge graphs from user stories by leveraging the advanced capabilities of Large Language Models. Utilizing the LangChain framework as a basis, the User Story Graph Transformer module was developed to extract nodes and relationships from user stories using an LLM to construct accurate knowledge graphs.This innovative technique was implemented in a script to fully automate the knowledge graph extraction process. Additionally, the evaluation was automated through a dedicated evaluation script, utilizing an annotated dataset for assessment. By enhancing the visualization and understanding of user requirements and domain concepts, this method fosters better alignment between software functionalities and user expectations, ultimately contributing to more effective and user-centric software development processes.

Figures

Figures reproduced from arXiv: 2506.11020 by the authors.

Figure 2.1
Figure 2.1. Conceptual model of a backlog [1]. This thesis proposes a knowledge graph design for representing user stories, also known as backlog items, within the Scrum framework [50]. The design adopts the ontology developed by Arulmohan et al. [1] (see [PITH_FULL_IMAGE:figures/full_fig_p015_2_1.png] view at source ↗
Figure 2.2
Figure 2.2. Example of a modeled product backlog of a web application following the ontology described in [PITH_FULL_IMAGE:figures/full_fig_p016_2_2.png] view at source ↗
Figure 4.1
Figure 4.1. Neo4j visualization of the graph created in Listing 4.1. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_4_1.png] view at source ↗
Figures from the paper (9 more)
Figure 4.2
Figure 4.2. Figure 4.2: Component diagram of the UserStoryGraphTransformer module. The USGT module, as illustrated in the component diagram in [PITH_FULL_IMAGE:figures/full_fig_p034_4_2.png]
Figure 4.3
Figure 4.3. Figure 4.3: Overview of automated KG extraction using the USGT module. To illustrate the behavioral architecture, refer to [PITH_FULL_IMAGE:figures/full_fig_p042_4_3.png]
Figure 4.4
Figure 4.4. Figure 4.4: Sub-diagram of activity 1: Configure LLM. Activity 2, transform user story into Document format, applies LangChain ready￾made function to convert the user story into a Document object type that is required for LLM interaction. Activity 3 ( [PITH_FULL_IMAGE:figures/f…
Figure 4.5
Figure 4.5. Figure 4.5: Sub-diagram of activity 3: Extract LLM-derived components using LLM Connector. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_4_5.png]
Figure 4.6
Figure 4.6. Figure 4.6: Sub-diagram of activity 4: Transform LLM-derived components into a Graph Documents using Graph Transformer. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_4_6.png]
Figure 4.7
Figure 4.7. Figure 4.7: Sub-diagram of activity 5: Store Graph Document into Neo4j. The final activity, which can be visualized in [PITH_FULL_IMAGE:figures/full_fig_p046_4_7.png]
Figure 4.8
Figure 4.8. Figure 4.8: Neo4j snapshot of knowledge graph extracted from example user story using USGT. how to code, so that I can build my own projects.” can be seen in [PITH_FULL_IMAGE:figures/full_fig_p047_4_8.png]
Figure 5.1
Figure 5.1. Figure 5.1: Average comparison of node types extraction using GPT-4o-mini model [PITH_FULL_IMAGE:figures/full_fig_p059_5_1.png]
Figure 5.2
Figure 5.2. Figure 5.2: Average comparison of node types extraction using LLama3. tive, and in this case both models show a good performance. Overall, GPT-4o-mini ( [PITH_FULL_IMAGE:figures/full_fig_p059_5_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 47 canonical work pages

  1. [39]

    Extracting domain models from user stories- repository, 2023

    S´ ebastien Mosser and Sathurshan Arulmohan. Extracting domain models from user stories- repository, 2023. Repository available on: https://github.com/ace- design/qualified-user-stories

    work page 2023

  2. [1]

    Extracting do- main models from textual requirements in the era of large language models

    Sathurshan Arulmohan, Marie-Jean Meurs, and S´ ebastien Mosser. Extracting do- main models from textual requirements in the era of large language models. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) , pages 580–587. IEEE, 2023

    work page 2023

  3. [2]

    Promptner: Prompting for named entity recognition

    Dhananjay Ashok and Zachary C Lipton. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444 , 2023

    arXiv 2023

  4. [3]

    An ontology-based approach to automate the software development process

    Kathirgamasegaran Athiththan, Selvaratnam Rovinsan, Srijeevahan Sathveegan, Nahanaa Gunasekaran, Kamila SAW Gunawardena, and Dharshana Kasthuri- rathna. An ontology-based approach to automate the software development process. In 2018 ieee international conference on information and automation for sustain- ability (iciafs), pages 1–6. IEEE, 2018

    work page 2018

  5. [4]

    The agile manifesto, 2001

    Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cunning- ham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, et al. The agile manifesto, 2001

    work page 2001

  6. [5]

    De- riving domain models from user stories: Human vs

    Maxim Bragilovski, Ashley T Van Can, Fabiano Dalpiaz, and Arnon Sturm. De- riving domain models from user stories: Human vs. machines. In 2024 IEEE 32nd International Requirements Engineering Conference (RE), pages 31–42. IEEE, 2024

    work page 2024

  7. [6]

    Domain modeling and domain engineering: Key tasks in require- ments engineering

    Manfred Broy. Domain modeling and domain engineering: Key tasks in require- ments engineering. Perspectives on the Future of Software Engineering: Essays in Honor of Dieter Rombach , pages 15–30, 2013

    work page 2013

  8. [7]

    Jumping nlp curves: A review of natural language processing research

    Erik Cambria and Bebo White. Jumping nlp curves: A review of natural language processing research. IEEE Computational intelligence magazine , 9(2):48–57, 2014

    work page 2014

Show all 60 references
  1. [8]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology , 2023

    2023

  2. [9]

    Langchain, 2022

    Harrison Chase. Langchain, 2022. https://github.com/langchain-ai/langchain. Ac- cessed on 2024-10-10

    2022

  3. [10]

    Llmgraphtransformer, 2023

    Harrison Chase. Llmgraphtransformer, 2023. 71

    2023

  4. [11]

    Generative ai for requirements engineering: A systematic literature review

    Haowei Cheng, Jati H Husen, Sien Reeve Peralta, Bowen Jiang, Nobukazu Yosh- ioka, Naoyasu Ubayashi, and Hironori Washizaki. Generative ai for requirements engineering: A systematic literature review. arXiv preprint arXiv:2409.06741, 2024

    2024

  5. [12]

    Know your customers’ jobs to be done

    Clayton M Christensen, Taddy Hall, Karen Dillon, and David S Duncan. Know your customers’ jobs to be done. Harvard business review, 94(9):54–62, 2016

    2016

  6. [13]

    User stories applied: For agile software development

    Mike Cohn. User stories applied: For agile software development . Addison-Wesley Professional, 2004

    2004

  7. [14]

    Software Engineering Technical Committee

    IEEE Computer Society. Software Engineering Technical Committee. IEEE Stan- dard Glossary of Software Engineering Terminology , volume 729. IEEE, 1983

    1983

  8. [15]

    Extracting knowledge graphs from user stories using langchain, 2024

    Thayn´ a Camargo da Silva. Extracting knowledge graphs from user stories using langchain, 2024. https://zenodo.org/record/14254058

    2024

  9. [16]

    Requirements data sets (user stories), July 2018

    Fabiano Dalpiaz. Requirements data sets (user stories), July 2018. dataset

    2018

  10. [17]

    Detecting terminological ambiguity in user stories: Tool and experimentation

    Fabiano Dalpiaz, Ivor Van Der Schalk, Sjaak Brinkkemper, Fatma Ba¸ sak Aydemir, and Garm Lucassen. Detecting terminological ambiguity in user stories: Tool and experimentation. Information and Software Technology, 110:3–16, 2019

    2019

  11. [18]

    Requirements engineering tools: Capa- bilities, survey and assessment

    Juan M Carrillo De Gea, Joaqu´ ın Nicol´ as, Jos´ e L Fern´ andez Alem´ an, Ambrosio Toval, Christof Ebert, and Aurora Vizca´ ıno. Requirements engineering tools: Capa- bilities, survey and assessment. Information and Software Technology, 54(10):1142– 1157, 2012

    2012

  12. [19]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018

    2018 arXiv

  13. [20]

    Automatic generation of uml se- quence diagrams from user stories in scrum process

    Meryem Elallaoui, Khalid Nafil, and Raja Touahni. Automatic generation of uml se- quence diagrams from user stories in scrum process. In 2015 10th international con- ference on intelligent systems: theories and applications (SITA) , pages 1–6. IEEE, 2015

    2015

  14. [21]

    Automatic transformation of user stories into uml use case diagrams using nlp techniques

    Meryem Elallaoui, Khalid Nafil, and Raja Touahni. Automatic transformation of user stories into uml use case diagrams using nlp techniques. Procedia computer science, 130:42–49, 2018

    2018

  15. [22]

    Formalizing natural language intent into program specifications via large language models

    Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. Formalizing natural language intent into program specifications via large language models. arXiv preprint arXiv:2310.01831 , 2023

    2023 arXiv

  16. [23]

    Large language models for software engineering: Survey and open problems

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (IC...

    2023

  17. [24]

    Knowledge graphs methodology, tools and selected use cases (2020)

    D Fensel, U Simsek, K Angele, E Huaman, E K¨arle, O Panasiuk, I Toma, J Umbrich, and A Wahler. Knowledge graphs methodology, tools and selected use cases (2020)

    2020

  18. [25]

    Transformerranker: A tool for effi- ciently finding the best-suited language models for downstream classification tasks

    Lukas Garbas, Max Ploner, and Alan Akbik. Transformerranker: A tool for effi- ciently finding the best-suited language models for downstream classification tasks. arXiv preprint arXiv:2409.05997 , 2024

    2024 arXiv

  19. [26]

    Connecting user stories and code for test development

    Adrian Genaid et al. Connecting user stories and code for test development. In 2012 third international workshop on recommendation systems for software engineering (rsse), pages 33–37. IEEE, 2012

    2012

  20. [27]

    Facts and fallacies of software engineering

    Robert L Glass. Facts and fallacies of software engineering . Addison-Wesley Pro- fessional, 2002

    2002

  21. [28]

    Large language models for software engineering: A systematic literature review

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620, 2023

    2023 arXiv

  22. [29]

    Unveiling llm evaluation focused on metrics: Chal- lenges and solutions

    Taojun Hu and Xiao-Hua Zhou. Unveiling llm evaluation focused on metrics: Chal- lenges and solutions. arXiv preprint arXiv:2404.09135 , 2024

    2024 arXiv

  23. [30]

    A study on the implementation method of an agent-based advanced rag system using graph

    Cheonsu Jeong. A study on the implementation method of an agent-based advanced rag system using graph. arXiv preprint arXiv:2407.19994 , 2024

    2024 arXiv

  24. [31]

    Research trends for the interplay between large lan- guage models and knowledge graphs

    Hanieh Khorashadizadeh, Fatima Zahra Amara, Morteza Ezzabady, Fr´ ed´ eric Ieng, Sanju Tiwari, Nandana Mihindukulasooriya, Jinghua Groppe, Soror Sahri, Farah Benamara, and Sven Groppe. Research trends for the interplay between large lan- guage models and knowledge graphs. arXi...

    2024 arXiv

  25. [32]

    Extracting queryable knowledge graphs from user stories: An empirical evaluation

    Ayodeji Ladeinde, Chetan Arora, Hourieh Khalajzadeh, Tanjila Kanij, and John Grundy. Extracting queryable knowledge graphs from user stories: An empirical evaluation. In International Conference on Evaluation of Novel Approaches to Soft- ware Engineering 2023, pages 684–692. S...

    2023

  26. [33]

    Conditional random fields: Probabilistic models for segmenting and labeling sequence data

    John Lafferty, Andrew McCallum, Fernando Pereira, et al. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Icml, volume 1, page 3. Williamstown, MA, 2001

    2001

  27. [34]

    Improving agile requirements: the quality user story framework and tool

    Garm Lucassen, Fabiano Dalpiaz, Jan Martijn EM van der Werf, and Sjaak Brinkkemper. Improving agile requirements: the quality user story framework and tool. Requirements engineering, 21:383–403, 2016

    2016

  28. [35]

    Extracting conceptual models from user stories with visual narrator

    Garm Lucassen, Marcel Robeer, Fabiano Dalpiaz, Jan Martijn EM Van Der Werf, and Sjaak Brinkkemper. Extracting conceptual models from user stories with visual narrator. Requirements Engineering, 22:339–358, 2017

    2017

  29. [36]

    An approach for knowledge graphs-based user stories in agile methodologies

    Marco Mancuso and Emanuele Laurenzi. An approach for knowledge graphs-based user stories in agile methodologies. In International Conference on Business Infor- matics Research, pages 133–141. Springer, 2023. 73

    2023

  30. [37]

    Transforming conversational ai

    Michael McTear and Marina Ashurkina. Transforming conversational ai. 2024

    2024

  31. [38]

    Modelling agile back- logs as composable artifacts to support developers and product owners

    S´ ebastien Mosser, Corinne Pulgar, and Vladimir Reinhar. Modelling agile back- logs as composable artifacts to support developers and product owners. J. Object Technol., 21(3):3–1, 2022

    2022

  32. [40]

    Natural language processing: an introduction

    Prakash M Nadkarni, Lucila Ohno-Machado, and Wendy W Chapman. Natural language processing: an introduction. Journal of the American Medical Informatics Association, 18(5):544–551, 2011

    2011

  33. [41]

    doccano: Text annotation tool for human, 2018

    Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. doccano: Text annotation tool for human, 2018. Software available from https://github.com/doccano/doccano

    2018

  34. [42]

    Towards a generation of class diagram from user stories in agile methods

    Samia Nasiri, Yassine Rhazali, and Mohammed Lahmer. Towards a generation of class diagram from user stories in agile methods. In Advancements in Model-Driven Architecture in Software Engineering, pages 135–159. IGI Global, 2021

    2021

  35. [43]

    Neo4j - the world’s leading graph database, 2024

    Neo4j. Neo4j - the world’s leading graph database, 2024. https://neo4j.com/. Accessed on 2024-06-12

    2024

  36. [44]

    Prompt engineering, 2023

    OpenAI. Prompt engineering, 2023. https://platform.openai.com/docs/guides/prompt- engineering?ref=blef.fr. Accessed on 2024-09-07

    2023

  37. [45]

    Chatgpt: Openai language model, 2024

    OpenAI. Chatgpt: Openai language model, 2024. https://platform.openai.com/docs/models/gpt-4. Accessed on 2024-07-14

    2024

  38. [46]

    Requirements engineering: An overview

    Klaus Pohl. Requirements engineering: An overview . Citeseer, 1996

    1996

  39. [47]

    User stories and natural language processing: A systematic literature review

    Indra Kharisma Raharjana, Daniel Siahaan, and Chastine Fatichah. User stories and natural language processing: A systematic literature review. IEEE access, 9:53811–53826, 2021

    2021

  40. [48]

    Automated extraction of conceptual models from user stories via nlp

    Marcel Robeer, Garm Lucassen, Jan Martijn EM Van Der Werf, Fabiano Dalpiaz, and Sjaak Brinkkemper. Automated extraction of conceptual models from user stories via nlp. In 2016 IEEE 24th international requirements engineering conference (RE), pages 196–205. IEEE, 2016

    2016

  41. [49]

    ace- design/qualified-user-stories: Version 1.0, July 2023

    Sathurshan Arulmohan, S´ ebastien Mosser, and Marie-Jean Meurs. ace- design/qualified-user-stories: Version 1.0, July 2023. Annotated dataset

    2023

  42. [50]

    The scrum guide

    Ken Schwaber and Jeff Sutherland. The scrum guide. Scrum Alliance, 21(1):1–38, 2011. 74

    2011

  43. [51]

    Revolu- tionizing mental health care through langchain: A journey with a large language model

    Aditi Singh, Abul Ehtesham, Saifuddin Mahmud, and Jong-Hoon Kim. Revolu- tionizing mental health care through langchain: A journey with a large language model. In 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), pages 0073–0078. IEEE, 2024

    2024

  44. [52]

    Software engineering (ed.)

    Ian Sommerville. Software engineering (ed.). America: Pearson Education Inc , 2011

    2011

  45. [53]

    Handbook on ontologies

    Steffen Staab and Rudi Studer. Handbook on ontologies. Springer Science & Business Media, 2013

    2013

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  47. [55]

    Unleashing chatgpt’s power: A case study on optimizing information retrieval in flipped classrooms via prompt engineering

    Mo Wang, Minjuan Wang, Xin Xu, Lanqing Yang, Dunbo Cai, and Minghao Yin. Unleashing chatgpt’s power: A case study on optimizing information retrieval in flipped classrooms via prompt engineering. IEEE Transactions on Learning Tech- nologies, 17:629–641, 2024

    2024

  48. [56]

    Chatgpt prompt patterns for improving code quality, refactoring, requirements elic- itation, and software design

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. Chatgpt prompt patterns for improving code quality, refactoring, requirements elic- itation, and software design. In Generative AI for Effective Software Development , pages 71–108. Springer, 2024

    2024

  49. [57]

    Fundamentals of codes, graphs, and iterative decoding, volume 714

    Stephen B Wicker and Saejoon Kim. Fundamentals of codes, graphs, and iterative decoding, volume 714. Springer Science & Business Media, 2002

    2002

  50. [58]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019

    1904 arXiv

  51. [59]

    Natural language pro- cessing for requirements engineering: A systematic mapping study.ACM Computing Surveys (CSUR), 54(3):1–41, 2021

    Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro. Natural language pro- cessing for requirements engineering: A systematic mapping study.ACM Computing Surveys (CSUR), 54(3):1–41, 2021

    2021

  52. [60]

    Univer- salner: Targeted distillation from large language models for open named entity recognition

    Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Univer- salner: Targeted distillation from large language models for open named entity recognition. arXiv preprint arXiv:2308.03279 , 2023. 75

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.