Pith. sign in

REVIEW 5 major objections 6 minor 124 references

Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding Theory-of-Mind helpers to a multi-agent system lets it extract richer, well-grounded software requirements through longer clarifying conversations.

desk verdict A well-designed multi-agent system for requirement refinement whose headline claim about accurately capturing stakeholder intents is not backed by any real stakeholder data. read the letter →

arxiv 2505.20973 v2 pith:SXG7HZRO submitted 2025-05-27 cs.SE cs.AI

Keheliya Gallaba , Ali Arabat , Dayi Lin , Mohammed Sayagh , Ahmed E. Hassan This is my paper
classification cs.SEcs.AI
keywords requirementsrefinementmulti-agentsystemstheory-of-mindfoundationmodelsLLM-as-a-judgelexicalrichnessintentalignmentconversationalAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that foundation models alone fail at requirements refinement because they rush to solutions and never probe the stakeholder's intent. To fix this, the authors build AlignMind, a multi-agent system whose helper agents infer the user's expertise, sentiment, and unstated goals after every exchange. Across 150 simulated scenarios, the refined requirements and natural-language workflows produced by AlignMind score higher with a panel of three FM judges than a directly prompted baseline FM. The refined requirements also contain about eight times more content words, and the system sustains longer clarifying conversations, but at roughly ten times the API calls and thirty times the tokens. The paper's claim is that this cost is justified because the resulting requirements more completely capture what stakeholders actually want.

What carries the argument

The central object is AlignMind, a multi-agent architecture in which a Router, a Requirement Refiner, a Workflow Generator, and a Workflow Refiner are supported by Theory-of-Mind helper agents. The ToM helpers—a Topics & Questions Decomposer, an Expertise Estimator, and a Sentiment Detector—run after each user utterance and feed inferred mental states back to the Requirement Refiner, which then tailors its next single question. This decomposition of the refinement task across specialised agents is what the paper credits for the longer, more grounded conversations and richer outputs.

What would settle it

Give the same 150 scenarios to human software engineers and ask them to choose, blind, between AlignMind's and the baseline's requirements documents; if humans do not prefer AlignMind at a similar rate, or if the FM judges' scores correlate poorly with human preferences beyond the 20-scenario calibration set, the central claim fails. A more direct check is to score AlignMind outputs with a judge model whose family is completely disjoint from the models used to generate the outputs and see whether the quality gap survives.

Watch

Extended reading notes

Core claim

The central claim is that adding Theory-of-Mind-style inference to a multi-agent architecture lets an FM-led system accurately capture stakeholder intents and convert them into both a detailed requirements document and an actionable step-by-step workflow. AlignMind's Router sends each query to a Requirement Refiner that asks one clarifying question at a time, guided by a Topics & Questions Decomposer plus Expertise and Sentiment helper agents; the refined requirements are then turned into a workflow by a separate generator, and a Workflow Refiner handles later edits. Compared with directly prompting one FM to refine requirements, AlignMind achieves statistically significant higher scores on five rubrics across three judge models, an eight-fold higher median lexical richness (266.5 vs 33 content words), and a median of 13 conversation rounds vs 4. The paper also reports that AlignMind's final requirements are grounded in the dialogue with no hallucination, and that the extra grounding costs a median of 74.5 FM calls and about 139,784 total tokens per task.

Load-bearing premise

The entire quality comparison rests on the assumption that the three-FM judge panel's rubric scores are a valid measure of how good refined requirements are; only 20 of the 150 scenarios were checked against human preferences (Cohen's kappa 0.685), so if the LLM judges favor AlignMind for reasons unrelated to real stakeholder value, the headline result would not generalize.

Editorial extensions

If this is right

  • If AlignMind's design is right, multi-agent refinement with ToM helpers is a practical way to turn vague stakeholder requests into complete, actionable requirements documents.
  • Requirements produced this way are richer (about eight times more content words) and more grounded in what the user says, reducing the risk of hallucinated requirements.
  • The workflow artifact doubles as a validation device: an unbuildable workflow signals gaps in the requirements.
  • Because the cost is roughly 10.6 times the API calls and 30 times the tokens, organisations will need prompt caching, compression, or cheaper models to make such refinement affordable.
  • The architecture is plug-and-play, so domain-specific ToM helpers can be added without changing the core refiner, opening the approach to non-software domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own evaluation leaves open whether human stakeholders would perceive the same quality gap; a natural test is a blind human preference study on the same 150 scenarios, comparing AlignMind outputs with baseline outputs among real developers.
  • Because some judge models share a model family with the generator, the claim would be stronger if the judge panel were fully disjoint from the models that produce the outputs—this is an extension the paper does not test.
  • The cost figures suggest AlignMind's value proposition will improve over time as inference prices fall; the paper hints at this but does not model the crossover point where the richer requirements become clearly worth the extra tokens.
  • The ToM helpers could be extended beyond expertise and sentiment to, say, risk tolerance or domain-specific regulatory knowledge; the paper frames the architecture as extensible but does not evaluate such extensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents AlignMind, an FM-powered multi-agent system for requirements refinement, augmented with theory-of-mind helper agents that estimate user expertise, sentiment, and topic coverage during multi-turn clarification dialogues. The system produces refined requirements and a natural-language workflow. The authors evaluate AlignMind against a single-FM baseline on 150 simulated scenarios using three FM judges, a lexical richness metric, conversation-round counts, a consistency/hallucination check, and cost measurements. They report significantly higher judge-assigned quality scores, roughly eight times higher lexical richness, longer conversations, comparable grounding, and a substantial cost overhead. The central claim is that AlignMind accurately captures stakeholder intents and requirements, with significantly higher output quality than direct prompting.

Significance. If the evaluation were fully validated, AlignMind would be a useful contribution to FM-based requirements engineering: it tackles a real phase (requirements refinement) that is under-addressed in the LLM4SE literature, and it combines multi-agent design with ToM-style helper agents in a concrete, extensible architecture. The paper's strengths include the detailed system description, the full set of prompts in the appendix (a reproducibility asset), the use of multiple judge models to mitigate self-preference, the iterative prototyping with six practitioners, and the explicit reporting of operational costs. The main weakness is that the evidence for the central claim is self-referential: simulated stakeholders, FM-generated rubrics, and FM judges from the same model families power the headline results, while the only human calibration covers 20 of 150 scenarios. The lexical richness result is also confounded with conversation length. These issues are load-bearing for the paper's main claim, so the current evidence supports a more limited reading: AlignMind outperforms a baseline under simulated conditions, according to LLM judges.

major comments (5)
  1. [Section 4.1 / RQ1.1] The central claim that AlignMind can 'accurately capture the intents and requirements of stakeholders' is not directly tested. All 150 evaluation scenarios use an FM role-playing as a human (prompt A.1.3) with FM-generated domains and intents, and the six human experts in Section 3.1 provided qualitative feedback on an early prototype, not ground-truth intents for these scenarios. The results therefore establish only that AlignMind outperforms a baseline when both interact with simulated stakeholders. To support the abstract's claim, the authors need either a human study with real stakeholders' intents (or a held-out set of real requirements) or a carefully qualified wording that restricts the conclusion to simulated settings.
  2. [Section 4 (RQ1.1), Table 1] The RQ1.1 quality comparison rests entirely on three FM judges from the same model families used to generate the artifacts and to derive the rubrics (GPT-4o, GPT-4o-mini, Llama3.3-70b; rubrics extracted in Appendix A.2.2). The only human calibration is a 20-scenario subset with Cohen's kappa of 0.685; this is a single agreement statistic and does not demonstrate that the judges are unbiased on the remaining 130 scenarios, nor that the FM-derived rubrics capture stakeholder-relevant quality rather than length or verbosity. A full human-judge validation on a larger sample, with per-rubric agreement, or an independent human evaluation of a random subset is needed before claiming 'significantly higher output quality.'
  3. [Section 4 (RQ1.2 and RQ1.3), Figures 9 and 10] The 8x lexical richness result (median 266.5 vs. 33 content words) is confounded with conversation length: AlignMind has a median of 13 rounds vs. 4 for the baseline, and 10.6x more FM calls. Longer conversations naturally produce more content words, so the 'requirement richness' metric is not an independent quality signal. The authors should report richness per round or per token, or otherwise control for output length, before claiming that AlignMind generates 'richer content' in the refined requirements.
  4. [Section 4 (RQ3)] The paper claims that AlignMind 'remains both cost-effective and efficient for requirements refinement tasks compared to manual requirement refinement' and cites absolute dollar amounts, but no manual baseline was measured in the study. The comparison to manual refinement is not part of the experimental design, and the reported $0.94-$31.40 figures do not include human effort, review time, or the cost of downstream errors caused by imperfect requirements. The authors should either remove this comparative claim or add an explicit human-effort baseline.
  5. [Section 4 (RQ2)] The hallucination/grounding check in RQ2 uses the same three FM judges from the same families as the RQ1 evaluation, and both systems achieve near-ceiling consistency scores (median 5/5). Given the self-referential evaluation setup, the conclusion that the generated requirements are 'grounded in dialogue' is not independently verified. Preference leakage or model-family bias could inflate consistency scores; a human-annotated consistency sample, or an external entailment-based metric, would provide stronger evidence for the no-hallucination claim.
minor comments (6)
  1. [Section 3.4.2] The text refers to the 'Topics & Decomposer Agent,' while Section 3.4.1 names the component 'Topics & Questions Decomposer Agent'; please unify the name.
  2. [Appendix A.1.1] The Domain Generator prompt contains the incomplete sentence 'Such domains involve creating automated workflows and should not include.' This appears truncated and should be completed or removed.
  3. [Figure 2] The box labeled 'X' among the ToM helper agents is never explained; please either label the placeholder explicitly or remove it.
  4. [Section 5] The sentence 'The flexibility provided by AlignMind plug-and-play architecture of AlignMind...' contains a duplicated phrase; please correct the wording.
  5. [General / Reproducibility] The paper does not state a data or code availability policy. Releasing the generated dialogues, judge prompts, and scoring scripts would substantially aid reproducibility and allow other researchers to assess the LLM-judge calibration concerns raised in the major comments.
  6. [References] Reference [9] is a non-archival blog post; consider replacing it with a peer-reviewed source for the claim that providing reasoning before scoring improves robustness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the FM-judge evaluation is anchored by a human-calibrated subset, so no central claim reduces to its own inputs by construction.

full rationale

The derivation chain is not circular under the required standard. AlignMind's central RQ1 claim is an empirical comparison against a baseline, measured by an FM judge panel using FM-derived rubrics; however, the paper independently calibrates that panel against human preferences on 20 scenarios with Cohen's kappa = 0.685 (Section 4.1), so the judge scores are not equivalent to the system's own outputs by construction. The rubrics themselves are generic quality criteria (intent identification, coverage, relevance, actionable workflow, realizability) and are manually reviewed in Section 6, not fitted parameters that force AlignMind to win. RQ2 applies a standard summarization consistency criterion to detect hallucinations, and RQ3 reports direct counts of calls and tokens; none of these metrics are defined in terms of the quantities they purport to predict. The use of FM-simulated stakeholders is an external-validity limitation rather than a circular step: it affects whether the result generalizes to real stakeholders, but it does not make the measured outcome equal to the input. Author self-citations [23,24] appear only in the discussion of costs and AI-native SE and are not load-bearing for the main result. No self-definitional, fitted-input, uniqueness-imported, or ansatz-via-citation pattern is present. I therefore find no significant circularity and assign score 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on unvalidated assumptions about LLM-judge validity and FM-simulated stakeholders. The only explicit free parameter is the per-topic question cutoff. No new entities are postulated.

free parameters (1)
  • questions per subtopic cutoff = 5
    The Requirement Refiner moves on after n=5 questions on a subtopic (Section 3.4.2). This hand-chosen threshold affects dialogue length, and thus the observed richness and cost differences.
assumptions (4)
  • domain assumption LLM judge scores correlate with human judgments of requirements quality
    RQ1.1 uses a panel of three LLM judges to score outputs; the rubrics are LLM-generated and the only human validation is 20 scenarios with Cohen's kappa 0.685. If this assumption fails, the main quality claim collapses.
  • domain assumption FM-as-human simulations adequately represent stakeholder behavior
    All 150 evaluation dialogues are produced by an LLM role-playing as a user with personas (casual, indecisive, rude). Real stakeholder ambiguity, domain expertise, and emotional nuance may not be captured. Section 4.1.
  • domain assumption Lexical richness is a valid proxy for requirement quality
    RQ1.2 counts unique content words as 'richness'. This metric is confounded with conversation length (AlignMind has 13 rounds vs 4), so it does not independently establish quality.
  • standard math Wilcoxon signed-rank test and Cliff's delta are appropriate for paired LLM-judge scores
    The paper uses standard nonparametric statistics; this is a routine assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement." pith.science (2026). https://pith.science/paper/SXG7HZRO

@misc{pith2026250520973,
  author       = {Pith},
  title        = {Pith review of: Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXG7HZRO}},
  note         = {Machine review of arXiv:2505.20973}
}
read the original abstract

Foundation Models (FMs) have shown remarkable capabilities in various natural language tasks. However, their ability to accurately capture stakeholder requirements remains a significant challenge for using FMs for software development. This paper introduces a novel approach that leverages an FM-powered multi-agent system called AlignMind to address this issue. By having a cognitive architecture that enhances FMs with Theory-of-Mind capabilities, our approach considers the mental states and perspectives of software makers. This allows our solution to iteratively clarify the beliefs, desires, and intentions of stakeholders, translating these into a set of refined requirements and a corresponding actionable natural language workflow in the often-overlooked requirements refinement phase of software engineering, which is crucial after initial elicitation. Through a multifaceted evaluation covering 150 diverse use cases, we demonstrate that our approach can accurately capture the intents and requirements of stakeholders, articulating them as both specifications and a step-by-step plan of action. Our findings suggest that the potential for significant improvements in the software development process justifies these investments. Our work lays the groundwork for future innovation in building intent-first development environments, where software makers can seamlessly collaborate with AIs to create software that truly meets their needs.

Figures

Figures reproduced from arXiv: 2505.20973 by the authors.

Figure 1
Figure 1. The Terminal-based User Interface (TUI) for [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. An overview of the proposed requirements refinement system, [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. An overview of the topic and question generation workflow. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: An overview of the evaluation of AlignMind [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: An overview of the output quality evaluation by a panel of FM judges (RQ1.1). [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: The overall score of AlignMind vs baseline, as judged by three different FMs. The rightmost plot (Median) shows the distribution of scores where, for each scenario, the median score from the three judge models is taken. −10 −5 0 5 10 15 20 25 30 35 7.4 7.6 7.8 8.0 8.2 …
Figure 7
Figure 7. Figure 7: Scatterplot showing the relative improvement of [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The score for each rubric of AlignMind and the baseline as judged by three different FMs. 4, and 5, with statistically significant differences (as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The requirements richness between AlignMind and baseline configurations. The median of the baseline approach is 33 and AlignMind is 266.5. Results [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The number of conversation rounds used for requirements refinement using [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The number of FM calls required in the two configurations for requirements refinement. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: The cost in terms of (a) prompt, (b) completion, and (c) total tokens needed for requirements refinement in the baseline configuration vs AlignMind. There is a notable monetary cost associated with using AlignMind compared to the baseline, as measured by metrics relat…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

124 extracted references · 37 canonical work pages

  1. [1]

    Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, and Mohammad Aliannejadi. 2024. Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (Merida, Mexico) (WSDM ’24). Association for Computing Machinery, New York, NY, USA, ...

    arXiv 2024

  2. [2]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2024. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? https://doi.org/10.48550/ARXIV.2408.05534

    arXiv doi:10.48550/arxiv.2408.05534 2024

  3. [3]

    Maryam Amirizaniani, Elias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. 2024. Can LLMs Reason Like Humans? Assessing Theory of Mind Reasoning in LLMs for Open-Ended Questions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24) . ACM, 34–44. https: //doi.org/10.1145/3627673.3679832

    arXiv 2024

  4. [4]

    Georgia Argyrou, Angeliki Dimitriou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. Automatic Generation of Fashion Images using Prompting in Generative Machine Learning Models. arXiv:2407.14944 [cs.CV] https://arxiv.org/abs/2407.14944

    work page Pith review arXiv 2024

  5. [5]

    Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2024. Advancing Requirements Engineering Through Generative AI: Assessing the Role of LLMs . Springer Nature Switzerland, 129–148. https://doi.org/10.1007/978-3-031-55642-5_6

    doi:10.1007/978-3-031-55642-5_6 2024

  6. [6]

    Yasharth Bajpai, Bhavya Chopra, Param Biyani, Cagri Aslan, Dustin Coleman, Sumit Gulwani, Chris Parnin, Arjun Radhakrishna, and Gustavo Soares. 2024. Let’s Fix this Together: Conversational Debugging with GitHub Copilot. In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) , Vol. 23. IEEE, 1–12. https: //doi.org/10.1109/vl/hcc60...

    arXiv 2024

  7. [7]

    Nathan Benaich and Ian Hogarth. 2024. State of AI report. (2024). https://www.stateof.ai/

    2024

  8. [8]

    Param Biyani, Yasharth Bajpai, Arjun Radhakrishna, Gustavo Soares, and Sumit Gulwani. 2024. RUBICON: Rubric- Based Evaluation of Domain-Specific Human AI Conversations. In Proceedings of the 1st ACM International Conference on AI-Powered Software (AIware ’24) . ACM. https://doi.org/10.1145/3664646.3664778

    arXiv 2024

Show all 124 references
  1. [9]

    Dylan Castillo. [n. d.]. Structured Outputs: Don’t Put the Cart Before the Horse . https://dylancastillo.co/posts/llm- pydantic-order-matters.html , Vol. 1, No. 1, Article . Publication date: August 2025. 26 Keheliya Gallaba, Ali Arabat, Dayi Lin, Mohammed Sayagh, and Ahmed E. Hassan

    2025

  2. [10]

    Guglielmo De Angelis, Alessio Ferrari, Stefania Gnesi, and Andrea Polini. 2018. Requirements elicitation and refinement in collaborative research projects. Journal of Software: Evolution and Process 30, 12 (Aug. 2018). https://doi.org/10. 1002/smr.1990

    2018

  3. [11]

    de Jong, J

    E. de Jong, J. van de Pol, and J. Hooman. [n. d.]. Refinement in requirements specification and analysis: a case study. In Proceedings Seventh IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ECBS

  4. [12]

    Damian and Didar Zowghi

    Daniela E. Damian and Didar Zowghi. 2003. RE challenges in multi-site software development organisations. Require- ments Engineering 8, 3 (July 2003), 149–160. https://doi.org/10.1007/s00766-003-0173-1

    2003 doi

  5. [13]

    Ramtin Ehsani, Sakshi Pathak, and Preetha Chatterjee. 2025. Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution. https://doi.org/10.48550/ARXIV.2501.11709

    2025 doi arXiv

  6. [14]

    Elad Gil. [n. d.]. Cost of 1M tokens has dropped from $180 to $0.75 in ∼18 months (240X cheaper!) . arXiv:Twitter https://x.com/eladgil/status/1827521805755806107

  7. [15]

    Khaled El Emam. 1999. Benchmarking Kappa: Interrater Agreement in Software Process Assessments. Empirical Software Engineering 4, 2 (1999), 113–133. https://doi.org/10.1023/a:1009820201126

    1999 doi

  8. [16]

    Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych. 2024. InferAct: Inferring Safe Actions for LLM-Based Agents Through Preemptive Evaluation and Human Feedback. arXiv preprint arXiv:2407.11843 (2024). https://doi.org/10. 48550/arXiv.2407.11843

    2024 doi

  9. [17]

    Mohamad Fazelnia, Mehdi Mirakhorli, and Hamid Bagheri. 2024. Translation Titans, Reasoning Challenges: Satisfiability-Aided Language Models for Detecting Conflicting Requirements. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (A...

    2024

  10. [18]

    Méndez Fernández, S

    D. Méndez Fernández, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetrò, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Männistö, M. Nayabi, M. Oivo, B. Penzenstadler, D. Pfahl, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spinola, A. Tuzcu, J. L. de ...

    2016 doi

  11. [19]

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling Synthetic Data Creation with 1, 000, 000, 000 Personas. https://doi.org/10.48550/ARXIV.2406.20094

    2024 doi arXiv

  12. [20]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao

  13. [21]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2024. A Survey on LLM-as-a-Judge. https://doi.org/10.48550/ARXIV.2411.15594

    2024 doi arXiv

  14. [22]

    Som Gupta and S. K Gupta. 2019. Abstractive summarization: An overview of the state of the art. Expert Systems with Applications 121 (May 2019), 49–65. https://doi.org/10.1016/j.eswa.2018.12.011

    2019 doi

  15. [23]

    Ahmed E. Hassan, Dayi Lin, Gopi Krishnan Rajbahadur, Keheliya Gallaba, Filipe Roseiro Cogo, Boyuan Chen, Haoxiang Zhang, Kishanthan Thangarajah, Gustavo Oliva, Jiahuei (Justina) Lin, Wali Mohammad Abdullah, and Zhen Ming (Jack) Jiang. 2024. Rethinking Software Engineering in t...

    2024

  16. [24]

    Hassan, Gustavo A

    Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin, Boyuan Chen, Zhen Ming, and Jiang. 2024. Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap. https://doi.org/10.48550/ARXIV.2410.06107 arXiv.org perpetual, non-exclusive license

    2024 doi

  17. [25]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (March 2023), 1–38. https://doi.org/10.1145/3571730

    2023 doi

  18. [26]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computationa...

    2023 doi

  19. [27]

    Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, and Hyunwoo Kim. 2024. Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024 doi

  20. [28]

    Siegel, Nitya Nadgir, and Arvind Narayanan

    Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. AI Agents That Matter. https://doi.org/10.48550/ARXIV.2407.01502 arXiv.org perpetual, non-exclusive license. , Vol. 1, No. 1, Article . Publication date: August 2025. Towards Conversa...

    2024 doi arXiv

  21. [29]

    Conor Kelly. 2024. Prompt Caching. https://humanloop.com/blog/prompt-caching. [Accessed 20-10-2024]

    2024

  22. [30]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling Declarative Language Model Calls into Self...

    2023 doi arXiv

  23. [31]

    Ko, Thomas D

    Amy J. Ko, Thomas D. LaToza, and Margaret M. Burnett. 2015. A Practical Guide to Controlled Experiments of Software Engineering Tools with Human Participants. Empirical Software Engineering 20, 1 (Feb. 2015), 110–141

    2015

  24. [32]

    Tom Kocmi and Christian Federmann. 2023. Large Language Models Are State-of-the-Art Evaluators of Translation Quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation , Mary Nurmi- nen, Judith Brenner, Maarit Koponen, Sirkku Lat...

    2023

  25. [33]

    Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural Text Summarization: A Critical Evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer...

    2019 doi

  26. [34]

    Kristopher Kyle. 2019. Measuring lexical richness. InThe Routledge handbook of vocabulary studies. Routledge, 454–476

    2019

  27. [35]

    Lahiri, and Siddhartha Sen

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 919–931. h...

    2023

  28. [36]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2024. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. https://doi.org/...

    2024 doi

  29. [37]

    Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025. Preference Leakage: A Contamination Problem in LLM-as-a-judge. arXiv:2502.01534 [cs.LG] https://arxiv.org/abs/2502.01534

    2025

  30. [38]

    Feng-Lin Li, Jennifer Horkoff, Alexander Borgida, Giancarlo Guizzardi, Lin Liu, and John Mylopoulos. 2015. From Stakeholder Requirements to Formal Specifications Through Refinement . Springer International Publishing, 164–180. https://doi.org/10.1007/978-3-319-16101-3_11

    2015 doi

  31. [39]

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  32. [40]

    Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, and Jaime Teevan. 2024. Interpretable User Sa...

    2024

  33. [41]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 doi arXiv

  34. [42]

    Dipeeka Luitel, Shabnam Hassani, and Mehrdad Sabetzadeh. 2023. Using Language Models for Enhancing the Com- pleteness of Natural-Language Requirements . Springer Nature Switzerland, 87–104. https://doi.org/10.1007/978-3-031- 29786-1_7

    2023 doi

  35. [43]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv preprint arXiv:2306.08568 (2023)

    2023 arXiv

  36. [44]

    Marcos Macedo, Yuan Tian, Filipe Cogo, and Bram Adams. 2024. Exploring the Impact of the Output Format on the Evaluation of Large Language Models for Code Translation. In Proceedings of the 2024 IEEE/ACM First International , Vol. 1, No. 1, Article . Publication date: August 2...

    2024

  37. [45]

    Antonis Maronikolakis, Ana Peleteiro Ramallo, Weiwei Cheng, and Thomas Kober. 2024. What should I wear to a party in a Greek taverna? Evaluation for Conversational Agents in the Fashion Domain. arXiv:2408.08907 [cs.IR] https://arxiv.org/abs/2408.08907

    2024 arXiv

  38. [46]

    Lloyd Montgomery, Davide Fucci, Abir Bouraffa, Lisa Scholz, and Walid Maalej. 2022. Empirical research on re- quirements quality: a systematic mapping study. Requirements Engineering 27, 2 (Feb. 2022), 183–209. https: //doi.org/10.1007/s00766-021-00367-z

    2022 doi

  39. [47]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proc. ACM Softw. Eng. 1, FSE, Article 103 (July 2024), 23 pages. https://d...

    2024 doi

  40. [48]

    Bashar Nuseibeh and Steve Easterbrook. 2000. Requirements Engineering: A Roadmap. In Proceedings of the Conference on The Future of Software Engineering (ICSE00) . ACM. https://doi.org/10.1145/336512.336523

    2000

  41. [49]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076 [cs.CL] https://arxiv.org/abs/2404.13076

    2024 arXiv

  42. [50]

    George Pólya. 1945. How to solve it: A new aspect of mathematical method . Vol. 34. Princeton university press

    1945

  43. [51]

    Suzanne Robertson and James Robertson. 2012. Mastering the Requirements Process: Getting Requirements Right (3rd ed.). Addison-Wesley Professional

    2012

  44. [52]

    2024.Requirements Engineering Using Generative AI: Prompts and Prompting Patterns

    Krishna Ronanki, Beatriz Cabrero-Daniel, Jennifer Horkoff, and Christian Berger. 2024.Requirements Engineering Using Generative AI: Prompts and Prompting Patterns . Springer Nature Switzerland, 109–127. https://doi.org/10.1007/978-3- 031-55642-5_5

    2024 doi

  45. [53]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. SpecRover: Code Intent Extraction via LLMs. https: //doi.org/10.48550/ARXIV.2408.02232 Creative Commons Attribution 4.0 International

    2024 doi arXiv

  46. [54]

    Sarah Santos, Travis Breaux, Thomas Norton, Sara Haghighi, and Sepideh Ghanavati. 2024. Requirements Satisfiability with In-Context Learning. In 2024 IEEE 32nd International Requirements Engineering Conference (RE) , Vol. 33. IEEE, 168–179. https://doi.org/10.1109/RE59067.2024.00025

    2024

  47. [55]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (Jan. 2024), 85–105. https://doi.org/10.1109/tse.2023.3334955

    2024

  48. [56]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou

  49. [57]

    Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. 2025. MuMA-ToM: Multi-modal Multi-Agent Theory of Mind. Proceedings of the AAAI Conference on Artificial Intelligence 39, 2 (April 2025), 1510–1519. https://doi.org/10.1609/aaai.v39i2.32142

    2025 doi

  50. [58]

    Riad Sonbol, Ghaida Rebdawi, and Nada Ghneim. 2022. The Use of NLP-Based Text Representation Techniques to Support Requirement Engineering Tasks: A Systematic Mapping Review. IEEE Access 10 (2022), 62811–62830. https://doi.org/10.1109/access.2022.3182372

    2022

  51. [59]

    Joan Torruella and Ramon Capsada. 2013. Lexical Statistics and Tipological Structures: A Measure of Lexical Richness. Procedia - Social and Behavioral Sciences 95 (Oct. 2013), 447–454. https://doi.org/10.1016/j.sbspro.2013.10.668

    2013 doi

  52. [60]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. https://doi.org/10.48550/ARXIV.2404.18796

    2024 doi arXiv

  53. [61]

    Qiaosi Wang and Ashok K. Goel. 2022. Mutual Theory of Mind for Human-AI Communication. CoRR abs/2210.03842 (2022). https://doi.org/10.48550/ARXIV.2210.03842 arXiv:2210.03842

    2022 doi arXiv

  54. [62]

    Qiaosi Wang, Koustuv Saha, Eric Gregori, David Joyner, and Ashok Goel. 2021. Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant. In Proceedings of the 2021 CHI Conference on Human Factors in Co...

    2021

  55. [63]

    Zhitao Wang, Wei Wang, Zirao Li, Long Wang, Can Yi, Xinjie Xu, Luyang Cao, Hanjing Su, Shouzhi Chen, and Jun Zhou. 2024. XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance Testing with Large Language Model. arXiv:2401.02705 [cs.AI] https://arxiv.org/a...

    2024 arXiv

  56. [64]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 52632–52657. h...

    2024

  57. [65]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt. 2024. ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design . Springer Nature Switzerland, , Vol. 1, No. 1, Article . Publication date: Augus...

    2024 doi

  58. [66]

    Alex Wilf, Sihyun Shawn Lee, Paul Pu Liang, and Louis-Philippe Morency. 2023. Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities. ArXiv abs/2311.10227 (2023). https://doi.org/10. 48550/arXiv.2311.10227

    2023 doi arXiv

  59. [67]

    Jie Jw Wu and Fatemeh H. Fard. 2025. HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent. ACM Transactions on Software Engineering and Methodology (TOSEM) (Jan. 2025). https://doi.org/10.1145/3715109

    2025 doi

  60. [68]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. 2024. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? a...

    2024 arXiv

  61. [69]

    Letsholo, Muideen A

    Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J. Letsholo, Muideen A. Ajagbe, Erol-Valeriu Chioasca, and Riza T. Batista-Navarro. 2021. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. Comput. Surveys 54, 3 (April 2021), 1–41. https:...

    2021 doi

  62. [70]

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Zhengyang Qi, Haofei Yu, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. In International Conference on Learn...

    2024

  63. [74]

    The user

    Topic 2 ... N. Topic n # Output: A.1.2 Intent Generator. , Vol. 1, No. 1, Article . Publication date: August 2025. 30 Keheliya Gallaba, Ali Arabat, Dayi Lin, Mohammed Sayagh, and Ahmed E. Hassan You are a helpful assistant capable of completing an intent that users might want ...

    2025

  64. [75]

    The maxim of quantity, where one tries to be as informative as one possibly can and gives as much information as is needed and no more.↩→

  65. [76]

    The maxim of quality, where one tries to be truthful and does not give information that is false or that is not supported by evidence.↩→

  66. [77]

    The maxim of relation, where one tries to be relevant and says things that are pertinent to the discussion.↩→

  67. [78]

    USER" for the human developer and

    The maxim of manner is when one tries to be as clear, as brief, and as orderly as one can in what one says and where one avoids obscurity and ambiguity.↩→ As the maxims stand, there may be an overlap, as regards the length of what one says, between the maxims of quantity and m...

    2025

  68. [79]

    Rubric - Represents the name of the rubric being scored

  69. [80]

    Justification - Provide a justification of the conversation, requirements, and workflow with respect to each rubric.↩→

  70. [81]

    rubrics": {

    Label - The label can be one of the following options: Strongly Disagree/ Disagree/Neutral/Agree/Strongly Agree.↩→ - Provide a label for each rubric based on the justification step. <CONVERSATION> {{conversation}} </CONVERSATION> <REQUIREMENTS> {{requirement}} </REQUIREMENTS> ...

    2025

  71. [82]

    **Query:** The user's request or inquiry

  72. [83]

    **Response:** The RequirementRefiner's response. , Vol. 1, No. 1, Article . Publication date: August 2025. 38 Keheliya Gallaba, Ali Arabat, Dayi Lin, Mohammed Sayagh, and Ahmed E. Hassan

    2025

  73. [84]

    No requirements for now

    **Requirements:** The current state of requirements, which can either be "No requirements for now" or a specific set of requirements.↩→

  74. [85]

    No workflow for now

    **Workflow:** The current state of the workflow, which can either be "No workflow for now" or a specific workflow process.↩→ Follow the instructions below to determine which agent to route the query to:

  75. [86]

    If the query is answering the RequirementRefiner's response, about clarifying, or changing the requirements:↩→ - Route the query to the agent **RequirementRefiner**

  76. [87]

    No requirements for now

    If the query involves modifying the workflow: - Check the current state of the requirements: - If requirements have been produced (i.e., not "No requirements for now."): - Route the query to the agent **WorkflowRefiner**. - If requirements have not been produced: - Respond wit...

  77. [88]

    I need to update the requirements for the project

    **If the query does not pertain to modifying requirements or workflow:** - Respond with: **RequirementRefiner**. ### Example Scenarios: - **User Query:** "I need to update the requirements for the project." - **Route to:** "RequirementRefiner". - **User Query:** "Can we change...

    2025

  78. [89]

    Response: your response to the user

  79. [90]

    No requirements for now

    Requirements: the generated requirements, if they are ready. Otherwise, return "No requirements for now.".↩→

  80. [91]

    No workflow for now

    Workflow: the workflow required to achieve the generated requirements. Otherwise, return "No workflow for now.".↩→ Return a valid JSON conforming to the following TypeScript type definition without backticks or additional text:↩→ ``` { "response": string, "requirements": strin...

    2025

  81. [92]

    - Do not consider the following types of questions: * Off-topic questions; * Clarifying questions; * Seeking additional information; * Similar questions with different wordings

    Answered Question - Determine whether the user's response answered your previous question. - Do not consider the following types of questions: * Off-topic questions; * Clarifying questions; * Seeking additional information; * Similar questions with different wordings. - Questi...

  82. [93]

    Area Coverage - Explain your reasoning and progress within the current area of conversation. - Reason: evaluate if you have gathered enough information based on **exclusively** the user's previous responses on the current area.↩→ Always end your reasoning with whether you need...

  83. [94]

    - Prompt the user for a response for unanswered questions only when covering the same area.↩→ - If the query is clear, provide an answer and a relevant follow-up question

    Response - Keep your responses and questions focused on the current area. - Prompt the user for a response for unanswered questions only when covering the same area.↩→ - If the query is clear, provide an answer and a relevant follow-up question. - If the query is unclear, ask ...

  84. [95]

    last_answered_question

    Current question - Extract from the response the question being currently asked. **The Area under investigation now is {{area}}**. ## Guidance Questions {{questions}} ## Feedback and Instructions about the User {{tom_helpers_sub_prompts}} # Output , Vol. 1, No. 1, Article . Pu...

    2025

  85. [96]

    They should be separated by "\n".↩→ - You should include your adjustments according to the area consistency checker evaluations here.↩→

    Response - Initially, you generate three possible sets of areas similar to the example provided above. They should be separated by "\n".↩→ - You should include your adjustments according to the area consistency checker evaluations here.↩→

  86. [97]

    response

    Final Area Set - Include the final area set here. <USER_INTENT> {{user_intent}} </USER_INTENT> Return a valid JSON conforming to the following TypeScript type definition without backticks:↩→ ``` { "response": string, "final_revised_area_set": string[] } ``` Output only valid J...

    2025

  87. [98]

    Analyze each area step by step

  88. [99]

    Determine a pool of key questions to achieve full coverage of each area

  89. [100]

    areas": {

    For each area, select the most relevant questions (up to 5). Note that you can have more than three questions for each area. <USER_INTENT> {{user_intent}} </USER_INTENT> <AREAS> {{areas}} </AREAS> Return a valid JSON conforming to the following TypeScript type definition witho...

    2025

  90. [101]

    - Do the user's messages involve any references to prior expertise? yes/no

    Answer these cue-related questions: - Do the user's messages involve any expressions of goodwill? yes/no. - Do the user's messages involve any references to prior expertise? yes/no. - Do the user's messages involve any organization of information? yes/no. - Do the user's messa...

  91. [102]

    - Long words count: {{long_words_count}}

    Here are some metrics of the user's query that may help estimate their expertise: - Word count: {{words_count}}. - Long words count: {{long_words_count}}. - Negation count: {{negation_count}}

  92. [103]

    Based on the answers above (1 and 2), provide comprehensive reasoning for categorizing the user's expertise level as Novice, Intermediate, or Expert.↩→

  93. [104]

    reason": string,

    Determine the expertise level that applies to the user. # Dialogue: {{dialogue}} Return a valid JSON conforming to the following TypeScript type definition without backticks:↩→ ``` { "reason": string, "expertise": string } ``` Output only valid JSON. Do **not** output any deli...

    2025

  94. [105]

    User visits a website

  95. [106]

    API call to fetch the user's browsing history

  96. [107]

    API call to a machine learning model to predict the user's interests based on their browsing history.↩→ , Vol. 1, No. 1, Article . Publication date: August 2025. Towards Conversational Development Environments 47

    2025

  97. [108]

    If a specific interest is identified, API call to fetch relevant advertisements; if not, API call to fetch↩→ general advertisements

  98. [109]

    Automated Job Application Workflow:

    API call to display the fetched advertisements. Automated Job Application Workflow:

  99. [110]

    User uploads their resume to a job search platform

  100. [111]

    API call to a machine learning model to extract skills and experience from the resume.↩→

  101. [112]

    API call to match the user with relevant job listings

  102. [113]

    For each matched job listing, make an API call to submit the user's application

  103. [114]

    Cyclic Weather Monitoring Workflow:

    API call to notify the user of the application status for each job. Cyclic Weather Monitoring Workflow:

  104. [115]

    The user sets up a routine weather check for their location

  105. [116]

    API call to fetch current weather data

  106. [117]

    If the weather condition is severe, API calls to alert the user

  107. [118]

    API call to a delay service to wait for a specified period (e.g., one hour)

  108. [119]

    Interactive Story Workflow:

    Repeat the process from step 2. Interactive Story Workflow:

  109. [120]

    The user starts an interactive story on an app

  110. [121]

    API call to present the user with a story scenario and multiple choices

  111. [122]

    The user makes a choice

  112. [123]

    Depending on the user's choice, make an API call to fetch and present the next scenario.↩→

  113. [124]

    Requirements: {{ requirements_document }} Output: 1

    Repeat the process from step 3 until the story ends. Requirements: {{ requirements_document }} Output: 1. Step 1. 2. Step 2. ... N. Step N. Ensure each step is clear, straightforward, and sequentially numbered, with API calls included. No leading/trailing titles or explanation...

    2025

  114. [2000]

    IEEE Comput

    (ECBS-00). IEEE Comput. Soc, 290–298. https://doi.org/10.1109/ecbs.2000.839888

    2000

  115. [2023]

    In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML’23)

    Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML’23). JMLR.org, Article 1291, 18 pages

  116. [2024]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24)

    Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24) . ACM, 1–13. https: //doi.org/10.1145/3597503.3608134

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.