Pith. sign in

REVIEW 3 major objections 4 minor 36 references

This paper formalizes text-to-story alignment—checking whether interview transcript chunks support each user story—and shows that an LLM judge can automate the check with 0.86 macro-F1 across four annotated datasets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 11:05 UTC pith:TE7D5YGA

load-bearing objection The framework is genuinely useful, but the headline F1 is measured on a curated test set with moderate label agreement; treat the numbers as provisional. the 3 major comments →

arxiv 2510.08622 v2 pith:TE7D5YGA submitted 2025-10-08 cs.CL cs.SE

Automated Alignment between Elicitation Interviews and Requirements

Francesco Dente , Fabiano Dalpiaz , Paolo Papotti This is my paper
classification cs.CL cs.SE
keywords text-to-story alignmentuser storiesrequirements elicitationLLM-as-a-judgeembedding blockingcorrectnesscompletenesssource grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tackles the problem of verifying that user stories derived from a stakeholder interview actually reflect what was said. It defines text-to-story alignment as a matching problem between transcript chunks and stories, and introduces two set-level metrics: correctness (the share of stories supported by the transcript) and completeness (the share of the transcript covered by stories). Across four annotated datasets, an LLM-based judge achieves 0.86 macro-F1 on chunk-story pairs, while an embedding-based blocking step cuts token cost by up to 3x without losing recall. The authors also show that completeness, not correctness, is the harder property for generated stories, and that their metrics are sensitive to semantic mismatch. If the result holds, requirements quality can be evaluated against source evidence automatically, complementing style-based checks.

Core claim

The central claim is that text-to-story alignment can be formalized as a chunk-level matching task and evaluated automatically with near-human accuracy. The paper introduces the T2SA task: given a transcript segmented into overlapping three-turn chunks and a set of user stories, decide for each pair whether the chunk supports the story. From these binary decisions, it derives correctness (stories with at least one supporting chunk) and completeness (chunks covered by at least one story). The empirical finding is that an LLM judge prompted with few-shot examples matches manual annotations with 0.86 macro-F1 on held-out pairs, outperforming cross-encoders and bi-encoders, and that an embedding

What carries the argument

The mechanism is the pair-level support relation y(ci, sj), decided by a matcher X. Chunking uses a sliding window of three speaker turns to create overlapping chunks, preserving context while localizing evidence. The paper's default matcher is an LLM judge prompted to output 1 if a knowledgeable reader could infer the story from the chunk alone. To make the Cartesian product practical, an embedding-based blocker keeps for each story only its top-K most similar chunks, implicitly labeling the rest as 0. The two metrics, Correctness and Completeness, aggregate these pairwise decisions into set-level measures that quantify how well the stories are grounded in the interview.

Load-bearing premise

The gold labels used to measure the 0.86 macro-F1 were created through a similarity-based sampling protocol with only moderate inter-annotator agreement (κ=0.470), so the reported accuracy may not generalize to arbitrary chunk-story pairs in full transcripts.

What would settle it

Take one full interview transcript, sample chunk-story pairs uniformly at random (not by embedding similarity), have three independent experts label them with a consensus rule, and compute the same LLM judge's macro-F1 on that sample. If the score falls well below 0.86—say, below 0.7—the reported quality is an artifact of easy, retrievable pairs; if it stays near 0.86, the result generalizes.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the 0.86 macro-F1 holds generally, requirements engineers can obtain an automatic, traceable first pass over story sets, with each positive match pointing to the transcript excerpt that justifies it.
  • Embedding blocking at recall 0.9 uses only about one third of the tokens, so the approach scales to long interviews and large story collections.
  • The finding that larger LLM generators produce more complete story sets suggests scaling generation size is a concrete lever for reducing missed requirements.
  • Correctness being easier to satisfy than completeness indicates evaluation resources should focus on coverage, not just validity.
  • The sensitivity to swapped transcripts confirms the metrics measure grounding rather than stylistic quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same chunk-story alignment could be repurposed as a traceability link between requirements and evidence for compliance or audit purposes, beyond generation evaluation.
  • A natural extension is to calibrate the LLM judge against expert labels from real industrial elicitation sessions, since the current datasets are mostly student projects.
  • One could test whether combining this source-grounding metric with style-based quality filters removes stories that are well-formed but unsupported, improving the precision of a requirements candidate list.
  • Since completeness is defined as chunk coverage, it may conflate one repeated idea across turns with multiple distinct requirements; a variant that clusters chunks semantically before counting coverage might better approximate requirements-level completeness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper formalizes text-to-story alignment (T2SA): interview transcripts are segmented into overlapping three-turn chunks, chunk–story pairs are labeled as supporting or not, and the labels are aggregated into two metrics, Correctness (Eq. 1) and Completeness (Eq. 2). The method is instantiated with bi-encoders, cross-encoders, and LLM judges, plus an embedding-based Top-K blocker to reduce the number of pairs evaluated. Experiments on 17 datasets, four of them manually annotated, report Llama3.3-70B at 0.859 macro-F1 on the labeled pairs and a roughly 3x token reduction from blocking. The authors argue that TEXT2STORIES offers a scalable, source-grounded complement to existing user-story quality criteria such as QUS.

Significance. If the headline results hold, this is a useful contribution to requirements engineering and NLP: it formalizes a previously informal activity, provides a concrete benchmark, and ships public code and prompts. The two metrics are interpretable, and the swapped-source robustness check in Table 6 is a valuable sanity test. Credit is due for the honest Limitations section and for making the code and public-dataset experiments reproducible. However, the central 0.86 macro-F1 is measured on a small, similarity-sampled, only moderately reliable gold set, so the contribution's validity is currently conditional on addressing those measurement issues.

major comments (3)
  1. [§5.2, Tables 2–3] The annotated test sets are not a random sample of P = C × S. The protocol in §5.2 selects the top-5 embedding-similar chunks per story, extends the ranking until at least two positives are found, and adds chunks the annotator already knew were supportive. This creates a balanced sample of near-miss negatives and retrievable positives, while excluding positives that have low embedding similarity. Since Tables 3–6 and the abstract's 0.86 macro-F1 are all computed on this curated distribution, the numbers do not estimate performance over the full Cartesian product or over arbitrary deployment pairs. Please annotate a random sample (or the full product for one dataset) to quantify the selection bias, or explicitly re-scope the claim as being conditional on the retrieval-based sampling protocol.
  2. [§5.2, App. B.4] Inter-annotator agreement is Fleiss κ=0.470 (moderate) on only 10 user stories, and the initial labels were produced by the first author with cross-checking by the other two on a subset. The support relation is therefore not stable enough to be a strong gold standard. With a reported 0.86 macro-F1, even a modest label-error rate can substantially move measured performance. Please report per-annotator macro-F1 on the re-annotated subset, provide adjudication statistics, and ideally a label-noise analysis (e.g., recomputing F1 after flipping a random fraction of labels). Without this, the 'manually labeled' premise is weaker than the paper suggests.
  3. [§6.3, Fig. 5] The blocking evaluation defines ground truth as Qwen-32B positives, while the same model family is used as the LLM judge, as the generator of synthetic labels for fine-tuning the blocker, and as the downstream matcher. This coupling means the reported token reductions and the fine-tuning gains may partly reflect one model matching itself rather than matching human-validated alignment. The four human-annotated test sets could provide an independent check on blocking recall; without that, the scalability claim (3x reduction) is not established for human-validated alignment.
minor comments (4)
  1. [Table 3, App. B.1] Cross-encoder scores use the best threshold per dataset; App. B.1 shows a drop of up to 0.085 after calibration. Since Table 3 is the main comparison table, please show calibrated values in the main table or label the 'best threshold' columns more prominently. The manuscript already discloses this, so I treat it as a presentation issue.
  2. [Figures 3–4] The horizontal axes are sorted by model size, but the ordering is not explicitly annotated. Adding model-size labels or an arrow would make the completeness-scaling trend easier to read.
  3. [Limitations, §4.1] The stride-1 overlapping chunks can inflate Completeness because one requirement-bearing turn appears in multiple chunks. The Limitations section acknowledges this, but the audit in App. B.5 would be stronger if it quantified how many chunks are shared between adjacent windows.
  4. [App. B.4] The κ estimate is based on only 10 stories. Please report a confidence interval and describe the random selection procedure (seed, strata) more fully in the appendix.

Circularity Check

1 steps flagged

Core 0.86 F1 rests on human labels and is not circular; circularity is partial in the blocking evaluation, where Qwen-32B supplies both the pseudo-ground-truth and the synthetic training labels for the fine-tuned blocker, making the efficiency/recall claims internal-consistency results.

specific steps
  1. fitted input called prediction [Section 6.3 (Blocking Efficiency), Figure 5 and fine-tuning paragraph]
    "Since full annotations of all Cartesian products are unavailable, we approximate ground truth with Qwen-32B. ... To obtain training data without gold annotations, we generate synthetic labels: for each of the 15 private interviews, we compute the full |C| × |S| Cartesian product and annotate pairs using Qwen-32B."

    Blocking recall is measured against positives identified by Qwen-32B, and the fine-tuned blocker is trained on synthetic labels generated by the same Qwen-32B model. Thus the reported recall and fine-tuning gains quantify agreement with the label-generating model, not with the human support labels that define T2SA. The claimed conclusion that 'blocking preserves alignment quality' is therefore an internal-consistency result: the evaluation target and the training-signal source are the same model. Table 3 shows Qwen-32B itself is imperfect against human labels (macro-F1 0.841), so recall with respect to Qwen-32B does not by itself establish preservation of human-defined alignment.

full rationale

The central claim—an LLM matcher reaches 0.86 macro-F1 on chunk–story alignment—is evaluated against manually created support labels (Section 5.2, Table 3). Those labels are human-produced, so measuring a matcher against them is not circular. The annotation protocol is similarity-driven and inter-annotator agreement is only moderate (Fleiss κ=0.470, Appendix B.4), and the Limitations admit that matching was annotated by one author and cross-checked on a subset; these are representativeness and reliability concerns, not construction-level circularity. The one self-referential part is the blocking evaluation: Qwen-32B is used as pseudo-ground-truth while also being the source of synthetic labels for fine-tuning the embedding blocker. Because the paper explicitly labels this an approximation, the blocking numbers are best read as an internal-consistency check of whether embeddings retrieve Qwen-32B positives, not as externally validated evidence that blocking preserves human-annotated alignment. Reuse of prior datasets by co-authors (Spijkman et al. 2023; Dalpiaz et al. 2020) is data reuse rather than load-bearing self-citation. Overall, the core F1 result is independent, but the blocking-quality claim is partially circular relative to its own pseudo-ground-truth.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The framework depends less on mathematical axioms than on four empirical/design assumptions: stable human judgments of 'support', the three-turn chunking unit, embedding similarity as a retrieval proxy, and LLM judge outputs as uncalibrated human approximations. Three design choices (chunk size, per-dataset thresholds, 50-story cap) are free parameters that directly affect reported numbers.

free parameters (5)
  • Chunking window (speaker turns) = 3 turns, stride 1
    Chosen in advance to keep annotation feasible (Section 4.1); Table 5 shows 2-turn windows improve blocking efficiency, so the 3-turn choice is not independently justified as optimal.
  • Cross-encoder decision threshold = best threshold per test dataset
    Table 3 reports scores at the best per-dataset threshold; Appendix B.1 shows calibrated thresholds drop (e.g., Qwen3-Reranker-8B: 0.829 to 0.744 on Public Interview), so headline numbers are optimistic.
  • Blocking Top-K = K tuned to reach target recall (0.9/0.95)
    Efficiency claims hold only after selecting K per recall target; no fixed operating point is recommended.
  • LLM story generation cap = 50 stories
    Human story sets are not capped; the human-vs-LLM completeness comparison in Figure 4 is confounded by the cap, as the authors note in Appendix B.5.
  • Synthetic label generation for embedding fine-tuning = Qwen-32B labels, negative downsampling
    The fine-tuned blocker is trained on oracle labels from the same model family used as judge (Section 6.3), so its gains are relative to that oracle.
axioms (5)
  • domain assumption Support relation y(ci,sj) is a well-defined semantic judgment with acceptable annotator agreement.
    The whole task is defined over an undefined 'support' relation; measured agreement is only κ=0.470 on a 10-story sample (Appendix B.4), so the ground truth is contested.
  • ad hoc to paper Three consecutive speaker turns form a coherent unit for deciding support.
    Adopted for feasibility (Section 4.1); results are sensitive to chunk size (Table 5) and no semantic validity check is provided.
  • domain assumption Embedding similarity retrieves the relevant candidates for support.
    Used both in the annotation protocol (Section 5.2) and in blocking (Section 4.3); if embeddings miss semantically supporting chunks, gold labels and recall estimates are biased.
  • domain assumption LLM judge binary outputs approximate human labels without calibration.
    The judge is prompted with few-shot examples (Appendix A.2) and compared against human labels, but no analysis of judge bias, prompt sensitivity, or variance is given.
  • standard math Basic set cardinality arithmetic in Eqs. 1-2.
    The metrics are proportions over finite sets; unproblematic.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Alignment between Elicitation Interviews and Requirements." pith.science (2026). https://pith.science/paper/TE7D5YGA

@misc{pith2026251008622,
  author       = {Pith},
  title        = {Pith review of: Automated Alignment between Elicitation Interviews and Requirements},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TE7D5YGA}},
  note         = {Machine review of arXiv:2510.08622}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Software requirements are derived from a variety of elicitation techniques, many of which have a conversational nature, like interviews. However, evaluating whether those derived requirements faithfully reflect the stakeholders' needs remains a challenging manual task. In this paper, we formalize the task of aligning the transcript of an interview with a collection of requirements represented as user stories. We propose two heuristic metrics for alignment, called (i) requirements faithfulness: the proportion of stories supported by the transcript, and (ii) interview coverage: the proportion of transcript supported by at least one story. Then, we run experiments with large language models and embedding models that assess the ability of evaluating these metrics automatically. Experiments over four datasets show that an LLM-based solution achieves 0.86 macro-F1 on manually labeled chunk-story pairs. We also show how embedding models can be used as blockers to make the approach more scalable. This work paves the way for more research on linking conversational artifacts with requirements. The formal framework and the automated matching techniques are basic components that can be used for emerging tasks such as tracing requirements to interviews and generating requirements from conversations.

Figures

Figures reproduced from arXiv: 2510.08622 by Fabiano Dalpiaz, Francesco Dente, Paolo Papotti.

Figure 1
Figure 1. Figure 1: Overview of the workflow. Elicitation from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example of Text-to-Story Alignment (T2SA). A snippet from the elicitation interview (left) is aligned [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average correctness of user stories generated by Qwen models of increasing size. Error bars denote ±1 standard deviation across datasets (n=15). 32B 14B 8B Human 4B 1.7B 0.6B Completeness 20 30 40 50 60 70 Percentage (%) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average completeness of user stories gener￾ated by Qwen models of increasing size. Error bars denote ±1 standard deviation across datasets (n=15). Completeness increases with generator size when measured with a fixed judge, indicating that larger LLMs produce story sets that cover more transcript chunks. This trend is orthogonal to matcher accuracy: the judge is held constant across tests, so differences a… view at source ↗
Figure 5
Figure 5. Figure 5: Recall of positive pairs vs. percentage of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Percentage of tokens that needs to be retrieved by the blocking operator ( [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages · 2 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mike Cohn. 2004. User Stories Applied: For Agile Software Development. Addison-Wesley Professional

    2004

  4. [4]

    Fabiano Dalpiaz, Arnon Sturm, and Patrizia Gieske. 2020. https://doi.org/10.5281/zenodo.4121935 Extraction of conceptual models: User stories vs. use cases . Zenodo dataset

    work page doi:10.5281/zenodo.4121935 2020

  5. [5]

    Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.naacl-main.187 Qafacteval: Improved qa-based factual consistency evaluation for summarization . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587...

    doi:10.18653/v1/2022.naacl-main.187 2022

  6. [6]

    D M \'e ndez Fern \'a ndez, Stefan Wagner, Marcos Kalinowski, Michael Felderer, Priscilla Mafra, Antonio Vetr \`o , Tayana Conte, M-T Christiansson, Des Greer, Casper Lassenius, Tomi Männistö, Maleknaz Nayabi, Markku Oivo, Birgit Penzenstadler, Dietmar Pfahl, Guenther Ruhe Rafael Prikladnicki, André Schekelmann, Sagar Sen, Rordrigo Spinola, and 3 others. ...

    work page doi:10.1007/s10664-016-9451-7 2017

  7. [7]

    Alessio Ferrari, Paola Spoletini, and Stefania Gnesi. 2016. https://doi.org/10.1007/s00766-016-0249-3 Ambiguity and tacit knowledge in requirements elicitation interviews . Requirements Engineering, 21(3):333--355

    doi:10.1007/s00766-016-0249-3 2016

  8. [8]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://doi.org/10.48550/arXiv.2001.08361 Scaling laws for neural language models . arXiv preprint arXiv:2001.08361

    Pith/arXiv arXiv doi:10.48550/arxiv.2001.08361 2020

  9. [9]

    Alexander Korn, Samuel Gorsch, and Andreas Vogelsang. 2025. https://doi.org/10.48550/arXiv.2507.02564 LLMREI : Automating requirements elicitation interviews with LLMs . In Proceedings of the 33rd IEEE International Requirements Engineering conference

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.02564 2025

  10. [10]

    Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.750 Evaluating the factual consistency of abstractive text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332--9346, Online. Association for Computational...

    doi:10.18653/v1/2020.emnlp-main.750 2020

  11. [11]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 SummaC : Re-visiting NLI -based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177

    doi:10.1162/tacl_a_00453 2022

  12. [12]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 11:157--173

    doi:10.1162/tacl_a_00638 2024

  13. [13]

    Garm Lucassen, Fabiano Dalpiaz, Jan Martijn E. M. van der Werf, and Sjaak Brinkkemper. 2016. https://doi.org/10.1007/s00766-016-0250-3 Improving agile requirements: The Quality User Story framework and tool . Requirements Engineering, 21:399--417

    work page doi:10.1007/s00766-016-0250-3 2016

  14. [14]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906--1919, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.173 2020

  15. [15]

    Preslav Nakov, David P. A. Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barr \' o n - Cede \ n o, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. https://doi.org/10.24963/IJCAI.2021/619 Automated fact-checking for assisting human fact-checkers . In Proceedings of the Thirtieth International Joint Conference on Artificial Inte...

    doi:10.24963/ijcai.2021/619 2021

  16. [16]

    Ani Nenkova and Rebecca Passonneau. 2004. https://aclanthology.org/N04-1019/ Evaluating content selection in summarization: The pyramid method . In Proceedings of the Human Language Technology Conference of the North A merican Chapter of the Association for Computational Linguistics: HLT - NAACL 2004 , pages 145--152. Association for Computational Linguistics

    2004

  17. [17]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. https://arxiv.org/abs/1901.04085 Passage re-ranking with BERT . arXiv preprint arXiv:1901.04085

    Pith/arXiv arXiv 2019

  18. [18]

    Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. https://doi.org/10.18653/v1/2021.naacl-main.383 Understanding factuality in abstractive summarization with FRANK : A benchmark for factuality metrics . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    doi:10.18653/v1/2021.naacl-main.383 2021

  19. [19]

    George Papadakis, Dimitrios Skoutas, Emmanouil Thanos, and Themis Palpanas. 2020. https://doi.org/10.1145/3377455 Blocking and filtering techniques for entity resolution: A survey . ACM Computing Surveys, 53(2)

    doi:10.1145/3377455 2020

  20. [20]

    Giovanni Quattrocchi, Liliana Pasquale, Paola Spoletini, and Luciano Baresi. 2025. https://doi.org/10.48550/arXiv.2507.15157 Can LLMs generate user stories and assess their quality? arXiv preprint arXiv:2507.15157

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.15157 2025

  21. [21]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982--3992, Hong Kong, Chi...

    doi:10.18653/v1/d19-1410 2019

  22. [22]

    Krishna Ronanki, Beatriz Cabrero-Daniel, and Christian Berger. 2022. https://doi.org/10.1007/978-3-031-48550-3_17 Chatgpt as a tool for user story quality evaluation: Trustworthy out of the box? In International Conference on Agile Software Development, pages 173--181. Springer

    doi:10.1007/978-3-031-48550-3_17 2022

  23. [23]

    Reine Santos, Gabriel Freitas, Igor Steinmacher, Tayana Conte, Ana Carolina Oran, and Bruno Gadelha. 2025. https://doi.org/10.5220/0013365500003929 User stories: Does ChatGPT do it better? In Proceedings of the 27th International Conference on Enterprise Information Systems, pages 47--58

    work page doi:10.5220/0013365500003929 2025

  24. [24]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Patrick Gallinari, and Ionut Sorodoc. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.102 Questeval: Summarization asks for fact-based evaluation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1312--1323, Online an...

    work page doi:10.18653/v1/2021.emnlp-main.102 2021

  25. [25]

    Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. https://doi.org/10.18653/v1/2020.acl-main.332 That is a known lie: Detecting previously fact-checked claims . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3607--3618

    work page doi:10.18653/v1/2020.acl-main.332 2020

  26. [26]

    Amol Sharma and Anil Kumar Tripathi. 2025. https://doi.org/10.1007/s10844-025-00939-3 Evaluating user story quality with LLMs : A comparative study . Journal of Intelligent Information Systems, pages 1--29

    work page doi:10.1007/s10844-025-00939-3 2025

  27. [27]

    Tjerk Spijkman, Fabiano Dalpiaz, and Sjaak Brinkkemper. 2022. https://doi.org/10.1109/RE54965.2022.00042 Back to the roots: Linking user stories to requirements elicitation conversations . In 2022 IEEE 30th International Requirements Engineering Conference (RE), pages 281--287. IEEE

    arXiv 2022

  28. [28]

    Tjerk Spijkman, Xavier de Bondt, Fabiano Dalpiaz, and Sjaak Brinkkemper. 2023. https://doi.org/10.1007/978-3-031-29786-1_9 Summarization of elicitation conversations to locate requirements-relevant information . In International Working Conference on Requirements Engineering: Foundation for Software Quality, pages 122--139. Springer

    work page doi:10.1007/978-3-031-29786-1_9 2023

  29. [29]

    Alistair Sutcliffe and Pete Sawyer. 2013. https://doi.org/10.1109/RE.2013.6636709 Requirements elicitation: Towards the unknown unknowns . In Proceedings of the 21st IEEE International Requirements Engineering Conference, pages 92--104

    arXiv 2013

  30. [30]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 809--819, New...

    doi:10.18653/v1/n18-1074 2018

  31. [31]

    Stefan Wagner, Daniel M \'e ndez Fern \'a ndez, Michael Felderer, Antonio Vetr \`o , Marcos Kalinowski, Roel Wieringa, Dietmar Pfahl, Tayana Conte, Marie-Therese Christiansson, Desmond Greer, Tomi Männistö Casper Lassenius, Maleknaz Nayebi, Markku Oivo, Birgit Penzenstadler, Rafael Prikladnicki, Guenther Ruhe, André Schekelmann, Sagar Sen, Rodrigo Spínola...

    work page doi:10.1145/3306607 2019

  32. [32]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. https://doi.org/10.18653/v1/2020.acl-main.450 Asking and answering questions to evaluate the factual consistency of summaries . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008--5020, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.450 2020

  33. [33]

    Asma Yamani, Malak Baslyman, and Moataz Ahmed. 2025. https://doi.org/10.1145/3727582.3728689 Leveraging LLMs for user stories in AI systems: UStAI dataset . In Proceedings of the 21st International Conference on Predictive Models and Data Analytics in Software Engineering, pages 21--30

    arXiv 2025

  34. [34]

    Olesya Zaremba and Sotirios Liaskos. 2021. https://doi.org/10.1109/RE51729.2021.00042 Towards a typology of questions for requirements elicitation interviews . In Proceedings of the 29th IEEE International Requirements Engineering Conference, pages 384--389. IEEE

    arXiv 2021

  35. [35]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328--11348, Toronto, Canada. Association for Co...

    doi:10.18653/v1/2023.acl-long.634 2023

  36. [36]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena . Preprint, arXiv:2306.05685. NeurIPS 2023 Datasets & Benchmarks track

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.