Pith. sign in

REVIEW 6 major objections 6 minor 44 references

Language Models to Support Multi-Label Classification of Industrial Data

T0 review · 6 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zero-shot classifiers can label industrial requirements into large taxonomies, and smaller sentence-embedding models beat massive LLMs.

desk verdict A useful, reproducible empirical comparison showing SBERT-style LMs beat raw LLM embeddings for zero-shot hierarchical multi-label requirements classification, but every ranking is tied to a single k=15 operating point; worth refereeing with revisions. read the letter →

arxiv 2504.15922 v2 pith:GFSAXW5U submitted 2025-04-22 cs.SE

Waleed Abdeen , Michael Unterkalmsteiner , Krzysztof Wnuk , Alessio Ferrari , Panagiota Chatzipetrou This is my paper · ORCID
classification cs.SE
keywords zero-shotlearningmulti-labelclassificationrequirementslanguagemodelshierarchicaltaxonomylabeldistancemetrictracingindustrialdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a zero-shot classifier, built by computing cosine similarity between embeddings of requirement text and embeddings of taxonomy class descriptions, can do useful multi-label classification of industrial requirements against large hierarchical taxonomies without any labeled training data. The motivation is practical: requirements engineering teams often have hundreds of classes but too few annotated requirements to train supervised models. The paper compares 14 language models, from 66 million to 70 billion parameters, on six taxonomy-based output spaces with 377 requirements and 1,968 labels. It reports that the sentence-to-sequence model T5-xl gives the best distance-aware performance on five of six output spaces, that smaller models beat much larger LLMs, and that model architecture matters more than parameter count. If true, this means traceability support can be deployed cheaply on modest hardware.

What carries the argument

The load-bearing mechanism is a zero-shot embedding-similarity pipeline. For each requirement, the text is concatenated with its document and section titles; for each taxonomy class, the node description is aggregated with its children's descriptions (the 'hierarchical strategy'). A language model produces embeddings for both sides, and cosine similarity ranks classes, of which the top $k$ are returned as labels. The paper's proposed evaluation device is the normalized label distance metric $D_n = D_a / D_{\max}$, where $D_a$ is the number of hops between predicted and true labels in the taxonomy and $D_{\max} = 2 \times \mathrm{depth}$; this continuous measure in $[0,1]$ is what makes T5-xl's advantage visible consistently across output spaces where $F_1$ and $F_\beta$ are ambiguous.

What would settle it

Re-running the same 14 models on the same six output spaces with k = 5, 10, 20, and 30 would settle the central claim: if T5-xl is not the distance-metric leader at other cutoffs, or if the negative correlation between model size and $F_\beta$ reverses, the architecture-not-scale conclusion fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that zero-shot language-model classification is a workable approach to multi-label requirements classification against large domain taxonomies, and that model selection should be driven by architecture and sentence-embedding fine-tuning rather than raw scale. In the reported experiment, T5-xl (an encoder-decoder model with 3B parameters) achieved the best label-distance performance on five of the six output spaces, with a maximum $F_eta = 0.78$ and a best normalized distance $D_n = 0.04$; a BERT-family MiniLM sentence-embedding model outperformed all other models on the remaining output space with $F_eta = 0.83$ and $D_n = 0.04$. Across the board, autoencoding and sequence-to-sequence models with up to 3B parameters outperformed autoregressive LLMs with up to 70B parameters, and the model type had a statistically significant effect on performance. The paper also claims that its proposed normalized label distance metric $D_n$ identifies the top-performing model more consistently across output spaces than $F_1$ or $F_eta$.

Load-bearing premise

All reported rankings and conclusions assume the classifier returns exactly 15 suggested labels per requirement; the authors tried other cutoff values but report only 15, so the model ordering and the smaller-beats-larger conclusion are conditioned on that unstated choice.

Editorial extensions

If this is right

  • Companies with scarce labeled requirements can deploy a zero-shot classifier immediately, cutting the label space an engineer must review from hundreds of classes to 15 candidates.
  • Model selection for this task should prefer sentence-embedding models and consider architecture first; parameter count beyond the 3B range does not pay off.
  • A human-in-the-loop tracing tool can use $D_n$ to display predicted labels and their neighbors within a small taxonomy distance, reducing search effort.
  • Supervised approaches for multi-label requirements classification now have a concrete zero-shot baseline to beat on this industrial dataset.
  • The $D_n$ metric gives a consistent cross-output-space ranking, making it a candidate for model selection in other hierarchical classification problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because only k=15 results are reported, the 'small models beat large models' ordering is untested at other k values; repeating the comparison at k=5 and k=30 would show whether it is a stable property of architecture or an artifact of the cutoff.
  • Editorial inference: the top models' shared trait is sentence-transformer fine-tuning for cosine similarity, so the paper's architecture conclusion is really about training objective; an autoregressive LLM fine-tuned for embedding generation might behave differently.
  • Editorial inference: the $D_n$ metric should transfer to other hierarchical multi-label tasks such as medical coding or product categorization, and its practical value could be tested by measuring whether recommendations within one hop speed up human label selection.
  • Editorial inference: because the classifier includes document and section titles as context, its success may depend on context-rich requirement documents; applying it to requirements without such context would likely lower performance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. This paper evaluates zero-shot classifiers for multi-label requirements classification on an industrial dataset. The classifier computes cosine similarity between embeddings of requirements (with document/section context) and embeddings of taxonomy node descriptions, then selects the k most similar labels. The authors compare 14 language models (autoencoding, autoregressive, and sequence-to-sequence, from 66M to 70B parameters) across six hierarchical output spaces, using 377 annotated requirements. Evaluation uses precision, recall, F1, a context-weighted Fβ (computed as F189), and a novel label distance metric Dn that measures the normalized hop distance between predicted and true taxonomy nodes. The paper reports that T5-xl is the top performer on most output spaces, that smaller sentence-transformer models outperform large autoregressive LLMs for this task, and that model type significantly affects performance. A replication package with code and data is provided.

Significance. If the findings hold, the paper would provide useful practical guidance for requirements tracing with scarce labeled data: a zero-shot pipeline built on a moderately sized SBERT-style model can outperform much larger LLMs, with implications for industrial deployment. The systematic comparison of 14 models on six output spaces is more extensive than previous zero-shot requirements classification studies, and the replication package is a concrete strength. The proposed Dn metric is a reasonable idea for hierarchical output spaces, though its multi-label aggregation needs to be precisely defined. The empirical claims are directionally plausible, but several load-bearing reporting and design issues currently prevent the paper's strong conclusions from being fully supported.

major comments (6)
  1. [Section IV-I, Table V] All reported metrics and every statistical result in RQ1/RQ2 are evaluated at a single cutoff k=15, although Section IV-I states that 'We experimented with different values of k' and that only space constraints motivated reporting k=15. Since precision, recall, F1, Fβ, and Dn are all functions of k (recall is monotonically non-decreasing in k, and precision is monotonically non-increasing), the headline conclusions—T5-xl is the top model, smaller LMs beat LLMs, and model type matters—are not established beyond this one operating point. Please report the k-sweep results, at least as a stability analysis of the model ranking and of the RQ2 tests, or provide a task-based justification for selecting k=15.
  2. [Section V-A, Abstract, Table V] The headline claim that T5-xl is the top-performing model on 5 of 6 output spaces is not directly supported by Table V. Under F189, MiniLM has the best value on OSA (0.83 vs. 0.78) and ties T5-xl on OST (both 0.75); under Dn, T5-xl ties with MiniLM and T5-large on OSA (all 0.04) and is second on OST. The text additionally says Multi-MPNet has Dn=0.08 on OST, but the table shows 0.17. These inconsistencies need to be corrected, and the criterion for 'top-performing' (Fβ, Dn, or a combination) must be stated explicitly and applied consistently.
  3. [Section II-A, Table II, Section VI-A] The comparison underlying RQ2 conflates model architecture and size with task adaptation and quantization. The autoencoding and sequence-to-sequence models are loaded from sentence-transformer checkpoints fine-tuned for sentence embeddings, while the autoregressive models are used as raw base models, and Llama2-70B, Mistral-7B, and Mixtral-8x7B are quantized to Q4. Therefore the observed superiority of SBERT-style models and the negative correlation with model size cannot be attributed solely to architecture or parameter count. A fairer test would compare all models under the same adaptation regime, or the conclusions should be explicitly restricted to off-the-shelf zero-shot embedders with the current adaptation differences.
  4. [Section IV-F, Eqs. (5)-(7)] The label distance metric Dn is underspecified for multi-label evaluation. Equations (5)-(7) define distance between a single predicted label p and a single true label t, but each requirement has up to k predicted labels and a variable number of true labels. The paper does not state how predicted and true labels are paired before averaging: if all predicted/true pairs are averaged, the metric is sensitive to k and to label-set size in a way that is not discussed; if a matching or nearest-neighbor rule is used, it must be defined. Since Dn is a central contribution and is used for model selection, this aggregation rule is load-bearing.
  5. [Section IV-E, Table IV, Table VI] F189 is applied uniformly to all output spaces even though Table IV computes output-space-specific β values (ranging from 189 to 617). The paper justifies this by noting that Fβ approximates recall for these β values, but then Fβ is still used as a dependent variable in the RQ2 statistical tests. Consequently, the 'model characteristics impact' results for Fβ are essentially recall results, and the precision dimension is absent from those tests. The authors' own statement in Section V-A that Fβ is 'not useful to identify the top-performing model' further undercuts its use as a primary outcome; this should be addressed in the analysis design.
  6. [Table VI, Section V-B] The Kruskal-Wallis test on embedding size includes groups with a single observation (embedding size 384 has n=1 for MiniLM, and 1024 has n=1 for RoBERTa-large). Comparing group medians with singleton groups is not meaningful, and the post hoc claim of a significant difference between 4096 and these singleton groups is not a robust result. Please re-analyze the embedding-size effect with a more appropriate method (e.g., treating embedding size as a variable with sufficient replication per level, or reporting per-level distributions without Kruskal-Wallis inference).
minor comments (6)
  1. [Abstract] The sentence 'The difficulties increases when a limited number of requirements is available' contains a subject-verb agreement error and should be revised.
  2. [Section VI-B] The phrase 'as depicted in Figure IV-F' is incorrect; the reference should be to Table V or Section IV-F, not a nonexistent figure.
  3. [Section IV-H] In the Internal Validity paragraph, 'To mitigate this treat' should read 'To mitigate this threat.'
  4. [Table II] The model listed as 'BERT (base)' uses checkpoint all-MiniLM-L12-v2, which is a distilled MiniLM model rather than BERT-base. Please rename the row to avoid misleading the reader.
  5. [Table IV] The lhead column sums to 442, which exceeds the 377 requirements reported in Section IV-B; please clarify how lhead is defined per output space and reconcile this with the dataset size.
  6. [Section V-B] The statistical analysis would benefit from effect sizes and confidence intervals for the significant Kruskal-Wallis and Spearman results, not only p-values.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark with metrics defined independently of the results.

full rationale

The paper does not claim to derive a prediction from an input; it reports a benchmark comparison of 14 zero-shot language-model classifiers on an annotated industrial dataset. The central claims (T5-xl top performer on 5/6 output spaces, smaller LMs beat LLMs, model architecture affects performance) are direct measurements on a held-out test set, not quantities reconstructed from fitted parameters. The label distance metric Dn is defined in Section IV-F by fixed formulas (Eqs. 5-8) over taxonomy hops and depth; it is not fitted to the outcome and is used as an evaluation measure, so selecting the best model by it is not circular. F-beta is computed with Berry's context-dependent formula using ground-truth label counts; this is a stated metric choice, not a prediction derived from the conclusion. Self-citations to the authors' prior study provide the dataset and taxonomy and describe the hierarchical aggregation strategy, but no load-bearing argument reduces to those citations: the present models and results are evaluated externally and the replication package is provided. The k=15 cutoff is disclosed as one chosen operating point and is a threat to generalizability of the ranking, not a circularity: it is not a fitted input disguised as a prediction. No step in the paper's argument makes the output equivalent to the input by definition.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the representativeness of the 377-requirement sample, the correctness of the ground-truth labels, the fairness of the LM versus LLM comparison given different fine-tuning, and the validity of the proposed distance metric. The only explicit hand-set evaluation parameter is k=15, with the Fβ evaluation beta also chosen post hoc.

free parameters (2)
  • k = 15
    Number of top labels returned per requirement; chosen by the experimenters. All reported results and rankings depend on this value; the authors state they experimented with other k values but only k=15 is shown in Section IV-I.
  • Fβ evaluation beta = 189 for all output spaces
    Although per-output-space β values range from 189 to 617 in Table IV, all Fβ results use F189, the smallest β. This changes the precision-recall tradeoff compared with each output space's recommended β and is acknowledged in Section V.
assumptions (4)
  • domain assumption The 377 sampled requirements are representative of the full 22,000-requirement industrial dataset.
    Section IV-B1 states sample size was calculated with 95% confidence and ±5% margin of error, but sampling and annotation details are from prior work [1] and are not fully re-derived here.
  • domain assumption Ground-truth labels, 1968 labels across 6 output spaces, are correct and consistent.
    The paper relies on annotations from previous study [1] but does not report inter-annotator agreement or validation of the labels. All metrics are computed against this ground truth.
  • ad hoc to paper Comparing sentence-transformer fine-tuned LMs with raw autoregressive LLM embeddings is a fair comparison of model architecture and size.
    Sections II-A and IV-C show LMs and seq2seq models were loaded from sentence-embedding fine-tuned checkpoints, while LLMs such as Llama2 and Mistral were used without sentence-embedding adaptation or prompting. This confounds model type and size with fine-tuning.
  • domain assumption The label distance metric, computed as normalized hop counts between labels via Eqs. 5 through 8, is a valid measure of classification quality for hierarchical multi-label tasks.
    Section IV-F adapts Eisner et al.'s metric but does not validate the metric's properties or the label-pair matching procedure for multi-label predictions. The metric is claimed useful for model selection but no external validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Models to Support Multi-Label Classification of Industrial Data." pith.science (2026). https://pith.science/paper/GFSAXW5U

@misc{pith2026250415922,
  author       = {Pith},
  title        = {Pith review of: Language Models to Support Multi-Label Classification of Industrial Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFSAXW5U}},
  note         = {Machine review of arXiv:2504.15922}
}
abstract

Multi-label requirements classification is a challenging task, especially when dealing with numerous classes at varying levels of abstraction. The difficulties increases when a limited number of requirements is available to train a supervised classifier. Zero-shot learning (ZSL) does not require training data and can potentially address this problem. This paper investigates the performance of zero-shot classifiers (ZSCs) on a multi-label industrial dataset. We focuse on classifying requirements according to a taxonomy designed to support requirements tracing. We compare multiple variants of ZSCs using different embeddings, including 9 language models (LMs) with a reduced number of parameters (up to 3B), e.g., BERT, and 5 large LMs (LLMs) with a large number of parameters (up to 70B), e.g., Llama. Our ground truth includes 377 requirements and 1968 labels from 6 output spaces. For the evaluation, we adopt traditional metrics, i.e., precision, recall, F1, and $F_\beta$, as well as a novel label distance metric Dn. This aims to better capture the classification's hierarchical nature and provides a more nuanced evaluation of how far the results are from the ground truth. 1) The top-performing model on 5 out of 6 output spaces is T5-xl, with maximum $F_\beta$ = 0.78 and Dn = 0.04, while BERT base outperformed the other models in one case, with maximum $F_\beta$ = 0.83 and Dn = 0.04. 2) LMs with smaller parameter size produce the best classification results compared to LLMs. Thus, addressing the problem in practice is feasible as limited computing power is needed. 3) The model architecture (autoencoding, autoregression, and sentence-to-sentence) significantly affects the classifier's performance. We conclude that using ZSL for multi-label requirements classification offers promising results. We also present a novel metric that can be used to select the top-performing model for this problem

Figures

Figures reproduced from arXiv: 2504.15922 by the authors.

Figure 1
Figure 1. Example of a multi-label requirement classification task with a hierarchical taxonomy. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Zero-Shot Classifier G2: Identify the model characteristics that impact the classi￾fier’s performance. In alignment with these goals, we wrote the following three research questions mapped one-to-one to the goals. RQ1: To what extent does the zero-shot classifier’s performance differ when using embedding created by different models? Motivation: A zero-shot classifier benefits from pre￾trained models to generate text… view at source ↗
Figure 3
Figure 3. An example demonstrating the measurement of the label distance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 32 canonical work pages

  1. [1]

    Multi-label requirements classification with large taxonomies,

    W. Abdeen, M. Unterkalmsteiner, W. Krzysztof, A. Chirtoglou, C. Schi- manski, and H. Goli, “Multi-label requirements classification with large taxonomies,” in 2024 IEEE 32st International Requirements Engineering Conference (RE) , ser. Requirements Engineering Conference. IEEE, 2024

    work page 2024

  2. [2]

    Zero-shot learning for require- ments classification: An exploratory study,

    W. Alhoshan, A. Ferrari, and L. Zhao, “Zero-shot learning for require- ments classification: An exploratory study,” Info Soft Techn , vol. 159, Jul. 2023. 3https://doi.org/10.6084/m9.figshare.25990825.v1

    work page doi:10.6084/m9.figshare.25990825.v1 2023

  3. [3]

    Requirements Classification for Smart Allocation: A Case Study in the Railway Industry,

    S. Bashir, M. Abbas, A. Ferrari, M. Saadatmand, and P. Lindberg, “Requirements Classification for Smart Allocation: A Case Study in the Railway Industry,” in 2023 IEEE 31st International Requirements Engineering Conference (RE) . Hannover, Germany: IEEE, Sep. 2023, pp. 201–211

    work page 2023

  4. [4]

    Empirical evaluation of tools for hairy requirements engineering tasks,

    D. M. Berry, “Empirical evaluation of tools for hairy requirements engineering tasks,” Empirical Software Engineering , vol. 26, no. 6, p. 111, 2021

    work page 2021

  5. [5]

    Panel: Context-dependent evaluation of tools for NL RE tasks: Recall vs. precision, and beyond,

    D. M. Berry, J. Cleland-Huang, A. Ferrari, W. Maalej, J. Mylopoulos, and D. Zowghi, “Panel: Context-dependent evaluation of tools for NL RE tasks: Recall vs. precision, and beyond,” in IEEE 25th Int Req Eng Conference (RE). IEEE, 2017, pp. 570–573, ISSN: 2332-6441

    work page 2017

  6. [6]

    Language Models are Few-Shot Learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, and T. Henighan, “Language Models are Few-Shot Learners,” Adv in neural inf processing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  7. [7]

    A Survey on Evaluation of Large Language Models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A Survey on Evaluation of Large Language Models,” ACM Trans on Intel Systems and Technology , vol. 15, no. 3, pp. 39:1–39:45, Mar. 2024

    work page 2024

  8. [8]

    W. J. Conover, Practical nonparametric statistics . john wiley & sons, 1999, vol. 350

    work page 1999

Show all 44 references
  1. [9]

    Requirements Classification and Reuse: Crossing Domain Boundaries,

    J. L. Cybulski and K. Reed, “Requirements Classification and Reuse: Crossing Domain Boundaries,” in Software Reuse: Advances in Software Reusability, W. B. Frakes, Ed. Springer, 2000, pp. 190–210

    2000

  2. [10]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” May 2019, arXiv:1810.04805

    2019 arXiv

  3. [11]

    Multiple comparisons using rank sums,

    O. J. Dunn, “Multiple comparisons using rank sums,” Technometrics, vol. 6, no. 3, pp. 241–252, 1964

    1964

  4. [12]

    Improving Protein Function Prediction using the Hierarchical Structure of the Gene Ontology,

    R. Eisner, B. Poulin, D. Szafron, P. Lu, and R. Greiner, “Improving Protein Function Prediction using the Hierarchical Structure of the Gene Ontology,” in 2005 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology. IEEE, Nov. 2005, pp. 1–10

    2005

  5. [13]

    Natural language requirements processing: a 4d vision,

    A. Ferrari, F. Dell’Orletta, A. Esuli, V . Gervasi, S. Gnesi et al., “Natural language requirements processing: a 4d vision,” IEEE SOFTWARE , vol. 34, no. 6, pp. 28–35, 2017

    2017

  6. [14]

    Con- structing and Using Software Requirement Patterns,

    X. Franch, C. Quer, S. Renault, C. Guerlain, and C. Palomares, “Con- structing and Using Software Requirement Patterns,” in Managing Re- quirements Knowledge, W. Maalej and A. K. Thurimella, Eds. Springer, 2013, pp. 95–116

    2013

  7. [15]

    Computing Semantic Relatedness Using Wikipedia- based Explicit Semantic Analysis,

    E. Gabrilovich, “Computing Semantic Relatedness Using Wikipedia- based Explicit Semantic Analysis,” in Proc. 20th Int. Conf on AI, vol. 20. IJCAI, 2007

    2007

  8. [16]

    NoRBERT: Transfer Learning for Requirements Classification,

    T. Hey, J. Keim, A. Koziolek, and W. F. Tichy, “NoRBERT: Transfer Learning for Requirements Classification,” in 2020 IEEE 28th Int. Requirements Engineering Conf (RE) . IEEE, Aug. 2020, pp. 169–179

    2020

  9. [17]

    Reporting Experiments in Software Engineering,

    A. Jedlitschka, M. Ciolkowski, and D. Pfahl, “Reporting Experiments in Software Engineering,” in Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London: Springer, 2008, pp. 201–228

    2008

  10. [18]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7B,” Oct. 2023, arXiv:2310.06825

    2023 arXiv

  11. [19]

    Mixtral of Experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, ...

    2024 arXiv

  12. [20]

    Vector Semantics and Embeddings,

    D. Jurafsky and J. H. Martin, “Vector Semantics and Embeddings,” in Speech and Language Processing , 3rd ed. Stanford University, Feb. 2024

    2024

  13. [21]

    One Model To Learn Them All,

    L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit, “One Model To Learn Them All,” Jun. 2017, arXiv:1706.05137 [cs, stat]

    2017 arXiv

  14. [22]

    A BERT-based transfer learning approach to text classification on software requirements specifications,

    D. Kici, G. Malik, M. Cevik, D. Parikh, and A. Bas ¸ar, “A BERT-based transfer learning approach to text classification on software requirements specifications,” Proc of the Canadian Conference on AI , Jun. 2021

    2021

  15. [23]

    Automatically Classifying Functional and Non-functional Requirements Using Supervised Machine Learning,

    Z. Kurtanovi ´c and W. Maalej, “Automatically Classifying Functional and Non-functional Requirements Using Supervised Machine Learning,” in 2017 IEEE 25th Int. Requirements Engineering Conf (RE) . IEEE, Sep. 2017, pp. 490–495

    2017

  16. [24]

    RoBERTa: A robustly optimized BERT pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” 2019

    2019

  17. [25]

    Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models,

    J. Ni, G. H. ´Abrego, N. Constant, J. Ma, K. B. Hall, D. Cer, and Y . Yang, “Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models,” Dec. 2021, arXiv:2108.08877

    2021 arXiv

  18. [26]

    Ultra-large-scale systems: The software challenge of the future,

    L. Northrop, P. Feiler, R. P. Gabriel, J. Goodenough, R. Linger, T. Longstaff, R. Kazman, M. Klein, D. Schmidt, K. Sullivan et al. , “Ultra-large-scale systems: The software challenge of the future,” 2006

    2006

  19. [27]

    Automatic Requirement Categorization of Large Natural Lan- guage Specifications at Mercedes-Benz for Review Improvements,

    D. Ott, “Automatic Requirement Categorization of Large Natural Lan- guage Specifications at Mercedes-Benz for Review Improvements,” in Requirements Engineering: Foundation for Software Quality, ser. LNCS, J. Doerr and A. L. Opdahl, Eds. Springer, 2013, pp. 50–64

    2013

  20. [28]

    A model for types and levels of human interaction with automation,

    R. Parasuraman, T. B. Sheridan, and C. D. Wickens, “A model for types and levels of human interaction with automation,” IEEE Transon systems, man, and cybernetics-Part A, vol. 30, no. 3, pp. 286–297, 2000

    2000

  21. [29]

    Improving Language Understanding by Generative Pre-Training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018

    2018

  22. [30]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proc Conf on Empirical Methods in NLP . Association for Computational Linguistics, 11 2019

    2019

  23. [31]

    The Two Word Test: A Semantic Benchmark for Large Language Models,

    N. Riccardi and R. H. Desai, “The Two Word Test: A Semantic Benchmark for Large Language Models,” Jun. 2023, arXiv:2306.04610

    2023 arXiv

  24. [32]

    The role of natural language in requirements engineering,

    K. Ryan, “The role of natural language in requirements engineering,” in

  25. [33]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” Feb. 2020, arXiv:1910.01108

    2020 arXiv

  26. [34]

    Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches,

    T. Schopf, D. Braun, and F. Matthes, “Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches,” in Proc 6th Int Conf on NLP Retrieval . Bangkok Thailand: ACM, Dec. 2022, pp. 6–15

    2022

  27. [35]

    MPNet: Masked and permuted pre-training for language understanding,

    K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu, “MPNet: Masked and permuted pre-training for language understanding,” 2020

    2020

  28. [36]

    EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models,

    Z. Tao, Z. Jin, X. Bai, H. Zhao, Y . Feng, J. Li, and W. Hu, “EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models,” May 2023, arXiv:2305.15268

    2023 arXiv

  29. [37]

    Llama 2: Open Foundation and Fine-Tuned Chat Models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  30. [38]

    TT-RecS: The taxonomic trace recommender system,

    M. Unterkalmsteiner, “TT-RecS: The taxonomic trace recommender system,” in IEEE 7th Int Workshop on IA for RE (AIRE) . IEEE, 2020, pp. 18–21

    2020

  31. [39]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017

    2017

  32. [40]

    What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?

    T. Wang, A. Roberts, D. Hesslow, T. L. Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel, “What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?” in Proceedings of the 39th International Conference on Machine Learning . PMLR,...

    2022

  33. [41]

    MiniLM: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MiniLM: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,” 2020

    2020

  34. [42]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl´en, Experimentation in Software Engineering. Springer, 2012

    2012

  35. [43]

    Natural language processing for requirements engineering: A systematic mapping study,

    L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, M. A. Ajagbe, E.-V . Chioasca, and R. T. Batista-Navarro, “Natural language processing for requirements engineering: A systematic mapping study,” ACM Comput- ing Surveys (CSUR) , vol. 54, no. 3, pp. 1–41, 2021

    2021

  36. [1993]

    IEEE, Jan

    Proceedings of the IEEE International Symposium on Require- ments Engineering. IEEE, Jan. 1993, pp. 240–242

    1993

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.