{"id":"8da3896e-8b96-4a4b-80b3-db1ffaca92c5","arxiv_id":"2504.15922","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot classifiers built on sentence-transformer language models (T5-xl, MiniLM) outperform much larger LLMs on multi-label, hierarchical requirements classification, and a hop-count label distance metric helps rank models.","lead":"This paper compares 14 language models as zero-shot classifiers for tagging industrial requirements with multiple hierarchical labels, finding that smaller sentence-embedding models such as T5-xl and MiniLM beat larger LLMs like Llama 2. A new label-distance metric is proposed to evaluate how close predicted taxonomy nodes are to the true ones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All model rankings and the small-beats-large conclusion are computed only at k=15, an unreported arbitrary cutoff. Because every metric (precision, recall, Fβ, Dn) is a function of k, the central claim that T5-xl is the top model is not yet established outside this single operating point.","rationale":"The reader's weakest_assumption is exactly the point I find most load-bearing. The paper's own reporting in Section IV-I concedes that only k=15 is shown, and all Table V values are conditioned on that choice. This is not a minor implementation detail: in a top-k recommender, every evaluation metric is a function of k, and the authors state that precision and recall trade off with k. Without the k sweep, the headline \"T5-xl is best\" and the negative size-performance correlation could be artifacts of the cutoff. The concern is addressable by a sensitivity analysis using the published replication package, and the paper's practical conclusion (smaller models suffice) may survive, so I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT. I do not separately elevate the Dn pair-matching ambiguity or the LM/LLM fine-tuning confound; both are real but secondary, and the k-dependence is the single assumption on which all reported comparisons rest.","tokens_in":16986,"tokens_out":5611,"duration_ms":53879,"concrete_test":"Using the publicly available replication package, rerun all 14 models × 6 output spaces for k in {1, 3, 5, 10, 15, 20, 30, 50}, recording precision, recall, F189, and Dn. Then check (a) whether T5-xl remains the top-ranked model on at least 5 of 6 output spaces by both F189 and Dn at every value of k, and (b) whether the Spearman correlations between model size and F1/F189/Dn (rs = -0.646, -0.665, +0.491) retain their sign and significance at every value of k. If rankings or correlations change across k, the reported k=15 results are an artifact of the cutoff and the central claim must be narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-I states: \"We experimented with different values of k but report, due to space constraints, only the results at k = 15.\" Every headline number in Table V — precision, recall, F1, F189, and Dn — is evaluated at that one cutoff. The classifier returns exactly k labels, so recall is monotonically non-decreasing in k, precision is monotonically non-increasing, F189 is essentially recall, and Dn depends on how many candidate labels are included before distances are averaged. The paper's strongest claims (\"T5-xl is the top performer on 5 of 6 output spaces\", \"smaller LMs beat LLMs\", \"model type significantly affects performance\") are therefore claims about a single operating point, not about the models in general. If, for example, k=1 or k=50 changes which model has the best recall or Dn, the model-selection conclusion and the architecture/size discussion would need to be revised. Since the k sweep was run but not reported, there is no evidence that the ranking is stable. This is the most load-bearing gap because it applies to every metric and every research question, and the authors themselves identify the omitted sweep in the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates zero-shot classifiers for multi-label requirements classification on an industrial dataset. The classifier computes cosine similarity between embeddings of requirements (with document/section context) and embeddings of taxonomy node descriptions, then selects the k most similar labels. The authors compare 14 language models (autoencoding, autoregressive, and sequence-to-sequence, from 66M to 70B parameters) across six hierarchical output spaces, using 377 annotated requirements. Evaluation uses precision, recall, F1, a context-weighted Fβ (computed as F189), and a novel label distance metric Dn that measures the normalized hop distance between predicted and true taxonomy nodes. The paper reports that T5-xl is the top performer on most output spaces, that smaller sentence-transformer models outperform large autoregressive LLMs for this task, and that model type significantly affects performance. A replication package with code and data is provided.","tokens_in":17227,"tokens_out":9238,"duration_ms":79999,"significance":"If the findings hold, the paper would provide useful practical guidance for requirements tracing with scarce labeled data: a zero-shot pipeline built on a moderately sized SBERT-style model can outperform much larger LLMs, with implications for industrial deployment. The systematic comparison of 14 models on six output spaces is more extensive than previous zero-shot requirements classification studies, and the replication package is a concrete strength. The proposed Dn metric is a reasonable idea for hierarchical output spaces, though its multi-label aggregation needs to be precisely defined. The empirical claims are directionally plausible, but several load-bearing reporting and design issues currently prevent the paper's strong conclusions from being fully supported.","major_comments":[{"comment":"All reported metrics and every statistical result in RQ1/RQ2 are evaluated at a single cutoff k=15, although Section IV-I states that 'We experimented with different values of k' and that only space constraints motivated reporting k=15. Since precision, recall, F1, Fβ, and Dn are all functions of k (recall is monotonically non-decreasing in k, and precision is monotonically non-increasing), the headline conclusions—T5-xl is the top model, smaller LMs beat LLMs, and model type matters—are not established beyond this one operating point. Please report the k-sweep results, at least as a stability analysis of the model ranking and of the RQ2 tests, or provide a task-based justification for selecting k=15.","section":"Section IV-I, Table V"},{"comment":"The headline claim that T5-xl is the top-performing model on 5 of 6 output spaces is not directly supported by Table V. Under F189, MiniLM has the best value on OSA (0.83 vs. 0.78) and ties T5-xl on OST (both 0.75); under Dn, T5-xl ties with MiniLM and T5-large on OSA (all 0.04) and is second on OST. The text additionally says Multi-MPNet has Dn=0.08 on OST, but the table shows 0.17. These inconsistencies need to be corrected, and the criterion for 'top-performing' (Fβ, Dn, or a combination) must be stated explicitly and applied consistently.","section":"Section V-A, Abstract, Table V"},{"comment":"The comparison underlying RQ2 conflates model architecture and size with task adaptation and quantization. The autoencoding and sequence-to-sequence models are loaded from sentence-transformer checkpoints fine-tuned for sentence embeddings, while the autoregressive models are used as raw base models, and Llama2-70B, Mistral-7B, and Mixtral-8x7B are quantized to Q4. Therefore the observed superiority of SBERT-style models and the negative correlation with model size cannot be attributed solely to architecture or parameter count. A fairer test would compare all models under the same adaptation regime, or the conclusions should be explicitly restricted to off-the-shelf zero-shot embedders with the current adaptation differences.","section":"Section II-A, Table II, Section VI-A"},{"comment":"The label distance metric Dn is underspecified for multi-label evaluation. Equations (5)-(7) define distance between a single predicted label p and a single true label t, but each requirement has up to k predicted labels and a variable number of true labels. The paper does not state how predicted and true labels are paired before averaging: if all predicted/true pairs are averaged, the metric is sensitive to k and to label-set size in a way that is not discussed; if a matching or nearest-neighbor rule is used, it must be defined. Since Dn is a central contribution and is used for model selection, this aggregation rule is load-bearing.","section":"Section IV-F, Eqs. (5)-(7)"},{"comment":"F189 is applied uniformly to all output spaces even though Table IV computes output-space-specific β values (ranging from 189 to 617). The paper justifies this by noting that Fβ approximates recall for these β values, but then Fβ is still used as a dependent variable in the RQ2 statistical tests. Consequently, the 'model characteristics impact' results for Fβ are essentially recall results, and the precision dimension is absent from those tests. The authors' own statement in Section V-A that Fβ is 'not useful to identify the top-performing model' further undercuts its use as a primary outcome; this should be addressed in the analysis design.","section":"Section IV-E, Table IV, Table VI"},{"comment":"The Kruskal-Wallis test on embedding size includes groups with a single observation (embedding size 384 has n=1 for MiniLM, and 1024 has n=1 for RoBERTa-large). Comparing group medians with singleton groups is not meaningful, and the post hoc claim of a significant difference between 4096 and these singleton groups is not a robust result. Please re-analyze the embedding-size effect with a more appropriate method (e.g., treating embedding size as a variable with sufficient replication per level, or reporting per-level distributions without Kruskal-Wallis inference).","section":"Table VI, Section V-B"}],"minor_comments":[{"comment":"The sentence 'The difficulties increases when a limited number of requirements is available' contains a subject-verb agreement error and should be revised.","section":"Abstract"},{"comment":"The phrase 'as depicted in Figure IV-F' is incorrect; the reference should be to Table V or Section IV-F, not a nonexistent figure.","section":"Section VI-B"},{"comment":"In the Internal Validity paragraph, 'To mitigate this treat' should read 'To mitigate this threat.'","section":"Section IV-H"},{"comment":"The model listed as 'BERT (base)' uses checkpoint all-MiniLM-L12-v2, which is a distilled MiniLM model rather than BERT-base. Please rename the row to avoid misleading the reader.","section":"Table II"},{"comment":"The lhead column sums to 442, which exceeds the 377 requirements reported in Section IV-B; please clarify how lhead is defined per output space and reconcile this with the dataset size.","section":"Table IV"},{"comment":"The statistical analysis would benefit from effect sizes and confidence intervals for the significant Kruskal-Wallis and Spearman results, not only p-values.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The core empirical contribution is potentially valuable, and the replication package is a clear strength. The main concern is the single-k reporting: the authors say they ran a k-sweep but do not report it, and all conclusions depend on k. This is likely fixable in a revision. The inconsistencies between the abstract, Section V-A, and Table V should also be corrected before the paper can be accepted. No issue of misconduct is apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: a solid, practically useful empirical paper, and the central directional claim—sentence-transformer LMs beat raw LLM embeddings for zero-shot multi-label requirements classification against a large taxonomy—is very likely correct. But the headline rankings all come from k=15, a single operating point the authors reached without reporting the sweep, so the conclusions are more conditional than the abstract suggests.\n\nWhat is new: this is the first systematic comparison I know of 14 LMs/LLMs on six hierarchical output spaces for multi-label requirements classification. That scale is useful, and the replication package with code and 342 annotated requirements is a real plus. The finding that architecture and sentence-embedding fine-tuning matter more than parameter count is actionable. The label distance metric Dn is adapted from Eisner et al., and the authors say so; as a model-selection diagnostic it is more discriminative than Fβ here, though it is under-specified for the multi-label setting (how are predicted and true labels paired before averaging?) and the authors themselves flag the need for a thorough evaluation.\n\nSoft spots, in order. First, the single k=15 cutoff. The stress-test note is right: precision, recall, Fβ and Dn all depend on k, and the authors admit they ran other values but don't report them. Without that sweep, 'T5-xl is best on 5/6 output spaces' and 'smaller beats larger' are claims about one operating point. Fixable in revision. Second, Fβ: they compute F189 for every output space even though the beta values differ; their justification is plausible—for beta ~200, Fβ approximates recall—but the upshot is that Fβ carries no precision information, and Dn is not yet validated as replacement. Third, no error bars or uncertainty estimates; the dataset is small (377 requirements), and model rankings vary across output spaces, so some stability analysis would help. Minor: 35 of 377 requirements are withheld for confidentiality; defensible, but full replication of the ground truth is limited. Citation pattern is honest; the self-citation to the dataset source is legitimate.\n\nWho this is for: requirements engineering and traceability researchers, and practitioners who want a low-compute zero-shot classifier. It deserves a serious referee. A referee should request the k-sweep results, a precise definition of how Dn handles multiple predicted and true labels, and some sensitivity analysis before acceptance.","headline":"A useful, reproducible empirical comparison showing SBERT-style LMs beat raw LLM embeddings for zero-shot hierarchical multi-label requirements classification, but every ranking is tied to a single k=15 operating point; worth refereeing with revisions.","tokens_in":17752,"tokens_out":3017,"would_cite":true,"duration_ms":28656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot classifiers can label industrial requirements into large taxonomies, and smaller sentence-embedding models beat massive LLMs.","keywords":["zero-shot learning","multi-label classification","requirements classification","language models","hierarchical taxonomy","label distance metric","requirements tracing","industrial dataset"],"falsifier":"Re-running the same 14 models on the same six output spaces with k = 5, 10, 20, and 30 would settle the central claim: if T5-xl is not the distance-metric leader at other cutoffs, or if the negative correlation between model size and $F_\\beta$ reverses, the architecture-not-scale conclusion fails.","tokens_in":16785,"feed_emoji":"🏷️","tokens_out":9333,"duration_ms":75887,"temperature":0.7,"pith_summary":"This paper tries to establish that a zero-shot classifier, built by computing cosine similarity between embeddings of requirement text and embeddings of taxonomy class descriptions, can do useful multi-label classification of industrial requirements against large hierarchical taxonomies without any labeled training data. The motivation is practical: requirements engineering teams often have hundreds of classes but too few annotated requirements to train supervised models. The paper compares 14 language models, from 66 million to 70 billion parameters, on six taxonomy-based output spaces with 377 requirements and 1,968 labels. It reports that the sentence-to-sequence model T5-xl gives the best distance-aware performance on five of six output spaces, that smaller models beat much larger LLMs, and that model architecture matters more than parameter count. If true, this means traceability support can be deployed cheaply on modest hardware.","feed_headline":"Small language models beat giant LLMs at tagging requirements","feed_subtitle":"A zero-shot T5-xl classifier hits Fβ 0.78 on industrial requirements, with no labeled training data.","key_machinery":"The load-bearing mechanism is a zero-shot embedding-similarity pipeline. For each requirement, the text is concatenated with its document and section titles; for each taxonomy class, the node description is aggregated with its children's descriptions (the 'hierarchical strategy'). A language model produces embeddings for both sides, and cosine similarity ranks classes, of which the top $k$ are returned as labels. The paper's proposed evaluation device is the normalized label distance metric $D_n = D_a / D_{\\max}$, where $D_a$ is the number of hops between predicted and true labels in the taxonomy and $D_{\\max} = 2 \\times \\mathrm{depth}$; this continuous measure in $[0,1]$ is what makes T5-xl's advantage visible consistently across output spaces where $F_1$ and $F_\\beta$ are ambiguous.","core_discovery":"On the paper's own terms, the central discovery is that zero-shot language-model classification is a workable approach to multi-label requirements classification against large domain taxonomies, and that model selection should be driven by architecture and sentence-embedding fine-tuning rather than raw scale. In the reported experiment, T5-xl (an encoder-decoder model with 3B parameters) achieved the best label-distance performance on five of the six output spaces, with a maximum $F_eta = 0.78$ and a best normalized distance $D_n = 0.04$; a BERT-family MiniLM sentence-embedding model outperformed all other models on the remaining output space with $F_eta = 0.83$ and $D_n = 0.04$. Across the board, autoencoding and sequence-to-sequence models with up to 3B parameters outperformed autoregressive LLMs with up to 70B parameters, and the model type had a statistically significant effect on performance. The paper also claims that its proposed normalized label distance metric $D_n$ identifies the top-performing model more consistently across output spaces than $F_1$ or $F_eta$.","pith_inferences":["Editorial inference: because only k=15 results are reported, the 'small models beat large models' ordering is untested at other k values; repeating the comparison at k=5 and k=30 would show whether it is a stable property of architecture or an artifact of the cutoff.","Editorial inference: the top models' shared trait is sentence-transformer fine-tuning for cosine similarity, so the paper's architecture conclusion is really about training objective; an autoregressive LLM fine-tuned for embedding generation might behave differently.","Editorial inference: the $D_n$ metric should transfer to other hierarchical multi-label tasks such as medical coding or product categorization, and its practical value could be tested by measuring whether recommendations within one hop speed up human label selection.","Editorial inference: because the classifier includes document and section titles as context, its success may depend on context-rich requirement documents; applying it to requirements without such context would likely lower performance."],"forward_implications":["Companies with scarce labeled requirements can deploy a zero-shot classifier immediately, cutting the label space an engineer must review from hundreds of classes to 15 candidates.","Model selection for this task should prefer sentence-embedding models and consider architecture first; parameter count beyond the 3B range does not pay off.","A human-in-the-loop tracing tool can use $D_n$ to display predicted labels and their neighbors within a small taxonomy distance, reducing search effort.","Supervised approaches for multi-label requirements classification now have a concrete zero-shot baseline to beat on this industrial dataset.","The $D_n$ metric gives a consistent cross-output-space ranking, making it a candidate for model selection in other hierarchical classification problems."],"supporting_citations":[{"why":"It supplies the 377-requirement, 1,968-label industrial dataset, the two taxonomies, and the hierarchical child-description aggregation strategy the classifier reuses.","marker":"[1]"},{"why":"It introduces the zero-shot requirements-classification approach based on pre-trained language model embeddings that this study extends to multi-label, large-taxonomy settings.","marker":"[2]"},{"why":"It defines the hairy-requirements-task concept and the context-dependent $F_\\beta$ formula used to set $\\beta=189$ for the evaluation.","marker":"[4]"},{"why":"It provides the hierarchical-distance idea that the proposed $D_n$ label distance metric adapts to taxonomy outputs.","marker":"[12]"},{"why":"It supplies the Sentence-T5 checkpoints, including the T5-xl model that is the paper's best performer on five of six output spaces.","marker":"[25]"},{"why":"It establishes the sentence-embedding and cosine-similarity approach shared by all top-performing autoencoding and sequence-to-sequence models.","marker":"[30]"},{"why":"It supplies the Llama 2 family, the largest autoregressive LLMs in the comparison that the smaller-beats-larger result depends on.","marker":"[37]"},{"why":"It defines the taxonomic-trace-link problem and the earlier word2vec classifier that motivates the need for better zero-shot classification.","marker":"[38]"}],"fun_headline_variants":["Small LMs beat 70B LLMs at zero-shot requirements tagging","T5-xl tops zero-shot industrial label classification, not Llama","Zero-shot T5-xl hits Fβ 0.78 for multi-label requirements","For requirements tagging, 3B T5 beats 70B LLMs","Novel metric D_n picks best zero-shot classifier for requirements"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All reported rankings and conclusions assume the classifier returns exactly 15 suggested labels per requirement; the authors tried other cutoff values but report only 15, so the model ordering and the smaller-beats-larger conclusion are conditioned on that unstated choice.","fun_headline_variants_meta":{"raw":{"variants":["Small LMs beat 70B LLMs at zero-shot requirements tagging","T5-xl tops zero-shot industrial label classification, not Llama","Zero-shot T5-xl hits Fβ 0.78 for multi-label requirements","For requirements tagging, 3B T5 beats 70B LLMs","Novel metric D_n picks best zero-shot classifier for requirements"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":3090,"prompt_tokens":1163,"completion_tokens":1927,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":779,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":779,"tokens_out":1927,"duration_ms":12917,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:14:37.081644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same 14 models on the same six output spaces with k = 5, 10, 20, and 30 would settle the central claim: if T5-xl is not the distance-metric leader at other cutoffs, or if the negative correlation between model size and $F_\\beta$ reverses, the architecture-not-scale conclusion fails.","supporting_citations":[{"cited_title":"Multi-label requirements classification with large taxonomies,","cited_arxiv_id":null,"evidence_quote":"It supplies the 377-requirement, 1,968-label industrial dataset, the two taxonomies, and the hierarchical child-description aggregation strategy the classifier reuses."},{"cited_title":"Zero-shot learning for require- ments classification: An exploratory study,","cited_arxiv_id":null,"evidence_quote":"It introduces the zero-shot requirements-classification approach based on pre-trained language model embeddings that this study extends to multi-label, large-taxonomy settings."},{"cited_title":"Empirical evaluation of tools for hairy requirements engineering tasks,","cited_arxiv_id":null,"evidence_quote":"It defines the hairy-requirements-task concept and the context-dependent $F_\\beta$ formula used to set $\\beta=189$ for the evaluation."},{"cited_title":"Improving Protein Function Prediction using the Hierarchical Structure of the Gene Ontology,","cited_arxiv_id":null,"evidence_quote":"It provides the hierarchical-distance idea that the proposed $D_n$ label distance metric adapts to taxonomy outputs."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"It establishes the sentence-embedding and cosine-similarity approach shared by all top-performing autoencoding and sequence-to-sequence models."},{"cited_title":"TT-RecS: The taxonomic trace recommender system,","cited_arxiv_id":null,"evidence_quote":"It defines the taxonomic-trace-link problem and the earlier word2vec classifier that motivates the need for better zero-shot classification."}],"review_version":1}