{"id":"ec67ee2d-2f58-408a-8d28-688ef1bcc5b6","arxiv_id":"2504.15427","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A retrieval-augmented LLM approach validates and recovers traceability links between stakeholder and system requirements for automotive diagnostic trouble codes, reporting 98.87% validation accuracy and 85.50% recovery correctness on industrial data.","lead":"TVR is a new system that uses large language models and retrieval-augmented generation to check whether automotive stakeholder requirements are correctly linked to system requirements. On a real industrial dataset of diagnostic trouble codes, it reports 98.87% validation accuracy and 85.50% correctness when recovering missing links.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recovery correctness is computed only over the 502 links TVR predicted, so the 85.50% claim would hold even if the step-3 message-overlap filter silently dropped true missing links.","rationale":"I agree with the reader that the ground-truth reliability and the recovery-precision-only reporting are the two soft spots, and that the headline validation result should not be accepted without qualification. However, the reader's phrasing leans on 'if a filter removes a real link the 85.50% result overstates completeness,' which is exactly the precision/recall conflation I consider the most load-bearing issue: the metric literally cannot see filter false negatives, so the claim 'recover missing ones' is under-supported. The correct fix is not merely a caveat; it determines what TVR can claim to do. The three-step filter is also presented as lossless on the basis of design intent rather than measurement, and Section 3.3's assertion that the steps 'preserve all potential missing links' is the decisive untested premise. For this I recommend keeping CONDITIONAL rather than moving to REJECT, because the validation result is strong, reproducible (code is public, temperature 0, LOO CV), and the recovery claim is explicitly about correctness rather than recall; but the paper must either add a filtered-samples check or soften the recovery claim to 'high precision on filtered candidates.'","tokens_in":20901,"tokens_out":1473,"duration_ms":13049,"concrete_test":"Re-verify a sample (e.g., 300-500) of the 14,494 pairs that survived Steps 1-2 but were removed by Step 3, using author verification or engineer review following the same annotation protocol; count how many are actually valid traceability links. Also repeat recovery on the full union of surviving and removed pairs for one DTC type, using TVR directly on the Cartesian product, and recompute recovery precision/recall with the preserved filter versus the unfiltered candidate set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The recovery result in Section 5.4 / Eq. 5 is a precision measure: of the 1,919 pairs surviving the Section 3.3 filters, TVR predicted 502 links, and authors verified 85.50% of those (k=429) as correct. This says nothing about recall or completeness. Section 7 claims the three rules 'were designed to ensure that no true traceability links were inadvertently removed,' but no analysis is reported to establish that preservation claim on this dataset. Step 3 is the fragile one: message-overlap matching tokens are extracted from stakeholder and system requirements after a domain-specific stop-word removal, and if a genuine stakeholder/system pair expresses the key message with different surface tokens or with a pseudonym mismatch, the pair never reaches TVR and is invisible to the 85.50% metric. Because recovery is evaluated only as correctness, a filter that removed half of the true missing links would not change any reported number. The same filter also raises potential circularity for validation: in leave-one-out validation, labels and the positive/negative examples both come from the same two-author annotation, so the accuracy estimate is tied to that ground truth; but the more acute concern for recovery is that the filter is itself a hypothesized traceability rule whose false-negative rate is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TVR, a retrieval-augmented generation approach for validating and recovering traceability links between stakeholder and system requirements for diagnostic trouble codes (DTCs) in an automotive industrial dataset. For validation, TVR retrieves k valid and k invalid labeled requirement pairs most similar to a given pair and includes them in the LLM prompt, asking the model to decide whether the system requirement covers the message or signal in the stakeholder requirement. The authors evaluate 13 LLMs, four prompting strategies, two similarity measures, and K = 1...8 using leave-one-out cross-validation on 2,132 links, reporting 98.87% accuracy and 96.72% Macro-F1 for their best configuration (Claude 3.5 Sonnet, cosine similarity, K = 3), significantly above LiSSA and TF-IDF/SBERT retrieval baselines. A variation-based cross-validation reports 97.13% accuracy. For recovery, a three-step filter plus TVR over the remaining 1,919 candidate pairs yields 502 predicted missing links with 85.50% human-verified correctness.","tokens_in":21145,"tokens_out":8921,"duration_ms":77003,"significance":"The paper addresses a genuine industrial problem with a plausible method and a largely transparent evaluation: leave-one-out cross-validation, multiple baselines, Fisher's exact tests, a broad model landscape, and a public replication package are strengths. If the performance holds out of sample, a RAG-based in-context demonstration approach for validating domain-specific requirement links is a useful practical contribution to automotive requirements engineering. However, the current evidence has two gaps that bear directly on the headline claims: the reported validation accuracy is selected over configurations assessed on the same test folds, and the recovery metric is precision-only with no assessment of the filter's false negatives. These are fixable with additional experiments, but the claims as written are stronger than the evidence.","major_comments":[{"comment":"The headline validation result (98.87% accuracy, 96.72% Macro-F1) is the best configuration selected from K = 1...8 and cosine/Euclidean similarity after observing the same leave-one-out test outcomes. Because the same test data were used both to choose K and the similarity measure and to report the final accuracy, the reported number is an optimistically biased estimate of the method's expected accuracy; the Fisher's exact tests cited in §5.2 do not account for this selection. Please either use a nested validation procedure (e.g., select K on a validation fold and evaluate on a held-out fold) or report the full distribution over configurations with an appropriate correction, and qualify the abstract and RQ2 conclusion accordingly.","section":"§5.2, Table 1, Fig. 5"},{"comment":"The recovery claim rests on Correctness (Eq. 5), which is precision over the 502 links that TVR predicted among the 1,919 candidate pairs surviving the three preprocessing steps. The assertion in §7 that the manually defined rules 'were designed to ensure that no true traceability links were inadvertently removed' is not backed by any analysis. In particular, Step 3 (message-overlap matching) can silently remove a true missing link if the stakeholder and system requirements express the same message with different surface tokens or pseudonyms, and such a pair is then invisible to the 85.50% metric, so the recovery result says nothing about recall. I request a false-negative analysis of the filter (e.g., check how many of the 1,913 known-valid links survive the three steps, or sample pairs removed at Step 3 and manually verify them) and reporting of recall and F1 for recovery; the abstract and RQ4 conclusion should be adjusted until such evidence is provided.","section":"§5.4 and Eq. (5), with §3.3 and §7"},{"comment":"In the leave-one-out setup, the test pair ⟨stakeReq, sysReq⟩ is removed from the retrieval database, but Dtrain still contains all other links that share the same sysReq (and sometimes the same stakeReq). Because the retriever embeds the concatenation of both requirements, those near-duplicate pairs are likely to dominate the k positive and k negative retrieved examples, effectively giving the LLM a small labeled set for the exact same system requirement. This may inflate accuracy relative to a deployment scenario where the link being validated concerns a requirement not present in the retrieval database. Please report an additional experiment where all links involving the test pair's sysReq are excluded from Dtrain, and discuss how performance changes.","section":"§4.3 and §5.2"}],"minor_comments":[{"comment":"The abstract and introduction state '98.87% accuracy' and '85.50% correctness' without qualification; these should be presented as the best configuration observed and as a precision-oriented recovery metric, respectively, ideally with a pointer to the relevant sections.","section":"Abstract and §1"},{"comment":"The dataset description would benefit from an explicit statement that the 48 system requirements are each typically linked to multiple stakeholder requirements; this makes clear why the retrieval near-duplicate issue in the leave-one-out protocol can arise.","section":"§4.3"},{"comment":"The term 'unseen variations' is somewhat strong: the four variation categories are all identified from the same dataset, so the evaluation shows generalization to held-out instances of known templates rather than to genuinely new template structures. A sentence qualifying this in the RQ3 answer would improve precision.","section":"§5.3, Table 2"},{"comment":"The Fisher's exact test p-values are reported for a few pairwise comparisons only; a compact table with all configuration comparisons or a statement about the number of tests performed would help readers evaluate the selection effect discussed in the major comments.","section":"§5.2"},{"comment":"The phrase 'preserving all potential missing links' before the description of the three-step filtering steps in Section 3.3 is an unverified claim; consider rephrasing it as a design goal or moving the supporting evidence from Section 7 into Section 3.3.","section":"§3.3"},{"comment":"Minor typographical and formatting issues include inconsistent capitalization of 'VARIATION' in the text and figures, and some reference entries (e.g., [5] and [6]) appear to have identical titles; these should be cleaned up in a revision.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-engineering empirical venue. The main threat to the validation claim is the configuration selection on the same test folds and the near-duplicate retrieval issue; the recovery claim additionally lacks a recall analysis. All of these are addressable within the manuscript's scope, so I recommend major revision rather than rejection. The non-disclosure of the dataset is handled honestly with fictitious examples, and the replication package is a positive point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on TVR (arXiv:2504.15427): this is a solid, transparent industrial case study of RAG-based LLM validation for automotive traceability, and the core empirical finding—retrieving balanced valid and invalid requirement pairs as in-context demonstrations beats zero-shot, CoT, few-shot, and LiSSA-style baselines—holds up on their data. Just don't take the headline numbers at face value: 98.87% is the best of a sixteen-configuration grid selected on the same test outcomes, and the 85.50% recovery figure is precision-only, with the candidate filter's false-negative rate untested.\n\nWhat's new: Hey et al. [61] and LiSSA both precede TVR on RAG-based traceability, so the abstract's claim that existing approaches don't address stakeholder-to-system traceability is contradicted by the paper's own related work. That said, TVR's specific twist—retrieving both positive and negative pairs as explicit demonstrations, and validating existing stakeholder-to-system DTC links rather than recovering links from scratch—is a legitimate new application, and it targets a real pain point (ASPICE/ISO 26262 compliance).\n\nWhat they do well: leave-one-out cross-validation, 13 LLMs, four prompting strategies, a k and similarity-measure sweep, Fisher's exact tests, a thoughtful cross-variation robustness experiment, error analysis on every recovery false positive, and a public replication package. Given the 89.7% valid-class imbalance, reporting macro-F1 and invalid-class F1 rather than leaning on accuracy is the right call.\n\nSoft spots, proportionately. First, configuration selection: k=3/cosine was chosen after seeing the same leave-one-out outcomes, so 98.87% is the maximum of a small grid. The Fisher p-values against self-consistency (0.0011) and few-shot (0.0004) survive a Bonferroni correction for 16 configs, so the relative conclusion is safe; just read the headline as best-of-grid, not expected performance. Second, and more load-bearing for the recovery claim: correctness is computed only over the 502 pairs TVR predicted. A true missing link dropped by the step-3 message-overlap filter never appears in any reported number, and Section 7's assertion that the rules were 'designed to ensure no true traceability links were inadvertently removed' is unsupported by any analysis. The stress-test note lands here. Third, the dataset is private and the labels are author-made; kappa of 0.962 helps, but the retrieved demonstrations come from the same annotation, so the estimates are conditional on that ground truth. Minor: the LiSSA baseline is reimplemented with Claude 3.5, which is reasonable given data privacy but is not the published LiSSA.\n\nWho it's for: requirements-engineering researchers and automotive practitioners who want an LLM-based validation tool with an honest evaluation. It deserves a serious referee, with required revision: either measure recovery recall on a sampled candidate set or hedge the recovery claim, tone down the abstract, and acknowledge the config-selection caveat.","headline":"Solid, transparent industrial case study—retrieved positive/negative demonstrations genuinely help LLM link validation—but the 98.87% headline is best-of-grid and the 85.50% recovery number is precision-only with an untested filter.","tokens_in":21664,"tokens_out":5278,"would_cite":true,"duration_ms":42839,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes TVR, a retrieval-augmented LLM method that validates and recovers requirement traceability links in automotive systems, reporting 98.87% validation accuracy and 85.50% recovery correctness on industrial DTC data.","keywords":["requirements traceability","retrieval-augmented generation","large language models","automotive systems","diagnostic trouble codes","traceability validation","traceability recovery","in-context learning"],"falsifier":"Re-annotate the same 2,132 links with independent automotive engineers who do not see TVR's outputs, then rerun leave-one-out evaluation; a material drop below 98.87% would show the ground truth and the retrieved examples were not independent of the tested labels. For recovery, manually inspect a random sample of the pairs removed by the three-step filter; finding genuine missing links there would show that the 85.50% figure measures precision only and overstates completeness.","tokens_in":20693,"feed_emoji":"🔗","tokens_out":6068,"duration_ms":53169,"temperature":0.7,"pith_summary":"This paper claims that requirement traceability in automotive systems can be checked automatically by an LLM that is shown similar already-labeled requirement pairs before it judges a new pair. On 2,132 industrial diagnostic-trouble-code (DTC) links between stakeholder and system requirements, the proposed method, TVR, reaches 98.87% accuracy and 96.72% Macro-F1 for validation, and identifies missing links with 85.50% correctness after human verification. The method holds up when tested on requirement variations not seen in the retrieval examples, keeping 97.13% accuracy. A sympathetic reader would care because traceability is currently checked by hand for weeks each year, and mistakes propagate into safety-critical automotive functions.","feed_headline":"Teaching an LLM with similar pairs validates 98.87% of links","feed_subtitle":"Retrieved valid and invalid requirement pairs guide the model to check message coverage, beating similarity-only baselines on industrial…","key_machinery":"The load-bearing mechanism is retrieval-augmented in-context learning with balanced demonstrations. The retriever embeds whole requirement pairs with the partner-approved embedding model, finds the k=3 closest valid pairs and k=3 closest invalid pairs by cosine similarity, and the generator then receives a prompt that defines the task narrowly: check only whether the message or signal from the stakeholder requirement is covered by the system requirement. The retrieved examples do the teaching work: they let the model infer the domain-specific notion of a valid link without retraining, and the explicit inclusion of invalid examples shows it what near-miss pairs look like. The paper's recovery mechanism relies on a three-step filter (same DTC type, same mature or demature condition, and at least one shared message) to shrink the candidate space before the same validator is applied.","core_discovery":"The central claim, stated on the paper's own terms, is that traceability validation for DTC requirements is a message-coverage question rather than a text-similarity question: a system requirement is correctly linked to a stakeholder requirement when the system requirement actually covers the message or signal named by the stakeholder requirement. TVR operationalizes this by retrieving the k most similar valid pairs and the k most similar invalid pairs from a labeled database, concatenating them into the prompt, and asking the LLM to answer Yes or No for the candidate pair. With cosine similarity and k=3, the paper reports 98.87% accuracy and 96.72% Macro-F1, a clear margin over plain zero-shot, chain-of-thought, few-shot, and self-consistency prompts and over retrieval-only baselines whose Macro-F1 peaks below 58%. For recovery, three filtering rules reduce a near-Cartesian candidate set to 1,919 pairs, TVR predicts 502 missing links, and human verification confirms 85.50% of them. The paper also reports robustness to unseen requirement variations at 97.13% accuracy.","pith_inferences":["This implies a bootstrapping strategy: a small seed set of validated links could start the loop, with engineers verifying only model-flagged pairs until the database is large enough to maintain 98%-class accuracy; the paper does not test how small a seed can be.","Because the prompt defines traceability narrowly as message or signal coverage, the method may transfer to other artifact pairs whose validity is similarly local, such as a code change covering a requirement's named variable; this transfer is an editorial inference, not a paper result.","The recovery filter's completeness assumption is untested: if any true missing link shares no message token, it is removed before TVR runs, so measuring recall would require sampling the filtered-out pairs.","One testable extension is to replace the fixed k=3 with per-DTC-type or per-variation retrieval counts, since the paper's robustness results suggest some variation categories are harder than others."],"forward_implications":["Engineers can use TVR to audit existing DTC traceability links at near-99% accuracy, turning a weeks-long manual review into a review of model-flagged exceptions.","The 85.50% recovery correctness means roughly 7 in 8 automatically proposed missing links are right, so recovery can be used as a suggestion engine with human sign-off on the remainder.","Because the retrieval database grows as engineers confirm links, each validation cycle should improve future prompts without retraining or prompt rewriting.","The large gap over similarity-only baselines suggests lexical or embedding distance alone is not sufficient for this requirement level, so traceability tooling in safety-critical domains needs label-guided reasoning rather than thresholds."],"supporting_citations":[{"why":"Supplies the main LLM-based validation baseline (retrieval plus simple prompts) that TVR must outperform, and motivates the retrieve-then-validate design.","marker":"[26]"},{"why":"Supports the idea of retrieving similar examples to serve as in-context demonstrations for the generator.","marker":"[35]"},{"why":"Provides the implementation pattern for the retrieval-only TF-IDF baseline used as an upper-bound comparator.","marker":"[15]"},{"why":"Supplies the sentence-embedding model used in the retrieval-only semantic baseline.","marker":"[43]"},{"why":"Justifies leave-one-out cross-validation, the evaluation design that reuses every labeled link as a test case.","marker":"[41]"},{"why":"Provides the embedding model used by the retriever to represent concatenated requirement pairs.","marker":"[37]"},{"why":"Provides the similarity-search library used to compute cosine similarity and retrieve the top-k pairs.","marker":"[38]"},{"why":"Grounds the expectation that LLMs handle task variation and in-context examples without retraining, motivating the RAG design.","marker":"[24]"},{"why":"Shows that prompt design matters for LLM-based traceability and is an early approach the paper distinguishes itself from.","marker":"[27]"}],"fun_headline_variants":["RAG-based traceability validator hits 98.87% accuracy","LLM with retrieved pairs recovers 85.5% missing links","Message coverage, not similarity, validates traceability links","TVR: 98.87% traceability accuracy via RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the two authors' manual labels of the 2,132 links are correct ground truth, and the recovery results also assume the three-step filter never discards a real missing link. If the labels are wrong or the filter drops true links, the reported accuracy and recovery correctness no longer measure what they appear to measure.","fun_headline_variants_meta":{"raw":{"variants":["RAG-based traceability validator hits 98.87% accuracy","LLM with retrieved pairs recovers 85.5% missing links","Message coverage, not similarity, validates traceability links","TVR: 98.87% traceability accuracy via RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4606,"prompt_tokens":966,"completion_tokens":3640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":3566}},"tokens_in":582,"tokens_out":3640,"duration_ms":24266,"temperature":1.0,"reasoning_tokens":3566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:26:57.540317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 2,132 links with independent automotive engineers who do not see TVR's outputs, then rerun leave-one-out evaluation; a material drop below 98.87% would show the ground truth and the retrieved examples were not independent of the tested labels. For recovery, manually inspect a random sample of the pairs removed by the three-step filter; finding genuine missing links there would show that the 85.50% figure measures precision only and overstates completeness.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/ACM 47th International Conference on Software Engineering","cited_arxiv_id":null,"evidence_quote":"Supplies the main LLM-based validation baseline (retrieval plus simple prompts) that TVR must outperform, and motivates the retrieve-then-validate design."},{"cited_title":"In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp","cited_arxiv_id":null,"evidence_quote":"Provides the implementation pattern for the retrieval-only TF-IDF baseline used as an upper-bound comparator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies leave-one-out cross-validation, the evaluation design that reuses every labeled link as a test case."},{"cited_title":"https: //docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models","cited_arxiv_id":null,"evidence_quote":"Provides the embedding model used by the retriever to represent concatenated requirement pairs."},{"cited_title":"https://github.com/facebookresearch/faiss","cited_arxiv_id":null,"evidence_quote":"Provides the similarity-search library used to compute cosine similarity and retrieve the top-k pairs."},{"cited_title":"In: 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), pp","cited_arxiv_id":null,"evidence_quote":"Shows that prompt design matters for LLM-based traceability and is an early approach the paper distinguishes itself from."}],"review_version":1}