{"id":"a49c3771-4390-4923-bbdd-0ddd275ac7e6","arxiv_id":"2504.20507","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Tracing artifacts through a shared domain taxonomy is feasible in an infrastructure project and could reduce trace maintenance effort, but the benefits over direct links remain unmeasured.","lead":"This paper proposes Taxonomic Trace Links (TTL), a way to connect software and system artifacts indirectly through a shared domain taxonomy instead of drawing direct links between each pair of artifacts. A small field study in a Swedish railway project suggests the approach is feasible and may cut the cost of maintaining trace links, though benefits are not yet measured against direct links.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classification consistency is the load-bearing premise for TTL feasibility, and the validation's reliability checks leave the error rate largely unmeasured.","rationale":"The paper's central claim, stated in Section 7, is that the validation demonstrates feasibility of implementing TTL to trace requirements to 3D models and to verify requirements in the model. This claim requires that the mediation via a taxonomy preserves semantic meaning: a shared SB11 code between a requirement and a model object must imply a genuine trace relationship. The validation tests exactly this premise in Section 5.2.2, but the tests are underpowered. Only ~30% of objects could be followed across model versions, leaving the intra-subcontractor consistency claim based on a subset. Inter-subcontractor consistency was sampled on 8 of 30 codes, and the method visually compares aggregate object sets rather than tracking individual objects. The paper is transparent about these limits, and it explicitly frames broader benefits as conjectures rather than established results. Therefore the correct verdict remains CONDITIONAL: the mechanism is plausible and the single successful verification of a fence-placement nonconformity is encouraging, but the reliability of the classification layer—on which all TTL benefits depend—is not yet demonstrated. The reader identified the same weakest assumption, and my concrete test would directly measure classification error rates and verification recall. This does not change the verdict because the conditional verdict already reflects the need for that evidence.","tokens_in":17252,"tokens_out":3168,"duration_ms":34204,"concrete_test":"Perform a blind validation on held-out data: have at least two independent domain experts classify a random sample of the 27 requirements and a random sample of model objects from the Eastlink project into SB11 codes. Compute inter-annotator agreement (e.g., Cohen's kappa) and measure precision and recall of TTL-derived requirement-to-object links against a gold standard of directly judged requirement-object correspondences. In addition, seed a small number of known nonconformities into a model (e.g., move a fence into a prohibited area) and check whether the Section 5.2.3 filtering-and-inspection procedure retrieves them. If agreement is high and seeded defects are recovered, the shared-code-as-semantic-relation premise is supported; if not, the feasibility claim weakens proportionally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim—that TTL can trace requirements to 3D design models and verify those requirements—depends on the premise that two artifacts sharing an SB11 code genuinely correspond, i.e., that both the requirement classification and the model-object classification are correct and consistent. The validation does not establish this with sufficient strength. In Section 5.2.2, only about 30% of objects could be traced between model versions; the conclusion that intra-subcontractor classification is consistent therefore rests on a minority of objects, while the remaining 70% could hide inconsistencies. Inter-subcontractor consistency was checked on only 8 of 30 codes, and the visual comparison found just two inconsistent objects; this does not bound the error rate for the unexamined codes or across all objects. Section 6.1 itself states that if classification is inconsistent or incomplete, 'the links may not be reliable, and consequently, the engineers may be unable to use them.' Because a TTL link is a logical conjunction of two classifications joined by a shared code, any classification error silently produces false trace links or missed links, with no immediate mechanism to detect the error. The single detected nonconformity (K29674) demonstrates the verification mechanism but is one selected instance; it does not measure recall of the TTL-based verification process. Thus the empirical support for the key enabling premise—reliable classification across artifacts and stakeholders—is thin, even though the paper honestly acknowledges this limitation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Taxonomic Trace Links (TTL), an approach in which trace links between artifacts are established indirectly by classifying each artifact against a domain-specific taxonomy. It argues that this indirection addresses three traceability challenges: granularity of links, lack of common data structures, and unclear responsibility. The paper reports a field validation in a railway infrastructure project: 27 requirements were classified using the SB11 taxonomy, the SB11 codes of 3D model objects were checked for consistency, and one design nonconformity was found through TTL-based verification. The paper identifies six classification challenges and proposes a research roadmap for realizing TTL in practice.","tokens_in":17541,"tokens_out":6941,"duration_ms":63328,"significance":"The TTL concept is a plausible and potentially useful contribution to traceability research and practice, especially in domains where a taxonomy already exists. The validation study is a strength: it uses real industrial artifacts (Eastlink project), involves practitioners, and honestly reports the limitations of the reliability checks (e.g., only about 30% of objects were traceable between model versions). The paper is also careful to frame most benefit claims as conjectures. However, the empirical evidence is limited to a single project, 27 requirements, and one detected defect; the maintenance-benefit argument is based on a toy example; and the classification-consistency premise is only partially measured. The paper is openly presented as an initial validation, which makes the remaining gaps more acceptable, but the conclusion should be tempered to match the evidence.","major_comments":[{"comment":"The central feasibility claim rests on the premise that classification is comprehensive and consistent across artifacts, but the validation leaves the classification error rate largely unmeasured. Only about 30% of objects could be traced between model versions A1/A2 and B1/B2, and inter-subcontractor consistency was checked on 8 of 30 codes. The paper itself notes in Section 6.1 that if classification is inconsistent or incomplete, TTL links may be unreliable. Because a TTL link is a logical conjunction of two classifications joined by a shared code, any classification error silently produces false or missed links. I recommend softening the Section 7 claim from 'demonstrates that it is feasible' to 'provides initial evidence that it is feasible', or adding a quantitative reliability analysis (for example, a sample-size calculation with a target error-rate bound, or a sensitivity analysis on the untraceable 70% of objects).","section":"Section 5.2.2 / Section 7"},{"comment":"The maintenance-effort benefit is stated as a general result: 'when tracing more than two types of artifacts to each other, indirect trace link maintenance requires less effort than direct trace links.' The only support is a toy example with one requirement, two classes, and two test cases, and the effort arithmetic depends on specific assumptions (e.g., a single requirement is split into two). The claim is not derived from a cost model and may not hold when an artifact is associated with many taxonomy nodes or when the taxonomy itself changes. The paper should either present a more general cost argument or explicitly frame the example as illustrative rather than as a general finding.","section":"Section 2.3"}],"minor_comments":[{"comment":"The paper states in Section 5.1 that it is not an evaluation study, yet the abstract refers to an 'initial validation with practitioners'; the distinction between validation (feasibility) and evaluation (utility) should be clarified in the introduction to set reader expectations.","section":"Section 5.1 / Abstract"},{"comment":"The claim that TTL creates 'up to twice as many links' is not defined in terms of what counts as a link; a precise definition (e.g., a binary association between an artifact and a taxonomy node) would make the maintenance comparison clearer.","section":"Section 2.3"},{"comment":"The text reads 'we found only two objects', but Figure 6 shows 'two apparent different objects'; please clarify whether these are two individual objects or two groups of objects.","section":"Section 5.2.2"},{"comment":"'When we had conflicted opinions' should be 'when we had conflicting opinions'.","section":"Section 5.1"},{"comment":"A table summarizing the six classification challenges with examples and potential mitigations would improve readability and help the reader connect the challenges to the research roadmap.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper relies substantially on the authors' own prior work [11, 15, 49, 50] for the TTL concept and for taxonomy-quality attributes, which creates a self-referential chain, though the validation study with external project artifacts (Trafikverket, HOCHTIEF, SB11) provides independent grounding. The study is honest about its limitations, but the sample is small (27 requirements, two subcontractors, one detected defect) and the classification-error rate is unmeasured. For SoSyM, the fit is reasonable given the modeling/taxonomy angle, but the transfer to software engineering remains speculative. The paper is likely acceptable after a major revision that tempers the feasibility claim and strengthens the reliability discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid conceptual-plus-feasibility paper, not a breakthrough. The TTL idea itself comes from the authors' earlier work, so the novelty here is the packaging: a careful argument mapping indirection through a taxonomy to three longstanding traceability problems, plus a first industrial feasibility study. That mapping is the strongest part. The authors write plainly, cite the relevant traceability literature, and label their benefit claims as conjectures rather than results.\n\nThe validation is genuinely useful as exploratory evidence. Twenty-seven real requirements from the Eastlink project, classified against SB11, with practitioners from HOCHTIEF, and an actual check of whether the traced requirements could be verified in 3D models. They find one concrete nonconformity, which shows the mechanism runs end-to-end. They also surface six implementation challenges that deserve attention. I think that part is worth a serious referee.\n\nThe soft spots are mostly in the gap between the claims and the supporting data. The central premise—that artifacts sharing a taxonomy code share meaning—rests on classification reliability, and the reliability checks are thin. Only about 30% of objects could be followed between model versions, and the 30% that could be followed may not represent the rest. Inter-subcontractor consistency was checked on 8 sampled codes out of 30, visually. That is not enough to bound the error rate, and a TTL link silently inherits any classification error. The authors acknowledge this in Section 6.1, which is honest, but it means the feasibility claim is weaker than the conclusion suggests. The maintenance-effort advantage is argued with toy examples, not measured against a baseline; that is fine for a position paper, less fine if the paper reads as empirical. I would also like to see the sampled requirements and model-classification data made available, because right now none of this can be reproduced.\n\nOn citations: there is a lot of self-citation, but it is legitimate—they are building on their own prior work, and the relevant taxonomy-quality and classification papers are theirs. I don't see a red flag.\n\nWho benefits: traceability researchers, especially people working in regulated domains where a taxonomy already exists. It is a useful roadmap and a source of testable hypotheses. I would accept it for peer review with expectations of major revision, mainly to either narrow the claims or add a proper measurement of classification consistency. I'd bring it to a reading group if the group cares about empirical traceability; otherwise it can wait.","headline":"TTL's conceptual argument is solid and the study is honestly scoped, but the validation is too thin to carry the reliability and maintenance claims; still worth serious refereeing with expectations of major revision.","tokens_in":18064,"tokens_out":2621,"would_cite":true,"duration_ms":29085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that trace links can be routed through a shared domain taxonomy instead of connecting artifacts directly, and reports a field trial that found a real design violation using such links.","keywords":["requirements traceability","taxonomic trace links","domain taxonomy","classification","trace link maintenance","requirements verification","3D design models","traceability challenges"],"falsifier":"Take the same railway setting and have a second, independent team re-classify the 27 requirements and a random sample of 3D model objects, then measure inter-rater agreement on the SB11 codes. If agreement is low, or if two artifacts that share a code are frequently judged by domain experts to be unrelated, the core premise that shared taxonomy classes encode traceability fails. Conversely, an audit of all 30 codes across both subcontractors' models, rather than the 8 sampled, would show whether the inconsistencies observed are rare human slips or systematic.","tokens_in":17113,"feed_emoji":"🛤️","tokens_out":7184,"duration_ms":70321,"temperature":0.7,"pith_summary":"This paper proposes that trace links do not have to connect artifacts directly: each artifact can be classified against a domain-specific taxonomy, and two artifacts are considered traced when they share a taxonomy class, or nearby classes. The argument is that this indirection fixes three practical failures of direct traceability: choosing the right granularity, coping with artifacts that have no common structure or tool, and deciding who owns the links. The paper backs this with a field study in an infrastructure railway project: 26 of 27 sampled natural-language requirements received at least one class from the SB11 taxonomy, objects in 3D design models already carried classes, and filtering by shared classes let the team verify the design against the requirements and find one real violation. The authors present the broader benefits as reasoned conjectures, not measured outcomes, and they list six implementation challenges to resolve next.","feed_headline":"Tracing through a shared taxonomy passes its first field test","feed_subtitle":"In a railway-project trial, 26 of 27 requirements were classified, and tracing them found a real design violation.","key_machinery":"The load-bearing device is the domain-specific taxonomy, a hierarchy of domain classes such as the SB11 construction-classification system with more than 2,000 nodes. Each artifact is multi-labeled with one or more classes, and tracing is a query: find artifacts whose labels equal, parent, child, sibling, or neighbor the source's labels. The taxonomy carries the argument in three ways: its parent-child structure supplies abstraction levels so granularity is chosen per query rather than per link; its fixed vocabulary gives heterogeneous tools a common interface; and splitting each trace into two artifact-to-taxonomy links assigns each artifact's creator responsibility for only their own artifact. Maintenance becomes local because a change to one artifact invalidates only its own taxonomy links, not links to every related artifact.","core_discovery":"The central claim is that a domain taxonomy can act as a neutral intermediary for traceability: classify every artifact against the same hierarchy of domain concepts, and the taxonomy class becomes the trace link. In the validation, 27 requirements from a railway project were manually mapped to SB11 codes, and 26 were mappable to at least one code; the one failure came from a vague, context-dependent requirement. The study then checked the codes already applied to objects in two versions of 3D models from two subcontractors: all objects were coded, a handful of errors were found (typos, one misclassified road element, two inconsistent surface-water classifications), and only about 30% of individual objects could be followed between model versions because exported files lost object identifiers. Despite those limitations, tracing one requirement about fence placement to fence objects in the model revealed fences positioned in a prohibited zone, demonstrating the intended verification use case. The paper's conclusion is phrased as feasibility, not proof of benefit: it is feasible to implement TTL to trace requirements to 3D design models and verify those requirements.","pith_inferences":["The strongest unstated test is classification reliability: the study sampled only 8 of 30 codes for cross-subcontractor consistency and could trace only about 30% of objects across versions, so a full audit of all codes could reveal silent inconsistencies that would undermine TTL's central premise.","In software engineering the bottleneck is likely the taxonomy itself: code, tests, and commits carry semantic names but no pre-existing classification system, so the approach's transfer depends on whether automatically built taxonomies reach the consistency that SB11 has from years of industry use.","A direct experiment could measure the claimed maintenance savings: subject the same requirement change to direct and TTL link maintenance and count link edits; the paper's example predicts TTL wins when more than two artifact types are involved.","The verification workflow is essentially a retrieval task, so its precision and recall could be compared against information-retrieval trace recovery on the same railway data; the paper reports finding one violation but does not quantify how many violations were missed."],"forward_implications":["If a requirement is split or changed, only the links from the requirement to the taxonomy need updating; links from source code and tests to the taxonomy stay valid, whereas direct links would all need rewriting.","Tracing multiple artifact types requires each artifact to carry taxonomy labels rather than links to every other artifact, so the number of links grows linearly with artifacts, not quadratically with artifact-type pairs.","Coverage and verification analyses can be done by filtering a repository by taxonomy class; the railway trial shows this can surface design non-conformances.","In regulated domains with existing taxonomies, new compliance sources only need to be classified against the same taxonomy to become traceable to all related artifacts.","Because each creator only needs to know their own artifact and the taxonomy, link creation can happen at artifact creation time instead of being deferred to a separate tracing role."],"supporting_citations":[{"why":"supplies the trace-link definition (source, target, primary and reverse links) that TTL modifies by adding the taxonomy as intermediary.","marker":"[1]"},{"why":"introduced taxonomic trace links and the pilot idea this paper extends into a field validation.","marker":"[11]"},{"why":"documents the practical traceability failures (granularity, structure, responsibility) that TTL claims to address.","marker":"[4]"},{"why":"frames the study as initial design-science validation rather than a full evaluation, which sets the scope of the feasibility claim.","marker":"[12]"},{"why":"provides the field-experiment methodology used for the practitioner workshops.","marker":"[30]"},{"why":"defines requirements-to-design verification as the usage scenario that the validation study executes.","marker":"[31]"},{"why":"shows traditional classifiers are not yet ready for multi-label taxonomy classification, supporting the roadmap's call for reliable classification.","marker":"[49]"}],"fun_headline_variants":["Taxonomy trace links pass rail trial, uncover fence violation","Shared taxonomy traces requirements, catches design error","26 of 27 requirements traced via taxonomy in field test","Feasibility shown: taxonomy-based tracing in real project"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach stands on the premise that a domain taxonomy can be applied comprehensively and consistently by everyone who creates or edits an artifact, so that a shared code really means a shared concept; if codes are wrong or applied differently by different teams, the resulting trace links silently inherit those errors.","fun_headline_variants_meta":{"raw":{"variants":["Taxonomy trace links pass rail trial, uncover fence violation","Shared taxonomy traces requirements, catches design error","26 of 27 requirements traced via taxonomy in field test","Feasibility shown: taxonomy-based tracing in real project"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3542,"prompt_tokens":937,"completion_tokens":2605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2541}},"tokens_in":553,"tokens_out":2605,"duration_ms":19764,"temperature":1.0,"reasoning_tokens":2541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:26:21.747822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same railway setting and have a second, independent team re-classify the 27 requirements and a random sample of 3D model objects, then measure inter-rater agreement on the SB11 codes. If agreement is low, or if two artifacts that share a code are frequently judged by domain experts to be unrelated, the core premise that shared taxonomy classes encode traceability fails. Conversely, an audit of all 30 codes across both subcontractors' models, rather than the 8 sampled, would show whether the inconsistencies observed are rare human slips or systematic.","supporting_citations":[{"cited_title":"In: Software and Systems Traceability, pp","cited_arxiv_id":null,"evidence_quote":"supplies the trace-link definition (source, target, primary and reverse links) that TTL modifies by adding the taxonomy as intermediary."},{"cited_title":"In: 2020 IEEE 28th International Requirements Engineer- ing Conference (RE), pp","cited_arxiv_id":null,"evidence_quote":"introduced taxonomic trace links and the pilot idea this paper extends into a field validation."},{"cited_title":"Journal of Systems and Software 192, 111389 (2022)","cited_arxiv_id":null,"evidence_quote":"documents the practical traceability failures (granularity, structure, responsibility) that TTL claims to address."},{"cited_title":"Springer, Berling, Heidelberg (2014)","cited_arxiv_id":null,"evidence_quote":"frames the study as initial design-science validation rather than a full evaluation, which sets the scope of the feasibility claim."},{"cited_title":"In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Tech- nology","cited_arxiv_id":null,"evidence_quote":"provides the field-experiment methodology used for the practitioner workshops."},{"cited_title":"In: International Work- ing Conference on Requirements Engineering: Foundation for Software Quality, pp","cited_arxiv_id":null,"evidence_quote":"defines requirements-to-design verification as the usage scenario that the validation study executes."},{"cited_title":"In: 2024 IEEE 32st International Requirements Engineering Conference (RE)","cited_arxiv_id":null,"evidence_quote":"shows traditional classifiers are not yet ready for multi-label taxonomy classification, supporting the roadmap's call for reliable classification."}],"review_version":1}