{"id":"1ad6d7b4-9802-4001-8324-12a144c0e2c8","arxiv_id":"2608.01592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study shows a quality-efficiency trade-off in LLM-generated SE taxonomies: TnT-LLM approaches human quality but over-generates, while CLIMB is 15-40x faster yet weaker on latent-concept categories.","lead":"This paper compares two automated methods for building taxonomies from software engineering data, using eight human-built taxonomies as references. It finds TnT-LLM matches human quality at higher cost and complexity, while CLIMB is far cheaper and faster but struggles when categories require technical inference.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3 reliability rests on unvalidated LLM annotators: the 'broadly usable' claim and Table 11 correlations are inter-LLM results, with no human annotation baseline; a targeted human annotation study would settle it.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: RQ3's reliability claims depend on LLM annotators whose behavior has never been validated against human annotators on the annotation task. My read agrees with this assessment. The paper does substantial methodological work for RQ1 (human validation of evaluator LLMs) and RQ2 (externally grounded alignment metrics), and the quality-efficiency trade-off is plausible and mostly supported. But the RQ3 conclusions, including the 'broadly usable' statement and the correlations in Table 11, are only about inter-LLM agreement. Since the central contribution is framed as a multi-perspective evaluation and the practical-use recommendation draws on reliability, this gap warrants a conditional verdict. The reader already reached CONDITIONAL, so my stress-test does not move the verdict; it reinforces the condition and specifies a concrete human annotation check that would resolve the concern. I do not see an additional concern that would require rejection: the RQ1 validation, RQ2 metrics, and RQ4 cost/runtime measurements are reported transparently, and the structural-matching issue is disclosed and partly addressed through Nodes Ratio reporting.","tokens_in":21412,"tokens_out":7887,"duration_ms":74642,"concrete_test":"Recruit three independent human annotators with SE background (not paper authors) to apply a stratified subset of taxonomies: two generated taxonomies per method, one from a surface-cue dataset such as Fairness API Issue Topic and one from a latent-inference dataset such as Dockerfile Flakiness, plus the corresponding human-defined taxonomies. Have them label 50-100 instances per taxonomy using the exact annotation prompt and category definitions from the replication package. Compute per-layer Krippendorff's alpha and label utilization for the human annotators, then compare Human-Human alpha against the LLM-annotator alpha in Tables 9-10 and against Human-LLM pairwise agreement. If the mean pairwise |Human-LLM - Human-Human| is below the Section 6.2 threshold (<0.15) and Human-Human alpha falls within the reported LLM ranges, RQ3 stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the substitution of LLM annotators for human annotators in RQ3 (Section 9.1). The paper carefully validates LLM-as-a-judge for RQ1 quality scoring against two human evaluators (Section 6, Tables 5-6) and even checks self-bias for RQ1, but no analogous validation exists for the RQ3 annotation task. The 'Human' rows in Tables 9-10 are LLM annotations of the human-defined taxonomies, not human annotations. As a result, the RQ3 claims (agreement 0.489-0.647, label utilization 0.859-0.981, 'broadly usable') describe inter-LLM agreement. LLM annotators share systematic biases from common pretraining and instruction-following conventions, so their agreement may be higher or lower than human annotators' agreement. This matters because RQ3 also feeds the practical-usability conclusion and the Table 11 correlations that guide using reliability as feedback for refining ambiguous categories. The paper's own Section 12 construct-validity note concedes that LLM judgments 'may not perfectly reflect how human experts would assess taxonomy quality (RQ1) or how annotators would apply the same taxonomies in practice (RQ3).' Without a human annotation baseline, the practical applicability claim remains conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This experience paper proposes a multi-perspective evaluation framework for automated taxonomy generation on software engineering (SE) artifacts, covering taxonomy quality (RQ1), alignment with human-defined taxonomies (RQ2), reliability under independent annotation (RQ3), and cost/runtime efficiency (RQ4). The authors curate seven SE papers with eight human-defined taxonomies, run two automated methods (TnT-LLM and CLIMB) with five LLMs, and validate their LLM-as-judge quality scoring against two human evaluators in a preliminary study, including a self-bias check. The main finding is a trade-off: TnT-LLM produces taxonomies whose judged quality is comparable to human-defined ones but at substantially higher cost and runtime and with a tendency toward overly complex structures, whereas CLIMB is 15--40x faster and 8--49x cheaper but scores lower on quality when technical inference beyond surface-level similarity is required. The paper also reports that inter-annotator agreement is moderately high and label utilization is high for generated taxonomies, with agreement correlated with category clarity and structural alignment. The study is framed as a first step toward systematic understanding of automated taxonomy generation in SE, with actionable guidance for method selection and human post-processing.","tokens_in":21782,"tokens_out":5915,"duration_ms":52545,"significance":"If the results hold, the paper makes a useful contribution: it is, to my knowledge, the first multi-dataset evaluation of seed-free LLM-based taxonomy generation on technically complex SE artifacts, and it provides a reusable evaluation framework and a curated dataset of eight human-defined taxonomies with public artifacts. The authors' care in validating the RQ1 LLM-as-judge setup against human evaluators on all datasets, checking for self-bias, and reporting a variability analysis for generation and evaluation are concrete methodological strengths. The paper's practical recommendation—to run a small pilot before committing to automated taxonomy generation—is sensible and actionable. However, the significance is conditional: the RQ3 reliability results, which support the 'broadly usable' claim, rest entirely on LLM annotators with no human-annotation baseline, and the structural constraints used for fair method comparison are not always respected. These two issues affect load-bearing parts of the central practical-usability conclusion, so the manuscript needs revision before the findings can be accepted at face value.","major_comments":[{"comment":"The RQ3 reliability results are computed from a panel of LLM annotators, and the rows labeled 'Human' in Tables 9 and 10 are LLM annotations of the human-defined taxonomies, not human annotations. The paper validates the LLM-as-judge setup for RQ1 quality scoring (Section 6) but provides no analogous validation for the RQ3 annotation task. Consequently, the claims 'generated taxonomies are broadly usable' (RQ3 answer) and the Table 11 correlations describe inter-LLM agreement. Because LLM annotators may share systematic biases from common pretraining and instruction-following conventions, their agreement and label utilization may differ from human annotators' agreement. The paper's own Section 12 Construct Validity concedes that LLM judgments 'may not perfectly reflect how human experts would assess taxonomy quality (RQ1) or how annotators would apply the same taxonomies in practice (RQ3).' To support the practical-usability conclusion, the authors should add a human-annotation baseline on a subset of taxonomies, or at minimum reword the RQ3 claims to refer to LLM-based annotation reliability rather than general usability.","section":"Section 9.1, Tables 9–10, Section 12"},{"comment":"The experimental setup states that both methods are instructed to match the structural properties of the human-defined taxonomy (number of layers, top-level categories, and leaf categories) to enable a fair comparison. However, Table 7 reports Nodes Ratios for TnT-LLM as high as 3.341 (Quantum Classical Issue) and 1.294 (Fairness API Issue Topic), which indicate that the structural constraints are not reliably followed. Since the abstract's central trade-off includes 'TnT-LLM tends to generate overly complex taxonomies,' the paper should clarify whether these constraints are enforced (e.g., by post-processing or re-prompting) or are only prompt-level instructions, and discuss how unenforced constraints affect the interpretation of over-generation as a property of the method rather than as an instruction-following artifact. This also bears on the fairness of the RQ1 quality comparison and the RQ3 agreement comparison between methods, because CLIMB's closer adherence to the target size may penalize or benefit it relative to TnT-LLM.","section":"Section 5.2 and Table 7"},{"comment":"The associations reported in Table 11 (e.g., OR-NonOv with rho=0.313 and CEDS with rho=0.461) are correlations between LLM-annotator agreement and LLM-evaluated quality/alignment. Since both sides of each correlation are produced by LLMs, common biases in how LLMs perceive category clarity and apply labels could inflate these associations independently of any true relationship with human annotation behavior. The self-bias check described in Section 12 only covers RQ1 evaluation, not the RQ3 annotation task. A human-annotation baseline would allow the authors to test whether these associations hold for human annotators; without it, the guidance to 'leverage annotation reliability as feedback for refining ambiguous categories' (Section 11.3) is built on inter-LLM agreement only.","section":"Section 9.2, Table 11"}],"minor_comments":[{"comment":"Please clarify the scale and normalization of Catalogue Edit Distance Similarity (CEDS); the reported values (e.g., 52.001 and 59.313 in Table 7) suggest a transformation of a normalized tree-edit distance, but the paper does not state the range or what value indicates perfect similarity. Similarly, Nodes Ratio is defined as a ratio, but the paper should state explicitly that values far from 1 indicate size mismatches and how ties to the instructed structure constraints are handled.","section":"Section 4, CEDS and Nodes Ratio definitions"},{"comment":"The implementation description mentions K-means clustering for CLIMB but does not state the random seed or number of restarts; specifying these details, or noting that default settings were used, would improve reproducibility of the clustering step.","section":"Section 5.2, implementation details"},{"comment":"The cross-dataset validation of the evaluator LLMs uses only taxonomies generated by GPT-5.2. Since RQ1 also reports results for other generator LLMs and for CLIMB, the validation coverage is incomplete; if feasible, extending the human-evaluator comparison to at least one CLIMB taxonomy and one weaker generator per dataset would strengthen the claim that the LLM-as-judge validation generalizes across methods.","section":"Section 6.2, cross-dataset validation"},{"comment":"The column header 'Human' in Tables 9 and 10 is misleading because the annotations are performed by LLMs on human-defined taxonomies. Rename these columns to something like 'LLM-annotated Human-defined' or 'Human taxonomy (LLM annotations)' to avoid implying that human annotators produced the agreement values.","section":"Section 9.1, Tables 9 and 10"},{"comment":"The text contains a garbled icon sequence 'hand-p⌢int-right' before several implication paragraphs; this appears to be a rendering artifact and should be removed or replaced with proper markup.","section":"Section 11.3"},{"comment":"The study selection process is called a systematic literature review, but it searches only two digital libraries and five venues; consider describing it as a targeted or structured search to avoid overclaiming generality, and acknowledge in the external-validity discussion the possibility of missing relevant taxonomies published in other venues or libraries.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid and unusually careful experience report: the dataset curation, the human validation of LLM-as-judge for RQ1, the self-bias check, and the variability analysis are all above the usual bar for this type of study. The main risk is the RQ3 reliability analysis, which is described in the abstract and conclusion as evidence of practical usability but is based entirely on LLM annotators with no human baseline. I would ask the authors to either run a small human-annotation validation (even on a subset of taxonomies) or substantially soften the 'broadly usable' claim. The structural-constraint issue (instructed but not enforced) is also worth pushing on, since the over-generation claim in the abstract depends on it. The paper fits the venue well and should be publishable after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about arXiv:2608.01592. First, the core empirical trade-off is real and worth reporting: TnT-LLM produces taxonomies whose quality scores match human-defined ones, while CLIMB is an order of magnitude cheaper and faster but scores lower on orthogonality when the task requires inferring latent concepts from code or logs. Second, the reliability analysis (RQ3) is weaker than the rest: it uses only LLM annotators, with no human annotation baseline, and the paper's own threats section concedes this. That part of the paper should not be taken as evidence that generated taxonomies are 'broadly usable' by people.\n\nWhat the paper does well: it puts together a useful eight-taxonomy benchmark from recent SE papers, covering mixed natural-language and formal-language artifacts, and evaluates two non-SE methods (TnT-LLM, CLIMB) with five generator LLMs. RQ1 quality scoring is validated against two human raters on all datasets, with Human–LLM agreement close to Human–Human. RQ2 alignment uses externally defined human taxonomies as ground truth, and the self-bias check comes back clean. RQ4 cost/runtime numbers are concrete and consistent with the methods' designs. For SE researchers deciding which method to use, the guidance – pick TnT-LLM for inference-heavy artifacts, CLIMB for surface-cue datasets when budget matters – is actionable and supported by the data.\n\nThe soft spots are real but addressable. RQ3's agreement and label utilization figures are inter-LLM results. The 'Human' rows in Tables 9–10 are LLMs labeling human-defined taxonomies, not human labelers. Because LLMs share pretraining and instruction-following conventions, their agreement can differ from human agreement in either direction. The paper acknowledges this in Section 12, but the phrase 'broadly usable' in the RQ3 answer overstates what the evidence supports. Also, the instructed structural matching is imperfectly followed – Nodes Ratio reaches 3.341 for TnT-LLM on Quantum Classical Issue – so the alignment metrics partly reflect how well the method obeys constraints, not just semantic recovery. That's a minor issue since the paper reports it transparently. The sample of eight taxonomies, six with only two layers, limits external validity; the authors note this.\n\nBottom line: this is a solid experience paper with a clear, reproducible methodology. The RQ1/RQ2/RQ4 findings stand on their own. RQ3 needs either a human-annotation baseline on a subset or a careful downgrade of the usability claim. I'd send it to peer review with that expectation. I'd cite it if I worked on LLM-based taxonomy generation in SE.","headline":"A solid empirical comparison of TnT-LLM and CLIMB on SE taxonomies, with a trustworthy RQ1/RQ2/RQ4 core and an RQ3 reliability analysis that rests on unvalidated LLM annotators.","tokens_in":22204,"tokens_out":3161,"would_cite":true,"duration_ms":25515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated taxonomy generation can match human quality in SE—when you pay for the slower method.","keywords":["taxonomy generation","large language models","software engineering artifacts","empirical evaluation","LLM-as-a-judge","inter-annotator agreement","TnT-LLM","CLIMB"],"falsifier":"Take any one of the eight datasets, recruit two human annotators to assign leaf categories from a generated taxonomy to a sample of the artifacts, and compare their agreement with the paper's LLM-panel agreement. If human-human alpha is substantially lower than the reported LLM-panel alpha (for example, below 0.4 where the paper reports 0.5–0.6), the reliability claim fails; if comparable, it is confirmed. A cheaper falsifier is to re-run RQ1 without the structural constraints that force generated taxonomies to match human layer counts, and check whether TnT-LLM's quality parity persists.","tokens_in":21225,"feed_emoji":"🤖","tokens_out":8171,"duration_ms":60055,"temperature":0.7,"pith_summary":"The paper sets out to determine whether automated taxonomy generation—building a hierarchy of categories from a collection of artifacts—can succeed on software engineering materials, which mix natural language and code and often demand inference beyond surface text. To answer this, it builds a multi-perspective evaluation framework that scores taxonomies on quality, alignment with human-defined taxonomies, reliability under independent annotation, and efficiency, then applies it to two state-of-the-art methods (TnT-LLM and CLIMB) across eight human-defined taxonomies from seven SE studies. The central claim is a trade-off: TnT-LLM produces taxonomies of quality comparable to human-built ones, while CLIMB is 15–40× faster and 8–49× cheaper but loses quality when latent technical concepts must be inferred, especially on orthogonality. A sympathetic reader would take this as evidence that automated taxonomy generation is practical in SE, with method choice depending on artifact type, taxonomy depth, and budget.","feed_headline":"Cheap LLM taxonomies lag on quality; costlier ones match humans","feed_subtitle":"Four-way evaluation reveals a cost-quality trade-off that decides which LLM taxonomy tool to use.","key_machinery":"The load-bearing machinery is the multi-perspective evaluation framework itself, together with the two competing methods. TnT-LLM is an end-to-end LLM pipeline that summarizes each input instance, then iteratively generates, updates, and reviews categories to build a hierarchy; CLIMB first embeds and clusters instances into coherent groups, then uses LLM agents to name leaf categories and synthesize higher levels. The framework scores each generated taxonomy on four axes: quality (12 criteria in clarity, hierarchical coherence, orthogonality, completeness, rated by LLM judges validated against human raters), alignment with human-defined taxonomies (Heading Soft Recall, Catalogue Edit Distance Similarity, and Nodes Ratio), reliability (Krippendorff's alpha and label utilization across a panel of LLM annotators), and efficiency (API cost and wall-clock runtime). The metrics are what carry the argument: they convert the vague idea of 'good taxonomy' into four measurable quantities that can disagree, which is exactly how the trade-off between the two methods becomes visible.","core_discovery":"On the paper's own terms, the discovery is that the two fully automated, seed-free methods transfer to SE with opposite strengths. TnT-LLM, which summarizes each artifact and iteratively generates, updates, and reviews categories, achieves quality scores within +0.018 of human-defined taxonomies on average and recovers more human-like categories (HSR 0.874–0.898), but it over-generates on deep structures—in the Quantum Classical Issue dataset it produced on average 307 nodes versus 92 human nodes—and costs 8–49× more than CLIMB. CLIMB, which embeds and clusters artifacts and then has LLM agents name categories, is structurally closer to human taxonomies (Nodes ratios near 1, higher CEDS in several datasets) but loses quality on orthogonality criteria when latent technical concepts must be inferred. Reliability measured as LLM annotator agreement ranges from 0.489 to 0.647 across generators, with label utilization 0.859–0.981, and agreement correlates with non-overlap and structural alignment. The paper concludes that both methods are usable in practice if researchers first probe cost and taxonomy complexity on a subset of their data.","pith_inferences":["If the structural constraints (matching human layer count and leaf count) were removed, TnT-LLM's quality advantage might shrink, since part of its parity may come from being told the target size; this is testable by re-running RQ1 with free structure.","Replacing the LLM annotator panel with human annotators on a subsample is the obvious validity check for RQ3; until that comparison exists, the claim that generated taxonomies are 'broadly usable' rests on an unvalidated proxy.","The significant correlation between CEDS and agreement suggests a practical repair loop: adjust a generated taxonomy until its tree-edit distance to a human reference drops, and annotation reliability should rise accordingly.","The cost estimates assume no prompt caching and treat all requests as cache misses, so real deployment costs with caching could be lower than reported, narrowing the gap between TnT-LLM and CLIMB in practice."],"forward_implications":["SE researchers can use TnT-LLM to obtain taxonomies whose quality matches human-built ones, at the price of higher API cost and runtime and with a tendency to over-generate on deep hierarchies.","CLIMB is the right choice when cost and runtime dominate and the artifacts are topic-oriented, where surface-level similarity aligns with category membership; on such datasets it approaches human quality.","Taxonomy reliability depends on category clarity, non-overlap, and structural alignment (CEDS ρ=0.461), so annotator agreement can serve as a feedback signal to identify ambiguous or overlapping categories.","For multi-layer taxonomies, annotator agreement decreases at deeper layers, so human post-processing effort should concentrate on lower-level subcategories.","Before adopting either method at scale, researchers should run a small subset of the target data to estimate taxonomy complexity and cost, as the paper recommends."],"supporting_citations":[{"why":"Supplies the TnT-LLM method under evaluation, with its iterative summarization, generation, and review pipeline.","marker":"[43]"},{"why":"Supplies the CLIMB method under evaluation, with its embedding-clustering and multi-agent bottom-up pipeline.","marker":"[24]"},{"why":"Supplies two datasets (LLM Code Error Semantic and Syntactic) and the 328-person-hour manual baseline for efficiency comparison.","marker":"[45]"},{"why":"Supplies the Dockerfile flakiness dataset and human taxonomy, the key case where latent inference beyond surface cues is required.","marker":"[36]"},{"why":"Provides the precedent and design for using LLMs as annotators in the reliability measurement of RQ3.","marker":"[3]"},{"why":"Supplies the 12-criteria quality rubric and the LLM-as-evaluator procedure used in RQ1.","marker":"[52]"},{"why":"Defines the Heading Soft Recall metric used to measure alignment with human-defined taxonomies.","marker":"[12]"},{"why":"Defines the Catalogue Edit Distance Similarity metric used to measure structural and semantic alignment.","marker":"[54]"}],"fun_headline_variants":["Costlier LLM taxonomies match humans; cheaper ones lag","SE taxonomy generation: quality and cost trade-off","TnT-LLM matches human taxonomies but over-generates","CLIMB is cheap and fast but weak on technical inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a panel of LLM annotators agrees with human annotators closely enough that the measured reliability numbers (Krippendorff's alpha 0.489–0.647, label utilization 0.859–0.981) tell us how well people could apply these taxonomies—and the paper never validates LLM annotation against human annotation on the same taxonomies.","fun_headline_variants_meta":{"raw":{"variants":["Costlier LLM taxonomies match humans; cheaper ones lag","SE taxonomy generation: quality and cost trade-off","TnT-LLM matches human taxonomies but over-generates","CLIMB is cheap and fast but weak on technical inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1975,"prompt_tokens":1118,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":734,"completion_tokens_details":{"reasoning_tokens":786}},"tokens_in":734,"tokens_out":857,"duration_ms":7696,"temperature":1.0,"reasoning_tokens":786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:05:44.892956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any one of the eight datasets, recruit two human annotators to assign leaf categories from a generated taxonomy to a sample of the artifacts, and compare their agreement with the paper's LLM-panel agreement. If human-human alpha is substantially lower than the reported LLM-panel alpha (for example, below 0.4 where the paper reports 0.5–0.6), the reliability claim fails; if comparable, it is confirmed. A cheaper falsifier is to re-run RQ1 without the structural constraints that force generated taxonomies to match human layer counts, and check whether TnT-LLM's quality parity persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 12-criteria quality rubric and the LLM-as-evaluator procedure used in RQ1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Heading Soft Recall metric used to measure alignment with human-defined taxonomies."}],"review_version":2}