{"id":"a1469ee4-062b-494e-a161-1e6ec2cb6071","arxiv_id":"2411.17045","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using GPT-4 to cluster test reports by bug type and then round-robin selecting across clusters yields faster discovery of distinct bugs, a 12.77% average APFD gain over DeepPrior.","lead":"LLMPrior is a new method that uses a large language model to sort crowdsourced test reports into groups by the type of bug they describe, then mixes reports from those groups to make a review order that finds different bugs quickly. The paper reports that this order beats the previous best method by about 13% in a benchmark of 1,417 real reports.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 12.77% APFD advantage may be inflated by a zero-based indexing convention, making the comparison with DeepPrior potentially unfair.","rationale":"The paper proposes a sensible cluster-then-round-robin pipeline and provides a meaningful ablation (Section 4.4) showing that the cluster-based strategy improves over direct prompting, which is credible. The token-efficiency analysis (Section 4.5) is also useful. However, the central quantitative claim—a 12.77% average APFD improvement over DeepPrior—is compromised by an apparent inconsistency in the APFD computation. The Ideal baseline values in Table 3 are incompatible with the standard formula as written: they are exactly 1/n higher, consistent with zero-based indexing of first-occurrence positions while retaining the +1/(2n) correction term. Because DeepPrior values are not re-run but 'derived from the results of DeepPrior [6]', the comparison likely mixes two different APFD conventions, inflating LLMPrior's advantage. This is more load-bearing than the prompt-overfitting concern raised in the reader's verdict, because it affects the validity of the headline number even before considering generalization to new apps. A simple recomputation can settle it. If the correction removes significance, the paper's central contribution is unsupported; if the advantage persists, the approach remains promising.","tokens_in":22202,"tokens_out":11097,"duration_ms":94058,"concrete_test":"Recompute all APFD values in Tables 2-4 using standard 1-based positions (T_fi = 1..n), then recalculate the L-D column against DeepPrior's published values. As a spot check, App A17 should drop from 1.000 to 0.750 if the first two reports reveal the two bugs. If the corrected average improvement is no longer statistically significant (Wilcoxon p < 0.05) or falls materially below the originally claimed 12.77%, the headline claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's APFD formula (Section 4.1.4) is written in the standard form, but the reported Ideal baseline values can only be reproduced if T_fi is the zero-based index of the first report revealing bug i, while still adding the +1/(2n) term. For example, App A17 has n=4 and M=2; an ideal ordering with the two distinct bugs in the first two positions gives standard APFD 1 - (1+2)/(4*2) + 1/8 = 0.75, yet Table 3 lists Ideal = 1.000. App A4 (n=9, M=8) would have standard Ideal = 0.556 but is listed as 0.667. These values match zero-based summation (0+1+...+M-1) plus the +1/(2n) term, which inflates every APFD by exactly 1/n. Since the DeepPrior baseline is 'derived from the results of DeepPrior [6]' rather than re-run with the same convention, a systematic +1/n inflation on LLMPrior's side biases the headline 12.77% improvement upward. Correcting LLMPrior's average APFD by subtracting the per-app 1/n (average roughly 0.057) reduces the mean advantage from 12.77% to roughly 5-6%, assuming DeepPrior's published values are standard 1-based. The paper provides no evidence that the baseline numbers were recomputed under its own convention, so the central quantitative claim rests on an unverified comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMPrior, a crowdsourced test-report prioritization approach that uses an LLM (GPT-4-Turbo) to cluster reports by the bug types revealed in their textual descriptions, represents the result as a hierarchical cluster tree, and then applies a recurrent round-robin selection algorithm to produce a prioritized sequence. The approach is evaluated on 1,417 MoocTest reports from 20 mobile apps, with APFD as the main effectiveness metric and a new TPR metric for token efficiency. Comparisons are made against BERT/XLNet-based analogues, Ideal and Random baselines, the state-of-the-art DeepPrior approach, and two LLM-based ablations. The paper claims an average APFD improvement of 12.77% over DeepPrior and reports statistical significance via Wilcoxon signed-rank tests.","tokens_in":22496,"tokens_out":7601,"duration_ms":69871,"significance":"If the empirical claims were fully supported, LLMPrior would be a useful first demonstration of LLM-based crowdsourced test-report prioritization, with a plausible mechanism: semantic clustering by bug type followed by algorithmic diversification avoids the token-limit and output-instability problems of asking an LLM to produce an end-to-end ordering. The ablation study against DirectLLMPrior and SimpleLLMPrior is a reasonable way to isolate the contribution of the cluster-based strategy and prompt engineering, and the TPR metric is a sensible addition for LLM cost. The main weakness is not the approach itself but the evidence for the headline quantitative claim: the APFD indexing convention is internally inconsistent, the DeepPrior baseline is not shown to have been re-run under the same convention, and the prompt was tuned on the same data used for evaluation. These are fixable with recomputation and additional disclosure, so I regard the contribution as potentially valid but not yet established.","major_comments":[{"comment":"The APFD values are computed under an indexing convention that contradicts the stated formula. The standard formula APFD = 1 - sum(T_fi)/(n*M) + 1/(2n) uses one-based ranks, so for App A17 (n=4, M=2) the ideal ordering of the two distinct bugs in the first two positions gives APFD = 1 - (1+2)/(4*2) + 1/8 = 0.75, yet Table 3 lists Ideal = 1.000. Similarly, App A4 (n=9, M=8) has a standard ideal APFD of 0.556, not the listed 0.667. These values are reproduced instead by taking T_fi as zero-based indices while still adding +1/(2n), which inflates every APFD by exactly 1/n relative to the one-based convention. Averaged over the 20 apps, this is roughly 0.057, reducing the reported LLMPrior mean of 0.888 to about 0.831. If DeepPrior's published values use the standard one-based convention, the headline 12.77% advantage shrinks to roughly 5-6%. The authors must state and apply a single convention for all methods, including the Ideal, Random, DeepPrior, and LLMPrior columns, and report corrected numbers.","section":"Section 4.1.4 and Table 3"},{"comment":"The DeepPrior baseline is described only as 'derived from the results of DeepPrior [6]'. No evidence is provided that these per-app APFD values were produced by a fresh run on the same 1,417 reports with the same ground-truth bug labels, or that the APFD convention in the original study matches the convention used for LLMPrior. Because the central claim of the paper is the 12.77% improvement over DeepPrior, the baseline needs to be reproduced on the current dataset (with code/model/data made available) or, at minimum, the paper must document the exact mapping from each per-app value in [6] to the current dataset and confirm that the same indexing convention was used.","section":"Section 4.3 and Table 3"},{"comment":"The prompt template was 'refined through continuous iterations' on what appears to be the same dataset later used for the reported APFD measurements. No held-out development set, validation split, or nested evaluation is described. Since the clustering instructions in the prompt directly determine which reports are spread across the prioritized sequence, prompt tuning on the evaluation data is a plausible source of in-sample inflation. The paper should report what variants were tried, what stopping criterion was used, and re-evaluate the final template on an untouched set of apps or reports, or at least quantify the sensitivity of the APFD results to the prompt design choices.","section":"Section 3.1"},{"comment":"The ground truth used for APFD is not operationally defined. The paper says the authors 'manually identify the index of each report that first reveals an unrevealed bug' and Table 1 reports per-app 'Bug' counts, but it does not state how distinct bugs were identified, whether a bug list from MoocTest was used, who performed the annotation, whether annotations were cross-checked, or whether the labels cover all 1,417 reports. Without a precise description and release of the ground-truth bug labels, the APFD values are not reproducible and the denominator M used in the metric is unverified.","section":"Section 4.1.1 and Section 4.1.4"}],"minor_comments":[{"comment":"The final average row of Table 4 is labeled inconsistently: the columns are 'L-S' and 'L-D', but the row places 4.10% under L-S and 16.12% under L-D, while the text and column headers indicate that L-S should be 16.12% and L-D should be 4.10%.","section":"Table 4"},{"comment":"Several values in Table 3 use commas as decimal separators (e.g., '39,87%' and '40,68%'), which is easy to misread as a thousands separator; the paper should use one decimal convention throughout.","section":"Table 3"},{"comment":"The text refers to 'app15' while the table uses the identifier A15; please use a single naming convention for the applications.","section":"Section 4.3 and Table 3"},{"comment":"The prompt template in Figure 3 is shown in English, while Section 6 states that the dataset is exclusively Chinese-language; please clarify whether the report texts were translated, whether the prompt mixed English instructions with Chinese report content, and whether this had any effect on clustering behavior.","section":"Figure 3 and Section 6"},{"comment":"The algorithm title contains a typo: 'GenrateSequence' should be 'GenerateSequence'.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about zero-based APFD indexing is confirmed by the Ideal-column values in Table 3, so the headline comparison should be re-examined before publication. In addition, I would ask the authors to make the ground-truth bug labels, prompt versions, and DeepPrior re-run artifacts available in a stable archival form rather than only a Google Sites page. The paper fits the scope of the journal, and the approach is worth pursuing, but the current evidence for the central quantitative claim is not yet conclusive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first LLM-based crowdsourced test report prioritization paper, and the cluster-then-round-robin design is a clever fix for LLM token limits. The evaluation is broad and the ablations are informative. But the headline 12.77% improvement over DeepPrior is not trustworthy as reported: the APFD numbers appear to use zero-based indexing while the formula is standard one-based, the DeepPrior baseline is borrowed from the previous paper rather than recomputed, and the prompt was iteratively tuned on the same 20 apps. These issues are fixable, so it deserves a serious referee, but the claim needs to be re-derived.\n\nWhat's genuinely new: applying an LLM to cluster reports by bug type, then using the hierarchy to interleave reports. The hierarchical tree with visits/active flags is clearly specified, and the algorithm makes sense. The token-efficiency analysis (TPR) is a nice addition, and the ablation against direct LLM prioritization shows why the cluster-then-round-robin matters. The paper also gives a fair treatment of threats to validity, including the Chinese-only dataset and the closed-source model.\n\nThe soft spots, in order of weight. First, the APFD inconsistency. The formula as printed is the standard one-based version, but the Ideal values in Table 3 (e.g., A17: n=4, m=2, Ideal=1.000) only reproduce if T_fi is the zero-based index. That adds 1/n to every LLMPrior APFD. The DeepPrior numbers are 'derived from the results of DeepPrior' and almost certainly use the standard convention. Subtracting the average 1/n (~0.057) cuts the mean advantage from 12.77% to somewhere in the 5-6% range. Second, the ground-truth bug labels are not documented: who labeled which reports map to which of the 141 bugs, and with what criteria? APFD is only as good as this mapping. Third, the prompt was refined 'through continuous iterations' on the exact apps used in the evaluation, which makes the result partly in-sample; the number of iterations and the relationship to validation isn't reported. Fourth, no code or data is available at the listed site, so replication is currently impossible.\n\nNone of this is fatal to the core idea. Even after the APFD correction, LLMPrior likely beats random and simple NLP baselines. The comparison against DeepPrior may still favor LLMPrior, but the margin is probably not as large as claimed. The right move is to send it to peer review with a request for major revision: rerun everything with one consistent APFD convention, release the data and the baseline recomputations, and either use a held-out set for prompt tuning or report the tuning procedure transparently.","headline":"First LLM-based test report prioritization with a clever token-limit workaround, but the reported 12.77% edge over DeepPrior is inflated by an APFD indexing mismatch and unverified baselines.","tokens_in":23017,"tokens_out":6301,"would_cite":false,"duration_ms":49627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM clustering beats prior best at ordering bug reports by 12.77%","keywords":["crowdsourced testing","test report prioritization","large language model","APFD","cluster-based prioritization","prompt engineering","mobile app testing","hierarchical cluster tree"],"falsifier":"Take an independent, held-out corpus of crowdsourced test reports with ground-truth bug labels (different apps, different language, different platform), apply LLMPrior unchanged, and compare APFD with DeepPrior; if the average APFD no longer beats DeepPrior, or if manual inspection shows the LLM's cluster assignments rarely match the true bug types, the central claim is refuted.","tokens_in":22015,"feed_emoji":"🐞","tokens_out":5645,"duration_ms":49342,"temperature":0.7,"pith_summary":"This paper tries to establish that the bottleneck in crowdsourced test report prioritization is semantic understanding, and that a large language model can supply it more reliably than the feature-embedding methods used before. Rather than asking the model to output a review order, LLMPrior asks it only to cluster reports by the type of bug they reveal, and then uses a small algorithm to interleave reports from different clusters. On 1,417 reports from 20 mobile apps, this raises the average APFD from 0.787 (DeepPrior) to 0.888, a 12.77% improvement, while remaining within about 5% of the theoretical ideal order. The design also avoids a practical failure mode: direct LLM prioritization truncates or omits reports on apps with over 100 reports, whereas the cluster-based approach finishes every app. A sympathetic reading is that LLM semantic clustering plus algorithmic ordering is the useful division of labor, not the LLM acting as an end-to-end ranker.","feed_headline":"LLM clustering beats prior best at ordering bug reports by 12.77%","feed_subtitle":"Spreading bug types evenly helps reviewers find distinct faults sooner, even on apps with hundreds of reports.","key_machinery":"The central object is the hierarchical cluster tree plus the round-robin selection rule. The LLM is prompted with zero-shot chain-of-thought instructions, fine-grained clustering guidance, and an example to output a tree whose leaves are reports and whose internal nodes are bug types at increasing granularity. Each node carries 'active' and 'visits' flags. SelectReport recursively descends into the alive child with the fewest visits, marks the chosen leaf inactive, and increments visits along the path; UpdateStatus recomputes aliveness bottom-up. The effect is an even interleaving of reports from different bug clusters without requiring the LLM to produce a sequence, which sidesteps token-limit truncation and severity-biased reordering.","core_discovery":"The paper's central claim is that crowdsourced test report prioritization is better solved by delegating only semantic grouping to an LLM and delegating ordering to an algorithm. Given the textual descriptions of all reports for one app, the LLM assigns reports to bug-type clusters organized as a tree; a recurrent selection algorithm then walks the tree, always choosing from the least-visited active cluster, so reports from the same cluster are spread evenly through the sequence. The measured consequence is an average APFD of 0.888 across 20 apps, versus 0.787 for DeepPrior and 0.631 for random order, with improvements statistically significant under Wilcoxon signed-rank tests. The authors therefore conclude that LLMs understand report semantics better than Word2Vec-style features, and that the cluster-then-select design makes LLM output stable and complete where direct LLM prioritization fails.","pith_inferences":["On datasets with lower duplication rates, the advantage over random ordering should shrink, because random interleaving already separates distinct bugs; the method's edge is likely tied to the high-redundancy regime the paper targets.","Since the pipeline treats clustering as the only LLM-dependent step, swapping in cheaper or open models with comparable bug-type understanding could preserve most of the APFD gain while cutting the per-report cost far below the paper's estimate.","The same two-stage pattern (semantic grouping by an LLM, then deterministic round-robin over the group tree) transfers naturally to other triage tasks, such as issue-tracker deduplication or user-feedback prioritization, whenever ground-truth fault labels are available for evaluation.","A direct test would be to re-run the experiment on an English-language platform; the paper's dataset is exclusively Chinese, and the authors argue generalizability from the LLM's multilingual ability rather than demonstrating it."],"forward_implications":["Reviewers using LLMPrior should encounter reports of distinct bugs earlier in the inspection process, since the measured APFD gain is largest on apps with heavy duplication where the prior best approach lags furthest behind.","The cluster-then-select design removes the LLM token-limit failure: LLMPrior produces complete sequences for all 20 apps, while both direct-prioritization baselines returned incomplete results on apps with more than 100 reports.","Prompt engineering is load-bearing, not cosmetic: removing it (SimpleLLMPrior) costs 16.12% average APFD, and skipping the tree-structured clustering (DirectLLMPrior) costs 4.10%.","The approach is token-efficient relative to direct LLM ranking, saving on average 53.3 tokens per report, with the savings statistically and practically significant against DirectLLMPrior.","LLMPrior never matches the theoretical ideal on most apps, so the remaining gap identifies headroom for better clustering or better interleaving rather than evidence that the problem is solved."],"supporting_citations":[{"why":"Supplies DeepPrior, the state-of-the-art baseline whose average APFD LLMPrior exceeds by 12.77%.","marker":"[6]"},{"why":"Introduces the crowdsourced test report prioritization problem and one of the earlier prioritization strategies.","marker":"[8]"},{"why":"Defines the APFD metric used to measure how quickly prioritized reports reveal distinct bugs.","marker":"[39]"},{"why":"Provides the 82% report-redundancy statistic that motivates the need for prioritization.","marker":"[7]"},{"why":"Supplies the chain-of-thought prompting insight behind the step-by-step instruction in the clustering prompt.","marker":"[27]"},{"why":"Supplies the zero-shot chain-of-thought technique the prompt template is based on.","marker":"[28]"},{"why":"Supports the instructions that push the LLM toward accurate output, which the paper credits for clustering quality.","marker":"[29]"}],"fun_headline_variants":["LLM groups bugs then spaces them for faster triage","Cluster-based LLM ordering finds faults in apps quicker","AI prioritizes bug reports better than previous methods","LLM clusters test reports to speed up crowd testing review","Smart bug report ordering with large language models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that GPT-4-Turbo's bug-type clustering of these particular Chinese-language reports is accurate enough that the round-robin order actually separates distinct bugs; if the model misclusters, or if the hand-tuned prompt only works on these 20 apps, the APFD advantage over DeepPrior would shrink or reverse elsewhere.","fun_headline_variants_meta":{"raw":{"variants":["LLM groups bugs then spaces them for faster triage","Cluster-based LLM ordering finds faults in apps quicker","AI prioritizes bug reports better than previous methods","LLM clusters test reports to speed up crowd testing review","Smart bug report ordering with large language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1352,"prompt_tokens":1004,"completion_tokens":348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":620,"tokens_out":348,"duration_ms":4413,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:34:36.479576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an independent, held-out corpus of crowdsourced test reports with ground-truth bug labels (different apps, different language, different platform), apply LLMPrior unchanged, and compare APFD with DeepPrior; if the average APFD no longer beats DeepPrior, or if manual inspection shows the LLM's cluster assignments rarely match the true bug types, the central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DeepPrior, the state-of-the-art baseline whose average APFD LLMPrior exceeds by 12.77%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the crowdsourced test report prioritization problem and one of the earlier prioritization strategies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting insight behind the step-by-step instruction in the clustering prompt."},{"cited_title":"Kojima, S","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot chain-of-thought technique the prompt template is based on."}],"review_version":1}