Pith. sign in

REVIEW 4 major objections 5 minor 66 references

Redefining Crowdsourced Test Report Prioritization: An Innovative Approach with Large Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM clustering beats prior best at ordering bug reports by 12.77%

desk verdict First LLM-based test report prioritization with a clever token-limit workaround, but the reported 12.77% edge over DeepPrior is inflated by an APFD indexing mismatch and unverified baselines. read the letter →

arxiv 2411.17045 v1 pith:NLTVJK2H submitted 2024-11-26 cs.SE

Yuchen Ling , Shengcheng Yu , Chunrong Fang , Guobin Pan , Jun Wang , Jia Liu This is my paper · ORCID
classification cs.SE
keywords crowdsourcedtestingtestreportprioritizationlargelanguagemodelAPFDcluster-basedpromptengineeringmobileapphierarchicalclustertree
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the bottleneck in crowdsourced test report prioritization is semantic understanding, and that a large language model can supply it more reliably than the feature-embedding methods used before. Rather than asking the model to output a review order, LLMPrior asks it only to cluster reports by the type of bug they reveal, and then uses a small algorithm to interleave reports from different clusters. On 1,417 reports from 20 mobile apps, this raises the average APFD from 0.787 (DeepPrior) to 0.888, a 12.77% improvement, while remaining within about 5% of the theoretical ideal order. The design also avoids a practical failure mode: direct LLM prioritization truncates or omits reports on apps with over 100 reports, whereas the cluster-based approach finishes every app. A sympathetic reading is that LLM semantic clustering plus algorithmic ordering is the useful division of labor, not the LLM acting as an end-to-end ranker.

What carries the argument

The central object is the hierarchical cluster tree plus the round-robin selection rule. The LLM is prompted with zero-shot chain-of-thought instructions, fine-grained clustering guidance, and an example to output a tree whose leaves are reports and whose internal nodes are bug types at increasing granularity. Each node carries 'active' and 'visits' flags. SelectReport recursively descends into the alive child with the fewest visits, marks the chosen leaf inactive, and increments visits along the path; UpdateStatus recomputes aliveness bottom-up. The effect is an even interleaving of reports from different bug clusters without requiring the LLM to produce a sequence, which sidesteps token-limit truncation and severity-biased reordering.

What would settle it

Take an independent, held-out corpus of crowdsourced test reports with ground-truth bug labels (different apps, different language, different platform), apply LLMPrior unchanged, and compare APFD with DeepPrior; if the average APFD no longer beats DeepPrior, or if manual inspection shows the LLM's cluster assignments rarely match the true bug types, the central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that crowdsourced test report prioritization is better solved by delegating only semantic grouping to an LLM and delegating ordering to an algorithm. Given the textual descriptions of all reports for one app, the LLM assigns reports to bug-type clusters organized as a tree; a recurrent selection algorithm then walks the tree, always choosing from the least-visited active cluster, so reports from the same cluster are spread evenly through the sequence. The measured consequence is an average APFD of 0.888 across 20 apps, versus 0.787 for DeepPrior and 0.631 for random order, with improvements statistically significant under Wilcoxon signed-rank tests. The authors therefore conclude that LLMs understand report semantics better than Word2Vec-style features, and that the cluster-then-select design makes LLM output stable and complete where direct LLM prioritization fails.

Load-bearing premise

The whole result rests on the assumption that GPT-4-Turbo's bug-type clustering of these particular Chinese-language reports is accurate enough that the round-robin order actually separates distinct bugs; if the model misclusters, or if the hand-tuned prompt only works on these 20 apps, the APFD advantage over DeepPrior would shrink or reverse elsewhere.

Editorial extensions

If this is right

  • Reviewers using LLMPrior should encounter reports of distinct bugs earlier in the inspection process, since the measured APFD gain is largest on apps with heavy duplication where the prior best approach lags furthest behind.
  • The cluster-then-select design removes the LLM token-limit failure: LLMPrior produces complete sequences for all 20 apps, while both direct-prioritization baselines returned incomplete results on apps with more than 100 reports.
  • Prompt engineering is load-bearing, not cosmetic: removing it (SimpleLLMPrior) costs 16.12% average APFD, and skipping the tree-structured clustering (DirectLLMPrior) costs 4.10%.
  • The approach is token-efficient relative to direct LLM ranking, saving on average 53.3 tokens per report, with the savings statistically and practically significant against DirectLLMPrior.
  • LLMPrior never matches the theoretical ideal on most apps, so the remaining gap identifies headroom for better clustering or better interleaving rather than evidence that the problem is solved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On datasets with lower duplication rates, the advantage over random ordering should shrink, because random interleaving already separates distinct bugs; the method's edge is likely tied to the high-redundancy regime the paper targets.
  • Since the pipeline treats clustering as the only LLM-dependent step, swapping in cheaper or open models with comparable bug-type understanding could preserve most of the APFD gain while cutting the per-report cost far below the paper's estimate.
  • The same two-stage pattern (semantic grouping by an LLM, then deterministic round-robin over the group tree) transfers naturally to other triage tasks, such as issue-tracker deduplication or user-feedback prioritization, whenever ground-truth fault labels are available for evaluation.
  • A direct test would be to re-run the experiment on an English-language platform; the paper's dataset is exclusively Chinese, and the authors argue generalizability from the LLM's multilingual ability rather than demonstrating it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LLMPrior, a crowdsourced test-report prioritization approach that uses an LLM (GPT-4-Turbo) to cluster reports by the bug types revealed in their textual descriptions, represents the result as a hierarchical cluster tree, and then applies a recurrent round-robin selection algorithm to produce a prioritized sequence. The approach is evaluated on 1,417 MoocTest reports from 20 mobile apps, with APFD as the main effectiveness metric and a new TPR metric for token efficiency. Comparisons are made against BERT/XLNet-based analogues, Ideal and Random baselines, the state-of-the-art DeepPrior approach, and two LLM-based ablations. The paper claims an average APFD improvement of 12.77% over DeepPrior and reports statistical significance via Wilcoxon signed-rank tests.

Significance. If the empirical claims were fully supported, LLMPrior would be a useful first demonstration of LLM-based crowdsourced test-report prioritization, with a plausible mechanism: semantic clustering by bug type followed by algorithmic diversification avoids the token-limit and output-instability problems of asking an LLM to produce an end-to-end ordering. The ablation study against DirectLLMPrior and SimpleLLMPrior is a reasonable way to isolate the contribution of the cluster-based strategy and prompt engineering, and the TPR metric is a sensible addition for LLM cost. The main weakness is not the approach itself but the evidence for the headline quantitative claim: the APFD indexing convention is internally inconsistent, the DeepPrior baseline is not shown to have been re-run under the same convention, and the prompt was tuned on the same data used for evaluation. These are fixable with recomputation and additional disclosure, so I regard the contribution as potentially valid but not yet established.

major comments (4)
  1. [Section 4.1.4 and Table 3] The APFD values are computed under an indexing convention that contradicts the stated formula. The standard formula APFD = 1 - sum(T_fi)/(n*M) + 1/(2n) uses one-based ranks, so for App A17 (n=4, M=2) the ideal ordering of the two distinct bugs in the first two positions gives APFD = 1 - (1+2)/(4*2) + 1/8 = 0.75, yet Table 3 lists Ideal = 1.000. Similarly, App A4 (n=9, M=8) has a standard ideal APFD of 0.556, not the listed 0.667. These values are reproduced instead by taking T_fi as zero-based indices while still adding +1/(2n), which inflates every APFD by exactly 1/n relative to the one-based convention. Averaged over the 20 apps, this is roughly 0.057, reducing the reported LLMPrior mean of 0.888 to about 0.831. If DeepPrior's published values use the standard one-based convention, the headline 12.77% advantage shrinks to roughly 5-6%. The authors must state and apply a single convention for all methods, including the Ideal, Random, DeepPrior, and LLMPrior columns, and report corrected numbers.
  2. [Section 4.3 and Table 3] The DeepPrior baseline is described only as 'derived from the results of DeepPrior [6]'. No evidence is provided that these per-app APFD values were produced by a fresh run on the same 1,417 reports with the same ground-truth bug labels, or that the APFD convention in the original study matches the convention used for LLMPrior. Because the central claim of the paper is the 12.77% improvement over DeepPrior, the baseline needs to be reproduced on the current dataset (with code/model/data made available) or, at minimum, the paper must document the exact mapping from each per-app value in [6] to the current dataset and confirm that the same indexing convention was used.
  3. [Section 3.1] The prompt template was 'refined through continuous iterations' on what appears to be the same dataset later used for the reported APFD measurements. No held-out development set, validation split, or nested evaluation is described. Since the clustering instructions in the prompt directly determine which reports are spread across the prioritized sequence, prompt tuning on the evaluation data is a plausible source of in-sample inflation. The paper should report what variants were tried, what stopping criterion was used, and re-evaluate the final template on an untouched set of apps or reports, or at least quantify the sensitivity of the APFD results to the prompt design choices.
  4. [Section 4.1.1 and Section 4.1.4] The ground truth used for APFD is not operationally defined. The paper says the authors 'manually identify the index of each report that first reveals an unrevealed bug' and Table 1 reports per-app 'Bug' counts, but it does not state how distinct bugs were identified, whether a bug list from MoocTest was used, who performed the annotation, whether annotations were cross-checked, or whether the labels cover all 1,417 reports. Without a precise description and release of the ground-truth bug labels, the APFD values are not reproducible and the denominator M used in the metric is unverified.
minor comments (5)
  1. [Table 4] The final average row of Table 4 is labeled inconsistently: the columns are 'L-S' and 'L-D', but the row places 4.10% under L-S and 16.12% under L-D, while the text and column headers indicate that L-S should be 16.12% and L-D should be 4.10%.
  2. [Table 3] Several values in Table 3 use commas as decimal separators (e.g., '39,87%' and '40,68%'), which is easy to misread as a thousands separator; the paper should use one decimal convention throughout.
  3. [Section 4.3 and Table 3] The text refers to 'app15' while the table uses the identifier A15; please use a single naming convention for the applications.
  4. [Figure 3 and Section 6] The prompt template in Figure 3 is shown in English, while Section 6 states that the dataset is exclusively Chinese-language; please clarify whether the report texts were translated, whether the prompt mixed English instructions with Chinese report content, and whether this had any effect on clustering behavior.
  5. [Algorithm 1] The algorithm title contains a typo: 'GenrateSequence' should be 'GenerateSequence'.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported 12.77% advantage is partly an in-sample fit: the prompt was tuned on the same 20-app dataset later used for APFD evaluation.

  1. fitted input called prediction [Section 3.1, Report Clustering (prompt template design)]
    "To enhance the LLM’s performance in report clustering, we applied various prompt engineering techniques and refined the template through continuous iterations."

    The final prompt template is the only fitted component of LLMPrior, and it was refined by iterating on the same 1,417 reports from the same 20 apps that are later scored in Section 4. The paper does not describe any held-out tuning/validation split. Consequently, the reported APFD (0.888 average) and the headline 12.77% improvement over DeepPrior are in-sample evaluations of a tuned artifact, not out-of-sample predictions. The tuning procedure can select a prompt that overfits these specific reports, so the measured advantage is partly forced by the fitting process rather than being an independent test of the approach.

full rationale

The paper's core contribution is the cluster-based LLM prioritization pipeline, and most of that pipeline (tree construction, recurrent selection, de-duplication) is a deterministic algorithm that does not reduce to its inputs. However, the prompt template is a fitted component: Section 3.1 says it was 'refined through continuous iterations,' and the Section 4 evaluation uses the same 20 apps/1,417 reports, with no held-out tuning set. The headline 12.77% APFD improvement over DeepPrior is therefore partly an in-sample score rather than an out-of-sample prediction; this is the fitted-input-called-prediction pattern. Separately, the DeepPrior baseline is taken from the authors' own prior work (reference [6]) and may use a different APFD index convention (the reported Ideal values are consistent with zero-based indices while the formula includes +1/(2n)); that is a serious comparison-validity concern but not a circular derivation, so it is not counted in the score. The paper contains no load-bearing uniqueness theorem and no chain of self-citations that forces the result. Given the partial in-sample nature of the central effectiveness claim, the circularity score is 6.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new scientific entities or numeric free constants; its load-bearing assumptions are the reliability of LLM semantic clustering on this dataset and the correctness of the ground-truth bug labels. The most fragile input is the hand-tuned prompt, which is a non-numeric but still fitted component.

free parameters (1)
  • Prompt template design = Hand-tuned via iterative refinement
    Section 3.1 states the prompt was refined through continuous iterations on the same dataset used for evaluation, effectively fitting a hyperparameter (prompt wording) to the test set.
assumptions (4)
  • domain assumption LLM clustering of reports by bug type is accurate enough for diversity-based prioritization.
    Section 3.1 relies on GPT-4 to produce a hierarchical cluster tree reflecting distinct bugs; the prioritization algorithm merely spreads reports across these clusters.
  • domain assumption The ground-truth bug labels used to compute APFD are correct and consistently applied.
    Section 4.1.4 defines APFD using the manually identified index of each first report of a bug, but does not document labeling procedure, annotators, or inter-rater agreement.
  • domain assumption The MoocTest dataset is representative of crowdsourced mobile test reports.
    Section 4.1.1 relies on 1,417 reports from 20 apps; Section 6 acknowledges Chinese-only content and uneven report counts across apps.
  • ad hoc to paper Prompt tuning on the evaluation dataset does not invalidate the reported performance.
    The prompt was refined on the same 20 apps later used for benchmarking (Section 3.1 vs Section 4), so reported APFD may be an in-sample estimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Redefining Crowdsourced Test Report Prioritization: An Innovative Approach with Large Language Model." pith.science (2026). https://pith.science/paper/NLTVJK2H

@misc{pith2026241117045,
  author       = {Pith},
  title        = {Pith review of: Redefining Crowdsourced Test Report Prioritization: An Innovative Approach with Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLTVJK2H}},
  note         = {Machine review of arXiv:2411.17045}
}
read the original abstract

Context: Crowdsourced testing has gained popularity in software testing, especially for mobile app testing, due to its ability to bring diversity and tackle fragmentation issues. However, the openness of crowdsourced testing presents challenges, particularly in the manual review of numerous test reports, which is time-consuming and labor-intensive. Objective: The primary goal of this research is to improve the efficiency of review processes in crowdsourced testing. Traditional approaches to test report prioritization lack a deep understanding of semantic information in textual descriptions of these reports. This paper introduces LLMPrior, a novel approach for prioritizing crowdsourced test reports using large language models (LLMs). Method: LLMPrior leverages LLMs for the analysis and clustering of crowdsourced test reports based on the types of bugs revealed in their textual descriptions. This involves using prompt engineering techniques to enhance the performance of LLMs. Following the clustering, a recurrent selection algorithm is applied to prioritize the reports. Results: Empirical experiments are conducted to evaluate the effectiveness of LLMPrior. The findings indicate that LLMPrior not only surpasses current state-of-the-art approaches in terms of performance but also proves to be more feasible, efficient, and reliable. This success is attributed to the use of prompt engineering techniques and the cluster-based prioritization strategy. Conclusion: LLMPrior represents a significant advancement in crowdsourced test report prioritization. By effectively utilizing large language models and a cluster-based strategy, it addresses the challenges in traditional prioritization approaches, offering a more efficient and reliable solution for app developers dealing with crowdsourced test reports.

Figures

Figures reproduced from arXiv: 2411.17045 by the authors.

Figure 1
Figure 1. General Workflow of Crowdsourced Testing [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. LLMPrior framework You are provided with a set of test reports. Each report includes a description of the bug and the operations that trigger it. The reports are listed below: — Textual Descriptions of All Reports — Categorize all the reports into different bug types with fine grains. For example, errors in display should be classified by their unique occurrences, not just as general display errors. Note that: 1. Si… view at source ↗
Figure 3
Figure 3. Prompt Template 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example Hierarchical Cluster Tree Algorithm 1: GenrateSequence Input: Hierarchical Cluster Tree Root Node n Output: Prioritized Test Report Sequence P // Generate the prioritization sequence of all the test reports 1 initiate prioritized report sequence P ← ∅ 2 initiat…
Figure 5
Figure 5. Figure 5: LLM-based Test Report Prioritization Approaches Data Comparison Analysis [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages

  1. [6]

    S. Yu, C. Fang, Z. Cao, X. Wang, T. Li, Z. Chen, Pri- oritize crowdsourced test reports via deep screenshot un- derstanding, in: 2021 IEEE/ACM 43rd International Con- ference on Software Engineering (ICSE), 2021, pp. 946–

    work page 2021

  2. [1]

    L. Wei, Y . Liu, S.-C. Cheung, Taming android fragmenta- tion: characterizing and detecting compatibility issues for android apps, in: Proceedings of the 31st IEEE /ACM In- ternational Conference on Automated Software Engineer- ing, ASE ’16, pp. 226–237. doi:10.1145/2970276. 2970312

    work page doi:10.1145/2970276

  3. [2]

    Zhang, J

    T. Zhang, J. Gao, J. Cheng, Crowdsourced testing services for mobile apps, in: 2017 IEEE Symposium on Service- Oriented System Engineering (SOSE), 2017, pp. 75–80. doi:10.1109/SOSE.2017.28

    work page doi:10.1109/sose.2017.28 2017

  4. [3]

    S. Yu, C. Fang, Y . Feng, W. Zhao, Z. Chen, LIRAT: Lay- out and Image Recognition Driving Automated Mobile Testing of Cross-Platform, in: 2019 34th IEEE /ACM In- ternational Conference on Automated Software Engineer- ing (ASE), 2019, pp. 1066–1069. doi:10.1109/ASE. 2019.00103

    arXiv 2019

  5. [4]

    R. Gao, Y . Wang, Y . Feng, Z. Chen, W. Eric Wong, Suc- cesses, challenges, and rethinking – an industrial investi- gation on crowdsourced mobile application testing 24 (2) (2019) 537–561. doi:10.1007/s10664-018-9618-5

    work page doi:10.1007/s10664-018-9618-5 2019

  6. [7]

    J. Wang, M. Li, S. Wang, T. Menzies, Q. Wang, Images don’t lie: Duplicate crowdtesting reports detection with screenshot information, Information and Software Tech- nology 110 (2019) 139–155. doi:10.1016/j.infsof. 2019.03.003

    doi:10.1016/j.infsof 2019

  7. [8]

    Y . Feng, Z. Chen, J. A. Jones, C. Fang, B. Xu, Test report prioritization to assist crowdsourced testing, in: Proceed- ings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC /FSE 2015, 2015, pp. 225–

    work page 2015

  8. [9]

    Y . Feng, J. A. Jones, Z. Chen, C. Fang, Multi-objective test report prioritization using image understanding, in: 2016 31st IEEE/ACM International Conference on Auto- mated Software Engineering (ASE), 2016, pp. 202–213

    work page 2016

Show all 66 references
  1. [10]

    D. Liu, Y . Feng, X. Zhang, J. A. Jones, Z. Chen, Cluster- ing crowdsourced test reports of mobile applications us- ing image understanding, IEEE Transactions on Software Engineering 48 (4) (2022) 1290–1308. doi:10.1109/ TSE.2020.3017514

    2022

  2. [11]

    Y . Li, Y . Feng, R. Hao, D. Liu, C. Fang, Z. Chen, B. Xu, Classifying crowdsourced mobile test reports with im- age features: An empirical study, J. Syst. Softw. 184 (C) (2022). doi:10.1016/j.jss.2021.111121

    2022

  3. [12]

    H. Li, X. Qi, M. Li, Y . Qu, X. Ge, Identifying high-impact bug reports with imbalance distribution by instance fuzzy entropy, International Journal of Software Engineering and Knowledge Engineering 32 (09) (2022) 1389–1417. doi:10.1142/S021819402250053X

    2022 doi

  4. [13]

    M. Du, S. Yu, C. Fang, T. Li, H. Zhang, Z. Chen, Sem- Cluster: a semi-supervised clustering tool for crowd- sourced test reports with deep image understanding, in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Founda- tions of ...

    2022

  5. [14]

    T. Kim, G. Yang, Predicting Duplicate in Bug Report Us- ing Topic-Based Duplicate Learning With Fine Tuning- Based BERT Algorithm, IEEE Access 10 (2022) 129666– 129675. doi:10.1109/ACCESS.2022.3226238

    2022

  6. [15]

    A. T. Nguyen, T. N. Nguyen, Incremental Relational Topic Model for Duplicate Bug Report Detection, in: 2022 29th Asia-Pacific Software Engineering Confer- ence (APSEC), 2022, pp. 99–108. doi:10.1109/ APSEC57359.2022.00022

    2022

  7. [16]

    X. Wu, W. Shan, W. Zheng, Z. Chen, T. Ren, X. Sun, An Intelligent Duplicate Bug Report Detection Method Based on Technical Term Extraction, in: 2023 IEEE/ACM International Conference on Automation of Software Test (AST), 2023, pp. 1–12. doi:10.1109/ AST58925.2023.00005

    2023

  8. [17]

    Y . Tong, X. Zhang, Crowdsourced test report prioriti- zation considering bug severity, Information and Soft- ware Technology 139 (2021) 106668. doi:10.1016/j. infsof.2021.106668

    2021

  9. [18]

    Mikolov, I

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, V ol. 26, 2013

    2013

  10. [19]

    Y . Shen, J. Liu, Comparison of text sentiment analysis based on bert and word2vec, in: 2021 IEEE 3rd Inter- national Conference on Frontiers Technology of Infor- mation and Computer (ICFTIC), pp. 144–147. doi: 10.1109/ICFTIC54370.2021.9647258

    2021

  11. [20]

    Huang, H

    S. Huang, H. Chen, Z. Hui, Y . Liu, A survey of the use of test report in crowdsourced testing, in: 2020 IEEE 20th International Conference on Software Quality, Reliabil- ity and Security (QRS), pp. 430–441. doi:10.1109/ QRS51102.2020.00062

    2020 arXiv

  12. [21]

    Tamkin, M

    A. Tamkin, M. Brundage, J. Clark, D. Ganguli, Under- standing the capabilities, limitations, and societal im- pact of large language models. doi:10.48550/arXiv. 2102.02503

    doi arXiv

  13. [22]

    J. Wei, Y . Tay, R. Bommasani, C. Ra ffel, B. Zoph, 18 S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Met- zler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, W. Fedus, Emergent abilities of large language models (2022). doi:10.48550/arXiv.2206.07682

    2022 doi arXiv

  14. [23]

    Chang, X

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, X. Xie, A survey on evalu- ation of large language models, ACM Trans. Intell. Syst. Technol. 15 (3) (2024). doi:10.1145/3641289

    2024 doi

  15. [24]

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language mod- els for software engineering: A systematic literature re- view. doi:10.48550/arXiv.2308.10620

    doi arXiv

  16. [25]

    Zheng, K

    Z. Zheng, K. Ning, J. Chen, Y . Wang, W. Chen, L. Guo, W. Wang, Towards an understanding of large language models in software engineering tasks. doi:10.48550/ arXiv.2308.11396

    doi arXiv

  17. [26]

    Belzner, T

    L. Belzner, T. Gabor, M. Wirsing, Large language model assisted software engineering: Prospects, challenges, and a case study, in: Bridging the Gap Between AI and Re- ality, Lecture Notes in Computer Science, pp. 355–374. doi:10.1007/978-3-031-46002-9\_23

    doi

  18. [27]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Proceedings of the 36th International Conference on Neu- ral Information Processing Systems, NIPS ’22, 2024

    2024

  19. [28]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, Y . Iwasawa, Large language models are zero-shot reasoners, in: Pro- ceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, 2024

    2024

  20. [29]

    Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, J. Ba, Large Language Models Are Human- Level Prompt Engineers (2023).doi:10.48550/arXiv. 2211.01910

    2023 doi arXiv

  21. [30]

    K. Mao, L. Capra, M. Harman, Y . Jia, A survey of the use of crowdsourcing in software engineering 126 57–84. doi:10.1016/j.jss.2016.09.015

    2016 doi

  22. [31]

    S. Yu, C. Fang, Q. Zhang, Z. Cao, Y . Yun, Z. Cao, K. Mei, Z. Chen, Mobile app crowdsourced test report consistency detection via deep image-and-text fusion understanding 1–20doi:10.1109/TSE.2023.3285787

    2023

  23. [32]

    K. Mao, Y . Yang, Q. Wang, Y . Jia, M. Harman, Developer recommendation for crowdsourced software development tasks, in: 2015 IEEE Symposium on Service-Oriented System Engineering, pp. 347–356. doi:10.1109/SOSE. 2015.46

    2015 doi

  24. [33]

    J. Wang, Y . Yang, R. Krishna, T. Menzies, Q. Wang, iSENSE: Completion-aware crowdtesting management, in: 2019 IEEE /ACM 41st International Conference on Software Engineering (ICSE), pp. 912–923. doi:10. 1109/ICSE.2019.00097

    2019

  25. [34]

    J. Wang, Y . Yang, T. Menzies, Q. Wang, iSENSE2.0: Improving completion-aware crowdtesting management with duplicate tagger and sanity checker 29 (4) 24:1– 24:27. doi:10.1145/3394602

    doi

  26. [35]

    J. Wang, Y . Yang, S. Wang, C. Chen, D. Wang, Q. Wang, Context-aware personalized crowdtesting task recom- mendation 48 (8) 3131–3144. doi:10.1109/TSE. 2021.3081171

    2021

  27. [36]

    Q. Wang, Z. Chen, J. Wang, Y . Feng, Crowdsourced Test- ing Task Management, Springer Nature Singapore, 2022, pp. 91–122. doi:10.1007/978-981-16-9643-5\_7

    2022 doi

  28. [37]

    Y . Yang, X. Chen, Crowdsourced Test Report Prioritiza- tion Based on Text Classification, IEEE Access 10 (2022) 92692–92705. doi:10.1109/ACCESS.2021.3128726

    2022

  29. [38]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray...

    2020 doi arXiv

  30. [39]

    Rothermel, R

    G. Rothermel, R. Untch, C. Chu, M. Harrold, Priori- tizing test cases for regression testing, IEEE Transac- tions on Software Engineering 27 (10) (2001) 929–948. doi:10.1109/32.962562

    2001 doi

  31. [40]

    J. Wang, S. Wang, Q. Cui, Q. Wang, Local-based active classification of test report to assist crowdsourced testing, in: 2016 31st IEEE /ACM International Conference on Automated Software Engineering (ASE), 2016, pp. 190–

    2016

  32. [41]

    J. Wang, Q. Cui, Q. Wang, S. Wang, Towards E ffec- tively Test Report Classification to Assist Crowdsourced Testing, in: Proceedings of the 10th ACM /IEEE Inter- national Symposium on Empirical Software Engineering and Measurement, Ciudad Real Spain, 2016, pp. 1–10. doi:10.114...

    2016

  33. [42]

    J. Wang, Q. Cui, S. Wang, Q. Wang, Domain adapta- tion for test report classification in crowdsourced testing, in: 2017 IEEE /ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 2017, pp. 83–92. doi:10.1109/ ICSE-SEIP.2017.8

    2017

  34. [43]

    Jiang, X

    H. Jiang, X. Chen, T. He, Z. Chen, X. Li, Fuzzy Cluster- ing of Crowdsourced Test Reports for Apps, ACM Trans- actions on Internet Technology 18 (2) (2018) 18:1–18:28. doi:10.1145/3106164

    2018 doi

  35. [44]

    R. Hao, Y . Feng, J. A. Jones, Y . Li, Z. Chen, CTRAS: Crowdsourced Test Report Aggregation and Summariza- tion, in: 2019 IEEE /ACM 41st International Conference on Software Engineering (ICSE), 2019, pp. 900–911. doi:10.1109/ICSE.2019.00096

    2019

  36. [45]

    L. Cai, N. Wang, M. Chen, J. Wang, J. Wang, J. Gong, Reports aggregation of crowdsourcing test based on fea- ture fusion, in: 2021 IEEE 21st International Confer- ence on Software Quality, Reliability and Security Com- panion (QRS-C), 2021, pp. 51–59. doi:10.1109/ QRS-C55045.2...

    2021

  37. [46]

    H. Chen, S. Huang, Y . Liu, R. Luo, Y . Xie, An e ffec- 19 tive crowdsourced test report clustering model based on sentence embedding, in: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), 2021, pp. 888–899. doi:10.1109/QRS54544. 2021.00098

    2021

  38. [47]

    Runeson, M

    P. Runeson, M. Alexandersson, O. Nyholm, Detection of duplicate defect reports using natural language process- ing, in: 29th International Conference on Software Engi- neering (ICSE’07), 2007, pp. 499–510. doi:10.1109/ ICSE.2007.32

    2007

  39. [48]

    C. Sun, D. Lo, X. Wang, J. Jiang, S.-C. Khoo, A discrim- inative model approach for accurate duplicate bug report retrieval, in: Proceedings of the 32nd ACM /IEEE Inter- national Conference on Software Engineering - V olume 1, ICSE ’10, 2010, pp. 45–54. doi:10.1145/1806799. 1806811

    2010 doi

  40. [49]

    C. Sun, D. Lo, S.-C. Khoo, J. Jiang, Towards more ac- curate retrieval of duplicate bug reports, in: 2011 26th IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE 2011), 2011, pp. 253–262. doi: 10.1109/ASE.2011.6100061

    2011

  41. [50]

    Alipour, A

    A. Alipour, A. Hindle, E. Stroulia, A contextual ap- proach towards more accurate duplicate bug report de- tection, in: 2013 10th Working Conference on Mining Software Repositories (MSR), 2013, pp. 183–192. doi: 10.1109/MSR.2013.6624026

    2013

  42. [51]

    Lazar, S

    A. Lazar, S. Ritchey, B. Sharif, Improving the accuracy of duplicate bug report detection using textual similarity measures, in: Proceedings of the 11th Working Confer- ence on Mining Software Repositories, MSR 2014, 2014, pp. 308–311. doi:10.1145/2597073.2597088

    2014

  43. [52]

    Ebrahimi, A

    N. Ebrahimi, A. Trabelsi, M. S. Islam, A. Hamou-Lhadj, K. Khanmohammadi, An HMM-based approach for auto- matic detection and classification of duplicate bug reports 113 98–109. doi:10.1016/j.infsof.2019.05.007

    2019 doi

  44. [54]

    S. Kang, J. Yoon, S. Yoo, Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction, in: 2023 IEEE /ACM 45th Inter- national Conference on Software Engineering (ICSE), 2023, pp. 2312–2323, iSSN: 1558-1225. doi:10.1109/ ICSE48619.2023.00194

    2023

  45. [55]

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, Q. Wang, Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing (2023). doi:10.48550/arXiv.2305.09434

    2023 doi arXiv

  46. [56]

    Feldt, S

    R. Feldt, S. Kang, J. Yoon, S. Yoo, Towards Autonomous Testing Agents via Conversational Large Language Mod- els, in: 2023 38th IEEE /ACM International Conference on Automated Software Engineering (ASE), 2023, pp. 1688–1693. doi:10.1109/ASE56229.2023.00148

    2023

  47. [57]

    S. Yu, C. Fang, Y . Ling, C. Wu, Z. Chen, Llm for test script generation and migration: Challenges, capabili- ties, and opportunities, in: 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 206–217. doi:10.1109/QRS60937. 2023.00029

    2023

  48. [58]

    MacNeil, A

    S. MacNeil, A. Tran, D. Mogil, S. Bernstein, E. Ross, Z. Huang, Generating Diverse Code Explanations using the GPT-3 Large Language Model, in: Proceedings of the 2022 ACM Conference on International Computing Education Research - V olume 2, V ol. 2 of ICER ’22, New York, NY , ...

    2022

  49. [59]

    M. L. Siddiq, A. Samee, S. R. Azgor, M. A. Haider, S. I. Sawraz, J. C. S. Santos, Zero-shot Prompting for Code Complexity Prediction Using GitHub Copilot, in: 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE), 2023, pp. 56–59. doi:...

    2023

  50. [60]

    Mashhadi, H

    E. Mashhadi, H. Ahmadvand, H. Hemmati, Method-Level Bug Severity Prediction using Source Code Metrics and LLMs, in: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 635–646. doi:10.1109/ISSRE59848.2023.00055

    2023

  51. [62]

    G. Weng, A. Andrzejak, Automatic Bug Fixing via De- liberate Problem Solving with Large Language Models, in: 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), 2023, pp. 34–36. doi:10.1109/ISSREW60843.2023.00040

    2023

  52. [63]

    Petrovi ´c, S

    N. Petrovi ´c, S. Koni ˇcanin, S. Suljovi ´c, ChatGPT in IoT Systems: Arduino Case Studies, in: 2023 IEEE 33rd International Conference on Microelectronics (MIEL), 2023, pp. 1–4. doi:10.1109/MIEL58498.2023. 10315791

    2023

  53. [64]

    X. Ding, L. Chen, M. Emani, C. Liao, P.-H. Lin, T. Van- derbruggen, Z. Xie, A. Cerpa, W. Du, HPC-GPT: In- tegrating Large Language Model for High-Performance Computing, in: Proceedings of the SC ’23 Workshops of The International Conference on High Performance Com- puting, Net...

    2023

  54. [65]

    Koreeda, T

    Y . Koreeda, T. Morishita, O. Imaichi, Y . Sogawa, LARCH: Large Language Model-based Automatic Readme Creation with Heuristics, in: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, 2023, pp. 5066–

    2023

  55. [201]

    doi:10.1145/2970276.2970300

  56. [236]

    doi:10.1145/2786805.2786862

  57. [956]

    doi:10.1109/ICSE43902.2021.00090

    2021

  58. [5070]

    doi:10.1145/3583780.3614744. 20

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.