REVIEW 5 major objections 5 minor 36 references
Automated Bug Triaging using Instruction-Tuned Large Language Models
T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A lightweight instruction-tuned LLM, forced to choose only from a known roster, turns bug triage into ranked shortlists, catching the true developer in 75% of Mozilla test bugs despite near-zero exact-match accuracy.
desk verdict A plausible LoRA+constrained-decoding pipeline for bug triage, with one strong multi-year result on Mozilla and an unsupported recent-snapshot claim that should not be trusted as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is candidate-constrained decoding: at inference the model is forbidden from emitting any identifier outside a roster of valid assignees built from training labels and/or the official list, then outputs a comma-separated Top-10 list that is regex-validated, deduplicated, and padded. This constraint is what converts the LLM's text understanding into a usable shortlist and prevents hallucinated identifiers. The other component is the training setup: a conversational JSONL prompt (system/user/assistant) with a low-rank adapter on an 8B-parameter instruction-tuned LLM, using 4-bit quantization, so the whole pipeline stays cheap to train and run. The paper argues the com
What would settle it
Run the same pipeline on a held-out set of issues whose true assignee was not present in the training-label roster (for example, brand-new contributors), and check whether Hit@10 collapses; if it does, the closed-world roster built from training labels is doing most of the work.
Extended reading notes
Core claim
The central claim is that an instruction-tuned LLM, combined with candidate-constrained decoding, provides a practical alternative to feature-engineered classifiers and graph neural networks for bug triage. The paper demonstrates this on EclipseJDT and Mozilla by following the same multi-year temporal windows and developer filtering as the graph-based baseline it compares against. With a low-rank-adapted 8B-parameter model and 4-bit quantization, it attains Top-1/Hit@10 of 0.156/0.475 on EclipseJDT and 0.013/0.753 on Mozilla; against the graph-based baseline it lags on EclipseJDT across all K (0.475 vs 0.675 at K=10) but surpasses it on Mozilla shortlists (0.753 vs 0.522 at K=10). On a Jan-J
Load-bearing premise
The central evaluation assumes every retained test issue has its gold assignee inside the candidate roster built from training labels or an official list, with no test-label leakage; if the test set were to include issues whose true assignee is a new or low-activity developer, or if the recorded assignee is not the actual fixer, the reported Hit@10 numbers would be conditional and could be inflated.
Editorial extensions
If this is right
- Maintainers can use the Top-10 shortlist as a recommendation panel: even a 0.75 Hit@10 on Mozilla means the right developer is usually on the menu, cutting search time even when rank 1 is wrong.
- The Jan-Jun 2025 snapshot shows the method is most useful on current rosters; periodic roster refresh and incremental low-rank fine-tuning can keep it aligned with team turnover.
- Because the approach needs no handcrafted features or graph construction, it can be adapted to a new project by formatting issue-tracker data and fine-tuning an adapter.
- The gap between Top-1 and Hit@10 motivates hybrid designs, such as graph-derived priors or developer profile embeddings to re-rank the LLM shortlist, and pairwise or listwise losses to improve intra-list ordering.
Reading between the lines
- Beyond the paper: the large jump from multi-year to six-month numbers suggests most of the difficulty is roster size and long-tail sparsity, not the model's text understanding; if so, shrinking or re-ranking the candidate set should help more than scaling the model.
- Beyond the paper: the Mozilla pattern (Top-1 0.013, Hit@10 0.753) implies a two-stage workflow—LLM proposes 10, a lightweight re-ranker or human picks—could outperform both pure LLM and pure graph methods; this is a testable design, not something the paper evaluates.
- Beyond the paper: because the candidate roster is built from training labels, an open-world deployment with new contributors would likely see lower Hit@K; a clean test is to evaluate on issues whose assignee never appeared in training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a bug-triaging framework built on an instruction-tuned 8B LLM (DeepSeek-R1-Distill-Llama-8B) with LoRA adapters and 4-bit NF4 quantization. Fine-tuned on EclipseJDT and Mozilla Bugzilla data converted to JSONL prompts, the model is evaluated with candidate-constrained decoding that restricts outputs to a known developer roster and produces ranked Top-K recommendations. Using what the authors describe as the NCGBT multi-year temporal windows and filtering protocol, the paper reports Top-1/Hit@10 of 0.156/0.475 on EclipseJDT and 0.013/0.753 on Mozilla, and on a Jan–Jun 2025 snapshot it reports 0.83/0.99 and 0.615/0.72 (n=200 per project). The paper positions the framework as a lightweight, practical alternative to feature-engineered and graph-based methods, emphasizing shortlist quality rather than exact assignment.
Significance. If the results are substantiated, the paper makes a useful empirical contribution: it shows that a relatively small instruction-tuned LLM with constrained decoding can produce ranked shortlists for bug triage without handcrafted features or graph construction, and it provides a direct comparison with a state-of-the-art graph baseline on the same datasets. The authors are honest about the mixed multi-year results and about the difference between Top-1 and Hit@K. The candidate-constrained decoding idea is sensible, and restricting the label space to training-roster developers is a standard way to avoid test-label leakage. However, several load-bearing evaluation details are currently undocumented or internally inconsistent, and the recent-snapshot claim at the center of the abstract is not yet supported.
major comments (5)
- [IV-F, Table VI] The Jan–Jun 2025 snapshot is undocumented. No source query, filtering rules, sampling method, or roster coverage rate is given. Since candidate-constrained decoding can only output developers from the training roster (Section III-C.3, IV-A), the reported Top-1/Hit@10 of 0.83/0.99 and 0.615/0.72 are interpretable only if every one of the 200 sampled issues has a gold assignee in the pre-2025 roster. The paper does not report the coverage rate or how the 200 issues were selected. The abstract's claim that 'accuracy rises sharply' on recent snapshots is therefore unsupported.
- [IV-D, Tables II and III] The test-set denominators are inconsistent. Table II reports ~2,000 (EclipseJDT) and ~12,000 (Mozilla) test issues, and Section IV-D repeats those numbers; the same paragraph then says n=1,612 and n≈11,050, which are the denominators implied by Table III counts (765/0.475 and 8,318/0.753). All ratios change by up to 20% depending on which denominator is used. Please align Table II, Section IV-D, and Table III, and state whether the <10-bug developer filter is applied before or after the split.
- [Table III, Mozilla row] Hit@1=0.013 (146) but Hit@2=0.743 (8,213), and Hit@3–10 increase by only ~0.01. This abrupt jump is unexplained and is central to the 'strong shortlist' claim. If true, it means the true assignee is the second-ranked candidate in ~74% of test issues; if it is an artifact of candidate-list ordering, deduplication, or a different prompt for Top-1 vs Top-K, the claimed Hit@K advantage is not valid. Please report per-rank statistics, examples, and the exact decoding procedure used for each metric.
- [III-E] 'Maximum steps: 500 (approximately one full epoch over the training set)' is inconsistent with the stated effective batch size of 8 and training sizes of ~16,000 and ~96,000. One epoch would require ~2,000 and ~12,000 steps, respectively. Please correct the reported training setup or clarify what subset is used; otherwise the 'fully reproducible' claim (Section IV-G) cannot be assessed.
- [V-D] The paper acknowledges that bootstrap confidence intervals, Wilcoxon tests, and multi-seed runs are missing. Because the Mozilla Hit@2 jump and the snapshot results are single-seed point estimates, no inferential statement about practical superiority can be made. Please add at least bootstrap CIs over test issues and, ideally, 3–5 seeds for the main tables.
minor comments (5)
- [III-A, Tables I-II] Table I lists 16,106/110,467 bugs after filtering, while Table II reports ~20,000/~120,000 totals. These should be reconciled so the split denominators are clear.
- [IV-G and footnote 1] The paper promises release of code, prompts, and per-issue predictions but provides no repository URL; with a 'fully reproducible' claim, a link or appendix is needed.
- [Table V] The 'Improve' row reports negative values for what appears to be a difference. Label it 'Difference' or explain the sign convention.
- [Figure 5] The caption is incomplete ('Radar chart comparing' ends abruptly).
- [III-A, example JSONL] The example JSONL record contains literal \n and quotes; format it as a code block for readability.
Circularity Check
No significant circularity: the evaluation is grounded in external Bugzilla labels and standard candidate-constrained decoding, so no claimed prediction reduces to its own construction.
full rationale
The paper's derivation chain is empirical rather than definitional. The model is fine-tuned on issue text with the gold assignee as the label, and at inference it is evaluated by exact-match Top-1 and Hit@K against held-out Bugzilla assignments. These metrics (e.g., Hit@K = N_hit / N_pred) are standard measurements, not quantities that are fitted and then re-reported as predictions. Candidate-constrained decoding restricts the output space to a roster built from training labels or an official list, but it does not guarantee that the gold assignee of a test issue is in that roster; if the assignee is absent, the model cannot score a hit. Thus the reported Hit@K is conditional on roster coverage and label-space filtering, which is an evaluation-design consideration rather than circularity. The Jan–Jun 2025 snapshot lacks collection and coverage details, so its high numbers are not fully supported as evidence of real-world utility, but this is an empirical-evidence gap, not a circular step: no result in that section is shown to be equal to an input by construction. The paper's self-citations appear in peripheral future-work and ethics contexts and are not load-bearing: they do not supply a uniqueness theorem, a fitted parameter, or an ansatz that the core claim depends on. The central comparison against NCGBT is an external benchmark, and the paper explicitly acknowledges mixed performance rather than claiming a forced superiority. Therefore no circular step meeting the quoted-evidence standard is present; the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- minimum developer bug count threshold =
10
- LoRA rank and scaling =
r=16, alpha=16, dropout=0
- training hyperparameters =
lr=2e-4, weight decay=0.01, warmup=0.03, max_steps=500, seq_len=2048, seed=3407
- candidate list size K =
10
- six-month snapshot test size =
200 per project
assumptions (5)
- domain assumption Recorded Bugzilla assignee equals the correct resolver
- domain assumption Title plus body text is sufficient input for triage
- domain assumption Candidate roster from training labels is complete and leakage-free
- domain assumption Replication of the NCGBT dataset protocol is exact
- ad hoc to paper A Jan-Jun 2025 snapshot exists for both projects
Cite this review
Pith. "Pith review of Automated Bug Triaging using Instruction-Tuned Large Language Models." pith.science (2026). https://pith.science/paper/Z45IYUWG
@misc{pith2026250821156,
author = {Pith},
title = {Pith review of: Automated Bug Triaging using Instruction-Tuned Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z45IYUWG}},
note = {Machine review of arXiv:2508.21156}
}
read the original abstract
Bug triaging, the task of assigning new issues to developers, is often slow and inconsistent in large projects. We present a lightweight framework that instruction-tuned large language model (LLM) with LoRA adapters and uses candidate-constrained decoding to ensure valid assignments. Tested on EclipseJDT and Mozilla datasets, the model achieves strong shortlist quality (Hit at 10 up to 0.753) despite modest exact Top-1 accuracy. On recent snapshots, accuracy rises sharply, showing the framework's potential for real-world, human-in-the-loop triaging. Our results suggest that instruction-tuned LLMs offer a practical alternative to costly feature engineering and graph-based methods.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
A spatial–temporal graph neural network framework for automated software bug triaging,
H. Wu, Y . Ma, Z. Xiang, C. Yang, and K. He, “A spatial–temporal graph neural network framework for automated software bug triaging,” Knowledge-Based Systems, vol. 241, p. 108308, 2022
work page 2022
-
[2]
C. Zhou, X. Xie, G. Chen, P. He, and B. Li, “Issuecourier: Multi- relational heterogeneous temporal graph neural network for open-source issue assignment,” arXiv preprint arXiv:2505.11205, 2025, https://arxiv. org/abs/2505.11205
work page Pith review arXiv 2025
-
[3]
Developer load balancing bug triage: Developed load balance,
A. Yadav, M. Baljon, S. Mishra, S. K. Singh, S. Saxena, and S. K. Sharma, “Developer load balancing bug triage: Developed load balance,” Expert Systems, vol. 41, no. 6, p. e13006, 2024
work page 2024
-
[4]
M. Samir, N. Sherief, and W. Abdelmoez, “Improving bug assignment and developer allocation in software engineering through interpretable machine learning models,” Computers, vol. 12, no. 7, p. 128, 2023, https://doi.org/10.3390/computers12070128
-
[5]
Factoring exper- tise, workload, and turnover into code review recommendation,
F. Hajari, S. Malmir, E. Mirsaeedi, and P. C. Rigby, “Factoring exper- tise, workload, and turnover into code review recommendation,” IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 884–899, 2024, https://doi.org/10.1109/TSE.2024.3366753
-
[6]
Expertise retrieval using adjusted tf-idf and keyword mapping to acm classification terms,
L. R. Aini and E. Yulianti, “Expertise retrieval using adjusted tf-idf and keyword mapping to acm classification terms,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 9, no. 3, pp. 427–435, 2025, https: //jurnal.iaii.or.id/index.php/RESTI/article/view/6397
work page 2025
-
[7]
Devrec: Multi-relationship embedded software developer recommendation,
X. Xie, X. Yang, B. Wang, and Q. He, “Devrec: Multi-relationship embedded software developer recommendation,” IEEE Transactions on Software Engineering, vol. 48, no. 11, pp. 4357–4379, 2021
work page 2021
-
[8]
Neighborhood contrastive learning-based graph neural network for bug triaging,
H. Dong, H. Ren, J. Shi, Y . Xie, and X. Hu, “Neighborhood contrastive learning-based graph neural network for bug triaging,” Science of Computer Programming, vol. 235, p. 103093, 2024
work page 2024
Show all 36 references
-
[9]
Bug triaging based on transformer models utilizing commit messages,
F. R. Arnob, R. H. Mollik, P. Goyal, and R. Bryce, “Bug triaging based on transformer models utilizing commit messages,” in International Conference on Information Technology-New Generations . Springer, 2025, pp. 354–366
2025
-
[10]
A comparative study of transformer- based neural text representation techniques on bug triaging,
A. K. Dipongkor and K. Moran, “A comparative study of transformer- based neural text representation techniques on bug triaging,” in 2023 38th IEEE/ACM International Conference on Automated Software En- gineering (ASE). IEEE, 2023, pp. 1012–1023
2023
-
[11]
A light bug triage framework for applying large pre-trained language model,
J. Lee, K. Han, and H. Yu, “A light bug triage framework for applying large pre-trained language model,” in Proceedings of the 37th ieee/acm international conference on automated software engineering , 2022, pp. 1–11
2022
-
[12]
An empirical assessment of different word embedding and deep learning models for bug assignment,
R. Wang, X. Ji, S. Xu, Y . Tian, S. Jiang, and R. Huang, “An empirical assessment of different word embedding and deep learning models for bug assignment,” Journal of Systems and Software , vol. 210, p. 111961, 2024
2024
-
[13]
Ealink: An efficient and accurate pre-trained framework for issue-commit link recovery,
C. Zhang, Y . Wang, Z. Wei, Y . Xu, J. Wang, H. Li, and R. Ji, “Ealink: An efficient and accurate pre-trained framework for issue-commit link recovery,” in 2023 38th IEEE/ACM International Conference on Auto- mated Software Engineering (ASE) . IEEE, 2023, pp. 217–229
2023
-
[14]
Git-anchor: Robust issue-to-code linking via structure-aware commit representations,
A. Akhavan, A. Hoseinpour, M. H. Aref, M. Keshani, and A. Hey- darnoori, “Git-anchor: Robust issue-to-code linking via structure-aware commit representations,” in Proceedings of the 40th IEEE/ACM In- ternational Conference on Automated Software Engineering (ASE) . IEEE/ACM, 20...
2025
-
[15]
Complex network neural dynamics framework for automated software bug triaging,
H. Cao and M. Cui, “Complex network neural dynamics framework for automated software bug triaging,” IEEE Access, 2025
2025
-
[16]
Structural contrastive learning based automatic bug triaging,
Y . Tao, J. Dai, L. Ma, Z. Ren, and F. Wang, “Structural contrastive learning based automatic bug triaging,” Automated Software Engineer- ing, vol. 32, no. 2, pp. 1–27, 2025
2025
-
[17]
Graph collab- orative filtering-based bug triaging,
J. Dai, Q. Li, H. Xue, Z. Luo, Y . Wang, and S. Zhan, “Graph collab- orative filtering-based bug triaging,” Journal of Systems and Software , vol. 200, p. 111667, 2023
2023
-
[18]
A comprehensive survey on graph neural networks,
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems , vol. 32, no. 1, pp. 4–24, 2020
2020
-
[19]
Automatic bug triage using text catego- rization,
G. Murphy and D. Cubranic, “Automatic bug triage using text catego- rization,” in Proceedings of the sixteenth international conference on software engineering & knowledge engineering . Citeseer, 2004, pp. 1–6
2004
-
[20]
Automating bug report assignment,
J. Anvik, “Automating bug report assignment,” in Proceedings of the 28th international conference on Software engineering , 2006, pp. 937– 940
2006
-
[21]
Towards effective bug triage with software data reduction techniques,
J. Xuan, H. Jiang, Y . Hu, Z. Ren, W. Zou, Z. Luo, and X. Wu, “Towards effective bug triage with software data reduction techniques,” IEEE transactions on knowledge and data engineering , vol. 27, no. 1, pp. 264–280, 2014
2014
-
[22]
Improving automated bug triaging with specialized topic model,
X. Xia, D. Lo, Y . Ding, J. M. Al-Kofahi, T. N. Nguyen, and X. Wang, “Improving automated bug triaging with specialized topic model,” IEEE Transactions on Software Engineering, vol. 43, no. 3, pp. 272–297, 2016
2016
-
[23]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
1907 arXiv
-
[24]
Deberta: Decoding-enhanced bert with disentangled attention,
P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv preprint arXiv:2006.03654 , 2020
2006 arXiv
-
[25]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...
2019
-
[26]
Deeptriage: Exploring the effectiveness of deep learning for bug triaging,
S. Mani, A. Sankaran, and R. Aralikatte, “Deeptriage: Exploring the effectiveness of deep learning for bug triaging,” in Proceedings of the ACM India joint international conference on data science and management of data , 2019, pp. 171–179
2019
-
[27]
Efficient estimation of word representations in vector space,
T. Mikolov, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , vol. 3781, 2013
2013 arXiv
-
[28]
Multi-triage: A multi-task learning framework for bug triage,
T. W. W. Aung, Y . Wan, H. Huo, and Y . Sui, “Multi-triage: A multi-task learning framework for bug triage,” Journal of Systems and Software , vol. 184, p. 111133, 2022
2022
-
[29]
Adptriage: Approximate dynamic programming for bug triage,
H. Jahanshahi, M. Cevik, K. Mousavi, and A. Bas ¸ar, “Adptriage: Approximate dynamic programming for bug triage,” IEEE Transac- tions on Software Engineering , vol. 49, no. 10, pp. 4594–4609, 2023, https://doi.org/10.1109/TSE.2023.3307243
2023
-
[30]
node2vec: Scalable feature learning for networks,
A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , 2016, pp. 855– 864
2016
-
[31]
Continuous-time dynamic network embeddings,
G. H. Nguyen, J. B. Lee, R. A. Rossi, N. K. Ahmed, E. Koh, and S. Kim, “Continuous-time dynamic network embeddings,” in Compan- ion proceedings of the the web conference 2018 , 2018, pp. 969–976
2018
-
[32]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023
2023 arXiv
-
[33]
Gpt-4 technical report,
OpenAI, “Gpt-4 technical report,” 2024, https://arxiv.org/abs/2303. 08774
2024
-
[34]
Exploring large language models for bug triaging: Opportunities and challenges,
N. Amini, W. Zhang, and R. Patel, “Exploring large language models for bug triaging: Opportunities and challenges,” Empirical Software Engineering, vol. 29, no. 4, p. 88, 2024
2024
-
[35]
Secure coding for web applications: Frameworks, challenges, and the role of LLMs,
K. Kiashemshaki, M. J. Torkamani, and N. Mahmoudi, “Secure coding for web applications: Frameworks, challenges, and the role of LLMs,” Aug. 2025, arXiv:2507.22223, doi:10.48550/arXiv.2507.22223. [Online]. Available: https://arxiv.org/abs/2507.22223
2025 doi
-
[36]
Llm-driven adaptive 6G-ready wireless body area networks: Survey and framework,
M. J. Torkamani, N. Mahmoudi, and K. Kiashemshaki, “Llm-driven adaptive 6G-ready wireless body area networks: Survey and framework,” aug 2025, arXiv:2508.08535, doi:10.48550/arXiv.2508.08535. [Online]. Available: https://arxiv.org/abs/2508.08535
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.