{"id":"aa5b0655-6306-4997-9144-f28e531f2802","arxiv_id":"2508.21156","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Instruction-tuned LLM with candidate-constrained decoding gives useful developer shortlists (Hit@10 up to 0.753) but weak exact Top-1 on multi-year bug data.","lead":"This paper fine-tunes an 8B LLM with LoRA and a candidate list to recommend developers for bug reports, reporting Hit@10 up to 0.753 on large issue datasets. It is positioned as a lightweight alternative to graph-based triage, but reproducibility and data provenance issues weaken the claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Jan–Jun 2025 snapshot (Table VI) is undocumented and likely filters to assignees in the training roster, so the 'practical' Hit@10 claim is not yet supported.","rationale":"The reader's weakest assumption identifies label-space restriction and long-tail filtering as the core threat, and I agree that this is the mechanism that can inflate Hit@K. However, the most load-bearing manifestation is not the multi-year NCGBT-aligned evaluation—which, even if filtered, is at least transparent about its protocol and is mixed (EclipseJDT lags NCGBT, Mozilla shows a strange Hit@2 jump)—but the Jan–Jun 2025 snapshot in Table VI. That snapshot is where the paper's 'practical alternative' claim comes from, yet it is introduced without any data-collection methodology, despite falling outside the dataset windows described in Section III-A. A 0.83/0.99 result on EclipseJDT is dramatically higher than the multi-year 0.156/0.475, and the only stated constraint is that the label space is restricted to developers in the training split. This implies the test issues had to be assigned to pre-2015/pre-2021 developers, which is a severe selection bias if not disclosed. My proposed check—evaluating the full Jan–Jun 2025 issue stream without the roster filter and reporting coverage—would settle whether the concern lands. If coverage is high and unfiltered Hit@10 remains close to Table VI, the concern does not land and the snapshot result stands. If not, the practical claim is unsupported until the authors provide an unfiltered evaluation. Since this is fixable by releasing data and recomputing, the reader's CONDITIONAL verdict remains appropriate; I do not see grounds to move to REJECT, and ACCEPT would be premature.","tokens_in":13823,"tokens_out":7375,"duration_ms":81607,"concrete_test":"Ask the authors for the exact collection script and raw issue IDs for the Jan–Jun 2025 snapshot. Re-run Table VI on the complete set of EclipseJDT and Mozilla issues with an assignee in Jan–Jun 2025, using the same training-roster candidate set but without dropping issues whose gold assignee is absent from the roster. Report (a) coverage = fraction of gold assignees present in the roster and (b) Hit@10 on this unfiltered set. If coverage is below 90% or the unfiltered Hit@10 drops by more than 0.1 absolute versus Table VI, the reported snapshot is an artifact of roster filtering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central practical claim rests on the recent six-month snapshot in Section IV-F: EclipseJDT Top-1/Hit@10 = 0.83/0.99 and Mozilla 0.615/0.72 (n=200 each). But Section III-A documents only EclipseJDT from Nov 2007–Nov 2015 and Mozilla from Jun 1999–Feb 2021; no collection details are given for the Jan–Jun 2025 snapshot. Candidate-constrained decoding in this framework can only return developers in a roster built from training labels (Section III-C.3), so for the reported Hit@10 to be meaningful, every one of the 200 test issues must have its gold assignee present in the pre-2025 training roster. If, instead, the authors selected 2025 issues whose assignees happen to be old developers—or dropped issues assigned to newer contributors—the snapshot result is conditional on a favorable subset and does not reflect real-world triage demand, where new contributors and long-tail developers are common. The paper does not report the coverage rate (fraction of Jan–Jun 2025 issues whose gold assignee is in the roster), nor does it describe how the 200 issues were sampled. Without this, the abstract's claim that 'accuracy rises sharply' on recent snapshots is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a bug-triaging framework built on an instruction-tuned 8B LLM (DeepSeek-R1-Distill-Llama-8B) with LoRA adapters and 4-bit NF4 quantization. Fine-tuned on EclipseJDT and Mozilla Bugzilla data converted to JSONL prompts, the model is evaluated with candidate-constrained decoding that restricts outputs to a known developer roster and produces ranked Top-K recommendations. Using what the authors describe as the NCGBT multi-year temporal windows and filtering protocol, the paper reports Top-1/Hit@10 of 0.156/0.475 on EclipseJDT and 0.013/0.753 on Mozilla, and on a Jan–Jun 2025 snapshot it reports 0.83/0.99 and 0.615/0.72 (n=200 per project). The paper positions the framework as a lightweight, practical alternative to feature-engineered and graph-based methods, emphasizing shortlist quality rather than exact assignment.","tokens_in":14091,"tokens_out":7421,"duration_ms":78151,"significance":"If the results are substantiated, the paper makes a useful empirical contribution: it shows that a relatively small instruction-tuned LLM with constrained decoding can produce ranked shortlists for bug triage without handcrafted features or graph construction, and it provides a direct comparison with a state-of-the-art graph baseline on the same datasets. The authors are honest about the mixed multi-year results and about the difference between Top-1 and Hit@K. The candidate-constrained decoding idea is sensible, and restricting the label space to training-roster developers is a standard way to avoid test-label leakage. However, several load-bearing evaluation details are currently undocumented or internally inconsistent, and the recent-snapshot claim at the center of the abstract is not yet supported.","major_comments":[{"comment":"The Jan–Jun 2025 snapshot is undocumented. No source query, filtering rules, sampling method, or roster coverage rate is given. Since candidate-constrained decoding can only output developers from the training roster (Section III-C.3, IV-A), the reported Top-1/Hit@10 of 0.83/0.99 and 0.615/0.72 are interpretable only if every one of the 200 sampled issues has a gold assignee in the pre-2025 roster. The paper does not report the coverage rate or how the 200 issues were selected. The abstract's claim that 'accuracy rises sharply' on recent snapshots is therefore unsupported.","section":"IV-F, Table VI"},{"comment":"The test-set denominators are inconsistent. Table II reports ~2,000 (EclipseJDT) and ~12,000 (Mozilla) test issues, and Section IV-D repeats those numbers; the same paragraph then says n=1,612 and n≈11,050, which are the denominators implied by Table III counts (765/0.475 and 8,318/0.753). All ratios change by up to 20% depending on which denominator is used. Please align Table II, Section IV-D, and Table III, and state whether the <10-bug developer filter is applied before or after the split.","section":"IV-D, Tables II and III"},{"comment":"Hit@1=0.013 (146) but Hit@2=0.743 (8,213), and Hit@3–10 increase by only ~0.01. This abrupt jump is unexplained and is central to the 'strong shortlist' claim. If true, it means the true assignee is the second-ranked candidate in ~74% of test issues; if it is an artifact of candidate-list ordering, deduplication, or a different prompt for Top-1 vs Top-K, the claimed Hit@K advantage is not valid. Please report per-rank statistics, examples, and the exact decoding procedure used for each metric.","section":"Table III, Mozilla row"},{"comment":"'Maximum steps: 500 (approximately one full epoch over the training set)' is inconsistent with the stated effective batch size of 8 and training sizes of ~16,000 and ~96,000. One epoch would require ~2,000 and ~12,000 steps, respectively. Please correct the reported training setup or clarify what subset is used; otherwise the 'fully reproducible' claim (Section IV-G) cannot be assessed.","section":"III-E"},{"comment":"The paper acknowledges that bootstrap confidence intervals, Wilcoxon tests, and multi-seed runs are missing. Because the Mozilla Hit@2 jump and the snapshot results are single-seed point estimates, no inferential statement about practical superiority can be made. Please add at least bootstrap CIs over test issues and, ideally, 3–5 seeds for the main tables.","section":"V-D"}],"minor_comments":[{"comment":"Table I lists 16,106/110,467 bugs after filtering, while Table II reports ~20,000/~120,000 totals. These should be reconciled so the split denominators are clear.","section":"III-A, Tables I-II"},{"comment":"The paper promises release of code, prompts, and per-issue predictions but provides no repository URL; with a 'fully reproducible' claim, a link or appendix is needed.","section":"IV-G and footnote 1"},{"comment":"The 'Improve' row reports negative values for what appears to be a difference. Label it 'Difference' or explain the sign convention.","section":"Table V"},{"comment":"The caption is incomplete ('Radar chart comparing' ends abruptly).","section":"Figure 5"},{"comment":"The example JSONL record contains literal \\n and quotes; format it as a code block for readability.","section":"III-A, example JSONL"}],"recommendation":"major_revision","confidential_remarks":"The self-citations [35] and [36] appear outside the paper's scope and are not used substantively; the editors may wish to check citation relevance. The reproducibility claim is also undermined by the absence of a repository link, though this is fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read through this one. The core idea is straightforward: fine-tune DeepSeek-R1-Distill-Llama-8B with LoRA on EclipseJDT and Mozilla issue text, constrain decoding to a roster of valid developers, and evaluate ranked recommendations with Hit@K. That combination hasn't been systematically tested on these NCGBT benchmarks before, and the multi-year results are worth taking seriously. On Mozilla they get Hit@10 = 0.753, beating the graph-based NCGBT baseline on shortlist quality despite a terrible Top-1 of 0.013. On EclipseJDT they lag NCGBT at every K. That mixed picture is actually the most credible part of the paper: it shows the method is not a universal win, and the authors are candid about the long-tail and cold-start failures.\n\nAlso solid: the paper is well-organized, with genuine threats-to-validity and ethics sections. They promise code, prompts, and per-issue predictions, though as of this version there's no repository link, so 'fully reproducible' is a promise, not an accomplishment.\n\nNow the soft spots. The six-month snapshot (Jan–Jun 2025) in Table VI is the biggest problem. The paper documents only the multi-year windows (EclipseJDT ends 2015, Mozilla 2021), yet the snapshot is used to claim 'accuracy rises sharply' and practical utility. We're given no collection description, no sampling method, and no coverage rate—specifically, how many 2025 issues had a gold assignee that actually appears in the training roster. Since candidate-constrained decoding can only output roster members, the high numbers (EclipseJDT Hit@10 0.99) are only meaningful if every test issue's assignee is in the roster. If they selected 200 issues whose assignees happen to be old devs, the result is baked in. The stress-test note is right: this claim is not yet supported.\n\nThere are also internal inconsistencies. The Mozilla Hit@2 jumps from 0.013 to 0.743 (146 to 8213 hits) with no explanation. That's the kind of jump that usually signals an evaluation bug, and it needs to be explained before I'd trust the Hit@K curve. The training description says max_steps=500 is 'approximately one epoch,' but with an effective batch size of 8 and 16k/96k training issues, one epoch would be 2,000/12,000 steps. Test set sizes also differ between Section IV (2,000/12,000) and Table III (1,612/11,050). No error bars or multi-seed runs, which they acknowledge.\n\nNone of this looks fraudulent—it looks like a rushed preprint. The multi-year comparison with NCGBT is a plausible empirical result that could survive scrutiny after the snapshot is either documented properly or removed, the Hit@2 anomaly is fixed, and the code is released. I'd send it to peer review, but I'd ask the reviewers to focus on those specific issues.\n\nWho's it for? Researchers working on bug triage or developer recommendation, and anyone using LLM evaluation on long-tail label spaces. It's a useful data point, not a breakthrough.","headline":"A plausible LoRA+constrained-decoding pipeline for bug triage, with one strong multi-year result on Mozilla and an unsupported recent-snapshot claim that should not be trusted as reported.","tokens_in":14651,"tokens_out":3355,"would_cite":false,"duration_ms":33464,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight instruction-tuned LLM, forced to choose only from a known roster, turns bug triage into ranked shortlists, catching the true developer in 75% of Mozilla test bugs despite near-zero exact-match accuracy.","keywords":["bug triaging","developer recommendation","instruction-tuned LLM","low-rank adapters","candidate-constrained decoding","Hit@K","EclipseJDT","Mozilla"],"falsifier":"Run the same pipeline on a held-out set of issues whose true assignee was not present in the training-label roster (for example, brand-new contributors), and check whether Hit@10 collapses; if it does, the closed-world roster built from training labels is doing most of the work.","tokens_in":13674,"feed_emoji":"🐛","tokens_out":10796,"duration_ms":93003,"temperature":0.7,"pith_summary":"The paper claims that bug triaging can be handled by a lightweight instruction-tuned large language model that needs no handcrafted features, graph construction, or complex preprocessing. The model is fine-tuned with low-rank adapters on issue-tracker data and is forced at decoding time to choose only from a known roster of developers, producing a ranked Top-K shortlist. On the same multi-year datasets used by a prior graph-based baseline, the model reaches Top-1/Hit@10 of 0.156/0.475 on EclipseJDT and 0.013/0.753 on Mozilla, meaning the true assignee appears in the top 10 for about three-quarters of Mozilla test issues even though exact-match accuracy is tiny. On a recent six-month snapshot with a smaller, current roster, accuracy jumps to 0.83/0.99 for EclipseJDT and 0.615/0.72 for Mozilla. If these results hold, the approach is a practical assistive tool for human-in-the-loop triage, with the main remaining gap being exact Top-1 assignment under large, long-tailed label spaces.","feed_headline":"Top-10 shortlist finds correct developer for 75% of Mozilla bugs","feed_subtitle":"A roster-constrained language model turns bug triage into ranked shortlists with little engineering overhead.","key_machinery":"The load-bearing mechanism is candidate-constrained decoding: at inference the model is forbidden from emitting any identifier outside a roster of valid assignees built from training labels and/or the official list, then outputs a comma-separated Top-10 list that is regex-validated, deduplicated, and padded. This constraint is what converts the LLM's text understanding into a usable shortlist and prevents hallucinated identifiers. The other component is the training setup: a conversational JSONL prompt (system/user/assistant) with a low-rank adapter on an 8B-parameter instruction-tuned LLM, using 4-bit quantization, so the whole pipeline stays cheap to train and run. The paper argues the com","core_discovery":"The central claim is that an instruction-tuned LLM, combined with candidate-constrained decoding, provides a practical alternative to feature-engineered classifiers and graph neural networks for bug triage. The paper demonstrates this on EclipseJDT and Mozilla by following the same multi-year temporal windows and developer filtering as the graph-based baseline it compares against. With a low-rank-adapted 8B-parameter model and 4-bit quantization, it attains Top-1/Hit@10 of 0.156/0.475 on EclipseJDT and 0.013/0.753 on Mozilla; against the graph-based baseline it lags on EclipseJDT across all K (0.475 vs 0.675 at K=10) but surpasses it on Mozilla shortlists (0.753 vs 0.522 at K=10). On a Jan-J","pith_inferences":["Beyond the paper: the large jump from multi-year to six-month numbers suggests most of the difficulty is roster size and long-tail sparsity, not the model's text understanding; if so, shrinking or re-ranking the candidate set should help more than scaling the model.","Beyond the paper: the Mozilla pattern (Top-1 0.013, Hit@10 0.753) implies a two-stage workflow—LLM proposes 10, a lightweight re-ranker or human picks—could outperform both pure LLM and pure graph methods; this is a testable design, not something the paper evaluates.","Beyond the paper: because the candidate roster is built from training labels, an open-world deployment with new contributors would likely see lower Hit@K; a clean test is to evaluate on issues whose assignee never appeared in training."],"forward_implications":["Maintainers can use the Top-10 shortlist as a recommendation panel: even a 0.75 Hit@10 on Mozilla means the right developer is usually on the menu, cutting search time even when rank 1 is wrong.","The Jan-Jun 2025 snapshot shows the method is most useful on current rosters; periodic roster refresh and incremental low-rank fine-tuning can keep it aligned with team turnover.","Because the approach needs no handcrafted features or graph construction, it can be adapted to a new project by formatting issue-tracker data and fine-tuning an adapter.","The gap between Top-1 and Hit@10 motivates hybrid designs, such as graph-derived priors or developer profile embeddings to re-rank the LLM shortlist, and pairwise or listwise losses to improve intra-list ordering."],"supporting_citations":[{"why":"Supplies the multi-year dataset windows, developer filtering protocol, and evaluation numbers the paper reproduces and compares against.","marker":"[8]"},{"why":"The zero-shot LLM bug-triaging baseline whose prompt sensitivity motivates instruction tuning and candidate constraints.","marker":"[34]"},{"why":"Documents the graph-based approach the paper positions against and the bug-tossing problem motivating triage.","marker":"[1]"},{"why":"Shows a recent graph-based pipeline that uses generative models only as components, highlighting the gap this paper fills.","marker":"[2]"},{"why":"Transformer text-representation comparison on bug triaging, a text-classification baseline the paper must be viewed against.","marker":"[10]"},{"why":"Lightweight fine-tuned pre-trained model for bug triage, the closest compute-efficient text-only baseline.","marker":"[11]"}],"fun_headline_variants":["LLM triage shortlists hit 75% on Mozilla, beating graph nets","Lightweight LLM ranks bug fixers, tops graphs on Mozilla shortlist","Top-10 bug fixers found for 75% of Mozilla cases via LLM","Roster-constrained LLM triage: strong shortlists, modest top-1","Instruction-tuned LLM: practical bug triage without heavy features"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central evaluation assumes every retained test issue has its gold assignee inside the candidate roster built from training labels or an official list, with no test-label leakage; if the test set were to include issues whose true assignee is a new or low-activity developer, or if the recorded assignee is not the actual fixer, the reported Hit@10 numbers would be conditional and could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["LLM triage shortlists hit 75% on Mozilla, beating graph nets","Lightweight LLM ranks bug fixers, tops graphs on Mozilla shortlist","Top-10 bug fixers found for 75% of Mozilla cases via LLM","Roster-constrained LLM triage: strong shortlists, modest top-1","Instruction-tuned LLM: practical bug triage without heavy features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2457,"prompt_tokens":675,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":1689}},"tokens_in":419,"tokens_out":1782,"duration_ms":13721,"temperature":1.0,"reasoning_tokens":1689,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:32:55.069133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a held-out set of issues whose true assignee was not present in the training-label roster (for example, brand-new contributors), and check whether Hit@10 collapses; if it does, the closed-world roster built from training labels is doing most of the work.","supporting_citations":[{"cited_title":"Neighborhood contrastive learning-based graph neural network for bug triaging,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-year dataset windows, developer filtering protocol, and evaluation numbers the paper reproduces and compares against."},{"cited_title":"Exploring large language models for bug triaging: Opportunities and challenges,","cited_arxiv_id":null,"evidence_quote":"The zero-shot LLM bug-triaging baseline whose prompt sensitivity motivates instruction tuning and candidate constraints."},{"cited_title":"A spatial–temporal graph neural network framework for automated software bug triaging,","cited_arxiv_id":null,"evidence_quote":"Documents the graph-based approach the paper positions against and the bug-tossing problem motivating triage."},{"cited_title":"IssueCourier: Multi-Relational Heterogeneous Temporal Graph Neural Network for Open-Source Issue Assignment","cited_arxiv_id":"2505.11205","evidence_quote":"Shows a recent graph-based pipeline that uses generative models only as components, highlighting the gap this paper fills."},{"cited_title":"A comparative study of transformer- based neural text representation techniques on bug triaging,","cited_arxiv_id":null,"evidence_quote":"Transformer text-representation comparison on bug triaging, a text-classification baseline the paper must be viewed against."},{"cited_title":"A light bug triage framework for applying large pre-trained language model,","cited_arxiv_id":null,"evidence_quote":"Lightweight fine-tuned pre-trained model for bug triage, the closest compute-efficient text-only baseline."}],"review_version":1}