{"id":"924f8a19-cac1-46fd-88a6-4e9bf975b985","arxiv_id":"2504.20911","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ChatGPT and DeepSeek correctly decompose only 8-9% of bug reports zero-shot and 19-23% with few-shot prompting, with over-decomposition as the dominant error.","lead":"This paper tests whether ChatGPT and DeepSeek can break complex bug reports into smaller, self-contained units, using 127 privacy-related Apache Jira reports. It finds that both models frequently fail with plain prompts, improve when given examples, but still make many errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Few-shot demonstrations may come from the same 127-report corpus, directly inflating the RQ2 gains; the paper never states they were held out.","rationale":"The reader's conditional verdict identifies the subjectivity of manual ground truth as the weakest assumption, with the possibility of few-shot leakage as an additional concern. I agree that the absence of inter-rater reliability is a serious weakness. However, I argue the most load-bearing concern for the paper's headline claim is the potential overlap between the few-shot demonstrations and the evaluation corpus. The central quantitative result is the improvement from zero-shot to few-shot prompting; if the demonstrations come from the same 127 reports, that improvement is contaminated by construction. For a test instance that appears as a demonstration, the model is literally shown the answer in the prompt. Even if only the three displayed examples overlap, they represent up to 21% of ChatGPT's gain (from 10 to 24) and 17% of DeepSeek's gain (from 11 to 29), and the broader stylistic leakage could inflate results beyond those three instances. The paper never states that demonstration reports were held out, and the threats-to-validity section omits this risk, so the concern is specific and empirically checkable. The replication package makes the check straightforward. I still credit the paper for releasing the package and for the qualitative error taxonomy, which may survive re-analysis with held-out demonstrations, so conditional acceptance remains the appropriate verdict rather than outright rejection. The reader's verdict already matches this position, so no change is needed.","tokens_in":15940,"tokens_out":5207,"duration_ms":54418,"concrete_test":"Inspect the replication package (github.com/LLMPrivacyResearch/LLMBugReport) to recover the Jira keys of the three demonstration bug reports used in Tables XI-XIII. If any of those keys appear in the 127-report evaluation set, rerun the RQ2 evaluation with the overlapping reports excluded and recompute the true-decomposition counts for both models. If ChatGPT's count drops from 24 to 21 or below, or DeepSeek's from 29 to 26 or below, the reported few-shot improvement is materially inflated and the paper needs revision to use held-out demonstrations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the RQ2 approach (Section IV), the authors select three demonstrations for few-shot prompting, noting that 'the decomposed bug results are based on prior LLM results and our manual enhancements.' They do not state that these examples were excluded from the 127-report evaluation set. The examples shown in Tables XI-XIII (Kafka website, generated stub, CouchDB) are privacy-related Apache Jira reports that appear consistent with the corpus. If any of these reports are in the evaluation set, the model receives a complete gold input-output pair in the prompt for that test instance, making that instance trivially correct. More broadly, demonstrations drawn from the same corpus mean the few-shot condition is evaluated on examples whose distribution is encoded in the prompt itself, so the reported improvement (ChatGPT: 140%, DeepSeek: 163.64%) may reflect in-context imitation of the authors' preferred output format rather than a transferable ability to decompose unseen bug reports. This directly threatens the paper's central conclusion that 'overall performance... strongly depends on the quality of the prompts.' The Threats to Validity section (Section VI) discusses manual subjectivity and model variability but does not mention this contamination risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of 127 privacy-related bug reports from Apache Jira, evaluating ChatGPT (GPT-4o) and DeepSeek (DeepSeek-V1) on the task of decomposing bug reports into smaller, self-contained units. The authors use zero-shot prompting for RQ1 and few-shot prompting with three hand-crafted demonstrations for RQ2, and manually label each model output as a correct or incorrect decomposition. They report that the true decomposition rate rose from 10 to 24 for ChatGPT and from 11 to 29 for DeepSeek after few-shot prompting, and they identify four failure modes: over-decomposition, over-analysis, incorrect interpretation of solutions, and lacking key information. The paper also includes a preliminary question on resolution time and priority levels, and a discussion section offering recommendations for bug reporters.","tokens_in":16114,"tokens_out":3085,"duration_ms":32903,"significance":"If the claims are supported, the study would be a useful first step in understanding LLM-based bug report decomposition, an under-explored task with practical relevance for issue triage and resolution. The error taxonomy (over-decomposition, over-analysis, incorrect solution interpretation, missing key information) is a plausible and potentially reusable contribution, and the paper provides a replication package. The improvement from few-shot prompting is dramatic and, if valid, would support the general conclusion that prompt quality matters for this task. However, the central quantitative claims currently rest on a manual labeling process with no reported inter-rater reliability, and on few-shot demonstrations that may not be held out from the evaluation set. Both issues must be resolved before the findings can be accepted as evidence.","major_comments":[{"comment":"The three few-shot demonstrations in Tables XI–XIII are described as based on 'prior LLM results and our manual enhancements,' but the paper never states that these bug reports were excluded from the 127-report evaluation set. The examples shown (Kafka website, generated stub, CouchDB) appear consistent with the privacy-related Apache Jira corpus. If any demonstration report is also in the evaluation set, that test instance is given a near-gold input-output pair in the prompt, making its correct decomposition trivial. More broadly, demonstrations drawn from the same corpus allow the few-shot condition to encode the authors' preferred output format on the evaluation distribution itself, inflating the reported gains (140% for ChatGPT and 163.64% for DeepSeek) and undermining the conclusion that performance 'strongly depends on the quality of the prompts.' The paper must either explicitly confirm that the demonstrations were excluded from the 127 reports, or re-run RQ2 with held-out demonstrations.","section":"§IV RQ2 (demonstration selection)"},{"comment":"The correctness labels are produced by the authors' manual review with no inter-rater reliability metric such as Cohen's kappa, despite the statement that two reviewers independently analyzed the results. Because the entire quantitative comparison (10 vs. 24 and 11 vs. 29) is based on these binary subjective judgments about the 'correct' number of decomposed bugs and the accuracy of each description, the central numbers are not verifiable without an agreement measure. Additionally, in Section D3 the paper claims the difference in resolution times between correctly and incorrectly decomposed reports is 'statistically significant,' but no statistical test, effect size, or p-value is reported anywhere in the manuscript. Either add the missing analysis or remove the significance claim.","section":"§IV RQ1/RQ2 and §VI (manual analysis)"},{"comment":"Table XIV contains internal arithmetic inconsistencies that affect the reported results. For ChatGPT, the false-case counts sum to 103 (56 over-decomposition + 28 over-analysis + 18 incorrect solution interpretation + 1 lacking key information), but the table reports an FP total of 101. For DeepSeek, the counts sum to 93 (52 + 25 + 12 + 4), but the table reports 98. Since the 'promotion' percentages in Table XVII and the error-distribution discussion in Section IV rely on these totals, they must be recomputed and corrected.","section":"Table XIV"}],"minor_comments":[{"comment":"The dataset collection description says the study 'focused only on resolved bug reports,' but then adds reports 'not explicitly labeled as resolved' whose status is 'fixed.' The paper should clarify whether 'fixed' is a subset of 'resolved' and how this affects the resolution-time analysis in PQ1.","section":"§III-B"},{"comment":"The column header 'Promotion (%)' is ambiguous: for the False rows, the values 88.03% and 84.48% represent the remaining fraction of false cases, not a promotion. A clearer label such as 'Percentage of RQ1 count' is needed.","section":"Table XVII"},{"comment":"The abstract says 'ChatGPT' while Section IV RQ1 states the model is GPT-4o; the paper should consistently identify the exact model versions and access dates for both ChatGPT and DeepSeek, since model updates are a known source of variability.","section":"Abstract and §IV"},{"comment":"The summary paragraph before the RQ2 approach is labeled 'Summary of RQ2' but reports RQ1 results; this appears to be a typographical error and should be fixed.","section":"§IV RQ1"},{"comment":"The replication package link is a bare GitHub URL with no version or archival DOI; providing a versioned or archived snapshot would improve reproducibility and long-term accessibility.","section":"Replication package"},{"comment":"Several illustrative tables have minor formatting issues (e.g., missing spaces in code snippets, inconsistent punctuation in decomposition outputs), and the paper should state whether these outputs are verbatim model transcripts or lightly edited for space.","section":"Tables VI–X"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the possible contamination of the few-shot demonstrations with the evaluation corpus. If the authors cannot confirm that the three demonstrations were excluded from the 127 reports, the RQ2 comparison cannot support the paper's central claim, and the paper would need substantial reworking or new experiments with properly held-out demonstrations. I would also encourage the editor to ask the authors to report inter-rater reliability and the actual statistical test behind the 'statistically significant' phrase in Section D3, as these are basic requirements for the empirical claims being made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does something genuinely new: it asks whether LLMs can take a complex bug report and split it into self-contained sub-bugs, and it backs the question with a small curated dataset of 127 privacy-related Apache Jira reports and a four-category error taxonomy. The qualitative finding—that LLMs over-decompose, over-analyze, misinterpret suggested solutions, and drop key information—is plausible and well illustrated. That taxonomy alone is a useful starting point for anyone building automated bug triage.\n\nThe soft spots are mostly about the quantitative claims. The ground truth is the authors' own manual judgment of what counts as a correct decomposition, with two independent reviewers and a mediator but no reported inter-rater agreement. That matters because the task is genuinely subjective: reasonable people can disagree about whether a report contains one bug or three. The paper also claims a 'statistically significant' difference in resolution time between correctly and incorrectly decomposed reports without reporting a test or p-value.\n\nThe bigger problem is the few-shot contamination risk. The three demonstrations appear to be drawn from the same 127-report corpus; the paper never states they were held out. The Kafka, generated-stub, and CouchDB examples in Tables XI–XIII are privacy-related Apache Jira reports and would plausibly be in the evaluation set. If they are, those instances become trivially correct, and the 140% and 163.64% improvements are inflated by in-context imitation of the authors' preferred output format. Even if the examples are not in the evaluation set, drawing demonstrations from the same distribution as the test set means the few-shot condition is not a clean measure of transfer. The Threats to Validity section discusses manual subjectivity but does not mention this.\n\nNone of this kills the paper. The core qualitative finding—few-shot prompting helps but the models still struggle—is consistent with the errors shown and probably would survive a cleaner setup. What is needed is a revision that reports inter-rater reliability, separates demonstration data from evaluation data explicitly, and reports the statistical test properly.\n\nI'd send it to peer review, but I'd ask for those changes before acceptance. It's a modest but honest contribution, and the error taxonomy is worth having.","headline":"First useful benchmark for LLM bug decomposition, undermined by subjective labels and likely few-shot contamination.","tokens_in":16653,"tokens_out":1814,"would_cite":false,"duration_ms":18236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that large language models can decompose bug reports only when given well-chosen examples: adding three demonstrations raised correct decompositions from 10 to 24 (ChatGPT) and 11 to 29 (DeepSeek) out of 127 reports.","keywords":["bug report decomposition","large language models","few-shot prompting","zero-shot prompting","prompt engineering","empirical study","privacy-related bug reports"],"falsifier":"Run the identical zero-shot and few-shot prompts on a fresh, independent sample of several hundred bug reports from other domains, with gold decompositions produced by independent annotators and inter-rater agreement reported. If correct-decomposition rates do not rise from about 8-9% to about 19-23%, or if the labels themselves show near-chance agreement, the claim that few-shot prompting improves LLM bug decomposition would be unsupported.","tokens_in":15731,"feed_emoji":"🐛","tokens_out":8672,"duration_ms":79585,"temperature":0.7,"pith_summary":"The paper sets out to test whether large language models can take a single complex bug report—one that often mixes several distinct issues, code, logs, and suggested fixes—and split it into smaller, self-contained bugs that could be assigned separately. Using 127 resolved privacy-related bug reports from a public open-source bug tracker, the authors find that both ChatGPT and DeepSeek do this poorly with a plain zero-shot prompt: only 10 and 11 reports are decomposed correctly. Adding three hand-picked worked examples to the prompt lifts the correct counts to 24 and 29, increases of 140% and 163.64%. The conclusion is that LLMs are usable for bug decomposition only with careful prompt design and human verification, and that over-decomposition is the most common failure mode.","feed_headline":"Few-shot prompts triple LLM success at bug decomposition","feed_subtitle":"With three worked examples, correct splits rose from 10 to 24 (ChatGPT) and 11 to 29 (DeepSeek) on 127 reports.","key_machinery":"The load-bearing mechanism is the few-shot prompt: a small set of three hand-crafted decomposition demonstrations inserted into the prompt, one for a straightforward report, one for a report containing code or logs, and one for a report with a suggested solution. These demonstrations are intended to teach the model the output format and, more importantly, to steer it around the failure modes found in the zero-shot round—especially the instruction not to analyze code or to treat solutions as separate bugs. The argument also rests on a manual annotation pipeline in which two reviewers independently judged each decomposition by whether the number of units was correct and whether each unit's description was accurate, with a third reviewer mediating disagreements.","core_discovery":"On the paper's own terms, the discovery is that prompt quality, not model capability alone, determines whether an LLM can decompose bug reports. With zero-shot instructions, ChatGPT correctly decomposed 10 of 127 reports and DeepSeek 11; after a few-shot prompt containing three curated examples, ChatGPT reached 24 and DeepSeek 29. The authors attribute the failures to four recurring behaviors: over-decomposition (splitting one bug into several versions of the same issue), over-analysis (turning logs and code into separate sub-bugs), incorrect interpretation of solutions (treating a suggested fix as a bug), and lacking key information (omitting links, code, or logs). The paper's central claim is that LLMs show genuine potential for decomposition, but are not yet reliable enough to automate it, and that their output must be checked by a human.","pith_inferences":["The few-shot demonstrations were selected from the same 127-report corpus and manually enhanced by the authors, so the measured 140% and 163.64% gains may overstate what a user would see with demonstrations drawn from a separate source; a held-out demonstration set would give a cleaner estimate.","A natural next experiment is to add a second-pass consistency check in which the model must map each decomposed unit back to a distinct sentence or paragraph of the original report; this could directly attack the dominant over-decomposition failure.","The finding that easier-to-resolve bugs are more likely to be decomposed correctly suggests LLM decomposition quality could be used as a proxy for bug complexity in triage, though the paper does not test this use.","Adding an inter-rater agreement measure in future work would make the ground-truth labels more transparent, since the current evaluation relies on two reviewers' judgment."],"forward_implications":["Tool builders cannot rely on zero-shot LLM calls for bug decomposition; a few demonstrations are a cheap but large accuracy lever, roughly tripling correct outputs in this dataset.","Over-decomposition is the dominant error, so practical systems should add a verification step that asks the model whether two units describe the same underlying issue before accepting a split.","LLM decomposition is safe to use only as a triage aid with human review, not as a fully automated pipeline.","Reporters can improve automated analysis by filing atomic, well-structured, concise reports, as the paper's discussion recommends.","Because correctly decomposed reports had shorter resolution times, decomposition success may serve as a rough indicator of bug complexity."],"supporting_citations":[{"why":"motivates the decomposition task by showing bugs and bug reports grow more complex over time","marker":"[7]"},{"why":"establishes which bug report elements developers rely on, informing what a good decomposition must preserve","marker":"[26]"},{"why":"documents that bug reports can be thousands of lines long, motivating automated simplification","marker":"[21]"},{"why":"shows earlier NLP extraction of bug report information, the task this study extends","marker":"[9]"},{"why":"asserts that prompt phrasing changes LLM output quality, the central variable under study","marker":"[30]"},{"why":"identifies ChatGPT as one of the two evaluated models","marker":"[20]"},{"why":"identifies DeepSeek as the other evaluated model","marker":"[32]"},{"why":"demonstrates LLM agents decomposing a complex request into subtasks, the capability tested here","marker":"[25]"},{"why":"shows reasoning-oriented prompts improve LLM task performance, informing the few-shot prompt design","marker":"[29]"}],"fun_headline_variants":["Prompt tuning boosts LLM bug decomposition by 140-164%","Few-shot prompts, not model choice, solve bug splitting","LLMs decompose bug reports, but prompt quality is key","Curated examples double LLM success at splitting bugs","Zero-shot fails, few-shot works for LLM bug splits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that the two reviewers' judgment of the correct number and wording of decomposed bugs is an objective ground truth, yet the paper reports no inter-rater agreement metric and the few-shot examples come from the same corpus and were manually enhanced by the authors.","fun_headline_variants_meta":{"raw":{"variants":["Prompt tuning boosts LLM bug decomposition by 140-164%","Few-shot prompts, not model choice, solve bug splitting","LLMs decompose bug reports, but prompt quality is key","Curated examples double LLM success at splitting bugs","Zero-shot fails, few-shot works for LLM bug splits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000975,"raw_usage":{"total_tokens":4149,"prompt_tokens":954,"completion_tokens":3195,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3112}},"tokens_in":570,"tokens_out":3195,"duration_ms":23632,"temperature":1.0,"reasoning_tokens":3112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:15:04.688844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical zero-shot and few-shot prompts on a fresh, independent sample of several hundred bug reports from other domains, with gold decompositions produced by independent annotators and inter-rater agreement reported. If correct-decomposition rates do not rise from about 8-9% to about 19-23%, or if the labels themselves show near-chance agreement, the claim that few-shot prompting improves LLM bug decomposition would be unsupported.","supporting_citations":[{"cited_title":"The significance of bug report elements","cited_arxiv_id":null,"evidence_quote":"establishes which bug report elements developers rely on, informing what a good decomposition must preserve"},{"cited_title":"Ausum: approach for unsupervised bug report summarization","cited_arxiv_id":null,"evidence_quote":"documents that bug reports can be thousands of lines long, motivating automated simplification"},{"cited_title":"What’s in a bug report? In Proceedings of the 8th ACM/IEEE Interna- tional Symposium on Empirical Software Engineering and Measurement , ESEM ’14, New York, NY , USA,","cited_arxiv_id":null,"evidence_quote":"shows earlier NLP extraction of bug report information, the task this study extends"},{"cited_title":"Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design","cited_arxiv_id":null,"evidence_quote":"asserts that prompt phrasing changes LLM output quality, the central variable under study"},{"cited_title":"Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,","cited_arxiv_id":null,"evidence_quote":"demonstrates LLM agents decomposing a complex request into subtasks, the capability tested here"}],"review_version":1}