{"id":"c86d18a0-351a-4f66-bdc5-95921e899ed1","arxiv_id":"2504.18806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new curated dataset combines 119,585 Mozilla bug reports with comments and 19,351 contributor profiles, plus a 10,351-report subset filtered for structure and quality.","lead":"BugsRepo gathers 119,585 resolved Mozilla bug reports with their comments, contributor profiles for 19,351 people, and a filtered subset of 10,351 supposedly well-structured reports. It is a convenient combined resource for bug triage, severity prediction, and summarization research, though the quality filtering is not independently validated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'well-structured' 10,351-report subset rests on an unvalidated transfer of the CTQRS 75% threshold from mobile crowdsourced test reports to Mozilla Bugzilla reports; no score distribution, regex definitions, or manual precision check is reported, so the 'higher-quality corpus' claim is not yet…","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the regex + CTQRS filtering pipeline is the only support for the 'higher-quality corpus' claim, and it is unvalidated. My stress-test agrees and narrows the concern to the domain transfer of the CTQRS threshold: CTQRS was built and evaluated for crowdsourced mobile test reports, and the paper provides no score distribution, regex patterns, or manual precision check to show that a 75% threshold has the same meaning for Mozilla Bugzilla reports. I do not see an internal inconsistency or a reason to reject the paper; the raw metadata, comments, and contributor data appear genuinely useful, and the authors provide a Zenodo dataset link and GitHub code, which are independent evidence that the resource exists and can be re-examined. The concern affects the strength of the characterization, not the usefulness of the raw components, so the appropriate outcome remains the reader's CONDITIONAL verdict pending the validation checks described above. Minor wording issues, malformed citation markers (e.g., the '[21?]' in Section 3 and duplicated Bugzilla guidelines references [29]/[30]/[31]), and vague phrases like 'high-quality' also support a conditional rather than full accept, but they are secondary to the threshold validation.","tokens_in":11288,"tokens_out":3582,"duration_ms":35248,"concrete_test":"Run the provided GitHub mining and CTQRS code on the full 119,585-report snapshot, record the CTQRS score distribution for the 12,614 regex-passing reports, and then have two independent annotators classify a random sample of at least 100 reports from each score band (e.g., below 50%, 50-74%, 75% and above) as 'follows Bugzilla guidelines' or not. Compute precision, recall, and Cohen's kappa at the 75% threshold; also report the overlap between regex-only and regex+CTQRS sets. If precision at >=12.75 is not substantially higher than the base rate of well-structured reports among the 12,614, or if the threshold does not separate the score bands, the higher-quality subset claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 2 is that the curated subset of 10,351 reports is a 'higher-quality corpus compared to typical Bugzilla data.' That claim depends on the Section 4 filtering pipeline: regex detection of Steps-to-Reproduce, Actual Behavior, and Expected Behavior sections reduces 119,585 reports to 12,614, and a CTQRS score above 75% (12.75 of 17) reduces the set to 10,351. The load-bearing weakness is that this pipeline is not validated for Mozilla bug reports. CTQRS (Zhang et al. [42]) was designed and presumably calibrated for crowdsourced mobile test reports, where indicators such as interface elements, user behavior, screenshots, and defect descriptions have a particular distribution. Mozilla Bugzilla reports are longer, often pasted logs, upstream discussions, or technical accounts, and the original CTQRS indicators may not rank them in the same way. The paper does not report the CTQRS score distribution on the 12,614 regex-passing reports, does not specify the regular expressions used to detect S2R/AR/ER, and reports no manual inspection, inter-rater agreement, or comparison against existing quality-labeled Bugzilla data. Section 6 lists scraping and rate-limiting challenges but does not acknowledge that the 'well-structured' subset lacks a precision check. If the threshold is miscalibrated for this domain, the 10,351 subset may contain many reports that merely have long or itemized text, while excluding concise high-quality reports. The raw metadata, comments, and contributor datasets would remain useful, but the headline higher-quality claim would not be established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"BugsRepo is a data paper that introduces a Mozilla Bugzilla dataset in three parts: a metadata/comment collection for 119,585 resolved bugs, a contributor profile collection for 19,351 users, and a filtered subset of 10,351 reports claimed to be well-structured. The mining pipeline uses the Bugzilla REST API and BeautifulSoup, and the filtering pipeline first applies regular expressions for Steps-to-Reproduce, Actual Behavior, and Expected Behavior sections and then applies the CTQRS quality score with a 75% threshold. The paper reports dataset sizes, gives a schema overview, lists applications such as triage and summarization, and provides Zenodo and GitHub links.","tokens_in":11591,"tokens_out":5582,"duration_ms":54498,"significance":"The raw resource is a plausible and potentially valuable contribution: the combination of metadata, comment threads, and contributor activity in one downloadable archive is not common among the existing Bugzilla datasets listed in Table 1, and the publication of both the data and the mining code supports reproducibility. If the dataset is as large and as complete as described, it can support work on triage, severity prediction, summarization, and developer recommendation. However, the paper's flagship claim that the 10,351-report subset is a 'higher-quality corpus' is not yet supported by the evidence presented, and the central filtering step needs validation before this claim can be accepted.","major_comments":[{"comment":"The central claim that the 10,351-report subset is a higher-quality corpus (Section 2, final paragraph) depends on the CTQRS filtering step in Section 4, but this step is not validated for Mozilla Bugzilla reports. The CTQRS framework was developed for crowdsourced mobile test reports, and the paper provides no evidence that its indicators or the 75% threshold transfer to long, log-heavy Mozilla reports. In particular, the paper does not report the CTQRS score distribution on the 12,614 regex-passing reports, does not report manual inspection of a sample, and does not compare the filtered subset with an independent quality label. Without this evidence, the 10,351-report subset may simply be a set of long reports that satisfy the score's length-related components, not necessarily well-structured reports. I ask the authors to add a validation section with a score distribution, a manual precision check (ideally with inter-rater agreement), and a justification or recalibration of the 75% threshold on Mozilla data.","section":"Section 4 (CTQRS paragraph) and Section 2"},{"comment":"The regular expressions used to detect S2R, AR/AB, and ER are not specified, and the paper's own counterexample suggests that naive section-label detection is insufficient. Figure 4, labeled as a low-quality/unstructured report, contains the literal heading 'Steps to reproduce:' followed by a long narrative description; a regex that merely checks for the presence of that label would admit this report. The paper reports that 119,585 reports reduce to 12,614 after regex filtering, but without the actual patterns, the set of reports that pass is not precisely defined. Please make the regular expressions available in the text or appendix and report a precision estimate for the regex step on a manually labeled sample.","section":"Section 4 (regular-expression filtering, Figure 4)"},{"comment":"The limitations section discusses scraping challenges and data inclusion criteria (enhancement excluded, resolved only) but does not acknowledge or discuss the absence of validation for the 'well-structured' subset. Since the paper's third contribution is exactly this subset, the limitation is load-bearing rather than peripheral. The authors should either provide the validation evidence in Section 4 or explicitly state in Section 6 that the structured subset has not yet been manually validated and should be treated as a large candidate set of structured reports.","section":"Section 6 (Limitations)"}],"minor_comments":[{"comment":"The sentence '... often focus primarily on bug metadata or comment text [21? ]' contains a malformed citation marker '[21? ]' that should be corrected to a proper reference.","section":"Section 3, text before Table 1"},{"comment":"The paper uses both 'AR' and 'AB' for actual behavior; for example, Section 2 uses 'AR' while Section 1 and Figure 3 use 'AB'. Choose one abbreviation and use it consistently.","section":"Sections 1, 2, and 4"},{"comment":"The text says 'using a regular expression as shown in ➌ of Figure 2', but the regular expression itself is not shown in the figure or elsewhere; include the actual patterns in the text, appendix, or repository.","section":"Section 4 (Figure 2)"},{"comment":"References [29], [30], and [31] all point to the same Mozilla Bug Writing Guidelines page; consolidate them into a single reference.","section":"References [29], [30], and [31]"}],"recommendation":"major_revision","confidential_remarks":"The paper is a dataset paper and the raw data appears to be real and accessible. The gap is in validating the quality-filtered subset; this is fixable with additional experiments and should not require re-curating the raw data. I would be comfortable with major revision, and I would not want to see this paper rejected solely because the CTQRS transfer is currently unvalidated, provided the authors close the gap by adding the requested validation and making the filtering artifacts concrete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read BugsRepo. Short version: the raw integrated dataset is a genuine contribution, and the code/data links make it verifiable. But the 'higher-quality structured subset' claim is not established; the CTQRS threshold transfer is unvalidated and the regex filter is underspecified. I would use the raw data, but be skeptical of the 10,351-report subset until it is validated.\n\nWhat is genuinely new: no one else combines metadata, comments, contributor profiles, and a filtered subset in one package. The contributor dataset with activity metrics is a nice addition for developer-recommendation research. The authors mined 119,585 resolved bugs from 50+ Mozilla projects, included comments and lifecycle, and shipped the mining code on GitHub with the dataset on Zenodo. That is a solid, reproducible artifact.\n\nThe main soft spot is Section 4. The regex filter that reduces the set to 12,614 reports is underdescribed: the actual regular expressions are not given, and Figure 4's 'low-quality' example actually contains a 'Steps to reproduce:' heading, which suggests the filter can catch narrative text that is not truly structured. Then CTQRS is applied with a 75% threshold adopted directly from crowdsourced mobile test reports. The paper reports no CTQRS score distribution on the 12,614 reports, no manual precision check, and no inter-rater agreement. So the claim that the 10,351 subset is a 'higher-quality corpus compared to typical Bugzilla data' is not supported. It may be true, but the evidence is missing. The authors do not even acknowledge the domain mismatch in Section 6; they only mention scraping and rate-limiting challenges.\n\nOther issues are minor: a broken citation marker '[21?]' in Section 3, duplicated Mozilla guidelines references, and a replication link that appears to be the same GitHub repo. The applications section is speculative but harmless.\n\nIf I were refereeing, I would ask for validation of the subset: report the CTQRS distribution, provide the regexes, run a manual inspection on a random sample with agreement metrics, and soften the language from 'high-quality' to 'reports matching the filter.' The raw data and contributor dataset can stand on their own.\n\nAudience: researchers in mining software repositories, bug triage, and bug report summarization. This paper deserves a serious referee rather than a desk reject. With the subset validated or the claim toned down, it would be a reasonable data paper; I would recommend conditional acceptance with required validation.","headline":"BugsRepo's raw integrated dataset is a real, useful resource, but the 'higher-quality structured subset' claim is not yet supported and needs validation before it is cited for that purpose.","tokens_in":12162,"tokens_out":2456,"would_cite":true,"duration_ms":24162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BugsRepo presents a curated Mozilla Bugzilla resource that pairs 119,585 bug records and comments with contributor profiles, and offers a 10,351-report quality-filtered subset.","keywords":["Bug-Fix Dataset","Mining Software Repositories","Software Maintenance","Bug report quality","Bugzilla","CTQRS","Contributor information","Bug triage"],"falsifier":"Draw a random sample of 100 reports from the 10,351-report subset, have independent annotators judge whether each report truly contains clear steps to reproduce, actual behavior, and expected behavior, and compare those judgments against the CTQRS scores and the regex pass. A weak match would show the filter is not measuring what the paper says it measures.","tokens_in":11060,"feed_emoji":"🐞","tokens_out":6102,"duration_ms":52885,"temperature":0.7,"pith_summary":"BugsRepo is a dataset paper that claims a single curated resource can give a \"holistic view\" of Mozilla bug history by joining three kinds of Bugzilla data: metadata and comment threads for 119,585 resolved bugs, account profiles for 19,351 contributors, and a 10,351-report subset that passes a two-stage quality filter. The authors argue that existing bug datasets focus on metadata or comment text separately, so they miss the connection between a bug's characteristics, its discussion, and the expertise of the people involved. If the curation works, software maintenance tasks such as bug triage, summarization, and severity prediction get both cleaner training data and a new contributor-information dimension.","feed_headline":"Dataset distills 119,585 Mozilla bugs into 10,351 structured reports","feed_subtitle":"It joins bug metadata, comments, and 19,351 contributor profiles for maintenance tasks.","key_machinery":"The load-bearing machinery is CTQRS (Crowdsourced Test Report Quality Score), a dependency-parsing-based scoring framework that rates a report on morphological, relational, and analytical indicators to a maximum of 17 points. Applied with a 75 percent threshold after the regular-expression section check, it is what converts the raw Bugzilla dump into the curated subset. The framework supplies the objective-looking quality gate that the paper's \"higher-quality corpus\" claim depends on.","core_discovery":"The paper's central claim is that the 10,351-report subset \"can be regarded as a higher-quality corpus compared to typical Bugzilla data.\" The claim rests on a filter that first uses regular expressions to keep reports containing Steps to Reproduce, Actual Behavior, and Expected Behavior sections, then applies the CTQRS quality score with a 75 percent threshold, reducing an initial 119,585 reports to 12,614 and finally to 10,351. The companion datasets—bug metadata with comment threads and 19,351 contributor profiles with activity metrics such as bugs filed, comments made, and patches reviewed—are presented as enabling tasks the structured subset alone cannot support, such as developer recommendation and workload-aware triage.","pith_inferences":["Beyond the paper: the CTQRS threshold was designed for crowdsourced mobile test reports, so a direct validation against human ratings of Mozilla reports would test whether the transferred threshold marks quality or merely template adherence.","Beyond the paper: the regex step likely over-selects reports whose authors followed the Bugzilla template closely; the subset may be skewed by project, reporter, or report age, so users should check representativeness before training.","Beyond the paper: the contributor data admits a testable prediction—if the holistic view is valuable, bugs filed by high-activity contributors should show measurably different severity assignment and resolution patterns than those from one-time reporters."],"forward_implications":["Models for bug triage and severity prediction can be trained on the 10,351 structured reports, reducing the noise that incomplete descriptions introduce.","Contributor activity metrics allow developer recommendation by matching a bug to members whose past bugs, comments, and reviewed patches fit the task.","Comment threads combined with structured S2R/ER/AR sections give summarization systems both discussion context and a reproducibility skeleton.","Coverage of more than 50 Mozilla projects lets triage policies be evaluated for cross-project generalization."],"supporting_citations":[{"why":"Supplies the CTQRS scoring framework and its 75 percent high-quality threshold.","marker":"[42]"},{"why":"Mozilla's bug writing guidelines define the S2R/AR/ER structure the regex filter checks for.","marker":"[29]"},{"why":"The Eclipse and Mozilla defect tracking dataset is the main prior-art baseline that BugsRepo extends.","marker":"[23]"},{"why":"The multi-extract Mozilla issue-tracking dataset is another comparison point for dataset novelty.","marker":"[44]"},{"why":"Bugzilla REST API is the source used to mine bug metadata and comments.","marker":"[32]"},{"why":"Establishes what makes a good bug report, motivating the structured-report filter.","marker":"[45]"}],"fun_headline_variants":["Bug dataset pairs 119k Mozilla reports with 19k contributor profiles","Structured bug corpus: 10k reports plus metadata and contributor stats","Mozilla bug dataset: 119k reports, 19k committers, 10k clean","Triple-layer bug repo: metadata, comments, and contributor profiles","From 119k to 10k: curated Mozilla bug reports with context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if the regular expressions that detect section labels, combined with a CTQRS score above 75 percent borrowed from crowdsourced mobile test reports, do not actually identify well-structured Mozilla bug reports.","fun_headline_variants_meta":{"raw":{"variants":["Bug dataset pairs 119k Mozilla reports with 19k contributor profiles","Structured bug corpus: 10k reports plus metadata and contributor stats","Mozilla bug dataset: 119k reports, 19k committers, 10k clean","Triple-layer bug repo: metadata, comments, and contributor profiles","From 119k to 10k: curated Mozilla bug reports with context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2057,"prompt_tokens":994,"completion_tokens":1063,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":959}},"tokens_in":610,"tokens_out":1063,"duration_ms":8931,"temperature":1.0,"reasoning_tokens":959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:08:56.967943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Draw a random sample of 100 reports from the 10,351-report subset, have independent annotators judge whether each report truly contains clear steps to reproduce, actual behavior, and expected behavior, and compare those judgments against the CTQRS scores and the regex pass. A weak match would show the filter is not measuring what the paper says it measures.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CTQRS scoring framework and its 75 percent high-quality threshold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mozilla's bug writing guidelines define the S2R/AR/ER structure the regex filter checks for."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multi-extract Mozilla issue-tracking dataset is another comparison point for dataset novelty."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Bugzilla REST API is the source used to mine bug metadata and comments."}],"review_version":1}