{"id":"48742fe9-71b6-41a6-ac47-5b103de553ab","arxiv_id":"2507.18319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across seven projects and all issue types, BM25 beats TF-IDF, LSI, and the bug-specific rVSM, which performs worst.","lead":"This paper builds a dataset and evaluation pipeline for predicting which code files need to change to resolve any type of software issue, not just bugs. It finds that a classic bug-specific retrieval method underperforms generic methods, and that results vary strongly across projects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'bug-specific methods underperform' claim rests on one confounded method (rVSM) without ablation or significance testing.","rationale":"The label-noise threat identified by the reader is real and is acknowledged in Section 5.2, but it affects all compared retrieval methods roughly symmetrically and does not directly undermine the causal interpretation of the BM25-vs-rVSM comparison. The more load-bearing weakness is that the central claim attributes rVSM's poor performance specifically to its bug-specific file-length heuristic, even though rVSM differs from plain VSM in two coupled ways and the length factor is never isolated. The reader's rationale does mention that the headline 'rests largely on a single method (rVSM)', so there is partial agreement, but their formal weakest_assumption is label noise rather than this attribution problem. A controlled ablation and paired significance test would settle whether the conclusion is a real effect of bug-specific heuristics or an artefact of the particular rVSM formulation. Since the paper already receives a CONDITIONAL verdict, this concern does not move the verdict; it sharpens the condition under which acceptance should be granted.","tokens_in":19221,"tokens_out":7941,"duration_ms":78666,"concrete_test":"Run an ablation on the same temporal 50/50 split: (a) rVSM with the sigmoid length factor in Eq. 2 set to 1 (pure log-frequency cosine VSM); (b) BM25 with the same sigmoid factor applied to its final score; (c) paired bootstrap/Wilcoxon tests between BM25 and rVSM across the seven projects (and per issue) for P@1, MRR, and H@10. If removing the length prior does not change rVSM's ranking, or if adding it to BM25 does not hurt, the headline 'bug-specific heuristics perform poorly' is unsupported and should be narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—that bug-specific methods perform poorly on general issue types—is inferred from rVSM being the worst method in Table 3. But rVSM is a single method whose score differs from plain VSM in two coupled ways: it replaces tf/|D| with log(f(w,D)+1), and it multiplies cosine similarity by the file-length sigmoid in Eqs. 2–3. Only the second component encodes the bug-specific 'longer files are more likely to contain bugs' prior; the first is a general IR tweak. The paper never ablates the length factor, so the observed rVSM deficit could come from its older term weighting or from the lack of BM25's term-frequency saturation and field weighting, rather than from the bug-specific heuristic. No paired significance test is reported for the method ordering; Table 3 averages over seven projects with no confidence intervals or per-project significance. A single confounded comparison cannot support the plural claim 'methods designed using bug-specific heuristics perform poorly.' At minimum the conclusion should be bounded to 'rVSM performs worst on this dataset' until the length prior is isolated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a pipeline for constructing issue-to-file localisation datasets from arbitrary Git branching and merging workflows, a new dataset of seven Jira-hosted projects with unrestricted issue types, and an evaluation of traditional IR baselines (VSM, LSI-500, LSI-1000, rVSM, BM25). RQ1 compares the methods, RQ2 tests the effect of issue type, and RQ3 tests the correlation between the number of identifiers/file names and retrieval performance. The paper reports that BM25 is the best method and rVSM the worst, that issue-type effects are small but statistically significant, that identifier presence has a weak effect, and that results are strongly project-dependent.","tokens_in":19368,"tokens_out":5326,"duration_ms":58381,"significance":"If the empirical claims hold, the main contribution is a reusable dataset-construction pipeline that removes bug-only filtering and deliberately addresses merge-commit ambiguity, together with baseline results for general-issue file localisation. The replication package, the temporal split, the reporting of effect sizes, and the explicit threats-to-validity discussion are strengths. However, the headline claim that 'methods designed using bug-specific heuristics perform poorly' is currently supported only by the behaviour of a single confounded method, rVSM, and the statistical claims in RQ2/RQ3 need a multiple-comparison check. With those points addressed, the paper would be a useful and appropriately cautious exploratory study.","major_comments":[{"comment":"The conclusion that 'methods designed using bug-specific heuristics perform poorly on general issue types' (Abstract, RQ1 takeaways, and §5.1) is supported only by rVSM being the worst method in Table 3. rVSM differs from plain VSM in two coupled ways: it uses log(f(w,D)+1) instead of f(w,D)/|D|, and it multiplies the cosine similarity by the file-length sigmoid in Eqs. (2)-(3). Only the second modification encodes the bug-specific 'longer files are more likely to contain bugs' prior; the first is a general IR term-weighting choice. The paper does not ablate the length factor, so the observed deficit could be explained by rVSM's older term weighting or by the absence of BM25's term-frequency saturation and field weighting rather than by the bug-specific heuristic. Table 3 also reports only unweighted means over seven projects, with no paired significance test or confidence intervals for the method ordering. Please add an ablation isolating the length factor and a paired test across projects, or restrict the claim to 'rVSM performs worst on this dataset'.","section":"§4.1, §5.1, Table 3, Eqs. (2)-(3)"},{"comment":"RQ2 and RQ3 each perform a large number of statistical tests (seven projects times several metrics) at alpha = 0.05 without any multiple-comparison correction. Under this many tests, the observation that many correlations in Table 7 are 'almost always significant' is expected even under the null, and the same concern applies to the Kruskal-Wallis results in Table 5. The paper should apply a correction such as Benjamini-Hochberg, or explicitly label these analyses as exploratory and rely on effect sizes rather than raw p-values. This matters because 'small, but statistically significant differences' is stated as a finding in the abstract and conclusion.","section":"§3.7.2, §3.7.3, Tables 5-7"},{"comment":"The label construction assumes that all files changed in a commit are relevant to every issue mentioned in that commit's message, and that every mentioned issue is relevant to all changes in the commit. The paper acknowledges this in §5.2, which is commendable, but since the dataset is a core contribution and all performance numbers inherit this noise, the manuscript should include at least a sensitivity analysis, e.g., restricted to issues linked to exactly one commit, or a discussion of the likely direction of bias. Without such an analysis, the absolute performance numbers in Tables 3 and 4 should be read as noisy estimates rather than reliable benchmarks.","section":"§3.3.2, §5.2"}],"minor_comments":[{"comment":"There is a typo in 'documennt length'; it should read 'document length'.","section":"§3.5.4"},{"comment":"The phrase 'statically significant correlations' should be 'statistically significant correlations'.","section":"§4.3"},{"comment":"Reference [28] lists the second author as 'Raghaven'; the correct spelling is 'Raghavan'.","section":"References"},{"comment":"Please clarify whether Table 3 reports an unweighted mean over projects or a mean over all issues; because projects vary greatly in dataset size, the two averages can behave differently.","section":"Table 3"},{"comment":"The project abbreviation is spelled 'TomEE' in Table 5 and the text but 'Tomee' in Table 6; please make the spelling consistent.","section":"Tables 5 and 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper's durable contribution is the dataset and pipeline, not the headline about bug-specific heuristics. The dataset handles merge commits and arbitrary branching in a way that existing bug-localization datasets don't, and it covers seven Jira projects across languages and domains with no issue-type filter. That is genuinely useful for anyone working on file localization, especially as a seed-finder for LLM-based agents. The baseline evaluation is clean and simple: BM25 wins on every metric, rVSM comes last. The secondary analyses report effect sizes and stress project-dependence, which is honest.\n\nThe soft spot is the load-bearing interpretive claim. The abstract says 'methods designed using bug-specific heuristics perform poorly on general issue types.' That is too strong. rVSM is one method, and it differs from plain VSM in two ways: it swaps tf/|D| for log(f+1), and it adds the file-length sigmoid. Only the second is bug-specific. There is no ablation isolating the length prior, and no significance test for the ordering in Table 3, just averages over seven projects. So the rVSM result is a data point, not a demonstrated failure of bug-specific heuristics. The authors hedge in Section 5.1 ('may not generalise'), but the abstract and conclusion do not. That mismatch should be fixed.\n\nThe other weaknesses are more minor. Ground truth comes from commit-message mentions, so tangled commits and revert mentions add noise; the authors acknowledge this in Section 5.2. The project selection—seven Jira projects, 2k-5k issues, excluding projects where more than 70% were bugs—means 'at large' is bounded. And the Kruskal-Wallis tests run many project-metric combinations without multiple-comparison correction, though the effect sizes are small.\n\nBottom line: the dataset, pipeline, and baseline numbers are reproducible and worth having. This paper deserves a serious referee. I would send it to review with a request to either ablate rVSM's length prior or soften the claim to 'rVSM performs worst on this dataset.' As it stands, I would cite the dataset, not the conclusion about bug-specific methods.","headline":"The dataset and pipeline are a real contribution; the bug-specific-heuristics claim overreaches because it rests on one confounded method (rVSM) without ablation or significance testing.","tokens_in":19929,"tokens_out":2884,"would_cite":true,"duration_ms":27133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For unrestricted issue types, plain BM25 beats bug-tuned localisation methods, and the bug-specific rVSM ranks last.","keywords":["file localisation","issue tracking systems","information retrieval","BM25","bug localisation","dataset construction","Jira","empirical software engineering"],"falsifier":"Manually untangle a sample of the issue-linked commits in these seven projects and rerun the same retrieval methods on the cleaned ground truth; if rVSM matches or beats BM25 once false links from tangled and revert commits are removed, the paper's main conclusion would be overturned. A second check: recompute the identifier correlation after removing issues that already name the target file; if the correlation vanishes, the identifier effect is label leakage rather than a real performance signal.","tokens_in":18982,"feed_emoji":"🎯","tokens_out":8260,"duration_ms":78711,"temperature":0.7,"pith_summary":"The paper argues that file localisation should be studied for all issue types, not only bugs, and that methods tuned for bugs do not carry over to this broader task. To make that case, the authors built a new dataset of seven medium-sized open-source projects, linking commits to Jira issues without filtering by issue type, and evaluated five information-retrieval baselines against it. Their central finding is that BM25 outperforms every other method on every evaluation metric, while rVSM, a vector-space model modified with bug-specific heuristics such as file-length weighting, performs worst. Statistical tests show that issue type and the number of identifiers in an issue have small but often statistically significant effects, and that these effects vary from project to project. If the result holds, it implies that general-purpose localisation models, not bug-specialised ones, should be the target for future work.","feed_headline":"BM25 beats bug-tuned localisation on every issue type","feed_subtitle":"A seven-project study finds plain BM25 wins on every metric; bug-tuned heuristics such as rVSM rank last.","key_machinery":"The carrying mechanism is the dataset-construction pipeline. It links commits to issues by matching a project key in the first line of the commit message, keeps only issues whose linked commits all lie on one path from the repository root to the current head, disambiguates merge commits by checking whether the merged branch already contains issue-linked commits, and uses the files changed by the first linked commit as ground truth for a 'first commit only' variant of the localisation problem. Against this corpus, the paper compares five retrieval methods. BM25, here used with a multi-field extension that scores file name and file content as separate weighted fields, is the strongest baseline, while rVSM's added file-length term is the bug-specific component that the results identify as harmful.","core_discovery":"On a new dataset containing unrestricted issue types from seven projects, the standard BM25 ranking function achieves the best average performance on every reported metric (for example P@1 of 0.321, hit@5 of 0.573, hit@10 of 0.666, and MRR of 0.438), while rVSM, which adapts TF-IDF with a logistic file-length weighting intended for bugs, ranks last on every metric. The authors interpret this as evidence that bug-specific heuristics degrade performance when the task is localising files for issues in general, and that dimensionality reduction via LSI does not help either. They further find that issue type and the presence of identifiers and file names in the issue text have statistically significant but small effects on performance, with the direction and size of the effects depending strongly on the project. No single issue type is detrimental to overall performance, and identifiers help most for bugs and improvements, not for new features.","pith_inferences":["A natural extension is to test whether transformer-based or LLM rankers can beat BM25 on this unrestricted dataset; if they cannot, lexical term matching may be the practical ceiling for issue-to-file retrieval.","The pipeline's single-path requirement implicitly discards issues developed on parallel branches, which may bias the dataset toward linearly developed issues; quantifying what is excluded would clarify how far the results generalise.","The rVSM result could be probed by ablating its components individually, such as log term-frequency weighting versus file-length weighting, to isolate which bug-specific heuristic hurts most outside the bug setting.","For practitioners, the modest positive correlation between identifiers and performance suggests that prompting issue writers to mention file and symbol names could improve automated localisation, but the project-dependence of the effect means the gain is not guaranteed."],"forward_implications":["New file-localisation work targeting general issue types should use BM25 as its reference baseline rather than bug-tuned models.","Dataset builders should avoid assuming a GitHub pull-request workflow; the proposed pipeline admits arbitrary branching and merging, making it usable across Git workflows.","Evaluating only on bugs can overstate the value of bug-specific heuristics, so future benchmarks should include a spread of issue types.","Small but real project-dependent differences in issue-type and identifier effects imply that adaptable, per-project models may be needed rather than one universal ranking method.","The 'first commit only' variant gives a tractable starting point; the harder variants defined in the paper, predicting all future files or exact successive commits, are left as open problems."],"supporting_citations":[{"why":"Supplies rVSM, the bug-tuned model whose worst performance anchors the paper's central conclusion.","marker":"[49]"},{"why":"Defines the BM25 ranking function that the paper finds best on every metric.","marker":"[36]"},{"why":"Defines BM25F, the multi-field extension that lets BM25 score file name and content separately.","marker":"[35]"},{"why":"Provides the BM25+ lower-bounding term-frequency extension used in the experiments.","marker":"[26]"},{"why":"Introduces the bug-localisation dataset biases (misclassification, localised bugs, non-buggy files) that the paper re-tests for general issues.","marker":"[23]"},{"why":"Replicates the bias analysis and supplies the comparison for the identifier-effect results.","marker":"[40]"},{"why":"Provides the Jira issue dataset and preprocessing pipeline on which the new dataset is built.","marker":"[27]"},{"why":"Supplies the public Jira repository dataset from which the project selection was drawn.","marker":"[29]"}],"fun_headline_variants":["BM25 beats bug-tuned heuristics on every issue type","General IR wins: BM25 tops bug-specific rVSM","Bug-tuned localisation fails on general issues","Plain BM25 outperforms bug-adapted methods for all issues","Issue type agnostic: BM25 beats rVSM across the board"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labels assume that every file changed in a commit whose message mentions an issue is exactly a file that should be localised for that issue, and that all issues named in a commit share all of that commit's changes.","fun_headline_variants_meta":{"raw":{"variants":["BM25 beats bug-tuned heuristics on every issue type","General IR wins: BM25 tops bug-specific rVSM","Bug-tuned localisation fails on general issues","Plain BM25 outperforms bug-adapted methods for all issues","Issue type agnostic: BM25 beats rVSM across the board"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1513,"prompt_tokens":975,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":452}},"tokens_in":591,"tokens_out":538,"duration_ms":6395,"temperature":1.0,"reasoning_tokens":452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:14:34.229201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually untangle a sample of the issue-linked commits in these seven projects and rerun the same retrieval methods on the cleaned ground truth; if rVSM matches or beats BM25 once false links from tangled and revert commits are removed, the paper's main conclusion would be overturned. A second check: recompute the identifier correlation after removing issues that already name the target file; if the correlation vanishes, the identifier effect is label leakage rather than a real performance signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BM25+ lower-bounding term-frequency extension used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Jira issue dataset and preprocessing pipeline on which the new dataset is built."}],"review_version":2}