{"id":"3b6207b1-098f-413f-9b11-fdfa35031365","arxiv_id":"2508.12232","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM agent with lazy access to commit history, issue threads, and code links issues to resolving commits, reporting Hit@1 of 0.86 to 0.89 and outperforming published baseline scores.","lead":"LinkAnchor is a new AI agent that uses a large language model to find the commit that fixes a software issue, letting the model request project data on demand. It reported higher hit rates than prior machine-learning methods, without any task-specific training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison is not apples-to-apples: LinkAnchor's Hit@1 uses redefined ground truth and whole-repo search, while baseline scores are published under the original protocol; claimed 60-262% improvements are unsubstantiated without re-running baselines.","rationale":"The paper's central claim is the 60–262% Hit@1 improvement over state-of-the-art ILR approaches. For this claim to hold, the comparison must use identical evaluation protocols. The paper openly changes the ground truth for LinkAnchor (to the latest linked commit) and uses a whole-repository search, while taking baseline numbers from published papers that used a different ground truth and a 100-candidate ranking setup. This is a concrete, textually verifiable flaw in the comparative methodology. The reader's weakest_assumption identifies exactly this issue, and I agree it is the most load-bearing concern. I considered data leakage (the LLM may have seen RQ1 issues during pre-training) as a competing threat, but the protocol mismatch is immediate and provable from the paper itself, whereas leakage, while plausible, is a hypothesis that would require additional testing. The recommended remedy—re-running baselines under the adjusted protocol—would settle the concern. The reader's CONDITIONAL verdict remains appropriate because the absolute results on RQ2 and the availability of a replication package leave room for the claim to hold after re-evaluation. Hence, no change to the verdict is needed.","tokens_in":18338,"tokens_out":7440,"duration_ms":84313,"concrete_test":"Re-run EALink (and, if available, T-BERT, DeepLink, VSM) on the same EALink dataset but with the adjusted ground truth used for LinkAnchor: for each issue, set the target to the latest linked commit. Use the same candidate-construction protocol as LinkAnchor (i.e., the set of commits in the safe lifespan, or a random sample of 100 commits that includes the target). Compute Hit@1 as the fraction of issues where the top-ranked candidate equals the adjusted target. Compare these recomputed scores to LinkAnchor's reported Hit@1. If LinkAnchor's advantage over EALink shrinks below 60% or reverses, the claimed superiority is not supported under a consistent protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.A.1 redefines the ground truth for LinkAnchor as the latest linked commit per issue, but reports baseline Hit@1 scores from EALink/T-BERT/DeepLink/VSM that were computed under EALink's original protocol: one true link per issue selected from the dataset and ranked within a 100-commit candidate set containing that true link. If the original positive instance differs from the latest commit (as it will for any issue with multiple linked commits), baseline Hit@1 is measured against a different target than LinkAnchor's. The paper does not recompute baselines under the adjusted ground truth. Furthermore, LinkAnchor's 'Hit@1' is the fraction of issues for which its single predicted commit equals the adjusted ground truth over the entire (safe-lifespan) commit set, not a ranking over a curated 100-commit candidate set with the answer guaranteed to be present. The paper asserts in Section V.A.2 that Hit@1 'allows for a fair comparison,' but this is unsupported; the two tasks differ in candidate construction, search space, and ground-truth definition. Consequently, the headline claim of outperforming SOTA by 60–262% is not established. The load-bearing assumption is that published baseline scores remain valid when the evaluation protocol is altered; this is the least secure link in the central comparative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LinkAnchor is an LLM-based agent for issue-to-commit link recovery. Given an issue URL and a Git repository, it lets an LLM invoke specialized functions to query commit history, issue threads, and code, finally returning a single commit hash. The paper evaluates LinkAnchor on six Apache projects from the EALink dataset (Table II) and on 120 recently resolved GitHub issues (Table V), reporting mean Hit@1 of 0.8632 and 0.89 respectively, and claims 60–262% improvement over four baselines. The approach is training-free, uses a lazy-access architecture to address context-window limits, and formulates ILR as a search problem rather than pairwise classification.","tokens_in":18708,"tokens_out":6547,"duration_ms":65297,"significance":"If the reported results hold, LinkAnchor would be a practical, training-free ILR approach that outperforms supervised methods while exploiting previously unused signals such as issue comments and codebase inspection. The modular design, public replication package, and three-run averaging for LLM variability are strengths. RQ2's use of post-training-cutoff issues is a thoughtful attempt to avoid data leakage. However, the headline comparative claim rests on a protocol mismatch in baseline evaluation, and the RQ1 sample sizes and RQ2 label construction are underspecified. These issues must be resolved before the significance of the claimed improvements can be assessed.","major_comments":[{"comment":"The comparison is not apples-to-apples. LinkAnchor's 'Hit@1' is the fraction of issues for which its single predicted commit equals the latest linked commit in the safe-lifespan commit set. The reported baseline scores come from EALink (and its baselines), where Hit@1 is measured by ranking a curated 100-commit candidate set guaranteed to contain exactly one true link. These differ in target definition (latest linked commit vs. any linked commit), candidate construction, and search space. The statement that Hit@1 'allows for a fair comparison' is unsupported. The 60–262% improvements in Table II are not established unless baselines are re-run under the adjusted ground truth and search protocol.","section":"Section V.A.2 / Table II"},{"comment":"The paper never states how many issues were run through LinkAnchor per project in RQ1. It refers to the full EALink dataset ('over 59k unique issues') and then reports Hit@1 proportions. Without the denominator and sampling method per project, the reader cannot judge whether a score of 0.79 on Netbeans is based on 10 or 1,000 issues. Please report the number of issues per project, the selection procedure, and ideally confidence intervals or per-run ranges. This also affects the cost analysis in Table VI, which appears to be based on a subset.","section":"Section V.A.1 / Table II"},{"comment":"RQ1 uses the EALink dataset, whose issues and commits almost certainly predate the October 2023 training cutoff of ChatGPT-4o-nano. Since the LLM was pretrained on public GitHub/Jira data, it may have memorized some issue-commit links, which would inflate LinkAnchor's scores relative to the supervised baselines. RQ2 explicitly avoids this risk by using post-October-2023 issues, but the main comparison in Table II does not. Please either replace RQ1 with post-cutoff data or analyze the leakage risk (e.g., by comparing performance across time splits) and discuss the effect on the claimed improvements.","section":"Section V.A.1"},{"comment":"The construction of the RQ2 ground truth is under-specified. How was the 'true resolving commit' determined for each of the 120 issues? Possibilities include GitHub's explicit closing-link metadata, commit-message keyword matching, or manual inspection; each yields different labels and thus different success rates. The paper also does not state how 'resolved after October 2023' was verified or how the 20 issues per project were sampled. Please provide the exact labeling procedure and dataset construction code.","section":"Section V.B.1"}],"minor_comments":[{"comment":"Inconsistent naming: 'T-Bert' vs. 'T-BERT', 'ChatGPT-4o-nano' vs. 'Chat-GPT4o-nano', 'EA-Link' vs. 'EALink', 'TREE-SITTER' vs. 'Tree-sitter', and 'ignite' (Table III) vs. 'Ignite' elsewhere. Please standardize.","section":"Throughout"},{"comment":"Equation (1) defines Hit@k on a candidate set Q, but LinkAnchor's metric is a success rate over issues, not a rank among 100 candidates. Suggest renaming LinkAnchor's metric (e.g., 'success@1') or explicitly describing it as an adapted metric to avoid confusion.","section":"Eq. (1) / Section V.A.2"},{"comment":"The sentence 'The number we report for LinkAnchor is calculated by dividing the number of correct commit hashes by the total number of issues available in the dataset' conflicts with the cost analysis, which suggests a subset was actually used. Please clarify the exact evaluation set.","section":"Section V.A.2"},{"comment":"The EALink time comparison (29,101 × 0.126s) assumes EALink scores every commit per issue; if EALink uses candidate generation or pruning, this comparison is misleading. Also, per-issue time for LinkAnchor is compared to per-1000-pairs scoring time for EALink, which is not a direct apples-to-apples comparison.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the baseline comparison protocol mismatch. Re-running EALink, T-BERT, DeepLink, and VSM under LinkAnchor's adjusted ground truth and whole-repo search space may be expensive but is necessary to support the claimed 60–262% improvements. If the authors cannot re-run the baselines, they should restrict LinkAnchor to the original 100-candidate protocol or substantially weaken the comparative claims. The RQ1 sample-size omission and leakage risk are also serious but fixable. The central idea is promising and worth another round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is, as far as I can tell, the first paper to apply an LLM agent to issue-to-commit link recovery, and the absolute results look solid. On EALink's six Apache projects, LinkAnchor gets a mean Hit@1 of 0.86; on 120 GitHub issues resolved after the model's training cutoff, it gets 0.89. The architecture does real work: deterministic, verifiable function calls over commit history, issue threads, and codebase, with pagination and feedback pruning to stay inside the token budget. The authors run each experiment three times because LLM output is nondeterministic, report per-issue cost and latency, and ship code and a replication package. A training-free tool at this absolute accuracy is a useful result in a niche but practical corner of software traceability.\n\nThe soft spot is the one the reader flagged, and I confirm it on reading the full text. For a sensible reason - an issue is often resolved by a chain of commits, so the final commit is the right target - the paper redefines ground truth as the latest linked commit per issue. But the baseline scores are the published numbers from EALink, T-BERT, DeepLink, and VSM, computed under the original protocol: any true link counts, and it must be ranked inside a 100-commit candidate set guaranteed to contain the answer. LinkAnchor instead searches the safe-lifespan window and must output the exact final hash, with no such guarantee. The stress-test note is right, with one correction: the search space is the safe-lifespan window, not literally the whole repository - but that does not rescue the comparison, because the two tasks still differ in target definition and candidate construction. The net bias is unknown, and the paper's assertion that Hit@1 'allows for a fair comparison' is asserted, not argued. So the headline 60-262% improvement is not established; the abstract's range also silently drops the 714% reported for DeepLink in Table II. To be fair, the comparison against baselines' Hit@10, though also protocol-mismatched, is a harsher test for LinkAnchor, and it still comes out ahead on five of six projects; that is decent evidence the qualitative result survives a fair re-run. The paper needs to do that re-run.\n\nSmaller gaps: RQ1 never states how many issues per project were actually run; RQ2's construction of the true resolving commits for the 120 GitHub issues is under-specified; and the give-up rate is never reported, so we don't know how often the agent quits or hits the iteration cap.\n\nWho this is for: people working on traceability, maintenance tooling, or LLM agents for software engineering. It deserves a serious referee - the approach is new, the engineering is transparent, and the main flaw is fixable. I would send it out and push hard on the baseline question.","headline":"Genuinely new LLM-agent application for issue-to-commit linking with strong absolute accuracy, but the 60-262% claim over SOTA is not established until baselines are re-run under the redefined ground truth.","tokens_in":19124,"tokens_out":9239,"would_cite":true,"duration_ms":93329,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LinkAnchor, a training-free LLM agent, recovers issue-to-commit links with a mean Hit@1 of 0.8632 on six Apache projects, outperforming state-of-the-art methods by 60–262%.","keywords":["issue-to-commit link recovery","LLM agent","software traceability","lazy-access architecture","function calling","commit resolution","GitHub issues","Jira"],"falsifier":"Take EALink's six-project dataset, relabel each issue's ground truth to the latest linked commit (as the paper does), and rerun EALink, T-BERT, DeepLink, and VSM on those labels; if any baseline's Hit@1 reaches or exceeds LinkAnchor's 0.8632 mean, the claimed 60–262% improvements collapse.","tokens_in":18292,"feed_emoji":"🔗","tokens_out":6153,"duration_ms":60416,"temperature":0.7,"pith_summary":"LinkAnchor addresses issue-to-commit link recovery, the task of finding which commit actually resolved a given issue. The paper argues that prior methods degrade because they must fit all context into a fixed window or score every issue-commit pair in isolation, ignoring chains of commits and issue-discussion clues. LinkAnchor instead lets a general-purpose LLM act as an autonomous search agent: it invokes functions to pull only the relevant commits, comments, and code on demand. The paper reports that LinkAnchor achieves a mean Hit@1 of 0.8632 across six Apache projects, a 60–262% improvement over four state-of-the-art baselines, and 89% accuracy on 120 unseen GitHub issues, all without any training. If these numbers hold, agent-based lazy retrieval is a practical, setup-free alternative to trained link-recovery models.","feed_headline":"LLM agent finds resolving commits with 86% top-1 accuracy","feed_subtitle":"No training, no feature engineering: on-demand retrieval from commits, issue threads, and code beats prior methods.","key_machinery":"The load-bearing mechanism is a lazy-access agent architecture: LinkAnchor exposes 20 deterministic functions in Git, Issue, Codebase, and Control categories that a general-purpose LLM can call iteratively. Git functions provide paginated commit lists filtered by author, file, or time, with a unified branch view and safe-lifespan filtering (one week before issue creation to one week after resolution); Codebase functions use a Tree-sitter AST to fetch definitions, docs, or line ranges at any commit; Issue functions expose title, description, comments, and participants; Feedback lets the model discard unhelpful large outputs to conserve tokens. This machinery reframes link recovery from exhaus","core_discovery":"On the paper's own terms, the central discovery is that an LLM equipped with a small set of read-only functions can solve issue-to-commit link recovery by searching rather than by classifying. LinkAnchor redefines the target as the latest linked commit per issue, and its agent searches the repository's commit history, issue discussion, and code through 20 deterministic functions. It reports a mean Hit@1 of 0.8632 across six Apache projects, with per-project scores between 0.79 and 0.95, and a mean of 0.89 on 120 GitHub issues resolved after the model's training cutoff. The paper attributes the gains to access to previously untapped data sources, especially issue comments and code inspection,","pith_inferences":["A natural stress test is to run LinkAnchor against an open-weight local LLM: if performance holds, the approach no longer depends on a specific API's pricing or training cutoff.","The paper's ground-truth redefinition (latest linked commit per issue) is reasonable but unilaterally applied; if prior baselines were recomputed on those labels, the size of the reported improvements could change, and a fair comparison would require releasing the relabeled dataset.","The same lazy-access agent pattern could plausibly transfer to other trace-link tasks, such as requirements-to-code or bug-report-to-fix, since the limiting factor in those tasks is also context-window capacity rather than the scoring model."],"forward_implications":["LinkAnchor can be applied to a new repository immediately, with no training data, feature engineering, or GPU infrastructure, and reports a median cost of about $0.01 and 23 seconds per issue.","The approach succeeds on projects where prior methods degrade (e.g., Ignite and NetBeans), suggesting that dynamic data selection rather than fixed features is the source of stability.","Because all function responses are deterministic, every step of the agent's search is auditable against the true repository state.","The same architecture extends beyond Jira and GitHub to any platform with a data extractor, and to any Tree-sitter-supported language."],"supporting_citations":[{"why":"Supplies the dataset of 59k issues across six Apache projects and is the strongest baseline; the paper reuses this dataset and compares directly to it.","marker":"[23]"},{"why":"Reports the 42.2% issue-commit link rate that motivates the problem and provides the DeepLink baseline.","marker":"[7]"},{"why":"Provides the T-BERT baseline, a BERT-based traceability model pre-trained on CodeSearchNet.","marker":"[21]"},{"why":"Provides the VSM baseline, an information-retrieval method used as a lower-bound comparison.","marker":"[24]"},{"why":"Tree-sitter parser that underlies the codebase functions, enabling language-agnostic retrieval of definitions and docs.","marker":"[25]"}],"fun_headline_variants":["LLM agent recovers issue-commit links without training","86% top-1 accuracy: LLM agent searches, not classifies","LinkAnchor: LLM agent finds resolving commits via lazy access","No training, just searching: LLM agent beats prior link recovery","Agentic LLM recovers GitHub issue-commit links with 86% Hit@1"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the published Hit@1 scores of the four prior methods remain valid when the ground truth is redefined as the latest linked commit per issue, even though those scores were never recomputed under that definition.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent recovers issue-commit links without training","86% top-1 accuracy: LLM agent searches, not classifies","LinkAnchor: LLM agent finds resolving commits via lazy access","No training, just searching: LLM agent beats prior link recovery","Agentic LLM recovers GitHub issue-commit links with 86% Hit@1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1261,"prompt_tokens":821,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":565,"tokens_out":440,"duration_ms":4523,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:34:08.664432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take EALink's six-project dataset, relabel each issue's ground truth to the latest linked commit (as the paper does), and rerun EALink, T-BERT, DeepLink, and VSM on those labels; if any baseline's Hit@1 reaches or exceeds LinkAnchor's 0.8632 mean, the claimed 60–262% improvements collapse.","supporting_citations":[{"cited_title":"Ealink: An efficient and accurate pre-trained framework for issue-commit link recovery,","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 59k issues across six Apache projects and is the strongest baseline; the paper reuses this dataset and compares directly to it."},{"cited_title":"Deeplink: Recovering issue- commit links based on deep learning,","cited_arxiv_id":null,"evidence_quote":"Reports the 42.2% issue-commit link rate that motivates the problem and provides the DeepLink baseline."},{"cited_title":"Traceability transformed: Generating more accurate links with pre-trained bert mod- els,","cited_arxiv_id":null,"evidence_quote":"Provides the T-BERT baseline, a BERT-based traceability model pre-trained on CodeSearchNet."}],"review_version":2}