Pith. sign in

REVIEW 4 major objections 6 minor 77 references

EffiHolmes: Differential Profiling-Guided Repository Level Time Inefficiency Fix Localization

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EffiHolmes claims that running a program at two workload scales, tracing the hot path, and then asking an LLM to reason upstream finds where to fix slow code better than retrieval, agent, and profiling baselines on every benchmark scenario.

desk verdict EffiHolmes is a well-built profiling-based localization pipeline with a genuinely useful new benchmark, but the headline comparison with static baselines is confounded because the method gets to run the reproducer and the baselines do not. read the letter →

arxiv 2608.03558 v1 pith:3V4WVEI3 submitted 2026-08-04 cs.SE

Haowen Yang , Yun Peng , Zishuo Ding This is my paper · ORCID
classification cs.SE
keywords repository-levelperformanceoptimizationtimeinefficiencydifferentialprofilingfixlocalizationLLMfault
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that repository-level time-inefficiency issues can be localized to their fix locations even when they produce no test failures or stack-traces, by running the same reproduction under a default and a scaled workload, extracting the execution paths linking the resulting hotspots to the entry point, and guiding an LLM with domain heuristics to reason upstream from symptoms to causes. Single-run profiling drowns inefficiency hotspots in execution noise, the paper argues, while differential profiling reveals which functions grow disproportionately with input scale. Because RepoEffi-Bench contains 140 real developer-reported and developer-fixed issues, the framework's best-or-tied-best result in all 18 evaluation scenarios is presented as evidence that the approach transfers beyond toy settings. A sympathetic reader would care because slow-but-correct code is a major source of cost and operational failure, and no previously existing fault-localization paradigm applies to it.

What carries the argument

The load-bearing mechanism is the differential trace pair and its derived normalized differential contribution (NDC). For each function node $n$, EffiHolmes measures $\Delta_{\mathrm{self}}(n)$ as scaled-execution self-time minus default-execution self-time, and $\mathrm{NDC}(n)=\Delta_{\mathrm{self}}(n)/\sum_m \Delta_{\mathrm{self}}(m)$; the top-$k$ nodes by NDC are the inefficiency hotspots. Those hotspots anchor a lower bound, while the reproduction script's entry function anchors an upper bound, and the framework enumerates entry-to-hotspot paths, compresses them by structural call-chain identity, and keeps the representative path with the highest cumulative differential self-time. That compact set of critical paths, together with a hybrid code context (full source for short functions, signature-plus-call-lines for long ones) and a guided-reasoning prompt that favors control-layer functions and requires source-code validation, is what carries the LLM from profiler symptom to fix location.

What would settle it

Take a held-out sample of time-inefficiency fixes that live in native code behind Python wrappers or in thin wrapper functions that profilers attribute entirely to built-ins, and run EffiHolmes on them; if its accuracy collapses to retrieval-baseline levels, the claim that differential execution-path evidence suffices to bridge hotspots and fix locations is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a differential-pair profiling pipeline can close the semantic gap between where time accumulates and where a fix belongs. EffiHolmes profiles the same reproduction under two workload scales, ranks functions by normalized differential self-time ($\mathrm{NDC}(n)=\Delta_{\mathrm{self}}(n)/\sum_m \Delta_{\mathrm{self}}(m)$), reconstructs compressed entry-to-hotspot call-chain paths, and prompts an LLM to prioritize upstream control-layer logic while validating candidates against source-code evidence and six measured signals. On RepoEffi-Bench this yields the best or tied-best result in all 18 scenarios (6 metrics × 3 models), including 4.29 percentage-point gains on GPT-5.1 file-level Acc@3 and 15.00 percentage-point gains on qwen3-4b function-level Acc@5 over the strongest baseline.

Load-bearing premise

The measured accuracy transfers to the real world of repository-level time inefficiency issues, because the 140 instances in RepoEffi-Bench are a representative and unbiased sample of that world rather than a sample shaped by the filtering pipeline.

Editorial extensions

If this is right

  • A 4B-parameter model using EffiHolmes reaches 83.57% file-level Acc@5 and 70.71% function-level Acc@5, matching or beating baselines that run on models with roughly 8× more parameters.
  • Execution-path extraction is the primary source of the gain: ablating it drops function-level Acc@3 by 39.29 percentage points on GPT-5.1.
  • Differential profiling isolates complexity growth that single-run profiles cannot show, so the approach works even when the inefficient operation is tiny relative to background noise.
  • Guided reasoning and execution paths are complementary: removing both drops function-level Acc@1 to 22.86%, below the least effective single-component ablation.
  • Function-level localization, the harder developer-facing task, is where EffiHolmes' advantage over retrieval-based methods is largest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the differential workload-scaling idea is language-agnostic and likely extends to memory or I/O inefficiencies, since any resource whose consumption grows superlinearly with input scale can be substituted for wall-clock self-time.
  • Beyond the paper: the benchmark's filtering stages—LLM classification with no audit of negatives, plus requiring reproducible slowdowns via provided test cases—may make RepoEffi-Bench easier than uncurated issue streams; applying EffiHolmes to non-reproducible or test-less issues would be a stricter check.
  • Beyond the paper: because the pipeline only ranks candidates on extracted paths, its recall ceiling is set by profiler visibility; adding static taint or native-code instrumentation could recover the off-path fixes the paper identifies as failure modes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EffiHolmes, an LLM-based framework for localizing the fix locations of repository-level time inefficiency issues. The framework constructs a differential test pair (default vs. scaled workload), profiles both executions with VizTracer, identifies hotspots by differential self-time and normalized differential contribution, extracts and compresses execution paths that connect the entry point to the hotspots, and then uses a guided-reasoning prompt with hybrid code context to rank candidate fix locations. The paper also introduces RepoEffi-Bench, a benchmark of 140 real-world Python inefficiency issues derived from popular data-science repositories, and reports that EffiHolmes outperforms retrieval-, agent-, procedure-, and profiling-based baselines in all 18 metric-by-model evaluation scenarios, with headline gains of 4.29 pp on GPT-5.1 file-level Acc@3 and 15.00 pp on qwen3-4b function-level Acc@5 over the strongest baselines.

Significance. If the results hold, the paper addresses a genuinely underexplored problem: repository-level inefficiency fix localization without stack traces or failing tests. The differential profiling idea is well motivated by the motivating example, and the paper has clear strengths: a reproducible artifact link, an ablation study isolating the contributions of execution paths and guided reasoning, a failure analysis with concrete issue examples, and a benchmark that is more repository-level and leakage-conscious than prior function-level or synthetic benchmarks. The central pipeline is clearly specified. However, several evaluation-protocol issues currently prevent the reported superiority from being interpreted as clean evidence for the framework's localization logic rather than for its access to reproduction artifacts or its benchmark construction choices.

major comments (4)
  1. [§3.2.1, §4.2, §4.3] The evaluation may be confounded by unequal access to reproduction artifacts. Section 3.2.1 says the scaled target is inferred 'from the baseline script and the issue report,' and Section 4.2 states that RepoEffi-Bench retains only cases where the slowdown is reproduced and the speedup verified 'using the provided test cases.' Section 4.3, however, says agent and procedure baselines receive only the repository and the cleaned issue description. If EffiHolmes is given the developer-provided test case or reproduction script while baselines are not, the reported gains (e.g., the 15.00 pp function-level Acc@5 gain over Direct Aggregated Profiling on qwen3-4b) include the effect of differential test-pair construction, not just the profiling-guided reasoning. The ablation in Table 2 does not resolve this, because all ablations retain the differential traces. Please state explicitly what reproduction artifacts each method receives, and add a control in which baselines receive the same test case/script or in which EffiHolmes is run without them.
  2. [§4.3, Table 1] The Direct Aggregated Profiling baseline is not specified. It is the strongest baseline on qwen3-4b and the comparison point for the paper's headline 15.00 pp gain, yet Section 4.3 describes only retrieval-, procedure-, and agent-based baselines; the profiling-based category is introduced only in Table 1 and RQ1. The paper must define how this baseline obtains per-function runtime statistics (which profiler, which prompt template, which model, what post-processing), otherwise the key comparison is not reproducible and its fairness cannot be assessed.
  3. [§3.2.1] The success rate of differential test pair generation is not reported. The pipeline uses a bounded retry budget of five rounds, but the paper never states how often the LLM-inferred scaling target fails to produce a valid pair, what happens when the budget is exhausted, or whether such instances are removed from the evaluation. If failed instances are dropped, the reported Acc@k values are conditional on successful pair generation and overstate the method's coverage. Please report the per-model generation success rate and re-run the evaluation counting generation failures as misses.
  4. [§4.2] The RepoEffi-Bench construction may systematically select for issues that are amenable to the EffiHolmes pipeline. Starting from 1,594 issue-PR pairs, the dataset keeps only 140 cases where the slowdown at the base commit and speedup at the fix commit could be reproduced using the provided test cases; the paper also notes that LLM-negative cases were not manually audited. Because EffiHolmes itself requires a baseline script and a differentially scalable workload, the benchmark could be enriched for exactly the conditions the method needs, limiting external validity. Please provide an analysis of the dropped 1,454 cases (e.g., reasons for exclusion, whether they lack test cases or do not scale) and discuss how this affects the generalizability of the reported accuracies.
minor comments (6)
  1. [Table 1] The EffiHolmes row for qwen3-4b contains a stray '0.00' between the File Acc@1 and File Acc@3 values; the table should be corrected so that File Acc@3 is reported consistently.
  2. [Figure 3] Figure 3 shows 'Test Case' as an input to the pipeline, while Section 3.1 states that the framework takes only the issue report and repository as inputs; the role of the test case/reproduction script in the pipeline should be stated unambiguously.
  3. [§4.3 and Table 3] CoSIL is listed under agent-based methods in Section 4.3 but appears as a procedure-based method in Table 3; the categorization should be made consistent.
  4. [§3.2.2 and §3.4.1] The symbol k is used both for the number of selected hotspots and for the line threshold for full code context; these are different parameters and should be renamed to avoid confusion.
  5. [Table 1] Many reported margins are very small (e.g., 0.71 pp, which is one instance out of 140), but no confidence intervals or significance tests are given; the claim of consistent superiority across all 18 scenarios would be strengthened by statistical analysis or at least a sensitivity discussion.
  6. [Tables 3 and 4] The '(NoReason)' variants of EffiHolmes and the baselines are not defined in the text; the paper should state exactly which reasoning component is removed and why baselines are also evaluated in this mode.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found; the central claim is empirical, and the benchmark-alignment, tuning, and self-citation concerns are external-validity issues rather than definitional circularity.

full rationale

EffiHolmes is an empirical pipeline rather than a derivation, so the classic circularity failure modes largely do not apply. The differential hotspot signal is computed directly from profiler self-time differences (Eqs. 1-2) and is independent of the ground-truth PR diff; path reconstruction anchors only on the reproduction-script entry point and the top-NDC hotspots; and the guided reasoning stage ranks candidates by structural evidence such as hotspot coverage, path coverage, entry proximity, and differential self-time. None of these stages defines the fix location as an input, and no parameter fitted from the ground-truth labels is subsequently reported as a localization prediction. The RepoEffi-Bench inclusion criterion that a case must reproduce the slowdown at the base commit and verify the speedup at the fix commit is an evaluation-scope constraint applied uniformly; it may create a favorable setting for profiling-based approaches, but it does not make EffiHolmes's output equal to the benchmark label by construction. The self-citations in the paper, such as [49] and [62], support peripheral choices (prior benchmark limitations and the prevalence of performance bugs in data-science libraries) and are not load-bearing; no author-uniqueness theorem is imported to force the design. Section 7 explicitly concedes that an off-path fix location cannot be recovered because the guided reasoning stage ranks candidates drawn from the extracted paths, and Section 4.2 concedes that LLM-negative issue candidates were not manually audited; both are honest external-validity and benchmark-curation limitations, not circular reductions. One minor non-circular concern is that the context-length setting k in {20,60,100} is selected on RepoEffi-Bench itself and then used in the final reported numbers, which is a tuning-on-test leakage risk rather than a definitional circularity. Therefore no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

EffiHolmes introduces no new physical or conceptual entities; its contribution is a new pipeline. The free parameters are mostly design choices with modest impact, but the top-k hotspot count and the inferred scaling target are not fully documented, which limits reproduction. The axioms are standard empirical assumptions for profiling-based localization, and none of them is ad-hoc to the point of circularity.

free parameters (4)
  • top-k hotspots
    The paper selects the top-k function nodes by NDC as hotspots, but does not state the value of k used in the experiments. This parameter affects which paths are extracted and therefore which candidates the LLM sees.
  • line threshold k for full code context = 100
    Section 3.4.1 compares k in {20, 60, 100} and chooses k=100. This is a hyperparameter selected based on validation performance.
  • retry budget = 5 rounds
    Section 3.2.1 bounds the differential test pair generation to five rounds. This is a hand-chosen budget that affects whether a valid scaled workload is produced.
  • scaling target inference
    When no explicit scaling cue exists, the LLM infers what to scale. This is an implicit free parameter: the quality of the inferred scaling target directly determines whether differential profiling exposes the hotspot.
assumptions (4)
  • domain assumption VizTracer's timing and hierarchical trace are accurate enough to reflect true algorithmic behavior under scaled workloads.
    Section 3.2.1 assumes the trace profiler's low overhead means observed performance differences reflect algorithmic behavior rather than tracing distortion.
  • domain assumption The differential-hotspot ranking by normalized self-time difference is a sufficient signal to identify the relevant inefficiency.
    Section 3.2.2 defines hotspots by NDC and uses them as the lower bound of the search space, implicitly assuming that the inefficiency manifests as self-time growth in some function node.
  • domain assumption The merged PR's functional code changes, verified to improve speed, are the correct ground-truth fix location.
    Section 4.2 derives ground truth from merged PRs, assuming the code changes directly address the inefficiency and that no other change would be equally valid.
  • domain assumption The LLM (GPT-5.1 or Qwen3) reasons correctly about control flow and source semantics when given structured runtime evidence.
    The guided reasoning stage depends on the LLM following the prompt's heuristics and interpreting source code accurately; this is an empirical assumption, not a formal guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EffiHolmes: Differential Profiling-Guided Repository Level Time Inefficiency Fix Localization." pith.science (2026). https://pith.science/paper/3V4WVEI3

@misc{pith2026260803558,
  author       = {Pith},
  title        = {Pith review of: EffiHolmes: Differential Profiling-Guided Repository Level Time Inefficiency Fix Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3V4WVEI3}},
  note         = {Machine review of arXiv:2608.03558}
}
read the original abstract

Large software systems often suffer from time inefficiencies that cause excessive execution time despite functional correctness. Localizing their fix locations is difficult because, unlike functional bugs, they produce neither test failures nor stack-trace clues, making traditional and recent LLM-based fault localization methods unsuitable. Runtime profiling provides alternative evidence but faces three challenges in repository-level settings: single-run profiling cannot reliably distinguish inefficiency hotspots from execution noise; existing profilers struggle to extract relevant execution paths from extensive background execution; and a semantic gap remains between observed hotspots and actual fix locations. We propose EffiHolmes, an LLM-based framework for repository-level time inefficiency fix localization. EffiHolmes uses differential profiling under default and scaled workloads to identify inefficiency hotspots, extracts compact execution paths connecting these hotspots to the reported inefficient function, and employs domain-guided LLM reasoning to locate the underlying inefficiency logic. We also introduce RepoEffi-Bench, the first benchmark for repository-level inefficiency localization, containing 140 high-quality issues collected from popular Python repositories. Experiments show that EffiHolmes consistently outperforms state-of-the-art retrieval-, agent-, and profiling-based baselines, improving file-level Acc@3 by 4.29 percentage points with GPT-5.1 and function-level Acc@5 by 15.00 percentage points with qwen3-4b. It also remains robust across model capacities.

Figures

Figures reproduced from arXiv: 2608.03558 by the authors.

Figure 1
Figure 1. Single-Run Profiling vs Differential Profiling. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Motivating Example of EffiHolmes. slower than in previous versions. We inspected the results of SOTA agent-based issue localization methods [11, 16, 32] and standard profiling techniques. None successfully identified the correct fix location. We analyze the failure modes of these approaches below. Failure of Agent-Based Methods. We traced the trajectory of SOTA agent-based method on this issue. As shown in [PITH_FU… view at source ↗
Figure 3
Figure 3. Overview of EffiHolmes. The framework takes a repository and an issue description as input. It generates a differential trace pair to identify hotspots, extracts and compresses critical execution paths, and guides the LLM to locate the fix location. computations and validate candidates against structural evidence, ensuring the search converges on the defective logical decision. 3 The EffiHolmes Framework 3.1 Overvie… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 28 canonical work pages

  1. [1]

    Issue 24840 in Scikit-learn

    n.d.. Issue 24840 in Scikit-learn. https://github.com/scikit- learn/scikit- learn/issues/24840. Accessed: [2025.12.17]

    work page 2025

  2. [2]

    Issue 9692 in Xarray

    n.d.. Issue 9692 in Xarray. https://github.com/pydata/xarray/issues/9692. Accessed: [2025.12.17]

    work page 2025

  3. [3]

    Pokémon Go makers call for calm as servers crash across Europe and US

    n.d.. Pokémon Go makers call for calm as servers crash across Europe and US. https://www.theguardian.com/technology/2016/jul/16/pokemon-go-server- crash-niantic-europe-us. Accessed: [2023.12.17]

    work page 2016

  4. [4]

    Scikit-learn

    n.d.. Scikit-learn. https://github.com/scikit-learn/scikit-learn. Accessed: [2025.12.17]

    work page 2025

  5. [5]

    Stop-loss orders are not stop-loss, ensuring a bad price and risk users capital unnecessarily

    n.d.. Stop-loss orders are not stop-loss, ensuring a bad price and risk users capital unnecessarily. https://github.com/freqtrade/freqtrade/issues/988. Accessed: [2026-01-17]

    work page 2026

  6. [6]

    Taylor Swift’s ’Midnights’ album crashes Spotify, leaving fans shocked; nearly 8,000 outages reported

    n.d.. Taylor Swift’s ’Midnights’ album crashes Spotify, leaving fans shocked; nearly 8,000 outages reported. https://www.foxbusiness.com/entertainment/ taylor-swifts-midnights-album-crashes-spotify-fans-shocked-8000-outages- reported. Accessed: [2023.12.17]

    work page 2023

  7. [7]

    n.d.. Xarray. https://github.com/pydata/xarray. Accessed: [2025.12.17]

    work page 2025

  8. [8]

    Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. InTesting: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, 89–98. doi:10.1109/TAIC.PART.2007.13

    doi:10.1109/taic.part.2007.13 2007

Show all 77 references
  1. [9]

    R. F. Abreu, P. Zoeteweij, and A. J. C. van Gemund. 2009. Spectrum-based Multi- ple Fault Localization. In24th International Conference on Automated Software Engineering (ASE’09), G. Taentzer and M. Heimdahl (Eds.). IEEE, Los Alamitos, CA, USA, 88–99. doi:10.1109/ASE.2009.25

    2009 doi

  2. [10]

    Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. 2024. SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement.arXiv preprint arXiv:2410.20285(2024). arXiv:2410.20285 doi:10.48550/arXiv.2410.20285

    2024 doi arXiv

  3. [11]

    Aorwall. 2025. Moatless Tools. https://github.com/aorwall/moatless-tools GitHub repository

    2025

  4. [12]

    Sebastian Baltes, Oliver Moseler, Fabian Beck, and Stephan Diehl. 2015. Navi- gate, understand, communicate: How developers locate performance bugs. In 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–10. doi:10.1109/ESEM....

    2015

  5. [13]

    Jianming Chang, Xin Zhou, Lulu Wang, David Lo, and Bixin Li. 2026. Bridging Bug Localization and Issue Fixing: A Hierarchical Localization Framework Leveraging Large Language Models.IEEE Transactions on Software Engineering52, 4 (2026), 1531–1548. doi:10.1109/TSE.2026.3668601

    2026

  6. [14]

    Junjie Chen, Yihua Liang, Qingchao Shen, Jiajun Jiang, and Shuochuan Li. 2023. Toward understanding deep learning framework bugs.ACM Transactions on Software Engineering and Methodology32, 6 (2023), 1–31. doi:10.1145/3587155

    2023 doi

  7. [15]

    Zimin Chen, Sen Fang, and Martin Monperrus. 2024. Supersonic: Learning to generate source code optimizations in C/C++.IEEE Transactions on Software Engineering50, 11 (2024), 2849–2864. doi:10.1109/TSE.2024.3423769

    2024

  8. [16]

    Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. 2025. Locagent: Graph- guided llm agents for code localization. InProceedings of the 63rd Annual Meeting ASE ’26, October 12–16, 2026, Munich, Germany Ha...

    2025

  9. [17]

    Luca Della Toffola, Michael Pradel, and Thomas R Gross. 2015. Performance problems you can fix: A dynamic analysis of memoization opportunities.ACM SIGPLAN Notices50, 10 (2015), 607–622. doi:10.1145/2858965.2814290

    2015

  10. [18]

    Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models.Advances in Neural Information Processing Systems37 (2024), 16601–16622. doi:10.52202/079017-0529

    2024 doi

  11. [19]

    Huerta, and Hao Peng

    Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A. Huerta, and Hao Peng. 2025. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. InFindings of the Association for Computationa...

    2025

  12. [20]

    Andreas Florath. 2023. LLM Interactive Optimization of Open Source Python Libraries – Case Studies and Generalization.arXiv preprint arXiv:2312.14949 (2023). arXiv:2312.14949 doi:10.48550/arXiv.2312.14949

    2023 doi arXiv

  13. [21]

    Freqtrade Developers. 2023. Freqtrade. https://github.com/freqtrade/freqtrade

    2023

  14. [22]

    Tian Gao and VizTracer Contributors. 2024. VizTracer: A Low-Overhead Log- ging/Tracing/Profiling Tool for Python. https://github.com/gaogaotiantian/vizt racer. Accessed: 2026-01

    2024

  15. [23]

    Spandan Garg, Roshanak Zilouchian Moghaddam, Colin B Clement, Neel Sun- daresan, and Chen Wu. 2022. Deepdev-perf: a deep learning-based approach for improving software performance. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on th...

    2022

  16. [24]

    Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2025. Rapgen: An approach for fixing code inefficiencies in zero-shot. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 124–135. doi:10.1...

    2025

  17. [25]

    Luca Gazzola, Daniela Micucci, and Leonardo Mariani. 2019. Automatic Software Repair: A Survey.IEEE Transactions on Software Engineering45, 1 (2019), 34–67. doi:10.1109/TSE.2017.2755013

    2019

  18. [26]

    Rafail Giavrimis, Michail Basios, Fan Wu, Leslie Kanthan, and Roman Bauer. 2025. Artemis ai: Multi-llm framework for code optimisation. In2025 IEEE Conference on Artificial Intelligence (CAI). IEEE, 1–6. doi:10.1109/CAI64502.2025.00057

    2025

  19. [27]

    Rafail Giavrimis, Alexis Butler, Constantin Cezar Petrescu, Michail Basios, and Santanu Kumar Dash. 2021. Genetic optimisation of C++ applications. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1180–1182. doi:10.1109/ASE51524.2021.9678650

    2021

  20. [28]

    Jingzhi Gong, Rafail Giavrimis, Paul Brookes, Vardan Voskanyan, Fan Wu, Mari Ashiga, Matthew Truscott, Mike Basios, Leslie Kanthan, Jie Xu, and Zheng Wang

  21. [29]

    Xue Han and Tingting Yu. 2016. An empirical study on performance bugs for highly configurable software systems. InProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10. doi:10.1145/2961111.2962602

    2016

  22. [30]

    Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, and Zejun Ma. 2025. SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?. InNeurIPS 2025 Fourth Workshop on Deep Learning for Code. https://arxiv.org/abs/2...

    2025 arXiv

  23. [32]

    Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu

  24. [33]

    Carlos E Jimenez, John Yang, Alexander Wettig, et al. 2023. SWE-bench: Can Lan- guage Models Resolve Real-World GitHub Issues?arXiv preprint arXiv:2310.06770 (2023). arXiv:2310.06770 doi:10.48550/arXiv.2310.06770

    2023 doi arXiv

  25. [34]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Ef- fibench: Benchmarking the efficiency of automatically generated code.Advances in Neural Information Processing Systems37 (2024). doi:10.52202/079017-0367

    2024 doi

  26. [35]

    Jones and Mary Jean Harrold

    James A. Jones and Mary Jean Harrold. 2005. Empirical Evaluation of the Taran- tula Automatic Fault-Localization Technique. InProc. of the 20th IEEE/ACM In- ternational Conference on Automated Software Engineering (ASE). ACM, 273–282. doi:10.1145/1101908.1101949

    2005 arXiv

  27. [36]

    In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    Issue Localization via LLM-Driven Iterative Code Graph Searching. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3034–3045. doi:10.1109/ASE63991.2025.00249

    2025

  28. [37]

    Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. InProceedings of the 25th interna- tional symposium on software testing and analysis. doi:10.1145/2931037.2931051

    2016

  29. [38]

    Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012. Understanding and detecting real-world performance bugs.ACM SIGPLAN Notices47, 6 (2012), 77–88. doi:10.1145/2254064.2254075

    2012

  30. [39]

    Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. InProceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 169–180. doi:10.1145/3293882.3330574

    2019

  31. [40]

    Sungmin Kang, Gabin An, and Shin Yoo. 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization.Proceedings of the ACM on Software Engineering1, FSE (2024), 1424–1446. doi:10.1145/3660771

    2024 doi

  32. [41]

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating Language Models for Efficient Code Generation. InFirst Conference on Language Modeling. https://openreview.net/forum?id=IBCBMe AhmC

    2024

  33. [42]

    Rahul Krishna, Md Shahriar Iqbal, Mohammad Ali Javidian, Baishakhi Ray, and Pooyan Jamshidi. 2020. CADET: Debugging and Fixing Misconfigura- tions Using Counterfactual Reasoning.arXiv preprint arXiv:2010.06061(2020). arXiv:2010.06061 doi:10.48550/arXiv.2010.06061

    2020 doi arXiv

  34. [43]

    Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2024. An Empirical Study on LLM-based Agents for Automated Bug Fixing.CoRRabs/2411.10213 (2024). arXiv:2411.10213 doi:10.48550/arXiv.2411.10213

    2024 doi

  35. [45]

    Adrian Nistor, Tian Jiang, and Lin Tan. 2013. Discovering, reporting, and fixing performance bugs. In2013 10th working conference on mining software repositories (MSR). IEEE, 237–246. doi:10.1109/MSR.2013.6624035

    2013

  36. [46]

    Yiling Lou, Qihao Zhu, Jinhao Dong, Xia Li, Zeyu Sun, Dan Hao, Lu Zhang, and Lingming Zhang. 2021. Boosting coverage-based fault localization via graph- based representation learning. InProceedings of the 29th ACM joint meeting on european software engineering conference and s...

    2021

  37. [47]

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2024. RepoGraph: Enhancing AI Software Engineering with Repository-Level Code Graph.CoRRabs/2410.14684 (2024). arXiv:2410.14684 doi:10.48550/arXiv.2410.14684

    2024 doi arXiv

  38. [48]

    Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014. Ask the mutants: Mutating faulty programs for fault localization. In2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 153–162. doi:10.1109/ICST.2014.28

    2014 doi

  39. [49]

    Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation.Proceedings of the ACM on Software Engineering 2, FSE (2025), 242–265. doi:10.1145/3715727

    2025 doi

  40. [50]

    Oswaldo Olivo, Isil Dillig, and Calvin Lin. 2015. Static detection of asymptotic performance bugs in collection traversals. InProceedings of the 36th ACM SIG- PLAN Conference on Programming Language Design and Implementation. 369–378. doi:10.1145/2737924.2737966

    2015

  41. [51]

    Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. 2025. SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization.IEEE Transactions on Software Engineering51, 4 (2025), 1173–1187. doi:10.1109/TSE.2025.3543187

    2025

  42. [52]

    Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: Mutation-Based Fault Localization.Software Testing, Verification and Reliability25, 5–7 (2015), 605–628. doi:10.1002/STVR.1509

    2015 doi

  43. [53]

    Stefano Rando, Luca Romani, Alessio Sampieri, Luca Franco, John Yang, Yuta Kyuragi, Fabio Galasso, and Tatsunori Hashimoto. 2025. LongCodeBench: Evalu- ating Coding LLMs at 1M Context Windows. InSecond Conference on Language Modeling. https://openreview.net/forum?id=GFPoM8Ylp8

    2025

  44. [54]

    Python Software Foundation. 2024. cProfile — Deterministic Profiling of Python Programs. https://docs.python.org/3/library/profile.html. Accessed: 2026-01

    2024

  45. [55]

    S. Reis, R. Abreu, and M. d’Amorim. 2019. Demystifying the Combination of Dynamic Slicing and Spectrum-based Fault Localization. InProceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19). International Joint Conferences on Artifici...

    2019

  46. [56]

    Asif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Has- san, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, and Dim- itrios S Nikolopoulos. 2025. MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Compu...

    2025 doi arXiv

  47. [57]

    Xingyao Wang, Boxuan Li, Yufan Song, et al. 2024. OpenHands: An Open Platform for AI Software Developers as Generalist Agents.arXiv preprint arXiv:2407.16741 (2024). arXiv:2407.16741 doi:10.48550/arXiv.2407.16741

    2024 doi arXiv

  48. [58]

    Raselimo and B

    M. Raselimo and B. Fischer. 2019. Spectrum-based Fault Localization for Context- Free Grammars. InProceedings of the 12th ACM SIGPLAN International Conference on Software Language Engineering (SLE 2019). Association for Computing Ma- chinery, New York, NY, USA, 15–28. doi:10.1...

    2019

  49. [59]

    W. E. Wong, V. Debroy, R. Gao, and Y. Li. 2014. The DStar Method for Effective Software Fault Localization.IEEE Transactions on Reliability63, 1 (2014), 290–308. doi:10.1109/TR.2013.2285319

    2014

  50. [60]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents.Proceedings of the ACM on Software Engineering2, FSE (2025), 801–824. doi:10.1145/3715754

    2025 doi

  51. [61]

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2023. Learning Performance-Improving Code Edits.arXiv preprint arXiv:2302.07867(2023). arXiv:2302.07867 doi:10....

    2023 doi arXiv

  52. [62]

    Haowen Yang, Zhengda Li, Zhiqing Zhong, et al. 2025. Towards Understanding Performance Bugs in Popular Data Science Libraries.Proceedings of the ACM on Software Engineering2, FSE (2025), 2335–2358. doi:10.1145/3729374

    2025 doi

  53. [63]

    Max Weber et al. 2021. White-box performance-influence models: A profiling and learning approach. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1059–1071. doi:10.1109/ICSE43902.2021.00099 EffiHolmes: Differential Profiling-Guided Repositor...

    2021

  54. [64]

    Junwen Yang, Pranav Subramaniam, Shan Lu, Cong Yan, and Alvin Cheung

  55. [65]

    Tong Ye, Weigang Huang, Xuhong Zhang, Tengfei Ma, Peiyu Liu, Jianwei Yin, and Wenhai Wang. 2025. LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctness.arXiv preprint arXiv:2502.18489(2025). arXiv:2502.18489 doi:10.48550/arXiv.2502.18489

    2025 doi arXiv

  56. [66]

    Chuyang Xu, Zhongxin Liu, Xiaoxue Ren, Gehao Zhang, Ming Liang, and David Lo. 2025. Flexfl: Flexible and effective fault localization with open-source large language models.IEEE Transactions on Software Engineering51, 5 (2025), 1455–

    2025

  57. [67]

    Shahed Zaman, Bram Adams, and Ahmed E. Hassan. 2012. A Qualitative Study on Performance Bugs. InProceedings of the 9th IEEE Working Conference on Mining Software Repositories (MSR 2012). IEEE, 199–208. doi:10.1109/MSR.2012.6224281

    2012

  58. [68]

    Zhang, M

    L. Zhang, M. Kim, and S. Khurshid. 2011. Localizing Failure-inducing Program Edits Based on Spectrum Information. In2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, 23–32. doi:10.1109/ICSM.2011.6080769

    2011

  59. [69]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. InAdvances in Neural Infor- mation Processing Systems 37. 50528–50652. doi:10.522...

    2024 doi

  60. [70]

    Yutong Zhao, Lu Xiao, Andre B Bondi, et al. 2023. A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source Projects.IEEE Transactions on Software Engineering49, 2 (2023), 924–946. doi:10.1109/TSE.2022.3167628

    2023

  61. [71]

    Yutong Zhao, Lu Xiao, Xiao Wang, Lei Sun, Bihuan Chen, Yang Liu, and Andre B Bondi. 2020. How are performance issues caused and resolved?-an empirical study from a design perspective. InProceedings of the ACM/SPEC International Conference on Performance Engineering. 181–192. d...

    2020

  62. [72]

    Yuwei Zhao, Yuan-An Xiao, Qianyu Xiao, Zhao Zhang, and Yingfei Xiong. 2026. SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis.ACM Trans- actions on Software Engineering and Methodology(2026). doi:10.1145/3820167 Received 2026-03-26; accepted 2026-06-18

    2026 doi

  63. [73]

    Zhongming Yu, Hejia Zhang, Yujie Zhao, Hanxian Huang, Matrix Yao, Ke Ding, and Jishen Zhao. 2025. OrcaLoca: An LLM Agent Framework for Software Issue Localization. InInternational Conference on Machine Learning. PMLR, 73416– 73436

    2025

  64. [76]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604. doi:10.1145/3650212.3680384

    2024

  65. [1471]

    doi:10.1109/TSE.2025.3553363

    2025

  66. [2018]

    InProceedings of the 40th International Conference on Software Engineering

    How not to structure your database-backed web applications: a study of performance bugs in the wild. InProceedings of the 40th International Conference on Software Engineering. 800–810. doi:10.1145/3180155.3180194

  67. [2025]

    doi:10.18653/v1/2025.findings-emnlp.1264

    23281–23298. doi:10.18653/v1/2025.findings-emnlp.1264

    2025 doi

  68. [4766]

    doi:10.24963/IJCAI.2019/661

    2019 doi

  69. [8727]

    doi:10.18653/v1/2025.acl-long.426

    2025 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.