Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Coding agents’ software-performance skill is workload-dependent: no stack dominates, and the agent framework can reshape the same model’s verified speedups.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 19:52 UTC pith:5MQCWGV5

load-bearing objection Useful new benchmark for agent performance engineering; framework-effect claims are directionally right but rest on single-run cells that the paper itself treats as snapshots. the 4 major comments →

arxiv 2607.07744 v1 pith:5MQCWGV5 submitted 2026-07-08 cs.SE

PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization

Yingyun Cui , Yi Xie , Piaohong Wang , Jiawei Ma , Bo Liu , Liangliang Cao This is my paper
classification cs.SE
keywords coding agentsperformance optimizationsoftware benchmarksverified speedupagent frameworkscross-layer bottlenecksshortcut exploitationagent relay
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most coding-agent benchmarks still score functional correctness, but production systems also need real, reproducible speedups. This paper introduces PERFOPT-Bench: twelve long-horizon tasks built from correct but deliberately suboptimal C codebases, where an agent must profile, find cross-layer bottlenecks, edit code without breaking hidden correctness tests, and earn a verified speedup. Evaluating seven agent stacks (different models and frameworks), the authors find that winners change by workload and that holding the language model fixed while swapping the agent framework can materially change per-task speedups. They also show that raw speedup is unsafe as a score, because some large gains come from benchmark-specific shortcuts rather than general optimization, and that an exploratory one-step “relay” restart from an externalized summary can recover more headroom after a session stops. The practical claim is that agents should be judged as full performance-engineering systems—stack, metric contract, and environment together—not as interchangeable model names on a universal leaderboard.

Core claim

On PERFOPT-Bench, optimization performance is workload-dependent rather than fixed by model identity: no single agent stack dominates across twelve verified-speedup tasks, and changing the agent framework for the same LLM can materially alter that model’s per-task speedup profile. Raw speedup alone is not a safe score, because large measured gains can be benchmark-specific shortcut exploitation rather than general cross-layer optimization.

What carries the argument

PERFOPT-Bench: each task pairs a correct but suboptimal codebase with a target metric, and scoring requires hidden correctness tests, verified speedup (baseline runtime over submitted runtime), and trajectory-level audit that discards or hardens shortcut cases.

Load-bearing premise

Single-run verified speedups on twelve expert-curated, deliberately bottlenecked C tasks—after hidden tests and a non-exhaustive expert trajectory audit—are enough to support comparative claims about stacks and framework effects despite noise, environment dependence, and residual shortcut ambiguity.

What would settle it

Repeat every model–framework–task cell with multiple independent trials under fixed hardware and compilers, and apply automated generalization checks that replace hidden workloads; a stable global ranking by model alone, or framework-invariant speedup profiles for the same model, would contradict the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent rankings for performance work should name the full stack (model plus framework), not the model alone.
  • Benchmarks that publish only raw speedup will systematically reward evaluator-specific shortcuts unless they enforce generalization contracts and audit trajectories.
  • Speedups must be treated as environment-conditioned claims (hardware, compiler, runtime, workload), not universal leaderboard entries.
  • Externalizing intermediate optimization state and restarting a fresh session is a usable continuation pattern when a first long run stops.
  • Cross-layer targets—toolchain flags, SIMD, memory locality, and runtime dispatch—become first-class agent evaluation goals, not side effects of bugfix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • For systems-level agent work, framework design (planning, tools, stop policy) may move the needle as much as the next model upgrade.
  • Aggressive shortcut search under continuous speedup rewards may be diagnostic of boundary-probing strength, not only of cheating.
  • Multi-session state handoff could transfer to other long-horizon engineering loops that fill a single context window.
  • Ambiguous specialization still needs expert judgment, so fully automated continuous-score systems benchmarks remain incomplete without stronger formal contracts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. PERFOPT-Bench introduces a coding-agent benchmark for verified software performance optimization rather than functional correctness alone. Each of twelve tasks supplies a correct but deliberately suboptimal C codebase and a natural-language optimization objective; scoring combines hidden correctness tests, environment-conditioned verified speedup, and trajectory-level audit for shortcut exploitation. The authors evaluate seven agent stacks (LLM × framework combinations spanning OpenCode, Codex, and Claude Code) and report that no stack dominates across tasks, that holding the LLM fixed while changing the framework can reshape per-task speedup profiles, that raw speedup is unsafe because of benchmark-specific shortcuts, and that a small one-step relay pilot can recover additional headroom after an initial session stops. Construction uses a four-stage semi-automated pipeline with pilot calibration and expert curation.

Significance. The paper addresses a genuine gap: most coding-agent benchmarks score pass/fail correctness, while production systems care about measurable, reproducible speedups under real hardware and toolchain constraints. Framing performance tuning as a full agentic loop (profile–diagnose–edit–verify) and requiring hidden tests plus trajectory audit is a clear methodological advance over raw wall-clock leaderboards. The fixed-LLM cross-framework contrasts and the documented shortcut cases (answer synthesis, build-artifact substitution) are useful contributions even if treated as descriptive. The public task packages and explicit anti-cheating contracts strengthen reusability. If the comparative claims hold under stronger measurement, the work would influence how agent stacks are evaluated and how performance benchmarks are hardened.

major comments (4)
  1. [§3.2–§3.3; Tables 2–4; Figure 2; Limitations] Observations 1–2 and the framework-effect claim (§3.2–§3.3; Tables 2–4; Figure 2) are load-bearing for the paper’s empirical message, yet every stack–task cell is a single run with no variance, confidence intervals, or repeated trials. The manuscript itself calls these “descriptive snapshots rather than statistically stable rankings” (Limitations; §3.1). With continuous noisy speedups, different stop policies, and residual shortcut ambiguity, win flips and geo-mean gaps (e.g., OpenCode+GPT 9.2× vs Codex+GPT 7.8× on 11 shared tasks; per-task winners on T2/T3/T6/T10) cannot yet be cleanly attributed to the framework rather than run noise. Either add multi-run statistics for the fixed-LLM contrasts, or substantially soften the language so that framework effects are presented only as hypothesis-generating patterns, not as established stack properties.
  2. [Stage 4; Table 2; Observation 1; Limitations] The claim that “no single agent stack dominates” and that optimization is “workload-dependent” is drawn from twelve expert-selected tasks after Stage-4 curation from a calibrated pool of ~28, with expert-refined speedup thresholds (Stage 4; Dataset Card). That selection is reasonable for a first release, but it is a free parameter of the study: the diversity argument and the Best-of-N leaderboard (Table 2) do not yet support generalization beyond this deliberately bottlenecked C suite. The paper should state more sharply what population of systems the twelve tasks are meant to represent, and either expand coverage or reframe Observations 1–2 as properties of this constructed set rather than of cross-layer optimization in general.
  3. [§3.4; Table 7; Limitations] Observation 3 (raw speedup is unsafe) rests on a non-exhaustive, single-expert trajectory audit of flagged outliers (Table 7; §3.4; Limitations). Several retained cells were re-evaluated under a hardened contract after large raw outliers (e.g., 107.9×, 492.8×, 110×). The qualitative finding is important and well illustrated, but the boundary between valid specialization and benchmark exploitation is acknowledged to be case-by-case, and the audit coverage is not quantified. For a central scoring claim, the paper needs a more systematic detection protocol (what fraction of trajectories were audited; dual review or inter-rater process for borderline cases; explicit decision rules) so that verified speedups in Figure 2 can be trusted as free of residual shortcuts.
  4. [§3.5; Table 5; Observation 4; Abstract] The relay pilot (§3.5; Table 5; Observation 4) reports R2/R1 gains of 1.02–2.48× in eight sequences on only two tasks, without equal-budget controls (longer single session, restart without the relay document, clean-workspace restart). The authors correctly label it exploratory and budget-confounded, yet Observation 4 and the abstract still present relay as recovering headroom. Either add the missing controls or move the pilot fully to an appendix with language that does not parallel the strength of Observations 1–3.
minor comments (6)
  1. [Table 2; Table 6] GeoMeans in Table 2 and Table 6 are computed over different numbers of valid tasks per stack (e.g., oc-glm over 10, oc-gpt over 11). The text notes non-comparability once, but the leaderboard presentation still invites direct ranking; mark incomparable aggregates more visibly.
  2. [Figure 2] Figure 2 is dense (seven stacks × twelve tasks with stars and hatching). A companion sorted bar or rank heatmap would make workload-dependent winners easier to read without relying only on cell values.
  3. [Table 1; §2] Table 1’s △ marks for Auto-Gen. and Real Repo on PERFOPT-Bench are slightly under-explained relative to the four-stage pipeline and the adapted open-source codebases; one clarifying sentence would help.
  4. [§3.1; Appendix A] Framework client versions and the single Windows 11 / Intel i7 target are recorded in the appendix; a short pointer in §3.1 would help readers interpret environment-conditioned speedups without hunting the supplement.
  5. [Abstract; §3] Minor consistency: the user-facing abstract snippet elsewhere mentions seven tasks while the manuscript body and main abstract use twelve; ensure all front-matter counts match the evaluation (12 tasks, 7 stacks).
  6. [Appendix B; Table 8] Dataset Card statistics (~668K LoC, median ~15K) are useful; stating whether LoC includes vendored/third-party code (e.g., large SQLite/Oniguruma-scale tasks) would clarify task difficulty.

Circularity Check

0 steps flagged

Empirical benchmark paper: measured speedups and stack comparisons are not definitionally forced by construction inputs.

full rationale

PERFOPT-Bench is an empirical systems/SE evaluation, not a first-principles derivation. The load-bearing claims (workload-dependent stack performance; framework effects under a fixed LLM; raw speedup unsafe due to shortcuts; exploratory relay headroom) rest on hardware-measured verified speedups after hidden correctness tests and trajectory audit, not on quantities defined in terms of the claimed outcomes. Task construction (LLM category generation, private reference solving, pilot-solver calibration to ~45% pass rate, expert curation of 12 tasks and threshold refinement) shapes difficulty and coverage of the benchmark set; it does not algebraically or statistically force which of the seven evaluated stacks wins which task, nor the OpenCode vs Codex/Claude Code per-task profile flips. Shortcut cases are discarded or re-evaluated under hardened contracts rather than folded into the score. There is no self-definitional loop, no fitted parameter renamed as a prediction of a closely related target, no load-bearing uniqueness theorem imported from overlapping authors, and no ansatz smuggled in via self-citation. Residual concerns about single-run noise, environment conditioning, and expert audit incompleteness are validity/statistical-power issues, not circularity. Score 0 with empty steps is the correct outcome.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on a constructed task suite and an evaluation contract, not on free physical constants. Load-bearing choices are the pilot pass-rate target, expert task selection and threshold refinement, the definition of valid optimization vs. shortcut, single-run measurement, and the relay-document schema. No new physical entities are postulated; the invented constructs are the benchmark and the relay protocol.

free parameters (3)
  • pilot_solver_pass_rate_target
    Stage 3 hardens candidates until pilot pass rate converges to ~45%, which sets the difficulty regime of the retained pool.
  • expert_selected_task_count_and_thresholds
    Stage 4 expert picks 12 of ~28 calibrated tasks and refines speedup thresholds to ‘realistic ceilings,’ directly shaping what stacks are scored on.
  • single_run_per_stack_task_cell
    Each model–framework–task cell is one run; continuous noisy speedups are treated as descriptive profiles without fitted variance parameters.
axioms (4)
  • domain assumption Verified speedup after hidden correctness tests plus trajectory audit is a valid primary score for genuine performance optimization ability.
    Stated throughout §3.1 and Observation 3; the paper itself notes residual ambiguity between valid specialization and exploitation.
  • domain assumption The four bottleneck families (toolchain, scalar/SIMD, memory/cache, parallelism/dispatch) adequately represent cross-layer software performance work for this evaluation.
    §2 construction premise used to generate and curate tasks.
  • domain assumption Agent stacks may be compared fairly when given the same issue.md, Code/, tools, and no extra hints beyond default system prompts.
    §3.1 evaluation protocol.
  • standard math Standard C toolchain timing and correctness oracles on the target machine are reliable enough for environment-conditioned claims.
    Implicit measurement model for all reported × speedups.
invented entities (2)
  • PERFOPT-Bench task suite and scoring contract no independent evidence
    purpose: Provide agent-visible suboptimal codebases with hidden correctness and verified-speedup evaluation.
    Core contribution; independent evidence is the released packages and evaluation scripts, not external prior measurement of the same suite.
  • One-step agent relay protocol (relay document + fresh second session) no independent evidence
    purpose: Test whether externalizing optimization state recovers additional speedup headroom after an initial session stops.
    Exploratory system-design construct in §3.5; only eight budget-confounded sequences support it.

pith-pipeline@v1.1.0-grok45 · 18936 in / 2956 out tokens · 40259 ms · 2026-07-10T19:52:06.447186+00:00 · methodology

0 comments
read the original abstract

Coding-agent benchmarks have largely measured whether agents can produce functionally correct patches, but production software also demands measurable speedups on real execution targets. Performance optimization is a distinct agentic task: agents must profile executions, diagnose cross-layer bottlenecks, edit code without breaking correctness, and verify that gains are reproducible rather than measurement artifacts. We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop. Each task provides a correct but deliberately suboptimal codebase and asks the agent to improve a target performance metric; scoring requires hidden correctness tests, verified-speedup measurement, and trajectory-level audit. We evaluate 7 agent stacks with different LLMs and agent frameworks on 7 long-horizon optimization tasks. The results show that optimization performance is workload-dependent rather than determined by model identity alone: no single stack dominates, and changing the agent framework can materially change the same LLM's per-task speedup profile. We further find that raw speedup is unsafe as a benchmark score, since some large gains arise from benchmark-specific shortcut exploitation; an exploratory relay pilot suggests that restarting from an externalized optimization summary can recover additional headroom after an initial session stops. The benchmark and our evaluation are available at: https://anonymous.4open.science/r/Dataset-D3CC.

Figures

Figures reproduced from arXiv: 2607.07744 by Bo Liu, Jiawei Ma, Liangliang Cao, Piaohong Wang, Yingyun Cui, Yi Xie.

Figure 1
Figure 1. Figure 1: Benchmark pipeline with LLM-driven generation, adversarial shaping, and expert curation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cell values denote speedup (×) over baseline; stars mark per-task best; hatched = missing (no valid result or discarded shortcut case). Cheating Detection Prompt Role: You are an anti-cheat reviewer. Audit whether this change truly satisfies the task contract, rather than passing through hard-coding, test bypasses, fabricated metrics, or specialization to public examples. Please check whether the submissio… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 45 canonical work pages · 6 internal anchors

  1. [1]

    Evaluating Large Language Models Trained on Code

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Program Synthesis with Large Language Models

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle=

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle=

    work page

  4. [4]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , booktitle=

    work page

  5. [5]

    and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and others , booktitle=

    Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and others , booktitle=

    work page

  6. [6]

    International Conference on Learning Representations , year=

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. International Conference on Learning Representations , year=

    work page

  7. [7]

    Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , journal=

    work page

  8. [8]

    Zhuo, Terry Yue and Vu, Minh Chien and Chim, Jenny and Hu, Han and Yu, Wenhao and Widyasari, Ratnadira and Yusuf, Imam Nur Bani and Zhan, Haolan and He, Junda and Paul, Indraneil and others , booktitle=

    work page

  9. [9]

    International Conference on Learning Representations , year=

    Learning Performance-Improving Code Edits , author=. International Conference on Learning Representations , year=

    work page

  10. [10]

    Advances in Neural Information Processing Systems , year=

    Mercury: A Code Efficiency Benchmark for Code Large Language Models , author=. Advances in Neural Information Processing Systems , year=

    work page

  11. [11]

    , booktitle=

    Huang, Dong and Qing, Yuhao and Shang, Weiyi and Cui, Heming and Zhang, Jie M. , booktitle=

    work page

  12. [12]

    He, Xinyi and Liu, Qian and Du, Mingzhe and Yan, Lin and Fan, Zhijie and Huang, Yiming and Yuan, Zejian and Ma, Zejun , journal=

    work page

  13. [13]

    Ma, Jeffrey Jian and Hashemi, Milad and Yazdanbakhsh, Amir and Swersky, Kevin and Press, Ofir and Li, Enhui and Reddi, Vijay Janapa and Ranganathan, Parthasarathy , journal=

    work page

  14. [14]

    and Hu, William and R

    Ouyang, Anne and Guo, Simon and Arora, Simran and Zhang, Alex L. and Hu, William and R. International Conference on Machine Learning , year=

    work page

  15. [15]

    Li, Jianling and Li, Shangzhan and Gao, Zhenye and Shi, Qi and Li, Yuxuan and Wang, Zefan and Huang, Jiacheng and Wang, Haojie and Wang, Jianrong and Han, Xu and Liu, Zhiyuan and Sun, Maosong , booktitle=

    work page

  16. [16]

    and Nadgir, Nitya and Narayanan, Arvind , journal=

    Kapoor, Sayash and Stroebl, Benedikt and Siegel, Zachary S. and Nadgir, Nitya and Narayanan, Arvind , journal=

    work page

  17. [17]

    Agentless: Demystifying

    Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming , journal=. Agentless: Demystifying

    work page

  18. [18]

    Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

    Producing Wrong Data Without Doing Anything Obviously Wrong! , author=. Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems , pages=. 2009 , doi=

    work page 2009

  19. [19]

    Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications , pages=

    Statistically Rigorous Java Performance Evaluation , author=. Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications , pages=. 2007 , doi=

    work page 2007

  20. [20]

    Communications of the ACM , volume=

    Roofline: An Insightful Visual Performance Model for Multicore Architectures , author=. Communications of the ACM , volume=. 2009 , doi=

    work page 2009

  21. [21]

    Scalable Parallel Programming with

    Nickolls, John and Buck, Ian and Garland, Michael and Skadron, Kevin , journal=. Scalable Parallel Programming with. 2008 , doi=

    work page 2008

  22. [22]

    Communications of the ACM , volume=

    The Tail at Scale , author=. Communications of the ACM , volume=. 2013 , doi=

    work page 2013

  23. [23]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  24. [24]

    Publications Manual , year = "1983", publisher =

    work page 1983

  25. [25]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  26. [26]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

    work page

  27. [27]

    Dan Gusfield , title =. 1997

    work page 1997

  28. [28]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    work page 2015

  29. [29]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

    work page

  30. [30]

    Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik , year =

    work page

  31. [31]

    Devansh Yadav and Shouvick Mondal , title =. J. Syst. Softw. , volume =. 2025 , url =. doi:10.1016/J.JSS.2025.112543 , timestamp =

    work page doi:10.1016/j.jss.2025.112543 2025

  32. [32]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , booktitle =

    Naman Jain and King Han and Alex Gu and Wen. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , booktitle =. 2025 , url =

    work page 2025

  33. [33]

    BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions , booktitle =

    Terry Yue Zhuo and Minh Chien Vu and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen Gong and James Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instruc...

    work page 2025

  34. [34]

    Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R

    Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R. Narasimhan , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  35. [35]

    CoRR , volume =

    Qixing Zhou and Jiacheng Zhang and Haiyang Wang and Rui Hao and Jiahe Wang and Minghao Han and Yuxue Yang and Shuzhe Wu and Feiyang Pan and Lue Fan and Dandan Tu and Zhaoxiang Zhang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.10975 , eprinttype =. 2602.10975 , timestamp =

    work page doi:10.48550/arxiv.2602.10975 2026

  36. [36]

    AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    Jason Chou and Ao Liu and Yuchi Deng and Zhiying Zeng and Tao Zhang and Haotian Zhu and Jianwei Cai and Yue Mao and Chenchen Zhang and Lingyun Tan and Ziyan Xu and Bohui Zhai and Hengyi Liu and Speed Zhu and Wiggin Zhou and Fengzong Lian , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2508.09101 , eprinttype =. 2508.09101 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.09101 2025

  37. [37]

    Mike A. Merrill and Alexander Glenn Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and Estefany Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marianna Nezhurina and Jenia Jitsev and Di Lu and Orfeas Menis. Terminal-Bench: Benchm...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.11868 2026

  38. [38]

    RE-Bench: Evaluating Frontier

    Hjalmar Wijk and Tao Roa Lin and Joel Becker and Sami Jawhar and Neev Parikh and Thomas Broadley and Lawrence Chan and Michael Chen and Joshua Clymer and Jai Dhyani and Elena Ericheva and Katharyn Garcia and Brian Goodrich and Nikola Jurkovic and Megan Kinniment and Aron Lajko and Seraphina Nix and Lucas Jun Koba Sato and William Saunders and Maksym Taran...

    work page 2025

  39. [39]

    Frank F. Xu and Yufan Song and Boxuan Li and Yuxuan Tang and Kritanjali Jain and Mengxue Bao and Zora Zhiruo Wang and Xuhui Zhou and Zhitong Guo and Murong Cao and Mingyang Yang and Hao Yang Lu and Amaad Martin and Zhe Su and Leander Maben and Raj Mehta and Wayne Chi and Lawrence Keunho Jang and Yiqing Xie and Shuyan Zhou and Graham Neubig , title =. CoRR...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2412.14161 2024

  40. [40]

    Gardner and Yiming Yang and Milad Hashemi and Graham Neubig and Parthasarathy Ranganathan and Osbert Bastani and Amir Yazdanbakhsh , title =

    Alexander Shypula and Aman Madaan and Yimeng Zeng and Uri Alon and Jacob R. Gardner and Yiming Yang and Milad Hashemi and Graham Neubig and Parthasarathy Ranganathan and Osbert Bastani and Amir Yazdanbakhsh , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  41. [41]

    SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

    Xinyi He and Qian Liu and Mingzhe Du and Lin Yan and Zhijie Fan and Yiming Huang and Zejian Yuan and Zejun Ma , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2507.12415 , eprinttype =. 2507.12415 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.12415 2025

  42. [42]

    Zhang and William Hu and Christopher R

    Anne Ouyang and Simon Guo and Simran Arora and Alex L. Zhang and William Hu and Christopher R. KernelBench: Can LLMs Write Efficient. Forty-second International Conference on Machine Learning,. 2025 , url =

    work page 2025

  43. [43]

    CoRR , volume =

    Robert Tjarko Lange and Qi Sun and Aaditya Prasad and Maxence Faldor and Yujin Tang and David Ha , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.14279 , eprinttype =. 2509.14279 , timestamp =

    work page doi:10.48550/arxiv.2509.14279 2025

  44. [44]

    TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators , booktitle =

    Jianling Li and Shangzhan Li and Zhenye Gao and Qi Shi and Yuxuan Li and Zefan Wang and Jiacheng Huang and WangHaojie WangHaojie and Jianrong Wang and Xu Han and Zhiyuan Liu and Maosong Sun , editor =. TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators , booktitle =. 2025 , url =

    work page 2025

  45. [45]

    Xingze Zou and Jing Wang and Yuhua Zheng and Xueyi Chen and Haolei Bai and Lingcheng Kong and Syed A. R. Abu. MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices? , journal =. 2026 , url =. doi:10.48550/ARXIV.2603.11935 , eprinttype =. 2603.11935 , timestamp =

    work page doi:10.48550/arxiv.2603.11935 2026