Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidden significant improvements in ~24% of previously non-significant tasks.
Robust benchmarking in noisy environments
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.
years
2026 4representative citing papers
AutoPass uses evidence from compiler states and runtime feedback to guide LLM agents in tuning LLVM optimizations, delivering 1.043x and 1.117x geometric-mean speedups over -O3 on x86-64 and ARM64.
Formalizes benchmarking as identifying the fastest program via consistent estimators of performance contrasts that cancel stateful biases under tenable assumptions.
Proposes a path-finding algorithm with A* as an alternative to Edmonds' blossom algorithm for computing minimal-weight-matching centrosymmetry parameter.
citing papers explorer
-
Rethinking Code Performance Benchmarks for LLMs
Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidden significant improvements in ~24% of previously non-significant tasks.
-
AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning
AutoPass uses evidence from compiler states and runtime feedback to guide LLM agents in tuning LLVM optimizations, delivering 1.043x and 1.117x geometric-mean speedups over -O3 on x86-64 and ARM64.
-
The Right Call for Software Benchmarking: Consistent Decisions in Stateful Environments
Formalizes benchmarking as identifying the fastest program via consistent estimators of performance contrasts that cancel stateful biases under tenable assumptions.
-
A path-finding algorithm for computing minimal-weight-matching centrosymmetry parameter
Proposes a path-finding algorithm with A* as an alternative to Edmonds' blossom algorithm for computing minimal-weight-matching centrosymmetry parameter.