CodeSpecBench shows LLMs achieve at most 20.2% pass rate on repository-level executable behavioral specification generation, revealing that strong code generation does not imply deep semantic understanding.
UTB oost: Rigorous evaluation of coding agents on SWE -bench
3 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
3
Pith papers citing it
1
external citations · OpenAlex
fields
cs.SE 3years
2026 3representative citing papers
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
An empirical study finds that injecting call/inheritance topology as comments improves LLM code agent localization by 2.2pp, shortens trajectories by 1.6 rounds, and halves run-to-run variance on medium repositories via a deterministic anchoring effect.
citing papers explorer
-
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
CodeSpecBench shows LLMs achieve at most 20.2% pass rate on repository-level executable behavioral specification generation, revealing that strong code generation does not imply deep semantic understanding.
-
What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
-
How Much Static Structure Do Code Agents Need? A Study of Deterministic Anchoring
An empirical study finds that injecting call/inheritance topology as comments improves LLM code agent localization by 2.2pp, shortens trajectories by 1.6 rounds, and halves run-to-run variance on medium repositories via a deterministic anchoring effect.