REVIEW 6 cited by
NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition
read the original abstract
Natural language-driven no-code development allows users to specify software functionality using natural language (NL) instead of editing source code, promising increased productivity and democratized development. Large language models (LLMs) show potential in enabling this paradigm. In this context, software documentation acts as an NL specification for functionality. This work introduces NoCode-bench, a benchmark designed to evaluate LLMs on real-world NL-driven feature addition tasks, consisting of 634 tasks across 10 projects and 114k code changes. Each task pairs documentation updates with corresponding code implementations, validated by developer-written test cases. A subset of 114 high-quality, human-verified instances, NoCode-bench Verified, ensures reliable evaluation. Our experiments reveal that, despite high token usage, the best LLMs achieve a task success rate of only 28.07%, highlighting challenges in cross-file editing, codebase understanding, and tool calling. These findings indicate that LLMs are not yet ready for fully NL-driven no-code development. NoCode-bench lays the foundation for future advances in this area.
Forward citations
Cited by 6 Pith papers
-
Names Are All You Need: Effective and Safe Regression Test Selection for Python
NameRTS uses name-based bipartite graph reachability for safe and effective regression test selection in Python, skipping 69.9% of tests on average with 99.6% safety on a new benchmark.
-
Reproduction Test Generation for Java SWE Issues
Presents the first benchmark and adapted solution for generating reproduction tests from Java software issues.
-
Neurosymbolic Repo-level Code Localization
LogicLoc combines LLMs with Datalog to achieve accurate repo-level code localization without relying on keyword shortcuts in benchmarks.
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A custom LLM agent achieves 94% manually verified success on a new benchmark of 35 software analysis setups, outperforming baselines at 77%, but struggles with stage mixing, error localization, and overestimating its ...
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.
-
Reproduction Test Generation for Java SWE Issues
Introduces the first benchmark for Java reproduction test generation from repository issues and adapts a prior Python tool to produce high performance on it.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.