REVIEW 24 cited by
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.
Forward citations
Cited by 24 Pith papers
-
DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
A production-grounded, sandbox-graded benchmark of 100 multi-engine data-engineering tasks shows frontier agents top out at 74.9 with no cross-engine winner.
-
SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner
SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.
-
FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
The paper introduces FEM-Bench, a 33-task computational mechanics benchmark, and shows that state-of-the-art LLMs complete at most 30/33 tasks with multiple attempts and fail entirely on geometric-stiffness-related tasks.
-
Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting
Iterative LLM refinement helps early in ideation and code, but in math only late under elaboration prompting; vague feedback tends to plateau or degrade quality.
-
Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data
Non-programmer business users often fail to spot critical mistakes in AI-generated data analyses, even when explicitly warned and incentivized.
-
More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation
Prompt specificity substantially raises LLM pass@1 on code benchmarks, with the largest gains on parallel-programming tasks.
-
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.
-
A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback subs...
-
ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation
A fully automated pipeline generates a 222.5K-pair synthetic chart dataset with 27 chart types and 11 plotting libraries, and a GPT-4o-judged benchmark shows current open-weights VLMs still underperform on chart-to-co...
-
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
SwiftEval, a 28-problem hand-crafted Swift benchmark, evaluates 44 code LLMs and shows large performance drops on Swift tasks, especially for smaller models.
-
Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation
The paper builds a 1000-task test-driven development benchmark for LLMs and concludes that instruction following and in-context learning matter more than general coding proficiency.
-
CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification
CLOVER is a new Python unit-test-generation benchmark with 845 problems, contexts up to 128k tokens, and coverage-based retrieval, where all 14 evaluated models score below 35% on the hardest task.
-
SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Code
SPDZCoder combines hand-crafted expert rules with LLM prompting to translate Python to MP-SPDZ, reporting 85.9% pass@1 on its new SPDZEval benchmark, far above the best direct-LLM baseline.
-
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models
A code LLM fine-tuned on winner responses from pairwise expert battles, with instructions mined from chat templates, beats same-size baselines without proprietary LLMs.
-
Evaluating and Aligning CodeLLMs on Human Preference
A new 397-sample, 44-language code Q&A benchmark with LLM-judged human preference reveals gaps between open and proprietary code models and a 20B-token synthetic instruction corpus.
-
FullStack Bench: Evaluating LLMs as Full Stack Coders
FullStack Bench and SandboxFusion offer a broader, execution-based benchmark for code LLMs with 3,374 problems in 16 languages and 11 domains.
-
Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
Across 56 tasks, 9 model configurations, and 10,584 runs, LLM-generated skill files provided no reliable performance improvement over task-only prompting for data-science workflows.
-
Trading Human Curation for Synthetic Augmentation in RLVR
Gated synthetic augmentations of a 10-task human base substitute for ~87 extra human RLVR tasks on aggregate held-out pass@1, with cost-adjusted trade rate ρ_cost in [1.4×, 11.6×].
-
COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models
COMPASS benchmark shows that evaluating code generation with only correctness misses large differences in runtime efficiency and code quality among frontier LLMs.
-
CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
The authors propose CompassVerifier, a lightweight multi-domain answer verifier, and VerifierBench, a benchmark of LLM outputs for evaluating verifiers.
-
HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics
A hierarchical Bayesian GLM framework, HiBayES, provides uncertainty-aware performance estimates and formal model comparison for AI evaluation, with a software package and demonstrations on LLM benchmarks.
-
CodeSCM: Causal Analysis for Multi-Modal Code Generation
A causal framework with latent mediators quantifies how prompt modalities affect code LLMs, finding that input-output examples and function-header names are influential beyond natural language instructions.
-
Multi-matrix Factorization Attention
MFA and MFA-KR factorize the attention QK circuit with shared low-rank key and value projections, matching or beating MHA accuracy at a small fraction of the KV cache.
-
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.
Discussion (0). Continue with ORCID to comment.