REVIEW 69 cited by
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated exceptional performance in code-related tasks. However, most existing models are solely pre-trained on extensive raw code data without instruction fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Through comprehensive experiments on four prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we unveil the exceptional capabilities of our model. It surpasses all other open-source Code LLMs by a substantial margin. Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+. Our code, model weights, and data are public at https://github.com/nlpxucan/WizardLM
Forward citations
Showing 60 of 69 Pith papers that cite this
-
ProSec: Fortifying Code LLMs with Proactive Security Alignment
ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.
-
Beware What You Autocomplete: Forensic Attribution of Backdoored Code Completions
CodeTracer attributes backdoored code completions to responsible fine-tuning examples via structured behavioral fingerprints, embedding retrieval, and LLM semantic reasoning under realistic post-deployment constraints.
-
K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs
K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.
-
SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner
SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.
-
COFFE: A Code Efficiency Benchmark for Code Generation
A new benchmark, COFFE, uses stressful test cases and CPU instruction counts to show LLM-generated code is often correct but time-inefficient.
-
Understanding and Improving Model Editing for Secure Code Generation
Editing a code model's parameters is a stronger defense against known vulnerability types than inference-time filtering, and a new post-edit tuning step largely fixes the coding-quality regression that editing causes.
-
Biased or Personalized? The Impact of Personal Information on AI-driven Development
Changing only the prompter's age and gender in AI coding prompts produces statistically significant differences in generated website interface design, template content, and code structure across 800 generated websites...
-
Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis
Refusal-ablated LLMs outperform aligned models on code-grounded localization and early executable patch generation, while aligned models retain advantages on shallow diagnostic tasks under neutral wording.
-
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).
-
SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis
SemOpt generates Semgrep static-analysis rules from LLM-summarized optimization commits and uses them to locate and apply optimization strategies, outperforming retrieval-based baselines on C/C++ code.
-
ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.
-
SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches
SketchAgent automates sketch-to-diagram conversion with a three-agent pipeline, but its benchmark replaces hand-drawn sketches with simplified renderings of the very diagrams the system must produce.
-
Enhancing Project-Specific Code Completion by Inferring Internal API Information
A retrieval-augmented code completion method that infers project-internal APIs from a rough draft and a static knowledge base, beating existing repo-level baselines.
-
Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing
A bootstrapping algorithm that trains program synthesis models on their own successful outputs and on repaired failures modestly improves pass@k over regular fine-tuning on MBPP, with mixed results on APPS.
-
Turning the Tide: Repository-based Code Reflection
A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
Recommending Variable Names for Extract Local Variable Refactorings
VarNamer recommends variable names for extract local variable refactorings using homogeneous-variable reuse and mined naming rules, improving exact-match precision over Eclipse, IntelliJ, and Incoder.
-
Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
Training on more validated GitHub issue-fix trajectories steadily raises SWE-bench Verified pass@1 from 6.4% to 38.0%, with 47.0% after test-time scaling, and no plateau yet.
-
How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison
LLMs improve only slightly and unstably from test-time experience on semantic reasoning games, while humans learn much faster.
-
SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
An agentic workflow with security guidelines and LLM-generated unit tests improves secure code generation on CWEval C tasks from 61% to 76% Func-Sec@1 with Sonnet-3.7, at roughly 98% of its original functionality.
-
Tag-Evol: Achieving Efficient Instruction Evolving via Tag Injection
Tag-Evol generates harder, more diverse instruction data by injecting sampled knowledge tags into seed instructions, improving downstream SFT accuracy across math, code, and general benchmarks.
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.
-
Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement
AlignMind, a multi-agent system with theory-of-mind helpers, refines software requirements through multi-round dialogue, outperforming a direct-prompt baseline in LLM-judged quality and lexical richness, at high token...
-
Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
Griffon-R generates its own grounding hints and rationale before answering, achieving state-of-the-art visual reasoning on VSR and CLEVR while improving MMBench, ScienceQA, and TextVQA.
-
A Solver-Aided Hierarchical Language for LLM-Driven CAD Design
A solver-aided hierarchical DSL lets an untuned LLM generate precise, editable 2D CAD geometry from text prompts, outperforming OpenSCAD slightly on CLIP alignment.
-
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
CSR-Bench and CSR-Agents show that LLM agents can complete under half of setup and data-download steps, and between 15 and 29 percent of training, inference, and evaluation steps, on 100 research repositories.
-
Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
An automated multi-fidelity search framework discovers layer-wise and depth-wise model merging recipes that improve single- and multi-objective LLM reasoning performance without retraining.
-
Extractive Schema Linking for Text-to-SQL
Extractive schema linking with decoder-only LLMs beats generative and cross-encoder baselines on Spider and BIRD while running more than twenty times faster than generative linking.
-
DeCon: Detecting Incorrect Assertions via Postconditions Generated by a Large Language Model
DeCon detects incorrect LLM-generated assertions by checking each assertion against LLM-generated postconditions that have been filtered against docstring I/O examples.
-
SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Code
SPDZCoder combines hand-crafted expert rules with LLM prompting to translate Python to MP-SPDZ, reporting 85.9% pass@1 on its new SPDZEval benchmark, far above the best direct-LLM baseline.
-
Dynamic Skill Adaptation for Large Language Models
A training pipeline that orders generated textbook and exercise data by a skill dependency graph and dynamically updates the data during fine-tuning improves LLM performance on calculus and social studies evaluations.
-
Repository Structure-Aware Training Makes SLMs Better Issue Resolver
Training 7B code models on automatically constructed repository-localization and code-edit data improves their SWE-Bench-verified resolution rate from about 1% to 7%.
-
HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
Fine-tuning DeepSeek-Coder on a new 122k synthetic HPC instruction dataset yields open-source models that reach 34.1 pass@1 on ParEval parallel code generation, besting other open baselines but trailing GPT-4.
-
ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis
ResoFilter keeps fine-tuning examples that produce small parameter updates in the last layers, matching full-data fine-tuning on GSM8k with 50% of the math data.
-
Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks
Token-level MCTS with CoT prompting lifts Qwen2.5-Coder-32B-Instruct to a 0.351 pass rate on LiveCodeBench-Hard, approaching O1-Mini's pass@1.
-
Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.
-
When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair
Federated fine-tuning of six code LLMs on private bug-fix data improves program repair to near-centralized levels, with negligible impact from heterogeneous code.
-
ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation
ConAIR iteratively corrects the most inconsistent LLM-generated test with user feedback, then repairs code, achieving large Pass@1 gains on HumanEval, HumanEval+, and MBPP in simulation.
-
LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues
A reflection-based experience pool raises LLM issue reproduction accuracy from 45% to 54% on SWE-bench Lite, with gains in downstream issue resolving.
-
Route-Align-Verify for Functional Correctness in Code Generation
RAV combines task-aware prompt routing, aligned LoRA fine-tuning, and public-test-based candidate selection to improve MBPP pass@1, but its verification step leaks evaluation tests.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
-
GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion
GRACE combines a multi-level code graph, hybrid text-structure retrieval, and graph fusion to improve repository-level code completion over vanilla and graph-based RAG baselines.
-
SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks
The full text describes LaTCoder, a block-wise chain-of-thought method for webpage design-to-code, reporting improved layout preservation on two benchmarks, while the title and abstract are for a different paper.
-
LaTCoder: Converting Webpage Design to Code with Layout-as-Thought
LaTCoder cuts a webpage screenshot into blocks, generates code for each block with chain-of-thought prompting, and assembles the blocks, improving layout fidelity over direct whole-page generation.
-
Detailed radial scale height profile of dust grains as probed by dust self-scattering in HL Tau
From the near-far side asymmetry and azimuthal contrast in HL Tau's polarized intensity, the authors infer a radial dust scale height profile and a turbulence parameter alpha increasing from 1e-5 at 100 au to 1e-2.5 at 20 au.
-
Multi-Agent Collaboration for Multilingual Code Instruction Tuning
A multi-agent collaboration framework for generating multilingual code instruction data is introduced, but the reported evaluations show the fine-tuned model underperforming its own base instruct model.
-
CodeSCM: Causal Analysis for Multi-Modal Code Generation
A causal framework with latent mediators quantifies how prompt modalities affect code LLMs, finding that input-output examples and function-header names are influential beyond natural language instructions.
-
OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training
OpenCSG released four open Chinese LLM training datasets, and 2B-scale tests report improved C-Eval, CMMLU, and Alignbench scores.
-
HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
HumanEval Pro and MBPP Pro test whether LLMs can first solve a base function and then call it inside a more complex solution, and frontier models lose about 10 to 15 points on this task.
-
Distilling Desired Comments for Enhanced Code Review with Large Language Models
Desiview identifies desired review comments from code review datasets using the perplexity difference of the actual fix with and without each comment, and the distilled data improves LLaMA-based code review models.
-
ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback
ADC fine-tunes Llama-3.1-8B with code execution traces and adversarial function-call data, improving BFCL v2 overall from 63.19 to 79.01, but with no released artifacts or error bars.
-
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
AceMath models use general-then-math fine-tuning and a listwise-trained reward model to report state-of-the-art math reasoning scores.
-
RedStone: Curating General, Code, Math, and QA Data for Large Language Models
RedStone turns Common Crawl into large, diverse pre-training datasets for language, code, math, and QA, claiming improved benchmark scores over existing open datasets.
-
On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code
Appending LLM-generated misleading code snippets to coding prompts lowers pass@1 by 12-34% for open-source and 3-24% for commercial code LLMs; guided prompting restores only part of the loss.
-
Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering
Interactive prompting (context enrichment, task decomposition, and question-asking) lifts LLM code snippet adaptation pass@1 from about 52% to 68-75% on ClassEval, with a multi-agent variant nearly matching human feedback.
-
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs
DSTC builds preference pairs by selecting the hardest self-generated test that the best self-generated code passes, then fine-tunes code LMs with DPO or KTO to improve pass@1 accuracy.
-
FastDraft: How to Train Your Draft
FastDraft trains small vocabulary-compatible draft models from scratch in under 24 hours, yielding up to 2x wall-clock and 3x memory-bound speedups for speculative decoding.
-
Infinite-Instruct: Synthesizing Scaling Code instruction Data with Bidirectional Synthesis and Static Verification
A synthetic data pipeline for code instruction tuning reports large benchmark gains, but the headline improvements are internally inconsistent.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
-
PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback
PlotGen, a five-agent LLM system, improves automated scientific chart generation by adding numeric, lexical, and visual feedback to iteratively fix plotting errors.
Discussion (0). Continue with ORCID to comment.