REVIEW 19 cited by
SantaCoder: don't reach for the stars!
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.
Forward citations
Cited by 19 Pith papers
-
Unified Communication Compression Beyond Global Error Bounds for Distributed Nonconvex Optimization
A unified compression algorithm for distributed nonconvex optimization achieves O(1/sqrt(T)) convergence for locally-bounded compressors, matching centralized 1-bit methods, with an improved O(1/T^{2/3}) rate after on...
-
Structure-Aware Fill-in-the-Middle Pretraining for Code
AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.
-
COFFE: A Code Efficiency Benchmark for Code Generation
A new benchmark, COFFE, uses stressful test cases and CPU instruction counts to show LLM-generated code is often correct but time-inefficient.
-
TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models
Pretrained code models linearly encode cross-lingual type representations, including result types of function application, that emerge from untyped code and survive partial lexical adversarial renaming.
-
Rethinking Code Performance Benchmarks for LLMs
Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidd...
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Investigating Training Data Detection in AI Coders
Most existing training-data detection methods perform poorly on code, while prefix-relative method ReCaLL consistently scores highest, though all degrade under code mutations.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
Recommending Variable Names for Extract Local Variable Refactorings
VarNamer recommends variable names for extract local variable refactorings using homogeneous-variable reuse and mined naming rules, improving exact-match precision over Eclipse, IntelliJ, and Incoder.
-
KEENHash: Hashing Programs into Function-Aware Embeddings for Large-Scale Binary Code Similarity Analysis
KEENHash hashes binaries into 8KB function-aware embeddings via LLM function embeddings, K-Means clustering, and Feature Hashing, enabling scalable program-level binary similarity analysis.
-
Seed-Coder: Let the Code Model Curate Data for Itself
Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.
-
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
A code retrieval benchmark with quality labels for correctness, efficiency, security, and maintainability shows most retrievers fail to prefer high-quality code, and quality-aware fine-tuning improves this without mea...
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.
-
LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.
-
Fault Localization via Fine-tuning Large Language Models with Mutation Generated Stack Traces
Fine-tuning LLMs on mutation-generated crash stack traces localizes faults in SAP HANA, SQLite, and DuckDB with 63 to 74 percent accuracy, beating weak baselines.
-
On the Generalizability of Transformer Models to Code Completions of Different Lengths
Across two languages and three metrics, no tested positional encoding scheme generalizes to code completion lengths unseen in training; mixed-length training is the recommended safe choice.
-
GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion
GRACE combines a multi-level code graph, hybrid text-structure retrieval, and graph fusion to improve repository-level code completion over vanilla and graph-based RAG baselines.
-
LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch
K2 Diamond is a fully open 65B-parameter LLM that reaches Llama 2 70B-level performance on standard benchmarks.
-
Evaluation of Large Language Model-Driven AutoML in Data and Model Management from Human-Centered Perspective
A 15-participant within-subjects study claims LLM-driven AutoML outperforms traditional AutoML on speed, accuracy, and usability, though supporting data are not provided and several numbers conflict.
Discussion (0). Continue with ORCID to comment.