Pith. sign in

REVIEW 49 cited by

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.04905 v3 pith:AN4GMTFG submitted 2024-11-07 cs.CL cs.PL

Siming Huang , Tianhao Cheng , J.K. Liu , Jiaran Hao , Liuyihan Song , Yang Xu , J. Yang , Jiaheng Liu
show 11 more authors
Chenchen Zhang Linzheng Chai Ruifeng Yuan Zhaoxiang Zhang Jie Fu Qian Liu Ge Zhang Zili Wang Yuan Qi Yinghui Xu Wei Chu
This is my paper · ORCID
classification cs.CLcs.PL
keywords codedatamodelsopentop-tierllmsopencoderreproducible
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 49 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ProSec: Fortifying Code LLMs with Proactive Security Alignment

    cs.CR 2024-11 conditional novelty 8.0 of 10

    ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.

  2. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner

    cs.CL 2025-06 conditional novelty 7.0 of 10

    SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.

  3. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.

  4. Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Left-shifting iterative compiler/test refinement into verified SFT data, then GRPO on difficulty-curated IO rewards, lifts Qwen3-8B Julia pass@1 past prior SOTA at 1/3 data and 1/6 cost, and bootstraps Ballerina.

  5. Fine-Tuning Masked Diffusion for Provable Self-Correction

    cs.LG 2025-10 conditional novelty 6.0 of 10

    PRISM fine-tunes any masked diffusion model with a binary-cross-entropy loss so its new head provably estimates per-token quality p(x_i=y_i|y⊕m_i) and can remask low-quality tokens at inference.

  6. Understanding Benchmark Language Under Weakened Formal Semantics

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICRAG, a retrieval-augmented iterative code-generation framework, improves accuracy on 13 NLP benchmarks by compiling questions into Python programs and executing them, though gains are partly inflated by in-distribut...

  7. Empirical Study of Code Large Language Models for Binary Security Patch Detection

    cs.SE 2025-09 conditional novelty 6.0 of 10

    Fine-tuned code LLMs detect binary security patches accurately on decompiler pseudo-code, and adding source-code patches to the training data improves accuracy and F1 further.

  8. Delta Activations: A Representation for Finetuned Large Language Models

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive com...

  9. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  10. VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    VERIRL applies reinforcement learning with a trace-back rescore mechanism and sample-balanced weighting to achieve state-of-the-art Verilog code generation.

  11. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    cs.CL 2025-08 conditional novelty 6.0 of 10

    AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.

  12. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  13. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.

  14. CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeEvo uses two interacting LLM agents with keyword-guided instruction evolution and hybrid compiler-plus-LLM feedback to synthesize high-quality instruction-code pairs for fine-tuning code models.

  15. Turning the Tide: Repository-based Code Reflection

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.

  16. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B masked-diffusion code model plus complementary-mask GRPO (coupled-GRPO) improves benchmark scores and shifts decoding away from strict left-to-right order.

  17. Essential-Web v1.0: 24T tokens of organized web data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.

  18. ProtocolLLM: RTL Benchmark for SystemVerilog Generation of Communication Protocols

    cs.AR 2025-06 conditional novelty 6.0 of 10

    A new benchmark, ProtocolLLM, evaluates LLM-generated SystemVerilog for SPI, I2C, UART, and AXI and finds most models fail timing-accurate functional checks.

  19. Seed-Coder: Let the Code Model Curate Data for Itself

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.

  20. Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.

  21. rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A large verified dataset of 418K competition code problems built by synthesis, test-input utility functions, and majority-vote output labeling improves small model code reasoning to frontier level.

  22. Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Code LLMs drop by more than 10% in accuracy when problem details are subtly changed, and fine-tuning on such counterfactual variants boosts performance on standard benchmarks.

  23. Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    The paper builds a 1000-task test-driven development benchmark for LLMs and concludes that instruction following and in-context learning matter more than general coding proficiency.

  24. CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts

    cs.LG 2025-05 conditional novelty 6.0 of 10

    CodeMixBench adds code-mixed prompt variants to BigCodeBench and shows that code-mixed inputs consistently reduce Pass@1 performance of code LLMs.

  25. CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Training on 3.5M code input-output prediction tasks with natural-language chain-of-thought improves LLM performance on math, logic, symbolic, scientific, and commonsense reasoning benchmarks.

  26. ACECODER: Acing Coder RL via Automated Test-Case Synthesis

    cs.SE 2025-02 conditional novelty 6.0 of 10

    ACECODER shows that automatically synthesized test cases can serve as scalable reward signals for code RL, yielding consistent gains on HumanEval, MBPP, BigCodeBench, and LiveCodeBench.

  27. BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

    cs.LG 2025-01 reject novelty 6.0 of 10

    BRiTE is an EM-style algorithm that uses RL to sample high-likelihood reasoning chains and fine-tunes the LLM on them, with a theoretical 1/T convergence rate and mixed empirical gains.

  28. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings

    cs.CL 2025-01 conditional novelty 6.0 of 10

    CodeElo benchmarks LLMs by submitting their solutions to CodeForces's official judge and converting results into human-comparable Elo ratings, with o1-mini at 1578 and most open models below the 20th human percentile.

  29. QUBE: Enhancing Automatic Heuristic Design via Quality-Uncertainty Balanced Evolution

    cs.NE 2024-12 conditional novelty 6.0 of 10

    QUBE adds a UCB-style uncertainty term to FunSearch's parent selection and finds better heuristics on bin packing and TSP, with cap set gains only over a same-hardware reproduction.

  30. Dream-Coder 7B: An Open Diffusion Language Model for Code

    cs.CL 2025-09 conditional novelty 5.0 of 10

    An open 7B diffusion language model for code, adapted from Qwen2.5-Coder, reaches 21.4% pass@1 on LiveCodeBench and shows adaptive, non-left-to-right generation patterns.

  31. Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    The authors introduce and test an optimization method for fine-tuning data mixtures that matches costly grid-search results within 0.66% validation loss.

  32. SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Two new MoE language models and a co-designed inference engine claim 20+ tokens/s CPU decoding under 1-8 GB memory with benchmark scores comparable to much larger models.

  33. A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat

    cs.SE 2025-07 conditional novelty 5.0 of 10

    On WeChat's closed-source codebase, similarity-based RAG with combined BM25 and GTE-Qwen retrieval improves open-source LLM code completion more than identifier-based retrieval, with gains growing for larger models.

  34. Solving Formal Math Problems by Decomposition and Iterative Reflection

    cs.AI 2025-07 conditional novelty 5.0 of 10

    An agent that decomposes Lean 4 goals into subproblems and iteratively repairs proofs achieves a 95.9% pass rate on miniF2F-test using a stock Gemini model.

  35. AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A 7B reasoning model trained with carefully balanced SFT and RL beats prior small models on math and code benchmarks, with the paper documenting scaling and temperature heuristics.

  36. HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration

    cs.DC 2025-06 conditional novelty 5.0 of 10

    A TVM-based pipeline generates 20,000 CUDA-CPU code pairs, and fine-tuning code LLMs on this data improves transpilation success and CPU performance, with an average speedup improvement of 43.8%.

  37. CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

    cs.SE 2025-05 conditional novelty 5.0 of 10

    CODE-DITING distills reasoning from a large reasoning LLM into 1.5B and 7B judges that evaluate generated code without reference solutions or test cases, beating larger judges under majority voting.

  38. LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LLM-KG-Bench 3.0 is an open, extensible benchmark framework with automatic scoring that compares more than 30 LLMs on RDF and SPARQL knowledge graph tasks.

  39. DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training

    cs.CL 2025-04 conditional novelty 5.0 of 10

    A two-stage SFT recipe using pass-rate and coefficient-of-variation based data selection from a 40M-response distilled dataset lifts Qwen2.5-72B to 79.2% on AIME2024, nearly matching RL-trained reasoning models.

  40. BitsAI-CR: Automated Code Review via LLM in Practice

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.

  41. InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion

    cs.CL 2025-01 reject novelty 5.0 of 10

    InfiFusion fuses multiple large language models into one pivot model using enhanced universal logit distillation, and reports that the fused model outperforms all source models on 11 benchmarks with a fraction of the ...

  42. HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation

    cs.SE 2024-12 conditional novelty 5.0 of 10

    HumanEval Pro and MBPP Pro test whether LLMs can first solve a base function and then call it inside a more complex solution, and frontier models lose about 10 to 15 points on this task.

  43. Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Typhoon 2 improves Thai LLM performance through continual pre-training on curated Thai data and post-training, releasing text, vision, audio, and safety models.

  44. ExecRepoBench: Multi-level Executable Code Completion Evaluation

    cs.CL 2024-12 reject novelty 5.0 of 10

    ExecRepoBench is an executable, repository-level code completion benchmark with unit tests, and Qwen2.5-Coder-Instruct-C is a 7B model fine-tuned on the accompanying Repo-Instruct corpus.

  45. Secure Code Generation at Scale with Reflexion

    cs.CR 2025-11 conditional novelty 4.0 of 10

    Three-round reflexion prompting raises LLM-generated code security from 70.74% to 79.43% as measured by the ICD static detector on the Instruct Prime benchmark.

  46. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  47. YuLan-Mini: An Open Data-efficient Language Model

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A 2.42B-parameter base model trained on 1.08T tokens matches or beats several industry baselines trained on 7T to 18T tokens across math, code, and general benchmarks.

  48. CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation

    cs.AI 2025-07 reject novelty 3.0 of 10

    A 7B model trained with GRPO and a sparse execution-correctness reward reaches 59.97% execution accuracy on BIRD dev, though the evaluation protocol and baseline numbers contain inconsistencies.

  49. Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study

    cs.CL 2025-05 conditional novelty 3.0 of 10

    Applying LD-DPO to the DeepDistill-32B model improves average benchmark scores by 3.3 points, but the gains are uneven, the tables are inconsistent, and the evidence is preliminary.

Pith tools