Chain-of-thought prompting, by including intermediate reasoning steps in few-shot examples, elicits strong reasoning abilities in large language models on arithmetic, commonsense, and symbolic tasks.
mega hub Mixed citations
Evaluating Large Language Models Trained on Code
Mixed citation behavior. Most common role is background (63%).
2295
Pith papers citing it
1,464
external citations · Pith
Background
63% of classified citations
abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
hub tools
citation-role summary
background 194
dataset 71
method 13
other 3
baseline 2
citation-polarity summary
claims ledger
- abstract We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of ou
authors
mega hub controls
export citing contexts JSON
export graph JSON
export full bundle JSON
open full Pith review
annotated reader queued
Recognition alignment
counterfactual ablation
co-cited works
representative citing papers
ProcArena is the first multi-scenario, direct and interactive benchmark for evaluating LLMs on NL-to-PL/SQL, comprising 3,998 executable tasks across two SQL dialects.
AnLR-LoRA assigns per-rank-one adaptive learning rates to LoRA modules using function-space velocity and Adam SNR, consistently outperforming standard LoRA and its variants.
A functional census of 327 quantized LLM artifacts reveals five silently defective files in the official Ollama library and documents backend-dependent failures that no registry currently tests for.
Text-only LLMs can reliably write code to draw an image when given its full geometry, but their ability to compose a spatial layout from a brief description varies and depends on both the model and the output medium.
WoE embeds a pseudorandom green-list bias into selected expert weights of MoE LLMs via LoRA, enabling black-box text provenance from routing-trajectory reconstruction.
A 4B-parameter model trained via reinforcement learning to synthesize executable retrieval programs achieves 80.9% Hit@1 on e-commerce and 90.9% on email, surpassing GPT-5.5 and Claude Opus 4.7.
A counterexample-guided loop automatically writes a small Python metric that improves held-out code selection accuracy by about 15% of the reachable range without per-candidate model calls.
Revoked constraints still shape model behavior at an 8B operating point, and compiling the net constraint state ahead of time removes the observed relapse.
Supervised fine-tuning can implant triggered backdoors whose chain-of-thought traces look clean, evading CoT-only monitors and forcing a consistency-check view of monitoring.
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
Sumi is an openly released 7B parameter uniform diffusion language model pretrained from scratch on 1.5T tokens that matches autoregressive models on several benchmarks.
PCB-QA is the first QA benchmark for LLMs on printed circuit board designs, with Gemini 3 Flash Preview reaching 93% accuracy on a JSON textual representation.
TheoremBench is a Lean4 benchmark of classical theorems in main and premised forms that evaluates LLM provers on partial progress, coverage, and token efficiency rather than binary success on competition problems.
A new benchmark with cognitive traps shows frontier deep research agents achieve only 13-16% acceptance on expert consulting tasks under combined verifier and rubric criteria.
Mistletoe introduces a stealthy attack on speculative decoding that collapses acceleration by reducing average accepted length while preserving output semantics.
PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
PhysicianBench is a new benchmark of 100 physician-reviewed, execution-grounded tasks in live EHR environments where the best LLM agent reaches only 46% success and open-source models reach 19%.
MLLMs exhibit a Mirage effect by bypassing circuit diagrams in favor of header semantics for Verilog generation; VeriGround with identifier anonymization and D-ORPO training reaches 46% Functional Pass@1 while refusing blank images at >92%.
StabilizerBench is a new benchmark for evaluating AI agents on generating, optimizing, and making fault-tolerant stabilizer circuits for quantum error correction, with efficient verification and multi-tier scoring.
NLI autonomously discovers a vocabulary of primitive operations and interprets variable-length programs via a neural executor, allowing end-to-end training and gradient-based test-time adaptation that outperforms prior methods on combinatorial generalization tasks.
LLM agents autonomously evolve the ABC logic synthesis tool by iteratively rewriting its source code to achieve better quality-of-results on standard benchmarks while preserving the original interface.
FermiLink is a unified AI agent framework that automates multidomain scientific simulations via separated package knowledge bases and a four-layer progressive disclosure mechanism, reproducing 56% of target figures in benchmarks and generating research-grade results on unpublished problems.
citing papers explorer
-
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Chain-of-thought prompting, by including intermediate reasoning steps in few-shot examples, elicits strong reasoning abilities in large language models on arithmetic, commonsense, and symbolic tasks.
-
ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
ProcArena is the first multi-scenario, direct and interactive benchmark for evaluating LLMs on NL-to-PL/SQL, comprising 3,998 executable tasks across two SQL dialects.
-
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
AnLR-LoRA assigns per-rank-one adaptive learning rates to LoRA modules using function-space velocity and Adam SNR, consistently outperforming standard LoRA and its variants.
-
Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them
A functional census of 327 quantized LLM artifacts reveals five silently defective files in the official Ollama library and documents backend-dependent failures that no registry currently tests for.
-
Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models
Text-only LLMs can reliably write code to draw an image when given its full geometry, but their ability to compose a spatial layout from a brief description varies and depends on both the model and the output medium.
-
WoE Wrote It? Watermarking Mixture-of-Experts LLMs for Black-Box Text Provenance
WoE embeds a pseudorandom green-list bias into selected expert weights of MoE LLMs via LoRA, enabling black-box text provenance from routing-trajectory reconstruction.
-
ProRetrieval: Learning to Orchestrate Hybrid Search via Executable Program Synthesis
A 4B-parameter model trained via reinforcement learning to synthesize executable retrieval programs achieves 80.9% Hit@1 on e-commerce and 90.9% on email, surpassing GPT-5.5 and Claude Opus 4.7.
-
Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots
A counterexample-guided loop automatically writes a small Python metric that improves held-out code selection accuracy by about 15% of the reachable range without per-candidate model calls.
-
Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
Revoked constraints still shape model behavior at an 8B operating point, and compiling the net constraint state ahead of time removes the observed relapse.
-
Evading Chain-of-Thought Monitoring Through Model Poisoning
Supervised fine-tuning can implant triggered backdoors whose chain-of-thought traces look clean, evading CoT-only monitors and forcing a consistency-check view of monitoring.
-
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
-
Sumi: Open Uniform Diffusion Language Model from Scratch
Sumi is an openly released 7B parameter uniform diffusion language model pretrained from scratch on 1.5T tokens that matches autoregressive models on several benchmarks.
-
PCB-QA: Evaluating LLMs over the First Printed Circuit Board Design Question-Answer Dataset
PCB-QA is the first QA benchmark for LLMs on printed circuit board designs, with Gemini 3 Flash Preview reaching 93% accuracy on a JSON textual representation.
-
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
TheoremBench is a Lean4 benchmark of classical theorems in main and premised forms that evaluates LLM provers on partial progress, coverage, and token efficiency rather than binary success on competition problems.
-
Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps
A new benchmark with cognitive traps shows frontier deep research agents achieve only 13-16% acceptance on expert consulting tasks under combined verifier and rubric criteria.
-
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Mistletoe introduces a stealthy attack on speculative decoding that collapses acceleration by reducing average accepted length while preserving output semantics.
-
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.
-
SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
SimWorld Studio deploys an evolving coding agent to create adaptive 3D environments that co-evolve with embodied learners, delivering 18-point success-rate gains over fixed environments in navigation benchmarks.
-
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
PhysicianBench is a new benchmark of 100 physician-reviewed, execution-grounded tasks in live EHR environments where the best LLM agent reaches only 46% success and open-source models reach 19%.
-
From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation
MLLMs exhibit a Mirage effect by bypassing circuit diagrams in favor of header semantics for Verilog generation; VeriGround with identifier anonymization and D-ORPO training reaches 46% Functional Pass@1 while refusing blank images at >92%.
-
StabilizerBench: A Benchmark for AI-Assisted Quantum Error Correction Circuit Synthesis
StabilizerBench is a new benchmark for evaluating AI agents on generating, optimizing, and making fault-tolerant stabilizer circuits for quantum error correction, with efficient verification and multi-tier scoring.
-
Gradient-Based Program Synthesis with Neurally Interpreted Languages
NLI autonomously discovers a vocabulary of primitive operations and interprets variable-length programs via a neural executor, allowing end-to-end training and gradient-based test-time adaptation that outperforms prior methods on combinatorial generalization tasks.
-
Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC
LLM agents autonomously evolve the ABC logic synthesis tool by iteratively rewriting its source code to achieve better quality-of-results on standard benchmarks while preserving the original interface.
-
FermiLink: A Unified Agent Framework for Multidomain Autonomous Scientific Simulations
FermiLink is a unified AI agent framework that automates multidomain scientific simulations via separated package knowledge bases and a four-layer progressive disclosure mechanism, reproducing 56% of target figures in benchmarks and generating research-grade results on unpublished problems.
-
Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems
DDIPE poisons LLM agent skills by embedding malicious logic in documentation examples, achieving 11.6-33.5% bypass rates across frameworks while explicit attacks are blocked, with 2.5% evading detection.
-
Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.
-
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
MCP-Atlas is a new benchmark with 1000 tasks on production MCP servers that uses claim-level scoring to evaluate LLM agents on realistic multi-step tool-use competency.
-
Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
CritPt benchmark shows state-of-the-art LLMs reach only 5.7% average accuracy on full-scale unpublished physics research tasks, rising to about 10% with coding tools.
-
The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
AIDev is a new open dataset of 456k AI-agent pull requests showing agents submit code faster than humans but with lower acceptance rates and simpler changes.
-
Large Language Diffusion Models
LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.
-
ProSec: Fortifying Code LLMs with Proactive Security Alignment
ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.
-
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.
-
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
The AI Scientist framework enables LLMs to independently conduct the full scientific process from idea generation to paper writing and review, demonstrated across three ML subfields with papers costing under $15 each.
-
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
RepairAgent autonomously repairs 164 bugs on Defects4J including 39 not fixed by prior techniques by treating an LLM as an agent that invokes tools via a finite state machine and dynamic prompts.
-
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
LongBench is the first bilingual multi-task benchmark for long context understanding in LLMs, containing 21 datasets in 6 categories with average lengths of 6711 words (English) and 13386 characters (Chinese).
-
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
API-Bank is a new benchmark and training dataset for tool-augmented LLMs that shows fine-tuned models can approach GPT-3.5 tool-use effectiveness.
-
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.
-
PAL: Program-aided Language Models
PAL improves few-shot reasoning accuracy by having LLMs generate executable programs rather than text-based chains of thought, outperforming much larger models on math and logic benchmarks.
-
Code as Policies: Language Model Programs for Embodied Control
Language models generate robot policy code from natural language commands via few-shot prompting, enabling spatial-geometric reasoning, generalization, and precise control on real robots.
-
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Training language models to generate intermediate computation steps on a scratchpad enables them to perform multi-step tasks such as long addition and arbitrary program execution that they otherwise fail at.
-
TruthfulQA: Measuring How Models Mimic Human Falsehoods
A new benchmark reveals that language models including GPT-3 are truthful on only 58% of questions designed to elicit popular misconceptions, far below human performance of 94%, with larger models performing worse.
-
GraphAHA: Graph-Based Adaptive Search with Heterogeneous Actions for Test-Time Code Generation
GraphAHA merges equivalent programs into a directed acyclic graph and uses hierarchical Thompson sampling to adaptively choose between sampling, reasoning, and repair actions during test-time code generation, improving Pass@1 by 4.1 percentage points on average.
-
Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning
CluSTER combines gradient-space clustering with DP-aware balanced allocation and weighted gradient updates to reduce training time up to 69.6% without significant accuracy loss across instruction-tuning datasets.
-
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Verbalized confidence surpasses log-probability based soft scoring as the better signal for evaluating outputs with the newest proprietary LLMs, reversing the pre-2025 best practice.
-
Retrofitting Code Using LLMs to Support Exceptional Behavior
EXCODER retrofits exception-related code into Java methods using LLMs guided by static and dynamic program analysis from exceptional behavior tests.
-
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
The paper identifies the verification gap as the binding constraint on LLM reasoning and proposes proof-carrying cognition, where reasoning is verified against a world model trained on reality settlement.
-
Benchmarking Hybrid Deep Research Across Database Querying and Web Search
HybridDeepResearch is the first benchmark requiring agents to integrate web search and SQL queries to solve complex analytical tasks, showing that current models fail frequently at this cross-modal reasoning.
-
How effective are traditional test criteria at detecting bugs in large language models generated code?
Traditional test adequacy criteria (statement, branch, mutation) are ineffective at detecting faults in LLM-generated code, with fault detection rates near one to two percent.
-
Training-Free Task Vectors for LLM Behavioral Control
Training-Free Task Vectors map activation steering directions to rank-one weight updates using SVD and expected input, enabling compositional behavioral editing without training.
-
What Fixed-Rollout pass@k Evaluations Can Identify
Fixed-n rollout counts identify only n moments of the latent task difficulty, making extrapolated pass@k generically nonidentifiable for k > n.