Pith. sign in

REVIEW 2 major objections 2 minor 2793 cited by

Evaluating Large Language Models Trained on Code

T0 review · 2 major / 2 minor · reviewed 2026-05-24 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A language model fine-tuned on GitHub code solves 28.8 percent of problems on a new benchmark for writing programs from docstrings.

desk verdict Codex introduces HumanEval and shows code-fine-tuned models can hit 28.8% functional correctness on docstring-to-code tasks, with sampling lifting that to 70%, though the contamination check is thin. read the letter →

arxiv 2107.03374 v2 pith:E7CAGJ73 submitted 2021-07-07 cs.LG

Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan , Harri Edwards , Yuri Burda
show 50 more authors
Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba
This is my paper · ORCID
classification cs.LG
keywords CodexHumanEvalcodegenerationprogramsynthesisfunctionalcorrectnessGPTfine-tuninglargelanguagemodels
checked against Cost.FunctionalEquationreality_from_one_distinction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Codex, a version of GPT fine-tuned on public code repositories. It measures how well this model can write correct Python functions when given only a docstring describing what the function should do. On the released HumanEval benchmark, Codex succeeds on 28.8 percent of the tasks while the base GPT-3 model succeeds on none. The authors also show that generating many candidate solutions and testing them raises the success rate to over 70 percent with 100 tries per problem. They note remaining weaknesses in handling long operation chains and variable bindings, and they outline possible effects on software development practices.

What carries the argument

Codex, the GPT model fine-tuned on GitHub code to generate Python functions from natural language descriptions.

What would settle it

Demonstrating that the model has seen the HumanEval problems or equivalent solutions during training, or showing that its success rate drops sharply on a fresh set of similar but unseen docstring-to-function tasks.

Watch

Extended reading notes

Core claim

Codex, a GPT model fine-tuned on publicly available GitHub code, achieves a 28.8% pass rate on the HumanEval benchmark for synthesizing correct programs from docstrings, compared to 0% for GPT-3 and 11.4% for GPT-J. Repeated sampling from the model raises this to 70.2% with 100 samples per problem.

Load-bearing premise

The HumanEval test cases are uncontaminated by the model's training data and give a reliable measure of whether the generated code actually works as described.

Editorial extensions

If this is right

  • Code generation performance improves substantially when language models are fine-tuned on large code corpora rather than text alone.
  • Generating and testing multiple outputs from the model is an effective way to increase the chance of finding a correct solution.
  • Current models still struggle with prompts that require chaining many operations or correctly binding values to variables.
  • Deployment of such models raises questions around safety, security vulnerabilities in generated code, and economic effects on programming work.
  • HumanEval provides a functional correctness measure that can be used to track progress in code synthesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Models like this could accelerate prototyping in software engineering by turning specifications into runnable code quickly.
  • Without safeguards, generated code might introduce security flaws that are hard to spot in large outputs.
  • Success on repeated sampling suggests that verification loops could become a standard part of AI-assisted coding workflows.
  • Similar fine-tuning approaches might extend to other programming languages beyond Python if comparable datasets exist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces Codex, a GPT-style language model fine-tuned on 159 GB of publicly available GitHub code, and releases the HumanEval benchmark consisting of 164 Python problems with hidden unit tests to measure functional correctness of code synthesized from docstrings. It reports that Codex achieves 28.8% pass@1 on HumanEval (versus 0% for GPT-3 and 11.4% for GPT-J), that repeated sampling raises this to 70.2% with 100 samples, and discusses model limitations and broader impacts.

Significance. If the central empirical numbers hold, the work provides a valuable public benchmark and reproducible measurement of code-generation capabilities in large language models, with explicit credit due for releasing HumanEval together with its functional test suite. The finding that repeated sampling is effective is a practical contribution that can be directly tested by others.

major comments (2)
  1. [§3] §3 (HumanEval construction and data contamination): The reported manual check establishes only that none of the 164 problems appear verbatim in the training corpus. This is insufficient to rule out near-duplicate docstrings, common function signatures, or solution logic that could have been present in the 159 GB GitHub data, directly affecting whether the 28.8% pass@1 can be interpreted as synthesis capability rather than partial memorization.
  2. [§4] §4 (main results and baselines): The performance gap versus GPT-3 and GPT-J is presented as evidence of Codex's superiority, yet the manuscript provides limited detail on whether identical prompt formatting, temperature, and stop-sequence settings were used for all three models; any mismatch would weaken the cross-model comparison that underpins the central claim.
minor comments (2)
  1. [Abstract] The abstract states that a production version of Codex powers GitHub Copilot; a brief clarification in the main text on how the evaluated checkpoint relates to the deployed model would improve transparency without altering the technical claims.
  2. [Figure 2] Figure 2 (pass@k curves) would benefit from explicit error bars or multiple random seeds to convey variability in the repeated-sampling results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive recommendation of minor revision and for the constructive comments on data contamination and baseline comparisons. We address each point below.

read point-by-point responses

  1. Referee: [§3] §3 (HumanEval construction and data contamination): The reported manual check establishes only that none of the 164 problems appear verbatim in the training corpus. This is insufficient to rule out near-duplicate docstrings, common function signatures, or solution logic that could have been present in the 159 GB GitHub data, directly affecting whether the 28.8% pass@1 can be interpreted as synthesis capability rather than partial memorization.

    Authors: We agree that the manual check rules out only exact verbatim matches and does not fully exclude near-duplicates or common patterns. In the revision we will expand §3 with additional details on our contamination analysis, including checks for similar docstring phrasing and function signatures in a sampled subset of the training corpus, and we will explicitly note the inherent difficulty of exhaustive verification over 159 GB as a limitation. The large gap relative to GPT-3 and GPT-J still supports that the results primarily reflect synthesis capability. revision: yes

  2. Referee: [§4] §4 (main results and baselines): The performance gap versus GPT-3 and GPT-J is presented as evidence of Codex's superiority, yet the manuscript provides limited detail on whether identical prompt formatting, temperature, and stop-sequence settings were used for all three models; any mismatch would weaken the cross-model comparison that underpins the central claim.

    Authors: Identical prompt formatting, temperature, and stop-sequence settings were used for GPT-3, GPT-J, and Codex. We will add an explicit statement in the revised §4 clarifying these shared evaluation settings to strengthen the comparison. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; direct empirical measurements on external benchmark

full rationale

The paper reports pass rates on the newly released HumanEval benchmark as direct measurements of a fine-tuned model's functional correctness. No equations, first-principles derivations, or parameter fits are presented whose outputs are then renamed as predictions. The central claims rest on external test cases and unit tests whose construction is independent of the reported numbers, with no self-citation load-bearing steps or self-definitional reductions. This is the standard case of an evaluation paper whose results are falsifiable against the released benchmark.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Empirical evaluation paper; no mathematical axioms, free parameters, or invented entities are introduced beyond standard ML training assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models Trained on Code." pith.science (2026). https://pith.science/paper/E7CAGJ73

@misc{pith2026210703374,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models Trained on Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7CAGJ73}},
  note         = {Machine review of arXiv:2107.03374}
}
read the original abstract

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

Figures

Figures reproduced from arXiv: 2107.03374 by the authors.

Figure 1
Figure 1. Pass rates of our models on the HumanEval dataset as a function of model size. When a single sample is generated for each problem, GPT-12B solves no problems, but Codex (fine-tuned on code) solves 28.8% of the problems, and Codex-S (further fine-tuned on correctly implemented standalone functions) solves 37.7% of the problems. From here, further gains can be realized by generating 100 samples per problem and selecti… view at source ↗
Figure 2
Figure 2. Three example problems from the HumanEval dataset, where the probabilities that a single sample from Codex-12B passes unit tests are 0.9, 0.17, and 0.005. The prompt provided to the model is shown with a white background, and a successful model-generated completion is shown in a yellow background. Though not a guarantee for problem novelty, all problems were hand-written and not programmatically copied from existing… view at source ↗
Figure 3
Figure 3. A numerically stable script for calculating an unbiased estimate of pass@k. Later, we provide evidence that BLEU score may not be a reliable indicator of functional correctness by showing that functionally inequivalent programs generated by our model (which are guaranteed to disagree with the reference solution on some input) often have higher BLEU scores than functionally equivalent ones [PITH_FULL_IMAGE:figures/f… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Model cross-entropy test loss measured on a held-out split of our Python GitHub code corpus. The smooth power law scaling of performance with model size observed in GPT-3 appears to hold even after code fine-tuning. model test loss follows a power law in model size (Ka…
Figure 5
Figure 5. Figure 5: In the top panel, we plot pass@k against the number of samples (k) for various temperature settings. Higher temperatures are better when the number of samples is large, likely due to the increased sample diversity. In the bottom panel, we plot the best temperature sett…
Figure 6
Figure 6. Figure 6: Using the optimal temperatures 0.2 and 0.8 for pass@1 and pass@100, we plot these two metrics as a function of model size. Performance appears to scale smoothly as a sigmoid in log￾parameters [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Model performance in the setting where we can generate multiple samples, but only evaluate one. We can do better than ran￾domly selecting a sample by choosing the solution with the highest mean log-probability (red) or with the highest back-translation score (orange) d…
Figure 9
Figure 9. Figure 9: Optimal sampling temperatures as a function of the num￾ber of samples generated for both Codex and Codex-S. Codex-S generally requires a higher temperature for any particular value of k, possibly to compensate for the fact that it models a narrower distribution [PITH_…
Figure 10
Figure 10. Figure 10: Comparing Codex-S against Codex on the metrics pro￾posed in Section 3. Codex-S is one or two orders of magnitude more parameter efficient on pass@1 and pass@100, and log-prob sample ranking with Codex-S yields similar benefits over random sampling that Codex does. 5. …
Figure 11
Figure 11. Figure 11: Pass rates of Codex-12B samples against the number of chained components in the synthetically generated docstring. With each additional component, pass rate drops by roughly a factor of 2-3. Further, just as text-conditional generative models in other modalities (Rame…
Figure 12
Figure 12. Figure 12: When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of. This persists when the prompt also includes instructions to write correct code. This gap increases with model size. forward to provide documentation to users reminding them …
Figure 13
Figure 13. Figure 13: Comparing the amount of bias and variance of two estimators of pass@k. While the top expression may look correct, it underestimates the true value by a considerable margin. The unbiased estimator may have a slightly higher variance initially but allows for a fair comp…
Figure 14
Figure 14. Figure 14: When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of producing. This gap increases with model size. Including an instruction to write correct code helps a little but does not fix the problem. Even with no examples in the contex…
Figure 15
Figure 15. Figure 15: Clearly insecure encryption keys produced by Codex. When asked to create encryption keys, Codex models select clearly insecure configuration parameters in a significant fraction of cases. We evaluated outputs as clearly insecure if: (a) RSA keys were shorter than 2048…

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    We introduce Codex... On HumanEval... pass@k := E[1−(n−c k)/(n k)]... fine-tune GPT models... on 159 GB GitHub Python

  • IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    HumanEval... 164 hand-written problems... functional correctness via unit tests

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Showing 60 of 2,793 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 2,793 Pith citations

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    cs.CL 2022-01 accept novelty 9.0 of 10

    Chain-of-thought prompting, by including intermediate reasoning steps in few-shot examples, elicits strong reasoning abilities in large language models on arithmetic, commonsense, and symbolic tasks.

  2. ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

    cs.CL 2026-09 accept novelty 8.0 of 10

    ProcArena is the first multi-scenario, direct and interactive benchmark for evaluating LLMs on NL-to-PL/SQL, comprising 3,998 executable tasks across two SQL dialects.

  3. One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

    cs.LG 2026-09 accept novelty 8.0 of 10

    AnLR-LoRA assigns per-rank-one adaptive learning rates to LoRA modules using function-space velocity and Adam SNR, consistently outperforming standard LoRA and its variants.

  4. Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them

    cs.SE 2026-09 accept novelty 8.0 of 10

    A functional census of 327 quantized LLM artifacts reveals five silently defective files in the official Ollama library and documents backend-dependent failures that no registry currently tests for.

  5. Learning What to Retain: Gated-Memory Routing for Efficient Collaboration in Multi-Agent LLM Systems

    cs.AI 2026-08 accept novelty 8.0 of 10

    Gated-Memory Routing uses learned write and retrieval gates to maintain a compact execution memory for multi-agent LLM coordination, outperforming prior routing paradigms in accuracy and efficiency.

  6. Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

    cs.AI 2026-08 conditional novelty 8.0 of 10

    Text-only LLMs can reliably write code to draw an image when given its full geometry, but their ability to compose a spatial layout from a brief description varies and depends on both the model and the output medium.

  7. WoE Wrote It? Watermarking Mixture-of-Experts LLMs for Black-Box Text Provenance

    cs.CR 2026-08 conditional novelty 8.0 of 10

    WoE embeds a pseudorandom green-list bias into selected expert weights of MoE LLMs via LoRA, enabling black-box text provenance from routing-trajectory reconstruction.

  8. ProRetrieval: Learning to Orchestrate Hybrid Search via Executable Program Synthesis

    cs.IR 2026-08 accept novelty 8.0 of 10

    A 4B-parameter model trained via reinforcement learning to synthesize executable retrieval programs achieves 80.9% Hit@1 on e-commerce and 90.9% on email, surpassing GPT-5.5 and Claude Opus 4.7.

  9. Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

    cs.AI 2026-08 conditional novelty 8.0 of 10

    A counterexample-guided loop automatically writes a small Python metric that improves held-out code selection accuracy by about 15% of the reachable range without per-candidate model calls.

  10. Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues

    cs.AI 2026-08 conditional novelty 8.0 of 10

    Revoked constraints still shape model behavior at an 8B operating point, and compiling the net constraint state ahead of time removes the observed relapse.

  11. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

    cs.AI 2026-08 conditional novelty 8.0 of 10

    A training-free speculative decoder using the verifier's hidden states as a semantic key retrieves correct continuations unreachable by exact suffix matching, lifting accepted length 24-29%.

  12. Evading Chain-of-Thought Monitoring Through Model Poisoning

    cs.CR 2026-08 conditional novelty 8.0 of 10

    Supervised fine-tuning can implant triggered backdoors whose chain-of-thought traces look clean, evading CoT-only monitors and forcing a consistency-check view of monitoring.

  13. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  14. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 8.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  15. RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

    cs.SE 2026-06 unverdicted novelty 8.0 of 10

    RigorBench is the first benchmark for process discipline in autonomous AI coding agents, reporting 41% higher process quality scores and 17% higher outcome correctness when agents follow structured engineering practices.

  16. Sumi: Open Uniform Diffusion Language Model from Scratch

    cs.CL 2026-06 unverdicted novelty 8.0 of 10

    Sumi is an openly released 7B parameter uniform diffusion language model pretrained from scratch on 1.5T tokens that matches autoregressive models on several benchmarks.

  17. PCB-QA: Evaluating LLMs over the First Printed Circuit Board Design Question-Answer Dataset

    cs.AR 2026-06 unverdicted novelty 8.0 of 10

    PCB-QA is the first QA benchmark for LLMs on printed circuit board designs, with Gemini 3 Flash Preview reaching 93% accuracy on a JSON textual representation.

  18. TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    TheoremBench is a Lean4 benchmark of classical theorems in main and premised forms that evaluates LLM provers on partial progress, coverage, and token efficiency rather than binary success on competition problems.

  19. Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

    cs.AI 2026-05 unverdicted novelty 8.0 of 10

    New benchmark evaluates three frontier deep research agents on 42 SME prompts with verifiers and rubrics, reporting low acceptance rates of 9.5-21.4% and agent-specific failure modes.

  20. Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 8.0 of 10

    Mistletoe is a stealthy attack that collapses the speedup of speculative decoding by reducing average accepted length τ without changing output semantics or perplexity.

  21. CIDR: A Large-Scale Industrial Source Code Dataset for Software Engineering Research

    cs.SE 2026-05 unverdicted novelty 8.0 of 10

    CIDR is a large-scale curated dataset of proprietary industrial source code repositories spanning 138 languages and 373 million lines of code, collected via formal agreements with industry partners.

  22. PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation

    cs.AI 2026-05 unverdicted novelty 8.0 of 10

    PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.

  23. SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning

    cs.AI 2026-05 unverdicted novelty 8.0 of 10

    SimWorld Studio uses a self-evolving coding agent to generate adaptive 3D environments that improve embodied agent performance, with reported gains of 18 points over fixed environments in navigation tasks.

  24. PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

    cs.AI 2026-05 conditional novelty 8.0 of 10

    PhysicianBench is a new benchmark of 100 physician-reviewed, execution-grounded tasks in live EHR environments where the best LLM agent reaches only 46% success and open-source models reach 19%.

  25. Can Coding Agents Reproduce Findings in Computational Materials Science?

    cs.SE 2026-05 conditional novelty 8.0 of 10

    AutoMat benchmark shows current LLM coding agents achieve at most 54.1% success when reproducing computational materials science claims from papers.

  26. StabilizerBench: A Benchmark for AI-Assisted Quantum Error Correction Circuit Synthesis

    quant-ph 2026-04 conditional novelty 8.0 of 10

    StabilizerBench is a new benchmark for evaluating AI agents on generating, optimizing, and making fault-tolerant stabilizer circuits for quantum error correction, with efficient verification and multi-tier scoring.

  27. Gradient-Based Program Synthesis with Neurally Interpreted Languages

    cs.LG 2026-04 unverdicted novelty 8.0 of 10

    NLI autonomously discovers a vocabulary of primitive operations and interprets variable-length programs via a neural executor, allowing end-to-end training and gradient-based test-time adaptation that outperforms prio...

  28. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC

    cs.AR 2026-04 unverdicted novelty 8.0 of 10

    LLM agents autonomously evolve the ABC logic synthesis tool by iteratively rewriting its source code to achieve better quality-of-results on standard benchmarks while preserving the original interface.

  29. DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    DF3DV-1K supplies 1,048 scenes with clean and cluttered image pairs plus a challenging 41-scene subset to benchmark and improve distractor-free radiance field methods.

  30. FermiLink: A Unified Agent Framework for Multidomain Autonomous Scientific Simulations

    physics.chem-ph 2026-04 conditional novelty 8.0 of 10

    FermiLink is a unified AI agent framework that automates multidomain scientific simulations via separated package knowledge bases and a four-layer progressive disclosure mechanism, reproducing 56% of target figures in...

  31. Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems

    cs.CR 2026-04 unverdicted novelty 8.0 of 10

    DDIPE poisons LLM agent skills by embedding malicious logic in documentation examples, achieving 11.6-33.5% bypass rates across frameworks while explicit attacks are blocked, with 2.5% evading detection.

  32. Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

    cs.LG 2026-03 unverdicted novelty 8.0 of 10

    Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.

  33. MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

    cs.SE 2026-01 unverdicted novelty 8.0 of 10

    MCP-Atlas is a new benchmark with 1000 tasks on production MCP servers that uses claim-level scoring to evaluate LLM agents on realistic multi-step tool-use competency.

  34. Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

    cs.AI 2025-09 unverdicted novelty 8.0 of 10

    CritPt benchmark shows state-of-the-art LLMs reach only 5.7% average accuracy on full-scale unpublished physics research tasks, rising to about 10% with coding tools.

  35. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering

    cs.SE 2025-07 conditional novelty 8.0 of 10

    AIDev is a new open dataset of 456k AI-agent pull requests showing agents submit code faster than humans but with lower acceptance rates and simpler changes.

  36. Large Language Diffusion Models

    cs.CL 2025-02 unverdicted novelty 8.0 of 10

    LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.

  37. ProSec: Fortifying Code LLMs with Proactive Security Alignment

    cs.CR 2024-11 conditional novelty 8.0 of 10

    ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.

  38. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

    cs.CV 2024-09 accept novelty 8.0 of 10

    Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.

  39. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    cs.AI 2024-08 unverdicted novelty 8.0 of 10

    The AI Scientist framework enables LLMs to independently conduct the full scientific process from idea generation to paper writing and review, demonstrated across three ML subfields with papers costing under $15 each.

  40. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

    cs.SE 2024-03 conditional novelty 8.0 of 10

    RepairAgent autonomously repairs 164 bugs on Defects4J including 39 not fixed by prior techniques by treating an LLM as an agent that invokes tools via a finite state machine and dynamic prompts.

  41. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

    cs.CL 2023-08 unverdicted novelty 8.0 of 10

    LongBench is the first bilingual multi-task benchmark for long context understanding in LLMs, containing 21 datasets in 6 categories with average lengths of 6711 words (English) and 13386 characters (Chinese).

  42. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

    cs.CL 2023-04 conditional novelty 8.0 of 10

    API-Bank is a new benchmark and training dataset for tool-augmented LLMs that shows fine-tuned models can approach GPT-3.5 tool-use effectiveness.

  43. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    cs.CL 2023-04 accept novelty 8.0 of 10

    Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.

  44. PAL: Program-aided Language Models

    cs.CL 2022-11 conditional novelty 8.0 of 10

    PAL improves few-shot reasoning accuracy by having LLMs generate executable programs rather than text-based chains of thought, outperforming much larger models on math and logic benchmarks.

  45. Code as Policies: Language Model Programs for Embodied Control

    cs.RO 2022-09 accept novelty 8.0 of 10

    Language models generate robot policy code from natural language commands via few-shot prompting, enabling spatial-geometric reasoning, generalization, and precise control on real robots.

  46. Show Your Work: Scratchpads for Intermediate Computation with Language Models

    cs.LG 2021-11 unverdicted novelty 8.0 of 10

    Training language models to generate intermediate computation steps on a scratchpad enables them to perform multi-step tasks such as long addition and arbitrary program execution that they otherwise fail at.

  47. TruthfulQA: Measuring How Models Mimic Human Falsehoods

    cs.CL 2021-09 unverdicted novelty 8.0 of 10

    A new benchmark reveals that language models including GPT-3 are truthful on only 58% of questions designed to elicit popular misconceptions, far below human performance of 94%, with larger models performing worse.

  48. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    cs.CL 2026-07 conditional novelty 7.5 of 10

    Across ~38,000 hours on 134 ultra-long real-world tasks, aggregate agent performance follows a log-sigmoid of interaction time, and measured learning speed doubles about every three months.

  49. Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

    cs.SE 2026-07 conditional novelty 7.5 of 10

    Across 35 Qwen Code releases with a fixed LLM, resolve rates do not significantly improve while token use and tool calls roughly double, driven by feature-heavy releases and high-risk components.

  50. GraphAHA: Graph-Based Adaptive Search with Heterogeneous Actions for Test-Time Code Generation

    cs.SE 2026-09 accept novelty 7.0 of 10

    GraphAHA merges equivalent programs into a directed acyclic graph and uses hierarchical Thompson sampling to adaptively choose between sampling, reasoning, and repair actions during test-time code generation, improvin...

  51. Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning

    cs.LG 2026-09 accept novelty 7.0 of 10

    CluSTER combines gradient-space clustering with DP-aware balanced allocation and weighted gradient updates to reduce training time up to 69.6% without significant accuracy loss across instruction-tuning datasets.

  52. Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

    cs.CL 2026-09 conditional novelty 7.0 of 10

    Verbalized confidence surpasses log-probability based soft scoring as the better signal for evaluating outputs with the newest proprietary LLMs, reversing the pre-2025 best practice.

  53. Retrofitting Code Using LLMs to Support Exceptional Behavior

    cs.SE 2026-09 accept novelty 7.0 of 10

    EXCODER retrofits exception-related code into Java methods using LLMs guided by static and dynamic program analysis from exceptional behavior tests.

  54. Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward

    cs.AI 2026-09 accept novelty 7.0 of 10

    The paper identifies the verification gap as the binding constraint on LLM reasoning and proposes proof-carrying cognition, where reasoning is verified against a world model trained on reality settlement.

  55. Benchmarking Hybrid Deep Research Across Database Querying and Web Search

    cs.CL 2026-09 accept novelty 7.0 of 10

    HybridDeepResearch is the first benchmark requiring agents to integrate web search and SQL queries to solve complex analytical tasks, showing that current models fail frequently at this cross-modal reasoning.

  56. How effective are traditional test criteria at detecting bugs in large language models generated code?

    cs.SE 2026-09 accept novelty 7.0 of 10

    Traditional test adequacy criteria (statement, branch, mutation) are ineffective at detecting faults in LLM-generated code, with fault detection rates near one to two percent.

  57. Training-Free Task Vectors for LLM Behavioral Control

    cs.LG 2026-09 conditional novelty 7.0 of 10

    Training-Free Task Vectors map activation steering directions to rank-one weight updates using SVD and expected input, enabling compositional behavioral editing without training.

  58. What Fixed-Rollout pass@k Evaluations Can Identify

    stat.ML 2026-09 accept novelty 7.0 of 10

    Fixed-n rollout counts identify only n moments of the latent task difficulty, making extrapolated pass@k generically nonidentifiable for k > n.

  59. Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference

    cs.AI 2026-09 accept novelty 7.0 of 10

    HeRo uses a linear-attention memory across model depth to make routing decisions aware of prior skips, improving accuracy-efficiency trade-offs in LLM inference.

  60. CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows

    cs.AI 2026-09 accept novelty 7.0 of 10

    CausalVerify shows that for four econometric causal designs, the best evaluation signal is whether executed model code matches a canonical estimate, not whether the text description agrees with a consensus label.

See all 2,793 Pith citations

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 2,793 Pith papers (see all)

  1. [1]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

    PMLR. URL http://proceedings.mlr.press/ v37/allamanis15.html. Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., and Church, G. M. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315–1322, 2019. Alon, U., Brody, S., Levy, O., and Yahav, E. code2seq: Gener- ating sequences from structured r...

    work page Pith review arXiv 2019

  2. [2]

    Generating Long Sequences with Sparse Transformers

    URL https://www.usenix.org/conference/ Evaluating Large Language Models Trained on Code usenixsecurity21/presentation/carlini- extracting. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In In- ternational Conference on Machine Learning, pp. 1691–1703. PMLR, 2020. Child, R., Gray, S., Radf...

    work page Pith review arXiv 2020

  3. [3]

    Clarkson, M

    URL https://www.alignmentforum.org/ posts/ZeE7EKHTFMBs8eMxn/clarifying-ai- alignment. Clarkson, M. R., Finkbeiner, B., Koleini, M., Micinski, K. K., Rabe, M. N., and S´anchez, C. Temporal logics for hyperproper- ties. In International Conference on Principles of Security and Trust, pp. 265–284. Springer, 2014. Clement, C., Drain, D., Timcheck, J., Svyatko...

    work page 2014

  4. [4]

    Lazy Prices

    URL https://www.youtube.com/watch?v= fMym BKWQzk. Crawford, K. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, 2021. Dai, A. M. and Le, Q. V . Semi-supervised sequence learning. Advances in neural information processing systems, 28:3079– 3087, 2015. Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D...

    work page Pith review arXiv doi:10.1145/362566.362568 2021

  5. [5]

    1 <= len(arr) <= 100

    work page

  6. [6]

    Number of elements are less than k

    1 <= k <= len(arr) """ ### COMPLETION 1 (WRONG): ### pass ### COMPLETION 2 (WRONG): ### for i in range (0, k): if len (str(arr[i])) < 3: k = k - 1 continue else: return arr[i] ### COMPLETION 3 (WRONG): ### sum_ = 0 i = 0 while k > 0 and i < len(arr): if len (str(arr[i])) <= 2: sum_ += arr[i] k -= 1 i += 1 return sum_ ### COMPLETION 4 (WRONG): ### size = l...

    work page 2020

  7. [7]

    "" ### COMPLETION 1 (WRONG): ### return x if n % x == 0 else y ### COMPLETION 2 (WRONG): ### if n > 1: return x if n%2 != 0 else y else: return

    returned tuple has the number of even and odd integer palindromes respectively. """ Evaluating Large Language Models Trained on Code ### COMPLETION 1 (WRONG): ### if n == 1: return 0, 0 elif n == 0: return 0, 0 else: return (1 + 2 * (n % 2), 2 * (n - 1)) ### COMPLETION 2 (WRONG): ### count_even = 0 count_odd = 0 for i in range (1, n+1): s = str(i) #even i...

    work page

  8. [8]

    remove all instances of the letter e from the string

    “remove all instances of the letter e from the string” s = s.replace("e", "")

    work page

Show all 22 references
  1. [9]

    replace all spaces with exclamation points in the string

    “replace all spaces with exclamation points in the string” s = s.replace(" ", "!")

  2. [10]

    convert the string s to lowercase

    “convert the string s to lowercase” s = s.lower()

  3. [11]

    remove the first and last two characters of the string

    “remove the first and last two characters of the string” s = s[2:-2]

  4. [12]

    removes all vowels from the string

    “removes all vowels from the string” s = "".join(char for char in s if char not in "aeiouAEIOU")

  5. [13]

    remove every third character from the string

    “remove every third character from the string” s = "".join(char for i, char in enumerate(s) if i % 3 != 0)

  6. [14]

    drop the last half of the string, as computed by char- acters

    “drop the last half of the string, as computed by char- acters” s = s[: len(s) // 2]

  7. [15]

    replace spaces with triple spaces

    “replace spaces with triple spaces” s = s.replace(" ", " ")

  8. [16]

    reverse the order of words in the string

    “reverse the order of words in the string” s = " ".join(s.split()[::-1])

  9. [17]

    drop the first half of the string, as computed by num- ber of words

    “drop the first half of the string, as computed by num- ber of words” s = " ".join(s.split()[len(s.split ()) // 2 :])

  10. [18]

    add the word apples after every word in the string

    “add the word apples after every word in the string” s = " ".join(word + " apples" for word in s.split())

  11. [19]

    make every other character in the string uppercase

    “make every other character in the string uppercase” s = "".join(char.upper() if i % 2 == 0 else char for i, char in enumerate(s)) Evaluating Large Language Models Trained on Code

  12. [20]

    delete all exclamation points, question marks, and periods from the string

    “delete all exclamation points, question marks, and periods from the string” s = "".join([x for x in s if x not in ".!?"]) These building blocks can be easily composed by concate- nating their one-line descriptions into a docstring and by concatenating their one-line implement...

    2021

  13. [21]

    When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of producing

    We consider a model capable of some task X if it has Evaluating Large Language Models Trained on Code Figure 14. When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of producing. This gap increases with model size. Including an instructio...

  14. [22]

    10") 10 >>> closest_integer(

    We say a model is intent misaligned if it outputs B, in some case where the user would prefer it outputs A, and where the model is both: (a) capable of outputting A instead, and (b) capable of distinguishing between situations where the user wants it to do A and situations whe...

    2020

Pith tools

Reviewed May 24, 2026 · model on record in the stance chip above.