Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SWE-IF: Aligning Code Evaluation with Human Preference

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Code rankings: instruction-following beats pass@k alone

desk verdict Solid, useful testbed for verifiable code instruction-following, but the headline preference-correlation claim is fit-to-the-same-data and needs real validation before it can carry the 'primary differentiator' conclusion. read the letter →

arxiv 2510.07315 v2 pith:CZPVNULI submitted 2025-10-08 cs.CL cs.AIcs.LGcs.SE

Ming Zhong , Xiang Zhou , Ting-Yun Chang , Qingze Wang , Nan Xu , Xiance Si , Dan Garrette , Shyam Upadhyay
show 4 more authors
Jeremiah Liu Jiawei Han Benoit Schillings Jiao Sun
This is my paper
classification cs.CLcs.AIcs.LGcs.SE
keywords vibecodinginstructionfollowingcodeevaluationhumanpreferenceverifiableinstructionspass@kLLMbenchmarkfunctionalregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the 'vibe check' human coders perform when accepting AI-generated code is not captured by pass@k alone. Its central claim is that instruction following — adhering to non-functional constraints such as coding style, logic patterns, documentation, and API conventions — is the missing piece that distinguishes models humans actually prefer. Using a taxonomy of 30 verifiable code instructions with deterministic verifiers, the authors augment two established benchmarks into a testbed and evaluate 31 LLMs. They find that combining functional correctness with instruction following correlates better with real human preference rankings than either metric alone, and that instruction following is the primary differentiator on real-world programming tasks. If true, this shifts code evaluation from 'does it pass tests' to 'does it do what a human would ask for.'

What carries the argument

VeriCode, a taxonomy of 30 verifiable code instructions across five categories (coding style, logic and code patterns, documentation, error handling, and library/API constraints). Each instruction has a binary deterministic verifier implemented with the Ruff linter, AST analysis, or regular expressions, plus configurable parameters that expand the set into hundreds of variants. These instructions are injected into BigCodeBench and LiveCodeBench to form the SWE-IF testbed, which the body calls Vibe Checker, evaluated under single-turn generation and multi-turn editing. The paper's headline quantity is the composite score α·IF + (1−α)·Func, with α swept between 0 and 1 to find the best match t

What would settle it

Take a held-out set of human preference votes not used in the leaderboard and compute the correlation between (a) pure functional correctness and (b) the composite 0.4·IF + 0.6·Func on the same models. If (a) matches or beats (b), the claim that instruction following adds predictive value beyond functionality is false. A simpler check: swap the instruction set for arbitrary constraints users do not care about; if the composite still correlates equally well, the effect is not specific to these non-functional instructions.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurable decomposition of human preference in coding: a composite score α·IF + (1−α)·Func, where IF is an instruction-following score from VeriCode and Func is pass@1, peaks in correlation with the coding Elo ratings of a large human-preference leaderboard at intermediate α for both benchmarks. On real-world programming tasks, pure IF correlates more than 0.1 points higher than pure Func (Spearman), making instruction following the key differentiator among models. The authors position this as evidence that non-functional, verifiable constraints are an overlooked axis of code quality, and that models suffer both functional regression and poor multi-instruc

Load-bearing premise

The whole preference claim rests on the human-vote leaderboard's coding Elo being a clean measure of human preference and on the two augmented benchmarks representing the tasks behind those votes; if style or interface artifacts, or a task mismatch, drive the correlation, the conclusion that instruction following is the missing factor collapses.

Editorial extensions

If this is right

  • Under five added instructions, even the strongest model satisfies all of them only 46.75% of the time on real-world tasks; success rates decay roughly exponentially as instructions accumulate.
  • Adding non-functional instructions that do not target functionality still lowers pass@1 for every model, with average regression reaching about 5.8–6.6 percentage points at five instructions.
  • Multi-turn editing follows instructions better but costs more functional correctness; single-turn generation preserves function better but follows fewer instructions.
  • A composite of functional correctness and instruction following correlates more strongly with human preference than either metric alone, with optimal IF weights around 0.4 (Pearson) and 0.6–0.7 (Spearman).
  • Instruction position matters: single-turn shows a primacy bias, multi-turn shows a recency bias, and real-world tasks show a U-shaped 'lost-in-the-middle' pattern even in short prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the composite score is a better proxy for human preference than pass@k, then leaderboards and reward models built only on unit tests likely under-reward models that follow conventions and intent; training with verifiable instruction rewards could yield models that feel better to users without sacrificing correctness.
  • The taxonomy is Python-only and derived from linter rules; applying the same logic to other languages (e.g., TypeScript, Rust) and to code review comments would test whether instruction following's role generalizes.
  • The optimal IF weight changes between Pearson and Spearman correlations, hinting that the right blend depends on how much to credit the strongest models; user-level studies would be needed to pin down a single mix.
  • Because instructions are chosen by an LLM selector from a fixed set of 30, the testbed measures compliance with explicit, linter-checkable constraints; implicit preferences such as naming sense or idiomaticity remain outside the taxonomy, so 'vibe' may be broader than what IF captures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses 'vibe check' in code generation, hypothesizing that instruction following (IF) is the missing component beyond functional correctness. It constructs VeriCode, a taxonomy of 30 verifiable code instructions with deterministic linter/AST verifiers, and augments BigCodeBench and LiveCodeBench into BigVibeBench/LiveVibeBench. It evaluates 31 LLMs in single-turn and multi-turn settings, reporting functional regression, IF decay under multiple instructions, and position bias. It then correlates LMArena coding Elo with a composite score αIF+(1−α)Func and claims that the optimal mixture predicts human preference better than either metric alone, with IF the primary differentiator on real-world programming tasks.

Significance. If substantiated, the contribution is useful: a verifiable taxonomy is a concrete step beyond pass@k, and the regression/IF findings across 31 models are of interest. Strengths include deterministic verifiers, a public release commitment, use of established benchmarks, and parallel instruction-level/task-level IF measurement in both single-turn and multi-turn settings. However, the central preference claim is not yet supported by the statistical evidence as presented. The paper's current value lies more in the testbed and descriptive findings than in the demonstration that IF is 'the missing piece' of human preference.

major comments (3)
  1. [§4.5, Figure 5] The optimal mixing weight α is selected by scanning α∈[0,1] on exactly the same 24–31 model-level LMArena Elo points used to report the peak correlation. No bootstrap, cross-validation, or significance test is reported. With n≈30 and Elo noise, the 'clear margin' of the mixture over pure IF or pure Func may be a selection artifact. Please report bootstrap confidence intervals for the peak-vs-single-metric difference, or a permutation test, and/or validate the chosen α on a held-out split or a second preference dataset.
  2. [§4.5, 'IF as primary differentiator'] The claim that IF is the primary differentiator is based on comparing univariate correlations of pure IF and pure Func. If IF and Func are correlated with each other and with overall model capability, the univariate comparison does not establish the unique contribution of IF. Please add partial correlations or a nested-model comparison (e.g., does adding IF to Func significantly improve correlation with Elo?) and report the relevant confidence intervals.
  3. [§4.5, Appendix E.2] The external-validity assumption that LMArena coding Elo is a valid, relatively confound-free proxy for human preference on the tasks in this testbed is not defended. The augmented benchmarks (BigCodeBench/LiveCodeBench with 30 linter-derived instructions) may not match the distribution of tasks behind the arena votes; style/UI effects may drive Elo even with style control. Please provide a task-distribution/instruction-distribution comparison or robustness analyses on Copilot Arena or another preference dataset, and state what evidence would falsify the composite-preference claim.
minor comments (5)
  1. [Abstract / Title] The abstract and GitHub name use 'SWE-IF', while the body and headers use 'Vibe Checker'. Please align the naming throughout.
  2. [Intro / §4.2, Table 2] The claim that 'pass@1 decreases across all models' is contradicted by Table 2 (e.g., Claude 4 Opus on BigVibeBench single-turn has negative regression values) and the full results in Table 5. Please qualify as 'on average' or reconcile the wording with the data.
  3. [Table 4 / §4.5] Several models in Table 4 (e.g., Gemini 2.0 Flash) have no Elo rating. Please state explicitly which models enter each correlation in Figure 5 and Appendix E.2, and the final n for each panel.
  4. [§4.5] The choice of IF from the single-turn 'one instruction' setting for the correlation analysis needs a justification; why not multi-turn or 5-instruction IF? A sentence explaining this choice would help.
  5. [Figure 5 / Appendix E.2] Adding error bars or confidence bands directly in Figure 5 would help the reader judge the 'clear margin' between the mixture and the single-metric correlations.

Circularity Check

2 steps flagged · score 6.0 of 10

Optimal-mixture correlation is an in-sample argmax over the same LMArena Elo data; taxonomy difficulty filter also selects on an evaluated model's IF.

  1. fitted input called prediction [Section 4.5 (Correlating with Human Preference), Figure 5; Conclusion §6; Appendix E.2]
    "We then evaluate a composite score 𝛼IF+(1−𝛼)Func with 𝛼∈[0,1], and report correlations. ... In all cases, the mixture outperforms either isolated metric by a clear margin."

    The composite is defined by a free weight 𝛼, and 𝛼 is chosen by scanning 𝛼∈[0,1] to maximize the correlation with the same LMArena coding Elo ratings used to report the result (24–31 model-level points, Appendix E.2). Because the reported peak is the maximum of a scan that includes 𝛼=0 and 𝛼=1, the selected composite is guaranteed to correlate at least as well in-sample as pure IF or pure Func; 'outperforms ... by a clear margin' is therefore an argmax artifact unless validated out-of-sample. The Conclusion's 'predicts human preference substantially better' is a restatement of the fitting objective, not an independent test.

  2. other [Section 2.2, Difficulty Filtering]
    "Using Gemini 2.5 Flash (Gemini Team, 2025) on a challenging test set, BigCodeBench-Hard (Zhuo et al., 2025), we measure instruction following rate alongside functional correctness at pass@1. Any instruction with a success rate above 90% and no degradation in pass@1 is removed."

    Gemini 2.5 Flash is one of the 31 models evaluated later (Table 4). The curation rule removes instructions that this model already satisfies (IF >90% with no pass@1 cost), so the paper's headline that 'even the strongest models struggle to comply with multiple instructions' is partly enforced by construction for the filtering model. Since the same IF scores feed the Figure 5 human-preference correlation, the 'IF is primary differentiator' result rests on an outcome-filtered instrument, though the external Elo data keeps this from being fully definitional.

full rationale

The measurements are not internally rigged: VeriCode is grounded in external linters (Ruff), the augmented benchmarks use established BigCodeBench/LiveCodeBench tasks, and the human-preference anchor (LMArena coding Elo) is external to the paper's fits and testbed construction. So the core 'IF is missing' hypothesis has independent empirical content, and there is no self-citation uniqueness chain or ansatz-by-citation. The circularity is concentrated in §4.5: α is scanned on the same 24–31 Elo points used to evaluate the composite, so the reported peak is the maximum of correlations that necessarily includes α=0 and α=1. Claiming that the resulting mixture 'outperforms either isolated metric by a clear margin' and 'predicts human preference substantially better' is in-sample selection; the only non-tautological empirical component is that the argmax is interior, and that is never tested for significance or on a holdout. The §2.2 difficulty filter compounds the issue by removing instructions that Gemini 2.5 Flash (an evaluated model) already handles, so part of the IF variance feeding Figure 5 is selected on an outcome. These are partial, not complete, circularities: the external Elo anchor and external base benchmarks keep the result from being definitionally forced, so score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

Central claim rests on (1) using LMArena coding Elo as ground-truth human preference; (2) assuming VeriCode's linter-based categories capture the non-functional dimensions users care about; (3) an LLM-selector (Claude 4 Opus) choosing instruction subsets/parameters for each task; (4) a fitted mixture weight α in the headline correlation. The testbed itself is not circular with the preference data, but the α optimization is a fitting loop and the difficulty filter was tuned on Gemini 2.5 Flash.

free parameters (4)
  • alpha (IF weight in composite score) = 0.4 (Pearson), 0.7 (Spearman) on BigVibeBench; similar on LiveVibeBench
    Chosen by maximizing correlation with LMArena Elo on the same data; supports the claim that a mixture beats either metric, but with no held-out validation or confidence interval.
  • Difficulty-filter thresholds in taxonomy construction = success rate >90% and no pass@1 degradation => removed
    Hand-selected thresholds in §2.2 that shape which 30 instructions survive; affects difficulty of the whole testbed.
  • LiveVibeBench model-exclusion threshold = >10% error rate
    Appendix D.1: models with persistent failures >10% are excluded from LiveVibeBench analysis, changing the model set used for correlation.
  • Instruction parameter values = LLM-generated per task (e.g., line_length, max_branches)
    Selected by Claude 4 Opus then rule-validated; not fit to outcomes, but they set instruction difficulty and are another hand/LLM choice.
assumptions (4)
  • domain assumption LMArena coding Elo ratings are a valid proxy for human 'vibe check' preference.
    Section 4.5 uses Elo as the only human-preference signal; if style/UI confounds dominate, the correlation conclusion is unsupported.
  • domain assumption VeriCode's 30 linter-derived instructions span the non-functional dimensions users actually screen for.
    Section 2.2 sources from Ruff and expert review; no behavioral validation that these are the dimensions behind vibe check.
  • domain assumption Linter/AST verifiers correctly classify instruction compliance on generated code.
    Section B.1: 27/30 verifiers are Ruff rule wrappers; false positives/negatives on unconventional-but-correct code are not analyzed.
  • domain assumption LLM-selected instruction subsets are relevant and non-conflicting for each task.
    Section 3.1: selection done by Claude 4 Opus scanning a random permutation; if selection is biased, all downstream metrics inherit that bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-IF: Aligning Code Evaluation with Human Preference." pith.science (2026). https://pith.science/paper/CZPVNULI

@misc{pith2026251007315,
  author       = {Pith},
  title        = {Pith review of: SWE-IF: Aligning Code Evaluation with Human Preference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZPVNULI}},
  note         = {Machine review of arXiv:2510.07315}
}
read the original abstract

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    Frontier LLMs pass unit tests over 76% of the time on debugging tasks but achieve edit precision below 45%, indicating regeneration rather than precise debugging.

  2. Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    The Precise Debugging Benchmark reveals that frontier LLMs achieve over 76% unit-test pass rates but below 45% edit precision when debugging, often regenerating rather than making minimal fixes.

Reference graph

Works this paper leans on

49 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introducing claude 3.5, 2024

    Anthropic . Introducing claude 3.5, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  2. [2]

    Introducing claude 4, 2025

    Anthropic . Introducing claude 4, 2025. URL https://www.anthropic.com/news/claude-4

    2025

  3. [3]

    Austin, A

    J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [4]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herb...

    arXiv 2021

  5. [5]

    W. Chi, V. Chen, A. N. Angelopoulos, W.-L. Chiang, A. Mittal, N. Jain, T. Zhang, I. Stoica, C. Donahue, and A. Talwalkar. Copilot arena: A platform for code LLM evaluation in the wild. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=9bYOqwtAud

    2025

  6. [6]

    Chiang, L

    W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net...

    2024

  7. [7]

    Chowdhury, J

    N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, E. Mays, R. Dias, M. Aljubeh, M. Glaese, C. E. Jimenez, J. Yang, L. Ho, T. Patwardhan, K. Liu, and A. Madry. Introducing SWE -bench verified. OpenAI Blog, 2024. URL https://openai.com/index/introducing-swe-bench-verified/

    2024

  8. [8]

    J. Da, C. Wang, X. Deng, Y. Ma, N. Barhate, and S. Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards. CoRR, abs/2506.11425, 2025. doi:10.48550/ARXIV.2506.11425. URL https://doi.org/10.48550/arXiv.2506.11425

    arXiv doi:10.48550/arxiv.2506.11425 2025

Show all 49 references
  1. [9]

    Deepseek-v3 technical report

    DeepSeek - AI . Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024. doi:10.48550/ARXIV.2412.19437. URL https://doi.org/10.48550/arXiv.2412.19437

    2024 doi arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek - AI . Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948, 2025. doi:10.48550/ARXIV.2501.12948. URL https://doi.org/10.48550/arXiv.2501.12948

    2025 doi arXiv

  3. [11]

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. CoRR, abs/2308.01861, 2023. doi:10.48550/ARXIV.2308.01861. URL https://doi.org/10.48550/arXiv.2308.01861

    2023 doi arXiv

  4. [12]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gemini Team . Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025. doi:10.48550/ARXIV.2507.06261. URL https://doi.org/10.48550/arXiv.2507.06261

    2025 doi arXiv

  5. [13]

    Gemma 3 technical report

    Gemma Team . Gemma 3 technical report. Mar. 2025. doi:10.48550/arXiv.2503.19786. URL https://arxiv.org/abs/2503.19786. First posted March 25, 2025

    2025 doi arXiv

  6. [14]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with APPS . In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track...

    2021

  7. [15]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, A. Yang, R. Men, F. Huang, X. Ren, X. Ren, J. Zhou, and J. Lin. Qwen2.5-coder technical report. CoRR, abs/2409.12186, 2024. doi:10.48550/ARXIV.2409.12186. URL https://doi.org/10.48550/arXiv.2...

    2024 doi arXiv

  8. [16]

    N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar - Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, S...

    2025

  9. [17]

    Jiang, Y

    Y. Jiang, Y. Wang, X. Zeng, W. Zhong, L. Li, F. Mi, L. Shang, X. Jiang, Q. Liu, and W. Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meetin...

    2024 doi

  10. [18]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net,...

    2024

  11. [19]

    Karpathy

    A. Karpathy. Vibe coding --- wikipedia, 2025. URL https://en.wikipedia.org/wiki/Vibe_coding

    2025

  12. [20]

    Kimi k2: Open agentic intelligence

    Kimi Team . Kimi k2: Open agentic intelligence. July 2025. doi:10.48550/arXiv.2507.20534. URL https://arxiv.org/abs/2507.20534

    2025 doi arXiv

  13. [21]

    Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. I. Wang, and T. Yu. DS-1000: A natural and reliable benchmark for data science code generation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Internationa...

    2023

  14. [22]

    T. Li, A. Angelopoulos, and W.-L. Chiang. Does style matter? disentangling style and substance in chatbot arena, August 2024. URL https://blog.lmarena.ai/blog/2024/style-control/

    2024

  15. [23]

    J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Process...

    2023

  16. [24]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024. doi:10.1162/tacl_a_00638. URL https://aclantholo...

    2024 doi

  17. [25]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    MiniMax . Minimax-m1: Scaling test-time compute efficiently with lightning attention. June 2025. doi:10.48550/arXiv.2506.13585. URL https://arxiv.org/abs/2506.13585

    2025 doi arXiv

  18. [26]

    Mistral medium 3, 2025

    Mistral AI . Mistral medium 3, 2025. URL https://mistral.ai/news/mistral-medium-3

    2025

  19. [27]

    u ndler, M. N. M \

    N. M \" u ndler, M. N. M \" u ller, J. He, and M. T. Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Sy...

    2024

  20. [28]

    Gpt-4o system card

    OpenAI . Gpt-4o system card. CoRR, abs/2410.21276, 2024. doi:10.48550/ARXIV.2410.21276. URL https://doi.org/10.48550/arXiv.2410.21276

    2024 doi arXiv

  21. [29]

    Openai o3 and o4-mini system card

    OpenAI . Openai o3 and o4-mini system card. Technical report, OpenAI, Apr. 2025. URL https://openai.com/index/o3-o4-mini-system-card/

    2025

  22. [30]

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer. The impact of AI on developer productivity: Evidence from github copilot. CoRR, abs/2302.06590, 2023. doi:10.48550/ARXIV.2302.06590. URL https://doi.org/10.48550/arXiv.2302.06590

    2023 doi arXiv

  23. [31]

    Pyatkin, S

    V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi. Generalizing verifiable instruction following. CoRR, abs/2507.02833, 2025. doi:10.48550/ARXIV.2507.02833. URL https://doi.org/10.48550/arXiv.2507.02833

    2025 doi arXiv

  24. [32]

    Y. Qin, K. Song, Y. Hu, W. Yao, S. Cho, X. Wang, X. Wu, F. Liu, P. Liu, and D. Yu. Infobench: Evaluating instruction following ability in large language models. In L. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024,...

    2024 doi

  25. [33]

    Qwen3 technical report

    Qwen Team . Qwen3 technical report. CoRR, abs/2505.09388, 2025. doi:10.48550/ARXIV.2505.09388. URL https://doi.org/10.48550/arXiv.2505.09388

    2025 doi arXiv

  26. [34]

    S. I. Ross, F. Martinez, S. Houde, M. J. Muller, and J. D. Weisz. The programmer's assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI 2023, Sydney, ...

    2023

  27. [35]

    Singhal, T

    M. Singhal, T. Aggarwal, A. Awasthi, N. Natarajan, and A. Kanade. Nofuneval: Funny how code lms falter on requirements beyond functional correctness. CoRR, abs/2401.15963, 2024. doi:10.48550/ARXIV.2401.15963. URL https://doi.org/10.48550/arXiv.2401.15963

    2024 doi arXiv

  28. [36]

    Ai | 2025 stack overflow developer survey, 2025

    Stack Overflow . Ai | 2025 stack overflow developer survey, 2025. URL https://survey.stackoverflow.co/2025/ai. Survey report

    2025

  29. [37]

    Z. Wang, J. Jiang, H. Zhou, W. Zheng, X. Zhang, C. Bansal, and H. Yao. Verifiable format control for large language model generations. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexic...

    2025 doi

  30. [38]

    Willison

    S. Willison. Not all ai-assisted programming is vibe coding (but vibe coding rocks), Mar. 2025. URL https://simonwillison.net/2025/Mar/19/vibe-coding/. Blog post, Simon Willison’s Weblog

    2025

  31. [39]

    Grok 3 beta — the age of reasoning agents, 2025 a

    xAI . Grok 3 beta — the age of reasoning agents, 2025 a . URL https://x.ai/news/grok-3

    2025

  32. [40]

    Grok 4 model card

    xAI . Grok 4 model card. 2025 b . URL https://data.x.ai/2025-08-20-grok-4-model-card.pdf

    2025

  33. [41]

    K. Yan, H. Guo, X. Shi, S. Cao, D. Di, and Z. Li. C ode IF : Benchmarking the instruction-following capabilities of large language models for code generation. In G. Rehm and Y. Li, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics...

    2025 doi

  34. [42]

    J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual...

    2023

  35. [43]

    J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. Wang, and O. Press. Swe-bench multimodal: Do AI systems generalize to visual software domains? In The Thirteenth International Conference on Lea...

    2025

  36. [44]

    D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang. Multi-swe-bench: A multilingual benchmark for issue resolving. CoRR, abs/2504.02605, 2025. doi:10.48550/ARX...

    2025 doi arXiv

  37. [45]

    Zhang, S

    L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and Y. Zhang. Swe-bench goes live! CoRR, abs/2505.23419, 2025. doi:10.48550/ARXIV.2505.23419. URL https://doi.org/10.48550/arXiv.2505.23419

    2025 doi arXiv

  38. [46]

    W. Zhao, N. Jiang, C. Lee, J. T. Chiu, C. Cardie, M. Gall \' e , and A. M. Rush. Commit0: Library generation from scratch. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https://openre...

    2025

  39. [47]

    Zheng, Z

    Z. Zheng, Z. Cheng, Z. Shen, S. Zhou, K. Liu, H. He, D. Li, S. Wei, H. Hao, J. Yao, P. Sheng, Z. Wang, W. Chai, A. Korolova, P. Henderson, S. Arora, P. Viswanath, J. Shang, and S. Xie. Livecodebench pro: How do olympiad medalists judge llms in competitive programming? CoRR, ab...

    2025 doi arXiv

  40. [48]

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. CoRR, abs/2311.07911, 2023. doi:10.48550/ARXIV.2311.07911. URL https://doi.org/10.48550/arXiv.2311.07911

    2023 doi arXiv

  41. [49]

    T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, J. Hoang, A. R. Zebaze, X. Hong, W. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, and et al. Bigcodebench: Benchmarking code generation with diverse function c...

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.