Pith. sign in

REVIEW 4 major objections 5 minor 97 references

This paper presents PROBE, a multi-dimensional code-generation benchmark, and claims that even the strongest current language models remain far from reliable: their one-shot correctness never exceeds 70 percent and falls to 20 percent on th

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:35 UTC pith:Q673SXFM

load-bearing objection A solid, well-documented benchmarking framework with a genuine empirical contribution, but the Qwen2.5-in-the-loop ground truth for 15% of problems is a real vulnerability that should be addressed before the field builds on the precise numbers. the 4 major comments →

arxiv 2607.13820 v1 pith:Q673SXFM submitted 2026-07-15 cs.SE

PROBE: Benchmarking Code Generation in Large Language Models

Rodrigo Pato Nogueira , Marco Vieira , Jo\~ao R. Campos This is my paper
classification cs.SE
keywords LLM code generationbenchmark frameworkpass@kfunctional correctnesscode qualityprogram synthesisunit test generationmulti-language evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PROBE is a benchmarking framework that evaluates language models on code generation along three dimensions at once: whether generated programs pass unit tests, how close they come to valid reference solutions, and how their cyclomatic complexity and length compare with human-written code. Applying it to six models, five programming languages, and three prompting strategies, the paper tries to establish that current models are not production-ready: the best model's one-shot success rate never exceeds 70 percent, and on the hardest problems even the strongest model solves only one in five. The paper also argues that problem difficulty, not prompting, is the dominant factor: feedback from failed executions buys a modest +0.05 pass@k by fixing compile errors, while one in-context example does essentially nothing. The motivating concern is that generated code fails in basic, avoidable ways — missing imports, out-of-bounds accesses, integer overflows, and brute-force algorithms — so developers who trust the output can land vulnerabilities or slowdowns in real systems.

Core claim

On the paper's own terms, the central discovery is that LLM code generation is predictable in its failure: performance is strongly ordered by model scale, by programming language (Python easiest, Rust hardest), and by problem difficulty, and no model comes close to reliable. Across the whole benchmark the best pass@1 is 0.70, and at the hardest difficulty level the strongest model drops to 0.20. The three-dimensional evaluation shows that correctness, similarity to reference solutions, and code quality are partially independent: correctness and proximity both collapse with difficulty, but the quality gap between generated and human-written solutions stays roughly flat, because on hard proble

What carries the argument

The load-bearing piece is PROBE's workload: 1,651 competitive-programming problems, each with three to ten unit tests, three to 250 validated reference solutions in the two main languages, and an execution-time limit set at three times the slowest reference solution. Correctness is measured by pass@k and an outcome-rate breakdown; proximity by a token, syntax-tree, and dataflow similarity score; quality by cyclomatic complexity and lines of code, computed only for correct solutions and scaled relative to references. Problem difficulty comes from clustering Python reference solutions on static metrics into four levels, a proxy that the paper validates by showing performance declines monotonic

Load-bearing premise

The correctness ground truth rests in part on LLM-generated unit tests (255 of 1,651 problems) that are kept only when three reference solutions agree; if those tests miss real edge cases, every pass@k and outcome rate shifts, and the benchmark's rankings could change.

What would settle it

Take the 255 problems with synthetically generated tests, have human experts write additional edge-case tests from the problem constraints, and re-run the six models; if pass@1 drops by more than a few points or a model that previously passed all tests fails a new test, the reported reliability figures are partly an artifact of test generation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the PROBE results are representative, no current model can be trusted for unattended code generation: the ceiling is 70 percent one-shot correctness, and harder, more realistic tasks push the best model to 20 percent.
  • A feedback loop based on compiler and test errors is a reliable, if small, lever: roughly +0.05 pass@k, mostly by fixing compilation errors, with the largest gains in strict compiled languages like Rust, C, and C++.
  • In-context examples add little when the prompt already spells out input/output format and constraints, so prompt-engineering effort in this setting is better spent on feedback.
  • Generated code tends to be shorter and structurally simpler than human reference solutions except in Python, and this quality gap does not widen with difficulty — meaning simpler output should not be read as better output.
  • The same benchmark infrastructure can be extended to new languages, metrics, and problem types, and its unit-test and reference-solution thresholds define when proximity and quality metrics are meaningful.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The difficulty proxy is only validated by monotone performance decline; an independent human difficulty rating on a subset would test whether the clustering tracks intrinsic problem hardness rather than implementation style.
  • Because the proximity metric scores the maximum similarity against available references, it likely understates how close a correct but stylistically divergent solution is; the Python results, in particular, may reflect implementation diversity rather than worse proximity.
  • Unit tests were generated without explicitly targeting edge cases; adding constraint-derived adversarial tests could push the reported pass rates below the already-low ceiling.
  • The finding that one-shot in-context learning is neutral suggests that the value of demonstrations may depend on prompt detail; in sparser real-world prompts, in-context learning could matter more than this benchmark suggests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PROBE, an extensible benchmarking framework for evaluating LLM code generation. The framework is constructed from 1,651 IBM CodeNet problems across five programming languages, and evaluates generated code along three dimensions: functional correctness (pass@k and Outcome Rate), proximity to valid solutions (CodeBLEU), and code quality (cyclomatic complexity and NLOC). Six models (four open-source, two proprietary) are evaluated under three prompting conditions (baseline, 1-shot ICL, and two-round feedback incorporation). The main reported findings are that model scale correlates with performance but even the best models remain far from reliable (pass@1 ≤ 0.70); Rust is the hardest language; feedback incorporation improves pass@k by about +0.05 on average while ICL has negligible effect; performance degrades sharply with problem difficulty; and a manual error analysis reveals basic, avoidable failures such as missing imports, integer overflows, out-of-bounds accesses, and inefficient brute-force algorithms.

Significance. If the quantitative results are supported, PROBE is a useful contribution to the software-engineering benchmarking literature. Its strengths include a public dataset and repository, a multi-dimensional evaluation design that goes beyond unit-test pass rates, a broad multilingual workload, a reproducible Docker-based procedure, and concrete, instructive error examples. The central qualitative claim — that current LLMs remain unreliable for code generation across languages and difficulty levels — is plausible and likely robust. However, several load-bearing methodological choices and reporting issues affect the precision of the headline numbers and the strength of the per-language, per-difficulty, and per-prompt comparisons. The paper deserves publication after those issues are addressed.

major comments (4)
  1. [§3.3.3, §6.2] The functional-correctness ground truth for 255 of 1,651 problems is generated by Qwen2.5-72B and validated only by consensus among reference solutions drawn from the same CodeNet pool. As the authors acknowledge in §6.2, this does not eliminate the risk of missing edge cases; the consensus filter cannot detect a shared blind spot, and the reported line/branch/mutation coverage (97.1/94.1/88.64%) is measured against those same reference implementations. Because Qwen2.5-family models are among the evaluated systems, there is a potential for differential bias on exactly the models the paper compares. This is load-bearing for the precise claim pass@1 ≤ 0.70 and for the per-language/per-difficulty comparisons. Please provide a sensitivity analysis: report pass@k and OutcomeRate separately for problems with original versus Qwen2.5-generated tests, and re-run the headline analyses excluding th
  2. [§4, §5.1.4] Problem difficulty is operationalized as k-means clustering (k=5, merged to 4) over Python static-complexity features of reference solutions. The only validation offered for this proxy is the same benchmark's observed monotone performance decline across difficulty levels. This is circular: the difficulty labels are derived from the reference solutions and then 'validated' by the pass-rate pattern on those same problems. The monotone decline is not an independent confirmation. Please validate the difficulty levels with an external signal (e.g., CodeNet/AtCoder difficulty metadata, acceptance rates, or human ratings), or reframe the levels as 'implementation complexity' and temper the difficulty-based claims accordingly. As written, the difficulty analysis in §5.1.4 rests on an unvalidated proxy.
  3. [§4, §5.1.2–§5.1.3] All estimates are based on N=5 samples per problem, but no confidence intervals, bootstrap intervals, or per-problem paired comparisons are reported. For ICL, differences of ±0.01 in Table 3 are entirely within sampling noise; the claim that ICL has 'negligible impact' is therefore not statistically supported as stated. Similarly, the feedback gain of +0.05 should be accompanied by a measure of uncertainty, and the cross-model and cross-language comparisons in Tables 2 and 5 need variance estimates. This is load-bearing for the prompting-strategy conclusions. Please add confidence intervals or equivalent uncertainty quantification for the main pass@k and OutcomeRate figures.
  4. [Table 6(f), Table 5(a)] Table 6(f), labeled 'No Code', contains exactly the same values as Table 6(d), 'Runtime Error', for every model and difficulty level. This is inconsistent with Table 2, where the No Code rate is 0.00 for all models and languages. The table appears to be an error and must be corrected; the difficulty-level analysis for No Code needs to be regenerated. In addition, Table 5(a) shows non-monotonic pass@1 values at difficulty level 3 (e.g., deepseek-coder-v2 0.04 at level 2 vs 0.06 at level 3; qwen2.5-coder:7b 0.02 vs 0.03). While the text only claims a large drop from level 0 to level 3, the individual values should be verified and, if they are correct, their sampling variability acknowledged.
minor comments (5)
  1. [§3.1.2] The text says 'generating complete programming problems'; this should presumably be 'generating complete programs' or 'complete solutions to programming problems'.
  2. [§3.3.1] The auxiliary model is described as 'Qwen2.5 model (72B parameters, 4-bit quantized)'. Please specify the exact variant (e.g., Qwen2.5-72B-Instruct) and quantization configuration to support reproducibility.
  3. [§5.3.2] The ICL and feedback effects on code quality are stated to be negligible, but no tables or numeric values are provided. For reproducibility, report the results in an appendix or provide a link to the full output data.
  4. [§5.4] The manual error analysis is based on 'approximately 30 samples per failure category' but does not state how the samples were selected or how many reviewers were involved and how disagreements were resolved. A brief description of the sampling and review procedure would strengthen the error analysis.
  5. [Tables 3–6] Model names are inconsistently capitalized (e.g., 'qwen2.5-coder:7b' vs 'Qwen-2.5:14B'). Please standardize the names and ensure the column/row labels in the difficulty tables clearly identify the model and metric.

Circularity Check

0 steps flagged

No significant circularity: PROBE is an empirical benchmark; the flagged Qwen2.5-assisted test generation and k-means difficulty proxy are acknowledged validity threats, not by-construction reductions.

full rationale

The paper's chain is measurement, not derivation. pass@k, OutcomeRate, CodeBLEU and code-quality scores are computed directly from execution or similarity against human-written reference solutions; no headline result is fitted from the metric definitions. The two loops that a reviewer might call circular are explicitly flagged by the authors themselves: Qwen2.5-generated unit tests for 255/1,651 problems (§3.3.3: 'While generating synthetic unit tests carries some risk, this was done only for the subset of problems that lacked sufficient tests initially (255 out of 1,651)') and difficulty clustering based on reference-solution static metrics (§4: 'these metrics capture implementation complexity rather than intrinsic problem difficulty'; §6.2: 'should be interpreted as a proxy rather than a ground-truth measure'). Neither is a circular step: generated test inputs are annotated by executing three reference implementations, not by the evaluated models, and difficulty labels are fitted to human-written reference complexity, not to model pass rates. The in-sample monotone performance decline is used only as a sanity check of the difficulty proxy; the labels are not defined from performance, so 'harder problems' is an empirical correlation, not a tautology. The only self-citation (Vieira & Madeira 2003) supports generic benchmark-design desiderata and carries no load in the empirical claims. Thus no prediction reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 8 axioms · 0 invented entities

The benchmark's quantitative conclusions rest on the construction choices listed above. Most are disclosed as threats to validity, but none is independently calibrated against external ground truth, and no sensitivity analyses are provided.

free parameters (5)
  • Difficulty cluster count and features = k=5 clusters (merged to 4 levels); features: mean cyclomatic complexity, effort, SLOC of Python reference solutions, z-s
    Chosen after visual inspection of cluster separation (§4, Problem Clustering); all difficulty-trend results depend on this grouping.
  • Code quality score sensitivity = β_i = γ_i = 1 / M_ref_i
    Ad hoc scaling in Eqs. (4)-(5) to balance absolute and proportional deviations; it determines all code-quality comparisons.
  • Execution time limit multiplier = 3× the slowest reference solution's longest-running test
    Chosen by hand to prevent infinite loops (§3.3.4); directly determines Timeout outcome rates.
  • Workload selection thresholds = ≥3 reference solutions in Python and C++; 3-10 unit tests per problem; ≥3 refs for proximity/quality; max 250 refs
    These thresholds fix the 1,651-problem dataset and which languages have proximity/quality scores (§4).
  • Unit-test generation policy = Qwen2.5-generated inputs retained if 3 reference solutions agree; repeat until 4 tests per problem
    Used for 255/1,651 problems; part of the correctness ground truth (§3.3.3).
axioms (8)
  • standard math The pass@k unbiased estimator in Eq. (1) is valid.
    Standard estimator from Chen et al. (2021), correctly quoted.
  • domain assumption The Python CodeBLEU library correctly computes CodeBLEU as defined by Ren et al. (2020).
    All proximity scores rely on this external tool; the paper does not ship an independent implementation (§3.5.3).
  • domain assumption CodeNet problems are a representative text-to-code workload.
    Stated in §3.3 and §6.2; contest-style self-contained I/O problems may not represent repository-level or interactive coding.
  • domain assumption CodeNet reference solutions filtered by unit tests and outlier removal are valid ground truth.
    All proximity and quality scores, and the expected outputs of generated tests, depend on this (§3.3.2).
  • domain assumption Qwen2.5-generated unit tests validated by three-reference agreement adequately capture correctness.
    255/1,651 problems use synthetic tests; missing edge cases could shift pass rates (§3.3.3).
  • domain assumption The evaluated models did not meaningfully memorize CodeNet problems during pretraining.
    Argued from low overall performance in the threats-to-validity section (§6.2), but no contamination-controlled comparison is made.
  • ad hoc to paper Implementation-complexity clustering approximates intrinsic problem difficulty.
    Introduced in §4; validated only by monotone performance degradation within the same benchmark.
  • ad hoc to paper The asymmetric saturation score in Eq. (4) with β=γ=1/M_ref is a meaningful code-quality distance.
    Introduced for this paper; no external validation that equal complexity implies equal quality.

pith-pipeline@v1.3.0-alltime-deepseek · 30563 in / 15133 out tokens · 132362 ms · 2026-08-02T03:35:59.254960+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks, particularly in automated code generation. Despite their widespread adoption, these models remain far from perfect, making systematic and fair evaluation essential to understand their strengths and limitations. In the context of code generation, existing benchmarks are limited: they often target a single programming language and rely primarily on unit test outcomes, while overlooking other critical dimensions such as the overall quality of the generated code and its closeness to a valid solution. To address these gaps, we introduce PROBE, an extensible benchmark framework that, unlike prior work, establishes a systematic structure built on diverse and well-defined metrics, representative workloads, varied prompt templates, and a robust experimental procedure. In practice, the code generated by the LLMs is evaluated along three complementary dimensions: functional correctness, proximity to valid solutions, and code quality, enabling a comprehensive assessment of performance. We use PROBE to evaluate four open-source and two proprietary models under three prompting strategies across five programming languages. We further complement this analysis with a study of common errors in the code and provide concrete examples, offering clearer insight into where LLMs tend to struggle. Our findings show that, while LLMs achieve promising results, they struggle with harder problems and, in the case of smaller models, with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.

Figures

Figures reproduced from arXiv: 2607.13820 by Jo\~ao R. Campos, Marco Vieira, Rodrigo Pato Nogueira.

Figure 1
Figure 1. Figure 1: Design of the PROBE Benchmarking Framework. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pass@k values for the different models and languages [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Code Quality Metrics reference implementations. C++ and Java exhibit a similar pattern, though with smaller magnitudes, while C shows moderate positive deviations across all models except GPT-4.1-mini, whose scores remain nearly neutral. We believe these differences are not solely attributable to the models but also reflect characteristics of the underlying dataset. For example, reference Python solutions … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

97 extracted references · 1 canonical work pages

  1. [1]

    Oppenheimer, David and Ganapathi, Archana and Patterson, David , year =. Why do. 4th

  2. [2]

    , title =

    Manna, Zohar and Waldinger, Richard J. , title =. 1971 , issue_date =. doi:10.1145/362566.362568 , journal =

    arXiv 1971

  3. [3]

    , title =

    Summers, Phillip D. , title =. 1977 , issue_date =. doi:10.1145/321992.322002 , journal =

    arXiv 1977

  4. [4]

    Backus, J. W. and Beeber, R. J. and Best, S. and Goldberg, R. and Haibt, L. M. and Herrick, H. L. and Nelson, R. A. and Sayre, D. and Sheridan, P. B. and Stern, H. and Ziller, I. and Hughes, R. A. and Nutt, R. , title =. 1957 , isbn =. doi:10.1145/1455567.1455599 , booktitle =

    arXiv 1957

  5. [5]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  6. [6]

    Alice and Rice, Andrew and Rifkin, Devon and Simister, Shawn and Sittampalam, Ganesh and Aftandilian, Edward , title =

    Ziegler, Albert and Kalliamvakou, Eirini and Li, X. Alice and Rice, Andrew and Rifkin, Devon and Simister, Shawn and Sittampalam, Ganesh and Aftandilian, Edward , title =. 2022 , isbn =. doi:10.1145/3520312.3534864 , booktitle =

    arXiv 2022

  7. [7]

    2024 , eprint=

    Qwen2.5-Coder Technical Report , author=. 2024 , eprint=

    2024

  8. [8]

    5 Technical Report , author=

    Qwen2. 5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  9. [9]

    2024 , publisher=

    Llama 3.1: An in-depth analysis of the next-generation large language model , author=. 2024 , publisher=

    2024

  10. [10]

    Language Models are Few-Shot Learners , url =

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winte...

  11. [11]

    2021 , eprint=

    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation , author=. 2021 , eprint=

    2021

  12. [12]

    2021 , eprint=

    Program Synthesis with Large Language Models , author=. 2021 , eprint=

    2021

  13. [13]

    2023 , eprint=

    xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval , author=. 2023 , eprint=

    2023

  14. [14]

    2021 , eprint=

    CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks , author=. 2021 , eprint=

    2021

  15. [15]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  16. [16]

    Long Short-Term Memory

    Graves, Alex. Long Short-Term Memory. Supervised Sequence Labelling with Recurrent Neural Networks. 2012. doi:10.1007/978-3-642-24797-2_4

    work page doi:10.1007/978-3-642-24797-2_4 2012

  17. [17]

    L. R. Medsker and L. Jain , title =. Design and Applications , volume =

  18. [18]

    2024 , eprint=

    A Survey on Evaluating Large Language Models in Code Generation Tasks , author=. 2024 , eprint=

    2024

  19. [19]

    2002 , publisher =

    Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing , title =. 2002 , publisher =. doi:10.3115/1073083.1073135 , booktitle =

    arXiv 2002

  20. [20]

    Text summarization branches out , pages=

    Rouge: A package for automatic evaluation of summaries , author=. Text summarization branches out , pages=

  21. [21]

    Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments , author=. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

  22. [22]

    2020 , eprint=

    CodeBLEU: a Method for Automatic Evaluation of Code Synthesis , author=. 2020 , eprint=

    2020

  23. [23]

    1992 , publisher=

    Benchmark handbook: for database and transaction processing systems , author=. 1992 , publisher=

    1992

  24. [24]

    Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis , url =

    Gupta, Kavi and Christensen, Peter Ebert and Chen, Xinyun and Song, Dawn , booktitle =. Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis , url =

  25. [25]

    2024 , eprint=

    Improving Code Generation by Training with Natural Language Feedback , author=. 2024 , eprint=

    2024

  26. [26]

    2023 , eprint=

    Teaching Large Language Models to Self-Debug , author=. 2023 , eprint=

    2023

  27. [27]

    2024 , eprint=

    Is Self-Repair a Silver Bullet for Code Generation? , author=. 2024 , eprint=

    2024

  28. [28]

    2017 , volume =

    Foundations and Trends® in Programming Languages , title =. 2017 , volume =. doi:10.1561/2500000010 , issn =

    doi:10.1561/2500000010 2017

  29. [29]

    Predicting a Correct Program in Programming by Example

    Singh, Rishabh and Gulwani, Sumit. Predicting a Correct Program in Programming by Example. Computer Aided Verification. 2015

    2015

  30. [30]

    SPoC: Search-based Pseudocode to Code , url =

    Kulal, Sumith and Pasupat, Panupong and Chandra, Kartik and Lee, Mina and Padon, Oded and Aiken, Alex and Liang, Percy S , booktitle =. SPoC: Search-based Pseudocode to Code , url =

  31. [31]

    Neural Attribution for Semantic Bug-Localization in Student Programs , url =

    Gupta, Rahul and Kanade, Aditya and Shevade, Shirish , booktitle =. Neural Attribution for Semantic Bug-Localization in Student Programs , url =

  32. [32]

    2021 , eprint=

    Unit Test Case Generation with Transformers and Focal Context , author=. 2021 , eprint=

    2021

  33. [33]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. Journal of Machine Learning Research , year =

  34. [34]

    Distributed Representations of Words and Phrases and their Compositionality , url =

    Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg S and Dean, Jeff , booktitle =. Distributed Representations of Words and Phrases and their Compositionality , url =

  35. [35]

    Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages=

    Glove: Global vectors for word representation , author=. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages=

    2014

  36. [36]

    and Fei-Fei, Li , title =

    Russakovsky, Olga and Deng, Jia and Su, Hao and Krause, Jonathan and Satheesh, Sanjeev and Ma, Sean and Huang, Zhiheng and Karpathy, Andrej and Khosla, Aditya and Bernstein, Michael and Berg, Alexander C. and Fei-Fei, Li , title =. International Journal of Computer Vision , volume =. 2015 , month =. doi:10.1007/s11263-015-0816-y , url =

    doi:10.1007/s11263-015-0816-y 2015

  37. [37]

    2024 , eprint=

    Code Llama: Open Foundation Models for Code , author=. 2024 , eprint=

    2024

  38. [38]

    2019 , eprint=

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. 2019 , eprint=

    2019

  39. [39]

    2021 , eprint=

    Measuring Coding Challenge Competence With APPS , author=. 2021 , eprint=

    2021

  40. [40]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  41. [41]

    2023 , eprint=

    InCoder: A Generative Model for Code Infilling and Synthesis , author=. 2023 , eprint=

    2023

  42. [42]

    2021 , eprint=

    CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation , author=. 2021 , eprint=

    2021

  43. [43]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    A survey on in-context learning , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  44. [44]

    2021 , eprint=

    What Makes Good In-Context Examples for GPT- 3 ? , author=. 2021 , eprint=

    2021

  45. [45]

    2022 , eprint=

    Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity , author=. 2022 , eprint=

    2022

  46. [46]

    , title =

    Winston, Patrick H. , title =. 1980 , issue_date =. doi:10.1145/359038.359042 , journal =

    arXiv 1980

  47. [47]

    2023 , eprint=

    Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers , author=. 2023 , eprint=

    2023

  48. [48]

    2023 , eprint=

    A Latent Space Theory for Emergent Abilities in Large Language Models , author=. 2023 , eprint=

    2023

  49. [49]

    arXiv preprint arXiv:1902.01007 , year=

    Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference , author=. arXiv preprint arXiv:1902.01007 , year=

    Pith/arXiv arXiv 1902

  50. [50]

    2019 , eprint=

    Probing Neural Network Comprehension of Natural Language Arguments , author=. 2019 , eprint=

    2019

  51. [51]

    2022 , eprint=

    TruthfulQA: Measuring How Models Mimic Human Falsehoods , author=. 2022 , eprint=

    2022

  52. [52]

    2023 , eprint=

    ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning , author=. 2023 , eprint=

    2023

  53. [53]

    2020 , eprint=

    RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models , author=. 2020 , eprint=

    2020

  54. [54]

    2023 , eprint=

    Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity , author=. 2023 , eprint=

    2023

  55. [55]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  56. [56]

    Training language models to follow instructions with human feedback , url =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul F and Leike, Jan and Lowe,...

  57. [57]

    2024 , eprint=

    Training Language Models with Language Feedback at Scale , author=. 2024 , eprint=

    2024

  58. [58]

    2023 , eprint=

    Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies , author=. 2023 , eprint=

    2023

  59. [59]

    arXiv preprint arXiv:2305.11747 , year=

    Halueval: A large-scale hallucination evaluation benchmark for large language models , author=. arXiv preprint arXiv:2305.11747 , year=

    Pith/arXiv arXiv

  60. [60]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  61. [61]

    Proceedings 2003 VLDB Conference , pages=

    A dependability benchmark for OLTP application environments , author=. Proceedings 2003 VLDB Conference , pages=. 2003 , organization=

    2003

  62. [62]

    2021 , publisher =

    Sid Black and Leo Gao and Phil Wang and Connor Leahy and Stella Biderman , title =. 2021 , publisher =. doi:10.5281/zenodo.5297715 , url =

    doi:10.5281/zenodo.5297715 2021

  63. [63]

    Wang, Ben and Komatsuzaki, Aran , title =

  64. [64]

    arXiv preprint arXiv:1808.09588 , year=

    Mapping language to code in programmatic context , author=. arXiv preprint arXiv:1808.09588 , year=

    Pith/arXiv arXiv

  65. [65]

    2024 , eprint=

    Large Language Models: A Survey , author=. 2024 , eprint=

    2024

  66. [66]

    2022 , eprint=

    Black-Box Tuning for Language-Model-as-a-Service , author=. 2022 , eprint=

    2022

  67. [67]

    2022 , eprint=

    Compilable Neural Code Generation with Compiler Feedback , author=. 2022 , eprint=

    2022

  68. [68]

    2024 , eprint=

    Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks , author=. 2024 , eprint=

    2024

  69. [69]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  70. [70]

    International conference on machine learning , pages=

    Improving language models by retrieving from trillions of tokens , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  71. [71]

    arXiv preprint arXiv:2012.04584 , year=

    Distilling knowledge from reader to retriever for question answering , author=. arXiv preprint arXiv:2012.04584 , year=

    Pith/arXiv arXiv 2012

  72. [72]

    arXiv preprint arXiv:2305.20050 , year=

    Let's verify step by step , author=. arXiv preprint arXiv:2305.20050 , year=

    Pith/arXiv arXiv

  73. [73]

    arXiv preprint arXiv:2205.12443 , year=

    Generating natural language proofs with verifier-guided search , author=. arXiv preprint arXiv:2205.12443 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2305.06161 , year=

    Starcoder: may the source be with you! , author=. arXiv preprint arXiv:2305.06161 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:1803.02324 , year=

    Annotation artifacts in natural language inference data , author=. arXiv preprint arXiv:1803.02324 , year=

    Pith/arXiv arXiv

  76. [76]

    arXiv preprint arXiv:2408.04667 , year=

    Llm stability: A detailed analysis with some surprises , author=. arXiv preprint arXiv:2408.04667 , year=

    Pith/arXiv arXiv

  77. [77]

    Large Language Models are Zero-Shot Reasoners , url =

    Kojima, Takeshi and Gu, Shixiang (Shane) and Reid, Machel and Matsuo, Yutaka and Iwasawa, Yusuke , booktitle =. Large Language Models are Zero-Shot Reasoners , url =

  78. [78]

    Density-Based Clustering in Spatial Databases: The Algorithm GDBSCAN and Its Applications , journal =

    J. Density-Based Clustering in Spatial Databases: The Algorithm GDBSCAN and Its Applications , journal =. 1998 , month =

    1998

  79. [79]

    arXiv preprint arXiv:2407.06204 , year=

    A survey on mixture of experts , author=. arXiv preprint arXiv:2407.06204 , year=

    Pith/arXiv arXiv

  80. [80]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    Pith/arXiv arXiv

Showing first 80 references.