Pith. sign in

REVIEW 4 major objections 6 minor 47 references

StackEval: Benchmarking LLMs in Coding Assistance

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that coding assistance can be benchmarked at scale with two Stack Overflow-derived datasets, and that an LLM judge using chain-of-thought plus a reference answer reaches 84.4% accuracy in deciding whether a generated…

desk verdict Useful benchmark artifacts but the main acceptance rates rest on an unnamed LLM judge validated on only 136 questions; the StackUnseen rankings are not yet supported. read the letter →

arxiv 2412.05288 v1 pith:X7YDW5XK submitted 2024-11-21 cs.SE cs.CLcs.LG

Nidhish Shah , Zulkuf Genc , Dogu Araci This is my paper · ORCID
classification cs.SEcs.CLcs.LG
keywords StackEvalLLM-as-a-judgecodingassistancebenchmarkOverflowacceptancescorechain-of-thoughtself-preferenceUnseen
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that coding-assistance quality can be measured at scale with two new benchmarks built from Stack Overflow. StackEval contributes 925 curated question-answer pairs across 25 languages and four task types, and StackUnseen contributes recent questions released after typical LLM training cutoffs, specifically to expose how models handle new material. The paper's central evaluation claim is that an LLM judge given the question, the candidate answer, the accepted Stack Overflow answer, and chain-of-thought instructions can decide whether the candidate answer is acceptable, matching human experts 84.4% of the time on a validated subset of 136 questions. If this is right, coding assistants no longer need human annotation for every evaluation run; acceptance rates from the judge can compare models across languages, task types, and difficulty levels. The reader should care because the benchmarks measure the kind of day-to-day assistance developers actually ask for, not just function completion.

What carries the argument

The load-bearing mechanism is the acceptance-score rubric implemented as an LLM judge prompt. A generated answer is scored 0 to 3 across accuracy, completeness, and relevance, with scores 2 and 3 counted as acceptable; before scoring, the judge is prompted to analyze the question, the accepted Stack Overflow reference answer, and the candidate answer step by step. The reference answer functions as an objective anchor, and the human-annotated 136-question subset is the calibration step that converts the judge's scores into a validity claim.

What would settle it

Take a fresh random sample of about 100 questions from StackEval that are not in the 136-question judge benchmark, have human experts score the same model outputs on the 0-3 rubric, and compare their binary acceptability labels with the LLM judge's labels; an agreement rate substantially below the reported 84.4%, or agreement that varies sharply by language or task type, would show the benchmark rankings do not generalize.

Watch

Extended reading notes

Core claim

The central discovery is that the reference answer is the strongest ingredient in an LLM judge for coding assistance. Across six judge models, adding the accepted Stack Overflow answer to the prompt improved evaluation accuracy for every model, while chain-of-thought alone sometimes hurt; the best configuration, chain-of-thought plus reference answer, reached 84.4% mean accuracy for GPT-4 Turbo. Using this judge, the paper reports acceptance rates for 24 models on StackEval, where O1 Preview leads at 95.5%, and on StackUnseen, where the same model leads at 83.0%; every tested model's acceptance rate drops on the recent questions, and the drop is smaller for models that score higher on StackEval. The paper also finds no statistically significant self-preference bias in coding evaluation when a reference answer is provided, and only marginal evidence of one without it.

Load-bearing premise

The acceptance rates reported for StackEval and StackUnseen are produced by an LLM judge that the paper never identifies, and the judge's reliability was validated on only 136 human-annotated questions before being extrapolated to all 925 StackEval and all StackUnseen questions.

Editorial extensions

If this is right

  • Combining chain-of-thought with a reference answer is the recommended judge configuration, and reference answers matter more than reasoning prompts alone.
  • Overall model rankings on StackEval, StackUnseen, HumanEval, and ChatBot Arena are highly correlated, so a single multi-task benchmark can summarize relative capability.
  • Models with higher StackEval acceptance rates show smaller performance drops on StackUnseen, implying that strong performance on established problems predicts better adaptation to new problems.
  • LLM judges do not exhibit statistically significant self-preference when a reference answer is present, so automated coding evaluations can be run without a bias correction for self-scoring.
  • Because StackUnseen is updated semi-annually, it can serve as a re-usable check for whether new model releases have improved on post-training-cutoff content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up is to run the same reference-answer judge on other objectively graded domains, such as mathematics or legal Q&A, to see whether the anchoring effect that removes self-preference generalizes beyond code.
  • Because StackUnseen samples questions from after model training cutoffs, the size of the acceptance-rate drop could be tested as a practical leakage detector for future base models.
  • The judge's accuracy was measured on a comparatively small annotated subset; re-running the benchmark on a larger human-annotated sample is the direct way to check whether the 84.4% figure holds.
  • The dataset structure invites a challenge-suite interpretation: teams could filter by language and task type to target specific weaknesses, such as advanced C++ debugging, and track improvement over releases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces two coding-assistance benchmarks derived from Stack Overflow: StackEval (925 questions, 25 languages, four task types) and StackUnseen (recent questions updated semi-annually), together with an LLM-as-a-Judge benchmark of 136 human-annotated questions. The authors evaluate four judge prompt configurations and report that GPT-4 Turbo with chain-of-thought and a reference answer achieves 84.4% accuracy against human acceptability labels. They then use an LLM judge to compute acceptance rates for 24 models on StackEval and StackUnseen, examine performance degradation on recent content, and test for self-preference bias. Datasets and evaluation code are released publicly.

Significance. If the judge-based evaluation is reliable, the benchmarks fill a real gap: they cover multiple languages and open-ended coding-assistance tasks with an absolute acceptability metric, complementing execution-based benchmarks like HumanEval and SWE-Bench. StackUnseen is a potentially useful dynamic resource, and the self-preference analysis with reference answers is a worthwhile contribution to the LLM-judge literature. The public release of datasets and code supports reproducibility. However, the paper's central evaluation claims depend on an LLM judge whose identity and transfer to the full StackEval and StackUnseen settings are not established, so the significance is conditional on the authors addressing the validation gaps described below.

major comments (4)
  1. [Section 5.2, Table 3] The LLM judge used to produce every acceptance rate in Table 3 and Table 5 is never identified. Section 5.2 only says that responses were assessed with the LLM-as-a-Judge framework and the prompt in Figure 8, but it does not name the model. The validation in Sections 3.2 and 5.1 covers only 136 human-annotated StackEval questions with answers generated by four models, whereas Table 5 scores 25 models including O1, Gemma, and Nemotron, and StackUnseen contains version-type questions not present in the validation set. Without naming the judge and validating it on the actual evaluation conditions, the reported acceptance rates, the ranking conclusions, and the generalization claims in Figure 4 are not reproducible and are not supported by the presented evidence.
  2. [Section 5.1 vs Section 5.2] Judge accuracy in Section 5.1 is computed on binarized acceptability, collapsing scores 0-1 versus 2-3, while the acceptance rates in Table 3 use the raw 0-3 scores with the threshold between 1 and 2. A judge can achieve the reported binary accuracy while misclassifying many near-threshold answers, which would directly change acceptance rates without changing the binary accuracy. The paper should report calibration at the threshold or show that the model rankings are robust to reasonable perturbations of the threshold, to establish that the acceptance-rate differences in Table 3 are not artifacts of threshold miscalibration.
  3. [Section 3.2] The human annotation process is described as involving a third domain expert who verified the annotations, but no inter-annotator agreement statistic (e.g., Cohen's kappa) and no number of annotators per item are reported. Since the human labels are the sole anchor for the LLM judge accuracy reported in Table 2, the reliability of the 84.4% figure is not established. Agreement statistics should be reported to show that the labels are stable enough to serve as a ground truth for judge evaluation.
  4. [Section 3.1 and Section 1] The leakage-mitigation claim for StackUnseen is asserted without verification. The paper states that recent questions 'mitigate the issue of data leakage' and 'help prevent potential test-train leakage,' but it does not check whether the evaluated models were trained on these Stack Overflow questions, nor does it report any contamination analysis. Moreover, Section 6.2 explicitly warns that the LLM judge may not transfer to newer or different questions, which is precisely the regime in which StackUnseen results are reported. The authors should include a judge-validation subset drawn from StackUnseen itself and report leakage checks before claiming that StackUnseen evaluates truly unseen content.
minor comments (6)
  1. [Section 3.1] The word 'Filteration' should be 'Filtration', and the list entry 'V ersion' contains an unintended space.
  2. [Figure 7 caption] The caption contains 'theStackEval' instead of 'the StackEval'.
  3. [Table 3 caption] The caption 'The StackEval and StackUnseen Benchmarks' is not informative; it should say that the table reports acceptance rates on StackEval and StackUnseen for representative models.
  4. [Figure 3] The correlation matrix reports Pearson correlations without confidence intervals or the number of models; given the small sample of models, the high correlations should be accompanied by uncertainty estimates.
  5. [Section 5.3] Table 4 reports only p-values; reporting effect sizes or median score differences would make the self-preference analysis more informative.
  6. [References] Reference [16] cites Mistral 7B, but the text in Section 3.2 refers to 'Mistral Medium'; please verify the citation or clarify the model used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the judge-accuracy result is anchored to independent human labels, and the benchmark acceptance rates are measurements rather than fitted predictions.

full rationale

The paper's central quantitative claim is the 84.4% LLM-judge agreement with human experts (Section 5.1, Table 2). That number is an empirical measurement against a human-annotated set of 136 StackEval questions (Section 3.2), with annotations produced by domain experts and verified by a third expert; it is not derived from the model being evaluated, and no parameter is fitted to the target acceptance rates. The StackEval and StackUnseen acceptance rates in Table 3 are subsequent applications of the LLM-as-a-Judge framework with a fixed rubric and reference answer, not predictions of quantities used to define the benchmark. The self-preference analysis (Section 5.3) uses a Wilcoxon signed-rank test on self-scores versus other-scores and is an independent statistical check rather than a circular justification. The paper itself flags in Section 6.2 that the judge benchmark is a static snapshot and that judge performance may vary on newer or different questions; this is an external-validity and generalization caveat, not circularity. The failure to identify the judge model used for Table 3 is a reproducibility weakness, but it does not make the reported scores circular, because the judge was not calibrated on the scored outputs. No load-bearing self-citation, ansatz-smuggling, or renaming of known results was found.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims of this paper are empirical benchmark results rather than mathematical derivations, so the ledger records the data-quality and evaluation assumptions that the results depend on. The largest burden is the chain of assumptions linking Stack Overflow accepted answers, GPT-4 annotation quality, human annotation reliability, and the transferability of the LLM judge's performance from 136 questions to the full benchmark. No free parameters are fitted to data in the derivation sense; the one hand-set numeric decision is the acceptability threshold.

free parameters (1)
  • Acceptance threshold = score >= 2
    The paper defines acceptable responses as those scoring 2 or 3 on the 0-3 rubric (Section 4). All acceptance rates in Tables 3 and 5 are computed with this cutoff; shifting it to 3 or to 1 would change every acceptance percentage reported.
assumptions (5)
  • domain assumption Accepted Stack Overflow answers with at least one upvote are correct and complete enough to serve as ground truth.
    Section 3.1 selects questions with an accepted answer that has at least one upvote, assuming community voting indicates correctness; this is not independently verified.
  • domain assumption GPT-4 Turbo's annotations of question type and complexity are sufficiently accurate for the reported analyses.
    Section 3.1 uses GPT-4 Turbo to assign each question a type and complexity level, with only a manual review that is not described in detail.
  • domain assumption The LLM judge validated on 136 human-annotated questions maintains similar accuracy on the full benchmark and on StackUnseen.
    Section 5.2 applies the LLM-as-judge methodology to all 925 StackEval questions and the StackUnseen set, but the judge's reliability was measured only on the 136-question LLM-as-judge benchmark.
  • domain assumption The human annotations of the LLM-as-judge dataset are consistent and reliable.
    Section 3.2 says answers are evaluated by human domain experts and a third expert verifies them, but no inter-annotator agreement statistic is reported.
  • domain assumption Stack Overflow questions accepted by the community represent real-world coding assistance queries.
    The paper acknowledges in Section 6.1 that Stack Overflow content is biased toward practical, popular topics, but still treats the dataset as representative of coding assistance needs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StackEval: Benchmarking LLMs in Coding Assistance." pith.science (2026). https://pith.science/paper/X7YDW5XK

@misc{pith2026241205288,
  author       = {Pith},
  title        = {Pith review of: StackEval: Benchmarking LLMs in Coding Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7YDW5XK}},
  note         = {Machine review of arXiv:2412.05288}
}
read the original abstract

We present two comprehensive benchmarks to evaluate the performance of language models in coding assistance tasks, covering code writing, debugging, code review, and conceptual understanding. Our main contribution includes two curated datasets: StackEval, a large-scale benchmark derived from Stack Overflow questions, and StackUnseen, a dynamic benchmark featuring the most recent Stack Overflow content. These benchmarks offer novel insights into the capabilities and limitations of LLMs, particularly in handling new and emerging content. Additionally, we assess LLMs' proficiency as judges for coding tasks using a curated, human-annotated dataset, exploring their evaluation capabilities and potential biases, including whether they favor their own generated solutions. Our findings underscore the potential of these benchmarks to advance LLM development and application in coding assistance. To ensure reproducibility, we publicly share our datasets and evaluation code at https://github.com/ProsusAI/stack-eval .

Figures

Figures reproduced from arXiv: 2412.05288 by the authors.

Figure 1
Figure 1. StackEval & StackUnseen Programming Language Distribution. The questions are subdivided based on the programming languages and type. The distribution of languages is sampled based on popularity of said languages as indicated in the Stack Overflow Developer Survey, 2023 [26]. 2 Related Works Coding Benchmarks. HumanEval [9], the most widely recognized coding benchmark, consists of 164 Python coding problems designed … view at source ↗
Figure 2
Figure 2. Evaluation methodology for assessing LLMs on coding tasks. a) LLM-as-a-Judge benchmark (CoT + Ref. Answer) comparing LLM-t (model under test) against human experts when evaluating answers from LLM-x models . b) Coding assistance evaluation where LLM-t generates StackOverflow answers, scored by an LLM judge. To standardize the evaluation process, we propose the acceptance score metric, inspired by the concept of acce… view at source ↗
Figure 3
Figure 3. Left: Correlation between model performance across different coding benchmarks shows strong positive correlations (0.72-0.92). Right: Model performance across different question types within the benchmark are very highly correlated (0.91-1.00), suggesting consistent performance across task categories. Analysis of the StackEval and StackUnseen benchmarks reveals consistent model rankings ( [PITH_FULL_IMAGE:figures/f… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Model Performance Degradation on Recent Problems. LLMs with higher StackEval scores show smaller acceptance rate drops on StackUnseen, suggesting better generalization to contemporary problems. 5.3 Self-Preference in LLM Judges Previous work has identified potential bi…
Figure 5
Figure 5. Figure 5: The performance of various LLMs across different question types on the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The performance of various LLMs across different question types on the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The performance of various LLMs across popular programming languages on the [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The LLM evaluation prompt (CoT + Ref. Answer) used to assess answer quality. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: StackEval Implementation. A C++ implementation sample question from the StackE￾val dataset. Call to undefined method Illuminate/Routing/RouteFileRegistrar::get() - Error after upgrading from Laravel 5.7 to 5.8. I have a running app written on Laravel 5.7. I tried to ch…
Figure 10
Figure 10. Figure 10: StackEval Debugging. A PHP debugging sample question from the StackEval dataset. Accessing something inside the object when you don’t know the key. I am getting a following object: { IuW1zvaSABwH4q: { label: ’Random Image of TypeScript not relavent to coworking’, thum…
Figure 11
Figure 11. Figure 11: StackEval Conceptual. A conceptual style sample question from the StackEval dataset. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: StackEval Optimization. A Rust optimization question from the StackEval dataset. Why ngModel doesn’t works on the last version of Angular 17? I am trying to make a form in my angular app, but when i want to implement ngModel on my form: <form (ngSubmit)="onSignUp()" #…
Figure 13
Figure 13. Figure 13: StackUnseen Versioning. A version-dependent question from the StackUnseen dataset. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 32 canonical work pages

  1. [1]

    Large Enough — mistral.ai

    Mistral AI. Large Enough — mistral.ai. https://mistral.ai/news/ mistral-large-2407/, 2024. [Accessed 29-10-2024]

    work page 2024

  2. [2]

    Mistral NeMo — mistral.ai

    Mistral AI. Mistral NeMo — mistral.ai. https://mistral.ai/news/mistral-nemo/,

    work page

  3. [3]

    Introducing claude 3.5 sonnet

    Anthropic. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet , 2024. [Accessed 29-10-2024]

    work page 2024

  4. [4]

    Introducing the Claude 3 family

    Anthropic. Introducing the Claude 3 family. https://www.anthropic.com/news/ claude-3-family, March 2024. Accessed: 2024-03-04

    work page 2024

  5. [5]

    Multi-lingual evaluation of code generation models, 2023

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudi...

    work page 2023

  6. [6]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. 10

    work page 2021

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  8. [8]

    Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022

    2022

Show all 47 references
  1. [9]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  2. [10]

    Meta large language model compiler: Foundation models of compiler optimization, 2024

    Chris Cummins, V olker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler optimization, 2024

    2024

  3. [11]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024

  4. [12]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024

  5. [13]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  6. [14]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, Oriol Vinyals, et al. Gemini 1.5: Unlocking multimodal under- standing acr...

    2024

  7. [15]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  9. [17]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  10. [18]

    Challenges in evaluating large language models

    Douwe Kiela, Shubham Bhooshan, Hamed Firooz, and Alun Preece. Challenges in evaluating large language models. arXiv preprint arXiv:2209.01186, 2022

    2022 arXiv

  11. [19]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  12. [20]

    Wildbench: Benchmarking language models with challenging tasks from real users in the wild, 2024

    Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking language models with challenging tasks from real users in the wild, 2024

    2024

  13. [21]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  14. [22]

    llama-3_1-nemotron-70b-instruct | NVIDIA NIM — build.nvidia.com

    Nvidia. llama-3_1-nemotron-70b-instruct | NVIDIA NIM — build.nvidia.com. https:// build.nvidia.com/nvidia/llama-3_1-nemotron-70b-instruct/modelcard , 2024. [Accessed 29-10-2024]

    2024

  15. [23]

    New models and developer products announced at devday

    OpenAI. New models and developer products announced at devday. https://openai.com/ index/new-models-and-developer-products-announced-at-devday , 2023. [Ac- cessed 29-10-2024]

    2023

  16. [24]

    Introducing openai o1-preview

    OpenAI. Introducing openai o1-preview. https://openai.com/index/ introducing-openai-o1-preview/ , 2024. [Accessed 23-10-2024]

    2024

  17. [25]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. Gpt-4 technical report, 2024

    2024

  18. [26]

    Stack overflow developer survey 2023, 2023

    Stack Overflow. Stack overflow developer survey 2023, 2023

    2023

  19. [27]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024

    2024

  20. [28]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational ...

    2002

  21. [29]

    Can foundation models label data like humans? Hugging Face Blog, 2023

    Nazneen Rajani, Nathan Lambert, Sheon Han, Jean Wang, Osvald Nitski, Edward Beeching, and Lewis Tunstall. Can foundation models label data like humans? Hugging Face Blog, 2023. https://huggingface.co/blog/llm-v-human-data

    2023

  22. [30]

    Codebleu: a method for automatic evaluation of code synthesis, 2020

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis, 2020

    2020

  23. [31]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024

  24. [32]

    Learning performance-improving code edits, 2024

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits, 2024

    2024

  25. [33]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. 12

    2024

  26. [34]

    Large language models are incon- sistent and biased evaluators, 2024

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are incon- sistent and biased evaluators, 2024

    2024

  27. [35]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  28. [36]

    Large language models are not fair evaluators, 2023

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023

    2023

  29. [37]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  30. [38]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations , 2024

    2024

  31. [39]

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024

    2024

  32. [40]

    Thinking before speaking: A role-playing model with mindset, 2024

    Baohua Zhang, Yongyi Huang, Wenyao Cui, and Huaping Zhang. Thinking before speaking: A role-playing model with mindset, 2024

    2024

  33. [41]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  34. [42]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023. 13 A Appendix Model Provider S...

    2023

  35. [44]

    Analyze question and reference answer for core requirements

  36. [45]

    Evaluate generated answer against requirements and reference

  37. [46]

    Reason on the acceptability of the generated answer based on the definition

  38. [47]

    questionAnalysis

    Assign final score based on rubric. Output Format The evaluation should be formatted as a JSON object: { "questionAnalysis": "Review core elements required for answer", "generatedAnswerAnalysis": "Evaluate coverage, strengths, and weaknesses", "acceptabilityEvaluation": "Asses...

  39. [2024]

    [Accessed 30-10-2024]

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.