Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces CPP-UT-Bench, a benchmark of 2,653 real-world C++ code-test pairs, and reports that fine-tuned LLMs beat their base models by more than 70 percent in nine of ten experiments.

desk verdict The dataset is a genuine contribution, but the unexecuted judge metric and missing train/test split mean the headline >70% gains are not supported. read the letter →

arxiv 2412.02735 v1 pith:G6UQNRON submitted 2024-12-03 cs.SE cs.LG

Vaishnavi Bhargava , Rajat Ghosh , Debojyoti Dutta This is my paper · ORCID
classification cs.SEcs.LG
keywords C++unittestingbenchmarkdatasetlargelanguagemodelscodegenerationfine-tuningin-contextlearningLLM-as-a-judgesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CPP-UT-Bench, a benchmark dataset of 2,653 {code, unit test} pairs drawn from 14 open-source C++ codebases across nine domains, to measure how well LLMs generate C++ unit tests. The authors argue that existing coding benchmarks are saturated and dominated by Python, so C++ test generation lacks a real-world evaluation instrument. They evaluate models under few-shot in-context learning, LoRA-based parameter-efficient fine-tuning, and full-parameter fine-tuning, using GPT-4o-mini as a judge to compute pairwise win rates against ground-truth tests. The central quantitative claim is that fine-tuned models outperform their base counterparts by an average of more than 70 percent in nine out of ten experiments. If the benchmark is sound, it gives the field a reusable tool for training and evaluating C++ test-writing models.

What carries the argument

The central object is the benchmark dataset itself: 2,653 {code, unit test} pairs extracted from 14 open-source C++ repositories spanning nine application domains. The extraction pipeline groups C++ source and test files by base name, then maps sources to their corresponding unit tests. For evaluation, the paper relies on LLM-as-a-Judge with GPT-4o-mini: a pairwise protocol in which the judge model compares two generated tests against a ground-truth test and returns a win, loss, or tie. For fine-tuning, the machinery is LoRA (low-rank adaptation) applied to attention and MLP projections, plus full-parameter fine-tuning, with two training epochs on the benchmark itself. The code chunker based on concrete syntax trees handles files longer than 200 lines.

What would settle it

Re-run the fine-tuning experiments with a held-out split of CPP-UT-Bench that is disjoint from the fine-tuning training data and see whether win rates against base models still exceed 70 percent; alternatively, replace the LLM judge with compile-and-run validation and see whether the win-rate ranking survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that CPP-UT-Bench functions as an effective benchmark for C++ unit test generation: it is diverse enough to reflect real-world codebases and sensitive enough that fine-tuning on it produces consistent gains. Across five base models (Mistral-7B-Instruct-v0.2, TinyLlama-1.1B-Chat, CodeLlama-7B-Instruct, Llama-3-8B-Instruct, and Llama-3.1-8B-Instruct), the authors compare LoRA-PEFT and full-parameter fine-tuned versions against the corresponding base models. With GPT-4o-mini acting as judge, win rates favor the fine-tuned models in nine of ten comparisons, with an average margin exceeding 70 percent. The paper also reports few-shot in-context learning results that reproduce known scaling trends across Llama-3, Phi-3, and Mistral-7B families, which the authors take as evidence that the benchmark behaves consistently with established coding benchmarks.

Load-bearing premise

The evaluation examples used to measure fine-tuning gains must be disjoint from the fine-tuning training examples, but the paper never states that a held-out split was used.

Editorial extensions

If this is right

  • Fine-tuning LLMs on CPP-UT-Bench yields consistent win-rate improvements over base models, supporting the use of the benchmark as a training signal for C++ test generation.
  • The benchmark reproduces known scaling trends in few-shot in-context learning, suggesting it behaves like established coding benchmarks and can serve as a saturation-resistant measure.
  • The released dataset, data-creation pipeline, and fine-tuned weights allow other researchers to adapt the benchmark to new models and domains without rebuilding from scratch.
  • Because C++ codebases have high verbosity and manual memory management, models tuned on this benchmark may help automate unit tests in real industrial C++ projects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to add compile-and-run validation, since LLM-as-a-judge compares generated tests against ground truth text rather than checking whether the tests actually compile and pass.
  • The dataset is imbalanced, with more than 60% of pairs coming from TensorFlow; a balanced split would test how well the gains generalize across codebase styles.
  • If the fine-tuning gains persist on held-out repositories, CPP-UT-Bench could play the role for C++ unit tests that HumanEval plays for Python function synthesis.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces CPP-UT-Bench, a dataset of 2,653 {code, unit test} pairs extracted from 14 open-source C++ repositories across nine domains. The authors evaluate several LLMs on few-shot in-context learning and fine-tuning, using GPT-4o-mini as a judge to compare generated tests against ground-truth tests. The central claim is that fine-tuned models outperform base models by an average of more than 70% in nine out of ten experiments, demonstrating the benchmark's effectiveness for measuring and improving C++ unit test generation.

Significance. If the benchmark and evaluation were sound, this would be a valuable contribution: real-world C++ unit test generation is an underexplored task, and a dataset with permissive licenses and released model weights could support further research. The paper's data curation pipeline and chunking approach are also potentially useful. However, the evaluation methodology is not validated, and the fine-tuning results are not shown to be based on held-out data, so the headline claims are not established. The paper does ship code and model weights, which is commendable, but the experimental evidence is currently insufficient to support the central assertions.

major comments (3)
  1. [Section 2.3, Section 4] The evaluation metric is an LLM-as-a-Judge protocol that never compiles, links, or executes the generated unit tests. The pipeline in Section 2.4 chunks source files and then simply appends the per-chunk generated tests, so the final artifact need not be a syntactically valid or runnable test file. The judge prompt in Figure 4 asks for 'alignment' with the ground truth, which is a textual similarity judgment, not a test-validity check. The paper cites MT-Bench for support but does not calibrate GPT-4o-mini against compilation, test execution, or mutation testing on this task. Consequently, the win rates reported in Figures 7-12, and the abstract's claim of >70% average improvement, may reflect lexical alignment or judge bias rather than the actual quality of the generated unit tests. This is load-bearing because the entire evaluation rests on this unvalidated metric.
  2. [Section 3, RQ-2, Figure 6] The fine-tuning experiments do not state a held-out split. The paper describes CPP-UT-Bench as the demonstration dataset for fine-tuning and also as the source of the 200 evaluation samples used to measure fine-tuning gains, but no train/test separation is mentioned. If the evaluation samples overlap with the training data, the reported improvements are inflated by memorization rather than generalization. The paper must specify the exact split and demonstrate that the evaluation set is disjoint from the fine-tuning set. This is load-bearing for the fine-tuning claims, which are a central contribution.
  3. [Section 4.2.1] The paper attributes the full-parameter fine-tuning underperformance of Mistral-7B-Instruct-v0.2 to its 'MoE architecture' [39]. Mistral-7B is a dense transformer, not a mixture-of-experts model; reference [39] is about ST-MoE, a different architecture. This is a factual error that invalidates the proposed explanation and suggests a lack of basic verification about the models under study. The authors should either correct the explanation or remove it, and they should verify all factual claims about model architectures throughout the paper.
minor comments (7)
  1. [Section 2, data collection] The text reads 'pairs from from 14 popular open-source' - remove the duplicated 'from'.
  2. [Equation (7)] The summation notation is malformed: 'UT(T(r)) = nX i=1 UT(Ci)' should use standard summation notation. Clarify the expression.
  3. [Section 4.2.3, Figure 10] The comparison label in the text and figure caption incorrectly names TinyLlama as the opponent for CodeLlama PEFT; it should be the CodeLlama-7B-Instruct-hf base model.
  4. [Table 2] Table 2 lists TinyLlama LoRA rank as 16, while Section 3 states that 'a rank of 8 and an alpha of 16' were used for all models. Please reconcile this discrepancy.
  5. [Section 4.1] The phrase 'we accessed the performance' should be 'we assessed the performance'.
  6. [Abstract] The phrase 'for nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%' is ambiguous: clarify whether 70% is the average win rate or the average relative improvement, and specify the baseline for the average.
  7. [Section 2.3] The paper states the evaluation prompt was 'carefully designed' and tuned to mitigate biases, but no details of the tuning process or validation are provided. Please describe the tuning procedure or provide evidence of bias mitigation.

Circularity Check

1 steps flagged · score 6.0 of 10

Fine-tuning improvements are evaluated on the same benchmark used for training, with no stated held-out split, so the headline >70% gain reduces to a training-fit quantity.

  1. fitted input called prediction [Section 3, RQ-2 (paragraph beginning 'To evaluate the performance of the fine-tuned models...'); Section 2.1]
    "The key value of a benchmark dataset such as CPP-UT-Bench comes from its value as a test data for few-shot in-context and a demonstration dataset for PEFT and full-parameter fine-tuning. ... The fine-tuning was performed over two epochs on our curated dataset ... To evaluate the performance of the fine-tuned models against their original counterparts, we used the process mentioned in RQ-1."

    The paper fine-tunes by maximizing log p(y|x) over {code, unit test} pairs (Eq. 2), and then evaluates on 200 samples using a judge that scores which generated test is 'more closely aligned' with the ground-truth test g (Eq. 4). CPP-UT-Bench is described both as the demonstration dataset for fine-tuning and as the source of the evaluation dataset; no held-out split is stated anywhere in Section 3 or Section 4. For any training pair also present in the evaluation sample, the fine-tuned model has directly optimized toward reproducing the ground-truth test, so the claimed >70% win rate is, as reported, a measure of training-set fit and memorization rather than an independent prediction of unit-test writing capability.

full rationale

CPP-UT-Bench itself is an externally constructed dataset of code/test pairs scraped from open-source repositories, so the dataset artifact and the few-shot in-context comparisons (RQ-1) retain independent content. No self-citations, imported uniqueness theorems, or ansatz-by-citation steps are load-bearing. The critical circularity is confined to RQ-2: the same benchmark is used as the fine-tuning demonstration dataset and as the source of the evaluation samples, and the paper never states that the 200 evaluation samples are disjoint from the training set. Since the fine-tuning objective (Eq. 2) directly optimizes the likelihood of ground-truth unit tests and the evaluation metric (Eq. 4) scores judged alignment with those same ground truths, the reported >70% improvement over base models is not demonstrably a generalization result; it is, on the paper's own description, a measure of how well the model reproduces the training distribution. The separate concern that GPT-4o-mini judges textual alignment without compiling or running any test is a validity/correctness risk rather than a circularity, and is not counted in the score beyond the way it amplifies the fit-based nature of the win rates.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset construction conventions and an unvalidated judge-based evaluation. No new physical or mathematical entities are introduced, but several domain assumptions are load-bearing.

free parameters (5)
  • LoRA rank = 8 (16 for TinyLlama)
    Chosen via grid search; affects the fine-tuning results reported in RQ-2.
  • LoRA alpha = 16
    Chosen via grid search alongside rank; part of the PEFT configuration.
  • Learning rate = 5e-5
    Set for both PEFT and full fine-tuning; mentioned as yielding stable training.
  • Sampling temperature = 0.1
    Selected in preliminary experiments; used for all inference.
  • Evaluation sample size = 200
    The number of benchmark items used in each evaluation set; selection described as 'somewhat random'.
assumptions (4)
  • domain assumption Source files and unit test files are correctly paired by basename matching
    The dataset construction in Section 2 assumes Foo.cc and Foo_test.cc with the same base name are related, which may not always hold.
  • domain assumption GPT-4o-mini judge scores alignment with ground truth as a valid proxy for unit test quality
    Section 2.3 describes the judge but provides no validation against human judgment or test execution.
  • domain assumption Evaluation samples are disjoint from fine-tuning training samples
    Never stated in Section 3; the benchmark is used as both training and evaluation source.
  • domain assumption LLMs have not memorized the benchmark code during pretraining
    Code is scraped from public GitHub repos that may be in LLM training data; no contamination analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?." pith.science (2026). https://pith.science/paper/G6UQNRON

@misc{pith2026241202735,
  author       = {Pith},
  title        = {Pith review of: CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6UQNRON}},
  note         = {Machine review of arXiv:2412.02735}
}
read the original abstract

We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%.

Figures

Figures reproduced from arXiv: 2412.02735 by the authors.

Figure 1
Figure 1. Data extraction pipeline for CPP-UT-Bench. It uses GitHub repos as upstream sources and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data distribution of CPP-UT-Bench from 14 different GitHub Repositories. The dominant [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The diversity in {code, unit test} pairs in terms of line lengths across 14 different opensource [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Prompt for pairwise evaluation of two LLM generated responses (Assistant A and Assistant [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Distribution of evaluation dataset for the few-shot in-context learning. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Distribution of evaluation dataset for the fine-tuning experiments. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Few-Shot in-context learning performance assessment for three LLM pairs: {Llama-3-70B [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Fine-tuning results for Mistral-7B-Instruct-v0.2 [ [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Fine-tuning results for TinyLlama [22]. The results corroborate with other general coding benchmarks [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Fine-tuning results for CodeLlama-7B [6] . The results corroborate with other general coding benchmarks. shows PEFT is working better than the base, winning 67% times. With full-parameter fine-tuning the model performance improves further to 75.5% w.r.t. the base [PI…
Figure 11
Figure 11. Figure 11: Fine-tuning results for Meta-Llama-3-8B-Instruct [ [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Fine-tuning results for Meta-Llama-3-8B-Instruct [ [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Prompt template for the unit test generation in C++. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SwiftEval, a 28-problem hand-crafted Swift benchmark, evaluates 44 code LLMs and shows large performance drops on Swift tasks, especially for smaller models.

  2. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

Reference graph

Works this paper leans on

39 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [39]

    St-moe: Designing stable and transferable sparse expert models

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022. A Appendix / supplemental material Model Name PEFT TechniqueRank Alpha Layers Targeted Mistral-7B-Instruct-v0.2 LoRA 8 16 up_proj, o_proj, gate_pro...

    arXiv 2022

  2. [1]

    https://github.com/abseil/abseil-cpp

    work page

  3. [2]

    https://docs.sweep.dev/blogs/chunking-improvements

    work page

  4. [3]

    https://huggingface.co/meta-llama/Meta-Llama-3-8B,

    work page

  5. [4]

    https://huggingface.co/meta-llama/Meta-Llama-3.1-8B,

    work page

  6. [5]

    https://github.com/google/cel-cpp

    work page

  7. [6]

    https://huggingface.co/codellama/CodeLlama-7b-hf

    work page

  8. [7]

    https://github.com/google/glog?tab=readme-ov-file

    work page

Show all 39 references
  1. [8]

    https://github.com/google/googletest

  2. [9]

    https://github.com/google/langsvr

  3. [10]

    https://github.com/google/leveldb

  4. [11]

    https://github.com/google/libaddressinput,

  5. [12]

    https://github.com/google/libphonenumber,

  6. [13]

    https://huggingface.co/casperhansen/llama-3-70b-instruct-awq,

  7. [14]

    https://huggingface.co/casperhansen/llama-3-8b-instruct-awq,

  8. [15]

    https://huggingface.co/mistralai/Mistral-7B-v0.1,

  9. [16]

    https://huggingface.co/mistralai/Mistral-7B-v0.2,

  10. [17]

    https://huggingface.co/microsoft/Phi-3-medium-128k-instruct, . 11

  11. [18]

    https://huggingface.co/microsoft/Phi-3-small-8k-instruct,

  12. [19]

    https://github.com/pytorch/pytorch

  13. [20]

    https://github.com/tensorflow/tensorflow,

  14. [21]

    https://github.com/google/tensorstore,

  15. [22]

    https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

  16. [23]

    https://github.com/google/tsl

  17. [24]

    Phi-3 technical re- port: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  18. [25]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  19. [26]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  20. [27]

    Multipl- e: a scalable and polyglot approach to benchmarking neural code generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on ...

    2023

  21. [28]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  22. [29]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  23. [30]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  24. [31]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  25. [32]

    Dynabench: Rethinking benchmarking in nlp

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021

    2021 arXiv

  26. [33]

    An introduction to kolmogorov complexity and its applications, 2008

    M Li. An introduction to kolmogorov complexity and its applications, 2008

    2008

  27. [34]

    How and why we end up with complex methods: a multi- language study

    Mateus Lopes and Andre Hora. How and why we end up with complex methods: a multi- language study. Empirical Software Engineering, 27(5):115, 2022

    2022

  28. [35]

    Full parameter fine-tuning for large language models with limited resources, 2024

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources, 2024. URL https: //arxiv.org/abs/2306.09782

    2024 arXiv

  29. [36]

    Mapping global dynamics of benchmark creation and saturation in artificial intelligence

    Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13(1):6793, 2022

    2022

  30. [37]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    2022 arXiv

  31. [38]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.