Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A deterministic n-gram benchmark for Japanese open-ended QA reproduces a frontier LLM judge's rankings at correlation $r = 0.9896$ across 50 models, offering a cheap, judge-free evaluation alternative.

desk verdict A practical, cheap, deterministic Japanese open-ended QA benchmark with a concrete recipe, but the headline correlation with LLM-as-judge needs a sensitivity analysis because the validation set includes the reference-set-building models. read the letter →

arxiv 2502.09316 v1 pith:YKXTKIP5 submitted 2025-02-13 cs.CL

Kentaro Imajo , Masanori Hirano , Shuji Suzuki , Hiroaki Mikami This is my paper · ORCID
classification cs.CL
keywords LLMevaluationopen-endedgenerationn-grammetricsdistributionalhypothesisreferenceanswersetLLM-as-a-judgeJapanesebenchmarkdeterministic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a benchmark that evaluates open-ended Japanese question answering without any human or LLM judge. The authors' claim is that three deterministic metrics—Fluency, Truthfulness, Helpfulness—computed from character-level n-gram statistics against a curated reference answer set, reproduce what a frontier LLM judge would say, with correlation $r = 0.9896$ across 50 models. If that claim holds, LLM evaluation for this class of tasks becomes cheap, deterministic, and usable on models that cannot follow instruction prompts. The paper positions this as evidence that the distributional hypothesis, that meaning correlates with word distributions, is still a workable tool for judging generated text, at least for short-answer tasks. The reference sets are built once per question from 1 million responses per generating model, then shrunk to 1,000 representative answers.

What carries the argument

The load-bearing object is the reference answer set: for each question, a distribution of 1,000 representative Japanese answers, generated by three large Japanese LLMs, normalized to about 100 characters, and filtered to remove hallucinated or atypical phrasing. The scoring machinery is character-level n-gram matching against this distribution: Fluency sums the reference-set probability of the response's 1-10 grams, Truthfulness thresholds the frequency of the response's 3-grams at 0.5%, and Helpfulness checks manually defined key terms. Length normalization matters throughout, with a linear discount that peaks at 100 characters and reaches zero at 150 characters, so that longer but padded answers do not inflate scores. The n-gram frequencies act as a proxy for the full answer distribution, and the distributional hypothesis is what justifies treating overlap with that proxy as agreement with 'good' answers.

What would settle it

Take the 50 questions and write high-quality answers that deliberately avoid every 3-gram present in the reference sets; if a panel of human judges (or a frontier judge model) rates them as correct and fluent while the Truthfulness score collapses, the central claim is refuted. A cheaper variant: insert one plausible wrong fact that happens to use frequent n-grams, such as answering the clock-overlap question with '23' instead of '22', and check whether the benchmark or the judge detects the error first.

Watch

Extended reading notes

Core claim

The central discovery is that the statistical overlap between a generated response and a precomputed reference answer set captures most of what an LLM judge rewards. For each of 50 Japanese questions, the authors assemble a reference set of 1,000 answers by generating 1 million responses from each of three Japanese LLMs, filtering out hallucinations with question-specific rules and a 5-gram frequency filter, then keeping length-typical and diverse answers. A test response is scored by (1) Fluency, the summed occurrence likelihood of its character-level 1- through 10-grams in the reference set; (2) Truthfulness, the fraction of its character-level 3-grams that appear in the reference set with at least 0.5% frequency, treating rarer n-grams as hallucination signals; and (3) Helpfulness, hand-built keyword rules per question. The average of the three metrics correlates at $r = 0.9896$ with a frontier LLM judge's scores on 50 models and at about 0.74 with two established Japanese LLM leaderboards. The reference set itself is stable: swapping any one of the three generating models leaves benchmark scores correlated above $r = 0.999$. The authors read this as confirmation that the distributional hypothesis, the idea that words that appear in similar contexts carry similar meaning, remains sufficient for evaluating open-ended generation when the answer space is tight.

Load-bearing premise

The reference answer sets, built from three specific LLMs and filtered by rules and n-gram frequency, faithfully represent the distribution of correct and desirable Japanese answers; if they do not, the n-gram scores measure agreement with a possibly biased sample rather than quality.

Editorial extensions

If this is right

  • Open-ended Japanese QA can be scored deterministically on a laptop, without paying for judge-model API calls.
  • The metrics work on base (non-instruction-tuned) models, which LLM-as-a-judge cannot meaningfully assess.
  • Scores are stable with respect to which LLMs produced the reference set, so the benchmark can be rebuilt with newer models without re-calibrating.
  • Some frontier models already outscore the reference set, indicating the benchmark tracks a manifold of acceptable answers rather than a single gold answer.
  • The paper expects the method to cover moderate future improvements but acknowledges that qualitatively different future outputs may escape n-gram detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to build the same reference-set construction for English questions and check whether the $r = 0.99$ correlation with a judge model survives; the character-level n-gram choice would likely need to become token- or word-level.
  • Because the correlation target is the judge model, the benchmark inherits whatever biases that judge has; a stronger claim would compare the n-gram scores directly against human ratings.
  • The near-1.0 correlation with a judge that supposedly reads for meaning suggests that much of judge reward is driven by local text-surface regularity on short, constrained answers; if so, this benchmark is also a probe of judge behavior, not just of the models being scored.
  • Applying the same construction to multi-turn conversation or idea generation, where the answer space is wider, would stress the distributional assumption; the authors note this as future work, and the approach would likely need sentence- or discourse-level units.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a deterministic, judge-free benchmark for evaluating open-ended Japanese QA responses of LLMs. For each of 50 questions, a reference answer set is constructed by sampling one million responses from each of three Japanese LLMs (stockmark-100b, PLaMo-100b, Swallow-MX-8x7b-NVE-v0.1), followed by rule-based and 5-gram-frequency filtering and refinement to 1,000 responses. Three metrics are defined: Fluency (Eqs. 1-3), Truthfulness (Eqs. 6-9), and Helpfulness (manually defined rules), all computed against the reference set. The authors report a correlation of r=0.9896 between the benchmark and GPT-4o-as-a-judge across 50 models, and correlations above 0.7 with Japanese MT-Bench and Nejumi, concluding that the n-gram-based approach approximates LLM-judge evaluation at much lower computational cost.

Significance. If the claimed correlation holds on genuinely held-out models, this would be a useful contribution: deterministic, reproducible, low-cost evaluation for Japanese short-form QA, with public code and data. The paper is also transparent in releasing full results and in performing a reference-stability analysis (Fig. 3). However, the headline validation is compromised by the fact that the three reference-generating models are part of the 50-model correlation set, which can inflate r because the metrics measure overlap with the very distribution those models created. Consequently, the empirical support for the central claim is currently weaker than stated, and the Truthfulness metric is not grounded in factual ground truth.

major comments (4)
  1. [5.2, Table 1, Section 3.2] The validation set used to compute r=0.9896 includes the three models that generated the reference answer sets: stockmark-100b, PLaMo-100b, and Swallow-MX-8x7b-NVE-v0.1 appear in Table 1, and Section 3.2 states that these three models produced the reference sets. Because Fluency (Eqs. 1-3) and Truthfulness (Eqs. 6-9) are calculated as n-gram overlap with those reference sets, the three generating models receive a constructional advantage over held-out models. Recomputing the GPT-4o correlation without these three models is necessary to establish that the benchmark predicts judge scores on models it was not built from; the paper should report both correlations and discuss any difference. The stability analysis in Figure 3 does not remedy this, since it compares scores obtained from reference sets built by the same three models rather than validating against held-out models or human judgments.
  2. [3.3.2, Eqs. (6)-(9), Section 3.2.3] Truthfulness is operationalized as the proportion of character 3-grams whose frequency in the reference set exceeds 0.5%, under the assumption that rare n-grams indicate hallucination. This is not validated against factual ground truth; the filtering in Section 3.2.3 removes only 5-grams that appear once, so any incorrect statement that is shared and frequent across the three generating LLMs will survive and be scored as truthful. The paper itself acknowledges in Section 4 that the validity of the metrics was not demonstrated mathematically or theoretically. A spot-check of the reference sets by human annotators, or a comparison of Truthfulness scores against a factual QA benchmark for a subset of questions, is needed before the metric can be called a measure of truthfulness.
  3. [5.2] The reported correlation r=0.9896 is given without a confidence interval or any measure of uncertainty, and the number of models is only 50. The GPT-4o judge scores themselves are stochastic; the paper does not state how many judge runs were averaged. Reporting a 95% confidence interval (e.g., via Fisher z-transformation or bootstrap) and, ideally, a bootstrap analysis that excludes the three reference-generating models would make the strength of the correlation interpretable.
  4. [3.3, Eqs. (2), (6), (9)] The benchmark contains several free parameters whose values are chosen without reported sensitivity analysis: the target length of 100 characters and discount to zero at 150, the 3-gram frequency threshold of 0.5% in Eq. (6), the n-gram order range 1-10 in Eq. (3), and the reference-set size of 1,000 per question. If the correlation with LLM-as-a-judge is robust to reasonable variations in these parameters, the claim is stronger; if not, the metric is tied to the particular choices. A sensitivity analysis over at least the threshold and length parameters should be added.
minor comments (5)
  1. [3.2.1] The text states that 1.5 billion responses were generated across 50 questions; with one million responses per question from each of three models, the total is 150 million, not 1.5 billion. Please correct the number or clarify the intended meaning.
  2. [3.3.2, Eq. (6)] The description says 'proportion of 3-grams with a frequency of at least 0.5%' but Eq. (6) uses min(L*,0.005)/0.0005, which is not exactly a proportion; the paper's own text acknowledges 'almost the same'. Please replace the prose with the exact formula.
  3. [Figure 3] The correlation above 0.999 between single-model and ensemble reference sets is reported as a stability indicator, but a rank correlation or a comparison of per-question score differences would be more informative for whether individual model rankings change.
  4. [Section 2 and References] Some citations are incomplete or informal (e.g., 'Bard (Google, 2023)' and 'Vicuna (2023)'); please provide full references with authors and venues.
  5. [Section 6] There is a typo 'muti-turn' for 'multi-turn'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported correlation with GPT-4o is an empirical result, not forced by the reference-set construction.

full rationale

The paper's central claim is that deterministic n-gram metrics (Fluency, Truthfulness, Helpfulness) correlate with GPT-4o-as-a-judge at r=0.9896. That correlation is measured, not derived: the n-gram scores are computed against LLM-generated reference answer sets, while the GPT-4o judge is given the human sample answer (Appendix B), so the two evaluations are not identical by construction. The three reference-generating models (stockmark-100b, PLaMo-100b, Swallow-MX-8x7b-NVE-v0.1) do appear among the 50 evaluated models, which is a genuine methodological limitation, but it does not make the correlation a by-construction artifact: Table 1 shows GPT-4o and GPT-4 score above all three reference-generating models, and the reference answer set itself scores higher than each of them, so the metrics do not simply reward self-similarity. The paper's stability check (Figure 3, correlation above 0.999 between single-model and ensemble reference sets) also shows that no single reference model, including the authors' own PLaMo-100b, is load-bearing. The identification of 'Truthfulness' with agreement with frequent n-grams in the LLM-generated reference distribution is an assumption about validity, not a circular derivation; the correlation with an external judge is the paper's evidence for that assumption. No self-citation chain, imported uniqueness theorem, or fitted parameter renamed as a prediction appears in the derivation. The Discussion honestly notes limits (e.g., future high-performing LLMs may not be adequately evaluated, and theoretical connections to the distributional hypothesis remain open), which further supports that the authors do not present the benchmark as a closed formal consequence of its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark depends on several hand-chosen design parameters and on the assumption that LLM-generated, filtered reference sets represent desirable answers. No invented entities are introduced.

free parameters (4)
  • Target answer length of about 100 characters = 100 characters, discount to zero at 150
    Chosen to match the manually written sample answers; the length discount peaks at 100 and affects Fluency, Truthfulness, and Helpfulness scores.
  • Truthfulness 3-gram frequency threshold = 0.5%
    Hand-chosen cutoff to label n-grams as unlikely or hallucinated; no sensitivity analysis is provided.
  • Fluency n-gram order range = 1-10 grams
    Design choice for the Fluency metric; no ablation shows why this range was selected.
  • Reference answer set size per question = 1000 responses
    Chosen after filtering from 30,000 responses; the size affects the stability and resolution of n-gram probability estimates.
assumptions (5)
  • domain assumption The distributional hypothesis applies to evaluating LLM open-ended generation.
    The paper assumes good generation means good agreement between the generated word distribution and the desirable answer distribution, despite citing work that challenges the hypothesis for LLMs.
  • domain assumption LLM-generated, filtered responses approximate the desirable answer distribution.
    The reference answer sets are generated by three LLMs and filtered by rules and n-gram frequency, with no human validation that they represent correct and desirable answers.
  • domain assumption Rare n-grams indicate hallucinations.
    The frequency-based filter and the Truthfulness metric both assume that very low-frequency n-grams are likely errors, which may remove rare but correct statements.
  • domain assumption Character-level n-grams suffice for Japanese evaluation.
    The paper states character-level analysis is preferable for Japanese, but provides no theoretical or empirical justification beyond this preference.
  • domain assumption Correlation with GPT-4o judge validates the benchmark.
    The main evidence of effectiveness is correlation with an LLM-as-a-judge method, not with human judgments or verified factual ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis." pith.science (2026). https://pith.science/paper/YKXTKIP5

@misc{pith2026250209316,
  author       = {Pith},
  title        = {Pith review of: A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKXTKIP5}},
  note         = {Machine review of arXiv:2502.09316}
}
read the original abstract

Evaluating the open-ended text generation of large language models (LLMs) is challenging because of the lack of a clear ground truth and the high cost of human or LLM-based assessments. We propose a novel benchmark that evaluates LLMs using n-gram statistics and rules, without relying on human judgement or LLM-as-a-judge approaches. Using 50 question and reference answer sets, we introduce three new metrics based on n-grams and rules: Fluency, Truthfulness, and Helpfulness. Our benchmark strongly correlates with GPT-4o-based evaluations while requiring significantly fewer computational resources, demonstrating its effectiveness as a scalable alternative for assessing LLMs' open-ended generation capabilities.

Figures

Figures reproduced from arXiv: 2502.09316 by the authors.

Figure 1
Figure 1. Evaluation outline developed based on this principle (Mikolov, 2013; Sarzynska-Wawer et al., 2021; Bojanowski et al., 2017; Devlin et al., 2019), leading to the modern LLMs. Despite arguments that the hypothesis can￾not fully explain LLMs’ generalization (Chiang and Yogatama, 2023; Enyan et al., 2024), we as￾sume it remains useful for evaluating LLM outputs. 3 Proposed Benchmark Our benchmark was designed to measure… view at source ↗
Figure 2
Figure 2. Relationships between Fluency, Truthfulness, and Helpfulness [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Benchmark score comparison between using one of three LLMs to construct a reference answer set [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Comparison between our benchmark and Japanese MT-Bench shift in LLM evaluations, but also highlights cur￾rent limitations. The success of our n-gram-based approach indicates that it could be expanded to gen￾eral open-ended generation evaluations. Currently, we have onl…
Figure 5
Figure 5. Figure 5: Comparison between our benchmark and Ne [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Test-time retrieval of committee-disagreement-mined synthetic images cuts a safety classifier's false-negative rate on a hard HoliSafe subset from 41.2% to 24.5%.

Reference graph

Works this paper leans on

43 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Z...

    arXiv 2023

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv. https://arxiv.org/abs/2108.07732

    arXiv 2021

  5. [5]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135--146

    2017

  6. [6]

    Tom Brown, Benjamin Mann, et al. 2020. Language Models are Few-Shot Learners . Advances in Neural Information Processing Systems, 33:1877--1901

    work page 2020

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  8. [8]

    Ting-Rui Chiang and Dani Yogatama. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.637 The distributional hypothesis does not fully explain the benefits of masked language model pretraining . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10305--10321, Singapore

    work page doi:10.18653/v1/2023.emnlp-main.637 2023

Show all 43 references
  1. [9]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41s...

    2024

  2. [10]

    Choi, Kristin E

    Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel B. Schwarcz. 2022. Chatgpt goes to law school. Journal of Legal Education, 71(3)

    2022

  3. [11]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vi...

    2023

  4. [12]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv. https://arxiv.org/abs/2110.14168

    2021 arXiv

  5. [13]

    Databricks. 2023. Dolly. https://github.com/databrickslabs/dolly

    2023

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associati...

    2019 doi

  7. [15]

    Zhang Enyan, Zewei Wang, Michael A Lepori, Ellie Pavlick, and Helena Aparicio. 2024. Are llms models of distributional semantics? a case study on quantifiers. arXiv. https://arxiv.org/abs/2410.13984

    2024 arXiv

  8. [16]

    Marc Eulerich, Aida Sanatizadeh, Hamid Vakilzadeh, and David A. Wood. 2024. https://doi.org/10.1007/s11142-024-09833-9 Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries . Review of Accounting Studies, 29(3):2318--2349

    2024 doi

  9. [17]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2021 doi

  10. [18]

    Google. 2023. Bard. https://bard.google.com/

    2023

  11. [19]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv. https://arxiv.org/abs/2306.11644

    2023 arXiv

  12. [20]

    Zellig S. Harris. 1954. https://doi.org/10.1080/00437956.1954.11659520 Distributional structure . WORD, 10(2-3):146--162

    1954

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  14. [22]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems

    2021

  15. [23]

    Kwan Yuen Iu and Vanessa Man-Yi Wong. 2023. https://doi.org/10.2139/SSRN.4339839 ChatGPT by OpenAI: The End of Litigation Lawyers? SSRN Electronic Journal. https://papers.ssrn.com/abstract=4339839

    2023 doi

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  17. [25]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv. https://arxiv.org/abs/2001.08361

    2020 arXiv

  18. [26]

    Tomas Mikolov. 2013. Efficient estimation of word representations in vector space. arXiv. https://arxiv.org/abs/1301.3781

    2013 arXiv

  19. [27]

    Harsha Nori, Nicholas King, Scott Mayer Mckinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 on Medical Challenge Problems . arXiv. https://arxiv.org/abs/2303.13375v2

    2023 arXiv

  20. [28]

    OpenAI. 2023 a . ChatGPT . https://openai.com/blog/chatgpt/

    2023

  21. [29]

    OpenAI. 2023 b . GPT-4 Technical Report . arXiv. https://arxiv.org/abs/2303.08774

    2023 arXiv

  22. [30]

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816--4828

    2021

  23. [31]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training . https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

    2018

  24. [32]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners . https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  25. [33]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark . In Proceedings of First Conference on Language Modeling. https://openreview.net/fo...

    2024

  26. [34]

    Justyna Sarzynska-Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. 2021. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304:114135

    2021

  27. [35]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model . arXiv. https://arxiv....

    2022 arXiv

  28. [36]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://openreview.net/forum?id=fR3wGCk-IXp Language models are multilingual chain-of-thought reasone...

    2023

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . LLaMA: Open and Efficient Foundation Language Models . arXiv. https://arxiv.org/abs/2302.13971

    2023 arXiv

  30. [38]

    Hugo Touvron, Louis Martin, et al. 2023 b . Llama 2: Open Foundation and Fine-Tuned Chat Models . arXiv. https://arxiv.org/abs/2307.09288v2

    2023 arXiv

  31. [39]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need . In Advances in Neural Information Processing Systems, volume 30, pages 5999--6009

    2017

  32. [40]

    Vicuna. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ https://vicuna.lmsys.org/

    2023

  33. [41]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Proceedings of Conference on Neural Informatio...

    2024

  34. [42]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. https://openreview.net/forum?id=yzkSU5zd...

    2022

  35. [43]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.