Pith. sign in

REVIEW 3 major objections 5 minor 20 references

TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TypyBench shows LLMs score near 0.80 on type similarity but far worse on repository-level type consistency.

desk verdict Useful benchmark and a thoughtful similarity metric, but the headline claim about repo-level consistency rests on an unnormalized error count that rewards conservative predictions. read the letter →

arxiv 2507.22086 v1 pith:EISBXXBS submitted 2025-07-28 cs.SE cs.AIcs.PL

Honghua Dong , Jiacheng Yang , Xun Deng , Yuhe Jiang , Gennady Pekhimenko , Fan Long , Xujie Si This is my paper · ORCID
classification cs.SEcs.AIcs.PL
keywords typeinferencePythonhintsLLMevaluationrepository-levelbenchmarkstaticcheckingSimCheckmypy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TypyBench, a benchmark that asks LLMs to infer the missing type annotations—as PEP 484 stub files—for 50 cleaned, well-typed Python repositories, and scores the predictions against the original human annotations. The benchmark's first metric, TypeSim, gives credit for semantically similar types by comparing the methods and operations types support and by recursing through nested generics and unions, instead of requiring exact matches. The second metric, TypeCheck, runs mypy on the predicted stubs and counts a fixed set of type errors as a measure of how coherently the inferred types work together across the whole repository. Across nine LLMs, the paper reports that state-of-the-art models reach TypeSim scores around 0.80, close to what human annotations achieve locally, but that most of them produce large numbers of type-checking errors; even the best model, Claude 3.5 Sonnet, has more errors than the ground truth on 29 of 50 repositories. The authors argue that the field should therefore shift from optimizing type similarity toward repository-level consistency.

What carries the argument

TypeSim is a continuous similarity scorer: for non-generic types it uses the Jaccard index of the sets of methods and operations two types support (e.g., float vs int ≈ 0.6, int vs str ≈ 0.06); for generic types it recursively compares root constructors and type arguments, averaging rather than multiplying root and argument scores so that bare List vs List[int] still scores 0.5; and for union types it computes an optimal matching between union members. TypeCheck is the count of mypy errors of five kinds—attr-defined, assignment, arg-type, union-attr, index—produced when the model's generated .pyi stub files are type-checked against the type-removed repository. The two metrics are designed to be complementary: TypeSim measures closeness to what a human wrote, while TypeCheck measures whether the predictions form a usable, internally consistent type system.

What would settle it

Run the same LLM predictions through a second static type checker (e.g., Pyright) or a normalized version of TypeCheck that divides mypy errors by the number of annotated functions or by the model's missing rate; if the ordering of models changes substantially or Claude 3.5 Sonnet's advantage disappears, then the claimed gap between local accuracy and global consistency is an artifact of the raw error count rather than a true consistency deficit.

Watch

Extended reading notes

Core claim

The central discovery is that local type accuracy and global type consistency are separable capabilities in LLM-based Python type inference. Top models cluster tightly on TypeSim—GPT-4o at 0.804, DeepSeek-V3 at 0.795, Grok-2 at 0.787, Claude 3.5 Sonnet at 0.788—while TypeCheck error counts vary widely, from 127.1 errors for Claude 3.5 Sonnet to 465.7 for Llama-3.1-8B, with the human ground truth averaging 141.8 errors. Because Claude 3.5 Sonnet beats the ground-truth TypeCheck average even though it loses to the ground truth on 29 repositories, the paper reads the result as showing that good TypeSim does not guarantee a coherent type system, and that consistency should be treated as a distinct evaluation axis. The paper also shows that deeper nested types widen the gap between TypeSim and exact match, and that giving the model whole-repository context reduces TypeCheck errors on three small repos at the cost of TypeSim.

Load-bearing premise

The central claim rests on treating the raw number of mypy errors in the generated stub files as a fair measure of repository-level type consistency, even though it is not adjusted for repository size, number of predicted annotations, or how many types the model left missing.

Editorial extensions

If this is right

  • If the claim holds, repository-level type consistency, not similarity to a single annotation, should become the primary target for LLM type inference systems.
  • Whole-repository context is a promising lever for consistency: on three small repositories, GPT-4o's TypeCheck errors dropped sharply when the full repo was provided, though TypeSim fell.
  • Complex nested types beyond depth 2 remain a weak spot for all tested models, and exact-match evaluation understates this difficulty more than TypeSim does.
  • Rare types still cost models points on many repositories, so frequency-aware training or evaluation is needed.
  • Newer repositories score lower than older ones, indicating that pretraining contamination should be accounted for when comparing models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A raw mypy error count conflates specificity with consistency: a model that predicts Any everywhere or omits annotations avoids many errors without producing a genuinely coherent type system, so an error-per-annotation or error-per-predicted-type normalization would separate the two.
  • TypeCheck as defined is tied to one checker and one error taxonomy; re-ranking models with Pyright or a different error filter would be a quick robustness test for the claimed gap.
  • One could extend TypeCheck to measure same-symbol agreement across files directly—whether the same function or variable receives compatible types in every reference—rather than relying on the checker's aggregate.
  • The full-repo context result suggests a scaling direction: methods that compress long repositories or stream relevant cross-file type constraints could improve consistency without the reported TypeSim loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. TypyBench introduces a repository-level benchmark for evaluating LLM type inference on untyped Python repositories, with two metrics: TypeSim, a semantic similarity measure based on method-set overlap and recursive structural comparison, and TypeCheck, defined as the number of mypy errors on predicted stub files. The authors curate 50 popular Python repositories, strip type annotations to create prediction tasks, and evaluate several API-based and local LLMs. They report TypeSim scores around 0.80 for state-of-the-art models but large TypeCheck error counts, concluding that local type accuracy is relatively strong while repository-level consistency is the main weakness and should be the focus of future research.

Significance. If the proposed metrics are sound, TipyBench is a potentially useful resource: it provides a large set of type-removed Python repositories with train/validation/test splits, a contamination-aware temporal split, per-repository results in the appendices, and a small full-repository-context experiment. The TypeSim metric is a reasonable first step beyond exact-match evaluation and the paper makes a concrete attempt at repository-level assessment. However, the central conclusion about a local-versus-global gap is almost entirely driven by TypeCheck, which is a raw, unnormalized error count. The current evidence does not firmly establish the claimed gap, because the metric appears to reward conservativeness and missing predictions rather than genuine cross-file consistency. The benchmark and TypeSim contribution remain valuable, but the headline claim requires substantial re-analysis.

major comments (3)
  1. [Section 4.2, Table 2] TypeCheck as defined in Section 4.2 is the raw number of mypy errors for the five error categories listed in Appendix C.2, with no normalization by repository size, number of annotations, or model missing rate. This makes the metric reward conservative predictions: a model that emits fewer annotations or uses permissive types such as Any or object will trigger fewer mypy errors. The paper's own Table 2 shows Claude 3.5 Sonnet with 127.1 errors versus the ground truth's 141.8, despite being worse than ground truth on 29 repositories; combined with its 11.9% missing rate, this indicates the metric conflates consistency with permissiveness. Please normalize TypeCheck (for example, errors per annotated site or per 1,000 lines of code), add an always-Any baseline, and compare models on the subset of sites where all models made predictions.
  2. [Section 6.1, Table 4] The main evaluation generates stub files file-by-file without repository context, whereas the ground truth annotations were produced with full knowledge of the repository. Reporting that LLMs 'struggle with repo-level consistency' is therefore partly an artifact of the protocol: a model cannot maintain cross-file coherence when it never sees the rest of the repository. The full-repository-context experiment on only three repositories shows large TypeCheck reductions (Table 4), which supports the importance of context, but it also means the headline claim about a local-versus-global gap should either be restricted to the single-file setting or supported by a larger full-repository evaluation.
  3. [Section 4.1, Algorithm 1, Table 2] The paper reports TypeSim both with and without missing predictions, but the handling of missing predictions in Algorithm 1 is not defined. It is not stated whether a missing site contributes 0 to the average or is excluded, nor how the reported 'TYPE SIM' (which includes missing) relates to 'TYPE SIM WO MISSING' and the missing rate. Without this definition, TypeSim values are not reproducible and comparisons across models with different missing rates are ambiguous. Please specify the exact scoring rule for missing annotations and provide the formula used to compute both reported quantities.
minor comments (5)
  1. [Section 5.1] The weighting parameters alpha, beta, and gamma in the repository quality score are not given values; please report them to allow reproduction of the repository selection process.
  2. [Table 4] The row labelled 'AVG = 0.705' appears under the TYPE CHECK column for single-file context; the two averages should be placed under the TYPE SIM columns and labelled clearly.
  3. [Figure 5] Figure 5 contains garbled text tokens (e.g., '/uni00000014'); please regenerate the figure so that axis labels and legends are legible.
  4. [Section 6.3, Figure 6] The text states that 40 of 50 repositories fall under the red line, but the scatter plot does not show a red line; either add the line to the figure or rephrase the sentence.
  5. [Appendix C.2] The choice of the five mypy error codes (attr-defined, assignment, arg-type, union-attr, index) is not justified; please explain why these codes, rather than other mypy categories, best capture repository-level consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TypyBench's metrics are defined independently of the predictions they evaluate, and the central claims are empirical measurements rather than derived equivalences.

full rationale

The paper's central claims—that SOTA LLMs reach TypeSim scores around 0.80 but show TypeCheck errors—are empirical measurements obtained by running an external static checker (mypy) on model-generated stubs and by comparing predicted types to human ground truth. TypeSim is defined through Jaccard similarity over type attributes and recursive structural comparison; TypeCheck is defined as a count of five mypy error codes. Neither metric is fitted to the model outputs, and no parameter is calibrated against the rankings that the paper reports, so the observed ordering is not forced by construction. The only self-citation is APPL (Dong et al., 2024), used as an evaluation implementation tool; it is not load-bearing for the benchmark's validity or for the paper's conclusions. Potential concerns about TypeCheck being an unnormalized raw error count, or about permissive predictions lowering error counts, are metric-design and interpretation issues rather than circularity, since the metric is not defined in terms of the conclusion it supports. The paper is self-contained against external benchmarks: ground-truth annotations are taken from real repositories and mypy is an independent type checker, so the evaluation is externally reproducible and falsifiable.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central evaluation does not introduce new physical or mathematical entities. It introduces two metrics, TypeSim and TypeCheck, which are measurement procedures rather than entities. The main assumptions are about the validity of mypy as a consistency oracle, the Jaccard method-overlap as a semantic similarity model, the completeness of type removal, and the quality of the original annotations. A few manually chosen values (weight 0.5, five error codes, repo score weights) act as free parameters.

free parameters (4)
  • TypeSim root/argument weight alpha = 0.5 (hand-chosen)
    Section 4.1.2 averages root and argument similarities; the paper states this is to give more weight to root types, but 0.5 is a manual design choice, not derived from data.
  • TypeCheck error category subset = 5 mypy error codes (attr-defined, assignment, arg-type, union-attr, index)
    Appendix C.2 lists the counted error types; selecting these five categories is a hand-made decision that affects TypeCheck scores.
  • Repository quality score weights (alpha, beta, gamma) = Unstated
    Section 5.1 defines S = alpha*Scoverage + beta*Spopularity + gamma*Scomplexity but does not report the weight values, making the selection of the 50 repositories not fully reproducible.
  • Missing-rate handling in TypeSim = Missing annotations excluded from TypeSim without-missing
    Table 2 reports TypeSim with and without missing; treating missing predictions as excluded rather than penalized changes the ranking and is a modeling choice.
assumptions (4)
  • domain assumption Mypy error counts are a valid proxy for type consistency across a codebase.
    Section 4.2 defines TypeCheck as the number of mypy errors and asserts these errors reflect what developers would need to fix; this assumes mypy's judgment is the ground truth for consistency, even though the original human-written annotations in the benchmark already contain 141.8 mypy errors on average (Table 2).
  • domain assumption The Jaccard index over supported methods captures functional similarity between types.
    Section 4.1.1 defines base similarity as the Jaccard index of attribute sets; this assumes method overlap is a faithful proxy for semantic interchangeability, e.g., s(int,float)=0.6. No user study or extrinsic validation is provided.
  • domain assumption Removing type annotations with the AST script preserves the semantic content needed for inference.
    Algorithm 4 in Section 5.2 strips annotations and docstring type hints; the paper asserts the code's runtime behavior is preserved, but the cleaned repo may lose type information embedded in docstrings beyond simple patterns.
  • domain assumption The original annotations in the 50 selected repositories are a reasonable ground truth for evaluation.
    Section 5.1 selects repos with high type coverage; however, many repos have nonzero mypy errors (Table 6), so the ground truth is itself imperfect, which complicates TypeCheck comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories." pith.science (2026). https://pith.science/paper/EISBXXBS

@misc{pith2026250722086,
  author       = {Pith},
  title        = {Pith review of: TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EISBXXBS}},
  note         = {Machine review of arXiv:2507.22086}
}
read the original abstract

Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce TypyBench, a benchmark designed to evaluate LLMs' type inference across entire Python repositories. TypyBench features two novel metrics: TypeSim, which captures nuanced semantic relationships between predicted and ground truth types, and TypeCheck, which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TypeSim scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TypyBench provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at https://github.com/typybench/typybench.

Figures

Figures reproduced from arXiv: 2507.22086 by the authors.

Figure 1
Figure 1. Overview of TYPYBENCH. We collect well-typed Python repositories and remove their typing information as the inputs. The outputs predicted by type inference methods are then evaluated using TYPESIM and TYPECHECK, where TYPESIM measures the functionality similarity between predicted and human-annotated types, and TYPECHECK evaluates type consistency across entire codebases through static type checking. To address thes… view at source ↗
Figure 2
Figure 2. Examples of type similarity computation. Left: List-wise comparison for generic types, where arguments are compared in order. Right: Set-wise comparison for Union types, where an optimal matching is computed between members. 4. Metric Design To effectively evaluate type inference systems, we introduce two complementary metrics: TYPESIM, which measures type prediction quality, and TYPECHECK, which assesses type coher… view at source ↗
Figure 3
Figure 3. Distribution of repository categories in our benchmark dataset, showing coverage across major Python application do￾mains. 5.3. Benchmark Statistics To facilitate the potential evaluation of learning-based meth￾ods, we split the 50 repositories into Train/Validation/Test splits, with 20/10/20 repositories correspondingly. The 20 test repositories are selected and further split into two sets with 10 repositories each… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Comparison of TYPESIM and TYPECHECK metrics across models. SOTA models (CLAUDE-3.5-SONNET, DEEPSEEK-V3, GPT-4O, GPT-4O-MINI, and GROK-2) achieve similar high TYPESIM scores while varying in TYPECHECK, showing TYPECHECK’s additional discriminative power. Small local-hos…
Figure 5
Figure 5. Figure 5: TYPESIM scores v.s. type complexity. Models achieve consistent high scores for simple types (depths 1) but show de￾graded performance and increased variance for more complex types (depths 2+). complex type inference remains a significant challenge in some cases. Impact…
Figure 7
Figure 7. Figure 7: TYPESIM scores v.s. repository creation dates. Better performance observed on older repositories (pre-2024 avg: 0.705) compared to newer ones (2024+ avg: 0.649), indicating potential data contamination effects. this, we compare the TYPESIM scores of repositories cre￾at…
Figure 8
Figure 8. Figure 8: TYPESIM scores between builtin types. C. Experimental Settings We use Python version 3.12 and Mypy version 1.11.1 for all our experiments. For API models with multiple versions, we use GPT-4O-2024-08-06, GPT-4O-MINI-2024-07-18, CLAUDE-3-5-SONNET-20240620, GROK-2-1212, …
Figure 9
Figure 9. Figure 9: Prompt Template for Single File Context You are given a repository with repository directory structure and the source files. # Inputs ==== directory structure ==== {director structure} ==== source ==== The contents of the source code are as follows: {source code} # Tas…
Figure 10
Figure 10. Figure 10: Prompt Template for Whole Repo Context D.2. Results for Each Model We present the detailed results in [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    The llama 3 herd of models

    9 TYPY BENCH : Evaluating LLM Type Inference for Untyped Python Repositories Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  3. [6]

    J., Bird, C., Barr, E

    Hellendoorn, V . J., Bird, C., Barr, E. T., and Allamanis, M. Deep learning type inference. In Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 152–162,

    work page 2018

  4. [8]

    Lehtosalo, J., van Rossum, G., Levkivskyi, I., and Sullivan, M. J. mypy. http://mypy-lang.org/. Available at http://mypy-lang.org/. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437 , 2024a. Liu, T., Xu, C., and McAuley, J. Repobench: Benchma...

    arXiv 2019

  5. [10]

    Peng, Y ., Wang, C., Wang, W., Gao, C., and Lyu, M. R. Generative type inference for python. In 2023 38th IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE), pp. 988–999. IEEE,

    work page 2023

  6. [12]

    P., Sabu, S., Wang, J., M

    Shivarpatna Venkatesh, A. P., Sabu, S., Wang, J., M. Mir, A., Li, L., and Bodden, E. Typeevalpy: A micro- benchmarking framework for python type inference tools. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Pro- ceedings, pp. 49–53,

    work page 2024

  7. [13]

    Repotrans- bench: A real-world benchmark for repository-level code translation

    Wang, Y ., Wang, Y ., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y ., Mao, M., Zhang, H., et al. Repotrans- bench: A real-world benchmark for repository-level code translation. arXiv preprint arXiv:2412.17744,

    arXiv

  8. [14]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al

    Available at: https: //x.ai/blog/grok-2. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

Show all 20 references
  1. [15]

    Y ., Vu, M

    Zhuo, T. Y ., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877,

    arXiv

  2. [16]

    TYPE SIM scores between builtin types. C. Experimental Settings We use Python version3.12 and Mypy version 1.11.1 for all our experiments. For API models with multiple versions, we use GPT-4 O-2024-08-06 , GPT-4 O-MINI -2024-07-18 , CLAUDE -3-5- SONNET -20240620 , GROK -2-1212...

    2024

  3. [17]

    The test sets are further split into two test1 and test2 based on the date created to test the data contamination issue

    The statistics of all repositories in TYPY BENCH , including the total number of tokens, functions, variables to be inferred, the ratio of functions being annotated, the category, and the date created. The test sets are further split into two test1 and test2 based on the date ...

    2014

  4. [18]

    TYPE CHECK and TYPE SIM scores for GPT-4O-MINI on each repository. REPO TYPE TYPE TYPESIM MISSING TYPESIM BY DEPTH CHECK SIM WO MISSING RATIO DEPTH1 D EPTH2 D EPTH3 D EPTH4 D EPTH5 AGENTS 53 0.756 0.887 0.148 0.751 0.793 0.599 0.000 APPWORLD N/A 0.571 0.921 0.380 0.626 0.429 0...

  5. [19]

    TYPE CHECK and TYPE SIM scores for GROK -2 on each repository. REPO TYPE TYPE TYPESIM MISSING TYPESIM BY DEPTH CHECK SIM WO MISSING RATIO DEPTH1 D EPTH2 D EPTH3 D EPTH4 D EPTH5 AGENTS 37 0.700 0.900 0.222 0.695 0.742 0.493 0.000 APPWORLD N/A 0.535 0.931 0.426 0.596 0.395 0.413...

  6. [20]

    TYPE CHECK and TYPE SIM scores for QWEN -2.5-7 B on each repository. REPO TYPE TYPE TYPESIM MISSING TYPESIM BY DEPTH CHECK SIM WO MISSING RATIO DEPTH1 D EPTH2 D EPTH3 D EPTH4 D EPTH5 AGENTS 158 0.695 0.865 0.196 0.706 0.683 0.507 0.000 APPWORLD N/A 0.462 0.921 0.498 0.502 0.38...

    2008

  7. [2015]

    Data contamination through the lens of time

    Roberts, M., Thakur, H., Herlihy, C., White, C., and Dooley, S. Data contamination through the lens of time. arXiv preprint arXiv:2310.10628,

    arXiv

  8. [2018]

    Monkeytype: A python library that generates static type annotations by collecting run- time types

    Instagram. Monkeytype: A python library that generates static type annotations by collecting run- time types. https://github.com/Instagram/ MonkeyType. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: ...

    arXiv

  9. [2019]

    Pyright: Static type checker for python

    Microsoft. Pyright: Static type checker for python. https: //github.com/microsoft/pyright. Mir, A. M., Latoˇskinas, E., and Gousios, G. Manytypes4py: A benchmark python dataset for machine learning-based type inference. In 2021 IEEE/ACM 18th International Conference on Mining ...

    2021

  10. [2021]

    J., and Si, X

    Dong, H., Su, Q., Gao, Y ., Li, Z., Ruan, Y ., Pekhimenko, G., Maddison, C. J., and Si, X. Appl: A prompt pro- gramming language for harmonious integration of pro- grams and large language model prompts. arXiv preprint arXiv:2406.13161,

    arXiv

  11. [2023]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  12. [2024]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J

    Avail- able at: https://www.anthropic.com/news/ claude-3-5-sonnet . Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information proces...

    1901

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.