The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages

Boqi Chen; D\'aniel Varr\'o; Gunter Mussbacher; Jos\'e Antonio Hern\'andez L\'opez

REVIEW 4 major objections 4 minor 60 references

The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that neural bug detectors for Python are trained and evaluated on datasets that include bugs a type checker can catch, and that removing those type-related bugs changes both measured performance and what the models learn.

desk verdict A careful empirical study showing type-checkable bugs inflate Python NBD benchmarks and filtering them from training helps recall; the same-line type-checker labeling is the main thing to probe, but the direction of the findings is credible. read the letter →

arxiv 2411.15368 v2 pith:7NMG253V submitted 2024-11-22 cs.SE cs.LGcs.PL

Boqi Chen , José Antonio Hernández López , Gunter Mussbacher , Dániel Varró This is my paper · ORCID

classification cs.SEcs.LGcs.PL

keywords neuralbugdetectionvariablemisusetypecheckingPythondynamicallytypedlanguagespytypemypytype-relatedbugs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that neural bug detectors for Python variable misuse are being trained and evaluated on datasets contaminated with bugs an ordinary type checker could already catch. It claims this contamination inflates reported performance and steers models toward easy, type-checkable cases rather than the harder bugs that remain after type checking. If right, existing benchmark numbers need to be re-read, and training data should be filtered of type-related bugs whenever a type checker is part of the workflow. The payoff is a more honest estimate of what neural detectors add beyond cheap static checking, especially for annotated Python code.

What carries the argument

The central object is the type-related bug, defined operationally as a variable misuse that pytype or mypy reports as an error on the same line as the bug when run on a stripped single-function snippet with missing imports auto-added. This labeling pipeline carries the whole argument because it partitions every dataset into type-related versus other bugs, and every research question reuses that partition. The second machinery piece is the pipeline setting, where a type checker runs first and the neural detector only faces code the type checker has cleared, mirroring how static typing makes type checking a precondition in statically typed languages. The third piece is the filtered training protocol, which replaces type-related bugs with oversampled non-type bugs to keep training-set size and class balance fixed while shifting the detector's focus.

What would settle it

Run pytype and mypy on full repository checkouts for the annotated 400-program sample and compare the same-line, stripped-snippet labels against full-project type-checker reports; if the two disagree on a substantial fraction, the prevalence figures and the RQ3/RQ4 conclusions would need to be re-estimated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a substantial share of variable misuse bugs in standard Python datasets are detectable by a type checker, and that this share distorts both evaluation and training of neural bug detectors. It finds that 19.56% of one synthetic dataset, 14.69% of another, and 4.95% of real-world bugs are caught by pytype even without type annotations, while annotated code yields 15.00% with pytype and 18.00% with mypy. When type-related bugs are removed from the evaluation set, precision and recall drop for every detector, showing the models perform better on exactly the bugs a type checker would already flag. When type-related bugs are removed from training data, recall on the remaining bugs improves in every tested case, with a smaller average precision loss of 0.56 percentage points, so the filtered training wins most comparisons when recall is prioritized. The paper also shows that running a type checker before the neural detector improves recall on unannotated real code by 2.31% to 3.97%, and improves both precision and recall on annotated code.

Load-bearing premise

The whole analysis depends on trusting the paper's definition of a type-related bug: a type checker error on the same line as the variable misuse, computed on isolated function snippets with auto-added imports, is treated as what a real project type checker would catch.

Editorial extensions

If this is right

Existing neural bug detector evaluations that mix type-checkable bugs with harder bugs are optimistic whenever a type checker is available.
Training on data filtered of type-related bugs improves recall on the remaining bugs across all tested detectors, at a small average precision cost.
Combining a type checker with a neural detector improves recall on unannotated Python code, and improves both precision and recall when the code is type-annotated.
At a recall-priority setting ($\beta \ge 1.62$), the type-checker-plus-detector pipeline beats the detector alone for all tested models.
Annotated code raises the share of bugs a type checker catches to 15–18%, making the pipeline more attractive and making unfiltered evaluation more misleading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the same-line labeling undercounts what a full-project type checker would catch, then the pipeline's real benefit is larger than measured and the contamination problem is worse than reported.
A testable extension is to apply the same filtering recipe to JavaScript or TypeScript variable-misuse datasets; the expectation is a similar but possibly smaller type-related contamination because the tooling and annotation cultures differ.
Oversampling existing non-type bugs rather than generating fresh synthetic non-type bugs likely understates the benefit of filtered training, since fresh bugs could recover part of the observed precision loss.
The paper does not model neural type inference, but if such tools become practical, the set of bugs a checker can catch will change, and the type-related partition used here would need to be recomputed rather than assumed stable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this paper actually delivers on its title. It quantifies how many variable misuse bugs in standard Python NBD datasets are detectable by pytype/mypy, and shows that these type-related bugs inflate measured NBD performance and that removing them from training shifts models toward harder, non-type bugs. The core numbers (19.56% of Synthetic 1 training, 4.95% of Real eval, 15-18% of annotated) are new and worth knowing.

What's good: the experimental design is more careful than most in this space. They use five architectures spanning GNN, transformer, and hybrid; they filter data leakage between real and synthetic splits; they manually annotate a balanced subset of real programs; and they report type-checker precision/recall separately. They also vary beta in F-beta to show when the pipeline wins. The threats-to-validity section honestly acknowledges the single seed and the lack of ground truth for type-relatedness.

The soft spot is exactly the one the stress-test flags: the type-related labeling is operationalized as 'pytype or mypy reports an error on the same line as the variable misuse' on a stripped single-function snippet with auto-added imports. That can miss type errors that only surface with full project context and can count unrelated errors on the same line. The paper concedes there is no ground truth. This matters because RQ3 and RQ4 inherit that labeling. But I don't think it's load-bearing in the sense of overturning the conclusions: the effect sizes are consistent across synthetic, real, and annotated data, and even if the label is noisy, the direction—type-checkable bugs exist in these benchmarks and NBDs do better on them—is robust.

Minor concerns: single seed with no confidence intervals is real but acknowledged; the Annotated set is 400 programs from one annotator, which is small but reasonable for a first pass. The oversampling strategy for RQ4 is a reasonable choice, though it could explain the precision drop.

Who this is for: anyone building or evaluating neural bug detectors for Python, and anyone designing variable misuse benchmarks. It deserves a serious referee; the central claim is credible and the paper is honest about its limits. I'd engage with it.

Recommendation: send to peer review. With a request for multiple seeds or at least variance reporting, and a validation of the type-related labeling against full-project type checking on a sample, this would be solid.

Referee Report

4 major / 4 minor

Summary. This paper studies whether bugs detectable by Python type checkers (pytype and mypy) contaminate neural bug detection (NBD) benchmarks for variable-misuse, and whether filtering them changes NBD training and evaluation. The authors type-check synthetic and real-world datasets, manually annotate 400 real programs, and compare five NBDs (GGNN, GREAT, CodeBERT, GraphCodeBERT, UniXcoder) under NBD-only versus pipeline settings and on full versus filtered training/evaluation data. They report that type-related bugs are prevalent, are easier for NBDs, inflate evaluation metrics, and that removing them from training improves recall for non-type-related bugs at some cost in precision.

Significance. If the findings hold, the paper makes a useful contribution: variable-misuse benchmark scores mix type-checkable and harder bugs, so evaluations involving type checkers should report the two categories separately, and training-data curation to exclude type-checkable bugs is a plausible way to improve complementarity. The study is commendable for covering five architectures, using both synthetic and real-world data, filtering potential train/test leakage (Section IV-B2), and releasing artifacts. The main empirical claims are, however, conditional on a same-line labeling of type-relatedness and on single-seed runs, so the exact effect sizes and the strength of the training-filtering recommendation are not yet established.

major comments (4)

[§III-D] The definition of a 'type-related bug' as any type-checker error on the same line as the variable misuse, computed on a stripped single-function snippet with auto-added imports, is the load-bearing construct for RQ1, RQ3, and RQ4. This operationalization is not validated against full-project type checking, and it can fail in both directions: context loss (missing aliases, class attributes, module-level state) can make real type errors invisible, while unrelated errors on the same line or artifacts of the reconstruction can be counted as type-related. The procedure never checks whether the error disappears when the correct variable is restored. I ask for a validation on a sample of projects, running pytype/mypy on the full repository or checkout context and comparing labels; at minimum, report the disagreement rate and a manual inspection of the same-line errors.
[§V-A, Tables III-IV] All experiments use a single random seed and no confidence intervals or significance tests, as acknowledged in Section V-A. Many of the reported differences are very small—for example, CodeBERT recall on Real moves from 31.02% to 31.06% and GraphCodeBERT precision from 28.80% to 28.69% in Table IV—so the RQ4 claim that filtering training 'causes a significant increase in recall' and the 9-out-of-10 F-beta statement are not supported without variance estimates. Please report multiple seeds (at least for the smaller models), or bootstrap confidence intervals and appropriate paired tests, and state which differences are robust.
[§III-C, Table II] The Annotated dataset underpins the RQ2 and RQ3 results on annotated code, but it consists of only 400 programs annotated by a single author, with no inter-rater reliability or agreement metric reported. Manual annotation choices directly change which bugs are labeled type-related (hence the 15-18% prevalence figures and the large Annotated deltas in Table III). Please provide details of the annotation protocol, the number of author discussions/resolutions, and ideally a second annotator on a subset with Cohen's kappa.
[§IV-E1, Table IV] The RQ4 comparison conflates filtering type-related bugs with a change in training distribution caused by random oversampling of 'other bugs.' Because filtered training keeps the same total size by duplicating a subset of non-type-related examples, any observed difference could reflect the oversampling distribution rather than the removal of type-related bugs per se. A control is needed: train on a random same-size subset of the unfiltered training data (or on an oversampled unfiltered baseline) and compare. Without this control, the training-filtering recommendation in Takeaway 3 is not uniquely supported.

minor comments (4)

[§VII] There is a typo in 'type annotaion' in the conclusion; it should be 'type annotation.'
[Figure 4] The bottom panel reports mypy category names that were 'manually converted' to pytype names; the mapping and any ambiguous cases should be documented for reproducibility.
[§IV-B4] Fine-tuning details (optimizer, warmup, batch size, number of gradient steps) are not specified; please add them for reproducibility.
[§III-B2] The relationship between the Real-Py150 and Real-PyPI counts (1,292 and 1,051 bugs) and the final merged Real dataset size is unclear; please state explicitly how many buggy and correct programs remain after merging and deduplication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the type-related bug classification is an operational measurement, and the headline claims are empirical outcomes rather than derived quantities.

full rationale

The paper's central claims (prevalence of type-related bugs in RQ1, the performance gap in RQ3, and the training-filter improvement in RQ4) are measured empirical results. 'Type-related bug' is operationally defined in Section III-D as a bug for which pytype or mypy reports an error on the same line as the variable misuse after preprocessing, so downstream analyses inherit that operational definition; however, this is a construct-validity threat about label noise, not circularity, because the NBD precision/recall/F-beta numbers are independent measurements and no fitted parameter is renamed as a prediction. The only self-citation to overlapping prior work ([9], Khan, Chen, Varró, McIntosh) is used to motivate the annotation process and to contextualize the 15% detection rate; it is not load-bearing for any derivation. The RQ2 recall improvement is partly a logical consequence of OR-combining a type checker with an NBD in the pipeline, but the paper reports measured magnitudes and precision trade-offs, making this a trivial entailment rather than a circular reduction. No equation-level circularity or self-citation chain forces the conclusions.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study introduces no new entities or fitted parameters. It relies on several domain assumptions about how type checkers and NBDs are deployed, the most load-bearing being the same-line error criterion that defines type-related bugs.

assumptions (4)

domain assumption A bug is type-related iff pytype or mypy reports an error on the same line as the variable misuse location.
Section III-D defines the labeling; this operationalization partitions all datasets and drives the RQ2, RQ3, and RQ4 analyses.
domain assumption The unannotated single-function snippet with auto-added imports faithfully represents real project code for type-checking purposes.
Section III-D preprocessing adds missing modules and ignores import and pyi errors, but the snippet still lacks surrounding project context that could change type inference results.
domain assumption In a pipeline, the type checker runs first and catches all type-related bugs, so the NBD only needs to handle remaining bugs.
This normative setup is introduced in Section I and Figure 1 and is the basis for the RQ2 pipeline evaluation.
domain assumption A hand-annotated balanced set of 400 programs (200 correct, 200 faulty) represents annotated real-world Python programs.
Section III-C describes the annotation by one author with discussion; the small and balanced sample may not capture the full distribution of annotation styles or class imbalance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages." pith.science (2026). https://pith.science/paper/7NMG253V

@misc{pith2026241115368,
  author       = {Pith},
  title        = {Pith review of: The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NMG253V}},
  note         = {Machine review of arXiv:2411.15368}
}

read the original abstract

Motivation: Automated bug detection in dynamically typed languages such as Python is essential for maintaining code quality. The lack of mandatory type annotations in such languages can lead to errors that are challenging to identify early with traditional static analysis tools. Recent progress in deep neural networks has led to increased use of neural bug detectors. In statically typed languages, a type checker is integrated into the compiler and thus taken into consideration when the neural bug detector is designed for these languages. Problem: However, prior studies overlook this aspect during the training and testing of neural bug detectors for dynamically typed languages. When an optional type checker is used, assessing existing neural bug detectors on bugs easily detectable by type checkers may impact their performance estimation. Moreover, including these bugs in the training set of neural bug detectors can shift their detection focus toward the wrong type of bugs. Contribution: We explore the impact of type checking on various neural bug detectors for variable misuse bugs, a common type targeted by neural bug detectors. Existing synthetic and real-world datasets are type-checked to evaluate the prevalence of type-related bugs. Then, we investigate how type-related bugs influence the training and testing of the neural bug detectors. Findings: Our findings indicate that existing bug detection datasets contain a significant proportion of type-related bugs. Building on this insight, we discover integrating the neural bug detector with a type checker can be beneficial, especially when the code is annotated with types. Further investigation reveals neural bug detectors perform better on type-related bugs than other bugs. Moreover, removing type-related bugs from the training data helps improve neural bug detectors' ability to identify bugs beyond the scope of type checkers.

Figures

Figures reproduced from arXiv: 2411.15368 by the authors.

**Figure 2.** Type annotation process for the Real dataset We include the correct programs from the test splits as well as faulty programs from all splits. In total, the dataset contains 1,292 bugs and 61,539 correct programs PyPI Real Bugs (Real-PyPI) [42] contains 2436 faulty functions with different types of bugs found in all 285k packages from the Python Package Index (PyPI). We only filter out the variable misuse bugs and us… view at source ↗

**Figure 3.** An overview of the type-related bugs labeling process [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Distribution of type-related bugs in the synthetic, real-world, and annotated datasets. In [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: (a) Precision and recall on the Real dataset. (b) F-β scores for different values of β TABLE II: Precision and recall of the bug detectors (a) and type checkers (b) on Annotated annotated with types for the NBD and pipeline performance on the unannotated (raw) and anno…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 58 canonical work pages

[1]

McConnell, Code complete - A practical handbook of software construction, 2nd Edition

S. McConnell, Code complete - A practical handbook of software construction, 2nd Edition . Microsoft Press, 2004

work page 2004
[2]

How many of all bugs do we find? A study of static bug detectors,

A. Habib and M. Pradel, “How many of all bugs do we find? A study of static bug detectors,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 . ACM, 2018, pp. 317–328

work page 2018
[3]

An empirical study on the correctness of formally verified distributed systems,

P. Fonseca, K. Zhang, X. Wang, and A. Krishnamurthy, “An empirical study on the correctness of formally verified distributed systems,” in Proceedings of the Twelfth European Conference on Computer Systems, EuroSys 2017, Belgrade, Serbia, April 23-26, 2017 . ACM, 2017, pp. 328–343

work page 2017
[4]

On the presence and causes of lingering defects in software: An industrial study of lingering defects,

S. Jemal, “On the presence and causes of lingering defects in software: An industrial study of lingering defects,” Ph.D. dissertation, Blekinge Institute of Technology, 2022

work page 2022
[5]

Software bug contributed to blackout,

K. Poulsen, “Software bug contributed to blackout,” 2004. [Online]. Available: https://www.theregister.com/2004/02/12/software bug contributed to blackout/

work page 2004
[6]

The real cost of software errors,

M. Zhivich and R. K. Cunningham, “The real cost of software errors,” IEEE Secur. Priv., vol. 7, no. 2, pp. 87–90, 2009

work page 2009
[7]

A large-scale study of programming languages and code quality in GitHub,

B. Ray, D. Posnett, P. T. Devanbu, and V . Filkov, “A large-scale study of programming languages and code quality in GitHub,” Commun. ACM, vol. 60, no. 10, pp. 91–100, 2017

work page 2017
[8]

To type or not to type: Quantifying detectable bugs in JavaScript,

Z. Gao, C. Bird, and E. T. Barr, “To type or not to type: Quantifying detectable bugs in JavaScript,” in Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Ar- gentina, May 20-28, 2017 . IEEE / ACM, 2017, pp. 758–769

work page 2017

Show all 60 references

[9]

An empirical study of type-related defects in Python projects,

F. Khan, B. Chen, D. Varr ´o, and S. McIntosh, “An empirical study of type-related defects in Python projects,” IEEE Trans. Software Eng. , vol. 48, no. 8, pp. 3145–3158, 2022

2022
[10]

Building useful program analysis tools using an extensible Java compiler,

E. Aftandilian, R. Sauciuc, S. Priya, and S. Krishnan, “Building useful program analysis tools using an extensible Java compiler,” in 12th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2012, Riva del Garda, Italy, September 23-24,

2012
[11]

Moving fast with software verification,

C. Calcagno, D. Distefano, J. Dubreil, D. Gabi, P. Hooimeijer, M. Luca, P. W. O’Hearn, I. Papakonstantinou, J. Purbrick, and D. Rodriguez, “Moving fast with software verification,” in NASA Formal Methods - 7th International Symposium, NFM 2015, Pasadena, CA, USA, April 27-29, ...

2015
[12]

Finding bugs is easy,

D. Hovemeyer and W. W. Pugh, “Finding bugs is easy,” ACM SIGPLAN Notices, vol. 39, no. 12, pp. 92–106, 2004

2004
[13]

Neural bug finding: A study of opportunities and challenges,

A. Habib and M. Pradel, “Neural bug finding: A study of opportunities and challenges,” CoRR, vol. abs/1906.00307, 2019

1906 arXiv
[14]

Learning to repre- sent programs with graphs,

M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to repre- sent programs with graphs,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018

2018
[15]

Probing pretrained models of source codes,

S. Troshin and N. Chirkova, “Probing pretrained models of source codes,” in Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , BlackboxNLP@EMNLP 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 8, 2022 . Association fo...

2022
[16]

Neural program repair by jointly learning to localize and repair,

M. Vasic, A. Kanade, P. Maniatis, D. Bieber, and R. Singh, “Neural program repair by jointly learning to localize and repair,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019

2019
[17]

Learning and evaluating contextual embedding of source code,

A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , ser. Proceedings of Machine Learning Researc...

2020
[18]

Global relational models of source code,

V . J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber, “Global relational models of source code,” in 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020

2020
[19]

The evolution of type annotations in Python: An empirical study,

L. D. Grazia and M. Pradel, “The evolution of type annotations in Python: An empirical study,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November ...

2022
[20]

Gated graph sequence neural networks,

Y . Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016

2016
[21]

CodeBERT: A pre-trained model for programming and natural languages,

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 20...

2020
[22]

GraphCodeBERT: Pre-training code representations with data flow,

D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “GraphCodeBERT: Pre-training code representations with data flow,” in 9th International Conf...

2021
[23]

UniXcoder: Unified cross-modal pre-training for code representation,

D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “UniXcoder: Unified cross-modal pre-training for code representation,” in Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22...

2022
[24]

Understanding neural code intelligence through program simplification,

M. R. I. Rabin, V . J. Hellendoorn, and M. A. Alipour, “Understanding neural code intelligence through program simplification,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 20...

2021
[25]

How to train your neural bug detector: Artificial vs real bugs,

C. Richter and H. Wehrheim, “How to train your neural bug detector: Artificial vs real bugs,” in 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 2023, pp. 1036–1048

2023
[26]

Learning to fix build errors with Graph2Diff neural networks,

D. Tarlow, S. Moitra, A. Rice, Z. Chen, P. Manzagol, C. Sutton, and E. Aftandilian, “Learning to fix build errors with Graph2Diff neural networks,” in ICSE ’20: 42nd International Conference on Software Engineering, Workshops, Seoul, Republic of Korea, 27 June - 19 July,
[27]

How often do single-statement bugs occur?: The ManySStuBs4J dataset,

R. Karampatsis and C. Sutton, “How often do single-statement bugs occur?: The ManySStuBs4J dataset,” in MSR ’20: 17th International Conference on Mining Software Repositories, Seoul, Republic of Korea, 29-30 June, 2020 . ACM, 2020, pp. 573–577

2020
[28]

Graph neural networks in program analysis,

M. Allamanis, “Graph neural networks in program analysis,” in Graph neural networks: foundations, frontiers, and applications . Springer, 2022, pp. 483–497

2022
[29]

Paper artifacts

B. Chen, J. A. Hern ´andez L ´opez, G. Mussbacher, and D. Varr ´o, “Paper artifacts.” [Online]. Available: https://github.com/20001LastOrder/ icse2025-type4py
[30]

Python 3 types in the wild: a tale of two type systems,

I. Rak-amnouykit, D. McCrevan, A. L. Milanova, M. Hirzel, and J. Dolby, “Python 3 types in the wild: a tale of two type systems,” in DLS 2020: Proceedings of the 16th ACM SIGPLAN International Symposium on Dynamic Languages, Virtual Event, USA, November 17,

2020
[31]

Pep 484 – type hints

G. van Rossum, J. Lehtosalo, and L. Langa, “Pep 484 – type hints.” [Online]. Available: https://peps.python.org/pep-0484/
[32]

mypy - optional static typing for Python

“mypy - optional static typing for Python.” [Online]. Available: https://mypy-lang.org/
[33]

Gradual typing for objects,

J. G. Siek and W. Taha, “Gradual typing for objects,” in ECOOP 2007 - Object-Oriented Programming, 21st European Conference, Berlin, Germany, July 30 - August 3, 2007, Proceedings , ser. Lecture Notes in Computer Science, vol. 4609. Springer, 2007, pp. 2–27

2007
[34]

ACM, 2020, pp. 57–70

2020
[35]

Deep learning type inference,

V . J. Hellendoorn, C. Bird, E. T. Barr, and M. Allamanis, “Deep learning type inference,” in Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista...

2018
[36]

Typilus: Neural type hints,

M. Allamanis, E. T. Barr, S. Ducousso, and Z. Gao, “Typilus: Neural type hints,” in Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2020, London, UK, June 15-20, 2020 . ACM, 2020, pp. 91–105

2020
[37]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long...

2017
[38]

pytype - a static type analyzer for Python code

“pytype - a static type analyzer for Python code.” [Online]. Available: https://google.github.io/pytype/
[39]

Semi-supervised classification with graph convolutional networks,

T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings, 2017

2017
[40]

On the properties of neural machine translation: Encoder-decoder approaches,

K. Cho, B. van Merrienboer, D. Bahdanau, and Y . Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” in Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 201...

2014
[41]

On distribution shift in learning-based bug detectors,

J. He, L. Beurer-Kellner, and M. T. Vechev, “On distribution shift in learning-based bug detectors,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp....

2022
[42]

A survey of transformers,

T. Lin, Y . Wang, X. Liu, and X. Qiu, “A survey of transformers,” AI Open, vol. 3, pp. 111–132, 2022

2022
[43]

How well static type checkers work with gradual typing? a case study on python,

W. Xu, L. Chen, C. Su, Y . Guo, Y . Li, Y . Zhou, and B. Xu, “How well static type checkers work with gradual typing? a case study on python,” in 2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC). IEEE, 2023, pp. 242–253

2023
[44]

A probabilistic interpretation of precision, recall and F-score, with implication for evaluation,

C. Goutte and ´E. Gaussier, “A probabilistic interpretation of precision, recall and F-score, with implication for evaluation,” in Advances in Information Retrieval, 27th European Conference on IR Research, ECIR 2005, Santiago de Compostela, Spain, March 21-23, 2005, Proceedin...

2005
[45]

Deep learning based identification of suspicious return statements,

G. Li, H. Liu, J. Jin, and Q. Umer, “Deep learning based identification of suspicious return statements,” in 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020 . IEEE, 2020, pp. 480–491

2020
[46]

Self-supervised bug detection and repair,

M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, “Self-supervised bug detection and repair,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , 2021, pp. 27 8...

2021
[47]

Learning realistic mutations: Bug creation for neural bug detectors,

C. Richter and H. Wehrheim, “Learning realistic mutations: Bug creation for neural bug detectors,” in 15th IEEE Conference on Software Testing, Verification and Validation, ICST 2022, Valencia, Spain, April 4-14,

2022
[48]

Semantic bug seeding: A learning-based approach for creating realistic bugs,

J. Patra and M. Pradel, “Semantic bug seeding: A learning-based approach for creating realistic bugs,” in ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23- 28, 2021. ACM, 2...

2021
[49]

Graph-based, self-supervised program repair from diagnostic feedback,

M. Yasunaga and P. Liang, “Graph-based, self-supervised program repair from diagnostic feedback,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event , ser. Proceedings of Machine Learning Research, vol. 119. PML...

2020
[50]

Learning defect predic- tion from unrealistic data,

K. Alrashedy, V . J. Hellendoorn, and A. Orso, “Learning defect predic- tion from unrealistic data,” CoRR, vol. abs/2311.00931, 2023

2023 arXiv
[51]

Type prediction with program decomposition and fill-in-the-type training,

F. Cassano, M. Yee, N. Shinn, A. Guha, and S. Holtzen, “Type prediction with program decomposition and fill-in-the-type training,” CoRR, vol. abs/2305.17145, 2023

2023 arXiv
[52]

Do machine learning models produce TypeScript types that type check?

M. Yee and A. Guha, “Do machine learning models produce TypeScript types that type check?” in 37th European Conference on Object-Oriented Programming, ECOOP 2023, July 17-21, 2023, Seattle, Washington, United States, ser. LIPIcs, vol. 263. Schloss Dagstuhl - Leibniz-Zentrum f¨...

2023
[53]

Typewriter: Neural type prediction with search-based validation,

M. Pradel, G. Gousios, J. Liu, and S. Chandra, “Typewriter: Neural type prediction with search-based validation,” in ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8- ...

2020
[54]

TypeT5: Seq2seq type inference using static analysis,

J. Wei, G. Durrett, and I. Dillig, “TypeT5: Seq2seq type inference using static analysis,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023

2023
[55]

SelfAPR: Self-supervised program repair with test execution diagnostics,

H. Ye, M. Martinez, X. Luo, T. Zhang, and M. Monperrus, “SelfAPR: Self-supervised program repair with test execution diagnostics,” in 37th IEEE/ACM International Conference on Automated Software Engineer- ing, ASE 2022, Rochester, MI, USA, October 10-14, 2022. ACM, 2022, pp. 9...

2022
[60]

OptTyper: Prob- abilistic type inference by optimising logical and natural constraints,

I. V . Pandi, E. T. Barr, A. D. Gordon, and C. Sutton, “OptTyper: Prob- abilistic type inference by optimising logical and natural constraints,” CoRR, vol. abs/2004.00348, 2020

2004 arXiv
[2012]

IEEE Computer Society, 2012, pp. 14–23

2012
[2020]

ACM, 2020, pp. 19–20

2020
[2022]

IEEE, 2022, pp. 162–173

2022
[9058]

Springer, 2015, pp. 3–11

2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

McConnell, Code complete - A practical handbook of software construction, 2nd Edition

S. McConnell, Code complete - A practical handbook of software construction, 2nd Edition . Microsoft Press, 2004

work page 2004

[2] [2]

How many of all bugs do we find? A study of static bug detectors,

A. Habib and M. Pradel, “How many of all bugs do we find? A study of static bug detectors,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 . ACM, 2018, pp. 317–328

work page 2018

[3] [3]

An empirical study on the correctness of formally verified distributed systems,

P. Fonseca, K. Zhang, X. Wang, and A. Krishnamurthy, “An empirical study on the correctness of formally verified distributed systems,” in Proceedings of the Twelfth European Conference on Computer Systems, EuroSys 2017, Belgrade, Serbia, April 23-26, 2017 . ACM, 2017, pp. 328–343

work page 2017

[4] [4]

On the presence and causes of lingering defects in software: An industrial study of lingering defects,

S. Jemal, “On the presence and causes of lingering defects in software: An industrial study of lingering defects,” Ph.D. dissertation, Blekinge Institute of Technology, 2022

work page 2022

[5] [5]

Software bug contributed to blackout,

K. Poulsen, “Software bug contributed to blackout,” 2004. [Online]. Available: https://www.theregister.com/2004/02/12/software bug contributed to blackout/

work page 2004

[6] [6]

The real cost of software errors,

M. Zhivich and R. K. Cunningham, “The real cost of software errors,” IEEE Secur. Priv., vol. 7, no. 2, pp. 87–90, 2009

work page 2009

[7] [7]

A large-scale study of programming languages and code quality in GitHub,

B. Ray, D. Posnett, P. T. Devanbu, and V . Filkov, “A large-scale study of programming languages and code quality in GitHub,” Commun. ACM, vol. 60, no. 10, pp. 91–100, 2017

work page 2017

[8] [8]

To type or not to type: Quantifying detectable bugs in JavaScript,

Z. Gao, C. Bird, and E. T. Barr, “To type or not to type: Quantifying detectable bugs in JavaScript,” in Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Ar- gentina, May 20-28, 2017 . IEEE / ACM, 2017, pp. 758–769

work page 2017