Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Deep Learning-based Code Completion: On the Impact on Performance of Contextual Information

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Giving a code completion model project context improves accuracy, and choosing predictions by model confidence yields a relative +22.4% over the baseline.

desk verdict Solid empirical study of code-completion contexts; the coding-context result holds up, but the headline +22% confidence-based ensemble needs baselines and calibration before I'd trust it. read the letter →

arxiv 2501.05062 v1 pith:KAAPTSFN submitted 2025-01-09 cs.SE

Matteo Ciniselli , Luca Pascarella , Gabriele Bavota This is my paper · ORCID
classification cs.SE
keywords codecompletiondeeplearningcontextualinformationT5empiricalstudyJavaconfidence-basedselectionsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a code completion model predicts missing Java statements better when it is given more than just the unfinished method, and its answer is yes. The authors define three families of context—coding (methods that call or are called by the target, the rest of the class, a retrieved similar method), process (the title or body of a related open issue), and developer (statements and method calls recently used by the person typing)—and fine-tune 18 T5 models on 85,266 real completion instances that differ only in context. Single contexts improve correct predictions by up to a relative +7%, the three coding contexts together give +11%, and a confidence-based rule that outputs, for each completion, the prediction of whichever model is most confident reaches 37.43% correct predictions versus 30.58% for the baseline, a relative +22.4%. The upshot is that feeding models project and developer signals can substantially improve code assistants, and that pooling several context models by confidence is a more effective use of those signals than training one model on one input format.

What carries the argument

The mechanism that carries the argument is a pool of T5 models with different input representations plus a selection rule. T5, a transformer-based text-to-text model, is fine-tuned on eight datasets that share the same target completions but differ in what context is appended to the incomplete method; combinations of contexts produce 18 models in total. Each generated prediction carries a log-likelihood, and the paper converts that value into a confidence score on a 0-to-1 scale. The confidence-based model is then the central object: it compares these scores across all context models for a given completion and outputs the prediction with the highest confidence. This design does the explanatory work because it turns context engineering from choosing one best input into building a set of complementary specialists and delegating each instance to the most confident one.

What would settle it

Re-run the confidence-based selector after separately calibrating each of the 18 models on a held-out set; if the +22.4% gain shrinks or vanishes once overconfident models are rescaled, the cross-model comparability assumption fails. Also compare the selector with a random baseline and a majority-vote ensemble of the same models, checking whether the gain comes from confidence calibration rather than from one model dominating the selection.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the context appended to an incomplete method is a first-order performance factor, and that different contexts are complementary. Using the same 85,266 statement-completion instances, the baseline T5 model (a transformer text-to-text model) that sees only the incomplete method produces 30.58% correct predictions, while a model augmented with the most similar method reaches 32.68% and one with method calls reaches 32.46%. Combining the three coding contexts (most similar method, method calls, class signatures) yields 33.88% correct predictions, a relative +11% over the baseline. The largest result comes from the confidence-based model, defined as follows: for each test instance, compute the T5 confidence score of every trained context model and emit the prediction with the highest score; this reaches 37.43% correct predictions, a relative +22.4% over the baseline. Complementarity analysis shows the context models and the baseline are not redundant: 20.90% of correct predictions come only from the confidence selector, while only 3.19% come only from the baseline. The intended takeaway is that contextual information should be supplied to multiple specialist models, and their confidence scores can be used to choose the best answer per instance.

Load-bearing premise

The load-bearing premise is that T5's confidence score for a generated completion is a real probability and is comparable across models fine-tuned on different contexts, so the prediction with the highest score is the one most likely to be correct.

Editorial extensions

If this is right

  • A single context already helps: most similar method gives a relative +7% over the baseline and method calls give +6%.
  • The three coding contexts together reach 33.88% correct predictions, a relative +11% over the baseline.
  • Choosing the highest-confidence prediction across all 18 models gives 37.43% correct predictions, a relative +22.4% over the baseline, with only 3.19% of correct predictions coming solely from the baseline.
  • Issue body and most similar statements are the only single contexts not statistically distinguishable from the baseline, and cross-family combinations do not beat the all-coding-context combination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if T5 confidence is approximately calibrated, the same highest-confidence selection rule can be applied to pools of models from different architectures or languages, without needing a fusion layer or retraining.
  • Editorial inference: the weaker performance of process and developer contexts may be a representation problem; dense embeddings of issue text or developer activity could transfer more of that signal than raw token strings do.
  • Editorial inference: the 85,266-instance intersection requires projects to have callable code, open issues, and developer history, so the measured gains are conditional on rich-project metadata; coding-context gains are the most likely to generalize to more modest repositories.
  • Editorial inference: a serving-time implementation could first consult the coding-context models, since they contribute the most correct predictions, and only fall back to other contexts when confidence is low.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical study of how different types of contextual information affect the performance of T5-based code completion at the Java method level. It defines three families of contexts (coding, process, and developer), builds 18 fine-tuned models on a dataset of 85,266 instances derived from 1,072 Java projects, and compares their correct-prediction rates against a baseline that sees only the incomplete method. The baseline achieves 30.58% correct predictions; the best within-family combination of coding contexts reaches 33.88% (a relative +11% improvement). A confidence-based model that selects the highest-confidence prediction among all 18 models reaches 37.43% (a relative +22.4% improvement, OR=6.56). The paper concludes that additional contextual information can substantially improve code completion and that combining models by confidence is promising.

Significance. If the results hold, this is a valuable large-scale empirical contribution that quantifies the benefit of different context families for code completion and demonstrates a practical way to combine context-augmented models. The study uses a substantial dataset, adopts a realistic masking strategy based on git-blame history, and applies standard significance testing (McNemar with Holm correction). The finding that coding contexts provide the largest gains is plausible and internally consistent. However, the headline confidence-based result rests on an unvalidated assumption about cross-model comparability of raw T5 log-likelihoods, and the 'most similar method' context may be inflated by same-project similarity. Both concerns are addressable with additional analysis.

major comments (3)
  1. [Section V, Table IV] The confidence-based model's +22.4% improvement relies on treating T5's raw sequence log-likelihood as a calibrated confidence score that is comparable across 18 models. This assumption is not justified. Completions are one or two statements of varying token length, so raw log-likelihood intrinsically favors shorter outputs; a correct longer completion can lose to an incorrect short one. In addition, the models are fine-tuned on inputs of very different lengths (Table I means range from 243 to 1,079 tokens), which can shift the score scales across models. The paper should provide a length-normalized or otherwise calibrated confidence score and show that the result is robust to that choice.
  2. [Section III-A, Section IV-A] The 'most similar method' context is retrieved from the training set, but the train/test split is at method level, not project level. Because the dataset is built from 1,072 repositories and methods from the same project can appear in both training and test, the retrieved similar method may come from the same project as the incomplete method, giving the model access to project-specific style and near-duplicate code. This could inflate the impact of the MSM context, which is part of the best-performing within-family combination (MSM+MC+CS, 33.88%). The authors should report the proportion of same-project similar-method retrievals and, ideally, re-evaluate the MSM context using a project-level split or a filter that excludes same-project methods.
  3. [Section V, Figure 3] The confidence-based selector is not compared against trivial baselines that would disentangle the effect of the confidence signal from the effect of having an ensemble. Since the best single contextual model achieves 33.88% and the combinational models already reach 33.54-33.75%, the paper should report the performance of always selecting the best single model, random model selection, and an oracle selector. Without these comparisons, it is unclear whether the jump to 37.43% comes from the confidence ranking or simply from having access to more models, and the OR=6.56 should be interpreted with caution.
minor comments (4)
  1. [Table II] The row label 'IT + IBy' appears to be a typo for 'IT + IB', and 'Best Code+ Best Developer' is missing a space after the plus sign.
  2. [References] Reference [9], the replication package, uses placeholder author 'A. authors' and a generic URL; if the package is available, the final version should provide the actual authors and URL. In addition, References [40] and [41] duplicate the same T5 paper and should be merged.
  3. [Figure 3] The x-axis of the confidence plots is labeled in a confusing way ('0.5 Confidence 10.1'); it should be a clear 0.0 to 1.0 scale with evenly spaced bucket labels.
  4. [Section IV-A] The hyperparameter configuration is tuned only on the baseline dataset, and the paper argues this may only improve the contextual models. This is a reasonable assumption but should be stated as an assumption rather than a fact, since a configuration that is optimal for the baseline is not guaranteed to be optimal for longer input representations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical context comparison and confidence-based model selection are evaluated against an independently reimplemented baseline; the self-citations to prior work are not load-bearing.

full rationale

The paper's derivation chain is empirical: it defines eight context types, builds one dataset per context, fine-tunes T5 models on each, measures exact-match correctness against a baseline, and finally combines the trained models by picking the output with the highest T5 log-likelihood. None of these steps reduces to its own inputs by construction. The baseline is inherited from Ciniselli et al. [13], but it is reimplemented in this study (the same T5 architecture and a reproduced 30.58% correctness, inline with the 29% reported there), so that self-citation is a comparison point rather than a load-bearing proof. The confidence-based model is an ad hoc selection rule applied to already-trained models; no parameter is fitted to the reported 37.43% figure, and correctness is measured by exact string match, not defined in terms of the confidence scores. Potential concerns about cross-model comparability of log-likelihoods, length bias, or making model-selection decisions on the test set are validity threats, but they are not circular reductions. No equation is equal to another by definition, and no cited uniqueness theorem or imported ansatz forces the choice of context or of the confidence rule. The central claim is therefore self-contained as an empirical comparison, and the score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several engineering choices (token budgets, top-k selections, masking limits) and domain assumptions (T5-base represents DL completion models, exact match is a valid metric, commit references indicate issue relevance). These are mostly reasonable choices, but the near-duplicate assumption for the similar-method context is the most fragile.

free parameters (6)
  • k (candidate similar methods) = 20
    Number of top Jaccard-similar methods re-ranked by CrystalBLEU to select the most similar method context; chosen as a scalability compromise, not optimized.
  • Token split (method/context) = 682/342
    Maximum tokens for the method representation and the additional context within the 1,024 input limit; chosen so the method gets two-thirds of the input.
  • Top-N developer statements = 10 statements
    Number of recent developer statements most similar to the incomplete method, included in the developer's most similar statements context.
  • Top-N frequent invocations = 100 calls
    Number of method calls frequently used by the developer, included in the frequent invocations context.
  • Max masked statements = 2
    At most two complete statements are masked to create instances, limiting the task complexity.
  • Fine-tuning hyperparameters = LR=0.001, batch=32, steps=160k
    Chosen on the baseline dataset only, then reused for all 18 models; possibly suboptimal for contextual models, but the authors note the bias would likely favor the baseline.
assumptions (5)
  • domain assumption T5-base is representative of DL-based code completion models.
    The study generalizes to 'DL-based code completion' from a single model architecture (T5-base, 220M); the external validity section explicitly acknowledges this limitation.
  • domain assumption Exact match (ignoring spaces) is a valid measure of completion quality.
    The paper uses correct predictions as the sole metric and discusses in the construct validity section that behaviorally equivalent predictions are counted as wrong.
  • domain assumption Commit message references to issue IDs indicate issue relevance for the method changed in the commit.
    Used to train TSDAE for retrieving the most relevant open issue for an incomplete method; weak labels come from commit messages, with MRR 0.34 vs 0.14 random.
  • domain assumption The 'high cohesion' principle (methods in a class are related) justifies the class signatures context.
    The paper relies on this software engineering principle to motivate appending signatures of other methods in the same class as input.
  • domain assumption The most similar method retrieved from the training set does not leak the expected completion.
    The study assumes that a method from the training set that is similar to the incomplete method does not contain the exact completion; near-duplicate methods are not explicitly excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning-based Code Completion: On the Impact on Performance of Contextual Information." pith.science (2026). https://pith.science/paper/KAAPTSFN

@misc{pith2026250105062,
  author       = {Pith},
  title        = {Pith review of: Deep Learning-based Code Completion: On the Impact on Performance of Contextual Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KAAPTSFN}},
  note         = {Machine review of arXiv:2501.05062}
}
read the original abstract

Code completion aims at speeding up code writing by recommending to developers the next tokens they are likely to type. Deep Learning (DL) models pushed the boundaries of code completion by redefining what these coding assistants can do: We moved from predicting few code tokens to automatically generating entire functions. One important factor impacting the performance of DL-based code completion techniques is the context provided as input. With "context" we refer to what the model knows about the code to complete. In a simple scenario, the DL model might be fed with a partially implemented function to complete. In this case, the context is represented by the incomplete function and, based on it, the model must generate a prediction. It is however possible to expand such a context to include additional information, like the whole source code file containing the function to complete, which could be useful to boost the prediction performance. In this work, we present an empirical study investigating how the performance of a DL-based code completion technique is affected by different contexts. We experiment with 8 types of contexts and their combinations. These contexts include: (i) coding contexts, featuring information extracted from the code base in which the code completion is invoked (e.g., code components structurally related to the one to "complete"); (ii) process context, with information aimed at depicting the current status of the project in which a code completion task is triggered (e.g., a textual representation of open issues relevant for the code to complete); and (iii) developer contexts, capturing information about the developer invoking the code completion (e.g., the APIs frequently used). Our results show that additional contextual information can benefit the performance of DL-based code completion, with relative improvements up to +22% in terms of correct predictions.

Figures

Figures reproduced from arXiv: 2501.05062 by the authors.

Figure 1
Figure 1. Experimented contexts: Examples from a real instance in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Method calls context helping the prediction. An OR=1.64 indicates 64% higher odds of obtaining a correct prediction using the contextual model as compared to the baseline. The complete statistical analysis is available in our replication package [9] [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Percentage of correct predictions when varying the confidence of the model. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Class signatures context helping the prediction of the confidence-based model. Internal validity. An important factor influencing DL per￾formance is the calibration of hyperparameters which, due to feasibility reasons, we limited to the baseline, assuming the others wo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 47 canonical work pages

  1. [1]

    Github copilot https://copilot.github.com

    “Github copilot https://copilot.github.com.”

    work page

  2. [2]

    Javalang https://github.com/c2nes/javalang/

    work page

  3. [3]

    ScrML Website, https://www.srcml.org/

    work page

  4. [4]

    Context-aware code recommen- dation in intellij IDEA,

    S. Abid, H. A. Basit, and S. Shamail, “Context-aware code recommen- dation in intellij IDEA,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE , 2022, pp. 1647–1651

    work page 2022

  5. [5]

    code2seq: Generating sequences from structured representations of code,

    U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019

    work page 2019

  6. [6]

    Structural language models of code,

    U. Alon, R. Sadaka, O. Levy, and E. Yahav, “Structural language models of code,” in International Conference on Machine Learning, ICML , 2020, pp. 245–256

    work page 2020

  7. [7]

    Context- sensitive code completion tool for better API usability,

    M. Asaduzzaman, C. K. Roy, K. A. Schneider, and D. Hou, “Context- sensitive code completion tool for better API usability,” in 30th IEEE International Conference on Software Maintenance and Evolution IC- SME, 2014, pp. 621–624

    work page 2014

  8. [8]

    A simple, efficient, context-sensitive approach for code comple- tion,

    ——, “A simple, efficient, context-sensitive approach for code comple- tion,” J. Softw. Evol. Process. , vol. 28, no. 7, pp. 512–541, 2016

    work page 2016

Show all 54 references
  1. [9]

    Replication package https://github.com/completion-context/ completion-context-replication

    A. authors, “Replication package https://github.com/completion-context/ completion-context-replication.”

  2. [10]

    Learning autocompletion from real-world datasets,

    G. A. Aye, S. Kim, and H. Li, “Learning autocompletion from real-world datasets,” in 43rd IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2021, Madrid, Spain, May 25-28, 2021 , 2021, pp. 131–139

    2021

  3. [11]

    Learning from examples to improve code completion systems,

    M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in 7th ACM Joint Meeting of the European Software Engineering Conference and the ACM/SIGSOFT In- ternational Symposium on Foundations of Software Engineering ESEC- FSE, 2009, pp...

    2009

  4. [12]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,...

    2021 arXiv

  5. [13]

    An empirical study on the usage of transformer models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella, A. Mastropaolo, E. Aghajani, D. Poshyvanyk, M. D. Penta, and G. Bavota, “An empirical study on the usage of transformer models for code completion,” IEEE Transactions on Software Engineering, TSE , vol. abs/2108.01585, no. 01, pp. 1–1, 2021

    2021 arXiv

  6. [14]

    An empirical study on the usage of BERT models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella, D. Poshyvanyk, M. D. Penta, and G. Bavota, “An empirical study on the usage of BERT models for code completion,” in 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021 , 2021, pp. 108–119

    2021

  7. [15]

    Smart formatter: Learning coding style from existing source code,

    F. Corbo, C. del Grosso, and M. di Penta, “Smart formatter: Learning coding style from existing source code,” in 2007 IEEE International Conference on Software Maintenance , 2007, pp. 525–526

    2007

  8. [16]

    Sampling projects in github for MSR studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for MSR studies,” in Proceedings of the 18th International Conference on Mining Software Repositories , ser. MSR’21, 2021, p. To appear. [Online]. Available: https://arxiv.org/abs/2103.04682

    2021 arXiv

  9. [17]

    Collective intelligence for smarter API recommendations in python,

    A. R. D’Souza, D. Yang, and C. V . Lopes, “Collective intelligence for smarter API recommendations in python,” in 16th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM, 2016, pp. 51–60

    2016

  10. [18]

    Crystalbleu: precisely and efficiently measuring the similarity of code,

    A. Eghbali and M. Pradel, “Crystalbleu: precisely and efficiently measuring the similarity of code,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022, pp. 341–342

    2022

  11. [19]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” in 10th Conference on Empirical Methods in Natural Language Processing, EMNLP , 2020, pp. 1536– 1547

    2020

  12. [20]

    A new algorithm for data compression,

    P. Gage, “A new algorithm for data compression,” The C Users Journal, vol. 12, no. 2, pp. 23–38, 1994

    1994

  13. [21]

    Google colab https://colab.research.google.com/

    Google, “Google colab https://colab.research.google.com/.”

  14. [22]

    java-callgraph https://github. com/gousiosg/java-callgraph

    G. Gousios, M. Vergne, and C. Laaber, “java-callgraph https://github. com/gousiosg/java-callgraph.”

  15. [23]

    Graphcodebert: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in 9th International Conf...

    2021

  16. [24]

    Complete completion us- ing types and weights,

    T. Gvero, V . Kuncak, I. Kuraj, and R. Piskac, “Complete completion us- ing types and weights,” in ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’13, Seattle, WA, USA, June 16-19, 2013 , 2013, pp. 27–38

    2013

  17. [25]

    Jaccard distance (jaccard index, jaccard similarity coefficient),

    J. M. Hancock, “Jaccard distance (jaccard index, jaccard similarity coefficient),” Dictionary of Bioinformatics and Computational Biology , 2004

    2004

  18. [26]

    Identifier-based context-dependent API method recommendation,

    L. Heinemann, V . Bauer, M. Herrmannsdoerfer, and B. Hummel, “Identifier-based context-dependent API method recommendation,” in 16th European Conference on Software Maintenance and Reengineering, CSMR, 2012, pp. 31–40

    2012

  19. [27]

    Are deep neural networks the best choice for modeling source code?

    V . J. Hellendoorn and P. Devanbu, “Are deep neural networks the best choice for modeling source code?” in 11th ACM/SIGSOFT Joint Meeting on Foundations of Software Engineering ESEC-FSE , 2017, p. 763?773

    2017

  20. [28]

    On the naturalness of software,

    A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. T. Devanbu, “On the naturalness of software,” in 34th IEEE/ACM International Conference on Software Engineering, ICSE , 2012, pp. 837–847

    2012

  21. [29]

    Linking developer experience to coding style in open-source repositories,

    H. Hokka, F. Dobslaw, and J. Bengtsson, “Linking developer experience to coding style in open-source repositories,” in 2021 IEEE Interna- tional Conference on Software Analysis, Evolution and Reengineering (SANER), 2021, pp. 516–520

    2021

  22. [30]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

    1979

  23. [31]

    Codefill: Multi-token code completion by jointly learning from structure and naming sequences,

    M. Izadi, R. Gismondi, and G. Gousios, “Codefill: Multi-token code completion by jointly learning from structure and naming sequences,” in Proceedings of the 44th International Conference on Software Engi- neering, 2022, p. 401?412

    2022

  24. [32]

    Maybe deep neural networks are the best choice for modeling source code,

    R. Karampatsis and C. A. Sutton, “Maybe deep neural networks are the best choice for modeling source code,” CoRR, vol. abs/1903.05734, 2019

    1903 arXiv

  25. [33]

    Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,

    T. Kudo and J. Richardson, “Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226 , 2018

    2018 arXiv

  26. [34]

    A proposal of code completion problem for java programming learning assistant system,

    H. H. S. Kyaw, S. T. Aung, H. A. Thant, and N. Funabiki, “A proposal of code completion problem for java programming learning assistant system,” in Conference on Complex, Intelligent, and Software Intensive Systems. Springer, 2018, pp. 855–864

    2018

  27. [35]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  28. [36]

    Beyond integrated development environments: adding context to software development,

    G. C. Murphy, “Beyond integrated development environments: adding context to software development,” in Proceedings of the 41st Interna- tional Conference on Software Engineering: New Ideas and Emerging Results, ICSE (NIER) 2019, Montreal, QC, Canada, May 29-31, 2019 , 2019, pp. 73–76

    2019

  29. [37]

    Graph-based pattern-oriented, context-sensitive source code completion,

    A. T. Nguyen, T. T. Nguyen, H. A. Nguyen, A. Tamrawi, H. V . Nguyen, J. Al-Kofahi, and T. N. Nguyen, “Graph-based pattern-oriented, context-sensitive source code completion,” in 2012 34th International Conference on Software Engineering (ICSE) , 2012, pp. 69–79

    2012

  30. [38]

    Type-directed completion of partial expressions,

    D. Perelman, S. Gulwani, T. Ball, and D. Grossman, “Type-directed completion of partial expressions,” in Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Imple- mentation, ser. PLDI ’12, 2012, p. 275?286

    2012

  31. [39]

    Intelligent code completion with bayesian networks,

    S. Proksch, J. Lerch, and M. Mezini, “Intelligent code completion with bayesian networks,” ACM Trans. Softw. Eng. Methodol. , vol. 25, no. 1, pp. 3:1–3:31, 2015

    2015

  32. [40]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2019

    2019

  33. [41]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    ——, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2019

    2019

  34. [42]

    Improving code completion with program history,

    R. Robbes and M. Lanza, “Improving code completion with program history,” Automated Software Engineering, ASE, vol. 17, no. 2, pp. 181– 212, 2010

    2010

  35. [43]

    Structured design,

    W. P. Stevens, G. J. Myers, and L. L. Constantine, “Structured design,” IBM Systems Journal , vol. 13, no. 2, pp. 115–139, 1974

    1974

  36. [44]

    Intellicode compose: code generation using transformer,

    A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: code generation using transformer,” in 28th ACM Joint European Software Engineering Conference and the ACM/SIGSOFT International Symposium on the Foundations of Software Engineering ESEC-FSE, 2020, pp...

    2020

  37. [45]

    Adding context to source code representations for deep learning,

    F. Tian and C. Treude, “Adding context to source code representations for deep learning,” in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2022, pp. 374–378

    2022

  38. [46]

    On the localness of software,

    Z. Tu, Z. Su, and P. T. Devanbu, “On the localness of software,” in 22nd ACM/SIGSOFT International Symposium on Foundations of Software Engineering, FSE, 2014, pp. 269–280

    2014

  39. [47]

    Unit test case generation with transform- ers,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transform- ers,” CoRR, vol. abs/2009.05617, 2020. [Online]. Available: https://arxiv.org/abs/2009.05617

    2009 arXiv

  40. [48]

    Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning,

    K. Wang, N. Reimers, and I. Gurevych, “Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning,” arXiv preprint arXiv:2104.06979 , 2021

    2021 arXiv

  41. [49]

    Towards full-line code completion with neural language models,

    W. Wang, S. Shen, G. Li, and Z. Jin, “Towards full-line code completion with neural language models,” arXiv preprint arXiv:2009.08603 , 2020

    2009 arXiv

  42. [50]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in 11st Conference on Empirical Methods in Natural Language Processing, EMNLP, 2021, pp. 8696–8708

    2021

  43. [51]

    Siri, write the next method,

    F. Wen, E. Aghajani, C. Nagy, M. Lanza, and G. Bavota, “Siri, write the next method,” in 43rd IEEE/ACM International Conference on Software Engineering (ICSE), 2021, pp. 138–149

    2021

  44. [52]

    Improve language modelling for code comple- tion by tree language model with tree encoding of context (S),

    Y . Yang and C. Xiang, “Improve language modelling for code comple- tion by tree language model with tree encoding of context (S),” in The 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019, Hotel Tivoli, Lisbon, Portugal, July 10-12, 20...

    2019

  45. [53]

    Improve language modelling for code completion through learn- ing general token repetition of source code,

    ——, “Improve language modelling for code completion through learn- ing general token repetition of source code,” in The 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019, Hotel Tivoli, Lisbon, Portugal, July 10-12, 2019 , 2019, pp. 667– 777

    2019

  46. [54]

    Apimatchmaker: Matching the right apis for supporting the development of android apps,

    Y . Zhao, L. Li, H. Wang, Q. He, and J. C. Grundy, “Apimatchmaker: Matching the right apis for supporting the development of android apps,” IEEE Trans. Software Eng. , vol. 49, no. 1, pp. 113–130, 2023

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.