Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs can verify and attribute code authorship with no training, reaching 65–69% top-1 accuracy with one reference per author.

desk verdict Genuinely useful first study of LLMs for code authorship attribution, with a load-bearing confound in the tournament experiment that likely overstates cross-project attribution. read the letter →

arxiv 2501.08165 v1 pith:6YYUDWAV submitted 2025-01-14 cs.SE cs.AI

Soohyeon Choi , Yong Kiam Tan , Mark Huasong Meng , Mohamed Ragab , Soumik Mondal , David Mohaisen , Khin Mi Mi Aung This is my paper · ORCID
classification cs.SEcs.AI
keywords codeauthorshipattributionlargelanguagemodelszero-shotpromptingfew-shotin-contextlearningtournamentstylometryadversarialrobustnesssourceforensics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that general-purpose large language models can attribute source code to its author without any task-specific training or labeled datasets. Using zero-shot prompting, the best models verify whether two C++ snippets share an author with Matthews correlation up to 0.78, and few-shot in-context learning attributes a query to one of a small candidate set with MCC up to 0.77. To get past the models' input-token limits, the paper introduces tournament prompting, which splits a large author pool into small subsets, picks the most similar author in each, and iterates; this reaches top-1 accuracy of 65% among 500 C++ authors and 68.7% among 686 Java authors using one reference per author. The authors also report partial resistance to two state-of-the-art misattribution attacks and argue the approach transfers across languages. Such training-free attribution would matter for code forensics, plagiarism detection, and protecting patch integrity.

What carries the argument

The load-bearing mechanism is few-shot in-context learning plus a tournament bracket built on top of it. In each round the query code is shown alongside reference snippets for a small subset of candidate authors (at most 12), and the LLM is asked to return the most similar author's ID; the winners of all subsets form the next round's pool, and the process repeats until one author remains. This converts an attribution problem too large for any prompt window into a sequence of small comparisons, at roughly USD 1.5 per final attribution on the tested datasets. The paper also varies prompt complexity (simple, detailed, complex) around the three stylometric feature families—layout, lexical, and syntactic—and finds that more guidance does not reliably help, with the simple prompt often winning.

What would settle it

Check whether a meaningful share of the crawled single-contributor 2024 files are AI-generated, templated, or co-authored; if so, the reported accuracy must be re-read as attributing code provenance rather than human authorship.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that state-of-the-art LLMs carry a latent stylistic fingerprint for code: asked in plain English whether two snippets are by the same programmer, GPT-4o and Gemini 1.5 Pro reach 0.78 MCC, and when given one to three reference snippets per candidate author in the prompt, they pick the true author from sets of three to ten candidates with accuracy up to 88.5% (Gemini, two-shot). The proposed tournament procedure extends this to 500 C++ and 686 Java authors with one reference each, yielding 65% and 68.7% top-1 accuracy on freshly crawled 2024 GitHub code. Against evasion and imitation attacks (MCTS and RoPGen), the LLMs keep attack success well below the 77–95% reported for machine-learning baselines, and an adversarial-aware prompt raises GPT-4o's accuracy to 70% under MCTS. Not all models succeed: Llama2, Llama3-base, and Mistral perform near chance or refuse to answer, so the capability belongs to the strongest current LLMs rather than to the whole family. The paper is careful to flag that the public Google Code Jam dataset may have leaked into training data, which is why the large-scale numbers are measured on GitHub code dated May–October 2024.

Load-bearing premise

The load-bearing premise is that each file in a single-contributor GitHub repository was actually written by that account's owner and not by AI or template reuse, and that the 2024 collection date keeps those files out of the models' training data; if either fails, the accuracy numbers measure something other than identifying a human writer.

Editorial extensions

If this is right

  • Code authorship verification becomes possible without labeled training data: a zero-shot pairwise check reaches MCC 0.78, which supports triage in forensic settings where per-author corpora do not exist.
  • Attribution over hundreds of authors is feasible with one reference per author, reaching 65% top-1 accuracy for 500 C++ authors and 68.7% for 686 Java authors.
  • The same prompts transfer from C++ to Java with no retraining, so the approach is language-agnostic at least across these two mainstream languages.
  • Standard misattribution attacks (MCTS, RoPGen) are less effective against LLMs than against machine-learning baselines, and adversarial-aware prompting further cuts attack success.
  • Deployment is bounded by cost and model choice: only a few LLMs perform well, and a single tournament attribution costs about USD 1.5 on the tested APIs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the single-contributor GitHub files include AI-generated code or shared templates, the reported accuracy is really code-provenance attribution, not human-style attribution, so the headline numbers should be re-interpreted as a lower bound on genuine authorship identification.
  • Beyond the paper: the tournament design implies a ranking among authors, not just a winner; the per-round winner lists could be recycled as a relevance ranking for code search or as a candidate filter for forensic tooling.
  • Beyond the paper: the same bracketing scheme should transfer to open-weights models fine-tuned on code, letting users run the pipeline offline and test whether the ability survives distillation or is unique to the large closed APIs.
  • Beyond the paper: a direct test of the leakage threat is to repeat the zero-shot verification on a purpose-built corpus of code written after the models' known training cutoffs, where one can compute how much of the MCC 0.78 survives.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using general-purpose LLMs for source-code authorship attribution without supervised training. It reports zero-shot pairwise authorship verification results with an MCC of up to 0.78 and few-shot attribution with MCC up to 0.77 on Google Code Jam C++ and Java data; it then introduces a tournament prompting scheme to scale attribution to large candidate pools and evaluates it on freshly crawled GitHub repositories (500 C++ authors, 686 Java authors), reporting Top-1 accuracy of 65% and 68.7%, respectively, with one reference sample per author. The paper also tests robustness against MCTS and RoPGen adversarial transformations and checks generalization from C++ to Java. The central claim is that state-of-the-art LLMs provide a practical, training-free authorship signal for verification, small-scale attribution, and, via tournament prompting, large-scale attribution.

Significance. If the results are correct, the paper is significant: it would establish that LLMs can perform code authorship attribution without task-specific labeled training, that the approach transfers across languages with unchanged prompts, and that a simple tournament procedure can extend the method to hundreds of authors. The authors also take several commendable precautions, including using post-2024 GitHub data for the large-scale experiments, fixing temperature to 0, scoring indeterminate outputs as wrong, and explicitly acknowledging the Google Code Jam data-leakage risk. However, the strongest scaling claim is currently undermined by an unstated and uncontrolled relationship between query samples and reference samples in the GitHub experiments, and several headline numbers rest on small test sets. The significance is therefore real but conditional on tightening these experimental controls.

major comments (4)
  1. [Sec. IV-B and Sec. III-A] The tournament experiment does not specify whether the one-shot reference sample for an author is drawn from a repository disjoint from the query sample. Since the GitHub dataset is built from single-contributor repositories with an average of 52.7 files per C++ author, a random reference for an author will typically come from the same repository as a random query by that author. The LLM could then attribute by matching repository-level conventions such as include headers, macros, file-header comments, and formatting rather than by recognizing the author's style across projects. This confound directly affects the headline claims in Table V and Table X. Please state the split explicitly, or better, re-run the large-scale experiments with references and queries drawn from disjoint repositories for each author, and report both settings.
  2. [Sec. III-C, Sec. III-D, and Sec. VI-B] The zero-shot and few-shot headline numbers (MCC up to 0.78 and 0.77) are computed on Google Code Jam 2017 data, a public dataset that may well be present in the training data of the closed commercial models. The paper acknowledges this threat in Sec. VI-B but does not resolve it. Since the only fresh-data experiments are the tournament runs, the abstract's verification and attribution claims remain vulnerable to training-data contamination. I would like to see the same zero-shot and few-shot protocols applied to the fresh GitHub data, or, failing that, a clear demotion of the GCJ-based numbers in the abstract and conclusions.
  3. [Sec. III-C, Sec. III-D, Sec. IV-B, Sec. V-B] The test sets are small: 200 pairs for verification, 200 instances per condition for few-shot attribution, and 300 queries per language for the tournament. For 300 samples, an accuracy of 65% has a 95% confidence interval of roughly ±5.4%; for 200 samples, an accuracy of 89% has a confidence interval of roughly ±4.3%. Several cross-prompt and cross-model differences reported in Tables III, IV, VIII, and IX are within this margin. The paper should report confidence intervals and, where feasible, increase the number of test samples, especially for the round-wise accuracy values in Tables V and X, which are currently presented as if the differences between models and rounds were precisely estimated.
  4. [Sec. III-A] The GitHub dataset defines an 'author' as the owner of a single-contributor repository and assumes that every file in the repository is the authentic stylistic product of that account. The paper does not control for AI-generated code, code copied from templates or tutorials, or vendored/third-party files. As a result, the large-scale experiments may be measuring attribution to a repository or account rather than to a human author. This limitation should be discussed explicitly, and ideally the dataset should be filtered to reduce such cases or the results should be reported separately for files with stronger evidence of human authorship.
minor comments (5)
  1. [Sec. IV-A, Algorithm 1] The parameter 'sample size' is used in the algorithm but is not defined in the pseudocode; the text later says the experiments use subsets of at most size 12. Please add a definition or a comment in the algorithm.
  2. [Tables II and IX] Several confusion-matrix rows sum to fewer than 200 because the models returned indeterminate answers, but the indeterminate count is not shown in the table. Adding an explicit 'Indeterminate' column, or reporting the number of indeterminate responses in a footnote, would make the tables much easier to read.
  3. [Table V and Table X] The '2nd round' and '3rd round' columns are described as accuracy values; it would be clearer to state explicitly that these are cumulative probabilities that the true author has not been eliminated by that round, rather than per-round classification accuracies.
  4. [Sec. VI-B] There is a typo in the sentence 'we carried our our tournament prompting experiments'; it should read 'we carried out our tournament prompting experiments.'
  5. [Sec. III-A and Sec. VI-A] For reproducibility, please record the exact API model versions and snapshot dates for GPT-4o and Gemini 1.5 Pro, since closed-source APIs are updated frequently and the reported results may not be reproducible with later versions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose reported accuracies are independent measurements, not fitted or definitionally forced outputs.

full rationale

This is an empirical benchmark study rather than a derivation, and no step was found in which a reported prediction is produced by construction from its inputs. The tournament scheme in Sec. IV-A defines each round's winner as argmax over HL(T,a) for a subset of authors and recursively reduces the pool; the reported Top-1 accuracy is an independent measurement of whether the LLM's choices retain the ground-truth author, not an identity or a fitted quantity. No parameter is fit to the evaluation set in a way that is later renamed as a prediction, no uniqueness theorem from the authors' prior work is invoked, and the few self-citations (e.g., [3], [4], [8]) are related-work references rather than load-bearing evidence. The GCJ contamination caveat (Sec. VI-B) is a data-leakage threat, and the single-repository GitHub construction could create a same-repository matching confound, but these are external validity concerns about label quality and task construction, not circular derivation; the paper explicitly flags the GCJ issue and attempts a fresh-data mitigation. Accordingly, no circular step is exhibited, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two data-labeling axioms (GCJ participant equals author; GitHub single-contributor repo equals one human author), one unverifiable data-hygiene axiom (fresh GitHub code is outside closed-model training data), and one statistical-representativeness axiom (small samples without error bars pin the metrics). No free parameters are fitted to data; the only hand-chosen settings are the tournament subset size (12) and the post hoc selection of the simple prompt for the Java and tournament runs. No new entities are postulated, so the graviton problem does not apply. The paper's own threats-to-validity section (VI-B) independently concedes the GCJ leakage risk, which supports this audit.

free parameters (2)
  • Tournament subset size (sample size) = 12
    Hand-chosen in Algorithm 1 and Sec. IV-B to fit the LLM token window ('at most size 12, which fits well into the input token window for the LLMs'). It determines the round structure 500 to 42 to 4 to 1 and hence how far attribution errors can propagate; no ablation is reported.
  • Selected prompt template for Java and tournament runs = P1 (simple)
    Chosen post hoc because the simple prompt usually won on the C++ runs (Sec. V-B). This selected configuration affects all RQ5 Java and tournament numbers.
assumptions (4)
  • domain assumption GCJ 2017 participant IDs are ground-truth authors of the C++/Java samples used in RQ1, RQ2, and RQ4.
    The verification and attribution experiments treat each GCJ participant as a distinct author and each code sample as their sole work (Sec. III-A). If teammates or shared competition solution patterns dilute authorship, the MCC values are inflated.
  • domain assumption Each single-contributor GitHub repository (May-Oct 2024) is the stylistic product of one human author.
    Underpins the RQ3/RQ5 headline Top-1 numbers (Sec. III-A Datasets). In 2024, a material share of GitHub code is LLM-generated or templated, so the 'author style' may not belong to a single human.
  • domain assumption Code committed May-Oct 2024 is outside the training data of GPT-4o and Gemini 1.5 Pro.
    Stated in Sec. VI-B as 'based on publicly known information'. Unverifiable for closed API models; if false, the GitHub results are contaminated in the same way the paper fears for GCJ.
  • domain assumption The 100-300 sample test sets are representative, with sampling noise small enough to support the stated accuracy and MCC comparisons.
    No confidence intervals or repeated sampling are reported (Tables II-X), so the paper implicitly assumes its random samples pin the metrics within a few percent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution." pith.science (2026). https://pith.science/paper/6YYUDWAV

@misc{pith2026250108165,
  author       = {Pith},
  title        = {Pith review of: I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YYUDWAV}},
  note         = {Machine review of arXiv:2501.08165}
}
read the original abstract

Source code authorship attribution is important in software forensics, plagiarism detection, and protecting software patch integrity. Existing techniques often rely on supervised machine learning, which struggles with generalization across different programming languages and coding styles due to the need for large labeled datasets. Inspired by recent advances in natural language authorship analysis using large language models (LLMs), which have shown exceptional performance without task-specific tuning, this paper explores the use of LLMs for source code authorship attribution. We present a comprehensive study demonstrating that state-of-the-art LLMs can successfully attribute source code authorship across different languages. LLMs can determine whether two code snippets are written by the same author with zero-shot prompting, achieving a Matthews Correlation Coefficient (MCC) of 0.78, and can attribute code authorship from a small set of reference code snippets via few-shot learning, achieving MCC of 0.77. Additionally, LLMs show some adversarial robustness against misattribution attacks. Despite these capabilities, we found that naive prompting of LLMs does not scale well with a large number of authors due to input token limitations. To address this, we propose a tournament-style approach for large-scale attribution. Evaluating this approach on datasets of C++ (500 authors, 26,355 samples) and Java (686 authors, 55,267 samples) code from GitHub, we achieve classification accuracy of up to 65% for C++ and 68.7% for Java using only one reference per author. These results open new possibilities for applying LLMs to code authorship attribution in cybersecurity and software engineering.

Figures

Figures reproduced from arXiv: 2501.08165 by the authors.

Figure 1
Figure 1. We will start by investigating RQ1 and RQ2. A. Experiment Setup Model Selection and Experiment Environment: Our em￾pirical study covers four mainstream LLM families, namely OpenAI’s GPT [12], Meta’s Llama [13], Mistral [20], and Google’s Gemini [11]. The GPT and Gemini models are closed-source, so we conduct experiments using their official APIs. For the remaining two model families (Llama and Mistral), we use the l… view at source ↗
Figure 1
Figure 1. An overview of LLM-based code authorship attribution. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Prompt templates (Simple, Detailed, and Complex) for the zero-shot code authorship verification experiment. TABLE II: Confusion matrix for LLM-based code authorship verification with zero-shot prompts over 200 random C++ code sample pairs. For prompts where the entries (TP, FN, TN, FP) sum to a value below 200, the LLM returned indeterminate answers for the remaining cases. P1 P2 P3 Models TP FN TN FP TP FN TN FP TP… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Prompt templates (Simple, Detailed, and Complex) for the few-shot (one-, two-, and three-shot) code authorship attribution experiment. LLMs, focusing attention on Gemini-1.5-p and GPT-4o as they were the most promising models from RQ1. For code attribution, we supplied…
Figure 4
Figure 4. Figure 4: Overview of tournament prompting. had the best results against 3 authors. However, the remaining experimental results do not always show such a clear trend. Second, we also expected that increasing the number of code samples per author would make the tasks simpler for …
Figure 5
Figure 5. Figure 5: Prompt templates Simple for the code authorship experiment with tournament prompting. TABLE V: Tournament prompting results for Gemini-1.5- p and GPT-4o were evaluated using GitHub C++ with 300 randomly sampled C++ query code samples. Models Accuracy(%) Gemini-1.5-p 2n…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Channel Spread-Spectrum Code Watermarking

    cs.CR 2026-07 unverdicted novelty 7.0 of 10

    A training-free post-hoc code watermark embeds 24-bit identifiers via multi-channel spread-spectrum encoding over naming conventions and semantic pattern pairs, with majority voting and Reed-Solomon recovery.

  2. I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A fine-tuned encoder-only CodeT5 model attributes LLM-generated C code to its source model with up to 97.56% binary and 95.40% five-class accuracy on a new 32,000-program benchmark.

  3. A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations

    cs.CR 2025-02 conditional novelty 2.0 of 10

    A literature review that taxonomizes LLM backdoor attacks and defenses by model construction phase, with no new experimental results.

  4. A Contemporary Survey of Large Language Model Assisted Program Analysis

    cs.SE 2025-02 conditional novelty 1.0 of 10

    A review that catalogs how large language models are used in static, dynamic, and hybrid program analysis, and outlines open challenges.

Reference graph

Works this paper leans on

44 extracted references · 34 canonical work pages · cited by 4 Pith papers

  1. [1]

    Code authorship attribution: Methods and challenges,

    V . Kalgutkar, R. Kaur, H. Gonzalez, N. Stakhanova, and A. Matyukhina, “Code authorship attribution: Methods and challenges,” ACM Comput. Surv., vol. 52, no. 1, pp. 3:1–3:36, 2019

    work page 2019

  2. [2]

    Source code authorship attribution using n-grams,

    S. Burrows and S. M. Tahaghoghi, “Source code authorship attribution using n-grams,” in ADCS, 2007, pp. 32–39

    work page 2007

  3. [3]

    Untargeted code authorship evasion with Seq2Seq transformation,

    S. Choi, R. Jang, D. Nyang, and D. Mohaisen, “Untargeted code authorship evasion with Seq2Seq transformation,” inCSoNet, ser. LNCS, M. H. H `a, X. Zhu, and M. T. Thai, Eds., vol. 14479. Springer, 2023, pp. 83–92

    work page 2023

  4. [4]

    Attributing chatgpt-generated source codes,

    S. Choi and D. Mohaisen, “Attributing chatgpt-generated source codes,” IEEE Trans. Dependable Secur. Comput. , 2024

    work page 2024

  5. [5]

    Source code authorship attribution using long short-term memory based networks,

    B. Alsulami, E. Dauber, R. E. Harang, S. Mancoridis, and R. Greenstadt, “Source code authorship attribution using long short-term memory based networks,” in ESORICS, ser. LNCS, S. N. Foley, D. Gollmann, and E. Snekkenes, Eds., vol. 10492. Springer, 2017, pp. 65–82

    work page 2017

  6. [6]

    Authorship attribution of source code: a language-agnostic approach and applicability in software engineering,

    E. Bogomolov, V . Kovalenko, Y . Rebryk, A. Bacchelli, and T. Bryksin, “Authorship attribution of source code: a language-agnostic approach and applicability in software engineering,” in FSE, D. Spinellis, G. Gousios, M. Chechik, and M. D. Penta, Eds. ACM, 2021, pp. 932–944

    work page 2021

  7. [7]

    On the feasibility of malware authorship attribution,

    S. Alrabaee, P. Shirani, M. Debbabi, and L. Wang, “On the feasibility of malware authorship attribution,” in FPS, ser. LNCS, F. Cuppens, L. Wang, N. Cuppens-Boulahia, N. Tawbi, and J. Garc ´ıa-Alfaro, Eds., vol. 10128. Springer, 2016, pp. 256–272

    work page 2016

  8. [8]

    Large-scale and language-oblivious code authorship identification,

    M. Abuhamad, T. AbuHmed, A. Mohaisen, and D. Nyang, “Large-scale and language-oblivious code authorship identification,” in CCS, D. Lie, M. Mannan, M. Backes, and X. Wang, Eds. ACM, 2018, pp. 101–114

    work page 2018

Show all 44 references
  1. [9]

    De-anonymizing programmers via code stylometry,

    A. Caliskan-Islam, R. E. Harang, A. Liu, A. Narayanan, C. R. V oss, F. Yamaguchi, and R. Greenstadt, “De-anonymizing programmers via code stylometry,” in USENIX Security , J. Jung and T. Holz, Eds. USENIX Association, 2015, pp. 255–270

    2015

  2. [10]

    Multi- χ: Identifying multiple authors from source code files,

    M. Abuhamad, T. AbuHmed, D. Nyang, and D. Mohaisen, “Multi- χ: Identifying multiple authors from source code files,” Proc. Priv. Enhancing Technol., vol. 2020, no. 3, pp. 25–41, 2020

    2020

  3. [11]

    Google, “Gemini,” https://ai.google.dev/gemini-api/docs, 2024, ac- cessed: 2024-05-12

    2024

  4. [12]

    ChatGPT,

    OpenAI, “ChatGPT,” https://openai.com/blog/chatgpt, 2024, accessed: 2024-05-12

    2024

  5. [13]

    Meta, “Llama,” https://llama.meta.com/, 2024, accessed: 2024-05-12

    2024

  6. [14]

    A survey of large language models,

    Wayne Xin Zhao et al., “A survey of large language models,” CoRR, vol. abs/2303.18223, 2023

    2023 arXiv

  7. [15]

    Translating natural language to planning goals with large-language models,

    Y . Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, “Translating natural language to planning goals with large-language models,” CoRR, vol. abs/2302.05128, 2023

    2023 arXiv

  8. [16]

    Wordcraft: Story writing with large language models,

    A. Yuan, A. Coenen, E. Reif, and D. Ippolito, “Wordcraft: Story writing with large language models,” in IUI, G. Jacucci, S. Kaski, C. Conati, S. Stumpf, T. Ruotsalo, and K. Gajos, Eds. ACM, 2022, pp. 841–852

    2022

  9. [17]

    A system- atic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A system- atic evaluation of large language models of code,” in MAPS@PLDI, S. Chaudhuri and C. Sutton, Eds. ACM, 2022, pp. 1–10

    2022

  10. [18]

    Using large language models to enhance programming error messages,

    J. Leinonen, A. Hellas, S. Sarsa, B. N. Reeves, P. Denny, J. Prather, and B. A. Becker, “Using large language models to enhance programming error messages,” in SIGCSE, M. Doyle, B. Stephenson, B. Dorn, L. Soh, and L. Battestilli, Eds. ACM, 2023, pp. 563–569

    2023

  11. [19]

    Can large language models identify authorship?

    B. Huang, C. Chen, and K. Shu, “Can large language models identify authorship?” CoRR, vol. abs/2403.08213, 2024

    2024 arXiv

  12. [20]

    Mistral 7b,

    Albert Q. Jiang et al., “Mistral 7b,” CoRR, vol. abs/2310.06825, 2023

    2023 arXiv

  13. [21]

    Misleading authorship attribu- tion of source code using adversarial learning,

    E. Quiring, A. Maier, and K. Rieck, “Misleading authorship attribu- tion of source code using adversarial learning,” in USENIX Security , N. Heninger and P. Traynor, Eds. USENIX Association, 2019, pp. 479–496

    2019

  14. [22]

    RoPGen: Towards robust code authorship attribution via automatic coding style transformation,

    Z. Li, Q. G. Chen, C. Chen, Y . Zou, and S. Xu, “RoPGen: Towards robust code authorship attribution via automatic coding style transformation,” in ICSE. ACM, 2022, pp. 1906–1918

    2022

  15. [23]

    Using metrics to evaluate student programs,

    R. J. Leach, “Using metrics to evaluate student programs,” ACM SIGCSE Bull., vol. 27, no. 2, pp. 41–43, 1995

    1995

  16. [24]

    Authorship analysis: identifying the author of a program,

    I. Krsul and E. H. Spafford, “Authorship analysis: identifying the author of a program,” Comput. Secur., vol. 16, no. 3, pp. 233–257, 1997

    1997

  17. [25]

    Finding plagiarisms among a set of programs with JPlag,

    L. Prechelt, G. Malpohl, and M. Philippsen, “Finding plagiarisms among a set of programs with JPlag,” J. Univers. Comput. Sci. , vol. 8, no. 11, p. 1016, 2002

    2002

  18. [26]

    On the use of discretized source code metrics for author identification,

    M. Shevertalov, J. Kothari, E. Stehle, and S. Mancoridis, “On the use of discretized source code metrics for author identification,” in SSBSE. IEEE, 2009, pp. 69–78

    2009

  19. [27]

    Robin: A novel method to produce robust interpreters for deep learning- based code classifiers,

    Z. Li, R. Zhang, D. Zou, N. Wang, Y . Li, S. Xu, C. Chen, and H. Jin, “Robin: A novel method to produce robust interpreters for deep learning- based code classifiers,” in ASE. IEEE, 2023, pp. 27–39

    2023

  20. [28]

    Document of chatgpt,

    OpenAI, “Document of chatgpt,” https://platform.openai.com/docs/ model-index-for-researchers, 2024, accessed: 2024-05-12

    2024

  21. [29]

    Gemini: A family of highly capable multimodal models,

    Rohan Anil et al., “Gemini: A family of highly capable multimodal models,” CoRR, vol. abs/2312.11805, 2023

    2023 arXiv

  22. [30]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,” ACM Trans. Intell. Syst. Technol., vol. 15, no. 3, pp. 39:1–39:45, 2024

    2024

  23. [31]

    Jigsaw: Large language models meet program synthesis,

    N. Jain, S. Vaidyanath, A. S. Iyer, N. Natarajan, S. Parthasarathy, S. K. Rajamani, and R. Sharma, “Jigsaw: Large language models meet program synthesis,” in ICSE. ACM, 2022, pp. 1219–1231

    2022

  24. [32]

    Refining ChatGPT-generated code: Characterizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X. D. Le, and D. Lo, “Refining ChatGPT-generated code: Characterizing and mitigating code quality issues,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 5, pp. 116:1–116:26, 2024

    2024

  25. [33]

    Investigating documented privacy changes in android OS,

    C. Yan, M. H. Meng, F. Xie, and G. Bai, “Investigating documented privacy changes in android OS,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, pp. 2701–2724, 2024

    2024

  26. [34]

    Chatting with GPT-3 for zero-shot human-like mobile automated GUI testing,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Chatting with GPT-3 for zero-shot human-like mobile automated GUI testing,” CoRR, vol. abs/2305.09434, 2023

    2023 arXiv

  27. [35]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in ISSTA, R. Just and G. Fraser, Eds. ACM, 2023, pp. 423–435

    2023

  28. [36]

    Evaluating large language models trained on code,

    Mark Chen et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021

    2021 arXiv

  29. [37]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in EMNLP, ser. Findings of ACL, T. Cohn, Y . He, and Y . Liu, Eds., vol. EMNLP 2020. Association for Comp...

    2020

  30. [38]

    An empirical study on learning bug-fixing patches in the wild via neural machine translation,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Trans. Softw. Eng. Methodol., vol. 28, no. 4, pp. 19:1–19:29, 2019

    2019

  31. [39]

    Google code jam,

    Google, “Google code jam,” https://codingcompetitions.withgoogle.com/ codejam/archive, 2024, accessed: 2024-05-12

    2024

  32. [40]

    The matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evalua- tion,

    D. Chicco, N. T ¨otsch, and G. Jurman, “The matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evalua- tion,” BioData Min., vol. 14, no. 1, p. 13, 2021

    2021

  33. [41]

    LLMs assist NLP researchers: Critique paper (meta- )reviewing,

    Jiangshu Du et al., “LLMs assist NLP researchers: Critique paper (meta- )reviewing,” CoRR, vol. abs/2406.16253, 2024

    2024 arXiv

  34. [42]

    A prompt pattern catalog to en- hance prompt engineering with ChatGPT,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to en- hance prompt engineering with ChatGPT,” CoRR, vol. abs/2302.11382, 2023

    2023 arXiv

  35. [43]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” CoRR, vol. abs/2402.07927, 2024

    2024 arXiv

  36. [44]

    A multiscale visualization of attention in the transformer model,

    J. Vig, “A multiscale visualization of attention in the transformer model,” in ACL, M. R. Costa-juss `a and E. Alfonseca, Eds. Association for Computational Linguistics, 2019, pp. 37–42

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.