{"id":"b5bcc890-1df0-4569-8f00-387f86548a4e","arxiv_id":"2501.08165","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot and few-shot LLM prompting can attribute source code authorship across C++ and Java, and a tournament prompting scheme scales this to 500 to 686 candidate authors with 65 to 69 percent top-1 accuracy.","lead":"This paper tests whether off-the-shelf large language models can guess who wrote a piece of source code without any training on labeled examples. The best models match same-author code pairs with about 88 percent accuracy and reach about 65 percent top-1 accuracy when picking one author from 500 candidates using a tournament-style prompting trick.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GitHub tournament's 65%/68.7% Top-1 may measure same-repository file matching, not authorship: the paper never states that the one reference per author comes from a different repository than the query, and its single-contributor-repo construction makes same-repo pairing the default.","rationale":"I read the paper as an empirical claim that frontier LLMs can attribute code authorship in zero-shot and few-shot settings, with the tournament prompting (Sec. IV) as the method for scaling to hundreds of authors. The evidence for the verification and small-scale attribution claims (RQ1/RQ2) is based on GCJ, where query and reference are different tasks, so those claims are not affected by my concern. The large-scale claim (RQ3) is the most novel and impactful, and it is the one that most needs an independent split between reference and query contexts. The reader's weakest assumption concerned the authenticity of GitHub 'author' labels (AI-generated or templated code). My concern is orthogonal: even if the labels are perfectly correct, the tournament may be matching repositories, not authors. This is the single most load-bearing issue because it determines whether the 65%/68.7% numbers support the paper's headline 'find you in seconds' claim. The paper's own validity discussion (Sec. VI-B) addresses training-data overlap but never mentions the reference/query split. I therefore recommend keeping the CONDITIONAL verdict, with the added condition that the authors clarify and validate the cross-repository setting. I do not see grounds for REJECT, because the small-scale results are internally consistent and the flaw is fixable; nor ACCEPT until the split is verified. Agreement with the reader: partial—we both target the GitHub dataset, but the reader focused on label authenticity and contamination, while I focus on the train/test (reference/query) independence.","tokens_in":20448,"tokens_out":7890,"duration_ms":79034,"concrete_test":"Recruit a subset of GitHub authors who own at least two single-contributor repositories in the same language. Run the identical tournament protocol with all reference codes taken from repository A and all query codes taken from repository B (no file overlap; if a query's true author has no second repo, exclude it). Compare the resulting Top-1 accuracy to the reported same-repo accuracy of 65% (C++, Gemini/GPT-4o) and 68.7% (Java, GPT-4o). If the cross-repo accuracy is substantially lower—e.g., near the final-stage chance level of ~25% or below 45%—the headline result is confounded by repository-level consistency. As a minimal check, the authors should at minimum report the proportion of queries whose reference came from the same repository in the current protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central large-scale claim (Sec. IV-B, Table V) rests on the GitHub dataset: 500 C++ authors, 26,355 samples, collected from single-contributor repositories (Sec. III-A). The tournament experiment selects 300 random query samples and 'a corresponding reference set of one-shot reference code samples for each author,' but the paper never specifies whether reference and query files for the same author are drawn from the same repository or from disjoint repositories. Since each author's files are overwhelmingly likely to come from one repository (52.7 files/author on average), a random reference for an author will typically be from the same repository as a random query from that author. The LLM can then attribute by matching repository-level stylistic consistency—shared include headers, macros, file-header comments, naming conventions, and formatting—rather than by identifying the author's personal style across projects. The reported final-round accuracy of 65% (Gemini) and 66% (GPT-4o) on C++, and 50%/68.7% on Java, may therefore be an artifact of the train/reference split rather than evidence of cross-project authorship attribution. The small-scale GCJ experiments avoid this confound because query and reference are different tasks, but the tournament is the paper's headline contribution for scaling. This is an internal control issue, distinct from the data-contamination and label-authenticity threats already noted by the reader.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using general-purpose LLMs for source-code authorship attribution without supervised training. It reports zero-shot pairwise authorship verification results with an MCC of up to 0.78 and few-shot attribution with MCC up to 0.77 on Google Code Jam C++ and Java data; it then introduces a tournament prompting scheme to scale attribution to large candidate pools and evaluates it on freshly crawled GitHub repositories (500 C++ authors, 686 Java authors), reporting Top-1 accuracy of 65% and 68.7%, respectively, with one reference sample per author. The paper also tests robustness against MCTS and RoPGen adversarial transformations and checks generalization from C++ to Java. The central claim is that state-of-the-art LLMs provide a practical, training-free authorship signal for verification, small-scale attribution, and, via tournament prompting, large-scale attribution.","tokens_in":20659,"tokens_out":6124,"duration_ms":66963,"significance":"If the results are correct, the paper is significant: it would establish that LLMs can perform code authorship attribution without task-specific labeled training, that the approach transfers across languages with unchanged prompts, and that a simple tournament procedure can extend the method to hundreds of authors. The authors also take several commendable precautions, including using post-2024 GitHub data for the large-scale experiments, fixing temperature to 0, scoring indeterminate outputs as wrong, and explicitly acknowledging the Google Code Jam data-leakage risk. However, the strongest scaling claim is currently undermined by an unstated and uncontrolled relationship between query samples and reference samples in the GitHub experiments, and several headline numbers rest on small test sets. The significance is therefore real but conditional on tightening these experimental controls.","major_comments":[{"comment":"The tournament experiment does not specify whether the one-shot reference sample for an author is drawn from a repository disjoint from the query sample. Since the GitHub dataset is built from single-contributor repositories with an average of 52.7 files per C++ author, a random reference for an author will typically come from the same repository as a random query by that author. The LLM could then attribute by matching repository-level conventions such as include headers, macros, file-header comments, and formatting rather than by recognizing the author's style across projects. This confound directly affects the headline claims in Table V and Table X. Please state the split explicitly, or better, re-run the large-scale experiments with references and queries drawn from disjoint repositories for each author, and report both settings.","section":"Sec. IV-B and Sec. III-A"},{"comment":"The zero-shot and few-shot headline numbers (MCC up to 0.78 and 0.77) are computed on Google Code Jam 2017 data, a public dataset that may well be present in the training data of the closed commercial models. The paper acknowledges this threat in Sec. VI-B but does not resolve it. Since the only fresh-data experiments are the tournament runs, the abstract's verification and attribution claims remain vulnerable to training-data contamination. I would like to see the same zero-shot and few-shot protocols applied to the fresh GitHub data, or, failing that, a clear demotion of the GCJ-based numbers in the abstract and conclusions.","section":"Sec. III-C, Sec. III-D, and Sec. VI-B"},{"comment":"The test sets are small: 200 pairs for verification, 200 instances per condition for few-shot attribution, and 300 queries per language for the tournament. For 300 samples, an accuracy of 65% has a 95% confidence interval of roughly ±5.4%; for 200 samples, an accuracy of 89% has a confidence interval of roughly ±4.3%. Several cross-prompt and cross-model differences reported in Tables III, IV, VIII, and IX are within this margin. The paper should report confidence intervals and, where feasible, increase the number of test samples, especially for the round-wise accuracy values in Tables V and X, which are currently presented as if the differences between models and rounds were precisely estimated.","section":"Sec. III-C, Sec. III-D, Sec. IV-B, Sec. V-B"},{"comment":"The GitHub dataset defines an 'author' as the owner of a single-contributor repository and assumes that every file in the repository is the authentic stylistic product of that account. The paper does not control for AI-generated code, code copied from templates or tutorials, or vendored/third-party files. As a result, the large-scale experiments may be measuring attribution to a repository or account rather than to a human author. This limitation should be discussed explicitly, and ideally the dataset should be filtered to reduce such cases or the results should be reported separately for files with stronger evidence of human authorship.","section":"Sec. III-A"}],"minor_comments":[{"comment":"The parameter 'sample size' is used in the algorithm but is not defined in the pseudocode; the text later says the experiments use subsets of at most size 12. Please add a definition or a comment in the algorithm.","section":"Sec. IV-A, Algorithm 1"},{"comment":"Several confusion-matrix rows sum to fewer than 200 because the models returned indeterminate answers, but the indeterminate count is not shown in the table. Adding an explicit 'Indeterminate' column, or reporting the number of indeterminate responses in a footnote, would make the tables much easier to read.","section":"Tables II and IX"},{"comment":"The '2nd round' and '3rd round' columns are described as accuracy values; it would be clearer to state explicitly that these are cumulative probabilities that the true author has not been eliminated by that round, rather than per-round classification accuracies.","section":"Table V and Table X"},{"comment":"There is a typo in the sentence 'we carried our our tournament prompting experiments'; it should read 'we carried out our tournament prompting experiments.'","section":"Sec. VI-B"},{"comment":"For reproducibility, please record the exact API model versions and snapshot dates for GPT-4o and Gemini 1.5 Pro, since closed-source APIs are updated frequently and the reported results may not be reproducible with later versions.","section":"Sec. III-A and Sec. VI-A"}],"recommendation":"major_revision","confidential_remarks":"The same-repository confound in the large-scale tournament is the main blocker: the current wording of Sec. IV-B does not allow a reader to determine whether the 65% and 68.7% results are authorship attribution or repository matching. This is fixable by re-running with repository-disjoint reference sets, but the paper should not be accepted with the scaling claims in their current form. The GCJ leakage issue is acknowledged and is a secondary concern, but it should be either resolved empirically or moved from a threat to a stated limitation in the abstract-level claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real first — a systematic evaluation of off-the-shelf LLMs for source-code authorship attribution, and the small-scale verification results are probably believable. But the large-scale tournament numbers (65% C++, 68.7% Java) should not be taken at face value: the paper never says the one-shot reference for an author comes from a different repository than the query, and with single-contributor repos, a random reference is usually from the same repo. So the model may be matching repository-level style, not personal style across projects.\n\nWhat's genuinely good: it asks a sensible question, uses MCC as well as accuracy, runs temperature-0 inference, treats indeterminate answers as wrong, reports confusion matrices, and tries to avoid training-data leakage by crawling GitHub commits from May–October 2024. The tournament prompting idea is simple and does solve the token-limit problem. The GCJ-based RQ1/RQ2 results avoid the same-repo confound because query and reference come from different tasks, and an MCC of 0.78 for GPT-4o/Gemini-1.5-p on verification is a finding worth knowing even if the exact number shifts with seeds. The adversarial robustness work is a bonus, though modest.\n\nSoft spots, in order: (1) the tournament confound above is the load-bearing one; it is internal to the design and fixable by splitting repositories per author. (2) The headline numbers come from 300 queries with no error bars or seed-fixing; some reported differences, e.g., Gemini 66 vs GPT-4o 68.7 on Java, could be noise. (3) There is no same-protocol non-LLM baseline on the GitHub data, so we don't know how much value the LLM adds over an n-gram or code-structure model on the same split. (4) The GitHub 'author' label assumes one contributor per repo wrote every file; 2024 repos may contain AI-generated or templated code. (5) The paper flags GCJ training-data leakage but cannot verify the freshness argument for closed APIs; that's acknowledged honestly.\n\nWho is this for? People working on code stylometry, software forensics, and LLM evaluation. It deserves a serious referee because it is the first of its kind and the tournament idea is reusable, but the revision needs a disjoint-repo evaluation and proper uncertainty quantification. I would send it to review, not desk-reject, but with the expectation of heavy revision before acceptance.","headline":"Genuinely useful first study of LLMs for code authorship attribution, with a load-bearing confound in the tournament experiment that likely overstates cross-project attribution.","tokens_in":21286,"tokens_out":2044,"would_cite":true,"duration_ms":19973,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can verify and attribute code authorship with no training, reaching 65–69% top-1 accuracy with one reference per author.","keywords":["code authorship attribution","large language models","zero-shot prompting","few-shot in-context learning","tournament prompting","code stylometry","adversarial robustness","source code forensics"],"falsifier":"Check whether a meaningful share of the crawled single-contributor 2024 files are AI-generated, templated, or co-authored; if so, the reported accuracy must be re-read as attributing code provenance rather than human authorship.","tokens_in":20185,"feed_emoji":"🕵️","tokens_out":6902,"duration_ms":62977,"temperature":0.7,"pith_summary":"The paper sets out to show that general-purpose large language models can attribute source code to its author without any task-specific training or labeled datasets. Using zero-shot prompting, the best models verify whether two C++ snippets share an author with Matthews correlation up to 0.78, and few-shot in-context learning attributes a query to one of a small candidate set with MCC up to 0.77. To get past the models' input-token limits, the paper introduces tournament prompting, which splits a large author pool into small subsets, picks the most similar author in each, and iterates; this reaches top-1 accuracy of 65% among 500 C++ authors and 68.7% among 686 Java authors using one reference per author. The authors also report partial resistance to two state-of-the-art misattribution attacks and argue the approach transfers across languages. Such training-free attribution would matter for code forensics, plagiarism detection, and protecting patch integrity.","feed_headline":"No-training LLM attribution finds code authors up to 68.7%","feed_subtitle":"Zero-shot GPT-4o and Gemini reach MCC 0.78 and rank one author among 686 using one reference.","key_machinery":"The load-bearing mechanism is few-shot in-context learning plus a tournament bracket built on top of it. In each round the query code is shown alongside reference snippets for a small subset of candidate authors (at most 12), and the LLM is asked to return the most similar author's ID; the winners of all subsets form the next round's pool, and the process repeats until one author remains. This converts an attribution problem too large for any prompt window into a sequence of small comparisons, at roughly USD 1.5 per final attribution on the tested datasets. The paper also varies prompt complexity (simple, detailed, complex) around the three stylometric feature families—layout, lexical, and syntactic—and finds that more guidance does not reliably help, with the simple prompt often winning.","core_discovery":"On the paper's own terms, the discovery is that state-of-the-art LLMs carry a latent stylistic fingerprint for code: asked in plain English whether two snippets are by the same programmer, GPT-4o and Gemini 1.5 Pro reach 0.78 MCC, and when given one to three reference snippets per candidate author in the prompt, they pick the true author from sets of three to ten candidates with accuracy up to 88.5% (Gemini, two-shot). The proposed tournament procedure extends this to 500 C++ and 686 Java authors with one reference each, yielding 65% and 68.7% top-1 accuracy on freshly crawled 2024 GitHub code. Against evasion and imitation attacks (MCTS and RoPGen), the LLMs keep attack success well below the 77–95% reported for machine-learning baselines, and an adversarial-aware prompt raises GPT-4o's accuracy to 70% under MCTS. Not all models succeed: Llama2, Llama3-base, and Mistral perform near chance or refuse to answer, so the capability belongs to the strongest current LLMs rather than to the whole family. The paper is careful to flag that the public Google Code Jam dataset may have leaked into training data, which is why the large-scale numbers are measured on GitHub code dated May–October 2024.","pith_inferences":["Beyond the paper: if the single-contributor GitHub files include AI-generated code or shared templates, the reported accuracy is really code-provenance attribution, not human-style attribution, so the headline numbers should be re-interpreted as a lower bound on genuine authorship identification.","Beyond the paper: the tournament design implies a ranking among authors, not just a winner; the per-round winner lists could be recycled as a relevance ranking for code search or as a candidate filter for forensic tooling.","Beyond the paper: the same bracketing scheme should transfer to open-weights models fine-tuned on code, letting users run the pipeline offline and test whether the ability survives distillation or is unique to the large closed APIs.","Beyond the paper: a direct test of the leakage threat is to repeat the zero-shot verification on a purpose-built corpus of code written after the models' known training cutoffs, where one can compute how much of the MCC 0.78 survives."],"forward_implications":["Code authorship verification becomes possible without labeled training data: a zero-shot pairwise check reaches MCC 0.78, which supports triage in forensic settings where per-author corpora do not exist.","Attribution over hundreds of authors is feasible with one reference per author, reaching 65% top-1 accuracy for 500 C++ authors and 68.7% for 686 Java authors.","The same prompts transfer from C++ to Java with no retraining, so the approach is language-agnostic at least across these two mainstream languages.","Standard misattribution attacks (MCTS, RoPGen) are less effective against LLMs than against machine-learning baselines, and adversarial-aware prompting further cuts attack success.","Deployment is bounded by cost and model choice: only a few LLMs perform well, and a single tournament attribution costs about USD 1.5 on the tested APIs."],"supporting_citations":[{"why":"Supplies the precedent that LLMs can do authorship analysis on natural language, motivating the zero-shot and few-shot prompts used here.","marker":"[19]"},{"why":"Provides a language-oblivious recurrent-neural-network baseline for code authorship attribution that the paper compares against.","marker":"[8]"},{"why":"Provides the code-stylometry approach with AST features that serves as the classical machine-learning baseline to beat.","marker":"[9]"},{"why":"Supplies the MCTS adversarial transformations and the implementations of the baseline models used in the robustness experiments.","marker":"[21]"},{"why":"Supplies the RoPGen style-transformation attack and the adversarial-training baseline used in the robustness experiments.","marker":"[22]"},{"why":"Provides the Google Code Jam 2017 dataset used for the small-scale zero-shot, few-shot, and adversarial evaluations.","marker":"[39]"},{"why":"Defines the Matthews correlation coefficient used as the main evaluation metric in the study.","marker":"[40]"}],"fun_headline_variants":["LLM style fingerprint IDs coders: 0.78 MCC, 68.7% top-1","No-training LLM code attribution beats baselines, scales to 686","Zero-shot LLM links code snippets to authors at 0.78 MCC","One code sample per author: LLM finds coder among 686 at 68.7%","LLM names code authors in plain English, no labeled data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each file in a single-contributor GitHub repository was actually written by that account's owner and not by AI or template reuse, and that the 2024 collection date keeps those files out of the models' training data; if either fails, the accuracy numbers measure something other than identifying a human writer.","fun_headline_variants_meta":{"raw":{"variants":["LLM style fingerprint IDs coders: 0.78 MCC, 68.7% top-1","No-training LLM code attribution beats baselines, scales to 686","Zero-shot LLM links code snippets to authors at 0.78 MCC","One code sample per author: LLM finds coder among 686 at 68.7%","LLM names code authors in plain English, no labeled data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1883,"prompt_tokens":1112,"completion_tokens":771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":662}},"tokens_in":728,"tokens_out":771,"duration_ms":8105,"temperature":1.0,"reasoning_tokens":662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:56.524562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether a meaningful share of the crawled single-contributor 2024 files are AI-generated, templated, or co-authored; if so, the reported accuracy must be re-read as attributing code provenance rather than human authorship.","supporting_citations":[{"cited_title":"Large-scale and language-oblivious code authorship identification,","cited_arxiv_id":null,"evidence_quote":"Provides a language-oblivious recurrent-neural-network baseline for code authorship attribution that the paper compares against."},{"cited_title":"De-anonymizing programmers via code stylometry,","cited_arxiv_id":null,"evidence_quote":"Provides the code-stylometry approach with AST features that serves as the classical machine-learning baseline to beat."},{"cited_title":"Misleading authorship attribu- tion of source code using adversarial learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the MCTS adversarial transformations and the implementations of the baseline models used in the robustness experiments."},{"cited_title":"RoPGen: Towards robust code authorship attribution via automatic coding style transformation,","cited_arxiv_id":null,"evidence_quote":"Supplies the RoPGen style-transformation attack and the adversarial-training baseline used in the robustness experiments."},{"cited_title":"Google code jam,","cited_arxiv_id":null,"evidence_quote":"Provides the Google Code Jam 2017 dataset used for the small-scale zero-shot, few-shot, and adversarial evaluations."},{"cited_title":"The matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evalua- tion,","cited_arxiv_id":null,"evidence_quote":"Defines the Matthews correlation coefficient used as the main evaluation metric in the study."}],"review_version":1}