Pith. sign in

REVIEW 3 major objections 6 minor 75 references

Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM-based assistants Bing CoPilot and Gemini frequently provide links that do not lead to the code they generated, a problem the authors call 'provenance debt'.

desk verdict Useful empirical study of LLM link relevance, but the abstract's headline percentages are unreconciled with the tables—fix that before citing. read the letter →

arxiv 2501.12134 v1 pith:ILXWFAUJ submitted 2025-01-21 cs.SE

Daniele Bifolco , Pietro Cassieri , Giuseppe Scanniello , Massimiliano Di Penta , Fiorella Zampetti This is my paper · ORCID
classification cs.SE
keywords codeprovenancelargelanguagemodelsLLM-basedassistantsclonedetectionBingCoPilotGoogleGeminilicensingempiricalsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether LLM-based assistants can tell a developer where an automatically generated code snippet actually came from. Using 243 snippets from Bing CoPilot and 194 from Google Gemini across six languages, it checks whether the links each assistant provides contain code similar enough to count as the likely origin. The answer, according to the study, is mixed: 66% of Bing CoPilot's links are relevant, but only 28% of Gemini's are, and many links are noisy, unreachable, or point to code in another language. If correct, this means developers cannot rely on the assistants' own citations to determine trust, attribution, or licensing terms for generated code.

What carries the argument

The argument is carried by a two-layer similarity analysis. First, the authors scrape code snippets from the landing pages of 1,520 links (1,006 from Bing CoPilot, 514 from Gemini) and run CCFinderSW, a token-based clone detector, with a threshold of at least 20 tokens, computing a cloning ratio between each linked snippet and the LLM-generated snippet. Second, they complement this with a cosine textual similarity over code tokens and, crucially, a manual annotation by five authors who judge link type, relevance to the query, and whether the linked snippet likely originated the generated code. The manual judgments are the ground truth; the automated metrics are evaluated against them.

What would settle it

Collect cases where the true origin of a generated snippet is independently known, for example, code that was published to a public repository with a timestamp clearly before the assistant's training cutoff; then prompt the assistant for that exact task and check whether its provided links include that known source and whether the clone ratio flags it. If, in such known-origin cases, the assistants' 'relevant' links are no more likely than chance to contain the true source, the provenance-debt conclusion would be weakened or overturned.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based assistants suffer from serious 'provenance debt': the external links they provide for a coding query only sometimes lead to code that resembles the snippet the assistant generated. Bing CoPilot gave at least one relevant link for the majority of the queries studied, with the share of queries having at least one relevant link ranging from 47.8% (Java) to 84% (PHP); Gemini rarely gave any relevant link, with shares mostly below 15% and 0% for Ruby. The authors also find that the assistants' link lists are heterogeneous and noisy, that they do not replicate what a straightforward search-engine query would return, and that even when a link is relevant there may be several candidate sources, making the actual origin ambiguous.

Load-bearing premise

The whole analysis treats high similarity between a generated snippet and a snippet reachable through a provided link as evidence that the link is the source, even though the paper cannot prove the direction of reuse and found 28 Bing sources and 1 Gemini source that were updated after the assistants' checkpoints.

Editorial extensions

If this is right

  • If the paper is right, developers cannot use assistant-provided links as reliable evidence of where generated code came from, so license compliance and trust decisions remain largely unsupported.
  • Users must expect to inspect several links per query, since many are irrelevant, unreachable, or written for a different programming language, and sometimes more than one plausible source exists.
  • Clone detection and textual similarity, as calibrated and used here, can partially distinguish relevant from irrelevant provenance links, suggesting these tools could be embedded in developer-facing assistants.
  • Because the assistants' link choices do not match what their underlying search engines return for the same query, provenance cannot be reproduced or audited by simply re-running a web search.
  • Gemini's near-absence of relevant links indicates that its attribution support, at least at the time of the study, is far weaker than Bing CoPilot's for code generation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 66% and 28% relevance figures likely understate or overstate the true provenance rate depending on direction of reuse: the paper itself notes 28 Bing sources and 1 Gemini source were updated after the models' checkpoints, so some 'relevant' links may have copied from the assistants rather than the reverse.
  • A practical extension would build a provenance-ranking tool that automatically takes an LLM's generated snippet, scrapes the provided links, runs clone and textual similarity, and ranks candidate sources; the paper's methodology is essentially a blueprint for that tool.
  • If provenance debt is a general property of post-generation attribution, then newer or differently architected assistants that rely on the same strategy would likely show similar patterns unless they are trained specifically to emit source links.
  • A testable next step would apply the same protocol to assistants that expose training-cutoff dates or to retrieval-augmented systems, to see whether explicit retrieval improves the relevance of provenance links.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports an empirical study of whether two LLM-based assistants, Bing CoPilot and Google Gemini, provide external links that contain code snippets similar to the code they generate. The authors use 99 CodeSearchNet tasks across six languages, issue prompts to the assistants, and analyze the returned links through manual annotation (with multiple annotators and reported Cohen's kappa) and automated clone detection/textual similarity. The main finding is that Bing CoPilot provides at least one snippet-relevant link for a majority of queries (about 48%–84% depending on language), while Gemini does so in only 0%–14% of queries. The paper concludes that LLM-provided links are noisy and suffer from 'provenance debt'.

Significance. If the results are reported accurately, this is a useful and timely empirical contribution: it is the first study, to the authors' knowledge, to evaluate the usefulness of LLM-provided links for code provenance. The study's strengths include a public replication package (Dataset DOI provided), a reasonably large manually annotated set of 1,520 links, the use of two independent annotators with reported inter-rater agreement (kappa values of 0.77–0.91), and the combination of manual assessment with automated clone detection and textual similarity analysis. The qualitative conclusion—that assistant-provided links often do not allow developers to verify the origin or licensing status of generated code—is practically relevant. However, the abstract's headline numbers (66% and 28%) are not consistently supported by the tables, and the aggregation method is not defined, which undermines the verifiability of the quantitative claim.

major comments (3)
  1. [Abstract; Tables IV and VI] The abstract's statement that '66% of the links from Bing CoPilot and 28% from Google Gemini are relevant' cannot be verified from the reported data. Table VI reports the percentage of queries (per language) with at least one snippet-relevant link, not the percentage of links. Weighting those percentages by the number of queries per language in Table I yields about 66% for Bing, so 66% appears to be a query-level aggregate mislabeled as a link-level figure. For Gemini, the same weighted calculation yields about 6%, and no table or combination of tables in the paper yields 28%. The abstract's 28% also contradicts the RQ2 summary in Section III.B, which states that Gemini's relevant-link percentages are below 15%. Please correct the abstract, state explicitly whether the figures are link-level or query-level, and ensure every reported aggregate is derivable from the tables or the replication package.
  2. [Section II.E] The analysis methodology does not describe how the aggregate percentages in the abstract are obtained. Section II.E explains the per-language analysis for RQ2 but does not state whether overall figures are weighted by number of queries, weighted by number of links, or simple averages of the per-language percentages. Without this description, the reader cannot reproduce the headline numbers. Please provide the formula (or the exact computation script) used to aggregate the per-language results, and consider adding the overall row to Table VI.
  3. [Section V] The threat that the direction of reuse may be reversed (i.e., some web sources updated after the LLM checkpoints may have copied the model's output) is acknowledged, but its impact on the central finding is not assessed. The paper reports that 28 Bing sources and 1 Gemini source were updated between the LLM checkpoints and the analysis, yet it does not say how many of the 'yes' relevant links fall into this set. Please quantify the overlap between the 'yes' links and the sources updated after the checkpoints, or otherwise bound the effect on the reported percentages. This would let the reader judge whether the 'relevant' figures are upper bounds and by how much.
minor comments (6)
  1. [Section III.A] The sentence 'For Bing CoPilot, we obtained higher percentages of relevant links (41.03% for Ruby up to 61.79% for Java vs 24.26% for Python up to 56% for Go)' is ambiguous because it is not clear which percentages refer to Bing CoPilot and which to Gemini; please label the two assistants explicitly.
  2. [Section III.B] The RQ2 summary sentence ends with a comma and no period: 'Bing CoPilot is able to provide at least one “relevant” link for the majority of the considered queries,'. Please correct the punctuation.
  3. [Table VI] The column headers 'One' and '> One' should be expanded to 'Exactly one' and 'More than one' to avoid ambiguity about the meaning of the columns.
  4. [Section II.C] The paper reports the number of links classified as 'Unreachable' and 'Not a Source' together as NA in Table IV; reporting these two categories separately would help interpret the often-high NA percentages and let readers judge how many links were simply unreachable at analysis time.
  5. [Section VII] The conclusion lists '(i) a general noise of the produced links, and (iii) a limited ability...' but no (ii); please renumber the items or insert the missing point.
  6. [Section III.B] The phrase 'computed across snippets from links classified as “not relevant” and “relevant” are statistically significant' is awkward; consider rephrasing to 'computed for links classified as relevant versus not relevant'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical measurement with no fitted parameter or self-citation chain doing load-bearing work.

full rationale

The paper reports an empirical study: it prompts two LLM-based assistants, collects their generated snippets and provided links, and assesses relevance through manual annotation supported by clone detection and textual similarity. There is no derivation chain in which an output is defined in terms of an input, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' own prior work. The central claims, such as 'Bing CoPilot is able to provide at least one relevant link for the majority of the considered queries' (RQ2 summary, Section III-B), are grounded in the annotators' yes/no judgments quantified in Table VI and in the complementary automated similarity analysis (Section II-D). The few self-references in the reference list, e.g., [19] and [63], are used as related-work context or as prior support for standard clone-detection methodology, not to justify the study's conclusions. The acknowledged limitation in Section V that 'we cannot prove the direction of the code reuse' is a validity threat about interpretation, not a circular step, and the paper explicitly reports it rather than hiding it. A skeptical concern that the abstract's aggregate percentages (66% and 28%) are not directly derivable from Tables IV or VI is a reporting or aggregation gap, not circularity: the underlying measurements are independent of the paper's own definitions. Therefore the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study relies on three domain assumptions: the benchmark tasks are representative, the clone threshold catches meaningful clones, and human annotation can judge likely provenance. None are fitted parameters or invented entities.

assumptions (3)
  • domain assumption CodeSearchNet queries represent realistic coding tasks
    Used as the basis for all queries; if unrepresentative, the results may not generalize.
  • domain assumption CCFinderSW token threshold of 20 tokens is appropriate
    The authors chose a lower bound from prior work; this affects which clones are detected in the automated analysis.
  • domain assumption Manual annotation by five authors is a valid ground truth for provenance
    The central 'relevant' label comes from human judgment, with Cohen's kappa reported but not perfect agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot." pith.science (2026). https://pith.science/paper/ILXWFAUJ

@misc{pith2026250112134,
  author       = {Pith},
  title        = {Pith review of: Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILXWFAUJ}},
  note         = {Machine review of arXiv:2501.12134}
}
read the original abstract

Large Language Models (LLMs) are currently used for various software development tasks, including generating code snippets to solve specific problems. Unlike reuse from the Web, LLMs are limited in providing provenance information about the generated code, which may have important trustworthiness and legal consequences. While LLM-based assistants may provide external links that are "related" to the generated code, we do not know how relevant such links are. This paper presents the findings of an empirical study assessing the extent to which 243 and 194 code snippets, across six programming languages, generated by Bing CoPilot and Google Gemini, likely originate from the links provided by these two LLM-based assistants. The study leverages automated code similarity assessments with thorough manual analysis. The study's findings indicate that the LLM-based assistants provide a mix of relevant and irrelevant links having a different nature. Specifically, although 66% of the links from Bing CoPilot and 28% from Google Gemini are relevant, LLMs-based assistants still suffer from serious "provenance debt".

Figures

Figures reproduced from arXiv: 2501.12134 by the authors.

Figure 1
Figure 1. Dataflow of the Study Design • RQ1: To what extent are LLM-based assistants able to provide links to external sources for code generation tasks? • RQ2: To what extent are the automatically generated code snippets “similar” to code snippets extracted from the LLM-provided links? First, we look at the number and types of links provided by the LLM-based assistants, along with the generated snippets. Then, we investigat… view at source ↗
Figure 2
Figure 2. Bing CoPilot: Cloning ratio and textual similarity [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Gemini: Cloning ratio and textual similarity [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 40 canonical work pages

  1. [1]

    Query: All permutations of a list,

    Baeldung, “Query: All permutations of a list,” Last Accessed: Nov 2024. [Online]. Available: https://baeldung.com/java-array-permutations

    work page 2024

  2. [2]

    Comparison and evaluation of clone detection tools,

    S. Bellon, R. Koschke, G. Antoniol, J. Krinke, and E. Merlo, “Comparison and evaluation of clone detection tools,” IEEE Trans. Software Eng., vol. 33, no. 9, pp. 577–591, 2007. [Online]. Available: https://doi.org/10.1109/TSE.2007.70725

    arXiv 2007

  3. [3]

    Replication package of the paper

    D. Bifolco, P. Cassieri, G. Scanniello, M. Di Penta, and F. Zampetti, “Replication package of the paper ”Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot”,” Aug. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.13151631

    work page doi:10.5281/zenodo.13151631 2024

  4. [4]

    Raters’ reliability in clone benchmarks construction,

    A. Charpentier, J. Falleri, F. Morandat, E. Ben Hadj Yahia, and L. R ´eveill`ere, “Raters’ reliability in clone benchmarks construction,” Empir. Softw. Eng. , vol. 22, no. 1, pp. 235–258, 2017. [Online]. Available: https://doi.org/10.1007/s10664-015-9419-z

    work page doi:10.1007/s10664-015-9419-z 2017

  5. [5]

    How to protect copyright data in optimization of large language models?

    T. Chu, Z. Song, and C. Yang, “How to protect copyright data in optimization of large language models?” in Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024 . AAAI Press, 2024, pp. 17 871–17 879. [Online]. Available: https://doi.org/10.1609/aaai. v38i16.29741

    doi:10.1609/aaai 2024

  6. [6]

    Answering ordinal questions with ordinal data using ordinal statistics,

    N. Cliff, “Answering ordinal questions with ordinal data using ordinal statistics,” Multivariate behavioral research, vol. 31, no. 3, pp. 331–350, 1996

    work page 1996

  7. [7]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement , vol. 20, no. 1, pp. 37–46, 1960

    1960

  8. [8]

    Applescript,

    W. R. Cook, “Applescript,” in Proceedings of the Third ACM SIGPLAN History of Programming Languages Conference (HOPL-III), San Diego, California, USA, 9-10 June 2007 . ACM, 2007, pp. 1–21. [Online]. Available: https://doi.org/10.1145/1238844.1238845

    arXiv 2007

Show all 75 references
  1. [9]

    Software bertillonage - determining the provenance of software development artifacts,

    J. Davies, D. M. Germ ´an, M. W. Godfrey, and A. Hindle, “Software bertillonage - determining the provenance of software development artifacts,” Empir. Softw. Eng. , vol. 18, no. 6, pp. 1195–1237, 2013. [Online]. Available: https://doi.org/10.1007/s10664-012-9199-7

    2013 doi

  2. [10]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE 2023, Melbour...

    2023

  3. [11]

    Method and system for matching unknown software component to known software component. patent no. wo2013177025a1,

    I. Fedorenko, V . Tatavu, J. Whitehouse, and B. E. Fox, “Method and system for matching unknown software component to known software component. patent no. wo2013177025a1,” 2013

    2013

  4. [12]

    Internet-scale code search,

    R. E. Gallardo-Valencia and S. E. Sim, “Internet-scale code search,” in 2009 ICSE workshop on search-driven development-users, infrastructure, tools and evaluation . IEEE, 2009, pp. 49–52

    2009

  5. [13]

    A taxonomy for human-llm interaction modes: An initial exploration,

    J. Gao, S. A. Gebreegziabher, K. T. W. Choo, T. J.-J. Li, S. T. Perrault, and T. W. Malone, “A taxonomy for human-llm interaction modes: An initial exploration,” in Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’24. New York,...

    2024

  6. [14]

    Query: Convert JSON to CSV,

    GeekFlare, “Query: Convert JSON to CSV,” Last Accessed: Nov 2024. [Online]. Available: https://geekflare.com/convert-json-to-csv-in-python

    2024

  7. [15]

    Query: Confusion matrix,

    GeeksForGeeks, “Query: Confusion matrix,” Last Accessed: Nov 2024. [Online]. Available: https://www.geeksforgeeks.org/ confusion-matrix-machine-learning

    2024

  8. [16]

    Query: Convert JSON to CSV,

    ——, “Query: Convert JSON to CSV,” Last Accessed: Nov 2024. [Online]. Available: https://www.geeksforgeeks.org/ convert-json-to-csv-in-python

    2024

  9. [17]

    Query: Extract data from HTML content,

    ——, “Query: Extract data from HTML content,” Last Accessed: Nov 2024. [Online]. Available: https://www.geeksforgeeks.org/ java-program-to-extract-content-from-a-html-document

    2024

  10. [18]

    Query: Output to HTML file,

    ——, “Query: Output to HTML file,” Last Accessed: Nov 2024. [Online]. Available: https://www.geeksforgeeks.org/ how-to-write-the-output-to-html-file-with-python-beautifulsoup

    2024

  11. [19]

    Code siblings: Technical and legal implications of copying code between applications,

    D. M. Germ ´an, M. Di Penta, Y . Gu ´eh´eneuc, and G. Antoniol, “Code siblings: Technical and legal implications of copying code between applications,” in Proceedings of the 6th International Working Conference on Mining Software Repositories, MSR 2009 (Co-located with ICSE), ...

    2009

  12. [20]

    GitHub Copilot,

    GitHub, “GitHub Copilot,” Last Accessed: Nov 2024. [Online]. Available: https://copilot.github.com

    2024

  13. [21]

    Query: Encrypt aes ctr mode,

    GitHub: abigbug/cowpro, “Query: Encrypt aes ctr mode,” Last Accessed: Nov 2024. [Online]. Avail- able: https://github.com/abigbug/cowpro/blob/06ac4c3/sites/all/modules/ cowpro/modules/jiupaipay/includes/libs/encrypt.php#L46

    2024

  14. [22]

    The fossology project,

    R. Gobeille, “The fossology project,” in Proceedings of the 2008 International Working Conference on Mining Software Repositories, MSR 2008 (Co-located with ICSE), Leipzig, Germany, May 10-11, 2008, Proceedings . ACM, 2008, pp. 47–50. [Online]. Available: https://doi.org/10.11...

    2008

  15. [23]

    Google search engine

    Google Inc., “Google search engine.” [Online]. Available: https: //www.google.com

  16. [24]

    Deep code search,

    X. Gu, H. Zhang, and S. Kim, “Deep code search,” in Proceedings of the 40th International Conference on Software Engineering , ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 933–944. [Online]. Available: https://doi.org/10.1145/3180155.3180167

    2018

  17. [25]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

    1979

  18. [26]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. C. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” CoRR, vol. abs/2308.10620,

    arXiv

  19. [27]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” CoRR, vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436

    1909 arXiv

  20. [28]

    Etude de la distribution florale dans une portion des alpes et du jura,

    P. Jaccard, “Etude de la distribution florale dans une portion des alpes et du jura,” Bulletin de la Societe Vaudoise des Sciences Naturelles , vol. 37, no. 142, pp. 547–579, 1901

    1901

  21. [29]

    Query: Regex case insensitive,

    JavaCodeExamples, “Query: Regex case insensitive,” Last Accessed: Nov 2024. [Online]. Available: https://www.javacodeexamples.com/ java-regex-case-insensitive-example/1498

    2024

  22. [30]

    CCFinder: A multilinguistic token-based code clone detection system for large scale source code,

    T. Kamiya, S. Kusumoto, and K. Inoue, “CCFinder: A multilinguistic token-based code clone detection system for large scale source code,” IEEE Trans. Software Eng. , vol. 28, no. 7, pp. 654–670, 2002. [Online]. Available: https://doi.org/10.1109/TSE.2002.1019480

    2002 arXiv

  23. [31]

    Copyright violations and large language models,

    A. Karamolegkou, J. Li, L. Zhou, and A. Søgaard, “Copyright violations and large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 . Association for Computational Linguistics,...

    2023 doi

  24. [32]

    An exploratory investigation into code license infringements in large language model training datasets,

    J. Katzy, R. M. Popescu, A. van Deursen, and M. Izadi, “An exploratory investigation into code license infringements in large language model training datasets,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering,...

    2024

  25. [33]

    An empirical study of code clone genealogies,

    M. Kim, V . Sazawal, D. Notkin, and G. C. Murphy, “An empirical study of code clone genealogies,” in Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon...

    2005

  26. [34]

    Query: Convert JSON to CSV,

    LearnPython, “Query: Convert JSON to CSV,” Last Accessed: Nov 2024. [Online]. Available: https://learnpython.com/blog/python-json-to-csv/

    2024

  27. [35]

    Resolving license dependencies for aggregations of legally-protectable content, patent no. us7552,093b,

    D. A. LevinPalle, M. Pedersen, and A. C. Shah, “Resolving license dependencies for aggregations of legally-protectable content, patent no. us7552,093b,” 2009

    2009

  28. [36]

    A survey of large language models attribution,

    D. Li, Z. Sun, X. Hu, Z. Liu, Z. Chen, B. Hu, A. Wu, and M. Zhang, “A survey of large language models attribution,” 2023. [Online]. Available: https://arxiv.org/abs/2311.03731

    2023 arXiv

  29. [37]

    Opportunities and challenges in code search tools,

    C. Liu, X. Xia, D. Lo, C. Gao, X. Yang, and J. Grundy, “Opportunities and challenges in code search tools,” ACM Comput. Surv., vol. 54, no. 9, oct 2021. [Online]. Available: https://doi.org/10.1145/3480027

    2021 doi

  30. [38]

    Evaluating verifiability in generative search engines,

    N. F. Liu, T. Zhang, and P. Liang, “Evaluating verifiability in generative search engines,” in Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 7001–7025

    2023

  31. [39]

    The Llama 3 Herd of Models,

    Llama Team, “The Llama 3 Herd of Models,” Last Accessed: Nov

  32. [40]

    The data provenance initiative: A large scale audit of dataset licensing & attribution in ai,

    S. Longpre, R. Mahari, A. Chen, N. Obeng-Marnu, D. Sileo, W. Brannon, N. Muennighoff, N. Khazam, J. Kabbara, K. Perisetla, X. Wu, E. Shippole, K. Bollacker, T. Wu, L. Villa, S. Pentland, and S. Hooker, “The data provenance initiative: A large scale audit of dataset licensing &...

    2023 arXiv

  33. [41]

    Crafting Knowledge: Exploring the Creative Mechanisms of Chat-Based Search Engines,

    L. Ma, X. Xu, and Y . Tan, “Crafting Knowledge: Exploring the Creative Mechanisms of Chat-Based Search Engines,” arXiv preprint arXiv:2402.19421, 2024

    2024 arXiv

  34. [42]

    Trained without my consent: Detecting code inclusion in language models trained on code,

    V . Majdinasab, A. Nikanjam, and F. Khomh, “Trained without my consent: Detecting code inclusion in language models trained on code,” CoRR, vol. abs/2402.09299, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.09299

    2024 doi arXiv

  35. [43]

    Experiment on the automatic detection of function clones in a software system using metrics,

    J. Mayrand, C. Leblanc, and E. Merlo, “Experiment on the automatic detection of function clones in a software system using metrics,” in 1996 International Conference on Software Maintenance (ICSM ’96), 4-8 November 1996, Monterey, CA, USA, Proceedings. IEEE Computer Society, 1...

    1996

  36. [44]

    Portfolio: Searching for relevant functions and their usages in millions of lines of code,

    C. McMillan, D. Poshyvanyk, M. Grechanik, Q. Xie, and C. Fu, “Portfolio: Searching for relevant functions and their usages in millions of lines of code,” ACM Trans. Softw. Eng. Methodol. , vol. 22, no. 4, oct

  37. [45]

    Bing Copilot,

    Microsoft, “Bing Copilot,” Last Accessed: Nov 2024. [Online]. Available: https://copilot.microsoft.com

    2024

  38. [46]

    Microsoft Bing

    Microsoft Inc., “Microsoft Bing.” [Online]. Available: https://www.bing. com

  39. [47]

    ChatGPT,

    OpenAI, “ChatGPT,” Last Accessed: Nov 2024. [Online]. Available: https://openai.com/blog/chatgpt

    2024

  40. [48]

    Leveraging statistical machine translation for code search,

    H. Phan and A. Jannesari, “Leveraging statistical machine translation for code search,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , 2024, pp. 191–200

    2024

  41. [49]

    Semantics-based code search,

    S. P. Reiss, “Semantics-based code search,” in 31st International Conference on Software Engineering, ICSE 2009, May 16-24, 2009, Vancouver, Canada, Proceedings. IEEE, 2009, pp. 243–253. [Online]. Available: https://doi.org/10.1109/ICSE.2009.5070525

    2009

  42. [50]

    Software provenance tracking at the scale of public source code,

    G. Rousseau, R. Di Cosmo, and S. Zacchiroli, “Software provenance tracking at the scale of public source code,” Empir. Softw. Eng., vol. 25, no. 4, pp. 2930–2959, 2020. [Online]. Available: https://doi.org/10.1007/s10664-020-09828-5

    2020 doi

  43. [51]

    A survey on software clone detection research,

    C. K. Roy and J. R. Cordy, “A survey on software clone detection research,” Technical Report No. 2007-541 School of Computing Queen’s University at Kingston Ontario, Canada, Tech. Rep., 2007. [Online]. Available: https://research.cs.queensu.ca/TechReports/Reports/2007-541. pdf

    2007

  44. [52]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scial...

    2023 arXiv

  45. [53]

    Sourcerercc: scaling code clone detection to big-code,

    H. Sajnani, V . Saini, J. Svajlenko, C. K. Roy, and C. V . Lopes, “Sourcerercc: scaling code clone detection to big-code,” in Proceedings of the 38th International Conference on Software Engineering, ser. ICSE ’16. New York, NY , USA: Association for Computing Machinery, 2016,...

    2016

  46. [54]

    Query: Confusion Matrix,

    Scikit-Learn, “Query: Confusion Matrix,” Last Accessed: Nov 2024. [Online]. Available: https://scikit-learn.org/stable/modules/generated/ sklearn.metrics.confusion matrix.html

    2024

  47. [55]

    CCFinderSW: clone detection tool with flexible multilingual tokenization,

    Y . Semura, N. Yoshida, E. Choi, and K. Inoue, “CCFinderSW: clone detection tool with flexible multilingual tokenization,” in 24th Asia-Pacific Software Engineering Conference, APSEC 2017, Nanjing, China, December 4-8, 2017 . IEEE Computer Society, 2017, pp. 654–659. [Online]....

    2017 doi

  48. [56]

    Improving code search with co-attentive representation learning,

    J. Shuai, L. Xu, C. Liu, M. Yan, X. Xia, and Y . Lei, “Improving code search with co-attentive representation learning,” in Proceedings of the 28th International Conference on Program Comprehension, ser. ICPC ’20. New York, NY , USA: Association for Computing Machinery, 2020, ...

    2020

  49. [57]

    What is this site’s policy on content generated by generative artificial intelligence tools?

    Stack Overflow, “What is this site’s policy on content generated by generative artificial intelligence tools?” Last Accessed: Nov 2024. [Online]. Available: https://stackoverflow.com/help/gen-ai-policy

    2024

  50. [58]

    Solving the search for source code,

    K. T. Stolee, S. Elbaum, and D. Dobos, “Solving the search for source code,” ACM Trans. Softw. Eng. Methodol. , vol. 23, no. 3, jun 2014. [Online]. Available: https://doi.org/10.1145/2581377

    2014 doi

  51. [59]

    A survey of source code search: A 3-dimensional perspective,

    W. Sun, C. Fang, Y . Ge, Y . Hu, Y . Chen, Q. Zhang, X. Ge, Y . Liu, and Z. Chen, “A survey of source code search: A 3-dimensional perspective,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 6, jun 2024. [Online]. Available: https://doi.org/10.1145/3656341

    2024 doi

  52. [60]

    Using the uniqueness of global identifiers to determine the provenance of python software source code,

    Y . Sun, D. M. Germ ´an, and S. Zacchiroli, “Using the uniqueness of global identifiers to determine the provenance of python software source code,” Empir. Softw. Eng., vol. 28, no. 5, p. 107, 2023. [Online]. Available: https://doi.org/10.1007/s10664-023-10317-8

    2023 doi

  53. [61]

    Coprotector: Protect open-source code against unauthorized training usage with data poisoning,

    Z. Sun, X. Du, F. Song, M. Ni, and L. Li, “Coprotector: Protect open-source code against unauthorized training usage with data poisoning,” in WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022 . ACM, 2022, pp. 652–660. [Online]. Available: h...

    2022

  54. [62]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  55. [63]

    Unveiling chatgpt’s usage in open source projects: A mining-based study,

    R. Tufano, A. Mastropaolo, F. Pepe, O. Dabic, M. D. Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,” in 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024 . ACM, ...

    2024

  56. [64]

    Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, ...

    2022

  57. [65]

    To distribute or not to distribute?: why licensing bugs matter,

    C. Vendome, D. M. Germ ´an, M. Di Penta, G. Bavota, M. L. V´asquez, and D. Poshyvanyk, “To distribute or not to distribute?: why licensing bugs matter,” in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03...

    2018

  58. [66]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” arXiv preprint arXiv:2305.07922 , 2023

    2023 arXiv

  59. [67]

    Alphonse Bertillon,

    Wikipedia, “Alphonse Bertillon,” Last Accessed: Nov 2024. [Online]. Available: https://en.wikipedia.org/wiki/Alphonse Bertillon

    2024

  60. [68]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” in Break- throughs in statistics: Methodology and distribution . Springer, 1992, pp. 196–202

    1992

  61. [69]

    Open source license inconsistencies on GitHub,

    T. Wolter, A. Barcomb, D. Riehle, and N. Harutyunyan, “Open source license inconsistencies on GitHub,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 5, jul 2023. [Online]. Available: https://doi.org/10.1145/3571852

    2023 doi

  62. [70]

    A systematic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” in MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022 . ACM, 2022, pp. 1–10. [Online]. Available: htt...

    2022

  63. [71]

    Privacy and copyright protection in generative AI: A lifecycle perspective,

    D. Zhang, B. Xia, Y . Liu, X. Xu, T. Hoang, Z. Xing, M. Staples, Q. Lu, and L. Zhu, “Privacy and copyright protection in generative AI: A lifecycle perspective,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN 2...

    2024

  64. [72]

    Expanding queries for code search using semantically related API class-names,

    F. Zhang, H. Niu, I. Keivanloo, and Y . Zou, “Expanding queries for code search using semantically related API class-names,” IEEE Trans. Software Eng., vol. 44, no. 11, pp. 1070–1082, 2018. [Online]. Available: https://doi.org/10.1109/TSE.2017.2750682

    2018

  65. [2013]

    Available: https://doi.org/10.1145/2522920.2522930

    [Online]. Available: https://doi.org/10.1145/2522920.2522930

  66. [2023]

    Available: https://doi.org/10.48550/arXiv.2308.10620

    [Online]. Available: https://doi.org/10.48550/arXiv.2308.10620

    doi arXiv

  67. [2024]

    Available: https://ai.meta.com/research/publications/ the-llama-3-herd-of-models/

    [Online]. Available: https://ai.meta.com/research/publications/ the-llama-3-herd-of-models/

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.