Pith. sign in

REVIEW 5 cited by

Studying LLM Performance on Closed- and Open-source Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.15100 v1 pith:UF3LDK2V submitted 2024-02-23 cs.SE cs.LG

Toufique Ahmed , Christian Bird , Premkumar Devanbu , Saikat Chakraborty This is my paper
classification cs.SEcs.LG
keywords codeperformanceproprietarydevelopmentfindllmsmodelssoftware
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language models (LLMs) are finding wide use in software engineering practice. These models are extremely data-hungry, and are largely trained on open-source (OSS) code distributed with permissive licenses. In terms of actual use however, a great deal of software development still occurs in the for-profit/proprietary sphere, where the code under development is not, and never has been, in the public domain; thus, many developers, do their work, and use LLMs, in settings where the models may not be as familiar with the code under development. In such settings, do LLMs work as well as they do for OSS code? If not, what are the differences? When performance differs, what are the possible causes, and are there work-arounds? In this paper, we examine this issue using proprietary, closed-source software data from Microsoft, where most proprietary code is in C# and C++. We find that performance for C# changes little from OSS --> proprietary code, but does significantly reduce for C++; we find that this difference is attributable to differences in identifiers. We also find that some performance degradation, in some cases, can be ameliorated efficiently by in-context learning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Empirical Study of Code Large Language Models for Binary Security Patch Detection

    cs.SE 2025-09 conditional novelty 6.0 of 10

    Fine-tuned code LLMs detect binary security patches accurately on decompiler pseudo-code, and adding source-code patches to the training data improves accuracy and F1 further.

  2. RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry

    cs.SE 2025-05 conditional novelty 6.0 of 10

    On a 160,000-file industrial C++ codebase, BM25-based retrieval-augmented generation outperformed fine-tuning for line-level code completion, and combining RAG with fine-tuning further improved accuracy.

  3. A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat

    cs.SE 2025-07 conditional novelty 5.0 of 10

    On WeChat's closed-source codebase, similarity-based RAG with combined BM25 and GTE-Qwen retrieval improves open-source LLM code completion more than identifier-based retrieval, with gains growing for larger models.

  4. Can LLMs Generate User Stories and Assess Their Quality?

    cs.SE 2025-07 conditional novelty 4.0 of 10

    LLMs generate user stories with human-like coverage and style but lower diversity, and they assess semantic quality well when given explicit codebook criteria, though human oversight is still needed.

  5. Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation

    cs.AI 2025-08 reject novelty 2.0 of 10

    This survey claims to be the first systematic review of LLMs for organic synthesis, but its central 'evaluation' is never actually performed.

Pith tools