REVIEW 7 cited by
When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?
read the original abstract
Retrieval-augmented generation (RAG) has increasingly shown its power in extending large language models' (LLMs') capability beyond their pre-trained knowledge. Existing works have shown that RAG can help with software development tasks such as code generation, code update, and test generation. Yet, the effectiveness of adapting LLMs to fast-evolving or less common API libraries using RAG remains unknown. To bridge this gap, we take an initial step to study this unexplored yet practical setting - when developers code with a less common library, they often refer to its API documentation; likewise, when LLMs are allowed to look up API documentation via RAG, to what extent can LLMs be advanced? To mimic such a setting, we select four less common open-source Python libraries with a total of 1017 eligible APIs. We study the factors that affect the effectiveness of using the documentation of less common API libraries as additional knowledge for retrieval and generation. Our intensive study yields interesting findings: (1) RAG helps improve LLMs' performance by 83%-220%. (2) Example code contributes the most to advance LLMs, instead of the descriptive texts and parameter lists in the API documentation. (3) LLMs could sometimes tolerate mild noises (typos in description or incorrect parameters) by referencing their pre-trained knowledge or document context. Finally, we suggest that developers pay more attention to the quality and diversity of the code examples in the API documentation. The study sheds light on future low-code software development workflows.
Forward citations
Cited by 7 Pith papers
-
BIM Information Extraction Through LLM-based Adaptive Exploration
LLM adaptive exploration via runtime code execution outperforms static query generation for information extraction from heterogeneous BIM models on the new ifc-bench v2 benchmark.
-
Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports
Agent-reactive bugs—failures caused by a specific LLM response colliding with harness code—fall into five symptom types and eight trigger behaviors, and most commonly surface as silent errors.
-
WebDesignIter: Co-Evolving Design Knowledge for Repository-Level Front-End Code Generation
A co-evolving design-and-code knowledge graph improves multi-iteration front-end repository generation by ~9.5 Pass@2 points and cuts input tokens 25–30× versus agent baselines.
-
Detecting Architectural Drift in Safety-Critical Firmware through Runtime Trace Analysis
Hardware traces abstracted to message exchanges, deterministically differenced against design sequence diagrams, detect architectural drift (confirmed/missing/additional/inverted) in ISO 26262 firmware, with constrain...
-
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation
Cross-lingual RACG shows non-trivial but unequal knowledge transfer across 13 programming languages, depending on linguistic affinity and pretraining diversity, with limited reliance on natural language information wh...
-
R+R: Reassessing Java Security API Misuse in Current LLMs: A Replication on JCA and JSSE APIs with External Security Knowledge
Replication finds Java security API misuse persists in current LLMs but is reduced by external knowledge in a model-dependent manner.
-
BODHI: Precise OS Kernel Specification Inference
A structured domain-knowledge translation guide injected into LLM prompts improves OS kernel specification generation from 55% to 97% Pass@1 across nine models.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.