REVIEW 3 cited by
Using an LLM to Help With Code Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Forward citations
Cited by 3 Pith papers
-
Neurosymbolic Characterization for Reliable Access Control Policy Analysis
PolicySummarizer generates a simplified, human-readable regular expression for the requests a cloud access-control policy allows, and uses automata-based model counting to check the simplification against the exact po...
-
Positional Biases Shift as Inputs Approach Context Window Limits
The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.
-
Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code
Mutation-based testing shows LLM-generated code summaries frequently fail to reflect behavior-changing edits, with accuracy dropping as code complexity rises.
Discussion (0). Continue with ORCID to comment.