REVIEW 11 cited by
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The rapid advancements in large language models (LLMs) have greatly expanded the potential for automated code-related tasks. Two primary methodologies are used in this domain: prompt engineering and fine-tuning. Prompt engineering involves applying different strategies to query LLMs, like ChatGPT, while fine-tuning further adapts pre-trained models, such as CodeBERT, by training them on task-specific data. Despite the growth in the area, there remains a lack of comprehensive comparative analysis between the approaches for code models. In this paper, we evaluate GPT-4 using three prompt engineering strategies -- basic prompting, in-context learning, and task-specific prompting -- and compare it against 17 fine-tuned models across three code-related tasks: code summarization, generation, and translation. Our results indicate that GPT-4 with prompt engineering does not consistently outperform fine-tuned models. For instance, in code generation, GPT-4 is outperformed by fine-tuned models by 28.3% points on the MBPP dataset. It also shows mixed results for code translation tasks. Additionally, a user study was conducted involving 27 graduate students and 10 industry practitioners. The study revealed that GPT-4 with conversational prompts, incorporating human feedback during interaction, significantly improved performance compared to automated prompting. Participants often provided explicit instructions or added context during these interactions. These findings suggest that GPT-4 with conversational prompting holds significant promise for automated code-related tasks, whereas fully automated prompt engineering without human involvement still requires further investigation.
Forward citations
Cited by 11 Pith papers
-
Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models
A fully static, LLM-driven pipeline estimates the execution trace from a single failure log, prunes the test code, and ranks faulty locations at function, block, and line levels, tested on 785 industrial Python test cases.
-
Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories
An empirical study of 1,262 prompt changes across 243 GitHub repositories shows that developers mainly add and modify prompt components during feature development, rarely document the changes, and sometimes introduce ...
-
Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation
Pressure-style prompt framings were associated with lower functional correctness and more security warnings than neutral prompts in LiveCodeBench, while most other influence tactics had minimal effects.
-
Extension Decisions in Open Source Software Ecosystem
A GitHub Actions graph study reports that most new CI tools duplicate existing functionality and that a handful of early tools become the templates for later copies, although the supporting calculation is missing from...
-
A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insights
A five-project pilot study found that a GPT-4o agent pipeline translates 70.7% of Android files to valid Swift after automated validation, with failures concentrated in internal references and platform-specific APIs.
-
Resilient LLM-Empowered Semantic MAC Protocols via Zero-Shot Adaptation and Knowledge Distillation
A hybrid MAC protocol that starts with LLM-generated control messages and switches to a distilled neural model after retraining improves resilience to user-count changes at lower compute than pure LLM inference.
-
METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries
METAMON uses EvoSuite-generated regression tests and metamorphic LLM prompts to flag test oracles that contradict method documentation, reaching 0.72 precision and 0.48 recall on Defects4J-derived pairs.
-
Improved IR-based Bug Localization with Intelligent Relevance Feedback
BRaIn uses LLM relevance judgments to expand queries and re-rank search results, improving IR-based bug localization on the Bench4BL dataset.
-
Mobile Application Review Summarization using Chain of Density Prompting
Adapting the Chain of Density prompt to define entities as app features yields denser and more readable summaries of mobile app reviews than the original prompt, vanilla prompting, or extractive baselines.
-
Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese
Fine-tuning on 3,000 culturally authentic Lebanese sentences appears to beat 140,000 translated sentences, but the evidence is weakened by a potentially non-independent evaluation set and conflicting FLoRes results.
-
A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering
PRIMES is a proposed four-stage checklist for prompt creation, pilot testing, multi-LLM evaluation, and output validation in LLM-based repository mining.
Discussion (0). Continue with ORCID to comment.