REVIEW 10 cited by
Optimizing Large Language Model Hyperparameters for Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs), such as GPT models, are increasingly used in software engineering for various tasks, such as code generation, requirements management, and debugging. While automating these tasks has garnered significant attention, a systematic study on the impact of varying hyperparameters on code generation outcomes remains unexplored. This study aims to assess LLMs' code generation performance by exhaustively exploring the impact of various hyperparameters. Hyperparameters for LLMs are adjustable settings that affect the model's behaviour and performance. Specifically, we investigated how changes to the hyperparameters: temperature, top probability (top_p), frequency penalty, and presence penalty affect code generation outcomes. We systematically adjusted all hyperparameters together, exploring every possible combination by making small increments to each hyperparameter at a time. This exhaustive approach was applied to 13 Python code generation tasks, yielding one of four outcomes for each hyperparameter combination: no output from the LLM, non executable code, code that fails unit tests, or correct and functional code. We analysed these outcomes for a total of 14,742 generated Python code segments, focusing on correctness, to determine how the hyperparameters influence the LLM to arrive at each outcome. Using correlation coefficient and regression tree analyses, we ascertained which hyperparameters influence which aspect of the LLM. Our results indicate that optimal performance is achieved with a temperature below 0.5, top probability below 0.75, frequency penalty above -1 and below 1.5, and presence penalty above -1. We make our dataset and results available to facilitate replication.
Forward citations
Cited by 10 Pith papers
-
DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis
DF3DV-1K supplies 1,048 scenes with clean and cluttered image pairs plus a challenging 41-scene subset to benchmark and improve distractor-free radiance field methods.
-
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Prompt language affects LLM code generation, but English is not consistently best: Chinese prompts improve Python correctness on CoderEval, while quality and lexicon effects vary by model and programming language.
-
Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding
LLM unlearning methods that pass greedy-decoding benchmarks leak forgotten facts when the model is sampled repeatedly, and the new leak@k metric quantifies this.
-
Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT
A large empirical study of GPT-4o shows that top-p affects Java method generation more than temperature, and that low temperature reduces the number of methods for which a correct implementation is found across repeat...
-
On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT
ChatGPT's generated Java methods are more likely to closely match copyleft code when given class or access-method context (up to 5X), less likely at temperature 2, and unaffected by an explicit anti-copy prompt.
-
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.
-
Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness
Feeding static-analysis warnings back to GPT-4o over ten iterations cut security issues from over 40% to about 13%, readability issues from over 80% to about 11%, and reliability warnings from over 50% to about 11%.
-
Investigating the Role of LLMs Hyperparameter Tuning and Prompt Engineering to Support Domain Modeling
Combining search-based hyperparameter tuning of Llama 3.1 with few-shot prompting improves domain-model generation quality over default settings on most of ten test domains, though only for a text-similarity metric.
-
AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
A new 70-challenge benchmark measures whether language models can autonomously discover and exploit AI/ML security vulnerabilities, finding Claude-3.7-Sonnet solves 61% of challenges while the best open-source model s...
-
Comparing Human and LLM Generated Code: The Jury is Still Out!
On 72 Python tasks, GPT-4 code passed 87.3% of tests versus 54.9% for one student's code, but was more complex and showed more severe security issues.
Discussion (0). Continue with ORCID to comment.