The reviewed record of science sign in
Pith

arxiv: 2312.04528 · v2 · pith:J5UITM2Z · submitted 2023-12-07 · cs.LG · cs.AI

Using Large Language Models for Hyperparameter Optimization

Michael R. Zhang , Nishkrit Desai , Juhan Bae , Jonathan Lorraine , Jimmy Ba This is my paper

Reviewed by Pithpith:J5UITM2Zopen to challenge →

classification cs.LG cs.AI
keywords hyperparameterllmsmodelsoptimizationmodelapproachesbenchmarkslanguage
0
0 comments X
read the original abstract

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    cs.LG 2026-06 unverdicted novelty 7.0

    ASAP integrates an LLM agent over a pool of HPO tools and adds system-level optimizations (prefix-stable prompts, speculation parallelism, Self-Tuner) to improve end-to-end wall-clock performance on diverse HPO tasks.

  2. GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs

    cs.LG 2026-06 unverdicted novelty 6.0

    GRIMIP integrates LLMs as probabilistic surrogates inside Bayesian optimization to perform instance-specific MIP solver configuration and reports over 40% reduction in primal-dual integral on hard benchmark instances.

  3. LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval

    cs.CV 2026-06 unverdicted novelty 6.0

    A phase-aware LLM agent for ANN index optimization outperforms Optuna TPE by 33.3% and VDTuner by 34.2% on the SIEVE metric for HICO-DET retrieval.

  4. LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    The paper reformulates industrial continual learning for LLMs as a closed-loop ecosystem problem, identifies three core challenges, and organizes solutions around five lifecycle design principles.

  5. AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipes

    cs.AI 2026-06 unverdicted novelty 4.0

    An LLM-based bounded controller adapts ML training parameters from structured telemetry to correct overfitting and exploration issues, shown on TinyStories and robotic RL tasks.