Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback

Nan Sun; Xukai Zhao; Yucheng Hu; Yuxing Lu

REVIEW 5 major objections 6 minor 5 references

Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three AI agents beat grid, random, and Bayesian tuning

desk verdict A plausible multi-agent LLM tuning loop, but the empirical claims rest on implausibly weak baselines and the theory is vacuous. read the letter →

arxiv 2508.15757 v1 pith:VBQ6GZBM submitted 2025-08-21 cs.AI cs.CLcs.LGcs.MA

Yuxing Lu , Yucheng Hu , Nan Sun , Xukai Zhao This is my paper

classification cs.AIcs.CLcs.LGcs.MA

keywords language-guidedtuningmulti-agentLLMtextualgradientshyperparameteroptimizationneuralarchitecturesearchconfigurationinterpretabilityautomatedmachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Language-Guided Tuning (LGT), a way to optimize machine-learning configurations by having three large-language-model agents work in a loop: an Advisor proposes changes across architecture, data augmentation, training strategy, and hyperparameters; an Evaluator judges whether each change helped; and an Optimizer rewrites the Advisor's instructions based on accumulated history. The central claim is that this qualitative, natural-language feedback—called textual gradients—complements numerical loss and finds better configurations than grid search, random search, neural architecture search, and Bayesian optimization across six datasets. If correct, it would mean configuration tuning no longer has to be a black box, because every optimization step comes with a readable justification, and tuning can adapt during training rather than only as a pre-training search. The headline numbers are up to 23.3 percentage points of absolute accuracy improvement and a 49.3 percent error reduction, with the largest gains on MNIST and CIFAR-10.

What carries the argument

Textual gradients: qualitative, natural-language feedback signals that stand beside numeric loss and gradients. Three agents carry the mechanism: the Advisor proposes configuration deltas, the Evaluator produces a binary success judgment from training history, and the Prompt Optimizer updates the Advisor's prompt with lessons learned. The load-bearing loop is the history accumulation H_t = H_{t-1} plus the current configuration, metrics, and prompt, which lets later suggestions refer to earlier failures and creates the claimed self-improving behavior.

What would settle it

Re-run the comparison with a properly tuned strong baseline: a standard MLP on MNIST (or a ResNet on CIFAR-10) with sensible defaults, plus the same 50-configuration budget for random search, grid search, and Bayesian optimization. If LGT does not beat that baseline by a statistically significant margin over 10 seeds, the claim of consistent superiority fails. Also, compare LGT against a single LLM call that proposes a configuration from prior knowledge with no iterative feedback; if that matches LGT's accuracy, the feedback loop is not the source of the gains.

Watch

Extended reading notes

Core claim

On the paper's own terms, LGT establishes that qualitative textual feedback can serve as an optimization signal alongside numerical gradients. At each epoch, an Advisor emits a configuration change, an Evaluator returns a success signal by comparing current and baseline training histories, and a Prompt Optimizer uses that history to refine the Advisor's prompt, forming a self-improving loop. The reported result is that this coordinated loop converges faster and to lower loss than all five baselines, with the sharpest gaps on MNIST (78.41% to 98.99% accuracy) and CIFAR-10 (49.01% to 69.64%), and roughly 49% error reduction on the regression tasks. The convergence claim is formalized in Equati

Load-bearing premise

The comparison is fair: the baselines are implemented well enough that beating them means something, especially the 'No Tuning' baseline which reaches only 78.41% on MNIST, far below what a standard model achieves.

Editorial extensions

If this is right

Configuration tuning tools could expose a readable log of why each change was made, letting practitioners audit and trust automated decisions.
Tuning could become dynamic, adjusting architecture, augmentation, loss, optimizer, and hyperparameters during training rather than only as a fixed pre-training search.
The same framework could improve automatically as backend language models improve, without changing the optimization algorithm.
The multi-agent loop applies to search spaces where dimensions are discrete and interdependent, which is exactly where numeric gradient methods are unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The largest reported gains may come more from the LLM's prior knowledge of sensible defaults than from the feedback loop; a controlled test with a one-shot LLM advisor (no history, no Evaluator) would separate the two effects.
The 'No Tuning' baselines appear far weaker than standard practice—a plain MLP on MNIST typically exceeds 98%, while the paper's baseline sits at 78.41%—so the improvement magnitudes are likely inflated even if the method itself works.
The paper does not report API cost or wall-clock overhead; LGT is most practical when each training run is expensive relative to a few LLM calls.
If textual gradients transfer, the same three-agent loop could optimize reward functions, prompts, or simulator parameters, extending beyond machine-learning configuration.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper is a composition of OPRO-style LLM optimization, TextGrad textual gradients, and an AgentHPO-like multi-agent loop, wrapped in a new three-role structure (Advisor/Evaluator/Optimizer). That combination is mildly novel, and the authors provide code and a clear framework description. The interpretability example in Figure 4 is a genuine feature.

The problem is the empirical and theoretical support. The headline gains over baselines are not credible because the baselines are not specified adequately. The 'No Tuning' MNIST accuracy of 78.41% is far below what any reasonable default MLP would achieve; this alone inflates the reported 23.3% improvement. The paper never states the architecture, training budget, or hyperparameters for the baselines, and 'statistically significant margins' is asserted without reporting any test. The comparison with NAS and Bayesian Optimization is similarly underspecified—we don't know the search spaces or the number of configurations they were allowed. LGT itself can alter architecture, so comparing it to a fixed default is not apples-to-apples.

The theory section is worse than weak: Eq. (12) gives a bound that grows with t, which contradicts the claimed convergence in Eq. (9). The 'approximation gap' delta is defined to make the statement true, so the contribution is tautological. There is no derivation of the stability condition Eq. (13). The paper would be no worse off if the theory were removed.

There are also presentation issues: Table 2 has formatting problems, the ablation numbers don't match the main table (MNIST AUC 99.99 vs 98.99 accuracy), and the paper plots test loss per epoch without a validation protocol—that is a potential leakage concern.

On balance: the framework idea is worth a serious look, but this manuscript does not establish that LGT outperforms competent conventional tuning. It might be salvageable if the authors redo the experiments with fair baselines, report compute budgets, and drop or fix the theory. As is, I would not trust the claims.

For you: not a must-read, but a useful example of how easy it is to overclaim in LLM-based optimization. I would not cite it in its current form. If I were an editor, I'd send it to review because the topic is timely and the code is available, but I'd expect a major revision or a rejection.

Referee Report

5 major / 6 minor

Summary. The paper introduces Language-Guided Tuning (LGT), a multi-agent LLM framework for configuration optimization. Three agents (Advisor, Evaluator, Optimizer) propose, assess, and refine configuration changes via natural-language 'textual gradients', with prompt updates forming a self-improving loop. The authors claim convergence to a local optimum and report on six datasets, with 'up to 23.3% absolute accuracy improvement and 49.3% error reduction' over five baselines (No Tuning, Random Search, Grid Search, NAS, Bayesian Optimization). The manuscript includes an ablation study, convergence plots, and example agent outputs.

Significance. If the empirical claims were reliable, LGT would be a useful contribution to LLM-guided AutoML and hyperparameter optimization, and the idea of interpretable, textual-gradient-driven configuration updates is appealing. The code link and 10-seed repeated runs are positive features. However, the central empirical claim is not currently supported: the baseline implementations appear seriously underpowered (e.g., No Tuning on MNIST at 78.41% accuracy), no statistical tests are reported despite the claim of 'statistically significant margins', and the computational budgets of LGT versus the baselines are not aligned. The theoretical convergence analysis is also circular and internally inconsistent. Thus the strengths of the idea are overshadowed by load-bearing problems in validation and theory.

major comments (5)

[Experimental Setup / Table 2] The No Tuning MNIST baseline reports 78.41% accuracy. A standard two-layer MLP with ReLU activations on MNIST typically exceeds 98% with 10 epochs of training, so this baseline is far below a competent default. The headline 'up to 23.3% accuracy improvement' is therefore driven largely by the weakness of the comparator. The authors must specify the exact model architecture, optimizer, learning rate, and preprocessing used for each baseline, and compare against properly configured standard defaults. The same concern applies to the CIFAR-10 No Tuning baseline (49.01%).
[Results / Table 2] The paper asserts that LGT 'consistently outperforms all baselines with statistically significant margins' but reports no significance test, p-value, confidence interval, or paired comparison across the 10 seeds. Some gaps are small (e.g., Water Potability: 66.92 vs. 66.67 for Bayesian Optimization). Without a specified test (paired t-test, Wilcoxon, or bootstrap), the significance claim is unsubstantiated. This undermines the central empirical conclusion.
[Theoretical Analysis, Eqs. (9) and (12)] The convergence analysis is not valid as written. Eq. (9) asserts lim E[L(c_t)] = L* + delta, where delta is defined as the 'approximation gap due to the discrete nature of LLM-based optimization'; this is a tautology rather than a proof. Eq. (12) states L(c_t) - L* <= O(1/sqrt(t)) + O(epsilon_p * t), and the second term diverges as t grows, contradicting the claimed convergence in Eq. (9). The section needs a corrected derivation, or it should be removed and replaced with an empirical convergence characterization.
[Experimental Setup / Baselines] The computational budget is not fairly specified. LGT trains a single model for 10 epochs with epoch-level updates, while the text states each baseline method evaluates 'maximum 50 configurations with 10 epochs per evaluation', implying up to 50 times the training cost. If this is the intended protocol, the comparison is unequal. The authors must report the total wall-clock or epoch budget per method and ensure all methods receive comparable computational effort.
[Methods, Eq. (2)] The augmentation selection rule is written as tau_{t+1} = arg max_{tau in T} E[L(tau(x), y)], i.e., it maximizes the expected loss L. Since L is defined in Eq. (1) as the loss to be minimized, this is a sign/optimization-direction error. The intended selection should minimize validation loss or maximize a performance metric. This is a technical error in the formal description of the method.

minor comments (6)

[Table 2] Several entries are formatted incorrectly, e.g., '77.882.3' and '97.550.4', which appear to be missing '±'. Please fix table formatting.
[Figure 4] The example agent output contains 'NgO' artifacts and layout issues. Present a clean, verbatim transcript to support the interpretability claim.
[Table 4 / Table 2] The ablation baseline 'Baseline (No LLM)' reports MNIST AUC 97.55, whereas Table 2's 'No Tuning' reports MNIST accuracy 78.41. Clarify whether these are the same configuration and why the metrics differ so dramatically; also clarify that Table 4 uses AUC while Table 2 uses accuracy.
[Eq. (13)] The operator norm ||A_adv ∘ A_eval ∘ A_opt|| <= 1 is not defined. These agents are not linear operators, so the inequality is not meaningful without formal definitions.
[Datasets] The 'House Price' dataset is described as having 50,000 samples with 5 features, but the cited reference (Harrison and Rubinfeld 1978) is the Boston housing dataset with 506 samples. Please specify the exact dataset source and preprocessing.
[References] The DeepSeek API is cited via the DeepSeek-V3 technical report, but the specific model/API version used is not stated. Also, the code repository link should be versioned or archived for reproducibility.

Circularity Check

1 steps flagged · score 3.0 of 10

Convergence result Eq. (9) is definitional: δ is introduced as the approximation gap, making the limit L*+δ true by construction; the empirical comparison is independent and no load-bearing self-citation chain exists.

self definitional [Theoretical Analysis, Convergence Analysis, Eq. (9)]
"Under the assumption that the configuration space C is compact and the loss functionL is Lipschitz continuous, the system converges to a local optimum: lim_{t→∞} E[L(ct)] = L∗ + δ (9) where L∗ is the optimal loss and δ ≥ 0 represents the approximation gap due to the discrete nature of LLM-based optimization."

δ is introduced as 'the approximation gap' between the limit and L*. With that meaning, the asserted identity lim E[L(ct)] = L* + δ is not derived from compactness/Lipschitz assumptions; it is just the definition of δ and it presupposes that the limit exists. Any bounded process can be written as L* plus its gap, so Eq. (9) has no predictive content. The preceding sentence asserts the very convergence ('the system converges to a local optimum') that the analysis purports to establish, and δ absorbs the entire discrepancy. This is pattern 1: the claimed convergence result is defined in terms of the gap it is supposed to quantify. The empirical results in Tables 2–3 do not rely on Eq. (9), so the circularity is localized to the theoretical section.

full rationale

The central empirical claim—up to 23.3% absolute accuracy improvement and 49.3% error reduction, with consistent gains over NAS and Bayesian optimization—rests on Tables 2 and 3. Those comparisons are externally falsifiable and are not derived from the convergence theory; no fitted parameter is renamed as a prediction. The weak 'No Tuning' MNIST baseline (78.41%) is a baseline-fairness concern, not a circular reduction. The convergence analysis in Eq. (9) is circular/definitional: δ is defined as the approximation gap, so the limit statement reduces to the definition of δ and assumes the limit exists. Separately, Eq. (12)'s O(ε_p·t) term grows with t and does not establish convergence, but that is an internal correctness defect, not a circular step. Self-citations (DoctorRAG, KARMA) appear only in related work and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The empirical core therefore has independent content, and the circularity is a localized, non-central theoretical step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

LGT introduces no new physical or mathematical entities. It borrows 'textual gradients' from TextGrad and reuses standard LLM-generated prompts. The central load-bearing assumptions are the reliability of the Evaluator, the boundedness of LLM updates, and the fairness of the baseline comparison.

free parameters (2)

Search space definitions = 2-5 layers, 32-512 hidden units, lr 1e-4 to 1e-1, class weights 0.1-10.0
Hand-defined in Experimental Setup; the relative performance of LGT versus baselines depends on these spaces being equally favorable to all methods.
LLM sampling temperature = 0.2
Chosen without sensitivity analysis; affects the exploration/exploitation balance of the Advisor and other agents.

assumptions (4)

domain assumption Configuration space C is compact and loss L is Lipschitz continuous.
Invoked in Theoretical Analysis (before Eq. 9) to assert convergence.
ad hoc to paper LLM update magnitude is bounded: ||Delta c_t|| <= epsilon and ||Delta prompt_t|| <= epsilon_p for t > T0.
Assumed in Eqs. (10)-(11) to stabilize the feedback loop; no mechanism or justification is given for these bounds.
domain assumption The Evaluator's binary success signal and textual gradient correctly indicate optimization progress.
The entire feedback loop relies on the Evaluator making correct judgments about whether a configuration change helped; the paper provides no validation of this.
domain assumption The test set is not used to guide configuration decisions.
Figure 2 plots test loss at each epoch, and the experimental setup does not state that a separate validation set is used for agent decisions, creating a leakage risk.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback." pith.science (2026). https://pith.science/paper/VBQ6GZBM

@misc{pith2026250815757,
  author       = {Pith},
  title        = {Pith review of: Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBQ6GZBM}},
  note         = {Machine review of arXiv:2508.15757}
}

read the original abstract

Configuration optimization remains a critical bottleneck in machine learning, requiring coordinated tuning across model architecture, training strategy, feature engineering, and hyperparameters. Traditional approaches treat these dimensions independently and lack interpretability, while recent automated methods struggle with dynamic adaptability and semantic reasoning about optimization decisions. We introduce Language-Guided Tuning (LGT), a novel framework that employs multi-agent Large Language Models to intelligently optimize configurations through natural language reasoning. We apply textual gradients - qualitative feedback signals that complement numerical optimization by providing semantic understanding of training dynamics and configuration interdependencies. LGT coordinates three specialized agents: an Advisor that proposes configuration changes, an Evaluator that assesses progress, and an Optimizer that refines the decision-making process, creating a self-improving feedback loop. Through comprehensive evaluation on six diverse datasets, LGT demonstrates substantial improvements over traditional optimization methods, achieving performance gains while maintaining high interpretability.

Figures

Figures reproduced from arXiv: 2508.15757 by the authors.

**Figure 2.** Train and test loss curves across six datasets com [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Configuration evolution showing coordinated op [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Example agent outputs showing multi-agent inter [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

5 extracted references · 5 linked inside Pith

[2]

arXiv preprint arXiv:2505.19538

DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients. arXiv preprint arXiv:2505.19538. Lu, Y .; and Wang, J. 2025. KARMA: Leveraging Multi- Agent LLMs for Automated Knowledge Graph Enrichment. arXiv:2502.06472. Ma, Z.; Guo, H.; Chen, J.; Peng, G.; Cao, Z.; Ma, Y .; and Gong, Y .-J. 2024. Llamoco: Instruction tuning of lar...

arXiv 2025
[5]

arXiv preprint arXiv:2303.18223, 1(2)

A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2). Z¨oller, M.-A.; and Huber, M. F. 2021. Benchmark and sur- vey of automated machine learning frameworks. Journal of artificial intelligence research, 70: 409–472

arXiv 2021
[2023]

arXiv preprint arXiv:2312.04528

Using large language models for hyperparameter op- timization. arXiv preprint arXiv:2312.04528. Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y .; Min, Y .; Zhang, B.; Zhang, J.; Dong, Z.; et al

arXiv
[2024]

arXiv preprint arXiv:2405.16434

The importance of directional feedback for llm-based optimizers. arXiv preprint arXiv:2405.16434. Pontes, F. J.; Amorim, G.; Balestrassi, P. P.; Paiva, A.; and Ferreira, J. R. 2016. Design of experiments and focused grid search for neural network parameter optimization. Neuro- computing, 186: 22–34. Pouyanfar, S.; Sadiq, S.; Yan, Y .; Tian, H.; Tao, Y .; ...

arXiv 2016
[2025]

arXiv preprint arXiv:2505.12285

Calm: Co-evolution of algorithms and language model for automatic heuristic design. arXiv preprint arXiv:2505.12285. Karmaker, S. K.; Hassan, M. M.; Smith, M. J.; Xu, L.; Zhai, C.; and Veeramachaneni, K. 2021. Automl to date and be- yond: Challenges and opportunities.Acm computing surveys (csur), 54(8): 1–36. Krizhevsky, A.; Hinton, G.; et al. 2009. Learn...

arXiv 2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2]

arXiv preprint arXiv:2505.19538

DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients. arXiv preprint arXiv:2505.19538. Lu, Y .; and Wang, J. 2025. KARMA: Leveraging Multi- Agent LLMs for Automated Knowledge Graph Enrichment. arXiv:2502.06472. Ma, Z.; Guo, H.; Chen, J.; Peng, G.; Cao, Z.; Ma, Y .; and Gong, Y .-J. 2024. Llamoco: Instruction tuning of lar...

arXiv 2025

[2] [5]

arXiv preprint arXiv:2303.18223, 1(2)

A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2). Z¨oller, M.-A.; and Huber, M. F. 2021. Benchmark and sur- vey of automated machine learning frameworks. Journal of artificial intelligence research, 70: 409–472

arXiv 2021

[3] [2023]

arXiv preprint arXiv:2312.04528

Using large language models for hyperparameter op- timization. arXiv preprint arXiv:2312.04528. Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y .; Min, Y .; Zhang, B.; Zhang, J.; Dong, Z.; et al

arXiv

[4] [2024]

arXiv preprint arXiv:2405.16434

The importance of directional feedback for llm-based optimizers. arXiv preprint arXiv:2405.16434. Pontes, F. J.; Amorim, G.; Balestrassi, P. P.; Paiva, A.; and Ferreira, J. R. 2016. Design of experiments and focused grid search for neural network parameter optimization. Neuro- computing, 186: 22–34. Pouyanfar, S.; Sadiq, S.; Yan, Y .; Tian, H.; Tao, Y .; ...

arXiv 2016

[5] [2025]

arXiv preprint arXiv:2505.12285

Calm: Co-evolution of algorithms and language model for automatic heuristic design. arXiv preprint arXiv:2505.12285. Karmaker, S. K.; Hassan, M. M.; Smith, M. J.; Xu, L.; Zhai, C.; and Veeramachaneni, K. 2021. Automl to date and be- yond: Challenges and opportunities.Acm computing surveys (csur), 54(8): 1–36. Krizhevsky, A.; Hinton, G.; et al. 2009. Learn...

arXiv 2021