{"id":"27235029-a17c-464b-9f30-b3a7b8644895","arxiv_id":"2505.19205","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"OptiMindTune assigns recommendation, evaluation, and decision roles to three Gemini-powered agents for hyperparameter search, but its reported 3-dataset advantage over Optuna lacks code, error bars, and a fair baseline comparison.","lead":"A preprint proposes OptiMindTune, a hyperparameter tuning system that splits the work among three AI agents powered by Google's Gemini models. It reports small accuracy gains over Optuna on two of three small datasets, but provides no code, prompts, or repeated trials, so the gains are not yet verifiable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section VI.A describes the reported results as 'expected' and 'anticipated' rather than as measured, so the empirical core of the central claim is internally undercut.","rationale":"I read the paper as an architecture proposal with preliminary empirical claims. The strongest assertion is that a three-agent LLM framework can reach near-Optuna accuracy with 3–4 trials on small tabular datasets and that the multi-agent interaction drives the improvement. For this to hold, the reported tables need to be actual measurements, and the measurements need to be reproducible. The most load-bearing weakness is the manuscript's own Section VI.A, which uses future/conditional language about the experimental results, saying the authors 'anticipate observing' the effects. Combined with the complete absence of artifacts (code, prompts, logs, seeds), the experimental section cannot be distinguished from a plan or a hypothetical illustration. Even granting that the experiments were run, the reader's concerns are correct: no implementation details, no error bars, no AgentHPO baseline run, and no ablation separating the multi-agent role split from a single LLM call. The comparison is asymmetric—Optuna tuned both models and reported the best, while OptiMindTune was constrained to one model per dataset. The three datasets are tiny, and the headline differences are about one percentage point, well within typical cross-validation variance without repeated runs. I credit the paper for a legible architecture and candid limitation statements, but the central claim is not substantiated by the submitted evidence. The Section VI.A observation strengthens the reader's rejection, so the verdict remains unchanged.","tokens_in":8181,"tokens_out":4437,"duration_ms":43376,"concrete_test":"Obtain the actual run artifacts from the authors: the execution script, the 3–4 agent conversation logs, exact prompt templates, Gemini configuration parameters, and the fixed random seeds. Rerun the full pipeline ten times and compare the resulting mean 5-fold accuracies to Table I. If the reported values (97.02%, 96.67%, 98.33%) do not reproduce within the variance expected from 5-fold cross-validation, or if the artifacts cannot be produced, the empirical claim fails. Additionally, ask the authors whether the experiments were completed before submission; if Section VI.A describes planned rather than completed work, Tables I and II must be treated as illustrative, not as results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that OptiMindTune improves convergence efficiency and robustness, evidenced by outperforming Optuna in two of three datasets with only 3–4 trials—rests entirely on Tables I and II. The manuscript itself, however, undermines those tables. In Section VI.A, the authors write that 'The experimental results are expected to validate OptiMindTune’s efficacy' and that they 'anticipate observing' the reported effects, using future or conditional tense. This is not a trivial stylistic slip: the entire experiments section is framed as a 'preliminary study,' and Section VI.A describes what the results are expected to show, not what they did show. If the tables are not actual measurements, the comparative accuracies (97.02%, 96.67%, 98.33%) are projections, and the claim has no empirical basis. Even if Section VI.A is read charitably as poorly worded boilerplate and the experiments were run, the reported numbers remain non-reproducible: there are no prompts, no agent conversation logs, no Gemini sampling parameters, no cross-validation fold details, no error bars, and no repeated-trial variance. The single number per cell could come from memorized defaults or a single lucky LLM call, so the load-bearing inference—that the three-agent role split is responsible for the result—is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptiMindTune, a multi-agent framework for hyperparameter optimization in which three LLM-powered agents (Recommender, Evaluator, Decision) collaborate to select scikit-learn models and hyperparameters. It reports preliminary experiments on three UCI datasets comparing mean cross-validated accuracy and computational efficiency against Optuna (TPE), claiming superiority in two of three datasets with fewer trials.","tokens_in":8459,"tokens_out":4271,"duration_ms":30611,"significance":"If substantiated, the claim that a three-agent LLM system can match or beat Optuna in 3–4 trials on small tabular benchmarks would be a notable result for LLM-based AutoML. However, the manuscript provides no code, prompts, agent logs, or repeated-run statistics, and Section VI.A frames the results in future tense, so the significance cannot currently be assessed. The high-level architecture is clearly described, but no artifact is provided to verify that the described framework actually produced the numbers.","major_comments":[{"comment":"The future/conditional tense in Section VI.A ('The experimental results are expected to validate OptiMindTune’s efficacy'; 'We anticipate observing...') directly contradicts the past-tense presentation of results in Section V and the tables. If the reported accuracies in Tables I and II are projections rather than measurements, the central claim of outperforming Optuna in two of three datasets has no empirical basis. This must be resolved before the paper can be evaluated.","section":"Section VI.A vs Sections IV–V"},{"comment":"The results are single point estimates with no standard deviations, confidence intervals, or repeated runs. Gemini inference is stochastic, and no temperature or sampling parameters are reported, so the reported accuracies could result from a single favorable LLM call. Without repeated independent runs, the 97.02% vs 96.14% and 98.33% vs 97.78% differences are within plausible noise and do not support the claim of superiority.","section":"Tables I and II"},{"comment":"The comparison is asymmetric: OptiMindTune is assigned a fixed model per dataset (LogisticRegression for Breast Cancer and Wine; RandomForestClassifier for Iris), while Optuna is allowed to search over both models and its best model is reported. This gives OptiMindTune a human-provided model choice as an advantage, and makes the 'outperforms Optuna's best model' comparison misleading. A fair comparison would require that both systems perform model selection or that OptiMindTune also be evaluated without that prior.","section":"Section IV.B"},{"comment":"The Decision Agent determines the termination criteria, so the lower trial counts in Table II (3–4 vs 10) are not evidence of sample efficiency; they are a product of the system's own stopping rule. The paper does not report the target accuracy or exploration ratio used for each dataset, nor does it show Optuna's accuracy as a function of trial count. Without this information, the trial-count comparison is uncontrolled and cannot support a claim of faster convergence.","section":"Sections III.B and IV.E"},{"comment":"The implementation details of the three agents are not specified: no prompt templates, message schemas, history handling, or LLM inference settings are reported beyond a single sentence saying 'Gemini 2.0 Flash was used to support agent reasoning.' As a result, the experiments are not reproducible, and the reported numbers cannot be attributed to the multi-agent architecture as opposed to memorized default hyperparameter recommendations.","section":"Sections III.A and IV.E"}],"minor_comments":[{"comment":"The sentence 'Our framework integrates principles from advanced large language models, and adaptive search' contains an unnecessary comma before 'and'; please revise.","section":"Abstract"},{"comment":"The keyword list repeats entries: 'Hyperparameter Optimization, Multi-Agent Systems, AutoML' appears twice. The list should be deduplicated.","section":"Keywords"},{"comment":"The table reports OptiMindTune times to two decimals (7.91 s, 14.86 s) but Optuna times to two decimals as well (7.28 s, 25.40 s, 6.08 s), which is actually consistent. However, the 'Trials/s' values are derived from time and n_trials, and the paper would benefit from stating this explicitly to avoid ambiguity.","section":"Table II"},{"comment":"The conclusion is largely speculative and contains repeated phrasing such as 'truly intelligent and autonomous hyperparameter optimization'; consider grounding the conclusion in the reported (or actually performed) experiments.","section":"Section VII"},{"comment":"Reference [16] for Gemini is given as a 2023 Google blog-style citation without authors or version; the current manuscript would be better served by citing the official Gemini technical report with full author list.","section":"References"}],"recommendation":"reject","confidential_remarks":"The future-tense framing in Section VI.A is a serious integrity concern. If the experiments were not actually run, the paper cannot be salvaged by revision; if they were run, the authors must rewrite the discussion and supply the missing implementation details. The asymmetric comparison and the self-determined stopping rule further undermine the central claims. I recommend rejecting the paper in its current form, but I would be open to a resubmission that presents genuinely implemented experiments with agent prompts, repeated runs, and a fair comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: Section VI.A says the experimental results \"are expected to validate\" OptiMindTune's efficacy and that they \"anticipate observing\" the outcomes. That's future tense next to Tables I and II. Either the tables are projections or the discussion is boilerplate, and both options are bad. The empirical core of the paper is internally undercut, and this isn't a minor wording issue.\n\nWhat the paper does well: the three-agent architecture (Recommender, Evaluator, Decision) is described legibly, the interaction loop is concrete, and the limitations section covers LLM dependency, interpretability, and search-space discretization honestly. For someone who wants a quick sketch of how LLM agents might split HPO roles, this is readable.\n\nThe problems go beyond the tense slip. The comparison is asymmetric: OptiMindTune is assigned one model per dataset while Optuna searches two models and the best result is reported. The reported accuracies are single point estimates with no standard deviations, repeats, or confidence intervals, and Gemini inference is stochastic. No code, prompts, agent logs, or reproducibility artifacts are provided. The trial-count advantage is partly manufactured because the Decision Agent sets its own termination criteria. AgentHPO is cited but never run as a baseline, so the central claim that the role split improves convergence efficiency is untested. The related work itself shows LLM-agent HPO already exists, and the three-agent split is a configuration change over AgentHPO's two-agent design.\n\nWho is this for? A reader working on LLM-powered AutoML might skim it as an architecture suggestion, but there is no evidence here to justify a serious referee assignment. The honest fix is to rerun the experiments, report distributions, provide the prompts and code, and compare against AgentHPO and single-agent LLM baselines. If that version exists, it would be a reasonable workshop or short-paper submission.\n\nFor now, I'd desk reject and tell the authors to come back with measurements.","headline":"The paper is a clear architecture sketch whose own discussion section says the reported results are expected rather than measured, so the central claim is currently unsupported.","tokens_in":8995,"tokens_out":3189,"would_cite":false,"duration_ms":31234,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-agent LLM loop can match or beat a standard optimizer's accuracy in a third of the trials.","keywords":["hyperparameter optimization","multi-agent systems","large language models","AutoML","Optuna","TPE","scikit-learn classifiers","Gemini"],"falsifier":"Run an ablation that replaces the Recommender Agent with random draws from the same scikit-learn model spaces, keeping the Evaluator and Decision Agents' trial count and reporting logic unchanged; if random draws reach the same 97–98 percent accuracies in 3–4 trials, the claimed convergence gain comes from the search spaces rather than from the multi-agent loop.","tokens_in":7973,"feed_emoji":"🤖","tokens_out":7637,"duration_ms":64496,"temperature":0.7,"pith_summary":"OptiMindTune is a proposed answer to the question of whether hyperparameter tuning has to be an expensive black-box search. The paper argues that splitting the job among three LLM-powered agents, one that recommends models and hyperparameters, one that evaluates them, and one that decides what to try next, lets the loop converge with only 3–4 trials on small tabular classification problems. In the reported experiments it beats Optuna's best of two scikit-learn models on Breast Cancer (97.02% vs 96.14%) and Wine (98.33% vs 97.78%), while losing narrowly on Iris (96.67% vs 98.00%). The reason this matters is that HPO cost is usually measured in training runs; if a handful of informed trials can rival a standard 10-trial optimizer, then LLM-driven division of labor becomes a credible ingredient in AutoML.","feed_headline":"AI agent trio tunes hyperparameters in 3–4 tries","feed_subtitle":"On two small UCI benchmarks the three-agent loop lands ~97–98 percent accuracy, beating Optuna's 10-trial best.","key_machinery":"The machinery is the three-agent optimization loop with a shared history. The Recommender Agent inspects dataset characteristics and past trial outcomes to propose a model plus hyperparameter configuration; the Evaluator Agent trains that configuration with 5-fold cross-validation and reports accuracy; the Decision Agent accepts or rejects the proposal, balances exploration against exploitation, and feeds guidance back to the Recommender. The optimization history acts as the collective memory, so every new trial is conditioned on previous outcomes rather than sampled from a static prior.","core_discovery":"The central claim is that structured communication between specialized LLM agents—Recommender, Evaluator, Decision—yields faster convergence and greater robustness than single-agent or monolithic optimizers. On the paper's own terms, this is shown by three UCI classification benchmarks: OptiMindTune achieves 97.02% mean cross-validated accuracy on Breast Cancer with LogisticRegression in 4 trials and 98.33% on Wine with LogisticRegression in 3 trials, exceeding Optuna's best-performing model (96.14% and 97.78%) under a 10-trial budget. On Iris, OptiMindTune's RandomForestClassifier reaches 96.67%, behind Optuna's LogisticRegression at 98.00%. The paper reads these results as evidence that the multi-agent approach is sample-efficient and competitive, particularly when the model family is fixed in advance.","pith_inferences":["The decisive test the paper does not run is an ablation that replaces the Recommender Agent with random sampling from the same model and hyperparameter spaces; if random draws match the reported 3–4 trial accuracies, the multi-agent loop itself is not what is buying sample efficiency.","Because the accuracy margins are a fraction of a percentage point, the most defensible reading of the result is about trial efficiency, not about finding better optima; a single cross-validation fold could flip the ordering on Iris.","An implicit dependence the paper leaves unexamined is the LLM's prior knowledge of common hyperparameter ranges; if that prior does most of the work, the Recommender is leaning on memorized defaults rather than learning from the dataset, which would determine how well the approach transfers to unfamiliar model families."],"forward_implications":["HPO on small tabular datasets can reach near-Optuna accuracy with roughly one-third to one-half of the trials Optuna uses.","OptiMindTune's wins on Breast Cancer and Wine cannot be attributed to model selection, because the framework was restricted to one model per dataset while Optuna chose between two.","Real-time logging of recommendations, evaluations, and decisions gives the HPO process an audit trail that black-box optimizers do not expose.","The architecture is designed to extend naturally toward model selection, deep learning support, cost-aware decisions, and multi-objective tuning, as the paper's future-work section describes."],"supporting_citations":[{"why":"Supplies the Optuna baseline with its TPE sampler, pruning, and 10-trial budget that OptiMindTune is compared against.","marker":"[10]"},{"why":"Defines the TPE algorithm that models P(x|y<y*) and P(x|y>=y*) and underpins Optuna's search.","marker":"[9]"},{"why":"Establishes random search as the traditional HPO baseline and motivates the sample-efficiency comparison.","marker":"[1]"},{"why":"Provides the Gemini model family that powers all three agents' reasoning in the reported experiments.","marker":"[16]"},{"why":"Describes AgentHPO, the two-agent LLM HPO system that OptiMindTune positions itself against as a complementary approach.","marker":"[23]"},{"why":"Represents the distributed AutoML approach that lacks the dynamic inter-agent collaboration the paper claims OptiMindTune adds.","marker":"[19]"}],"fun_headline_variants":["Three AI agents collaborate to speed hyperparameter search","Agent trio beats Optuna's 10-trial best on two UCI sets","Multi-agent framework tunes models in under five attempts","Gemini-powered agents converge on hyperparameters faster","Collaborative LLM agents match Optuna with fewer trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 3–4 trial results depend on the unstated implementation of the three agents—exact prompts, Gemini inference settings, and message parsing—and on the LLM reliably converting dataset characteristics into useful hyperparameter ranges.","fun_headline_variants_meta":{"raw":{"variants":["Three AI agents collaborate to speed hyperparameter search","Agent trio beats Optuna's 10-trial best on two UCI sets","Multi-agent framework tunes models in under five attempts","Gemini-powered agents converge on hyperparameters faster","Collaborative LLM agents match Optuna with fewer trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2357,"prompt_tokens":910,"completion_tokens":1447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1367}},"tokens_in":526,"tokens_out":1447,"duration_ms":12771,"temperature":1.0,"reasoning_tokens":1367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:18:41.841126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an ablation that replaces the Recommender Agent with random draws from the same scikit-learn model spaces, keeping the Evaluator and Decision Agents' trial count and reporting logic unchanged; if random draws reach the same 97–98 percent accuracies in 3–4 trials, the claimed convergence gain comes from the search spaces rather than from the multi-agent loop.","supporting_citations":[{"cited_title":"Optuna: A next-generation hyperparameter optimization framework,","cited_arxiv_id":null,"evidence_quote":"Supplies the Optuna baseline with its TPE sampler, pruning, and 10-trial budget that OptiMindTune is compared against."},{"cited_title":"Algorithms for hyper-parameter optimization,","cited_arxiv_id":null,"evidence_quote":"Defines the TPE algorithm that models P(x|y<y*) and P(x|y>=y*) and underpins Optuna's search."},{"cited_title":"Random search for hyper-parameter optimization,","cited_arxiv_id":null,"evidence_quote":"Establishes random search as the traditional HPO baseline and motivates the sample-efficiency comparison."},{"cited_title":"Gemini: A family of multimodal models,","cited_arxiv_id":null,"evidence_quote":"Provides the Gemini model family that powers all three agents' reasoning in the reported experiments."},{"cited_title":"Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA,","cited_arxiv_id":null,"evidence_quote":"Represents the distributed AutoML approach that lacks the dynamic inter-agent collaboration the paper claims OptiMindTune adds."}],"review_version":1}