{"id":"46ea2ede-5129-45da-8d83-1eb36200afcc","arxiv_id":"2504.14126","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Injecting LLM suggestions into Particle Swarm Optimization reduces the number of deep learning training runs needed to reach target accuracy or error in the tested cases.","lead":"This paper combines Particle Swarm Optimization for hyperparameter search with a large language model that suggests better candidate settings during the search. On a benchmark function, an LSTM air-quality task, and a CNN image classification task, the hybrid reaches the target quality with fewer training model calls than plain PSO, though the supporting experiments are small.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 20–60% model-call reduction is not yet attributable to LLM suggestions, because the baseline PSO runs a fixed 10 iterations while the LLM variant stops as soon as it matches the baseline's final cost; no random-replacement or early-stopping control is reported.","rationale":"The paper proposes a plausible hybrid: use an LLM to propose particle positions when PSO stagnates. The Rastrigin experiments show only small gains (1.5–8.5%) and one negative result; the DL experiments are the sole source of the 20–60% headline. The load-bearing question is whether those savings are produced by the LLM's suggestions or by the comparison protocol. I find the reader's concern exactly on target and would sharpen it: the protocol is not 'PSO vs PSO+LLM' under equal stopping rules; it is PSO run for a fixed 10 iterations vs. LLM-PSO stopped at a target taken from the baseline's final cost. Without an early-stopping baseline or a random-replacement control, the reduction is confounded. I would not call this a rejection: the idea is coherent, the reported costs are at least self-consistent (20 calls = 4 iterations × 5 particles; 30–40 calls = 6–8 iterations × 5 particles), and the Rastrigin tables show some directionally positive numbers. But the core quantitative claim is under-supported: n=3 per method, no code or data, no convergence curves, and overlapping confidence intervals. The minimal experiment I propose is cheap and would settle the attribution. Therefore I agree with the CONDITIONAL verdict and would not change it.","tokens_in":15533,"tokens_out":6307,"duration_ms":54308,"concrete_test":"Run the regression case again with 5 particles but log the full PSO-only curve: record the cost after every iteration (calls 5, 10, ..., 50). Compute t* = the first iteration at which PSO-only cost is no greater than 0.1343 (the reported final cost). If t* < 10, then PSO-only itself reaches the target in fewer than 50 calls and the 20-call LLM result is not a fair comparison. Separately, implement a control identical to Algorithm 2 but replace the LLM query with random positions drawn uniformly from the same layer/neuron bounds, and stop under the same 'match PSO final cost' rule. Report the mean and confidence interval over at least 10 seeds for PSO-only, LLM-PSO, and random-replacement PSO. If the random control achieves a similar reduction, the LLM-specific contribution is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim—20–60% fewer deep learning model calls at equal error—rests on an asymmetric stopping protocol. In Section IV-B/C, standard PSO is always run for 10 iterations (50 calls with 5 particles), whereas Algorithm 2 stops as soon as 'the cost matched or surpassed that of PSO alone' (Section IV-B). Thus the comparison is between a fixed-horizon baseline and a target-threshold variant. If standard PSO reaches its own final cost before iteration 10, or if any diversity-increasing replacement (random reinitialization, fresh PSO particles) also reaches that threshold by iteration 4–8, the reported reduction has nothing to do with the LLM. The paper reports no PSO convergence curves (cost vs. call number), no random-replacement control, and no fixed-budget comparison that would rule out early stopping as the source of the saving. The Rastrigin results actually undercut a general convergence-speed claim: with 20 particles ChatGPT-3.5 gave 173.9 mean iterations versus 168.4 for PSO (Tables 2–3), so the LLM does not consistently accelerate PSO; only the DL experiments carry the 20–60% claim, and those are exactly the ones lacking the control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-driven PSO, a hybrid hyperparameter optimization method that replaces poorly performing PSO particles with positions suggested by ChatGPT-3.5 or Llama3. The method is evaluated on the Rastrigin benchmark, on LSTM-based AQI regression, and on CNN-based material classification. The authors report a 20% to 60% reduction in the number of deep learning model calls required to reach a target cost while maintaining comparable accuracy and error rates.","tokens_in":15833,"tokens_out":5923,"duration_ms":44428,"significance":"If the claimed reductions are real, the method would offer a practical way to cut the cost of hyperparameter search in deep learning, and the paper's idea of coupling LLM suggestions with swarm updates is timely and of interest to the optimization and AutoML communities. However, the experimental evidence as presented does not yet support the attribution of the savings to the LLM component, because the comparison protocol differs between the baseline and the proposed method and because key controls are missing. The paper does test on three scenarios, uses two different LLMs, and reports confidence intervals for the deep learning results, which is a useful level of transparency; nevertheless, the central claim of a 20% to 60% reduction in model calls is not yet established.","major_comments":[{"comment":"The reported 20% to 60% reduction in model calls is confounded by an asymmetric stopping protocol. The PSO baseline is always run for a fixed 10 iterations (50 model calls with 5 particles), whereas the LLM-driven variant is stopped as soon as its cost 'matched or surpassed that of PSO alone' (Section IV-B), which in Table 5 requires only 4 iterations (20 calls) for ChatGPT-3.5. This design cannot separate the effect of the LLM suggestions from the effects of early stopping and of the additional diversity introduced by replacing particles. A control in which the worst particles are replaced by random samples or by freshly initialized PSO particles under the same early-stopping rule is needed to attribute the savings to the LLM.","section":"Section IV-B and Tables 5-6"},{"comment":"The Rastrigin results undercut the general convergence-speed claim stated in the abstract and conclusion. For 20 particles with balanced coefficients (c1=c2), standard PSO converges in a mean of 168.4 iterations, while ChatGPT-3.5-driven PSO takes 173.9 iterations (Tables 2 and 3). The 20% to 60% reduction in model calls is observed only in the deep learning experiments, which are the ones lacking the control described above. Without a demonstrated advantage on a standard benchmark, the claim that LLM suggestions accelerate PSO is not generally supported.","section":"Section V-A, Tables 2 and 3"},{"comment":"The statistical support is weak: each method was run only three times, and the 95% confidence intervals overlap substantially. For the classification task, PSO accuracy is 0.8626 with CI (0.8601, 0.8652), while ChatGPT-3.5 and Llama3 give CIs (0.8442, 0.8640) and (0.8480, 0.8612), respectively, so the claim of comparable accuracy is not statistically established. Similarly, the RMSE intervals for the regression task overlap. Additional repetitions or a formal equivalence test are needed before the 'preserving accuracy and error rates' claim can be accepted.","section":"Section V-B and V-C, Tables 5 and 6"},{"comment":"The prompt fed to the LLM includes the current cost values for every particle (the fifth entry of the particle string), and the LLM is asked to output new positions that reduce the cost. Because the LLM sees the exact objective values, the improvement could simply reflect a response to the presented costs rather than a transferable search heuristic. The paper provides no ablation that removes or masks the cost information, so the contribution of the LLM's generative capabilities versus its use of the cost vector remains untested.","section":"Section III-C and Section IV prompt"}],"minor_comments":[{"comment":"The layer velocity entry 'Min = 1, Max = 0.2(MaxLayers - MinLayers)' is ambiguous; it should clarify whether these are the minimum and maximum bounds for the random initialization of velocities or a formula for a velocity range.","section":"Table 1"},{"comment":"The annotations '20 %' and '60 %' on the model-call bars are not defined in the captions; please indicate that they refer to the percentage reduction relative to the PSO baseline.","section":"Figures 6 and 8"},{"comment":"The lesson that 'reducing the initial PSO iterations at each step resulted in earlier convergence' is not supported by Table 7, where the number of iterations is non-monotonic (for example, with 20 particles it moves from 123 to 140 to 147 to 168 to 165 as the initial PSO horizon increases).","section":"Section VI"},{"comment":"The term 'groundbreaking methodology' in the abstract is subjective and should be replaced with a neutral description of the proposed approach.","section":"Abstract and Introduction"},{"comment":"The statement that 'Our method consistently converged within four iterations' is inconsistent with Table 5, where Llama3 uses 6 to 8 iterations in the regression task.","section":"Section VI"},{"comment":"The prompt template lists Npop values 5, 10, 15, 20, 50, or 100, but Table 4 only reports 5, 10, 15, 20 for the deep learning experiments; please reconcile the numbers.","section":"Section IV prompt"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the LLM-PSO hyperparameter paper. The new thing is the specific hybrid: after a few PSO iterations, an LLM (ChatGPT-3.5 or Llama3) suggests replacement positions for the worst particles. That combination is not in the cited prior work, and the authors clearly describe Algorithm 2 and the prompt. The Rastrigin benchmark is more thorough than the rest—10 runs per setting, varying swarm size and c1/c2—though the results are mixed.\n\nThe problem is that the headline claim—20–60% fewer model calls on the LSTM and CNN—does not test the LLM's contribution. In the DL experiments, standard PSO always runs 10 iterations (50 calls with 5 particles), while the LLM variant runs PSO for 2 iterations, queries the LLM, and stops as soon as the cost \"matched or surpassed that of PSO alone.\" That is a fixed-horizon baseline versus an early-stopping variant with a known target. Any diversity-increasing replacement—random reinitialization, fresh PSO particles—could plausibly hit that threshold before iteration 10. There is no random-replacement control and no convergence-curve plot for the DL cases. With three runs per method, the 95% intervals are wide, and no code or data is provided to check the Rastrigin numbers.\n\nThe Rastrigin results themselves undercut the general convergence story: with 20 particles, ChatGPT-3.5 takes 173.9 mean iterations versus 168.4 for PSO alone. The authors acknowledge \"no improvement\" there but still advertise 20–60% overall in the abstract and conclusion. That is an overreach.\n\nThe idea is plausible—LLM suggestions could add diversity and reduce stagnation—but the current evidence does not separate that mechanism from simple early stopping or randomization. A proper control with matched stopping rules and diversity-equivalent replacements would be enough to find out.\n\nThis paper is for researchers curious about LLM-guided metaheuristics; the algorithm description and prompt design are worth reading, but the quantitative claims should not be cited. I would not send it to a serious referee in its current form; the main comparison is flawed and the sample sizes are too small. It needs substantial revision before it is credible.","headline":"Promising hybrid idea, but the headline 20–60% model-call reduction is unsupported because the baseline runs a fixed 10 iterations while the LLM variant stops at the baseline's final cost—no random-replacement control.","tokens_in":16357,"tokens_out":3547,"would_cite":false,"duration_ms":29799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that occasionally replacing PSO's worst particles with LLM-suggested positions cuts deep-learning hyperparameter-tuning model calls by 20–60 percent with no loss in accuracy or error.","keywords":["LLM-driven PSO","hyperparameter tuning","particle swarm optimization","large language models","deep learning optimization","LSTM","CNN classification","Rastrigin function"],"falsifier":"Run the same Algorithm 2 but replace the LLM block with random samples from the search bounds (or reinitialized PSO particles), keeping the same prompt format, early-stopping target (the PSO-only cost), and evaluation budget; if random replacement matches the 20–60 percent reduction in model calls, the LLM-specific contribution is not confirmed.","tokens_in":15364,"feed_emoji":"🤖","tokens_out":5936,"duration_ms":49081,"temperature":0.7,"pith_summary":"This paper claims that a hybrid optimization loop—standard particle swarm optimization (PSO) with occasional suggestions from a large language model—reaches the same deep-learning model quality in fewer model evaluations than PSO alone. In three settings (the Rastrigin benchmark, an LSTM air-quality regression, and a CNN material-classification task), the authors report cutting the number of model calls by 20 to 60 percent while keeping RMSE and accuracy statistically unchanged. The mechanism is selective replacement: after a few PSO iterations, the LLM is asked to propose improved particle positions, and the worst-performing PSO particles are swapped for the best of those suggestions. If the claim holds, the practical consequence is cheaper hyperparameter search for expensive deep models, especially on resource-constrained hardware.","feed_headline":"PSO tuning gets 20–60 percent cheaper with LLM suggestions","feed_subtitle":"Replacing the swarm's worst particles with ChatGPT-3.5 or Llama3 picks reaches the same accuracy in fewer model runs.","key_machinery":"The load-bearing mechanism is LLM-driven particle replacement, Algorithm 2: after initial PSO iterations, the LLM receives a prompt containing particle positions, velocities, and cost values; it returns an equal number of new positions; and the swarm replaces its worst particles with the best suggestions. This introduces externally generated diversity and guidance into the swarm's search while leaving the standard PSO velocity and position update equations intact. The paper's claimed effect—fewer model evaluations at equal cost—is carried by this replacement step.","core_discovery":"On its own terms, the paper's central claim is that injecting LLM-generated candidate positions into a running PSO swarm accelerates convergence without degrading the final model. The proposed algorithm runs standard PSO for a small number of iterations, sends the current particle positions, velocities, and costs to ChatGPT-3.5 or Llama3, receives an equal number of candidate positions, and replaces the worst-performing PSO particles with the best LLM suggestions; this loop repeats until the global best stops changing or the iteration budget is exhausted. In the LSTM regression case, PSO alone needed 50 model calls to reach the target cost while the LLM-enhanced version needed 20 calls with ChatGPT-3.5 and 30–40 with Llama3; in the CNN classification case both LLMs reduced calls from 50 to 20 at comparable accuracy. On the Rastrigin benchmark the savings are smaller (up to about 8.5 percent fewer iterations with Llama3), which the paper reads as evidence the method helps most when model evaluations are expensive.","pith_inferences":["The prompt includes the current cost values of particles. A testable extension is to remove cost values from the prompt; if the savings vanish, the LLM is partly acting as a value-guided searcher rather than a pure generator of candidate architectures.","A random-replacement control under the same early-stopping protocol would isolate how much of the savings comes from the LLM's content versus from the diversity introduced by any replacement; the paper does not run this control.","The regression gains (60 percent with ChatGPT-3.5) are larger than the Rastrigin gains (up to 8.5 percent), which suggests the benefit scales with evaluation cost; one could test this by applying the method to progressively more expensive model families."],"forward_implications":["If the reported savings generalize, LLM queries can substitute for some deep-learning training calls, which matters because a single model evaluation can cost minutes or hours.","The same replacement rule could be applied to other population-based metaheuristics; the paper explicitly suggests genetic-algorithm parent selection and mutation as natural targets.","The method's efficiency claim implies that hyperparameter tuning can move onto resource-constrained settings such as IoT and edge devices, a direction the authors name.","Because the savings differ by LLM (ChatGPT-3.5 saved 60 percent in regression; Llama3 saved 20–40 percent), the choice of LLM and prompt becomes a tunable component of the optimizer's cost.","If the stopping rule is kept fixed, the 20–60 percent reduction in model calls translates directly into a training-budget reduction for comparable accuracy or RMSE."],"supporting_citations":[{"why":"Defines the standard PSO velocity and position update equations that the hybrid method builds on.","marker":"[8]"},{"why":"Provides the PSO-for-CNN-hyperparameter-tuning baseline the paper extends and cites for computational-efficiency claims.","marker":"[16]"},{"why":"Earlier PSO-based deep-learning parameter optimization that supplies the comparison frame for reduction in model calls.","marker":"[20]"},{"why":"Demonstrates LLMs as optimizers generating solutions from previous results and values, the direct intellectual source for prompting the LLM with particle costs.","marker":"[25]"},{"why":"Shows LLMs can outperform random search and Bayesian optimization on small hyperparameter-optimization budgets, supporting the premise that LLM suggestions can accelerate tuning.","marker":"[30]"},{"why":"Supplies the air-quality dataset, sensors, and LSTM prediction setup used in the regression experiments.","marker":"[37]"},{"why":"Documents the Llama3 model family used as one of the two LLMs in the experiments.","marker":"[35]"}],"fun_headline_variants":["LLM-boosted PSO cuts DL tuning costs by up to 60%","ChatGPT-3.5 makes PSO tuning 60% cheaper","LLM suggestions replace weak PSO particles, cutting costs 20-60%","PSO with LLM picks: 20-60% fewer model calls, same accuracy","LLM-accelerated PSO: up to 60% less compute for DL tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedup is measured against PSO-only runs with no control where worst particles are replaced by random or fresh PSO particles under identical early-stopping, so part or all of the savings could come from replacement diversity or the stopping rule rather than from the LLM's suggestions.","fun_headline_variants_meta":{"raw":{"variants":["LLM-boosted PSO cuts DL tuning costs by up to 60%","ChatGPT-3.5 makes PSO tuning 60% cheaper","LLM suggestions replace weak PSO particles, cutting costs 20-60%","PSO with LLM picks: 20-60% fewer model calls, same accuracy","LLM-accelerated PSO: up to 60% less compute for DL tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3674,"prompt_tokens":1079,"completion_tokens":2595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":2486}},"tokens_in":695,"tokens_out":2595,"duration_ms":17421,"temperature":1.0,"reasoning_tokens":2486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:56:05.861448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Algorithm 2 but replace the LLM block with random samples from the search bounds (or reinitialized PSO particles), keeping the same prompt format, early-stopping target (the PSO-only cost), and evaluation budget; if random replacement matches the 20–60 percent reduction in model calls, the LLM-specific contribution is not confirmed.","supporting_citations":[{"cited_title":"Hyper- parameter optimization of convolutional neural network based on par- ticle swarm optimization algorithm,","cited_arxiv_id":null,"evidence_quote":"Provides the PSO-for-CNN-hyperparameter-tuning baseline the paper extends and cites for computational-efficiency claims."},{"cited_title":"Parameters optimization of deep learning models using particle swarm optimization,","cited_arxiv_id":null,"evidence_quote":"Earlier PSO-based deep-learning parameter optimization that supplies the comparison frame for reduction in model calls."},{"cited_title":"Using large language models for hyperparameter optimization,","cited_arxiv_id":null,"evidence_quote":"Shows LLMs can outperform random search and Bayesian optimization on small hyperparameter-optimization budgets, supporting the premise that LLM suggestions can accelerate tuning."},{"cited_title":"Deep learning based multimodal urban air quality prediction and traffic analytics,","cited_arxiv_id":null,"evidence_quote":"Supplies the air-quality dataset, sensors, and LSTM prediction setup used in the regression experiments."}],"review_version":1}