{"id":"47d9ca8c-6245-4d87-a160-bf7cebe04e9a","arxiv_id":"2501.15661","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A probe-then-fit portfolio of five metaheuristics attains the best rank in average test accuracy on 16 benchmarks for probabilistic neural networks, but the test set appears to be used as the training objective.","lead":"The paper proposes a training strategy for probabilistic neural networks that briefly tries five optimization algorithms, then keeps the one with the lowest error rate and gives it a larger budget. A generalist might read it as a simple example of algorithm selection, but the reported gains are small and the evaluation appears to reuse the test set inside the training loop.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported test accuracies are contaminated because the fitness function in Eq. (10) is evaluated on the same test set later used for the ranking in Tables 9-11; the paper never introduces a separate validation set.","rationale":"The reader's weakest assumption identified exactly this issue: the fitness in Eq. (10) may be computed on the held-out test set, making the comparison circular. My reading of the full text strengthens that concern from an assumption to a near-certainty. Section 3.1 says each test sample is evaluated during the fitness count, Algorithm 1 explicitly stops on the test set, and Section 4.1 describes only train/test splits with no validation split. Because the proposed algorithm's two phases both use this fitness to select a metaheuristic and tune smoothing parameters, the reported test set is not unseen. The central claim of superiority over single metaheuristics therefore lacks the required independent evaluation. No new concern beyond the reader's is needed; agreeing with the reader's REJECT verdict is the honest outcome. A rerun with a proper validation set, or an explicit confirmation from the code that the fitness split is disjoint from the evaluation split, would settle the matter; absent that, the empirical support for the headline claim is not credible.","tokens_in":18224,"tokens_out":2816,"duration_ms":29313,"concrete_test":"Rerun the cHM and single-metaheuristic experiments on at least the datasets where cHM wins (e.g., Iris, Cancer, ILPD, Ecoli, Pima, Climate) with a three-way split: a training set, a validation set used for the Eq. (10) fitness in probing/fitting and for the convergence check in Algorithm 1, and the original 20% test set used only once for the final Tables 9-11. Recompute the ranks; if cHM's rank or average margin over the baselines drops materially, the leakage is confirmed. As a cheaper first check, inspect the released code's data partition to determine whether the split consumed by the fitness function is identical to the split reported in Tables 9-11.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that cHM 'overperforms' single metaheuristics in PNN training, supported by the rank advantage in Tables 9-11 for average test accuracy, precision, and recall. That claim depends on the test accuracies being independent of the optimization process. The text indicates they are not. Eq. (10) defines the error-rate fitness as 1 minus correct predictions divided by 'cardinality of test sample.' Section 3.1 states that maxFE counts 'a single evaluation of each test sample of each individual.' Algorithm 1 stops when 'the error rate is equal to 0 on the test set.' Section 4.1 says the data were split into train and test sets and that 'the particular sets were then used to calculate train and test metrics to evaluate the methods on unseen data,' but it never describes a validation set. If the same 20% held-out test set is used both as the fitness signal in probing/fitting and as the evaluation set in Tables 9-11, then the reported test accuracies are not estimates of generalization: they are training-set scores on the test data. The comparison may still tell us which optimizer best fits a particular set of labels, but it no longer supports the conclusion that cHM produces PNNs with superior classification performance on unseen data. This is a correctness risk in the central empirical argument, not merely a missing significance test; the absence of error bars and statistical tests is secondary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the constrained Hybrid Metaheuristic (cHM) algorithm for training Probabilistic Neural Networks (PNNs) by optimizing smoothing parameters. cHM runs a portfolio of five metaheuristics (PSO, BAT, BFO, SA, FPA) in a two-phase procedure: a probing phase evaluates each method on a fixed computational budget and selects the one with the lowest error rate, and a fitting phase continues with the selected method. The procedure is repeated for a fixed number of iterations or until zero error. The authors evaluate cHM on 16 benchmark datasets, comparing average test accuracy, precision, and recall against the five individual metaheuristics and a plug-in baseline. They report that cHM achieves the highest rank on all three metrics and claim that it 'overperforms' single metaheuristics in PNN training.","tokens_in":18541,"tokens_out":7002,"duration_ms":57070,"significance":"If the empirical claims were valid, the paper would make a modest but useful contribution by showing that a simple portfolio-based selection strategy can improve PNN training without gradient-based optimization. The algorithm is clearly described and the evaluation covers a diverse set of datasets. However, the central empirical claim is currently unsupported because the fitness function in Eq. (10) is defined on the test sample and the paper never introduces a validation set. This makes the reported test accuracies the same objective used for model selection and early stopping, i.e., the comparison is partly circular. In addition, the absence of standard deviations or significance tests makes the rank-based superiority claim difficult to assess. The contribution can only be evaluated after a corrected experimental protocol is applied.","major_comments":[{"comment":"The error-rate fitness function in Eq. (10) is defined as 1 minus the number of correct predictions on the 'test sample', and Section 3.1 states that maxFEprobing/fit counts evaluations of each test sample of each individual. Algorithm 1 also stops when 'the error rate is equal to 0 on the test set'. The dataset split in Section 4.1 is described only as train and test sets; no validation set is mentioned. Therefore, the test accuracies reported in Tables 9-11 are computed on data that were used to select the metaheuristic and to tune the smoothing parameters. This makes the central comparison circular and the conclusion that cHM generalizes better unsupported. The authors must either show that the 'test sample' in Eq. (10) was a separate validation set (e.g., obtained by a three-way split or cross-validation) or redo the experiments with a proper validation set and report metrics on a truly held-out test set.","section":"Section 4.1, Eq. (10); Section 3.1; Algorithm 1"},{"comment":"The tables present averages over 10 runs without any measure of dispersion (e.g., standard deviation, confidence intervals) or statistical significance tests. Since the compared methods are stochastic metaheuristics, the observed differences (e.g., cHM accuracy 0.954 vs BAT 0.947 on Cancer) may be within run-to-run variability. The statement that 'a fixed random seed was applied to all stochastic operations' also needs clarification: if the same seed was used for all 10 repetitions, the runs are not independent and the averages are meaningless. Please report the seed policy and provide variance information or significance tests to support the rank-based claims.","section":"Tables 9-11 and Section 4.1"}],"minor_comments":[{"comment":"The 'Rank' definition counts the number of datasets in which a method is among the best (ties counted for each method). This is not a standard rank and the total across methods exceeds the number of datasets; please rename or clarify the metric.","section":"Section 4.3.1"},{"comment":"The claim that 'cHM is roughly N times faster than testing each of the N metaheuristics separately' is not supported by the budget: the probing phase runs all N methods, so the total evaluation count is N*maxFEprobing + maxFEfit, not (maxFEprobing+maxFEfit)/N. Please provide a precise computational-complexity statement.","section":"Conclusions"},{"comment":"Typo: 'Simmulated Annealing' should be 'Simulated Annealing'.","section":"Section 3.2"},{"comment":"In the data description, 'Vecivle' should be 'Vehicle', and the E. coli class-balance column in Table 7 is incomplete ('143/77/52/35/20/').","section":"Section 4.2 and Table 7"},{"comment":"The dot plots in Figures 2 and 4 would benefit from explicit axis labels and a legend explaining the dot sizes; the captions mention that the size indicates the count, but this is not visible in the text.","section":"Figures 1-4"},{"comment":"Some references are incomplete (e.g., [16] 'Vaswani, et al.' has no initials or journal) and some are generic AI papers not cited for a specific claim; a careful reference check is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage issue is, in my view, the main barrier to acceptance. If the authors can provide a corrected experimental protocol with a true validation set and re-run the experiments, the paper may become publishable. The fixed-seed policy also requires clarification. The paper is within the scope of Information Sciences, though the presentation could be more concise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know the central claim of this paper is not supported: the cHM algorithm optimizes the error rate on the test set itself, so the reported test accuracies in Tables 9–11 are training scores, not generalization estimates.\n\nWhat's actually new is the two-phase structure: probe five vanilla metaheuristics (BAT, SA, FPA, BFO, PSO) on a budget, pick the one with lowest error, then spend a larger budget fitting it, with the population carried between iterations. That's a reasonable algorithm-selection idea, and applying it to PNN smoothing parameters is new, even if the components are standard. The paper also deserves credit for a fairly broad 16-dataset comparison, fixed seeds, explicit evaluation budgets, and the selection-frequency plots that show the method doesn't just lock onto one optimizer.\n\nThe soft spot is load-bearing. Eq. (10) defines the error rate as 1 minus correct predictions over the cardinality of the test sample. Section 3.1 says maxFE counts evaluations of each test sample; Algorithm 1 stops when the error rate hits 0 on the test set; Section 4.1 describes only train/test splits, never a validation set. So the fitness used in both probing and fitting is the same held-out test set on which accuracy is later reported. The rank advantage of cHM is therefore partly an artifact of fitting to the test labels. This is not a mere absence of confidence intervals — though those are also missing, since Tables 9–11 give averages over 10 runs with no standard deviations or significance tests. The authors would need to redo the experiments with a separate validation set for selection and tuning, use the test set only once at the end, and report variance.\n\nMinor issues: the introduction is padded with generic AI commentary; the Rank metric is just a count of wins, not a rigorous comparison; and the gains are small and inconsistent (cHM loses on Wine, Vehicle, and Parkinson on some metrics).\n\nWho gets value from this? Researchers working on metaheuristic-based PNN training might find the probe-then-fit construction worth exploring, but the empirical evidence as presented is not usable. I would not cite it, and it is not a good reading-group paper until the leakage is fixed.\n\nRecommendation: desk reject, with a clear note explaining the test-set contamination and what a revision would need. If the authors return with a proper validation split and variance reporting, it could become a modest but useful paper.","headline":"The cHM probe-then-fit idea is reasonable, but the evaluation is circular: Eq. (10) trains on the test set, so the reported accuracy gains are not evidence of generalization.","tokens_in":19048,"tokens_out":3426,"would_cite":false,"duration_ms":30401,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C59"],"pacs":[],"model":"deepseek-v4-flash","headline":"The constrained Hybrid Metaheuristic (cHM), which probes five optimizers and commits to the best one, outperforms each single method in PNN training across 16 datasets.","keywords":["Probabilistic Neural Networks","learning procedure","metaheuristic","hybrid metaheuristic","synergy","smoothing parameters","classification"],"falsifier":"Re-run the cHM protocol with the error-rate fitness computed on a validation split disjoint from the test split, keeping every other setting identical; if cHM no longer accumulates the highest rank on average test accuracy across the 16 datasets, the result depends on using test labels during training.","tokens_in":18054,"feed_emoji":"🧠","tokens_out":8219,"duration_ms":70277,"temperature":0.7,"pith_summary":"Probabilistic neural networks classify by estimating density functions, and their accuracy depends on smoothing parameters that gradient-based training handles poorly. This paper tries to show that a constrained Hybrid Metaheuristic (cHM) can train those parameters better than any of its five component optimizers alone. cHM runs a short probing phase across PSO, BAT, BFO, SA, and FPA, selects the one with the lowest error rate, then lets that optimizer refine the network under a fixed evaluation budget. Across 16 datasets the hybrid ranks first far more often than single methods on average test accuracy, precision, and recall, and it also beats a classical plug-in bandwidth rule on six of the sixteen datasets. If the comparison holds, hybrid selection is a cheap way to get strong PNN performance without knowing in advance which optimizer suits a dataset.","feed_headline":"Hybrid metaheuristic ranks first in PNN training across 16 datasets","feed_subtitle":"Probing five optimizer families and focusing on the best yields higher average test accuracy than any single method.","key_machinery":"The central object is the constrained Hybrid Metaheuristic (cHM) procedure: a two-phase wrapper around a portfolio of five population-based optimizers, PSO, BAT, BFO, SA, and FPA, used to tune the smoothing-parameter vector $h_{III}$ of a PNN built with a product Cauchy kernel. The two phases are probing and fitting, each capped by a maximum number of fitness evaluations ($\\mathrm{maxFE_{probing}}$, $\\mathrm{maxFE_{fit}}$). Probing runs every weak optimizer on an equal budget and keeps the population of the lowest-error-rate method; fitting continues that method alone; the best population is carried into the next cHM iteration. The fitness function is the error rate of Eq. (10), so the mechanism is an adaptive selection-and-concentration strategy over smoothing parameters.","core_discovery":"On the paper's own terms, the central discovery is that combining several population-based metaheuristics into one constrained, two-phase procedure makes PNN training more reliable than any single metaheuristic. In the probing phase cHM gives each of five weak optimizers the same population and the same number of function evaluations, then selects the one with the lowest error rate; in the fitting phase that optimizer continues with the saved population until a second evaluation budget is exhausted. The cycle repeats five times, passing the best population forward. Reported average test accuracy gives cHM a rank of 10 wins across the 16 datasets, versus 3 for the next-best single method, with similar rankings for precision and recall; in the best-accuracy comparison cHM wins 6 datasets against the plug-in method's 5. The paper reads this as evidence that cHM effectively selects and exploits the right optimizer for each dataset and each stage of training.","pith_inferences":["A fair-minded next test would compare cHM against an oracle that, after all runs, picks the best single metaheuristic per dataset; the gap would quantify the cost of online selection and whether probing adds value beyond chance.","The probe-then-fit wrapper is not PNN-specific: it could be transplanted to any kernel density estimation task where a bandwidth parameter must be chosen without gradients, such as anomaly detection or density-based clustering.","Sweeping the probing-to-fitting budget ratio would show how much exploration is needed before committing to an optimizer and could turn cHM into a budget-aware anytime algorithm; the authors list this ratio as future work."],"forward_implications":["A user no longer needs to know in advance which optimizer suits a dataset: cHM's probing phase selects among PSO, BAT, BFO, SA, and FPA under a fixed evaluation budget.","Because one cHM run tests all component methods during probing, it gives roughly an N-fold speedup over running the N single metaheuristics separately, assuming equal per-evaluation costs.","The population of the chosen optimizer is carried into the next cycle, and later cycles can switch to a different optimizer, so the method adapts its search strategy as training proceeds.","Since the rank advantage appears for average test accuracy, precision, and recall alike, the improvement is not specific to one evaluation metric.","The same wrapper can be applied to other PNN smoothing-parameter schemes, such as per-class, per-feature, or matrix forms, without changing the probe-and-fit loop."],"supporting_citations":[{"why":"Defines the probabilistic neural network architecture whose smoothing parameters cHM is designed to train.","marker":"[37]"},{"why":"Earlier study of metaheuristic adjustment of PNN parameters that cHM builds on and benchmarks against.","marker":"[26]"},{"why":"Original particle swarm optimization, one of the five inside optimizers used in the probing phase.","marker":"[55]"},{"why":"BAT algorithm variant specialized to probabilistic neural networks; source of BAT parameters and baseline.","marker":"[57]"},{"why":"Bacterial foraging optimization, one of the five weak metaheuristics in the cHM portfolio.","marker":"[58]"},{"why":"Simulated annealing, one of the five weak metaheuristics; supplies the SA baseline.","marker":"[59]"},{"why":"Flower pollination algorithm, one of the five weak metaheuristics; supplies the FPA baseline.","marker":"[60]"},{"why":"Reflection technique used to keep proposed smoothing parameters inside the nonnegative feasible range.","marker":"[61]"},{"why":"Public benchmark dataset repository supplying most of the 16 datasets used in the comparison.","marker":"[63]"}],"fun_headline_variants":["Two-phase hybrid metaheuristic boosts PNN training reliability","cHM: five optimizers, one winner for PNN smoothing","Probabilistic neural nets learn better with hybrid metaheuristic","Constrained hybrid metaheuristic wins 10 of 16 PNN datasets","Hybrid optimizer selection improves PNN accuracy across 16 datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the error-rate score used to pick the best optimizer is computed on data that were not used to measure the final accuracy; the paper never states that a separate validation set was used, so the reported rankings could be circular if the same test labels guided both selection and reporting.","fun_headline_variants_meta":{"raw":{"variants":["Two-phase hybrid metaheuristic boosts PNN training reliability","cHM: five optimizers, one winner for PNN smoothing","Probabilistic neural nets learn better with hybrid metaheuristic","Constrained hybrid metaheuristic wins 10 of 16 PNN datasets","Hybrid optimizer selection improves PNN accuracy across 16 datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1410,"prompt_tokens":1006,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":314}},"tokens_in":622,"tokens_out":404,"duration_ms":3576,"temperature":1.0,"reasoning_tokens":314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:03:57.974445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the cHM protocol with the error-rate fitness computed on a validation split disjoint from the test split, keeping every other setting identical; if cHM no longer accumulates the highest rank on average test accuracy across the 16 datasets, the result depends on using test labels during training.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Public benchmark dataset repository supplying most of the 16 datasets used in the comparison."},{"cited_title":"Yang, Flower pollination algorithm for global optimization, in: J","cited_arxiv_id":null,"evidence_quote":"Flower pollination algorithm, one of the five weak metaheuristics; supplies the FPA baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the probabilistic neural network architecture whose smoothing parameters cHM is designed to train."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier study of metaheuristic adjustment of PNN parameters that cHM builds on and benchmarks against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BAT algorithm variant specialized to probabilistic neural networks; source of BAT parameters and baseline."},{"cited_title":"Links in Surfaces and Laplacian Modules","cited_arxiv_id":"2002.10040","evidence_quote":"Bacterial foraging optimization, one of the five weak metaheuristics in the cHM portfolio."},{"cited_title":"Kirkpatrick, C","cited_arxiv_id":null,"evidence_quote":"Simulated annealing, one of the five weak metaheuristics; supplies the SA baseline."}],"review_version":1}