{"id":"1fa2c116-3ba0-451b-a185-2b30736cf186","arxiv_id":"2512.19739","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OASI, a simulated-annealing-based, objective-aware initialization for multi-objective Bayesian optimization, improves hypervolume and memory-feasible deployment for TinyML keyword spotting models, though the statistical support is limited.","lead":"This paper proposes a new way to start multi-objective Bayesian optimization for tiny speech-recognition models, seeding it with accuracy-and-memory-aware candidates produced by simulated annealing. The authors report better memory-feasible models on STM32 microcontrollers, but the statistical evidence is weak and the budget comparison is uneven.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OASI's reported advantage may be an artifact of unequal evaluation budgets: its MOSA initialization performs many extra objective evaluations, yet comparisons appear to hold MOBO iterations fixed rather than total trainings.","rationale":"The reader identified the budget-equivalence premise as the weakest assumption, and the manuscript's own Algorithm 1 and timing tables confirm that OASI spends substantially more computation and, more importantly, more objective evaluations during initialization. This is the single most load-bearing concern because the paper's headline result—'always achieving zero generational distance with equal budgets'—depends on the comparison being equal in terms of the expensive evaluations, not just MOBO iterations. The concrete test of fixing total evaluations would settle whether the advantage is intrinsic to objective-aware initialization. The non-significant Kruskal–Wallis result supports the need for such a check. Given that the issue is addressable with a fairer comparison, the conditional verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":7150,"tokens_out":3080,"duration_ms":32903,"concrete_test":"Re-run Table II with a fixed total evaluation budget, e.g., 100 trained models total. For LHS/Random/Sobol use n = 10 initial points + 90 MOBO iterations; for OASI use Nchains × N_iter MOSA evaluations plus MOBO iterations summing to 100, and compare all methods after the same number of trainings (or same wall-clock time). If OASI still yields GD = 0 and higher HV at equal evaluation count, the budget confound is ruled out; if not, the central claim must be qualified as 'given extra initialization evaluations.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract; Section IV, Table II; Conclusion) is that OASI improves MOBO 'under the same budget constraints' and 'without incurring extra optimization costs.' The paper's own numbers contradict this premise: Table II lists OASI total time as 1934.88 s versus 1501.79–1562.11 s for LHS/Random/Sobol, and Table I reports OASI initialization overhead as 1934.9 s versus 1501.8 s for plain MOBO. More fundamentally, Algorithm 1 does not merely select n initial points; it runs Nchains chains of N_iter perturbed candidates, evaluating the objective f on every candidate, storing all evaluations in I_A, and only then selecting D_0. Each evaluation requires full training of a DS-CNN on the training set, so OASI consumes many more expensive model trainings before MOBO begins. If the evaluation budget is measured in number of trained models, the comparison is not equal: OASI gets a much larger search budget to condition the surrogate. The HV/GD advantage, including the headline GD = 0, may reflect this extra search effort rather than objective-aware initialization. The reported Kruskal–Wallis test (H = 5.40, p = 0.144) is non-significant, further weakening confidence that the observed differences are due to the initialization method rather than budget or noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OASI, an initialization strategy for multi-objective Bayesian optimization (MOBO) in TinyML keyword spotting. OASI runs short multi-objective simulated annealing (MOSA) chains before MOBO, collects all evaluated configurations in an archive, and then selects a diverse, Pareto-biased subset to seed the Gaussian-process surrogate. The authors compare OASI-MOBO against NSGA-II, MOSA, and plain MOBO, and compare OASI initialization against LHS, Sobol, and random initialization on a DS-CNN keyword-spotting model. They report that OASI achieves the highest hypervolume and lowest generational distance, including GD=0, and that hardware-in-the-loop deployment on STM32 boards verifies memory-feasible models. The central claim is that OASI improves Pareto-front convergence and stability under the same budget constraints and without extra optimization cost.","tokens_in":7574,"tokens_out":3021,"duration_ms":34338,"significance":"If the claims are sustained, OASI would be a practically useful contribution: low-budget multi-objective optimization for TinyML is genuinely important, and the hardware-in-the-loop evaluation on real STM32 microcontrollers goes beyond simulation-only studies. The paper also makes its code publicly available, which supports reproducibility. However, the central empirical claim is not yet fully supported. The paper's own tables show OASI consumes substantially more wall-clock time than the baselines, so the 'same budget' premise is not met. The only reported statistical test is non-significant (Kruskal-Wallis p=0.144), and HV/GD are reported as single values with no run-to-run variance, so the claimed convergence robustness is not established. The definition of the reference Pareto front used to compute GD=0 is also missing. Given the targeted venue and the practical relevance, these issues are addressable, but they require a substantive revision rather than cosmetic changes.","major_comments":[{"comment":"The budget-equivalence claim is contradicted by the paper's own numbers. Algorithm 1 evaluates f(h) for every MOSA candidate in every chain (line 8), and each evaluation is a full DS-CNN training plus validation. Yet Table I lists OASI-MOBO's overhead as 1934.9 s versus 1501.8 s for plain MOBO, and Table II reports OASI total time 1934.88 s versus 1501.79–1562.11 s for LHS/Random/Sobol. Thus OASI consumes more model trainings and more wall-clock time. The claim in the Abstract and Conclusion that OASI works 'under the same budget constraints' and 'without incurring extra optimization costs' is therefore unsupported. The comparison must be made under the same total number of objective evaluations (trained models), and the exact values of N_chains, N_iter, and the number of evaluations drawn into D_0 must be reported.","section":"§III, Algorithm 1 and §IV, Tables I-II"},{"comment":"The paper reports HV and GD as single numbers per initialization method, with no run-to-run variance, and then states that OASI provides 'reduced inter-run variance' and 'improved convergence stability.' The only statistical evidence, the Kruskal-Wallis test, is non-significant (H=5.40, p=0.144, η²=0.0007). This does not support the central claim that OASI robustly outperforms the alternatives. Please run multiple repetitions with fixed seeds, report distributions of HV/GD (or mean±std), and use an appropriate paired or repeated-measures test across the same evaluation budgets. This is load-bearing because the convergence-robustness advantage is a headline contribution.","section":"§IV, Table II and Kruskal-Wallis test"},{"comment":"The text says GD 'measures convergence to the reference Pareto front' but never defines how that reference front is constructed. A GD of exactly 0 for OASI is either a remarkable result or an artifact: if the reference front is derived from the same runs that include OASI's points, the zero is uninterpretable. Please specify the independent Pareto front (e.g., a merged front from all methods with a non-dominated filter) or a known ground-truth set, and explain how the zero is achieved. Without this, the headline GD=0 cannot be assessed.","section":"§IV, Table II and GD definition"}],"minor_comments":[{"comment":"The column 'Init. Overhead (s)' appears to contain total runtimes (e.g., NSGA-II 1499.9 > its iteration time 1389.5). Rename the column or separate initialization overhead from total time, and make consistent with Table II.","section":"Table I"},{"comment":"Report the actual values of N_chains, N_iter, T0_acc, T0_size, alpha_acc, alpha_size, and the dataset size n. These are free parameters and their values are essential to reproduce the search cost and the results.","section":"Algorithm 1 / Reproducibility"},{"comment":"The indicator I[...] should be written with a clear notation, e.g., mathbb{1} with a condition, and the resource limits B^{(u)}_h should be defined explicitly before the equation.","section":"Eq. (6)"},{"comment":"The text mentions a latency measurement on STM32N6570-DK (0.315 ms) and Neural-ART offloading, but Table III lists only H7, F469, and F401 platforms. Add this board to the table or clarify that it is an additional experiment.","section":"§IV, Hardware-in-the-loop"},{"comment":"The phrase 'objective-aware initialization strategies lead to infeasible (out-of-memory) models' in the contribution list appears to describe the opposite of the result; the baselines lead to OOM, not OASI. Please reword for clarity.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"I agree with the conditional verdict of the reader. The central concern is not circularity but budget accounting: OASI's apparently superior HV/GD may simply reflect extra consumed evaluations and wall-clock time. The Kruskal-Wallis result is non-significant, so the empirical claim currently rests on point estimates. A revision that reruns the comparison under equal total evaluations, reports variances, and clarifies the reference Pareto front would substantially strengthen the paper. The topic and hardware-in-the-loop methodology are appropriate for the venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: this paper proposes something sensible — initialize MOBO with a Pareto-biased archive from a short multi-objective simulated annealing run, then select a diverse subset for the initial surrogate. That is a reasonable thing to try, and the authors actually did the work: public code, real STM32 measurements, and a deployability index that captures the multi-resource bottleneck. The specific combination for TinyML KWS appears new, and the hardware-in-the-loop data, especially the latency difference between models with identical MACC counts, is a genuine contribution.\n\nThe problem is that the central claim — same budget, better HV and GD — is not supported as stated. Algorithm 1 evaluates f on every MOSA candidate and stores them all in I_A; each evaluation is a full model training. Table I lists OASI's initialization overhead at 1934.9 s versus 1501.8 s for plain MOBO, and Table II gives total times of 1934.88 s versus roughly 1502–1562 s for LHS/Random/Sobol. So OASI consumes many more trained models before MOBO even starts. The observed HV/GD advantage can easily be explained by extra search effort rather than better surrogate conditioning. The abstract's claim of 'without incurring extra optimization costs' does not match the paper's own tables.\n\nThe statistics are also weaker than the prose implies. The Kruskal–Wallis test is non-significant (p = 0.144, eta² = 0.0007), and Table II reports HV and GD as single numbers with no variance. An exact GD of 0 is suspicious unless the reference Pareto front is constructed from evaluations that include OASI's own archive — and even then, one would normally see some nonzero distance. The authors do show dispersion elsewhere, but the key comparative result lacks confidence intervals.\n\nWhat is still worthwhile: the OASI concept is not flawed, and the hardware-in-the-loop validation on STM32MCUs demonstrates that memory-feasible models can be deployed. The deployability index is a sensible heuristic, though it is not deeply analyzed.\n\nBottom line: this deserves a serious referee, but the revision bar is substantial. The evaluation budget must be fair — either match the number of trained models or give the baselines an equally expensive initialization. Reporting variance or confidence intervals on HV/GD is essential. If the advantage survives a fairer test, this will be a solid niche contribution.\n\nRegards","headline":"OASI is a plausible method with real hardware validation and public code, but the budget accounting contradicts the 'same budget' claim and the non-significant statistical test leaves the central empirical case unproven.","tokens_in":7996,"tokens_out":2916,"would_cite":false,"duration_ms":34199,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Seeding Bayesian optimization with Pareto-biased points from simulated annealing yields better accuracy-memory trade-offs for TinyML keyword spotting than standard space-filling initializations.","keywords":["multi-objective Bayesian optimization","surrogate initialization","simulated annealing","TinyML","keyword spotting","hypervolume","generational distance","deployment feasibility"],"falsifier":"Run OASI and LHS/Sobol/Random initializations under strictly equal wall-clock time or strictly equal number of trained models; if OASI no longer achieves higher hypervolume and zero generational distance under those conditions, the central claim of objective-aware initialization superiority collapses.","tokens_in":7066,"feed_emoji":"🎯","tokens_out":4668,"duration_ms":42385,"temperature":0.7,"pith_summary":"Multi-objective Bayesian optimization (MOBO) is sensitive to the points used to seed its surrogate model, especially when the evaluation budget is tiny, as in TinyML model design. This paper introduces OASI, which seeds MOBO with candidates generated by short chains of multi-objective simulated annealing (MOSA), biased toward configurations that already balance accuracy against memory footprint. On a keyword-spotting model search under equal budgets, OASI reports the highest hypervolume, zero generational distance, and the best Tchebycheff score among LHS, Sobol, random, and OASI initializations, while also avoiding SRAM-violating configurations. Hardware-in-the-loop runs on STM32 microcontrollers confirm that OASI-selected models are deployable, and the paper adds a Deployability Index to quantify hardware slack. If the equal-budget claim holds, objective-aware initialization is a cheap way to make MOBO feasible in low-budget embedded settings.","feed_headline":"Pareto-biased seeds beat space-filling starts in TinyML MOBO","feed_subtitle":"Simulated-annealing seeding gives higher hypervolume and exact Pareto-front convergence, verified on STM32 hardware.","key_machinery":"Multi-objective simulated annealing (MOSA) chains: short stochastic walks that accept a neighbor with probability 1 when it improves accuracy, and probability exp(−Δ/T) otherwise, with a separate acceptance condition for model size, so accepted points accumulate in a Pareto-biased archive. A maximin selection rule picks a diverse subset of this archive to form the initial dataset D₀ for the Gaussian-process surrogates driven by Expected Hypervolume Improvement (EHVI). The work of the machinery is to replace objective-agnostic space-filling with surrogate conditioning that is already concentrated near feasible accuracy–memory trade-offs.","core_discovery":"On its own terms, the paper establishes that the initial surrogate dataset—not only the acquisition function—controls MOBO's success in low-budget TinyML searches. OASI runs a few short MOSA chains (40–50 iterations each) that accept perturbed hyperparameter configurations only when both accuracy improves and model size improves, or probabilistically otherwise; all evaluated configurations go into an archive, and a maximin rule selects a diverse subset to seed Gaussian-process surrogates. Compared with Latin hypercube, Sobol, and random seeding, this Pareto-biased initialization gives higher hypervolume (0.062748 vs ≤0.059255), a generational distance of exactly zero, and a lower Tchebycheff","pith_inferences":["The paper's equal-budget claim is strained: OASI's total runtime is 1934.9 s versus 1501.8 s for LHS/Random and 1562.1 s for Sobol, so the extra time could itself buy better results. A time-equated comparison would separate the initialization effect from the extra compute.","Because the Kruskal–Wallis test does not reach significance (p = 0.144), the GD = 0 result may be a single-seed artifact; replication across many runs and seeds is needed before relying on the convergence claim.","OASI's MOSA chains evaluate dozens of untrained candidate configurations before the budget — if these evaluations are part of the same total budget, the architecture is really a hybrid optimizer that spends part of the budget on biased exploration, which could be compared directly against other hybrid optimizers rather than pure initializations.","The Deployability Index could be embedded into the acquisition function as a constraint or third objective, not just a post-hoc validation metric, to directly optimize for hardware slack."],"forward_implications":["If OASI provably yields GD = 0 on equal budgets, then surrogate initialization, not acquisition alone, can dominate MOBO convergence in low-budget settings, and other initialization methods could be benchmarked the same way.","The Pareto-biased seeding should transfer to other TinyML objectives (latency, energy) as long as cheap proxy evaluations exist for the MOSA chains.","The Deployability Index gives a quantitative, hardware-specific way to compare optimization results, so model search and deployment validation can be unified.","Avoiding SRAM-violating seeds cuts wasted full training runs, lowering the real cost of search in constrained embedded development.","OASI's benefit should grow as the evaluation budget shrinks, making it relevant for on-device or federated learning where training is costly."],"fun_headline_variants":["OASI: Bias seeds with Pareto fronts to improve TinyML MOBO","Surrogate initialization: Pareto-biased seeds help TinyML MOBO","Better MOBO starts: Pareto-biased seeding beats LHS and Sobol","Pareto-bias initial data improves TinyML Bayesian optimization","Seeding surrogates with Pareto samples improves MOBO in TinyML"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes equal evaluation budgets, but OASI requires about 433 seconds more wall-clock time than the other initializations (1934.9 s vs 1501.8 s); if that extra time corresponds to more objective evaluations or search effort, the observed improvement in hypervolume and generational distance may be a budget effect, not an initialization effect.","fun_headline_variants_meta":{"raw":{"variants":["OASI: Bias seeds with Pareto fronts to improve TinyML MOBO","Surrogate initialization: Pareto-biased seeds help TinyML MOBO","Better MOBO starts: Pareto-biased seeding beats LHS and Sobol","Pareto-bias initial data improves TinyML Bayesian optimization","Seeding surrogates with Pareto samples improves MOBO in TinyML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001161,"raw_usage":{"total_tokens":4644,"prompt_tokens":744,"completion_tokens":3900,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":3806}},"tokens_in":488,"tokens_out":3900,"duration_ms":26103,"temperature":1.0,"reasoning_tokens":3806,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T15:43:32.952336+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OASI and LHS/Sobol/Random initializations under strictly equal wall-clock time or strictly equal number of trained models; if OASI no longer achieves higher hypervolume and zero generational distance under those conditions, the central claim of objective-aware initialization superiority collapses.","supporting_citations":[],"review_version":1}