{"id":"01f30ee5-8ef2-4f14-973c-c8a3646eb56c","arxiv_id":"2607.27645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Bayesian-optimization-based wavelength selector produces more consistent and locally stable sugar-content predictions than genetic algorithms or simulated annealing on a small plum NIR dataset.","lead":"This paper tests a Bayesian-optimization method for choosing which near-infrared wavelengths to use when predicting sugar content in plums. The method picks wavelength regions that give consistent, stable predictions across repeated runs, unlike common genetic-algorithm and simulated-annealing baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported RMSE and robustness metrics are computed on the same cross-validation folds used for selection, with n=60 chosen after model inspection; no independent test set is held out, so the claimed improvements may reflect selection bias.","rationale":"The reader identified the lack of an independent test set as the weakest assumption; my analysis agrees. The paper's central claim is empirical, and the reported metrics are computed on the same validation folds used for model selection, with n=60 itself chosen adaptively. These are textbook sources of selection bias, and with only 40 samples the variance of cross-validation estimates is large. The paper's own note in Sec. 5 explicitly concedes that nested validation with an independent test set is needed. While the method design and internal consistency of the surrogate-based search are interesting, the primary evidence for practical usefulness is not yet out-of-sample. No other concern—such as the one-sided one-bit perturbations or the budget comparison with GA/SA—is as central, because even those would need an unbiased test set to be interpretable. Thus the correct verdict remains CONDITIONAL, contingent on an independent validation study with fixed hyperparameters.","tokens_in":10329,"tokens_out":5191,"duration_ms":52776,"concrete_test":"Hold out an independent test set prior to any optimization. Fix n=60 and K=5, and pre-register all hyperparameters including t0. For each of 10 random 30/10 splits of the 40 spectra, run BOCS(SA), GA, and SA on the training split only, using L(x) (Eq. 3) to select a binary subset. Then train PLS with the selected subset on the full training split and evaluate RMSE on the held-out test split. Record test RMSE and the mean one-bit-removal ΔRMSE on the test split. If BOCS's mean test RMSE is not significantly lower than GA/SA, or its ΔRMSE advantage does not reproduce on the test set, the claimed robustness and consistency are in-sample artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that BOCS-based wavelength selection improves prediction accuracy and yields more consistent, locally robust subsets than GA/SA. Evidence is Table I: mean Hamming distance 19.2 vs 25.6/27.1 and mean ΔRMSE under one-bit removal 0.044 vs 0.147/0.219. However, these numbers are all derived from the same K-fold cross-validation objective L(x) (Eq. 3) that the selection procedures optimize. No independent test set is held out; the methods are selected on, and evaluated against, the same validation folds. This is textbook selection bias: the reported RMSE is the minimum over a search on a fixed validation split and is optimistically biased. The paper itself acknowledges this in Sec. 5: 'a more reliable estimate of the generalization performance may require nested validation with an independent test set, which may also clarify the extent of any selection bias in the reported errors.' Compounding this, the wavelength-region resolution n=60 was chosen after inspecting prediction results for n∈{20,30,40,60,75,100} (Sec. 3.1 and 4.1), another layer of data-dependent model selection. With only N=40 spectra, the cross-validation estimates have high variance, and the missing specification of K (number of folds) and t0 (initial dataset size in Algorithm 2) makes the bias impossible to assess. This is the most load-bearing concern because the paper's headline is an empirical generalization claim, and the entire evidence base is in-sample. If the BOCS advantage is an artifact of overfitting to the validation folds, the consistency and local-stability results do not transfer to new samples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formulates wavelength-region selection for near-infrared PLS-based sugar content estimation as a binary black-box optimization problem, proposes a combinatorial Bayesian optimization method (BOCS) with a sparse quadratic surrogate regularized by a horseshoe prior, and uses Thompson sampling with the acquisition function solved by simulated or quantum annealing. Experiments on 40 plum NIR spectra compare BOCS to GA and SA in terms of cross-validated RMSE, run-to-run consistency (mean Hamming distance and per-bit Shannon entropy), and local robustness (mean RMSE increase under one-bit removal). The authors report that BOCS achieves lower and more consistent RMSE than GA and SA, with mean Hamming distance 19.2 vs 25.6/27.1 and mean ΔRMSE under one-bit removal 0.044 vs 0.147/0.219. The paper is clearly written and includes an explicit discussion of limitations, but its central empirical claim rests entirely on cross-validated metrics computed on the same folds used for selection.","tokens_in":10710,"tokens_out":4974,"duration_ms":51435,"significance":"If substantiated with proper out-of-sample evaluation, the work would be a useful application of combinatorial Bayesian optimization to spectral feature selection, and the proposed consistency and local-robustness metrics are a practical addition to the wavelength-selection literature. The formulation is clear, the dataset is publicly referenced, and the authors are candid about the need for nested validation. However, the headline generalization claim is not yet established: all key numbers (Table I, Figs. 2–4) come from the same K-fold cross-validation objective L(x) (Eq. 3) that is optimized, and the resolution n=60 was selected after inspecting prediction results. The paper's strengths—careful algorithmic description, explicit evaluation metrics, and honest discussion—are offset by this selection-bias issue and by missing experimental hyperparameters.","major_comments":[{"comment":"All accuracy and robustness claims (Table I, Figs. 2–4) are computed on the same K-fold cross-validation objective L(x) (Eq. 3) that the selection procedures optimize, and the resolution n=60 was chosen after inspecting prediction results for n∈{20,30,40,60,75,100} (§4.1). The paper acknowledges in Sec. 5 that a more reliable estimate may require nested validation with an independent test set. Because the abstract's claim is a generalization claim, this selection bias is load-bearing. The reported RMSEs are minima over a search on a fixed validation split and are optimistically biased; the local-robustness improvement under one-bit removal is likewise a property of L(x) on those same folds. An independent test set or a nested CV protocol is required to substantiate the claimed advantage.","section":"§4.1, §4.2, §5"},{"comment":"The number of folds K in L(x) and the initial dataset size t0 in Algorithm 2 are never specified. With N=40 spectra, K determines the fold size and the variance of the cross-validated estimate, and t0 controls the amount of data used to fit the initial surrogate. Both are necessary to assess the magnitude of selection bias and to reproduce the experiments. Please report these values and include a sensitivity analysis (e.g., K=5 vs K=10).","section":"§3.2.3, Algorithm 2"},{"comment":"The sample-efficiency comparison is confounded. The text says the GA requires 60 objective evaluations per iteration while BOCS uses one, but the SA baseline also uses 60 evaluations per iteration; the 'SA_1flip' variant uses one evaluation and, according to the text, reaches the same validation loss as BOCS-SA after 200 iterations. Thus the claim of 'fewer objective evaluations' holds only against the GA baseline, not against SA. The advantage over SA lies only in reproducibility and local robustness, and this should be stated explicitly to avoid overclaiming.","section":"§4.2"}],"minor_comments":[{"comment":"The displayed formula for R_i(r,x) is not typeset correctly in the manuscript; the inner norm appears broken. Please fix the equation.","section":"Eq. (1)"},{"comment":"The per-bit Shannon entropy is said to be normalized to [0,1], but the table header reads 'Shannon entropy/# bits', which is ambiguous. Clarify the normalization and the maximum value used.","section":"§3.3, Table I"},{"comment":"The number of repeated runs R used to compute mean Hamming distance and entropy is not stated. Please report R.","section":"§3.3"},{"comment":"Typo: 'consisitent' should be 'consistent'.","section":"Fig. 4 caption"},{"comment":"'advantage system4.1' should be 'Advantage system 4.1'.","section":"§3.2.3"},{"comment":"The total number of BOCS iterations T and the number of independent trials are not explicitly stated; please include these to aid reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application of a known optimization method with no new physics or algorithmic theory, but it is within the journal's applied statistical/machine-learning scope. The main issue is the missing independent test set and the unstated K and t0, which are fixable with additional experiments and reporting. I see no evidence of misconduct; the limitation is openly acknowledged in Sec. 5. After adding a proper out-of-sample evaluation, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean application of BOCS to wavelength-region selection for PLS sugar prediction, and the robustness analysis (run-to-run consistency plus one-bit-removal sensitivity) is the right thing to look at for this problem. But the headline numbers all come from the same cross-validation folds the optimizers were searching over, so the strong claim — that BOCS gives more consistent and locally stable subsets — is not yet supported. The paper knows this; Sec. 5 says nested validation with an independent test set is needed.\n\nWhat's good: the framing is honest. They use the same objective L(x) for GA, SA, and BOCS, give standard errors, and don't oversell the QA part — QA vs SA as the internal optimizer makes no difference here, and they say so. The local robustness metric is genuinely useful for chemometrics: rather than just reporting a single selected mask, they show most one-bit removals barely change RMSE, and the few sensitive regions coincide with the consistently selected bands. That is a real argument for the method's interpretability.\n\nWhere it's soft: N=40 spectra is small. n=60 was chosen after looking at trajectories for six resolutions, which is an extra layer of model selection. K (number of folds) and t0 (initial dataset size) are never stated, so the exact protocol can't be reproduced from the text alone. More importantly, the consistency and robustness metrics are computed on the same folds used for selection. A method that overfits the validation set could look smoother and more reproducible in-sample and transfer badly. That isn't hypothetical — GA and SA directly minimize validation error, and BOCS's surrogate is trained on the same values, so all three are vulnerable. The paper's Sec. 5 admission is accurate.\n\nOne additional observation: they report that random one-flip SA after 200 iterations reached the same validation loss as BOCS. That weakens the sample-efficiency claim; the argument then rests mostly on consistency, which is exactly the metric needing out-of-sample confirmation.\n\nWho it's for: chemometricians using NIR, and people in the quantum-annealing application community who want another BOCS/QA case study. It deserves a serious referee: the method is sensible, the write-up is clear, and the limitation is explicit. My recommendation: send it to review, and have the referee ask for a held-out test set or nested CV, fixed hyperparameters, and preferably a second dataset. If that comes back clean, the robustness claim becomes genuinely interesting.","headline":"Competent BOCS-for-wavelength-selection paper with an honest limitations section, but every headline number is in-sample — gets a serious referee, not a desk reject.","tokens_in":11202,"tokens_out":2407,"would_cite":false,"duration_ms":23439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A combinatorial Bayesian optimizer selects wavelength regions that are more reproducible and locally stable than genetic-algorithm or simulated-annealing selection for sugar-content prediction.","keywords":["wavelength selection","near-infrared spectroscopy","combinatorial Bayesian optimization","partial least squares","Thompson sampling","QUBO","sugar content estimation","horseshoe prior"],"falsifier":"Hold out a test set of plum spectra, select wavelengths using only the training folds, then measure RMSE and one-bit-removal sensitivity on the test set. If BOCS's selected regions show no better test RMSE, or if its one-bit-removal advantage disappears, compared with GA and SA, the robustness claim is refuted.","tokens_in":10215,"feed_emoji":"🍑","tokens_out":5408,"duration_ms":53385,"temperature":0.7,"pith_summary":"The paper tries to establish that wavelength-region selection for near-infrared sugar-content prediction can be cast as a binary black-box optimization problem and solved with combinatorial Bayesian optimization in a way that is more sample-efficient, more reproducible, and more locally robust than direct optimization with a genetic algorithm or simulated annealing. Over repeated runs, the proposed method returns nearly the same spectral bands each time, and removing any one chosen band barely changes the cross-validated prediction error. That matters because a practical method must give domain experts stable, interpretable regions rather than a different answer on every rerun, and because the framework needs very few expensive partial-least-squares evaluations. The authors also show the acquisition step is a quadratic unconstrained binary optimization problem that can be minimized by simulated or quantum annealing, opening the door to quantum-hardware acceleration.","feed_headline":"Bayesian search finds stable wavelength bands for sugar estimates","feed_subtitle":"Repeated runs pick nearly the same bands, and dropping any one selected band barely changes prediction error.","key_machinery":"The machinery is a quadratic surrogate model, f(x)=α0+Σαi xi+Σαij xi xj, with a horseshoe prior on the coefficients that shrinks most terms to zero while letting a few one-body and two-body terms stay large. At each iteration the model's coefficients are sampled from the posterior via Gibbs sampling, and Thompson sampling proposes the next selection vector by minimizing the sampled surrogate. Because the surrogate is quadratic in binary variables, that minimization is a QUBO problem, solvable by simulated annealing or by quantum annealing on Ising hardware. The expensive PLS cross-validation objective is evaluated only once per proposed vector, which is what makes the search sample-efficient","core_discovery":"The central claim is that Bayesian optimization with a sparse quadratic surrogate finds wavelength subsets that lie in broad, flat basins of the prediction-error landscape rather than isolated overfitted minima. Quantitatively, repeated runs give a mean Hamming distance of 19.2±0.4 between selected subsets, versus 25.6±0.5 for GA and 27.1±0.5 for SA, and a mean RMSE increase under one-bit removal of 0.044±0.003, versus 0.147±0.005 and 0.219±0.007. The few bands whose removal does hurt coincide with bands that are consistently selected across runs. The paper interprets this as convergence to smoother regions of the error landscape, and reports that the objective value after 500 iterations imp","pith_inferences":["A direct extension would be to run the same comparison with a held-out test set that is never used in selection or in choosing n; the paper itself notes this is missing, and it would settle how much of the reported stability reflects genuine generalization.","The coincidence of consistently selected bands with high-sensitivity removal bands suggests a practical protocol: use BOCS selection frequencies as a confidence map to propose wavelength bands for chemical assignment, then validate those bands spectroscopically.","Because the surrogate is strictly quadratic, the method can only capture pairwise interactions among wavelength regions; for targets where higher-order spectral interactions matter, the flat-basin advantage may weaken, and a test on a different analyte would show how far the claim extends.","The n=60 resolution was chosen after looking at prediction results across n; an automatic, data-driven resolution choice would make the method fully parameter-free and would test whether the consistency advantage persists at finer resolutions."],"forward_implications":["With BOCS, repeated runs select nearly the same spectral regions, so the bands can be handed to analysts as stable candidates for chemical interpretation rather than stochastic artifacts.","Because one-bit removal barely changes prediction error, the chosen model is resilient to a noisy or failing spectrometer channel at the selected wavelength.","The expensive PLS objective is evaluated only once per proposed subset, meaning the method reaches low RMSE with fewer objective evaluations than the GA baseline.","Since the acquisition step is QUBO, the method is naturally compatible with quantum annealing hardware and should scale to finer wavelength resolution or denser pairwise interactions as annealers improve.","The same surrogate-and-Thompson-sampling recipe transfers to any high-dimensional feature-selection task with an expensive black-box objective and a demand for reproducible subsets."],"fun_headline_variants":["Bayesian search yields stable, consistent bands for sugar estimates","Robust wavelength selection for sugar via Bayesian optimization","Smoother error landscape from Bayesian-chosen wavelength sets","Bayesian method pinpoints reliable sugar-measuring wavelengths"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole comparison assumes that robustness and consistency measured on the same cross-validation folds used to pick wavelengths predict how the method will behave on genuinely new samples; because no independent test set is held out, the reported advantages could partly reflect tuning to the validation set.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian search yields stable, consistent bands for sugar estimates","Robust wavelength selection for sugar via Bayesian optimization","Smoother error landscape from Bayesian-chosen wavelength sets","Bayesian method pinpoints reliable sugar-measuring wavelengths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1631,"prompt_tokens":712,"completion_tokens":919,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":854}},"tokens_in":456,"tokens_out":919,"duration_ms":7843,"temperature":1.0,"reasoning_tokens":854,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:07:00.390270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a test set of plum spectra, select wavelengths using only the training folds, then measure RMSE and one-bit-removal sensitivity on the test set. If BOCS's selected regions show no better test RMSE, or if its one-bit-removal advantage disappears, compared with GA and SA, the robustness claim is refuted.","supporting_citations":[],"review_version":1}