{"id":"7d551309-3104-4e6a-858a-12a33e8e46c9","arxiv_id":"2505.17151","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A bilevel Bayesian optimization strategy pairing EI and UCB acquisition functions improves RoBERTa fine-tuning by up to 2.7 percent on a subset of GLUE tasks, but the results lack statistical validation.","lead":"This paper tests a two-level Bayesian optimization scheme that mixes two standard search rules, EI and UCB, to fine-tune a language model, reporting a 2.7 percent average improvement over plain fine-tuning on four GLUE tasks. The finding could matter if it holds up, because better automated fine-tuning would lower the cost of adapting large language models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim lacks a well-defined bilevel EI-UCB procedure: §5.3 describes only single-level BO with EI, no algorithm or code is given, and §3/§6 even assign EI-UCB to opposite loops. The 2.7% result is therefore not attributable to the claimed method.","rationale":"The reader's weakest assumption correctly identifies the load-bearing gap: the paper's central claim requires a concrete, well-defined bilevel EI-UCB procedure, but no such procedure is specified. My reading of Sections 3, 5.3, and 6 confirms the contradiction: Section 5.3 explicitly describes only a Gaussian process surrogate with Expected Improvement, which is ordinary single-level BO, while Section 3 and Section 6 assign EI and UCB to the inner and outer loops in opposite ways. This is not a matter of taste or a disagreement with the field's consensus; it is an internal inconsistency between the claimed method and the reported experimental setup. Without an algorithm, pseudocode, or runnable code, the reader cannot verify that any bilevel EI-UCB search was actually performed, and Table 1's improvement could plausibly come from a single-level EI search or from noise, especially given the single seed and dev-set-based early stopping and reporting. The paper's own limitations section acknowledges only narrow model coverage and computational cost, not this missing implementation detail. I therefore find no reason to alter the reader's REJECT verdict: the central assertion is unsupported as written. I am not alleging any author misconduct; the issue is that the evidence presented does not establish the claimed contribution.","tokens_in":7366,"tokens_out":3541,"duration_ms":30005,"concrete_test":"Obtain or implement the exact pipeline from §5.3 and run two variants on the same four GLUE tasks with identical search budget and seeds: (A) the described procedure as written, a GP with EI, 50 trials, early stopping on dev; (B) a true bilevel loop with EI in one level and UCB in the other, plus SWA fusion. If variant A reproduces the EI-UCB row (76.82), or if variant B does not beat variant A by more than seed-to-seed variation, then the reported 2.7% gain is not evidence for the bilevel acquisition-pairing claim. The decisive check is the presence of a nested loop in the runnable code: without two acquisition functions operating at two levels, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires that a true bilevel BO procedure with EI and UCB in separate nested loops was run and beat standard fine-tuning. That procedure is never defined. Section 3.3 gives only the generic formulation min_θ F(θ, φ(θ)) with φ(θ) = argmin_φ G(θ, φ); it never states which acquisition function is used at which level or how the GP is shared between loops. Sections 3.1-3.2 and 6 disagree on the assignment: §3.1 says EI is applied to the outer loop and UCB to the inner loop, while §6 defines EI-UCB as EI in the inner loop and UCB in the outer loop. Section 5.3, the only concrete experiment description, says: We use a Gaussian Process surrogate with Expected Improvement as the acquisition function, running 50 trials. That is standard single-level BO with EI; there is no inner loop, no UCB, and no SWA or model-fusion step anywhere in the setup. Figure 2 is a schematic, not an algorithm. If the runs were actually single-level EI, then Table 1's EI-UCB row (76.82 vs. 74.80, 2.70%) is mislabeled and the claimed contribution, pairing acquisition functions across nested loops, is unsupported. This is an internal inconsistency, not a disagreement with consensus: the paper's own experimental section contradicts its headline method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Bilevel-BO-SWA, a bilevel Bayesian optimization framework for fine-tuning RoBERTa-base on GLUE tasks, in which different acquisition functions (Expected Improvement and Upper Confidence Bound) are assigned to inner and outer optimization loops, combined with stochastic weight averaging. The central empirical claim is that the EI-UCB configuration achieves an average GLUE score of 76.82 versus 74.80 for standard fine-tuning, an improvement of 2.7%, reported in the abstract, Section 6, and Table 1. The paper does not provide pseudocode, an algorithmic description of the nested loops, or a description of how SWA is integrated.","tokens_in":7619,"tokens_out":3417,"duration_ms":28292,"significance":"If the described method were well defined and the experiments reproducible, the question of how acquisition functions should be paired across nested optimization loops for language-model fine-tuning would be a legitimate and reasonably interesting contribution. However, as written, the paper does not define the claimed bilevel EI-UCB procedure, does not describe the SWA component, presents results from a single fixed seed without error bars, and contains direct contradictions between the method description and the experimental setup. These problems concern the central claim rather than presentation, so the paper in its current form does not provide evidence for its headline result.","major_comments":[{"comment":"The assignment of acquisition functions to loops is internally inconsistent. Section 3.1 states that EI is applied to the outer optimization loop and Section 3.2 states that UCB is suitable for the inner loop, while Section 6 defines 'EI-UCB' as EI in the inner loop and UCB in the outer loop (and 'UCB-EI' as the reverse). The paper never resolves this contradiction or states which assignment was actually used in the reported experiments.","section":"Section 3 vs. Section 6"},{"comment":"The only concrete experimental description says: 'We use a Gaussian Process surrogate with Expected Improvement as the acquisition function, running 50 trials.' This is a standard single-level BO setup; there is no inner loop, no UCB acquisition function, no bilevel structure, and no SWA or model-fusion step. Consequently, Table 1's rows labeled UCB-EI and EI-UCB cannot be attributed to the proposed bilevel method, and the central claim that pairing EI and UCB across nested loops improves fine-tuning is unsupported by the described experiments.","section":"Section 5.3"},{"comment":"The name 'Bilevel-BO-SWA' promises stochastic weight averaging as a model-fusion component, but SWA is never formally defined or described anywhere in the manuscript. There is no equation, algorithm, or experimental detail showing how SWA is combined with the Bayesian optimization procedure, so the experiments are not reproducible from the text.","section":"Section 4 and throughout"},{"comment":"All runs use a fixed random seed and no variance or confidence information is reported. Many of the differences between configurations are small (e.g., EI at 75.52 versus UCB-EI at 75.45, or RTE values of 70.8 versus 70.9), and without multiple seeds or error bars it is impossible to know whether the reported 2.7% improvement over fine-tuning is statistically meaningful. In addition, the 'single-level' baseline row is not described anywhere in the experimental setup.","section":"Section 6, Table 1"}],"minor_comments":[{"comment":"The text contains numerous grammatical and typographical errors, for example 'down stream tasks Model gets messy' in the abstract and 'Bilievel-BO' in the keywords; the manuscript would benefit from careful proofreading.","section":"Abstract and Section 1"},{"comment":"The 'Imp. Rate (%)' column is computed inconsistently: for the single-level row, 1.18 is the absolute improvement in average score, while for EI-UCB, 2.70 is the relative percentage improvement; the column should use one consistent definition.","section":"Table 1"},{"comment":"The reference list contains duplicates (Devlin et al. appears as [8] and [13]; Wang et al. as [11] and [17]; Snoek et al. as [9] and [21]) and reference [6] is cited for the UCB formula but is actually a paper on predictive entropy search; the citations need to be cleaned up.","section":"References"},{"comment":"The caption says 'Cumulative Maximum Accuracy Comparison,' but the surrounding text says the figure tracks loss and accuracy across trials; the figure itself is not described in enough detail to interpret the convergence claim.","section":"Figure 3"},{"comment":"The sentence 'RditionalBO relies on acquisition functions' appears to be a typo for 'Traditional BO'; in addition, the claim that SWA is less effective in NLP 'due to differences in loss functions like cross-entropy in NLP versus softmax loss used in vision tasks' is not substantiated by a citation or analysis.","section":"Section 2"}],"recommendation":"reject","confidential_remarks":"The internal contradiction between Section 3, Section 6, and Section 5.3 is not a matter of presentation; it means the manuscript does not actually describe the method it evaluates. The authors would need to define the algorithm, provide code or pseudocode, and rerun experiments with multiple seeds to make the central claim credible. Given the current state, I cannot recommend acceptance even with major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper is not a coherent report of a bilevel EI-UCB procedure. The central empirical result—76.82 average on four GLUE tasks versus 74.80 for fine-tuning—is real in the sense that the table exists, but the method that supposedly produced it is never pinned down, and the experimental description contradicts the headline claim.\n\nWhat is genuinely new: the idea of pairing EI and UCB in separate nested loops for LLM hyperparameter search is a reasonable extension of standard BO. The paper also compares several configurations (EI, UCB, EI-UCB, UCB-EI, single-level) on a small GLUE subset, which is a useful check on whether acquisition-function choice matters at all. That is the extent of the credit.\n\nThe soft spots are load-bearing. Section 3.1 says EI goes in the outer loop and UCB in the inner loop; Section 3.2 says UCB is suitable for the inner loop; Section 4 says the inner-EI/outer-UCB configuration performs best; Section 6 defines \"EI-UCB\" as EI in the inner loop and UCB in the outer loop. Section 5.3, the only concrete setup, says: \"We use a Gaussian Process surrogate with Expected Improvement as the acquisition function, running 50 trials.\" No inner loop, no UCB, no SWA. SWA is in the title and abstract but never described as a fusion step anywhere in the experiments. There is no algorithm or pseudocode and no code release. Results are from a single seed with no error bars, and early stopping uses the dev set that is then reported as the outcome—an optimistic-bias risk the paper does not address. The limitations section is honestly worded, but it concedes that traditional grid search may beat BO when only a few hyperparameters are tuned, which further undercuts the claimed advantage.\n\nWho this is for: a reader interested in the broad question of acquisition-function selection in BO for NLP fine-tuning could skim the comparative table for inspiration. As a contribution, though, it does not hold together; a serious referee would have to reconstruct the method from contradictory text, which is not an acceptable starting point.\n\nI would desk-reject this one. The underlying question is worth a better paper, but this manuscript is not ready for review.","headline":"The paper reports a 2.7% GLUE gain from a bilevel EI-UCB BO scheme, but the method is never coherently described: Section 5.3 describes only single-level BO with EI, and Sections 3, 4, and 6 disagree on which acquisition function sits in which loop.","tokens_in":8200,"tokens_out":2658,"would_cite":false,"duration_ms":21120,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairing EI and UCB acquisition functions in nested Bayesian optimization loops improves language-model fine-tuning by 2.7 percent.","keywords":["Bayesian optimization","acquisition function","bilevel optimization","language model fine-tuning","model fusion","GLUE benchmark","RoBERTa","hyperparameter optimization"],"falsifier":"Re-run the four GLUE tasks with a single-level GP-EI search (as Section 5.3 describes) and compare the average with the reported 76.82; if single-level EI reaches the same score, the benefit attributed to the EI-UCB pairing disappears. Alternatively, inspect the implementation to confirm that two nested acquisition-function loops exist and that the reported numbers come from that procedure.","tokens_in":7081,"feed_emoji":"🤖","tokens_out":6808,"duration_ms":48202,"temperature":0.7,"pith_summary":"The paper tries to establish that the choice and arrangement of acquisition functions inside a bilevel Bayesian optimization loop materially change how well a fine-tuned language model generalizes. It claims that pairing Expected Improvement (EI) in the inner loop with Upper Confidence Bound (UCB) in the outer loop yields an average GLUE score of 76.82 for RoBERTa-base, beating standard fine-tuning's 74.80 by 2.7 percent. A sympathetic reader would care because fine-tuning is expensive and hyperparameter choice is usually treated as a generic search problem, not as a design decision about exploration versus exploitation. If the pairing claim holds, it gives practitioners a concrete, low-cost lever for improving fine-tuning without changing model architecture.","feed_headline":"EI-UCB pairing lifts RoBERTa fine-tuning by 2.7 percent","feed_subtitle":"On four GLUE tasks, inner EI + outer UCB scores 76.82 versus 74.80 for standard fine-tuning.","key_machinery":"The load-bearing mechanism is a bilevel Bayesian optimization loop in which two different acquisition functions are assigned to two nested levels: an inner loop that minimizes training loss and an outer loop that optimizes validation metric. Expected Improvement (EI), which prefers regions with high improvement potential, and Upper Confidence Bound (UCB), which adds an uncertainty term $\\kappa\\sigma(\\theta)$ to the predicted mean, are the two acquisition functions combined into configurations such as EI-UCB. A Gaussian Process surrogate models the validation-performance landscape, and the nested structure is meant to let the inner loop refine model parameters while the outer loop gathers better hyperparameters, with the acquisition-function pairing determining how exploration and exploitation are balanced at each level. This machinery carries the argument because the reported gains are attributed specifically to the interaction of EI and UCB in the two loops.","core_discovery":"The central claim, stated on the paper's own terms, is that Bilevel-BO-SWA—a framework that couples model fusion with bilevel Bayesian optimization—improves language-model fine-tuning, and that the configuration using EI in the inner optimization loop and UCB in the outer loop achieves the best results: an average score of 76.82 on RTE, MRPC, CoLA, and STS-B, 2.7 percent above standard fine-tuning. The paper further claims that this EI-UCB arrangement also produces the lowest average loss (0.76) across the four tasks, and that the search space of the best configuration is 'notably larger' than alternatives, which it ties to more thorough exploration. The authors present this as validation that acquisition-function design, not just the choice of hyperparameters, is a meaningful axis of optimization for large language models.","pith_inferences":["The paper does not report standard deviations or multiple seeds, so a natural next test is whether the 2.7 percent gap is stable across random seeds; the fixed-seed setup leaves the improvement vulnerable to noise.","The description in Section 5.3 of a standard GP-EI search sits uneasily with the bilevel EI-UCB narrative; if the implementation is in fact single-level, the paper's contribution would reduce to a conventional BO fine-tuning study with a different framing.","The SWA component is introduced but not isolated; a reader could test whether the gain comes from weight averaging rather than from the acquisition-function pairing by running EI-UCB without SWA.","The authors themselves note (Section 7) that only RoBERTa was tested and that the overhead of a GP model can exceed the benefit when only one or two hyperparameters matter; both caveats bound the scope of the 2.7 percent claim."],"forward_implications":["If the central claim is right, then acquisition-function pairing is a transferable design choice for fine-tuning other transformer models.","The 2.7 percent gain implies that replacing a single acquisition function with a complementary pair could improve downstream task performance at no architectural cost.","The bilevel scheme's success on four GLUE tasks suggests it is worth testing on the full nine-task GLUE suite and on larger models.","Because the best configuration also has the lowest average loss, the pairing may improve both optimization and generalization simultaneously."],"supporting_citations":[{"why":"Supplies RoBERTa-base, the model on which all reported fine-tuning experiments are run.","marker":"[14]"},{"why":"Defines the GLUE benchmark and its task metrics, used for the 76.82 average comparison.","marker":"[11]"},{"why":"Provides the standard Bayesian optimization machinery (GP surrogate and acquisition functions) that the paper extends to a bilevel setting.","marker":"[21]"},{"why":"Introduces Stochastic Weight Averaging, the model-fusion technique the framework is built around.","marker":"[12]"},{"why":"The prior model-fusion-via-BO approach that this work positions against and aims to improve.","marker":"[1]"},{"why":"Gives the general bilevel optimization formulation that motivates the nested inner/outer loop structure.","marker":"[3]"},{"why":"Is the source for the UCB acquisition function formula used in the outer loop.","marker":"[6]"}],"fun_headline_variants":["Bilevel BO with EI-UCB boosts RoBERTa by 2.7%","Acquisition fusion: EI inner, UCB outer lifts GLUE score","Nested loops: EI+UCB fine-tunes LLMs 2.7% better","Bilevel optimization picks acquisition functions for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumption that the experiments actually ran a two-level optimization with EI in one loop and UCB in the other, rather than a single standard Bayesian optimization search with one acquisition function.","fun_headline_variants_meta":{"raw":{"variants":["Bilevel BO with EI-UCB boosts RoBERTa by 2.7%","Acquisition fusion: EI inner, UCB outer lifts GLUE score","Nested loops: EI+UCB fine-tunes LLMs 2.7% better","Bilevel optimization picks acquisition functions for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2395,"prompt_tokens":931,"completion_tokens":1464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1381}},"tokens_in":547,"tokens_out":1464,"duration_ms":9138,"temperature":1.0,"reasoning_tokens":1381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:59:31.889906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four GLUE tasks with a single-level GP-EI search (as Section 5.3 describes) and compare the average with the reported 76.82; if single-level EI reaches the same score, the benefit attributed to the EI-UCB pairing disappears. Alternatively, inspect the implementation to confirm that two nested acquisition-function loops exist and that the reported numbers come from that procedure.","supporting_citations":[{"cited_title":"In: International Conference on Learning Representations (ICLR)(2019)","cited_arxiv_id":null,"evidence_quote":"Defines the GLUE benchmark and its task metrics, used for the 76.82 average comparison."},{"cited_title":"P.: Practical Bayesian Optimization of Machine Learning Algorithms","cited_arxiv_id":null,"evidence_quote":"Provides the standard Bayesian optimization machinery (GP surrogate and acquisition functions) that the paper extends to a bilevel setting."},{"cited_title":"In:Conference on Uncertainty in Artificial Intelligence (UAI)(2018)","cited_arxiv_id":null,"evidence_quote":"Introduces Stochastic Weight Averaging, the model-fusion technique the framework is built around."},{"cited_title":"Model Fusion through Bayesian Optimization in Language Model Fine-Tuning","cited_arxiv_id":"2411.06710","evidence_quote":"The prior model-fusion-via-BO approach that this work positions against and aims to improve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the general bilevel optimization formulation that motivates the nested inner/outer loop structure."},{"cited_title":"Predictive Entropy Search for Multi-objective Bayesian Optimization","cited_arxiv_id":"1511.05467","evidence_quote":"Is the source for the UCB acquisition function formula used in the outer loop."}],"review_version":1}