{"id":"696e99e2-a7fa-477a-a0ab-b22cfef0fe02","arxiv_id":"2502.04206","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper reviews methods that cast hyperparameter selection as multiple hypothesis testing to deliver formal risk guarantees.","lead":"This paper is a tutorial review of the Learn-Then-Test (LTT) framework, which selects AI hyperparameters by treating each candidate as a statistical hypothesis and keeping only those with formal guarantees. It summarizes extensions for quantile risks, multi-objective settings, prior knowledge, and adaptive testing, aimed at reliable AI deployment.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The null hypothesis is reversed: Eq. (1) defines Hλ: R(λ) < α, but Eq. (3)'s p-value and Eq. (6)'s guarantee are only coherent for Hλ: R(λ) ≥ α; this invalidates the formal FWER/FDR claim as written.","rationale":"The reader's weakest assumption targeted the external validity of cited p-value and e-value bounds for quantile and mutual-information risks. That is a legitimate concern, but it is about material the paper reviews rather than about the paper's own formalization. The more load-bearing issue is internal: the definition of the null hypothesis in Eq. (1) is the opposite of what the p-value in Eq. (3), the guarantee in Eq. (6), and the e-value in Eq. (13) presuppose. Since the paper's central claim is that practitioners can certify hyperparameter subsets via FWER/FDR control, an inconsistency in the very null hypothesis invalidates the formal argument as written, regardless of whether the cited bounds are correct. The deterministic example in the concrete test shows the failure immediately and unambiguously. I therefore agree with the conditional verdict, but for a different primary reason than the reader. The verdict remains CONDITIONAL rather than REJECT because the necessary correction is a typo-level sign fix plus consistency checks across equations, and the underlying LTT literature is independently established. No ad hominem is intended; this is purely a statement about the manuscript's internal coherence.","tokens_in":8986,"tokens_out":5366,"duration_ms":54840,"concrete_test":"Run the following analytic check: set rλ(Z) ≡ 0.4 for all Z, α = 0.5, and n = |Z| arbitrary. Under Eq. (1), H0: R(λ) < α is true. Eq. (3) yields pλ = exp(−2n(0.1)^2) < 1, a constant. The validity requirement for a p-value is P_H0(pλ ≤ u) ≤ u for all u ∈ [0,1]; taking u = pλ gives P_H0(pλ ≤ pλ) = 1 > pλ, so Eq. (3) is not a valid p-value for the null as defined. Re-run this check after changing Eq. (1) to Hλ: R(λ) ≥ α; the same computation then satisfies P_H0(R̂ < α − t) ≤ e^{−2nt^2}. If the paper's Eq. (6) is also changed to require R(λ) < α, the guarantee becomes coherent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that LTT-based selection returns a subset with FWER/FDR guarantees. That claim depends on every null hypothesis being paired with a valid p-value. As written, the paper is internally inconsistent about what the null hypothesis is. Section II defines Hλ : R(λ) < α (Eq. 1) and calls this 'unreliable', but the following paragraph defines an unreliable configuration as one violating (1), i.e. R(λ) ≥ α. The p-value in Eq. (3) is derived from Hoeffding's inequality and is small when the empirical risk is below α; this is valid only if the null is R(λ) ≥ α. If Eq. (1) is taken literally, Eq. (3) fails the defining validity property P_H0(pλ ≤ u) ≤ u. Eq. (13) uses Hλ : R(λ_i) > α, and Eq. (6) states the guarantee with R(λ) ≤ α while 'reliable' was defined as R(λ) < α. These are not cosmetic: a practitioner using Eq. (1) would reject precisely the configurations that are most consistent with the null, and the claimed control of unreliable configurations would not hold. The fix is simple (state Hλ : R(λ) ≥ α and keep strict/non-strict thresholds consistent), but until applied, the review's central guarantee is not supported by its own equations. The validity of quantile/MI p-values cited from [2] and [3] is a separate, external concern; this internal sign error is more immediately load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This tutorial-style paper reviews the Learn-Then-Test (LTT) framework for hyperparameter selection, in which candidate hyperparameters are cast as multiple hypothesis tests and p-values or e-values computed on a held-out calibration set are fed into a multiple-hypothesis-testing procedure. The paper claims that the output subset of hyperparameters comes with formal FWER or FDR guarantees on population risk measures. It surveys extensions to quantile risk, mutual-information risk, multi-objective Pareto testing, side information encoded as reliability graphs, and adaptive e-value-based testing, and it illustrates the methods with communication-system applications. The central claim is that practitioners can certify hyperparameter choices with statistical reliability guarantees rather than relying only on empirical validation.","tokens_in":9289,"tokens_out":6210,"duration_ms":67434,"significance":"If the formal inconsistencies identified below are corrected, this would be a useful compact introduction to an emerging area, with a clear tabular comparison and concrete engineering examples. The paper's strengths are its organizing Table I, the reproduction of the core LTT equations, and the connection to wireless-system use cases. However, the central formal guarantee is not currently supported by the paper's own definitions: the null hypothesis in Eq. (1) has the opposite direction from the p-value it is paired with and from the guarantees stated in Eqs. (6) and (9). Because this sign error propagates to the reviewed extensions, the manuscript needs substantial revision before its main claim can be accepted. The tutorial value is real, but the load-bearing formalism must be made internally consistent.","major_comments":[{"comment":"The null hypothesis is defined as Hλ: R(λ) < α and is called \"unreliable\". This is the reverse of the standard LTT convention and is inconsistent with the paper's own subsequent equations. Under the stated null, rejecting Hλ should be interpreted as evidence that R(λ) ≥ α, i.e., that λ is unreliable, but the text says rejecting means deeming λ reliable. Furthermore, the p-value in Eq. (3) is small when the empirical risk R̂(λ|Z) is below α; for the null Hλ: R(λ) ≥ α, Hoeffding's inequality gives P_{H0}(pλ ≤ u) ≤ u, so Eq. (3) is a valid p-value only for that null, not for Hλ: R(λ) < α. As a result, the FWER/FDR guarantees in Section IV do not follow from the stated testing procedure. This is the load-bearing formal claim of the paper and must be corrected, e.g., by defining Hλ: R(λ) ≥ α, and propagated consistently throughout.","section":"Section II, Eq. (1)"},{"comment":"Even after fixing Eq. (1), the threshold conventions are inconsistent. Eq. (6) states Pr(R(λ) ≤ α for all λ ∈ Λ̂Z) ≥ 1 − δ, while Eq. (9) defines an unreliable hyperparameter as one with R(λ) > α, and Eq. (13) tests Hλi: R(λi) > α. The paper should adopt a single convention for \"reliable\" (e.g., R(λ) ≤ α) and use it consistently in Eq. (1), Eq. (6), Eq. (9), and Eq. (13). With the current mixture of strict and non-strict inequalities, the claimed control of \"no unreliable configurations\" is ambiguous.","section":"Section IV, Eqs. (6) and (9)"},{"comment":"The paper asserts that the works cited as [2] and [3] provide valid p-values for quantile and mutual-information risks, but it does not state the direction of the null hypothesis for these p-values or the validity inequality that they satisfy. Because the same sign error in Eq. (1) propagates to these extensions, the reader cannot verify that the FWER guarantee holds for QLTT and IB-MHT. Please include, at least for each risk measure, the null being tested and the p-value validity condition (P_{H0}(p ≤ u) ≤ u), or explicitly state that this is inherited from the cited works.","section":"Section III, QLTT and IB-MHT"}],"minor_comments":[{"comment":"The notation in Eq. (3) is inconsistent: the exponent uses |Z| in the second occurrence of the equation in the manuscript but |Ẑ| in the first occurrence, while the data set is defined as Z. Please make the notation uniform.","section":"Section II, Eq. (3)"},{"comment":"The block diagram for LTT-based hyperparameter selection appears twice, once as Fig. 1 and once as Fig. 4, with identical content. One of the two figures should be removed.","section":"Figures 1 and 4"},{"comment":"The sentence following Eq. (5), \"controlling the q-quantile risk ... guarantees that the worst-case risk among the top-q fraction of test instances is no smaller than α,\" is confusing and appears to describe the complementary fraction. Please rephrase it in terms of the fraction 1 − q of instances whose risk lies below α.","section":"Section III, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The paper is largely a digest of the authors' own prior work: QLTT [2], IB-MHT [3], and RG-PT [5] are by Farzaneh and Simeone, and aLTT [6] is by Zecchin and Simeone. The only externally sourced core framework is LTT [1] and PT [4]. This is not by itself a reason to reject, but the editor should consider whether the review provides a balanced treatment of the external literature or functions mainly as an advertisement for the authors' extensions. The central null-hypothesis sign error in Eq. (1) is the critical issue to fix before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a review/tutorial, not a new research contribution. There are no new theorems, algorithms, or experiments. What it does well is organize the growing LTT literature into a single picture: the core framework from Angelopoulos et al., plus extensions to quantile risk, mutual information, multi-objective control, reliability graphs, and adaptive e-value testing. The comparison table in Section I is genuinely helpful, and the expository structure makes it easy to see how the extensions relate. A practitioner looking for a map of this niche would get value from the paper.\n\nThe problem is in the formal core. Section II defines the null hypothesis as H_λ: R(λ) < α, calling that state unreliable. But then the p-value in Eq. (3) is derived from Hoeffding and is small when the empirical risk is below α, which is only valid if the null is R(λ) ≥ α. The guarantee in Eq. (6) also concerns R(λ) ≤ α, and the aLTT section in Eq. (13) explicitly uses H: R(λ) > α. So the paper contradicts itself about what the null is. This is not cosmetic. A reader following Eq. (1) literally would compute p-values that are valid for the opposite hypothesis, and the advertised FWER/FDR control would not follow. The stress-test note is right: this is a load-bearing internal inconsistency.\n\nThe soft spots beyond that are milder. The title says “Advances,” but the paper is a review and the extensions are mostly the authors’ own prior work, described without independent reproduction. That is normal for a tutorial, but it does limit critical distance. Figure 2 uses a different calibration data set without reporting it, and there is no code or data release. None of these would sink a well-formed survey, but they matter given the paper’s purpose.\n\nWho is this for? Someone who wants a compact overview of LTT and its variants, and who is willing to check the original papers before relying on the formulas. As written, I would not trust the equations without fixing the null definition first. A corrected version would deserve a serious referee; the current one does too, but with a clear request for major revision.\n\nRecommendation: engage with it, but only after asking the authors to fix the null hypothesis in Eq. (1) and to reconcile the strict/non-strict thresholds. It should not be accepted as is.","headline":"A useful but flawed survey of LTT-based hyperparameter selection whose central null hypothesis is defined backwards; with the sign error fixed it would be a solid tutorial, but as written the guarantees do not follow.","tokens_in":9866,"tokens_out":3110,"would_cite":false,"duration_ms":33417,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Statistical testing turns hyperparameter selection into a reliability-certified procedure.","keywords":["hyperparameter selection","Learn-Then-Test","multiple hypothesis testing","family-wise error rate","false discovery rate","e-values","risk control","reliability"],"falsifier":"Feed an unbounded risk function into the Hoeffding-based p-value (3) on i.i.d. data and count the fraction of selected configurations whose true risk exceeds $\\alpha$; if the empirical family-wise error exceeds $\\delta$ at the nominal level, the tested bound is invalid and the guarantee collapses.","tokens_in":8749,"feed_emoji":"🎯","tokens_out":5212,"duration_ms":43794,"temperature":0.7,"pith_summary":"Hyperparameter selection is usually a heuristic search, but this paper reviews a statistical reformulation: treat each candidate hyperparameter as a null hypothesis to be tested and return only the configurations that pass a multiple-testing filter. The Learn-Then-Test (LTT) framework promises that the returned set contains no unreliable configuration with probability at least $1-\\delta$ (family-wise error control), or only a controlled fraction of unreliable ones (false discovery rate control). The paper shows how this guarantee extends to quantile risks, mutual information risks, multi-objective constraints, prior-knowledge graphs, and adaptive testing with e-values. If right, practitioners can certify hyperparameters for safety-critical deployments rather than rely on empirical validation alone.","feed_headline":"A testing framework certifies hyperparameters with controlled errors","feed_subtitle":"FWER and FDR guarantees extend to quantile, mutual-information, multi-objective, and adaptive e-value settings.","key_machinery":"The central object is the multiple hypothesis testing structure: each hyperparameter $\\lambda$ is attached to a null hypothesis $H_\\lambda : R(\\lambda) < \\alpha$, and a valid $p$-value (e.g., the Hoeffding bound in (3) for average risk) or $e$-value (e.g., the product betting score in (13)) is computed from held-out data. A multiple-testing algorithm then decides which nulls to reject, yielding a subset $\\hat\\Lambda_Z$ with a family-wise error rate bound like (6) or a false discovery rate bound like (9). The extensions reviewed—quantile and mutual-information risk $p$-values, Pareto testing, reliability graphs with DAGGER, and adaptive $e$-processes—all plug into this same testing pipeline, replacing the risk measure or the multiple-testing control.","core_discovery":"Learn-Then-Test (LTT) reframes hyperparameter selection as a multiple hypothesis testing problem: each candidate $\\lambda$ is assigned a null hypothesis $H_\\lambda : R(\\lambda) < \\alpha$ and a valid $p$-value or $e$-value computed from a held-out calibration set $Z$, and a multiple-testing algorithm rejects the nulls that deem hyperparameters unreliable, yielding a subset $\\hat\\Lambda_Z$ guaranteed by (6) to exclude every unreliable configuration with probability at least $1-\\delta$. The paper reviews how this same pipeline extends to quantile risks and mutual-information risks (via dedicated $p$-values), to Pareto-front pruning for multi-objective constraints, to reliability graphs that inject prior knowledge with FDR control, and to adaptive e-value-based testing that can stop early without losing anytime-valid guarantees. The review indicates that the whole framework achieves its goal by ensuring that the selected subset $\\hat\\Lambda_Z$ satisfies the FWER bound (6) or its FDR analogue (9), regardless of the risk measure plugged in.","pith_inferences":["A natural extension the authors only gesture at: applying the same testing pipeline to contextual or instance-dependent risk functions (they mention contextual MHT as future work) could certify reliability for personalized models.","The validity of the whole scheme hinges on the calibration data being i.i.d.; in non-stationary deployment, the guarantee may degrade—a testable extension would be to use distribution-free or anytime-valid bounds for drifting data.","The review's confidence-interval duality (Eq. 8) suggests LTT-style control could be composed with conformal prediction to deliver end-to-end reliability guarantees for predictive systems.","One could empirically compare the conservativeness of FWER versus FDR variants on the same candidate set to decide which guarantee is economically appropriate for a given deployment."],"forward_implications":["If the LTT reformulation is correct, hyperparameter tuning can move from picking the best-looking validation score to certifying that the chosen configuration meets a risk target with probability at least $1-\\delta$.","The extension to quantile and mutual-information risks means safety-performance metrics like 90th-percentile latency or information-theoretic sufficiency can be controlled with the same guarantees, not just averages.","Multi-objective Pareto testing lets an engineer search over trade-offs while still guaranteeing that no selected configuration violates any constraint.","Reliability graphs incorporate prior knowledge (e.g., energy-use ordering in telecom) and enable FDR control, reducing conservatism compared to Bonferroni correction.","Adaptive e-value testing allows data to be collected sequentially and stopping when evidence is sufficient, cutting evaluation cost without losing anytime-valid guarantees."],"supporting_citations":[{"why":"Supplies the fundamental LTT pipeline and the Hoeffding p-value for average risk.","marker":"[1]"},{"why":"Supplies the valid p-value for quantile risk, enabling QLTT.","marker":"[2]"},{"why":"Supplies the p-value for mutual information risk in the information bottleneck setting.","marker":"[3]"},{"why":"Introduces the Pareto-front reduction and combined p-value for multi-objective control.","marker":"[4]"},{"why":"Introduces reliability graphs and FDR control via RG-PT.","marker":"[5]"},{"why":"Introduces e-value-based adaptive testing with anytime-valid stopping.","marker":"[6]"},{"why":"Supplies the theory of e-values and e-processes used by aLTT.","marker":"[18]"},{"why":"Provides the DAGGER algorithm for FDR control on directed acyclic graphs, used by RG-PT.","marker":"[31]"}],"fun_headline_variants":["Hyperparameter selection gets a statistical safety net","Statistical guarantees for picking AI hyperparameters","Learn-Then-Test: certified AI tuning","Controlling errors when tuning AI models","Statistical testing for reliable hyperparameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's guarantees are only as strong as the validity of the p-value or e-value bounds computed from i.i.d. held-out data, so any risk measure without a valid bound—or any calibration set that is not i.i.d.—would void the claimed family-wise or false-discovery control.","fun_headline_variants_meta":{"raw":{"variants":["Hyperparameter selection gets a statistical safety net","Statistical guarantees for picking AI hyperparameters","Learn-Then-Test: certified AI tuning","Controlling errors when tuning AI models","Statistical testing for reliable hyperparameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3025,"prompt_tokens":875,"completion_tokens":2150,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2087}},"tokens_in":491,"tokens_out":2150,"duration_ms":15352,"temperature":1.0,"reasoning_tokens":2087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:08:59.961610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed an unbounded risk function into the Hoeffding-based p-value (3) on i.i.d. data and count the fraction of selected configurations whose true risk exceeds $\\alpha$; if the empirical family-wise error exceeds $\\delta$ at the nominal level, the tested bound is invalid and the guarantee collapses.","supporting_citations":[{"cited_title":"Quantile learn-then-test: Quantile-based risk control for hyperparameter optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the valid p-value for quantile risk, enabling QLTT."},{"cited_title":"Statistically Valid Information Bottleneck via Multiple Hypothesis Testing","cited_arxiv_id":"2409.07325","evidence_quote":"Supplies the p-value for mutual information risk in the information bottleneck setting."},{"cited_title":"Efficiently controlling multiple risks with Pareto testing,","cited_arxiv_id":null,"evidence_quote":"Introduces the Pareto-front reduction and combined p-value for multi-objective control."},{"cited_title":"Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs","cited_arxiv_id":"2501.13018","evidence_quote":"Introduces reliability graphs and FDR control via RG-PT."},{"cited_title":"A sequential algorithm for false discovery rate control on directed acyclic graphs,","cited_arxiv_id":null,"evidence_quote":"Provides the DAGGER algorithm for FDR control on directed acyclic graphs, used by RG-PT."}],"review_version":1}