{"id":"16c9d6f1-bfb5-45af-b532-0515b9713a1e","arxiv_id":"1908.07380","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Training neural networks with PAC-Bayes objectives yields MNIST test error of 1.4% and a non-vacuous risk bound of 2.3%, much tighter than prior PAC-Bayes certificates.","lead":"This paper trains probabilistic neural networks by directly minimizing PAC-Bayes generalization bounds, reporting MNIST test errors around 1.4% with risk certificates around 2.3%. If those certificates are valid, it suggests a network can be learned and certified from the same training data without a separate holdout set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-dependent prior-scale selection voids the PAC-Bayes certificate behind the claimed 2.3% MNIST bound.","rationale":"I agree with the reader's weakest assumption. The central empirical claim is that PBB provides a valid 2.3% PAC-Bayes certificate for a 1.4%-error MNIST predictor. For that to be true, the prior Q0 must be fixed before the training sample is observed. The paper's Section 6 protocol selects the prior scale and the overall run by minimizing the bound on the training set, which is exactly the kind of data-dependent prior selection PAC-Bayes bounds exclude. Section 6.1's argument about random initialization is insufficient because it does not cover the scale sweep. I also note a related, independently serious ambiguity: the training objectives use a bounded cross-entropy surrogate, so the plotted 'risk upper bound' may upper-bound the surrogate risk rather than the 0-1 error reported in Table 1; if so, 2.3% does not certify 1.4% test error even with a fixed prior. Both issues undermine the self-bounding claim, and the data-dependent prior is the most direct. The proposed rerun with a pre-fixed prior scale and 0-1-loss bound would settle the question.","tokens_in":10190,"tokens_out":10166,"duration_ms":102094,"concrete_test":"Re-run the MNIST pipeline after fixing the prior scale (and all hyperparameters) before seeing any training labels, with a single random initialization, and compute the PAC-Bayes bound for both the bounded cross-entropy loss and the 0-1 loss at the end of training. If the 0-1-loss bound is still below 0.05, the selection concern is refuted; if it is vacuous or above 0.1, the reported tightness is an artifact of the grid sweep and/or surrogate-loss mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported 2.3% MNIST risk bound is not a valid PAC-Bayes certificate because the prior Q0 was not fixed before seeing the data. Section 6 states that \"we performed a grid sweep over all the hyper parameters and then selected the run with the best risk upper bound,\" and the swept hyperparameters include the prior scale (standard deviation values in [0.1, 0.09, ..., 1e-3]), with the posterior variance initialized to the prior variance. Section 6.1 defends only the randomness of the weight initialization as \"data-free\"; it does not address the data-dependent selection of the prior scale or of the run. Theorems 1 and 2 require Q0 to be a fixed, data-free distribution; a Q0 chosen by minimizing the bound on the same training sample is data-dependent and the high-probability statement no longer holds. Even treating each grid configuration as a separate data-free prior, reporting the minimum over many data-dependent runs without a union bound overestimates tightness. The certificate is the central claim, so the 1.4%/2.3% headline is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two PAC-Bayes training objectives for probabilistic neural networks: the objective flamb in Eq. (7), derived from the PAC-Bayes-lambda bound of Thiemann et al., and fquad in Eq. (6), derived from a quadratic inversion of the PAC-Bayes-kl bound. Both objectives are minimized by backpropagation-style gradient estimates. The authors report experiments on MNIST and five UCI datasets, claiming test error around 1.4% on MNIST together with non-vacuous risk upper bounds around 2.3%, which they describe as much tighter than earlier results by Dziugaite and Roy. They argue that these results suggest 'self-bounding learning' with fixed, data-free priors and classical SGD, without a data-splitting protocol.","tokens_in":10400,"tokens_out":6258,"duration_ms":63694,"significance":"If the reported certificates were valid, the paper would be a significant step in PAC-Bayes training of neural networks: it would show that a simple method using classical SGD and fixed priors can produce both competitive predictions and non-vacuous risk guarantees, and it would provide a useful baseline against methods that rely on data-dependent priors or specialized optimizers. The theoretical derivations of Theorems 1 and 2 appear mathematically sound under the stated assumptions, and the connection to the Bayes-by-Backprop objective is informative. However, the central empirical claim depends on the validity of the reported risk certificates, and that validity is undermined by the data-dependent selection of the prior scale and other hyperparameters. The significance is therefore conditional on repairing the experimental protocol.","major_comments":[{"comment":"The reported PAC-Bayes certificate is voided by data-dependent prior-scale selection. The paper states that 'we performed a grid sweep over all the hyper parameters and then selected the run with the best risk upper bound,' and the swept hyperparameters include the prior distribution scale parameter with standard deviation values in [0.1, 0.09, ..., 1e-3], as well as pmin, learning rate, and momentum. Theorems 1 and 2 require Q0 to be a fixed, data-free distribution. A prior scale selected by minimizing the bound on the same training data is data-dependent, so the high-probability statement no longer holds for the selected run. Section 6.1 defends only the randomness of weight initialization as external to the training process; it does not address the selection of the prior scale or the selection of the best run. No union bound over the configuration grid is given. Consequently, the 2.3% bound in Table 1 is not a valid PAC-Bayes risk certificate, and the comparisons in Tables 1 and 2 do not support the paper's headline claim of tighter valid bounds.","section":"Section 6, Section 6.1"},{"comment":"The claim of 'self-bounding learning' is unsupported by the experimental protocol. The abstract explicitly frames the contribution as obtaining competitive test error and simultaneously computing a valid risk bound 'with no need to follow a data-splitting protocol.' That framing relies entirely on the prior being data-free. Since the prior scale and the reported run are selected using the training data, the bound is not a valid high-probability guarantee, and the conclusion in Section 7 that PBB 'output models that come with a performance guarantee' is not justified. A correct protocol would require fixing the prior before seeing the training data, using a separate validation set for hyperparameter selection, or applying a union bound over the grid; none of these is present.","section":"Abstract, Section 7"}],"minor_comments":[{"comment":"There is a typo in the sentence 'there are many other PC-Bayes bounds available'; this should read 'PAC-Bayes bounds.'","section":"Section 3"},{"comment":"The boundedness of the cross-entropy loss is not fully specified. Lower-bounding network probabilities by pmin makes the loss finite but does not by itself place it in the interval [0,1] required by Theorems 1 and 2; if the bound is evaluated for the cross-entropy loss, a normalization step or explicit statement is needed, and if the reported bounds are for the zero-one loss, this should be stated clearly.","section":"Section 4"},{"comment":"The inversion step leading to Eq. (3) uses the inequality kl(p||p') >= (p'-p)^2/(2p'), which is stated to be valid for p' < p, but this side condition is not mentioned in the text surrounding Eq. (3).","section":"Section 6, Eq. (3)"},{"comment":"The paper does not specify how the plotted and tabulated 'risk upper bound' is computed at the end of training: which loss function, which value of delta, and whether the bound is evaluated at the last iterate or at the iterate with the minimal bound. This information is needed for reproducibility.","section":"Section 6.2"}],"recommendation":"reject","confidential_remarks":"The theoretical framework is standard and the derivations of the two bounds appear correct, but the main empirical result is not a valid PAC-Bayes certificate under the described protocol. Because the paper's contribution is primarily the empirical demonstration of tight self-bounding learning, this is a load-bearing issue rather than a local fix. A resubmission that repeats the experiments with a valid protocol—for instance, fixing the prior scale before data access or using a separate selection set—could be considered, provided the claimed bounds survive the corrected procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The only genuinely new thing here is the empirical claim: vanilla SGD with fixed Gaussian or Laplace priors gives a 2.3% PAC-Bayes risk bound on MNIST, an order of magnitude tighter than Dziugaite and Roy. That claim does not survive contact with the paper's own experimental description. Section 6 says plainly that on every dataset they grid swept all hyperparameters and selected the run with the best risk upper bound, and the sweep includes the prior scale. So the prior Q0 is data-dependent: the chosen scale depends on the training data through the bound. Theorems 1 and 2 require Q0 fixed before seeing data. No union bound over grid configurations is applied. The 2.3% number is therefore not a valid high-probability certificate. The stress-test note is right.\n\nCredit where due: the paper is transparent about its setup, which is why the flaw is so easy to see. Theorem 1 is correctly derived from the cited kl-inequality, though it is a few algebraic lines and not really novel. The experiments are clean and the comparison to BBB and SGD is useful. They also honestly report the CIFAR-10 failure, which limits their own self-bounding claim. The theoretical derivations are mathematically sound.\n\nA second soft spot, minor by comparison: the bound applies to the clamped cross-entropy loss, while the reported test error is 0/1 loss. So the claimed tight gap between 1.4% test error and 2.3% bound is not on the same loss. That is a comparability issue, not the main problem.\n\nWho is this for? People working on PAC-Bayes and self-bounding learning. It is a good cautionary example of how easy it is to void a bound via hyperparameter selection. I would not accept it as is, but I would not desk-reject it either. A revision that applies a proper union bound over the grid or uses a hold-out set for prior selection could make the empirical claim solid. As it stands, the central result is unsupported, so my vote is reject.","headline":"The 2.3% MNIST bound is not a valid PAC-Bayes certificate because the prior scale was picked by grid search on the training data; the rest of the paper is correct but standard.","tokens_in":10928,"tokens_out":2647,"would_cite":false,"duration_ms":28383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that backprop-minimized PAC-Bayes bounds on MNIST yield 1.4% test error together with a 2.3% risk certificate, an order of magnitude tighter than previous non-vacuous bounds.","keywords":["PAC-Bayes","probabilistic neural networks","risk certificates","non-vacuous bounds","self-bounding learning","Bayes by backprop","MNIST","stochastic gradient descent"],"falsifier":"Fix every prior hyperparameter, including the prior scale, before seeing the MNIST labels, run the same PBB training, and recompute the risk bound; if the certificate is materially above 2.3%, the reported number depends on data-driven prior selection rather than on the training method.","tokens_in":10001,"feed_emoji":"🧠","tokens_out":10515,"duration_ms":95457,"temperature":0.7,"pith_summary":"This paper is trying to show that a neural network can be trained by minimizing PAC-Bayes risk bounds directly, so that the output of training is not just a predictor but also a valid numerical guarantee on its misclassification risk. On MNIST, the two proposed training objectives—one built on a known PAC-Bayes-lambda bound and one on a new quadratic bound—achieve roughly 1.4% test error while producing a risk upper bound of about 2.3%, an order of magnitude tighter than the previously reported 0.21 certificate. The authors read this as evidence for self-bounding learning: the same training data yields both the model and the certificate, with no separate data split needed. They use classical fixed data-free priors centered at randomly initialized weights and ordinary SGD, in contrast to earlier work that needed data-dependent priors.","feed_headline":"PAC-Bayes training hits 1.4% test error with a 2.3% risk bound","feed_subtitle":"Two backprop objectives turn MNIST training into self-certifying learning, tightening the risk guarantee tenfold.","key_machinery":"The load-bearing objects are two PAC-Bayes upper bounds on the average risk $Q[L]$ of a randomized weight distribution $Q$. The PAC-Bayes-quadratic bound (Theorem 1, Eq. (6)) comes from inverting a binary-KL inequality and solving a quadratic inequality for the square root of $Q[L]$; the PAC-Bayes-$\\lambda$ bound (Theorem 2, Eq. (7)) follows from the inequality $\\sqrt{ab}\\le(\\lambda a + b/\\lambda)/2$ and is minimized by alternating over $\\lambda$ and $Q$. Both bounds are functions of the empirical risk, the KL divergence to a fixed data-free prior $Q_0$, the sample size $n$, and the confidence $\\delta$. To make the objectives trainable by backprop, the paper uses the pathwise gradient estimator to differentiate through samples drawn from Gaussian or Laplace posteriors centered at the randomly initialized weights.","core_discovery":"On the paper's own terms, the central claim is that PAC-Bayes with Backprop (PBB) works: minimizing the two objectives in Eqs. (6) and (7) with SGD produces a stochastic neural network whose average risk $Q[L]$ is both small and certified. The headline numbers are a test-set error near 0.014 on MNIST and a simultaneous non-vacuous upper bound near 0.023, compared with the previous best non-vacuous bound of 0.21 from data-dependent priors; on Binary MNIST the paper reports 0.015 test error with a 0.022 bound. The authors conclude that fixed data-free priors and standard SGD may suffice for self-bounding learning on MNIST, and they report similarly close bound/error pairs on five UCI data sets, while conceding that initial CIFAR-10 experiments gave certificates about three times the test error.","pith_inferences":["A fair re-test that fixes the prior scale before seeing the training data would likely loosen the reported 2.3% value, because the paper's grid sweep selected the best bound over that same data; this is testable and should be done before treating the certificate as valid.","The KL term is the main obstacle on larger networks, so structured posteriors, weight coupling, or hierarchical priors are the natural next levers for extending PBB to CIFAR-10-class problems.","Because the same machinery emits a certificate for any posterior, PBB could be used to fine-tune or compress an already trained network while producing a fresh, valid bound.","The near-equality of bound and test error suggests the bound itself could serve as the training signal for architecture search, provided the prior is kept data-free."],"forward_implications":["If the central claim is right, a single training run on the available data produces both a neural network and a high-probability guarantee on its true risk, so the test set is no longer required to certify generalization.","A certificate near the test error means PAC-Bayes bounds can be used as practical model-selection and early-stopping criteria, not just as theoretical afterthoughts.","On MNIST, the data-dependent-prior machinery used in earlier work is not necessary for tight certificates; plain SGD with fixed priors reaches the same goal.","The UCI and Binary MNIST results suggest the bound/error gap stays small on small and mid-size problems, while the CIFAR-10 gap shows the method has not yet scaled to large natural-image tasks."],"supporting_citations":[{"why":"Supplies the PAC-Bayes-lambda bound used as the first training objective and the alternating-minimization procedure for lambda.","marker":"Thiemann et al. [2017]"},{"why":"Provides the Bayes-by-Backprop objective, the pathwise gradient estimator for differentiating through sampled weights, and the test-error baseline.","marker":"Blundell et al. [2015]"},{"why":"Gives the previous best non-vacuous MNIST bound of 0.21 and the data-dependent-prior method that PBB is compared against.","marker":"Dziugaite and Roy [2018]"},{"why":"Supplies the earlier non-vacuous bound on Binary MNIST and the architecture/experimental setting reused for comparison.","marker":"Dziugaite and Roy [2017]"},{"why":"Underlies the PAC-Bayes-kl theorem from which the quadratic bound in Eq. (6) is derived.","marker":"Langford and Seeger [2001]"},{"why":"Supplies the inequality used to invert the PAC-Bayes-kl bound and derive the quadratic form leading to Theorem 1.","marker":"Boucheron et al. [2013]"}],"fun_headline_variants":["Backprop-trained nets self-certify: 1.4% error, 2.3% bound","PAC-Bayes backprop yields 1.4% error and 2.3% certified risk","Self-bounding learning via backprop: MNIST at 1.4% error, 2.3% bound","Backprop for PAC-Bayes: tight 2.3% bound, 1.4% error on MNIST","Tighter risk bounds from backprop: 1.4% error, 2.3% certificate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 2.3% guarantee holds only if the prior distribution, including its scale, is fixed without using the training data; the reported run was chosen from a grid sweep that picked the best bound on that same data.","fun_headline_variants_meta":{"raw":{"variants":["Backprop-trained nets self-certify: 1.4% error, 2.3% bound","PAC-Bayes backprop yields 1.4% error and 2.3% certified risk","Self-bounding learning via backprop: MNIST at 1.4% error, 2.3% bound","Backprop for PAC-Bayes: tight 2.3% bound, 1.4% error on MNIST","Tighter risk bounds from backprop: 1.4% error, 2.3% certificate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3148,"prompt_tokens":885,"completion_tokens":2263,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2125}},"tokens_in":501,"tokens_out":2263,"duration_ms":15476,"temperature":1.0,"reasoning_tokens":2125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:35:08.722603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix every prior hyperparameter, including the prior scale, before seeing the MNIST labels, run the same PBB training, and recompute the risk bound; if the certificate is materially above 2.3%, the reported number depends on data-driven prior selection rather than on the training method.","supporting_citations":[{"cited_title":"A strongly quasiconvex PAC-Bayesian bound","cited_arxiv_id":null,"evidence_quote":"Supplies the PAC-Bayes-lambda bound used as the first training objective and the alternating-minimization procedure for lambda."},{"cited_title":"Weight uncertainty in neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the Bayes-by-Backprop objective, the pathwise gradient estimator for differentiating through sampled weights, and the test-error baseline."},{"cited_title":"Data-dependent PAC-Bayes priors via differential privacy","cited_arxiv_id":null,"evidence_quote":"Gives the previous best non-vacuous MNIST bound of 0.21 and the data-dependent-prior method that PBB is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the earlier non-vacuous bound on Binary MNIST and the architecture/experimental setting reused for comparison."}],"review_version":1}