{"id":"3933dd1b-107b-4a2e-b6f5-7f8184097dee","arxiv_id":"2411.14860","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Stochastically rounding a single pretrained model into several low-precision copies yields a training-free ensemble that improves NLL and calibration on large models.","lead":"This paper proposes making an ensemble of neural networks by stochastically rounding one pretrained model's weights to low-precision values, so each ensemble member is a slightly different copy. It reports that this cheap, training-free ensemble improves predictions and calibration on several large vision and language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Large-model LPE-BSR claim rests on a single post-hoc bit width with no error bars; without robustness across B and noise baselines, 'consistently improves' is underdetermined.","rationale":"The reader's CONDITIONAL verdict is reasonable, and my read does not move it. I focused on the empirical robustness of the headline large-model result rather than the single-basin premise: the method is stochastic, yet Table 3 and Figure 7 report no error bars; the choice of INT-5 for the billion-scale models is not justified as a pre-registered or validation-selected choice; and the paper's own Table 1 documents that the benefit flips sign at lower precision for smaller models. The comparison to Gaussian noise in Table 6 is encouraging but is limited to CLIP-ViT-L/14 and itself lacks error bars. None of this is an internal inconsistency, and the diversity-decomposition mechanism is plausible; the weakness is that the central 'consistently improves' assertion is broader than the evidence currently supports. The proposed concrete test would settle whether the improvement is robust or an artifact of bit-width selection and stochastic variability. I also note that the diversity-analysis equation in Appendix B is corrupted in the provided source, but since the headline NLL/ERR/ECE numbers do not depend on that equation, I did not treat it as the primary concern.","tokens_in":23158,"tokens_out":8260,"duration_ms":103755,"concrete_test":"Rerun the Section 4.5 protocol on CLIP-ViT-G/14 (ImageNet) and LLaMa-3-8B (MMLU) with B in {4,5,6}, S=20, and at least 10 independent stochastic-rounding seeds, reporting mean plus/minus standard deviation for NLL and ECE, alongside the RTN INT-5 single-model NLL and a Gaussian-perturbation baseline with variance tuned on a held-out 1% split. If the best B is not 5, if the improvement over the pretrained checkpoint is within seed noise, or if the Gaussian baseline matches LPE-BSR within error bars, the central claim needs to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim ('LPE-BSR consistently improves upon the pre-trained checkpoint, even for larger models such as CLIP-ViT-G/14 and LLaMa-3', Section 4.5) is supported for billion-scale models only by Table 3 and Figure 7 at INT-5, with no standard errors for a stochastic procedure and no alternative bit widths. This matters because the method's own motivating Table 1 shows the effect is not uniform in B: at INT-4, LPE-BSR is worse than the pre-trained checkpoint for ViT-T/16 (NLL 1.30 vs 0.932) and ViT-S/16 (0.821 vs 0.667). Section 3 declares B=5 'for simplicity', but the text provides no evidence that this choice was fixed before seeing the reported test numbers; if it was selected from the same evaluations, then the 'consistently improves' statement is not a predictive claim. Additionally, Table 6 shows that a simple Gaussian-perturbation ensemble, with variance tuned on the same checkpoint, can reduce CLIP-ViT-L/14 NLL from 0.948 to 0.934, so part of the improvement may be generic averaging of nearby accurate models rather than something specific to low-precision number systems. The load-bearing condition is empirical: the billion-scale gains must persist across neighboring bit widths and seeds and must beat training-free noise baselines at the same scale.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LPE-BSR, a training-free ensemble method that samples ensemble members by Bernoulli stochastic rounding of a single pre-trained or fine-tuned model's weights into a symmetric uniform INT-B number system and then averages the predictive probabilities. Experiments cover ViT-T/S/B/L on ImageNet, CLIP-ViT-L/14 fine-tuned with SGD, SWAG, and IVON, comparisons with SWAG/IVON Bayesian model averaging, combinations with SSE and CSGLD, and zero-shot CLIP-ViT-L/14, CLIP-ViT-G/14, and LLaMa-3-8B. The central claims are that quantization error in low-precision number systems can serve as a source of ensemble diversity and that LPE-BSR improves negative log-likelihood and calibration for large pre-trained models without additional training on downstream data.","tokens_in":23460,"tokens_out":6544,"duration_ms":61203,"significance":"If the empirical claims hold, LPE-BSR is a genuinely cheap way to obtain ensemble diversity from a single checkpoint, which is relevant to the scalability problem of Bayesian deep learning for large models. The paper is an empirical study rather than a theoretical one; it explicitly disclaims Bayesian status for LPE-BSR and relies on the single-basin picture and the accurate-and-diverse condition. Strengths include the breadth of architectures tested, the comparisons with several established baselines, the public code repository, and the honest limitation statement about fake quantization and lack of latency benchmarking. The main risk is that the headline large-model claim rests on a single bit width and on tables without error bars, so the acceptance decision hinges on experimental robustness rather than on any formal guarantee.","major_comments":[{"comment":"The headline results for CLIP-ViT-G/14 and LLaMa-3 are reported only at B=5, with no standard errors or confidence intervals for a stochastic rounding procedure; Table 4 in Appendix C.1 shows that trial-to-trial standard deviations are feasible for the smaller ViT experiments. Section 3 says B=5 is chosen 'for simplicity', but Table 2 shows INT-5 is the best of INT-4/5/6 in the fine-tuned CLIP-ViT-L/14 setting, and Table 1 shows the method is not uniformly beneficial across B (ViT-T/16 at INT-4 degrades from 0.932 to 1.30 NLL). The claim that LPE-BSR 'consistently improves' the pre-trained checkpoint therefore needs results for neighboring bit widths and multiple seeds at the billion-scale models, and the text should clarify whether B=5 was fixed before inspecting the reported comparisons.","section":"Section 4.5, Table 3, Fig. 7"},{"comment":"The comparison with training-free baselines is incomplete. On CLIP-ViT-L/14, a Gaussian-perturbation ensemble with variance tuned on the same checkpoint reaches NLL 0.934, within 0.005 of LPE-BSR's 0.929, and Monte Carlo dropout reaches 0.938, with no standard errors reported for any of these numbers. Because no Gaussian or MCD results are given for CLIP-ViT-G/14 or LLaMa-3, the paper has not established that the improvement is specific to low-precision number systems rather than a generic effect of averaging nearby accurate models; a same-memory-budget comparison with uncertainty estimates is needed.","section":"Section 4.5, Appendix C.4, Table 6"},{"comment":"The blanket phrase 'consistently improves' is metric-dependent. For LLaMa-3, LPE-BSR improves NLL from 1.03 to 0.923 and ECE from 0.160 to 0.087, but classification error worsens from 0.361 to 0.364, and for CLIP-ViT-G/14 the error is unchanged at 0.206. The text should state explicitly which metrics improve and should report accuracy along with NLL and ECE for the large-model results.","section":"Section 4.5, Table 3 (LLaMa-3 row)"}],"minor_comments":[{"comment":"There are several typos that should be corrected: 'commoly' in Section 3, 'star-shapred' in Section 4.1, 'perfomance' in Section 4.3, 'reserach' in Section 4.4, 'vi sualize' in the Fig. 6 caption, and 'fdetails' in Appendix A.","section":"Throughout"},{"comment":"The displayed formula for the generalized ambiguity decomposition is corrupted by stray symbols and is unreadable; the definitions of average loss, ambiguity, and ensemble loss need to be typeset cleanly.","section":"Appendix B, Eq. (9)"},{"comment":"The 'Memory budgets' column would benefit from an explicit formula, such as number of parameters times bits per weight times ensemble size, so that the reader can verify the reported bit budgets.","section":"Appendix C.3, Table 5"},{"comment":"The memory-budget plots use a different x-axis range from the corresponding ensemble-size plots in Fig. 2; unifying the axes or adding a note would make the comparison clearer.","section":"Appendix C.2, Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical short paper with an interesting and potentially useful idea. The central claim about billion-scale models is exactly the part that currently lacks error bars and robustness checks; the B=5 selection concern is the closest thing to a circularity issue and should be addressed explicitly rather than described as 'for simplicity'. I would be willing to accept after the authors add neighboring bit widths, multiple seeds, and training-free baseline comparisons at the large-model scale."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a simple, memorable trick—stochastic rounding of a single pretrained model to make a training-free ensemble—and the paper has enough honest evidence that the core claim is plausible, but the headline large-model results are thinner than the prose suggests. The best parts are the motivating ViT table (Table 1/4), where INT-4 LPE-BSR beats RTN and sometimes the original checkpoint on ViT-B/L, and the fine-tuned CLIP experiments in Table 2, which come with error bars and show LPE-BSR matching or beating SWAG/IVON on NLL when centered on the same MAP. The comparison against a tuned Gaussian-noise baseline (Table 6) is a useful sanity check: LPE-BSR wins, though the Gaussian baseline gets within 0.005 NLL at CLIP-ViT-L/14, so a chunk of the gain looks like generic averaging of nearby accurate models rather than something unique to low-precision rounding.\n\nSoft spots, in order of severity. First, the billion-scale results (CLIP-ViT-G/14, LLaMa-3, Table 3 and Fig. 7) have no error bars for a stochastic procedure, and only INT-5 is shown. The paper's own Table 1 shows the effect is not monotonic in B—at INT-4 LPE-BSR is worse than the checkpoint for ViT-T/S—so 'consistently improves' needs at least a couple of bit widths and multiple seeds at scale. Second, B=5 looks post-hoc: the text says 'for simplicity', but Table 2 shows a clean INT-4/5/6 comparison and B=5 is the sweet spot there. That is a selection-on-the-evaluated-data worry, not a fatal one, because the mechanism is explicit, but the 'consistently' phrasing is stronger than the evidence. Third, the diversity-analysis equation in Appendix B is corrupted in the source I have (mojibake rather than math); the authors need to replace it with the actual decomposition. Also worth noting: all quantization is fake (simulated), which the authors disclose in Limitations, so inference speedups are untested.\n\nIs the central idea solid? For small and medium models, yes, the evidence holds up and the paper is honest that it breaks for small models at INT-4. For the 1B/8B headline, it's plausible but under-supported. The citation pattern looks appropriate—they engage with the low-precision Bayesian works (Ferianc et al., Zhang et al.) and differentiate on scale and training-free setting. No red flags there.\n\nWho's this for? People working on ensembling, uncertainty quantification, and efficient deployment of large models; a reading group could get a decent discussion out of it. I'd send it to review, with the expectation that the authors add error bars and bit-width sweeps for the large models and fix the appendix equation. My own verdict would be a borderline accept/major revision.","headline":"A simple, memorable trick for training-free ensembles via stochastic rounding, with solid small/medium-model evidence but thin large-model headline results; worth reviewing, not yet fully convincing.","tokens_in":23955,"tokens_out":2571,"would_cite":true,"duration_ms":24216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stochastically rounding a single pre-trained model's weights into a 5-bit number system produces an ensemble that improves the original checkpoint without any training.","keywords":["low precision ensembling","Bernoulli stochastic rounding","training-free ensemble","quantization error","ensemble diversity","pre-trained large models","Bayesian model averaging","calibration"],"falsifier":"Independently reproduce the LLaMa-3 result: at INT-5 with S=20 members on MMLU, the paper reports NLL dropping from 1.03 to 0.923 and ECE from 0.160 to 0.087; if a rerun shows the ensemble NLL above the checkpoint's NLL for any large pre-trained model, the central claim fails. The paper's own Table 1 already provides a small-model falsifier: ViT-T/16 at INT-4 gives ensemble NLL 1.30 versus the checkpoint's 0.932, showing the method can fail when rounded members are no longer accurate.","tokens_in":22954,"feed_emoji":"🎲","tokens_out":10947,"duration_ms":93218,"temperature":0.7,"pith_summary":"Ensembling usually requires training several models, which becomes prohibitive for billion-parameter networks. This paper proposes Low Precision Ensembling with Bernoulli Stochastic Rounding (LPE-BSR), which builds an ensemble from a single pre-trained or fine-tuned checkpoint by independently rounding each weight to the floor or ceiling of its value in a low-precision number system, then averaging the predictions of several such rounded copies. The paper's central claim is that the quantization error introduced by this rounding, normally treated as a defect to be corrected, is actually a usable source of ensemble diversity, and that for sufficiently large models the rounded copies remain accurate enough that the ensemble improves over the original checkpoint. The evidence covers vision transformers up to 307M parameters, fine-tuned CLIP-ViT-L/14 compared against Bayesian model averaging baselines, and zero-shot CLIP and LLaMa-3 models up to 8B parameters. If the claim holds, low-precision rounding offers a training-free route to cheaper ensemble uncertainty estimates for large deployed models.","feed_headline":"Rounding one model's weights into 5 bits beats the original","feed_subtitle":"No extra training: rounded copies of one checkpoint outperform it on ImageNet and MMLU.","key_machinery":"The engine of the method is the Bernoulli stochastic rounding distribution: for each weight $w$, the rounded value is $\\lfloor w \\rfloor$ with probability $(\\lceil w \\rceil - w)$ and $\\lceil w \\rceil$ with probability $(w - \\lfloor w \\rfloor)$, where floor and ceiling are taken in a symmetric uniform integer number system with per-channel scales. Repeating draws from this distribution produces $S$ distinct low-precision copies of the same checkpoint, and averaging their predictions implements Eq. 4 as a Monte Carlo approximation of a posterior over weights. The paper verifies the two conditions that make ensembling effective—individual accuracy and collective diversity—using the generalized ambiguity decomposition of Wood et al. (2023), and shows that lower precision increases ambiguity while larger models retain accuracy, which is why the method works best precisely where ensembling is most expensive.","core_discovery":"The central discovery is that quantization error, which post-training quantization schemes spend effort minimizing, can instead be harvested as ensemble diversity. LPE-BSR defines, per weight, a Bernoulli distribution that rounds the weight to its floor or ceiling in a symmetric uniform INT-B system with probability proportional to distance, and averages the softmax predictions of S independent draws. The paper demonstrates that for models above a few hundred million parameters, these stochastic-rounding copies are simultaneously accurate and diverse: individual copies have slightly higher loss than the checkpoint, but their averaged predictions reach lower negative log-likelihood and better calibration than the FP32 checkpoint itself. In a fine-tuning setting, LPE-BSR centered at the same MAP solution matches or exceeds the Bayesian model averaging of SWAG and IVON. For zero-shot pre-trained models, LPE-BSR at INT-5 improves NLL over the checkpoint on CLIP-ViT-L/14, CLIP-ViT-G/14, and LLaMa-3, cutting LLaMa-3's expected calibration error on MMLU from 0.160 to 0.087 at S=20.","pith_inferences":["The paper leaves the number system fixed to symmetric uniform INT; a natural extension is that better low-precision number systems (non-uniform, per-group scales, or learned roundings) would shift the accuracy-diversity frontier further, since the same diversity argument applies to any stochastic rounding scheme inside a flat basin.","The small-model failures in Table 1 (ViT-T/16 at INT-4 raises NLL from 0.932 to 1.30) suggest the method operates in a regime between 'rounding preserves accuracy' and 'rounding creates diversity'; mapping that boundary across architectures and bit-widths would tell practitioners when training-free low-precision ensembling is applicable.","If the flat-basin assumption holds for future architectures too, LPE-BSR-style perturbations could serve as a general cheap alternative to Bayesian marginalization, effectively decoupling ensemble quality from the cost of repeated training runs.","A testable bet implied by the results: Gaussian noise and dropout at matched memory budgets underperform quantization rounding (Table 6), which suggests that the discrete, bounded structure of rounding—not just randomness—is what makes the sampled members land in high-probability regions of the basin."],"forward_implications":["A memory-efficient ensemble of a large pre-trained model can be created from a single checkpoint with no training passes, replacing multi-run deep ensembles or expensive Bayesian posterior sampling.","The memory budget of an ensemble, measured in total bits, can stay below that of a single FP32 checkpoint for practical ensemble sizes, because members are stored in INT-5 rather than FP32.","Fast ensembling methods such as SSE and CSGLD can trade their high-precision snapshots for LPE-BSR samples and obtain better performance at equal training and memory budgets.","In zero-shot settings, calibration improves markedly (LLaMa-3 ECE drops from 0.160 to 0.087 on MMLU at 20 members), making low-precision rounding a candidate mechanism for uncertainty estimation in deployed large models."],"supporting_citations":[{"why":"Introduces Bernoulli stochastic rounding, the technique LPE-BSR uses to generate ensemble members from a single checkpoint.","marker":"Gupta et al., 2015"},{"why":"Supplies the accurate-and-diverse condition the paper uses to explain when low-precision rounding can improve ensembles.","marker":"Dietterich, 2000"},{"why":"Defines SWAG, a Gaussian-posterior Bayesian baseline whose model averaging is compared with LPE-BSR.","marker":"Maddox et al., 2019"},{"why":"Defines IVON, the variational optimizer used as a MAP solver and as a Bayesian-model-averaging baseline in Section 4.2.","marker":"Shen et al., 2024"},{"why":"Provides the generalized ambiguity decomposition used to quantify ensemble diversity (and its average-loss/ambiguity split).","marker":"Wood et al., 2023"},{"why":"Supports the single-basin view of pre-trained loss landscapes that motivates sampling nearby points rather than multiple modes.","marker":"Sadrtdinov et al., 2023"},{"why":"Defines snapshot ensembling, the fast ensembling method LPE-BSR is combined with and compared against in Section 4.4.","marker":"Huang et al., 2017"},{"why":"Frames deep ensembles as approximate Bayesian marginalization, the perspective LPE-BSR borrows for its predictive averaging.","marker":"Wilson and Izmailov, 2020"}],"fun_headline_variants":["Quantization error as diversity: rounded copies beat original","No training: round weights to 5 bits, ensemble beats checkpoint","Low-precision rounding creates ensembles that outperform baseline","Stochastic rounding yields diverse ensembles without extra training","Round to 5 bits, average predictions, beat the original model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pre-trained or fine-tuned model's weights sit inside a broad, flat basin of the loss landscape, so that stochastically rounding them to low precision yields many different models that are each still accurate.","fun_headline_variants_meta":{"raw":{"variants":["Quantization error as diversity: rounded copies beat original","No training: round weights to 5 bits, ensemble beats checkpoint","Low-precision rounding creates ensembles that outperform baseline","Stochastic rounding yields diverse ensembles without extra training","Round to 5 bits, average predictions, beat the original model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1607,"prompt_tokens":867,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":660}},"tokens_in":483,"tokens_out":740,"duration_ms":7954,"temperature":1.0,"reasoning_tokens":660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:47:09.001911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently reproduce the LLaMa-3 result: at INT-5 with S=20 members on MMLU, the paper reports NLL dropping from 1.03 to 0.923 and ECE from 0.160 to 0.087; if a rerun shows the ensemble NLL above the checkpoint's NLL for any large pre-trained model, the central claim fails. The paper's own Table 1 already provides a small-model falsifier: ViT-T/16 at INT-4 gives ensemble NLL 1.30 versus the checkpoint's 0.932, showing the method can fail when rounded members are no longer accurate.","supporting_citations":[],"review_version":1}