Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems

Giung Nam; Juho Lee

REVIEW 3 major objections 4 minor 15 references

Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Stochastically rounding a single pre-trained model's weights into a 5-bit number system produces an ensemble that improves the original checkpoint without any training.

desk verdict A simple, memorable trick for training-free ensembles via stochastic rounding, with solid small/medium-model evidence but thin large-model headline results; worth reviewing, not yet fully convincing. read the letter →

arxiv 2411.14860 v1 pith:MGGRI3XX submitted 2024-11-22 cs.LG

Giung Nam , Juho Lee This is my paper · ORCID

classification cs.LG

keywords lowprecisionensemblingBernoullistochasticroundingtraining-freeensemblequantizationerrordiversitypre-trainedlargemodelsBayesianmodelaveragingcalibration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ensembling usually requires training several models, which becomes prohibitive for billion-parameter networks. This paper proposes Low Precision Ensembling with Bernoulli Stochastic Rounding (LPE-BSR), which builds an ensemble from a single pre-trained or fine-tuned checkpoint by independently rounding each weight to the floor or ceiling of its value in a low-precision number system, then averaging the predictions of several such rounded copies. The paper's central claim is that the quantization error introduced by this rounding, normally treated as a defect to be corrected, is actually a usable source of ensemble diversity, and that for sufficiently large models the rounded copies remain accurate enough that the ensemble improves over the original checkpoint. The evidence covers vision transformers up to 307M parameters, fine-tuned CLIP-ViT-L/14 compared against Bayesian model averaging baselines, and zero-shot CLIP and LLaMa-3 models up to 8B parameters. If the claim holds, low-precision rounding offers a training-free route to cheaper ensemble uncertainty estimates for large deployed models.

What carries the argument

The engine of the method is the Bernoulli stochastic rounding distribution: for each weight $w$, the rounded value is $\lfloor w \rfloor$ with probability $(\lceil w \rceil - w)$ and $\lceil w \rceil$ with probability $(w - \lfloor w \rfloor)$, where floor and ceiling are taken in a symmetric uniform integer number system with per-channel scales. Repeating draws from this distribution produces $S$ distinct low-precision copies of the same checkpoint, and averaging their predictions implements Eq. 4 as a Monte Carlo approximation of a posterior over weights. The paper verifies the two conditions that make ensembling effective—individual accuracy and collective diversity—using the generalized ambiguity decomposition of Wood et al. (2023), and shows that lower precision increases ambiguity while larger models retain accuracy, which is why the method works best precisely where ensembling is most expensive.

What would settle it

Independently reproduce the LLaMa-3 result: at INT-5 with S=20 members on MMLU, the paper reports NLL dropping from 1.03 to 0.923 and ECE from 0.160 to 0.087; if a rerun shows the ensemble NLL above the checkpoint's NLL for any large pre-trained model, the central claim fails. The paper's own Table 1 already provides a small-model falsifier: ViT-T/16 at INT-4 gives ensemble NLL 1.30 versus the checkpoint's 0.932, showing the method can fail when rounded members are no longer accurate.

Watch

Extended reading notes

Core claim

The central discovery is that quantization error, which post-training quantization schemes spend effort minimizing, can instead be harvested as ensemble diversity. LPE-BSR defines, per weight, a Bernoulli distribution that rounds the weight to its floor or ceiling in a symmetric uniform INT-B system with probability proportional to distance, and averages the softmax predictions of S independent draws. The paper demonstrates that for models above a few hundred million parameters, these stochastic-rounding copies are simultaneously accurate and diverse: individual copies have slightly higher loss than the checkpoint, but their averaged predictions reach lower negative log-likelihood and better calibration than the FP32 checkpoint itself. In a fine-tuning setting, LPE-BSR centered at the same MAP solution matches or exceeds the Bayesian model averaging of SWAG and IVON. For zero-shot pre-trained models, LPE-BSR at INT-5 improves NLL over the checkpoint on CLIP-ViT-L/14, CLIP-ViT-G/14, and LLaMa-3, cutting LLaMa-3's expected calibration error on MMLU from 0.160 to 0.087 at S=20.

Load-bearing premise

The pre-trained or fine-tuned model's weights sit inside a broad, flat basin of the loss landscape, so that stochastically rounding them to low precision yields many different models that are each still accurate.

Editorial extensions

If this is right

A memory-efficient ensemble of a large pre-trained model can be created from a single checkpoint with no training passes, replacing multi-run deep ensembles or expensive Bayesian posterior sampling.
The memory budget of an ensemble, measured in total bits, can stay below that of a single FP32 checkpoint for practical ensemble sizes, because members are stored in INT-5 rather than FP32.
Fast ensembling methods such as SSE and CSGLD can trade their high-precision snapshots for LPE-BSR samples and obtain better performance at equal training and memory budgets.
In zero-shot settings, calibration improves markedly (LLaMa-3 ECE drops from 0.160 to 0.087 on MMLU at 20 members), making low-precision rounding a candidate mechanism for uncertainty estimation in deployed large models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves the number system fixed to symmetric uniform INT; a natural extension is that better low-precision number systems (non-uniform, per-group scales, or learned roundings) would shift the accuracy-diversity frontier further, since the same diversity argument applies to any stochastic rounding scheme inside a flat basin.
The small-model failures in Table 1 (ViT-T/16 at INT-4 raises NLL from 0.932 to 1.30) suggest the method operates in a regime between 'rounding preserves accuracy' and 'rounding creates diversity'; mapping that boundary across architectures and bit-widths would tell practitioners when training-free low-precision ensembling is applicable.
If the flat-basin assumption holds for future architectures too, LPE-BSR-style perturbations could serve as a general cheap alternative to Bayesian marginalization, effectively decoupling ensemble quality from the cost of repeated training runs.
A testable bet implied by the results: Gaussian noise and dropout at matched memory budgets underperform quantization rounding (Table 6), which suggests that the discrete, bounded structure of rounding—not just randomness—is what makes the sampled members land in high-probability regions of the basin.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a simple, memorable trick—stochastic rounding of a single pretrained model to make a training-free ensemble—and the paper has enough honest evidence that the core claim is plausible, but the headline large-model results are thinner than the prose suggests. The best parts are the motivating ViT table (Table 1/4), where INT-4 LPE-BSR beats RTN and sometimes the original checkpoint on ViT-B/L, and the fine-tuned CLIP experiments in Table 2, which come with error bars and show LPE-BSR matching or beating SWAG/IVON on NLL when centered on the same MAP. The comparison against a tuned Gaussian-noise baseline (Table 6) is a useful sanity check: LPE-BSR wins, though the Gaussian baseline gets within 0.005 NLL at CLIP-ViT-L/14, so a chunk of the gain looks like generic averaging of nearby accurate models rather than something unique to low-precision rounding.

Soft spots, in order of severity. First, the billion-scale results (CLIP-ViT-G/14, LLaMa-3, Table 3 and Fig. 7) have no error bars for a stochastic procedure, and only INT-5 is shown. The paper's own Table 1 shows the effect is not monotonic in B—at INT-4 LPE-BSR is worse than the checkpoint for ViT-T/S—so 'consistently improves' needs at least a couple of bit widths and multiple seeds at scale. Second, B=5 looks post-hoc: the text says 'for simplicity', but Table 2 shows a clean INT-4/5/6 comparison and B=5 is the sweet spot there. That is a selection-on-the-evaluated-data worry, not a fatal one, because the mechanism is explicit, but the 'consistently' phrasing is stronger than the evidence. Third, the diversity-analysis equation in Appendix B is corrupted in the source I have (mojibake rather than math); the authors need to replace it with the actual decomposition. Also worth noting: all quantization is fake (simulated), which the authors disclose in Limitations, so inference speedups are untested.

Is the central idea solid? For small and medium models, yes, the evidence holds up and the paper is honest that it breaks for small models at INT-4. For the 1B/8B headline, it's plausible but under-supported. The citation pattern looks appropriate—they engage with the low-precision Bayesian works (Ferianc et al., Zhang et al.) and differentiate on scale and training-free setting. No red flags there.

Who's this for? People working on ensembling, uncertainty quantification, and efficient deployment of large models; a reading group could get a decent discussion out of it. I'd send it to review, with the expectation that the authors add error bars and bit-width sweeps for the large models and fix the appendix equation. My own verdict would be a borderline accept/major revision.

Referee Report

3 major / 4 minor

Summary. The paper proposes LPE-BSR, a training-free ensemble method that samples ensemble members by Bernoulli stochastic rounding of a single pre-trained or fine-tuned model's weights into a symmetric uniform INT-B number system and then averages the predictive probabilities. Experiments cover ViT-T/S/B/L on ImageNet, CLIP-ViT-L/14 fine-tuned with SGD, SWAG, and IVON, comparisons with SWAG/IVON Bayesian model averaging, combinations with SSE and CSGLD, and zero-shot CLIP-ViT-L/14, CLIP-ViT-G/14, and LLaMa-3-8B. The central claims are that quantization error in low-precision number systems can serve as a source of ensemble diversity and that LPE-BSR improves negative log-likelihood and calibration for large pre-trained models without additional training on downstream data.

Significance. If the empirical claims hold, LPE-BSR is a genuinely cheap way to obtain ensemble diversity from a single checkpoint, which is relevant to the scalability problem of Bayesian deep learning for large models. The paper is an empirical study rather than a theoretical one; it explicitly disclaims Bayesian status for LPE-BSR and relies on the single-basin picture and the accurate-and-diverse condition. Strengths include the breadth of architectures tested, the comparisons with several established baselines, the public code repository, and the honest limitation statement about fake quantization and lack of latency benchmarking. The main risk is that the headline large-model claim rests on a single bit width and on tables without error bars, so the acceptance decision hinges on experimental robustness rather than on any formal guarantee.

major comments (3)

[Section 4.5, Table 3, Fig. 7] The headline results for CLIP-ViT-G/14 and LLaMa-3 are reported only at B=5, with no standard errors or confidence intervals for a stochastic rounding procedure; Table 4 in Appendix C.1 shows that trial-to-trial standard deviations are feasible for the smaller ViT experiments. Section 3 says B=5 is chosen 'for simplicity', but Table 2 shows INT-5 is the best of INT-4/5/6 in the fine-tuned CLIP-ViT-L/14 setting, and Table 1 shows the method is not uniformly beneficial across B (ViT-T/16 at INT-4 degrades from 0.932 to 1.30 NLL). The claim that LPE-BSR 'consistently improves' the pre-trained checkpoint therefore needs results for neighboring bit widths and multiple seeds at the billion-scale models, and the text should clarify whether B=5 was fixed before inspecting the reported comparisons.
[Section 4.5, Appendix C.4, Table 6] The comparison with training-free baselines is incomplete. On CLIP-ViT-L/14, a Gaussian-perturbation ensemble with variance tuned on the same checkpoint reaches NLL 0.934, within 0.005 of LPE-BSR's 0.929, and Monte Carlo dropout reaches 0.938, with no standard errors reported for any of these numbers. Because no Gaussian or MCD results are given for CLIP-ViT-G/14 or LLaMa-3, the paper has not established that the improvement is specific to low-precision number systems rather than a generic effect of averaging nearby accurate models; a same-memory-budget comparison with uncertainty estimates is needed.
[Section 4.5, Table 3 (LLaMa-3 row)] The blanket phrase 'consistently improves' is metric-dependent. For LLaMa-3, LPE-BSR improves NLL from 1.03 to 0.923 and ECE from 0.160 to 0.087, but classification error worsens from 0.361 to 0.364, and for CLIP-ViT-G/14 the error is unchanged at 0.206. The text should state explicitly which metrics improve and should report accuracy along with NLL and ECE for the large-model results.

minor comments (4)

[Throughout] There are several typos that should be corrected: 'commoly' in Section 3, 'star-shapred' in Section 4.1, 'perfomance' in Section 4.3, 'reserach' in Section 4.4, 'vi sualize' in the Fig. 6 caption, and 'fdetails' in Appendix A.
[Appendix B, Eq. (9)] The displayed formula for the generalized ambiguity decomposition is corrupted by stray symbols and is unreadable; the definitions of average loss, ambiguity, and ensemble loss need to be typeset cleanly.
[Appendix C.3, Table 5] The 'Memory budgets' column would benefit from an explicit formula, such as number of parameters times bits per weight times ensemble size, so that the reader can verify the reported bit budgets.
[Appendix C.2, Fig. 8] The memory-budget plots use a different x-axis range from the corresponding ensemble-size plots in Fig. 2; unifying the axes or adding a note would make the comparison clearer.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: LPE-BSR claims are empirical comparisons against independent baselines, with one non-load-bearing self-citation.

full rationale

The paper makes no formal derivation from first principles; its central claim is empirical: applying Bernoulli stochastic rounding within an INT-B system to a pre-trained or fine-tuned checkpoint, then averaging S predictions, improves NLL, ERR, and ECE relative to the checkpoint and to training-free baselines. This claim is not equivalent to its input by construction. The ensemble predictor in Eqs. 3, 4, and 5 is defined from w_MAP, but the reported improvement is a measured property of the quantized ensemble, not an algebraic consequence of the definition; indeed Table 1 shows the same construction degrades performance at INT-4 for ViT-T/16 (NLL 1.30 vs 0.932), so the outcome is empirically contingent. The bit width B=5 is a declared experimental choice ('for simplicity,' Section 3), not a parameter fitted to the reported test numbers within the paper, and the comparisons to SWAG, IVON, SSE, CSGLD, Gaussian noise, and Monte Carlo dropout are independent external benchmarks. The only self-citation, Lee et al. (2024), appears in a list of prior work on the pre-trained basin and is not load-bearing: the basin premise is also supported by Wortsman et al. (2022) and Sadrtdinov et al. (2023), and the paper's own radial landscape plots provide independent evidence. The ambiguity decomposition is imported from Wood et al. (2023) and is used for analysis, not as the source of the improvement. The stated limitation that fake quantization prevents latency benchmarking concerns deployment measurement, not the validity of the NLL and ECE findings. The absence of standard errors for CLIP-ViT-G/14 and LLaMa-3 at INT-5 and the absence of neighboring-bit-width robustness checks weaken the strength of the 'consistently improves' claim, but that is a soundness and evidence concern, not circularity, because the reported numbers are not forced by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper contributes an empirical method, not a derivation. Its central result depends on the basin assumption around pretrained weights, on the accuracy-diversity principle, and on a prior decomposition identity. The hand-chosen hyperparameters that materially affect the headline results are bit-width B and ensemble size S. No new physical or mathematical entities are introduced.

free parameters (2)

Quantization bit-width B = 5 (INT-5)
Chosen by hand after comparing INT-4, INT-5, and INT-6 in the authors' own diversity analysis (Table 2); B=5 gives the best ensemble-loss values and is then used for all headline large-model results. This is a hyperparameter selected with knowledge of the reported outcomes, not a derived constant.
Ensemble size S = 10 in Table 1, 20 in Tables 2 and 3, 5 per snapshot in Section 4.4
Set by the authors per experiment. The paper shows performance improving with S, so the headline large-model results use the largest tested size. No theoretical or validation-based rule fixes S.

assumptions (3)

domain assumption The loss landscape around a pretrained or fine-tuned MAP solution contains many distinct, individually accurate models reachable by low-precision rounding.
Invoked in Section 1 with the single-basin picture and in Section 4.1 with the accurate-and-diverse condition. If rounding leaves the basin or destroys accuracy, LPE-BSR fails; the paper's own Table 1 shows failure for small models at INT-4.
domain assumption Accuracy plus diversity are sufficient for an ensemble to beat its individual members.
Section 4.1 relies on Dietterich (2000) for this principle, but the paper never proves it holds for the specific neural network ensembles constructed by low-precision rounding.
standard math The generalized ambiguity decomposition of Wood et al. (2023) applies to the logit ensembles used for diversity analysis.
Used in Section 4.3 and Appendix B to split ensemble loss into average loss and ambiguity; the paper borrows this identity from prior literature without reproving it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems." pith.science (2026). https://pith.science/paper/MGGRI3XX

@misc{pith2026241114860,
  author       = {Pith},
  title        = {Pith review of: Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGGRI3XX}},
  note         = {Machine review of arXiv:2411.14860}
}

read the original abstract

While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.

Figures

Figures reproduced from arXiv: 2411.14860 by the authors.

**Figure 1.** Concepts of low precision ensembling. It shows a two-dimensional schematic, where the x and y axes represent the neural network weights, while the contours above visualize the loss surface. (a) Let the pre-trained weights, denoted by a yellow star-shaped marker (☆), be positioned within a basin on the loss landscape. In general, (b) post-training quantization methods introduce lower precision number systems, and the… view at source ↗

**Figure 2.** Comparing low precision ensembling to Bayesian methods. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Comparison between IVON and LPE-BSR samples. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Comparison between snapshot and LPE-BSR samples. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Combining with fast ensembling methods. Negative log-likelihood and expected calibration error for fast ensembling methods, SSE and CSGLD, in terms of training budgets, i.e., the number of backward passes, and memory budgets, i.e., the total number of bits for represe…

**Figure 6.** Figure 6: Radial landscapes for zero-shot CLIP-ViT-L/14 model. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Constructing low precision ensemble of large models. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Comparing low precision ensembling to Bayesian methods. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: Combining with ensembling methods. Negative log-likelihood and expected calibration error for ensembling methods, SSE, DE, and MultiIVON, in terms of training budgets, i.e., the number of backward passes, and memory budgets, i.e., the total number of bits for represen…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

[2]

MCD is particularly relevant as it uses a q(w) form similar to Eq

Monte Carlo Dropout (MCD; Gal and Ghahramani, 2016), which constructs an ensemble by applying the dropout technique during inference. MCD is particularly relevant as it uses a q(w) form similar to Eq. 5 of LPE-BSR, employing δ(0) and δ(w). Table 6 summarizes the results for CLIP-ViT-L/14 with an ensemble size ofS =

work page 2016
[4]

Deep ensembles for low-data transfer learning

Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. arXiv preprint arXiv:2010.06866,

arXiv 2010
[6]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

arXiv
[7]

Ethical and social risks of harm from language models

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359,

arXiv
[8]

URL https://cims.nyu.edu/~andrewgw/deepensembles/. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, ...

work page 2020
[9]

URL https://www.aclweb.org/anthology/2020

As- sociation for Computational Linguistics. URL https://www.aclweb.org/anthology/2020. emnlp-demos.6. Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research (JMLR), 24(359):1–49,

work page 2020
[11]

The evaluation of MMLU was conducted using the template provided in the official repository2, and the computation was based on a micro-average

for LLaMa. The evaluation of MMLU was conducted using the template provided in the official repository2, and the computation was based on a micro-average. B Evaluation metrics Let pi ∈ [0, 1]K represent the predicted categorical probabilities and yi ∈ {1, . . . , K} denote the ground truth label for the ith data point, where the total number of data point...

work page 2023
[12]

Table 5 summarizes our experimental results using the Adam optimizer (Kingma and Ba, 2015)

would be valuable. Table 5 summarizes our experimental results using the Adam optimizer (Kingma and Ba, 2015). We observed that in LPE-BSR, (a) each ensemble member had relatively lower performance (= 0.513). However, (b) due to high ensemble diversity (≥ 0.025), (c) there was a significant improvement in the final ensemble performance. Consequently, it a...

work page 2015

Show all 15 references

[14]

The code is available at https://github.com/cs-giung/lpe-bsr

and Transformers (Wolf et al., 2020), both licensed under Apache-2.0.3 We conducted experiments using TPUv2/v3/v4 cores, with flexibility in selecting the cores based on the memory requirements of each experiment. The code is available at https://github.com/cs-giung/lpe-bsr. E...

2024
[15]

Efforts to develop variational methods for implementing Bayesian inference on neural net- work models have continued over time (Graves, 2011; Blundell et al., 2015)

IVON. Efforts to develop variational methods for implementing Bayesian inference on neural net- work models have continued over time (Graves, 2011; Blundell et al., 2015). However, these at- 3https://www.apache.org/licenses/LICENSE-2.0 18 Table 7: Hyperparameters in SGD and IV...

2024
[2018]

A survey of quantization methods for efficient neural network inference

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630,

arXiv
[2020]

Why are bootstrapped deep ensembles not better? In ”I Can’t Believe It’s Not Better!” NeurIPS 2020 workshop,

Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In ”I Can’t Believe It’s Not Better!” NeurIPS 2020 workshop,

2020
[2021]

Deep ensembles: A loss landscape per- spective

Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv:1912.02757,

1912 arXiv
[2022]

14 A Models and datasets The pre-trained weights utilized in our experiments are listed below. We refer readers to the respec- tive papers fdetails on each model: ViT (Dosovitskiy et al., 2021), CLIP (Radford et al., 2021; Cherti et al., 2023), and LLaMa (Touvron et al., 2023)...

2021
[2023]

Why m heads are better than one: Training a diverse ensemble of deep networks

Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [2]

MCD is particularly relevant as it uses a q(w) form similar to Eq

Monte Carlo Dropout (MCD; Gal and Ghahramani, 2016), which constructs an ensemble by applying the dropout technique during inference. MCD is particularly relevant as it uses a q(w) form similar to Eq. 5 of LPE-BSR, employing δ(0) and δ(w). Table 6 summarizes the results for CLIP-ViT-L/14 with an ensemble size ofS =

work page 2016

[2] [4]

Deep ensembles for low-data transfer learning

Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. arXiv preprint arXiv:2010.06866,

arXiv 2010

[3] [6]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

arXiv

[4] [7]

Ethical and social risks of harm from language models

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359,

arXiv

[5] [8]

URL https://cims.nyu.edu/~andrewgw/deepensembles/. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, ...

work page 2020

[6] [9]

URL https://www.aclweb.org/anthology/2020

As- sociation for Computational Linguistics. URL https://www.aclweb.org/anthology/2020. emnlp-demos.6. Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research (JMLR), 24(359):1–49,

work page 2020

[7] [11]

The evaluation of MMLU was conducted using the template provided in the official repository2, and the computation was based on a micro-average

for LLaMa. The evaluation of MMLU was conducted using the template provided in the official repository2, and the computation was based on a micro-average. B Evaluation metrics Let pi ∈ [0, 1]K represent the predicted categorical probabilities and yi ∈ {1, . . . , K} denote the ground truth label for the ith data point, where the total number of data point...

work page 2023

[8] [12]

Table 5 summarizes our experimental results using the Adam optimizer (Kingma and Ba, 2015)

would be valuable. Table 5 summarizes our experimental results using the Adam optimizer (Kingma and Ba, 2015). We observed that in LPE-BSR, (a) each ensemble member had relatively lower performance (= 0.513). However, (b) due to high ensemble diversity (≥ 0.025), (c) there was a significant improvement in the final ensemble performance. Consequently, it a...

work page 2015